SpatiaLab

Can vision–language models perform spatial reasoning in the wild?

ICLR 2026

1,400 visual QA pairs

6 categories • 30 subcategories

MCQ + Open-ended

SpatiaLab: Can Vision–Language Models Perform Spatial Reasoning in the Wild?

Azmine Toushik Wasi, Wahid Faisal, Abdur Rahman, Mahfuz Ahmed Anik, Munem Shahriar, Mohsin Mahmud Topu, Sadia Tasnim Meem, Rahatun Nesa Priti, Sabrina Afroz Mitu, Md. Iqramul Hoque, Shahriyar Zaman Ridoy, Mohammed Eunus Ali, Majd Hawasly, Mohammad Raza, Md Rizwan Parvez

Computational Intelligence and Operations Laboratory (CIOL) • Shahjalal University of Science and Technology (SUST) • Monash University • Qatar Computing Research Institute (QCRI)

Accepted to The Fourteenth International Conference on Learning Representations (ICLR 2026)

📄 OpenReview 📄 arXiv 🤗 Hugging Face 🤗 HF (Paper) Kaggle GitHub

Spatial reasoning is fundamental to human intelligence and real-world embodied AI. SpatiaLab provides a comprehensive evaluation suite (1,400 QA pairs) across six core spatial categories: Relative Positioning, Depth & Occlusion, Orientation, Size & Scale, Spatial Navigation, and 3D Geometry. It exposes large gaps between SOTA models and humans.

Benchmark Structure and Categorization

Category	Example sub-tasks (5 each)
Relative Positioning	Left/Right, Above/Below, Between, Adjacency, Corner/Angle
Depth & Occlusion	Partial occlusion, Complete occlusion, Layer order, Reflection/visibility, Hidden feature
Orientation	Rotation angle, Facing, Tilt, Tool handedness, Mirror
Size & Scale	Relative size, Scale ratio, Big/Small, Proportion, Size consistency
Spatial Navigation	Path existence, Obstacle avoidance, Turn sequence, Viewpoint visibility, Accessibility
3D Geometry	3D containment, Intersection, Volume ordering, Pose matching, Stability

Data Collection and Annotation

Web crawling, targeted retrieval, and manual snapshots. 3-stage annotation / verification to produce the final 1,400 QA items.

Data pipeline figure (export Figure 3 here)

Task formats & sample instances

Every subcategory includes MCQ and open-ended forms so we can measure discriminative accuracy and generative reasoning capacity. Representative examples below.

Annotation & QC

Annotators were trained; items passed a 3-stage review to produce gold-standard annotations. Fleiss’ kappa among annotators: 0.774; LLM judge agreement: Cohen’s kappa ≈ 0.738 (details in appendix).

Results

Tables below are transcribed from the uploaded paper PDF.

Multiple Choice Evaluation Accuracy (%) on SpatiaLab-MCQ

Model	3D Geom. (#238)	Dep. & Occu. (#259)	Orientation (#202)	Relat. Posit. (#212)	Size & Scale (#252)	Spati. Navig. (#237)	Overall (#1400)
Random Choice	25.00	25.00	25.00	25.00	25.00	25.00	25.00
GPT-4o-mini	47.06	39.00	47.03	47.17	49.60	49.79	46.50
GPT-5-mini	48.74	54.83	60.40	62.74	44.84	56.54	54.29
Gemini-2.0-Flash	47.06	55.21	53.96	58.02	54.37	46.84	52.50
Gemini-2.5-Flash	44.96	48.26	48.02	56.13	42.46	51.05	48.29
Claude 3.5 Haiku	42.44	42.08	46.53	46.23	35.71	45.99	42.93
Mistral Medium 3.1	46.64	49.81	47.52	61.79	41.67	41.77	47.93
InternVL3.5-1B	33.61	32.43	23.27	37.26	31.75	30.80	31.64
InternVL3.5-2B	34.03	31.66	31.68	40.57	32.54	32.49	33.71
Qwen-VL2.5-3B-Instruct	41.18	35.52	46.04	40.09	47.22	39.24	41.43
InternVL3.5-4B	42.86	42.86	42.08	54.72	36.51	42.19	43.29
Gemma-3-4B-it	43.70	34.36	46.53	45.75	37.30	37.97	40.57
Qwen-VL2.5-7B-Instruct	42.86	37.84	42.57	46.23	42.06	35.44	41.00
Llama-3.2-11B-Vision-Instruct	26.47	30.50	20.30	42.92	30.56	32.07	30.50
Gemma-3-27B-it	43.28	40.15	48.02	54.25	48.02	47.26	46.57
Qwen-VL2.5-32B-Instruct	41.18	40.15	46.53	45.28	45.24	41.77	43.21
InternVL3.5-72B	50.00	57.14	53.47	66.04	49.21	54.85	54.93
Qwen-VL2.5-72B-Instruct	47.06	48.65	51.98	54.25	43.65	48.95	48.86
Llama-3.2-90B-Vision-Instruct	46.22	52.12	50.50	58.96	46.83	48.52	50.36
o4-mini-medium	51.26	58.30	54.95	64.15	40.87	51.48	53.21
Gemini-2-Flash-Thinking	37.82	41.31	41.58	45.75	50.40	43.04	43.36
Gemini-2.5-Flash-Thinking	45.80	53.67	52.97	56.60	55.16	53.59	52.93
SpaceOm	42.44	38.61	48.02	37.74	42.86	39.24	41.36
SpaceThinker-Qwen2.5VL-3B	40.34	37.84	47.03	38.21	43.25	37.97	40.64
SpaceQwen2.5-VL-3B-Instruct	31.51	35.14	37.62	37.74	50.79	47.26	40.14
Human Baseline	93.70	74.13	91.58	91.51	88.89	87.76	87.57

Open-ended Evaluation Accuracy (%) on SpatiaLab-OPEN

Model	3D Geom. (#238)	Dep. & Occu. (#259)	Orientation (#202)	Relat. Posit. (#212)	Size & Scale (#252)	Spati. Navig. (#237)	Overall (#1400)
GPT-4o-mini	23.53	16.60	23.27	30.66	17.86	21.94	26.00
GPT-5-mini	45.38	34.75	37.13	49.53	42.46	37.13	40.93
Gemini-2.0-Flash	31.93	24.32	27.23	31.13	26.19	24.47	27.43
Gemini-2.5-Flash	34.03	26.64	31.68	38.68	26.59	29.54	30.93
Claude 3.5 Haiku	26.05	18.92	24.75	25.94	20.24	21.10	22.64
Mistral Medium 3.1	25.21	19.31	21.78	29.25	15.08	16.88	21.00
InternVL3.5-1B	05.88	09.65	09.90	13.68	09.13	10.13	09.64
InternVL3.5-2B	12.18	11.20	10.89	23.58	11.90	18.14	14.50
Qwen-VL2.5-3B-Instruct	15.55	08.49	15.35	10.85	18.25	09.28	12.93
InternVL3.5-4B	19.33	17.76	15.84	19.81	16.27	18.99	18.00
Gemma-3-4B-it	20.17	13.13	14.85	23.58	15.08	19.83	17.64
Qwen-VL2.5-7B-Instruct	15.13	15.83	20.30	27.83	15.87	19.83	18.86
Llama-3.2-11B-Vision-Instruct	16.81	16.99	22.28	25.00	13.49	18.57	18.57
Gemma-3-27B-it	22.69	16.22	24.75	34.43	22.62	21.94	23.43
InternVL3.5-72B	22.69	20.46	20.30	31.60	19.84	26.16	23.36
Qwen-VL2.5-72B-Instruct	26.89	20.85	25.25	30.66	24.60	20.68	24.64
Llama-3.2-90B-Vision-Instruct	22.69	23.17	21.29	28.30	21.83	27.00	24.00
GLM-4.5V-106B-MoE	31.09	20.46	25.25	26.42	24.21	24.47	25.21
o4-mini-medium	40.76	32.82	32.18	42.92	44.05	34.18	37.86
Gemini-2-Flash-Thinking	31.09	27.41	31.19	34.43	29.37	29.54	30.36
Gemini-2.5-Flash-Thinking	37.14	45.45	36.36	37.14	21.74	22.22	32.77
SpaceOm	12.61	06.95	15.84	11.79	18.65	12.24	12.93
SpaceThinker-Qwen2.5VL-3B	13.45	09.27	17.82	10.38	19.44	10.13	13.36
SpaceQwen2.5-VL-3B-Instruct	12.61	03.86	13.86	09.43	11.90	11.39	10.36
Human Baseline	73.53	50.19	70.30	69.81	65.48	62.87	64.93

Key Takeways

Overall Findings:
- SpatiaLab-MCQ:
  - Model accuracies vary from ~30–55% (random choice = 25%), human baseline = 87.57%.
  - Top models include InternVL3.5-72B (54.93%), GPT-5-mini (54.29%), o4-mini (53.21%), and Gemini-2.5-Flash-Thinking (52.93%).
  - Model scale alone is not determinative (e.g., Llama-3.2-11B = 30.50%).
  - Orientation and 3D Geometry subtasks show highest performance (>60% for several models); Spatial Navigation, Depth & Occlusion, Size & Scale are harder.
  - Spatially specialized models (SpaceOm, SpaceThinker, SpaceQwen) achieve mid-40s to low-40s, showing specialization does not guarantee higher MCQ accuracy.
  - Depth & Occlusion and Size & Scale performance varies across architectures, highlighting differences in inductive biases or training data.
- SpatiaLab-Open:
  - Open-ended accuracy ranges from ~9.6% (InternVL3.5-1B) to 40.9% (GPT-5-mini), human baseline = 64.9%.
  - Proprietary and reasoning-tuned models lead (GPT-5-mini 40.93%, o4-mini 37.86%, Gemini-2.5-Flash-Thinking 32.77%).
  - Small open-source models cluster at the bottom (~10–15%), very large models improve modestly (20–25%).
  - Spatial specialists show limited open-ended performance (10–13%), indicating architecture or task specialization alone is insufficient.
  - Orientation and relative position subtasks have comparatively higher performance (GPT-5-mini Rel.Pos = 49.53%, Ori = 37.13%), while depth, size, and navigation remain difficult (<30%).
  - Multi-step grounding and perceptual reasoning are bottlenecks for open-ended generation.
5.2 Performance Drop in Open-Ended Evaluation Compared to MCQ
- Average MCQ→Open-ended gap across 25 models = 23.0% (σ = 5.5%), subtask gaps range 22.89% (navigation) to 24.57% (3D geometry).
- Specialist models exhibit largest gaps (~27%), especially in navigation (up to 36.68%) and orientation (34.44%).
- Reasoning-oriented models have smaller gaps (~19%) and lower variance, reflecting stabilizing effect of instruction-tuning and CoT decoding.
- Negative or near-zero gaps (e.g., Llama-3.2-11B: -1.98 in Depth & Occlusion; o4-mini: -3.18 in Relative Position) indicate MCQ distractors can misrepresent competence.
- Format sensitivity likely arises from MCQ structure, specialization bias, sequential reasoning challenges, and generative calibration differences.
- Recommendation: complement MCQs with open-ended evaluations, audit distractors, use stepwise generation or instruction-tuning, and report per-subtask diagnostics.

Error Analysis

SPATIALAB-MCQ:
- VLMs achieve selective peaks but lack holistic spatial competence .
- Closed-source models reach high scores (e.g., 85.71% in Stacking Orientation) but collapse to near-chance in Relative Size Comparison.
- Open-source scaling improves ceilings (80.56% in Corner/Angle Positioning) but catastrophic failures remain (2.0% in Object Rotation).
- Reasoning-augmented and spatially specialized models add localized gains (occlusion inference, navigation) but remain below 55% in many physical abstraction tasks (Gravity Effects, Stability Prediction).
- Error patterns reveal weaknesses in:
  - Embodied reasoning (Tool Handedness)
  - Recursive relational chaining (Pathway Existence)
  - Non-local cues (Reflective Surfaces)
- Stronger results in Obstacle Avoidance suggest shortcut exploitation rather than robust planning.
- Conclusion: Current VLMs rely on surface correlations and lack stable encodings for orientation, physics, and compositional logic. Progress requires richer training distributions and architectural mechanisms for geometric and reference-frame grounding.
SPATIALAB-OPEN:
- Top closed-source models (e.g., GPT-5-mini: 58.14% in Directional Relations) collapse on tasks like Proximity Gradients.
- Large open-source models show strong Depth & Occlusion performance (e.g., 60%) but fail on Proximity (9.3%).
- Smaller models exhibit isolated strengths (e.g., 26% in Relative Size Comparison) but often break down completely (0% in Betweenness).
- Reasoning-tuned models show promise (Gemini-2.5-Flash-Thinking = 75% in Tool Handedness) yet similar models fail in single digits.
- Specialized spatial models perform worst overall (20%) and frequently fail core relational tasks.
- Failure patterns highlight systemic weaknesses in:
  - Occlusion handling
  - Orientation
  - Multi-step relational chaining
- Partial success on size & scale cues indicates reliance on superficial correlations.
- Root causes: missing spatial analogy representations, architectural limits, brittle reasoning pipelines. Moving forward requires unified geometric encodings, physics-aware reasoning, and embodied data.
Qualitative Error Analysis:
- Failures cluster into recurring classes, not random noise.
- Common error types:
  - Spatial mislocalization: Confusing referents in crowded scenes.
  - Perspective and scale mistakes: Overreliance on object-size priors.
  - Occlusion/ordering failures: Especially with thin or partially hidden structures.
  - Attribute confusion: Mixing perceptual and functional properties.
  - Ungrounded open-ended reasoning: Fluent narratives not grounded in visual input.
  - Multi-cue integration failures: Struggle to combine depth, relative size, and ordering.
  - Poor confidence calibration: Especially in open-ended generative tasks.
- Diagnostic findings: Models ignore minimal decisive visual features, rely on brittle heuristics, and fail to update with counterfactual edits.
- Root causes: insufficient object-centric binding, lack of geometric supervision, training objectives favor plausibility over grounding.
- Conclusion: Current VLMs achieve strong coarse perception but struggle with multi-cue integration and grounded reasoning, highlighting the need for geometry-aware supervision, multi-scale feature retention, and verification pipelines.

Methods

Images collected via web crawling, targeted retrieval, and manual capture; annotations via trained annotators with 3-tier QC. Evaluation uses direct prompting for MCQ and LLM judge with human validation for open-ended items. See paper for exact prompts and appendices.

Evaluation protocol

MCQ: models return option number; automatic comparison against ground truth.
Open-ended: free-form answers judged by an LLM judge and compared against human majority vote (LLM-human agreement reported).
Metrics: accuracy; Cohen’s kappa and Fleiss’ kappa for inter-annotator / judge agreement.

Performance Improvement Approaches

Inherent Reasoning Mechanisms: Models with built-in reasoning consistently outperform baselines across both MCQ and open-ended tasks, with the largest gains in relational and orientation categories (e.g., +13.1% in Relative Positioning for MCQs). While reasoning stabilizes open-ended performance, improvements remain uneven—particularly declining in Size & Scale—indicating that reasoning modules boost logical consistency but do not fully solve grounding or scale sensitivity.
Chain-of-Thought (CoT) Prompting: CoT prompting alone provides limited benefits and sometimes reduces accuracy, with orientation tasks showing the only consistent improvement. Step-by-step reasoning aids directional alignment but struggles on perceptually grounded tasks such as depth, scale, and navigation, occasionally amplifying flawed priors.
Chain-of-Thought (CoT) Prompting with Self-Reflection: Incorporating self-reflection with CoT yields modest gains in MCQ tasks, particularly for geometry and depth, but does not generalize to open-ended tasks.
Supervised Fine-Tuning (SFT): SFT consistently improves MCQ accuracy across all spatial reasoning categories, but transfer to open-ended tasks is limited or negative, suggesting overfitting to task-specific distributions. The drop in generative performance may result from biased internal representations and catastrophic forgetting of linguistic priors during fine-tuning.
AI Agents for Spatial Reasoning: SpatioXolver, a multi-agent system adapted from Xolver, is designed for structured spatial reasoning on images in SpatioLab. Sub-tasks on objects, attributes, relations, or transformations are assigned to specialized agents. Results show strong gains in orientation (+36% open-ended) but declines in depth, occlusion, and navigation.

Citation

Please cite the paper as below:

@inproceedings{
wasi2026spatialab,
title={SpatiaLab: Can Vision{\textendash}Language Models Perform Spatial Reasoning in the Wild?},
author={Azmine Toushik Wasi, Wahid Faisal, Abdur Rahman, Mahfuz Ahmed Anik, Munem Shahriar, Mohsin Mahmud Topu, Sadia Tasnim Meem, Rahatun Nesa Priti, Sabrina Afroz Mitu, Md. Iqramul Hoque, Shahriyar Zaman Ridoy, Mohammed Eunus Ali, Majd Hawasly, Mohammad Raza, Md Rizwan Parvez},
booktitle={The Fourteenth International Conference on Learning Representations},
year={2026},
url={https://openreview.net/forum?id=fWWUPOb0CT}
}