ISSUE 09 · 2026 / W41

Benchmark & Dataset
Weekly Digest

DATE RANGE2026.10.05 — 2026.10.11

SELECTED IN THIS ISSUE09 PICKS

BENCHMARK / DATASET08 / 01

01 / OVERVIEW

Selection overview

3Executable agent environments

3D design, urban analysis and rule learning require iterative interaction with feedback.

3Process and evidence audits

Repository policies, evidence topologies and prerequisite questions localize failures.

3Real multimodal usability

UI engineering, spatial prediction and fact checks separate plausible from correct.

ISSUE INSIGHT

Trend insight

Executable environments are becoming standard, while scoring expands from final outputs to process and structure. BrickBench, DUDA-Bench and Learn2Play put agents inside feedback loops; SWE-CC, TopoGraphRAG and SAVU-Bench audit traces, evidence topology and prerequisite failures.

Atomic Tasks + Deterministic Verifiers + Trace Audits

02 / TOP PICKS

Issue 09 selections

Ranked by research value, evaluation innovation and practical transferability

01TOP PICK

BenchmarkAgent × 3D / Physical DesignHigh

BrickBench

arXiv ↗Project / Data ↗

300 text-driven LEGO design tasks jointly evaluate buildability, semantic alignment and design quality.

AUTHOR: Peter Kulits et al.
INSTITUTION: Stanford University, MPI-IS et al.
PUBLISHED: 2026.10.08 (v1)

SCALE / DESIGN

300 prompts across Model, Set and Alt-Build settings.

EVALUATION / METRICS

Combines connection, collision and stability checks with VQA alignment and preference-calibrated Elo.

KEY RESULT

With the environment, leading agents usually build valid assemblies and satisfy up to about 95% of semantic requirements.

WHY IT MATTERS

It closes the loop from executable programs to physical objects and multi-axis verification.

DESIGN TAKEAWAYS

Verify hard constraints first, rank soft quality second, and return localized failures.

02TOP PICK

BenchmarkMultimodal / 3D Spatial ReasoningHigh

SpaceCast-Bench

arXiv ↗Project / Data ↗

3,862 questions across 182 real scenes test observation, spatial transformation and inference about unseen outcomes.

AUTHOR: Hongxing Li et al.
INSTITUTION: Zhejiang University
PUBLISHED: 2026.10.08 (v1)

SCALE / DESIGN

182 scenes, 3,862 questions, 16 task types and 21 evaluated models.

EVALUATION / METRICS

Reports separate accuracy for static perception, local prediction and global prediction.

KEY RESULT

The best model reaches 58.0% versus 87.2% for humans; synthetic-data tuning reaches 65.7%.

WHY IT MATTERS

It turns spatial state updates and hidden-outcome prediction into diagnostic layers.

DESIGN TAKEAWAYS

Anchor and bridge views, layered tasks, and human/random baselines.

03TOP PICK

BenchmarkCoding Agent / Process ComplianceHigh

SWE-CC

arXiv ↗Project / Data ↗

Compiles repository contribution rules into 823 machine-checkable policies and audits 500 extended SWE-bench tasks.

AUTHOR: Hai Dang Truong et al.
INSTITUTION: Singapore Management University et al.
PUBLISHED: 2026.10.05 (v1)

SCALE / DESIGN

12 repositories, 823 atomic policies and 500 end-to-end contribution tasks.

EVALUATION / METRICS

Deterministic checkers audit both execution traces and final artifacts.

KEY RESULT

Even functionally correct agents violate 43.1% of applicable policies; nearly half occur mid-process.

WHY IT MATTERS

It bridges the gap between solving a ticket and producing a mergeable contribution.

DESIGN TAKEAWAYS

Atomic policies, trace auditing and separate functionality/compliance metrics.

BenchmarkAgent / Urban Multimodal AnalysisMedium-high

DUDA-Bench

arXiv ↗

Covers five urban-problem families with 86 atomic tasks and 22 end-to-end workflows across 12 cities.

AUTHOR: Yizhi Song et al.
INSTITUTION: HKUST (Guangzhou)
PUBLISHED: 2026.10.07 (v1)

SCALE / DESIGN

86 atomic and 22 workflow tasks across four analysis stages.

EVALUATION / METRICS

Reports recognition, localization, indicator F1, workflow score and cost.

KEY RESULT

The best workflow score remains below 0.3.

WHY IT MATTERS

It is a strong template for converting expert analysis into hierarchical agent tasks.

DESIGN TAKEAWAYS

Atomic/workflow layers, evidence propagation and cost tracking.

DatasetMultimodal Coding / UIMedium-high

TaoD2C-Bench

arXiv ↗Project / Data ↗

Uses 2,861 production designs and 97,652 expert annotations to evaluate requirement inference, constraint implementation and React generation.

AUTHOR: Chengwei Shi et al.
INSTITUTION: Zhejiang University et al.
PUBLISHED: 2026.10.07 (v1)

SCALE / DESIGN

2,861 designs, 183,697 layers and 97,652 annotations; 421 designs in the public evaluation subset.

EVALUATION / METRICS

Separates requirement prediction, code implementation and visual fidelity.

WHY IT MATTERS

Visual similarity does not guarantee an engineering-correct implementation.

DESIGN TAKEAWAYS

Score requirements, constraints and visuals separately; mark valid alternatives explicitly.

BenchmarkMultimodal RAG / Evaluation InfrastructureMedium-high

TopoGraphRAG-Bench

arXiv ↗Project / Data ↗

Controls single-hop, bridge-chain and multi-source evidence topologies across long, visually rich documents.

AUTHOR: Ruochi Li et al.
INSTITUTION: North Carolina State University et al.
PUBLISHED: 2026.10.07 (v1)

SCALE / DESIGN

201 documents, 2,024 questions and three evidence-topology families.

EVALUATION / METRICS

Scores retrieval, answers, faithfulness and topology-aware step coverage.

WHY IT MATTERS

It evaluates whether systems reconstruct complete evidence paths, not merely retrieve related pages.

DESIGN TAKEAWAYS

Store gold topologies and use counterfactual evidence swaps to detect shortcuts.

BenchmarkAudio-Visual / Spatial ReasoningMedium

SAVU-Bench

arXiv ↗

Pairs a 1,066-question leaderboard with 611 diagnostic questions spanning perception, alignment and reasoning.

AUTHOR: Yu Chen et al.
INSTITUTION: University of Western Australia et al.
PUBLISHED: 2026.10.07 (v1)

SCALE / DESIGN

1,066 main questions, 611 diagnostic questions and 81 spatial-reasoning cases.

EVALUATION / METRICS

Same-scene grounding and alignment prerequisites locate the source of high-level reasoning failures.

WHY IT MATTERS

Real spatial audio remains a major blind spot for multimodal models.

DESIGN TAKEAWAYS

Prerequisite graphs, semantic/spatial negatives and expert-agreement gates.

BenchmarkAgent Learning / Long-Horizon InteractionMedium

Learn2Play Bench

arXiv ↗Project / Data ↗

Twenty unfamiliar or counterintuitive text games separate learning from experience from pretraining memory.

AUTHOR: Yibo Li et al.
INSTITUTION: National University of Singapore
PUBLISHED: 2026.10.06 (v1), 10.08 (v2)

SCALE / DESIGN

20 games; five scored episodes for standard games and ten for challenge games.

EVALUATION / METRICS

Reports peak, mean, gain and slope, then reshuffles instances to test transfer.

WHY IT MATTERS

It intentionally removes memorized-rule shortcuts and exposes iterative learning.

DESIGN TAKEAWAYS

Novel rules, learning curves, instance reshuffling and harness ablations.

BenchmarkMultimodal Fact VerificationMedium

WorldFact-Bench

arXiv ↗

Uses 1,274 source-aligned real/fake image pairs to test open-world factual consistency.

AUTHOR: Zhuohong Chen et al.
INSTITUTION: Ant Group, Tsinghua University et al.
PUBLISHED: 2026.10.08 (v1)

SCALE / DESIGN

1,274 real–fake pairs across four verification mechanisms and ten domains.

EVALUATION / METRICS

Pair accuracy requires both images in a pair to be correct and suppresses label bias.

WHY IT MATTERS

It extends image verification from visual plausibility to real-world consistency.

DESIGN TAKEAWAYS

Paired samples, one verifiable conflict per fake and explicit evidence state.

Benchmark & DatasetWeekly Digest

Selection overview

Trend insight

Issue 09 selections

BrickBench

SpaceCast-Bench

SWE-CC

DUDA-Bench

TaoD2C-Bench

TopoGraphRAG-Bench

SAVU-Bench

Learn2Play Bench

WorldFact-Bench

Benchmark & Dataset
Weekly Digest