Benchmark & Dataset
Weekly Digest

Validated on SWE-bench Verified, TerminalBench and Toolathlon.

Jointly measures cost savings, prediction accuracy, resolve-rate disturbance and ranking preservation.

Cuts 13%–26% of agent steps with 89%–97% prediction accuracy and about 1–2 points of resolve-rate movement.

It attacks the per-task runtime cost that increasingly bottlenecks agent evaluation.

2,280 questions across three domains and eight relation types, evaluated on 10 SOTA MLLMs.

Measures question accuracy and the human–model gap, sliced by logical, temporal and spatial relations.

It mirrors how information is organized in real multimodal documents, webpages, tutorials and interfaces.

Eight physical phenomena, controlled real scenes and thousands of generations from six SOTA video models.

Uses a calibration-independent consistency score plus VLM-based physics-violation detection.

No model exceeds 0.42 consistency; the best VLM violation detection is about 67%.

Relative relations avoid dependence on absolute scale, frame rate or camera calibration.

Nine MLLMs, four task families and about 100 episodes, including 20 four-drone command episodes.

Success requires entering a five-meter radius and correctly issuing finished, separating navigation from termination protocol.

Best approaching success is about 65%; Qwen3.5-9B enters the radius 90% of the time but succeeds only 35%.

It shows that reaching the target and correctly declaring completion are distinct abilities.

100K trajectories, 10K held-out queries and 831 bimanual assembly videos for OOD validation.

Combines Recall@20, mAP@20 and nDCG@20 with downstream robot-policy gains.

Recall@20 reaches 0.9964, while hard downstream tasks improve by up to about 59.8 percentage points.

It organizes embodied retrieval around motion structure instead of semantic or visual similarity alone.

About 700K audio clips and 15M captions, averaging roughly 24 descriptions per clip.

Uses human quality checks and CLAP retrieval and zero-shot classification to validate multi-caption sampling.

Multi-caption sampling consistently improves retrieval and zero-shot classification.

It treats multiple valid descriptions as a supervision distribution for multimodal caption datasets.