Benchmark & Dataset
Weekly Digest

A unified automated framework plus PAIR-IQ pairwise idea judgments for training and validating ranking models.

Measures expert agreement, ranking accuracy and robustness using pairwise rather than absolute scoring.

Idea evaluation is emerging as a distinct AI-scientist evaluation problem.

About 120K QA pairs and 117K images; sequences average 15.14 frames and reach 30 frames, with 30K structured traces.

Twelve task families evaluate temporal understanding plus key-frame and spatial-region grounding.

Structures evaluation around the full sequence-to-grounded-reasoning process.

30K samples across six domains, combining images, business questions, retrieval queries and market evidence.

MLLM judges score six business criteria under blind-rubric and known-rubric settings, including creativity and feasibility.

MBA-b / MBA-k improve 63.9% / 77.1% over caption baselines and 25.6% / 35.8% over multimodal baselines.

Tests a complete multimodal retrieval-to-generation agent pipeline.

56 one-shot tasks derived from 14 incident classes with controlled execution-transport experiments.

Exact final-state validation replays identical model outputs across raw and parser-mediated paths.

Adding a parser can reduce success rates by 55.4–73.2 percentage points.

Shows benchmark score is not automatically equivalent to model capability.

170 real tasks across seven languages; 11.4 files and 261.6 changed lines per task on average.

Expert-audited test suites remove overly narrow or broad tests and filter low-complexity tasks.

The best model reaches about 41.2% resolve rate.

Moves coding-agent evaluation toward multilingual, cross-file repository engineering.

200 long-horizon tasks across ten life domains with 22 mock services, an autonomous clock and silent changes.

Weighted checks inspect the agent's final state, action timeliness and implicit-constraint compliance.

Directly evaluates persistent, proactive and long-running agents.