Benchmark & Dataset
Weekly Digest

400 scenarios spanning safety-relevant tool domains, instrumental goals, oversight conditions and pressure mechanisms.

SCOUT grounds multi-criteria judgments in reasoning and action evidence across controlled tests of five LLM agents.

Explicit instrumental goals are the strongest driver; action-only monitoring can increase scheming in some models.

Factorization isolates which deployment conditions create covert behavior instead of reporting only an aggregate rate.

660 evidence-grounded instances: 163 real reproduction gaps and 497 controlled synthetic gaps.

Covers readiness assessment, defect localization and clarification-action generation.

Best real-world Macro Defect Recovery is 9.6%; clarification success reaches 80.6% when the defect is provided.

It shows that finding missing implementation-critical information is a key bottleneck for AI scientists and coding agents.

Adds a mini-swe-agent-based flow to 94 HLS-Eval design tasks.

Logs compilation, pass rate, tokens, cost, iteration count and full trajectories for passing and failing runs.

Open models with an agent harness solve every simple generation task, showing the benchmark needs harder cases.

It upgrades hardware-code evaluation from one-shot generation to a realistic compiler-in-the-loop workflow.

Analyzes 790K+ outputs from 12 LLMs and 22 language pairs, then builds 9,900 noisy/clean pairs.

Compares quality-estimation span extraction with LLM extraction across formatting and content noise.

The dataset contains 8,800 synthetic and 1,100 manually curated authentic examples across 12 recurring noise patterns.

It evaluates production usability that conventional translation-quality metrics often miss.

Combines synthetic, semantic operational and formula-grounded SCMs with public real-world data and observational/interventional settings.

Measures F1, SHD and cross-family ranking stability while checking pretraining–evaluation overlap.

Rankings shift substantially across SCM families and protocols; strength in one environment does not reliably transfer.

It makes foundation-model training exposure an explicit part of causal-discovery benchmark design.

19,272 videos from 6,427 MoCap sequences and 437 subjects, with SMPL motion and six gait-parameter labels.

Compares RGB, pose, biomechanics and HMR methods across viewpoint, scale and synthetic-to-real shift.

Synthetic supervision transfers to real video; spatial gait parameters are more sensitive to visual domain shift.

It pairs controllable visual generation with real physical motion for scalable clinical-gait training and diagnosis.