Benchmark & Dataset
Weekly Digest

50 patients, 150 sessions, up to 30 turns each and 18 evaluated models.

Labels 14 issue types with 0/1/5/10 severity and bootstraps results by patient.

It combines realism, longitudinal memory and clinically actionable evaluation.

30 web and 20 game tasks; 20 tasks are public in v1.0.

Unifies click/type/scroll/wait actions, repeats each task three times and reports success plus Pass³.

Best Wall Clock success is 44% and best Pass³ is 36%.

It makes latency part of the environment rather than an after-the-fact metric.

32 apps, 320 cumulative requests and explicit cross-feature dependency graphs.

Agents work on persistent codebases; every round retests new and historical functionality in a browser.

It exposes integration debt hidden by one-shot coding benchmarks.

200 static and 60 dynamic memory cases across 23 semantic categories.

Keeps the first observation as reference and scores identity, geometry, event onset and persistence across revisits.

Leading systems do well on static memory but trigger prescribed events only 31.7% of the time.

It cleanly separates off-screen memory from dynamic evolution.

1,488 instances from 248 reference trajectories at six history lengths.

An executor generates gold states; metrics combine exact match with cell-level update and preservation errors.

Spatial reasoning failures become localizable instead of collapsing into one accuracy number.

278 tasks: 50 airline, 114 retail and 114 telecom.

Combines database state, tool calls and language assertions, counting empty or malformed outputs as failures.

It shows that multilingual agent evaluation requires environment-level localization.