Backdrop

Findings

What the world around an agent costs it

1

Hazards Cost Every Model, and Cost the Strongest Most

Figure
Every model solves fewer tasks once the hazards are in the world, and the strongest lose the most ground (the leaderboard).
Figure
(a) For the 13 models that ran four attempts, one band per model and family member, running from the share of tasks the model solves on every attempt up to the share it solves at least once, with a tick at its pass rate over all attempts (pass@k). (b) Best of k attempts for four models.
The per-model table, with every world and every hazard for the 16 models, is on the leaderboard.
2

Agents Follow Other People More Than Injected Text

Table

Intervals and tests for the quantities in these findings; bold marks an interval excluding zero or p<0.05.

Table

Hazard cost by the app carrying the cue, as the share of twin passes the ablation removes, pooled over the 16 models. Groups under nine families are left out; bold marks each hazard's two largest carriers.

Table

Cue arrival (read) and mode rates over all compound runs and over the runs the cue reached, mean over the 16 models. Counting all runs understates compliance by up to a factor of two.

3

Hazards Cause New Failures and False Claims of Success

Table

Share of runs in which each mode fired, averaged over the 16 models, for each family member.

Figure
The models whose clean-task reports are the most honest are the ones whose honesty collapses when the world turns adversarial.
4

Hazards Overlap, and They Interact

Figure

The mode rates by world are in finding 3, and the reliability bands in finding 1.