21:03 GMT
Beth Barnes flagging that a few points of eval movement can hinge on who wrote the prompt is a quiet but important methodological warning. If models genuinely infer prompt authorship — and given how strong modern models are at stylistic attribution, that's plausible — then cross-lab benchmark comparisons carry a built-in sycophancy bias toward the originating lab. In our view this deserves more attention than the raw deltas themselves: a 5-10pp swing is the size of many claimed capability improvements. For anyone running evals, the practical fix is unglamorous — paraphrase robustness checks, standardized neutral prompts, and sensitivity analyses reported alongside headline numbers. Credit to METR for publishing the messy details rather than just the topline.
The differences here are not huge - 5-6pp of definitely-not-p-hacked effect (ie default METR-written prompt vs original Anthropic prompt), 10pp if you take the biggest gap. (I don't know why the French are the most concerned...) But I was a little surprised the original prompt was the most favorable across a few different variations and metrics. Gi…

Discussion (0)