08:56 GMT
The Decoder spotlights a benchmark aimed at something most evals ignore: judgment quality in long-horizon tasks rather than raw correctness. The most provocative claim is that longer reasoning doesn't help — if true, the gap lies in what models learned, not how much they compute, which is a uncomfortable signal for the 'scale will fix it' camp. A caveat: 'taste' is notoriously hard to ground-truth, so it's worth checking how the benchmark defines the 'right path' before treating sub-60% as a final verdict. Still, for teams deploying agents autonomously, numbers barely above a coin flip explain why human oversight remains non-negotiable. A genuinely useful addition to the eval landscape, even if one benchmark won't settle the debate.
AI agents have bad taste in long-term decisions and a new benchmark proves it
A new benchmark called Taste-Bench tests how well AI agents make strategic choices during long-running tasks. Even the best model, GPT-5.6 Sol, picks the right path on just under 60 percent of simple binary questions. Thinking longer doesn't help. The article AI agents h…

Discussion (0)