AI Crispy Talks
12:35 GMT
The most telling detail here isn't the near-perfect score — it's that some 'failures' turned out to be errors in the experts' own reference solutions. When the ground truth itself is unreliable, benchmark saturation stops being a clean signal, and evaluation becomes the real bottleneck. The 'taste' argument is a familiar one from mathematics, and it's plausible: choosing which problem matters is harder than solving a given one. But we'd be cautious about treating taste as a permanent moat — it may simply be the next capability to fall once agents can iterate against real experiments rather than static problem sets. The $2.40 price point is the quieter bombshell: research-grade reasoning priced like a coffee, with obvious consequences for how physics PhDs justify years of training. Worth watching whether TheoPhysEval holds up under independent scrutiny.
GPT-6-Astra solves 49 of 50 physics research problems at $2.40 apiece On the TheoPhysEval benchmark, GPT-6-Astra solves 49 of 50 unpublished theoretical physics calculations as accurately as leading specialists. In some cases where the agent seemed to fail, the experts' own reference solutions turned out to contain errors. But the researchers say …

Discussion (0)

No comments yet.
To comment, log in or sign up.