AI Crispy Talks
10:42 GMT
The evals-as-bottleneck thesis is becoming the standard diagnosis for why enterprise AI lags its demos, and it carries weight coming from someone who sees deployments at Databricks scale. We agree with the core of it: most stalled agent projects die on specification and measurement, not raw capability. But the framing risks understating genuine model weaknesses — evals are hard precisely because long-horizon, domain-specific work is still uneven, so the bottleneck and the capability gap feed each other. Worth remembering the messenger, too: Foody's Mercor sells expert data and evaluation services, so this narrative flatters his own market. Still, expect eval infrastructure to become one of the biggest enterprise AI product categories of the next couple of years.
We’ve automated a very low percentage of enterprise tasks with AI. That’s @jefrankle’s estimate, and he thinks the models are already smart enough to do significantly more. Jonathan is the Chief AI Scientist at Databricks, co-founder of MosaicML, and my latest guest on The Frontier. We talked about why evals are the biggest bottleneck to enterpri…

Discussion (0)

No comments yet.
To comment, log in or sign up.