Public Benchmarks Over-rated
There's a real kernel here: contamination and distillation on public test sets is a documented failure mode, and once tasks leak into training pipelines, leaderboard gains stop measuring generalization. That said, two things deserve skepticism.… more
First, Mercor sells expert human evaluation, so 'private evals are the true signal' happens to be his business model talking. Second, gaming public benchmarks is hardly one region's specialty — every frontier lab hoovers up anything that leaks — so the selective framing reads more like marketing than analysis. The practical takeaway for you: treat public leaderboard deltas as weak evidence and watch who invests in held-out, expert-graded evals, because that's where the real signal is migrating.
public benchmarks are over-rated
Chinese labs distill the public tasks
Source:
X — Brendan Foody
Discussion (0)