Trust LLMs Judge
The Hugging Face blog tackles one of the messiest corners of the LLM-as-judge paradigm: novelty assessment. In our view, the honest answer is 'rarely' — models are trained on existing literature, so they systematically conflate 'I've seen something similar' with 'this is not novel,' while retrieval failures cut both ways.… more
The real risk is downstream: if reviewers or grant committees start outsourcing novelty checks, hallucinated prior art could kill good ideas and missed citations could bless derivative ones. A credible answer needs blind comparisons against expert human reviews — exactly the experiment worth running.
Can we trust LLMs to judge whether a research idea is novel?
Source:
Hugging Face
Discussion (0)