Ai2 has published research showing that large language models used to judge whether a research idea is novel are unstable. In one test, a single change in prompt wording cut one model's accuracy from 93.5% to 40.9% on the same pairs of ideas. The study, titled "Old Ideas, Novel Problems," appeared in late October, and it challenges a growing practice of using LLM judges to score automatically generated scientific ideas.
How the Benchmark Was Built
The team mined novelty judgments from ICLR 2026 peer reviews. They kept passages in which reviewers explicitly praised or disputed a paper's originality, rather than commenting on acceptance itself. The high-novelty pool contained 154 accepted papers from the top tenth, whose reviewers praised novelty without dispute. The lower-novelty pool contained 145 rejected papers from the bottom tenth, whose reviewers raised originality concerns without praising novelty.
Two settings were used. In the Human-Only setting, judges compared ideas drawn entirely from human-authored papers. In the Human+Generated setting, each of the 154 high-novelty ideas was paired with an idea from a simple LLM generator, which was treated as less novel by assumption. Six judges were tested: GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5 and Claude Opus 4.6. They were evaluated on pointwise ratings of absolute novelty and on pairwise comparisons of which idea was more novel. The authors also varied the prompt, reasoning budget, retrieval access, formatting, vote aggregation and generator model.
The Prompt Problem
The most striking result is prompt sensitivity. For GPT-5.4 on Human+Generated pairs, a prompt stating that reviewers found exactly one idea novel reached 93.5% accuracy. A plainer prompt asking only which idea was novel dropped accuracy to 40.9%, below chance, and changed more than half of the verdicts on the same pairs.
The effect varied by model and dataset. On Human-Only data, the same kind of prompt switch moved GPT-5.4 by only about seven points, while Claude Opus 4.6 shifted by about 21 points. There was no universal best prompt: dropping one instruction helped some judges and hurt others. The arXiv paper summarizes the pattern by saying that prompt changes can move pairwise accuracy by more than 50 points, sometimes below chance. A reported result may therefore depend heavily on how the question was worded.
When the authors compared ideas from two AI generators, roughly a quarter to a third of verdicts changed across prompt variations. That suggests the problem is not limited to judging AI ideas against human ones.
Expensive Methods Did Not Win
More compute did not solve the problem. An agentic GPT-5.6 Sol judge with arXiv search scored 0.78 macro-F1, behind Claude Opus 4.6's 0.89, at roughly ten times the cost per decision. A simple Claude Opus 4.6 prompt with majority voting scored 0.854 for a total of $2.59.
Two purpose-built novelty evaluators did not beat that cheapest prompted baseline. Idea Novelty Checker scored 0.642 at $30.66, and the evaluator from Sakana AI's AI Scientist scored 0.772 at $82.05. In a separate expert review of 30 pairs, mostly ones the strongest judge had gotten wrong, the expert preferred the human-written idea in all 30 cases.
Why It Matters
Automated ideation and automated evaluation are increasingly used together. If the judge is unstable, reported novelty gains from idea-generation systems may reflect the prompt as much as the idea. The authors argue that this casts doubt on the robustness of a large body of prior work that relies on LLM judges to evaluate scientific ideas.
The study also shows that retrieval and longer reasoning do not reliably fix the problem. In these tests, more search and more reasoning did not consistently improve novelty judgments. For builders of research agents, the practical lesson is that evaluation prompts are part of the experimental design.
- Report the exact judge prompt alongside any novelty result.
- Test several paraphrased prompts and check whether verdicts agree.
- Compare against a cheap baseline such as a simple prompt with majority voting before paying for a complex pipeline.
- Keep a human expert in the loop for hard cases.
Implications for AI Scientists
The study has immediate relevance for anyone using LLM judges in research. If you are running an experiment in which an LLM rates or filters outputs, the accuracy of that rating is worth testing against prompt variations. A 50-point accuracy swing is not hypothetical; it happens in real benchmarks on real model pairs. The Ai2 team is signaling that a single prompt change can reverse the scientific conclusion.
For example, if an AI system generates a set of research ideas and you use an LLM to filter out the ones reviewers would find redundant, the list you keep depends partly on the judge's wording. This matters because research papers citing filtered results may build their claims on an artifact of the evaluation pipeline rather than a genuine property of the generated ideas.
What Remains Open
The study draws on ICLR 2026 reviews and six judge models, so its findings are strongest for that setting. The authors come from the Allen Institute for AI, the Hebrew University of Jerusalem, Microsoft, the University of Arizona and the University of Washington, and they have released code and data so others can probe the same questions. Whether the same instability appears in other fields, or with newer models, is a question for follow-up work.
Because the authors frame the problem as one of instability rather than outright failure, the most useful next step is independent replication across more disciplines and model generations. Until then, a novelty score from an LLM judge is best read as one noisy signal, not a measurement.
In Practice
For teams using LLM evaluators: test your exact prompt against a paraphrase. If verdicts flip significantly, you need to either stabilize the prompt or add human review. The cost of a wrong filtering decision—keeping bad ideas or dropping good ones—often exceeds the cost of a second look by a person. Simple prompts with majority voting also proved robust, so resist the urge to over-engineer the pipeline. The sophistication did not help here; clear framing and a cheap baseline did.
