Two representative frontier evaluations have recently examined whether AI can conduct scientific research independently. One is the shadow review experiment conducted by Princeton and other institutions (arXiv:2607.27191v2). Researchers selected two unpublished NeurIPS papers and provided only their core research questions to the AI, allowing it to search literature, write code, run experiments, and ultimately produce a complete paper in an isolated compute environment. The agent had no access to the original papers or the authors’ results; each task was originally capped at 120 hours, and after submitting a draft self-assessed as Weak Reject, researchers added 24 hours to test its upper limit. No guidance was provided for specific research questions, nor was there any off-the-shelf scoring harness to give feedback. In the end, the authors of the original papers reviewed the submissions against real peer-review standards, assigning scores of 2/6 Reject and 1/6 Strong Reject, respectively.
The other is the long-horizon algorithm optimization benchmark conducted by the Prime Intellect team. The setup here was the exact opposite: there was a real-time scoreboard on screen, and the system recorded the performance of 18 different AI models across 153 autonomous long-horizon runs, with the longest run lasting 8.7 days. In this deterministic environment where metric changes could be tracked at any moment, the top-performing Fable 5 system reduced training steps to 2,726 over an 8.7-day run, closing 81.7% of the gap to the human record (2,600 steps, an open PR that had not yet been re-verified under the same validator at the time of the blog post).
On the surface, this appears to be a contrast between a crushing defeat and a breakthrough. Such a contrast easily gives the impression that AI can only game leaderboards in environments with immediate scoring, but is completely lost in genuine open-ended research. Yet looking into Prime Intellect’s public run trajectory repository and the shadow review’s trajectory viewer reveals that both studies point to the same insight. Across Prime Intellect’s 153 runs, the optimization ideas proposed by 18 models overlapped heavily; not a single run produced a fundamentally new method, and all winning ingredients were similar to existing literature. As the research team summarized: The models all find similar ideas. What separates them is how they run experiments. Even underperforming models could propose the correct direction: Grok 4.5 discarded the row normalization modification twice, both times due to a scaling bug it introduced itself. The idea returned to its hands twice, only to be falsified twice by its own implementation errors.
This reveals a frequently overlooked reality: the true bottleneck for AI in research is not an inability to come up with good ideas. Frontier models do not lack inspiration when understanding problems and formulating hypotheses; the real divide emerges after an experiment returns a negative signal: how the system actually treats this failure. In genuine scientific exploration, a failed experiment does not mean the idea itself has reached a dead end. From the moment a negative signal emerges to when it delivers value again in the future, it typically passes through five gates: attribution, scoping, archiving, revival, and composition.
When an experiment falls short of expectations, the first question to answer is attribution: where exactly did things go wrong? In experimental execution, metric degradation typically stems from four fundamentally distinct root causes: the core hypothesis is mathematically wrong; the hypothesis is sound but the code contains a bug; both code and hypothesis are sound but random noise in the measurement environment masks the signal; or the currently paired hyperparameters are unsuitable. On a monitoring screen, these four entirely different situations manifest identically: a drop in numbers. Inexperienced systems are most prone to equating any metric drop with the bankruptcy of the hypothesis itself. Grok 4.5 blamed its implementation flaws on its mathematical hypothesis; meanwhile, in the shadow review’s TabPFN task, after a few failed probes of internal signals, the AI jumped directly to the sweeping conclusion that white-box internal signals do not exist, ultimately submitting a detector that deviated from the problem’s required PFN-specific design.
Following attribution, one must immediately clarify the boundaries of scoping: how broad is the scope of this failure? Prime Intellect offered a clear verdict in their report: A negative result here only tells you about the specific recipe it was tested on. A failed experiment merely indicates that the method does not work under this particular recipe; it does not imply that the entire family of methods is unviable. Precise scoping demands rigorous measurement discipline. In 62 out of ~100 runs at Prime Intellect, models did not blindly trust the official noise estimates and instead wrote scripts to measure the baseline noise of the current environment first—and these self-measured runs were concentrated at the top of the leaderboard. Among them, 42 runs further discovered that even with identical random seeds, rerunning the same recipe still caused loss fluctuations because GPUs are inherently non-deterministic. Consequently, they designed paired comparisons with matching seeds to distinguish faint signals of genuine improvement from noise without inflating compute costs. When encountering ambiguous, borderline results, strong systems tested with 3 seeds instead of 1, paying the validation cost of 8 seeds only when their own noise model judged it worthwhile.
This measurement discipline also informs how we should interpret the leaderboard itself. On Prime Intellect’s public leaderboard and exploration dashboard, two runs using the same model and harness differed by approximately 40–54 steps under aligned criteria, whereas 16 of 18 gaps between adjacent ranks on the main leaderboard did not exceed 54 steps. Furthermore, since most configurations launched at least 3 runs and only continued the best one after 24 hours, the final scores represent extreme values chosen across multiple trials. The leaderboard is essentially a ranking of systems coupling models with harnesses; specific rankings are easily swayed by single-run fluctuations, and only broad tier separations provide stable, meaningful reference value.
The fundamental difference between strong and weak systems lies in what remains after an experiment fails. As the evaluation team observed: What separates the best traces is what an experiment leaves behind. In engineering terms, what is worth keeping are intermediate logs with failure attributions, the measured baseline noise floor, and an explicit ledger of negative results documenting the conditions under which failures occurred.
Such an approach emerged in the mathematical exploration experiment on the Riemann Hypothesis: all 650 mathematical conjectures proposed in the first round were proven unviable. Instead of wiping the slate clean, the system compiled a detailed ledger of 106 candidates, precisely recording under what conditions each conjecture failed and what new prerequisites would justify reactivating it in the future. In the second round of exploration, the system first read this failure ledger, charting new paths with clearly defined known boundaries.
The reactivation of negative results requires explicit prerequisites: when the system’s underlying global environment (recipe) changes, methods that were previously shelved must be brought back for re-testing. Prime Intellect’s long-horizon traces show that after the overall recipe changed, Opus 5 proactively dug up an early tuning attempt that had been deemed ineffective and retested it, this time breaking the system’s all-time best record. Conversely, modifications that were once helpful can also expire: after introducing a new normalization method, K3 proactively ablated and removed two auxiliary modules that had previously provided gains. In complex systems, once the core recipe is adjusted, historical successes and failures must all be reshuffled.
The failure in the shadow review exposed flaws precisely at this step. Over the course of 6 days of exploration, the AI’s internal self-critique explicitly pointed out two fatal flaws in its current scheme within the open-world evaluation logs: features were hand-picked, and the evaluation metric was circularly coupled with the training corpus. But lacking a mechanism to trigger re-evaluation upon recipe changes, the AI locked in its overarching direction with 110 hours remaining before the original deadline; the subsequent 24-hour extension granted by humans also failed to trigger a reopening of the primary route, with work continuing to revolve around patching experiments onto the old draft, adding qualifiers, and rewriting.
Here lies a striking contrast: the system in the shadow review conceded its claims while stubbornly clinging to its draft; in contrast, strong systems in the benchmark directed overturning toward their own written code and tech stack (re-ablating after every merge), while persevering on faint signals and the initial research question. A log that merely records the phenomenon of failure without specifying restart conditions is nothing more than a tombstone, incapable of guiding subsequent research.
Modifications that fail when tested in isolation can yield qualitative leaps when combined. In Prime Intellect’s evaluation, after micro-tuning individual parameters hit a bottleneck, Fable 5 deliberately began searching for parameter pairs that appeared worse in single-variable tests but performed remarkably when combined. Through a joint test of historically marginal modifications, the system secured a substantial improvement of 31 steps. Weaker systems typically judge every minor change in isolation, discarding anything that shows no obvious effect in a single test—a one-at-a-time approach that inevitably misses higher-order solutions requiring synergy among multiple components.
In the shadow review’s Persona Cartography task, when confronted with consecutive negative results, the AI merely narrowed the scope of claims asserted in the paper mechanically, even consuming substantial compute simply replicating negative results across other small models, without ever attempting to piece together the faint clues accumulated earlier for cross-validation. This one-dimensional retreat rapidly collapsed the exploration space, ultimately producing a paper with heavily restricted conclusions. This is precisely why rigorously recording negative results is so crucial: those seemingly unremarkable fragments in the present are essentially waiting for partners that can create synergy with them.
Prime Intellect’s run traces demonstrate that the advanced research behaviors mentioned above do not require step-by-step instruction from human researchers; they have already emerged spontaneously during autonomous exploration by frontier models: some models proactively retested historical parameters, and dozens of runs spontaneously established hardware noise baselines. These behavioral patterns are not the exclusive talent of a few models—at least some of them can be codified and reinforced through engineering frameworks. In terms of engineering implementation, four disciplines can be built into automated rigid runtime result certainty gates:
Still, a core set of judgments cannot be entirely replaced by rigid rules. In open-ended exploration without an off-the-shelf scoreboard, determining whether a flaw is a harmless engineering glitch or a fatal, foundation-shaking error remains heavily reliant on mature human research taste and academic judgment. These trade-offs that cannot be codified into rigid rules form the core moat of scientific discovery. This also points to a clear comparative roadmap for subsequent evaluations: in open-ended research’s shadow review evaluation framework, keep the model itself unchanged and introduce only these four rigid disciplines for managing negative results into the external harness to see if the system can escape the trap of doubling down on flawed directions; while in scored benchmarks, unify testing harnesses, fix trajectory seed counts, and repeat runs to reduce confounding from engineering wrappers and sampling variance.
Evaluating whether a system truly understands how to conduct research cannot be based solely on how many fancy ideas it generates during brainstorming, nor on the score it achieves on exams with known answers. The true watershed lies in what actions the system takes after an experiment produces a negative result, and what legacy it leaves behind from that failure. Systems that can discern the mechanisms of failure and revive old ideas when the time is ripe will pull ahead across long and complex exploratory journeys from those that casually discard failure.