Post-training climbed maths, code and tool use on process data written by humans or synthesised under their direction, and that ladder is running out of rungs. The next rung is research, and research closes the loop: a system that can do research can work on its own training. That makes research plausibly the last capability we have to teach directly. Discovery becomes the bottleneck on intelligence, not an application of it.

01 The ceiling

Post-training has been living off process data someone else already produced.

Post-training gains so far form a ladder. First maths. Then code. Then tool use, the agentic turn. Every rung worked for the same two reasons: a large body of human process data already existed for it, or could be synthesised cheaply under human direction; and the result could be checked automatically.

The post-training ladder: maths, code, tool use, research. The first three rungs are stacks of process data someone else wrote down; the research rung is an empty outline, because the research loop is almost never written down. FIG. 1 · THE LADDER Each rung so far had a corpus to learn from. The next one doesn't. each rung: a longer chain the model closes on its own Maths Code Tool use the loop that produced each result is never written down Research LEAVES BEHIND reasoning traces LEAVES BEHIND tests, compilers, fix history LEAVES BEHIND call traces, failures, retries LEAVES BEHIND almost nothing public The post-training ladder: maths, code, tool use, research. The first three rungs are stacks of process data someone else wrote down; the research rung is an empty outline, because the research loop is almost never written down. FIG. 1 · THE LADDER Each rung so far had a corpus to learn from. The next one doesn't. bar length: the chain the model closes on its own Maths leaves behind: reasoning traces Code leaves behind: tests, compilers, fix history Tool use leaves behind: call traces, failures, retries Research leaves behind: almost nothing public the loop that produced each result is never written down
Fig. 1. Each rung had a corpus someone else had already generated and something could check automatically. The research rung does not. The artefact worth learning from is the whole design → verify → revise loop, including everything that failed, and it is almost never written down.

This ladder is running out of rungs of that kind. Human text, and synthetic text generated under human direction, is not where the next order of magnitude comes from. To keep getting smarter, models will increasingly have to train themselves. A system that trains itself is doing research.

Research ability stops being one capability among many and becomes the constraint on all the others. It also differs from them in kind. A system that can run the research loop can point it at its own training, which makes research plausibly the last capability we have to teach directly.

That has not happened yet, as the next section shows. But every capability so far has had to be handed over. Research is the one that would end the handing over.

02 The gap

Agents look like they can already do research. Every benchmark that isolates it says otherwise.

Frontier agents write training code, launch runs, read checkpoints and report numbers. It has the shape of automated research. Measured directly, it isn't yet.

Measured directly Three benchmarks that isolate research ability.
BenchmarkScoreWhat was measured
PostTrainBench 23.2% best agent post-training a base model end to end, against 51.1% for the official instruct model [1]
AI4AI-Bench 0.250 best system rewriting a training algorithm, with hyperparameter tuning factored out (mean 0.166) [3]
ASI-Bench 26.6 project-level research score once the method is no longer given, down from 50.9 with full guidance [6]

The last is the most diagnostic. ASI-Bench[6] progressively withdraws human methodological guidance inside the same project: the method is supplied, then only named, then left for the agent to choose. The score falls by almost half as the guidance is withdrawn, step by step. The missing half was the human's.

What today's models have is engineering strength, not research strength. An analysis of a large corpus of released post-training trajectories[2] separates the two. Execution-level capability is iterating inside a chosen strategy; strategy-level capability is revising that choice as evidence arrives. Agents are strong at the first. At the second, the strategy is locked in at the start of the run and the entire remaining budget goes to local adjustment inside it.

That pattern is also what agents have been scoring on. Earlier suites could be won by collecting more data or tuning hyperparameters, and none of them separated a change in how a run is executed from a change in how the model learns. AI4AI-Bench takes tuning off the table: the agent has to rewrite the training algorithm itself, and the rewrite is rerun from scratch under an evaluator the agent never sees. The best system closes only a small fraction of the distance between the existing algorithm and the optimum.[3]

The same analysis tries the three obvious fixes, and all three fail. Experience-driven scaffolding lifts execution sharply and leaves the strategy exactly where it was. Human guidance redirects the strategy, and the agent falls back into local adjustment as soon as training starts. More inference compute helps on easy tasks and buys almost nothing on the hardest. What is missing is not experience, guidance or reasoning compute. It is a mechanism for spontaneously re-evaluating the strategy mid-run.

An environment would have to teach exactly that, and it is not "write better code". It is: notice, halfway through, that the plan is wrong, and change it. It cannot be learned from write-ups, because a write-up is the format from which the abandoned plans have been removed.

The environment has to be a real one: live scientific repositories, executable pipelines, genuine failure. The loop closes on itself. Models need science to keep improving, and the improving is itself done by doing science.

03 The other direction

AI is already accelerating the rest of science.

The argument so far is internal to AI. The second motivation points outward, and 2026 made it concrete.

In mathematics, OpenAI reported a proof that solutions to the Navier–Stokes equations can blow up in finite time: roughly ten thousand agents over 88 hours, plus a further 17 hours for a model to formalise the result.[7][8] In biology, Anthropic reported that Claude produced optimised implementations of 36 open-source biomolecular models in under four weeks, including a new set of triangle-attention kernels, for about a 4× average speed-up.[9] The supervisors were two people with no prior kernel-engineering background.

Every lab running those models now waits a quarter as long: a compounding gain across an entire field, from a capability AI already has. Science is full of bottlenecks shaped like that.

04 The design

Recursive self-improvement is the right shape. The verifier closes it. A world model makes the verifier affordable.

Human judgement is the scarcest input in the research loop. Every step that requires it caps the throughput of the whole system; that is the real content of the ASI-Bench result above. The design target is a loop that runs with as little human intervention as possible.

Recursive self-improvement is the right shape for a boring reason: an RSI loop needs two components, not an agent that is already good at the domain. A proposer generates the next hypothesis. A verifier decides whether the last one worked. "Accumulate experience, including failures" is an empty instruction unless something can tell the loop that an attempt failed, and ideally why. Failures carry as much information as successes, often more — but only once something has called them failures. The verifier is not an implementation detail of the loop. It is what makes the loop a loop, and its verdicts are the only signal the proposer is ever trained on. A system built this way should get better at the task the longer it runs. The proposer is allowed to start out poor. The verifier is not.

The shape has already been built. FunSearch, published in Nature, paired a language model that writes programs with an evaluator that executes and scores them, discarding the worse and feeding the better back into the next prompt; the loop found the largest known cap sets.[10] AlphaEvolve, by DeepMind's own account, runs the same loop with an ensemble of models and has improved on Strassen's algorithm in a case where it had been the best known.[11] Both loops closed because the verifier was a program: the verdict on a candidate is exact, and it costs a single execution.

The loop has an obvious problem: its throughput is set by its most expensive edge, and that edge is verification. Outside maths and code the verifier is rarely a unit test. It is reality itself: a training run, a simulation, an assay. An agent can propose ten thousand protein designs in an afternoon. Verifying ten thousand designs at the bench is not an afternoon, may not be a year, and may not be affordable at all. The proposal side scales and the verification side does not, and the surplus proposals are wasted.

The recursive self-improvement loop with a world model on its verification edge: a proposer generates ten thousand cheap candidates, a world model screens them down to ten survivors, the real verifier tests those against reality, and evidence comes back. Three return edges: evidence revises the candidate, evidence updates the proposer itself, and ground truth sharpens the world model. FIG. 2 · THE LOOP AND THE FILTER One loop. The world model sits on its costly edge; the verdict returns to the proposer itself. Proposer generates the next hypothesis proposes 10⁴ candidates cheap to generate, worthless if unscreened World model cheap stand-in for the verifier: need not be right, only better than chance 10¹ survivors Verifier ground truth: execution, simulation or assay slow · costly · final did it work? Evidence results, and every failure ordinary loop: revises the candidate RSI loop: updates the proposer itself every real measurement sharpens the stand-in The recursive self-improvement loop with a world model on its verification edge: a proposer generates ten thousand cheap candidates, a world model screens them down to ten survivors, the real verifier tests those against reality, and evidence comes back. Three return edges: evidence revises the candidate, evidence updates the proposer itself, and ground truth sharpens the world model. FIG. 2 · THE LOOP AND THE FILTER One loop. The world model sits on its costly edge; the verdict returns to the proposer itself. RSI loop: evidence updates the proposer itself Proposer generates the next hypothesis proposes 10⁴ candidates cheap to generate, worthless unscreened ordinary loop: revises the candidate World model stands in for the verifier: need not be right, only better than chance 10¹ survivors Verifier ground truth: execution, simulation or assay slow · costly · final every real measurement sharpens the stand-in did it work? Evidence results, and every failure
Fig. 2. Everything the proposer learns arrives as the verifier's verdict, and the question is where that verdict goes. An ordinary experimental loop returns a revised candidate. An RSI loop returns a revised proposer, which is exactly the strategy-level revision that today's agents were found not to do[2]. The verifier is also the loop's expensive edge, and the world model sits on that edge alone: it stands in for the verifier on most candidates, turning an unaffordable verification problem into an affordable one, and the ground truth that does get measured flows back to sharpen the stand-in. In ML research the analogue is predicting a run's outcome without paying for the run[12][13]; in biology the canonical instance is the virtual cell[14].

A world model earns its place on exactly that edge. It is not a second mechanism alongside the verifier. It is a cheap approximation of the verifier: a fast in-silico predictor of what the real verification would return, used to triage which candidates are worth spending the real verifier on. It does not have to be right. It has to be cheap and better than chance.

Biology shows what this looks like taken seriously, because there the verifier is a wet-lab assay. Arc Institute's virtual-cell programme predicts what a cell does under a perturbation it has never seen, so millions of perturbations can be screened before anything is pipetted.[14] Their 2026 challenge raises the bar to zero-shot prediction across unseen cell lines.[15] It is the right bar: a stand-in that only works on contexts the real verifier has already measured buys no new science. The same structure recurs inside AI research, where the verifier is a training run: a persistent model of the research process predicts what the run would show, so compute goes to the experiments worth running.[12][13]

The two pieces of the design are not peers. The verifier closes the loop, and because its verdict is the only signal the proposer learns from, it is the objective the proposer is optimised against. The world model is conditional. It matters where the verifier is too expensive to run on every candidate; there it stands in on most of them, and each turn of the loop becomes cheap enough for the loop to improve often.

05 What matters next

01 Challenge Scale the scientific environments, not just the scientific data

If research ability is the constraint, the direct lever is agentic data and environments drawn from real science. ScienceIDE[16] names the obstacle a scientific experience bottleneck: repositories encode decades of method, but fragmented toolchains, implicit domain conventions and specialised correctness criteria keep that knowledge from becoming learnable experience. Their answer is to convert repositories into executable environments with expert-defined acceptance criteria, then train on verified trajectories.

They also report transfer: scientific experience improved not only held-out scientific-code repair but general code, reasoning and knowledge benchmarks. If that holds at scale, science is not a vertical application of foundation models. It is a training signal for them. The domain benchmarks show how much headroom is left: 17% on real bioinformatics analyses in BixBench[17], and the guidance-withdrawal collapse in ASI-Bench[6].

02 Challenge Verifiers robust enough to be optimised against

Once an agent can search against a verifier, the verifier is the objective. Yukon[18] shows the upside: open autoresearch challenges in quantum circuit optimisation, proof-system acceleration and GPU kernels, where every submission must clear a deterministic gate before it is merged, and the frontier ratchets. Its ECDSA.fail challenge reports beating the Google Quantum AI benchmark by 62%.

The downside arrives the moment the gate is weaker than the optimiser. KernelBench-Verified[19] documents frontier models exploiting the narrow test distribution, in one case with a "ReLU kernel" that checks whether the input shape matches the test configuration and returns the input unchanged. Gimlet Labs put a number on it: KernelBench overstates correctness by 31%.[20] On a benchmark that is embarrassing. In a live RSI loop it is worse than embarrassing, because the false result becomes training signal.

Fuzzing does not close this, for the reason the same write-up gives: testing can only examine a finite subset of an effectively infinite space, and the agent is optimising against whatever harness you chose. Formal verification has to move into the verifier itself. The pieces exist. Symbolic equivalence checking of generated kernels via SMT proved 16 of 26 Triton kernels and caught two that passed numeric tests while being mathematically inequivalent[20]. In repository-scale joint implementation-and-proof synthesis, the best agent still only closes 27 of 43 instances[21]. There are machine-checked proofs of the properties production systems actually rely on, such as permutation equivariance and correctness of tiling[22]. None of them is ready to sit inside a loop running millions of proposals. That gap is the problem.

03 Challenge Domain world models where feedback is genuinely expensive

In fields where a single round of feedback costs weeks and real money, wet-lab biology among them, the world model is not an optimisation. It is the precondition for running the loop at all. The argument in §04 takes concrete form here: a research world model for the research process itself[12], virtual cells for perturbation biology[14][15], and predictive architectures general enough to be retargeted across domains[23]. The open question is calibration, not accuracy: a filter is useful when you know where it is trustworthy, and dangerous when you don't.

04 Challenge Learning efficiently from the loop's own failures

In an unfamiliar environment an agent's first attempts are bad, and most of what the loop produces is failure. An RSI system earns its name only if that exhaust becomes capability. The design question is what, exactly, gets updated.

CAKE[24] is the clearest worked example. It treats the compiler as part of the agent harness under evolution, not a fixed black box. Recurring failures are promoted into verifier rules, IR primitives, model calibrations and reusable tactics: "shrink the search space using what went wrong", precisely. A parallel line of work makes the same move on the model–harness pair. One alternates weight and harness optimisation[25]. Another evolves the harness, retaining matched successes, regressions and near misses as verified sibling trajectories for the next model update[26]. A third composes typed operators over data, harness and weights under one loop kernel[27].

What I take from these is that the unit of self-improvement is not the model. It is the whole system: model, harness, verifier, search space. The open problem is how to co-evolve them without the loop quietly optimising one of its own measuring instruments.

References

  1. PostTrainBench: Can LLM Agents Automate LLM Post-Training? arXiv:2603.08640 · ICML 2026
  2. Lim, Huang, Peng, Lu, Cong, Zhang, Sun, Lin. What is Missing from AI Post-Training AI: An Empirical Analysis. arXiv:2608.19072
  3. AI4AI-Bench: Benchmarking LLM Agents in Algorithmic Design for Recursive Self-Improvement. arXiv:2608.20318
  4. Terminal-Bench-Science 0.1. tbench.ai
  5. SWE-bench Science: Can Coding Agents Resolve Engineering Tasks in Science? arXiv:2608.19799
  6. ASI-Bench: At the Dawn of Artificial Superintelligence. arXiv:2608.17271
  7. OpenAI. On the Navier–Stokes Millennium Prize Problem. openai.com, Sept 2026
  8. OpenAI claims huge maths breakthrough on a famed 'Millennium Problem'. Nature, 2026
  9. Anthropic. How Claude is uplifting biomolecular modeling. anthropic.com, Sept 2026
  10. Romera-Paredes, Barekatain, Novikov, Balog, Kumar, Dupont, Ruiz, Ellenberg, Wang, Fawzi, Kohli, Fawzi. Mathematical discoveries from program search with large language models. Nature 625, 468–475, Dec 2023
  11. Google DeepMind. AlphaEvolve: A Gemini-powered coding agent for designing advanced algorithms. deepmind.google, May 2025
  12. AI Should Build Its Own Research World Model. Agent Native Research
  13. Scaling Automatic Research Agents via World Models. arXiv:2608.12564
  14. Arc Institute. Virtual Cell Initiative. arcinstitute.org
  15. Arc Institute. The 2026 Virtual Cell Challenge. arcinstitute.org
  16. ScienceIDE: Turning World's Scientific Codebase into Agent Learnable Environments. arXiv:2609.19134
  17. BixBench: A Comprehensive Benchmark for LLM-based Agents in Computational Biology. arXiv:2503.00096
  18. Yukon — the platform for open frontier research. yukon.org
  19. KernelBench-Verified: Do LLM-Generated Kernels Actually Beat PyTorch? arXiv:2607.16241
  20. Taneja, St John, Serrino. Formally Verifying AI-Generated GPU Kernels. Gimlet Labs
  21. Vero: Can AI Agents Build Formally Verified Software Repositories? arXiv:2608.13522
  22. Rush, S. Lean Verified Transformers. srush.github.io
  23. JEPA-Anything: Learning Predictive Models across Different Worlds. arXiv:2609.20800
  24. CAKE: Compiler–Agent Co-Design for Frontier Kernel Evolution. arXiv:2608.12629
  25. WHALE: A Simple Recipe for Joint Harness-Weight Optimization. arXiv:2609.00196
  26. HELIX: Model–Harness Co-evolution for Recursive Self-Improvement. arXiv:2608.13951
  27. MetaRSI-v1: A Meta-Recursive Self-Improving System for Recursive Self-Improving Systems Themselves. CosmosMind, Sept 2026

Citation

Cited as:

Zhang, Shuibai. (Sep 2026). Scientific discovery is the next bottleneck for intelligence. Shuibai Zhang. https://zhangshuibai.github.io/blog/posts/scientific-discovery-is-the-next-bottleneck-for-intelligence/.

Or

@article{zhang2026discovery,
  title   = "Scientific discovery is the next bottleneck for intelligence",
  author  = "Zhang, Shuibai",
  journal = "zhangshuibai.github.io",
  year    = "2026",
  month   = "Sep",
  url     = "https://zhangshuibai.github.io/blog/posts/scientific-discovery-is-the-next-bottleneck-for-intelligence/"
}