: speedrunning SWE-bench

Over the past three years, SWE-bench has served as a north star for the development of coding agents. With ever larger training budgets, frontier models now essentially solve the benchmark.¹ We instead ask: how far can we go with just $1000 of train compute?

Our starting point is Andrej Karpathy's nanochat. We add support for long-context training, and swap the train data for Qwen Coder trajectories from existing post-training datasets.² No train data comes from any of the SWE-bench codebases; they are strictly held out. We then train models from randomly initialized weights, with train compute ranging from $10 to $1000. Dollar figures throughout refer to train compute only.³

Surprisingly, even the $10 model⁴ achieves non-zero pass@1 (0.53%)⁵. Our $60 speedrun⁶ achieves 5.0% pass@1⁷, matching Claude 2 (4.4%), the state of the art at SWE-bench's release. Solve rate continues to improve roughly log-linearly with train compute (Figure 1), reaching 11.0% at $1000⁸, landing between Claude 3 Haiku (7.2%) and Claude 3 Opus (15.8%). All from randomly initialized weights!

Is it fair to compare our speedruns against Claude 2 and Claude 3? No, for two reasons. First, our speedruns train on hundreds of thousands of bug-fixing trajectories, whereas the Claude models likely did not target SWE-bench as much.⁹ Second, the Claude models use orders of magnitude more training compute than we do, with our $60 speedrun likely using 1000x fewer training FLOPs than Claude 2.¹⁰ Thus, the comparisons are unfair along two dimensions: test-task training and scale [4]. That our tiny speedruns can compete with Claude 3 is striking, and speaks to the power of targeting specific capabilities at train time.¹¹

That power is precisely what we set out to study: how far can $1000 of train compute ultimately take us when aimed solely at SWE-bench?

Speedrunning & autoresearch

In a speedrun, participants compete (or collaborate) to maximize performance within a fixed budget [6–9].¹² Speedruns give researchers a shared experimental setup in which to compete, with the current record serving as a strong baseline. Because the training budget is fixed, simply out-scaling the competition isn’t an option; any advantage has to come from somewhere else.

Though training compute is capped at $1000, we deliberately impose no other budget constraints. This leaves several promising avenues for improvement. One is data, specifically, synthesizing trajectories that best elicit SWE-bench capability. Such synthesis should likely be student-aware [13,14], since our student is maximally weak. Another axis of improvement is inference-time compute. Finally, we place no cap on experimentation itself: researchers can test as many ideas as they can afford. This makes autoresearch agents particularly compelling, as they can massively scale the experimentation loop.

We expect nanoswe to be a challenging setting for autoresearch. At $1000 per run, blindly trying ideas and keeping what sticks quickly becomes unaffordable. A successful agent must instead create faithful smaller-scale setups that support inexpensive iteration. This is substantially harder for SWE-bench than for perplexity-based environments. Scores on the Verified test set sit near zero. Worse, extrapolating solve rate from smaller-scale experiments is notoriously challenging. Data remains the most promising axis for improvement, but synthesizing tens of billions of tokens is itself too expensive to iterate on carelessly. These challenges make nanoswe a natural testbed for evaluating how efficiently autoresearch systems convert experimentation budget into performance gains.

Just distill from stronger models?

Does SWE-bench speedrunning boil down to picking the strongest model available and simply distilling from it? Not necessarily. We show as much by swapping our teacher model, Qwen Coder 30B A3B (29% pass@1¹³), for the much stronger Qwen 3.6 35B A3B (58% pass@1).

We use Qwen 3.6 to synthesize about 1.3B tokens, in identical fashion to our Qwen Coder pipeline. As expected, the new dataset contains a much greater share of correct trajectories: 49%, versus 22% for our Qwen Coder dataset. We then re-train the 12 B200-hour speedrun, replacing mini-coder-trajs with the newly generated Qwen 3.6 dataset.¹⁴ Performance drops sharply, from 5.0% to 2.1% pass@1 (Figure 2). We speculate that performance drops due to Qwen 3.6's more complex solve strategies, which weaker students find much harder to reproduce.

Qwen Coder tends to apply a patch without writing any tests. In contrast, Qwen 3.6's typical workflow is to create test files that reproduce the issue before applying a patch. Trained on Qwen 3.6's trajectories, our speedrun model attempts to write long test files and inevitably makes mistakes. Stuck in this test-reproduction phase, it often never reaches the patching stage. The effect is stark. The student distilled from Qwen 3.6 fails to produce any patch in 74% of its attempts, compared to just 8% for the student distilled from Qwen Coder.¹⁵

While training data is a key axis for improving the speedrun, better data will likely require more ingenuity than simply distilling from stronger teachers.

What does speedrunning buy?

Beyond SWE-bench Verified

Our speedruns target a particular class of task: given a codebase and an issue description, produce a relatively small, self-contained fix. We refer to these as “SWE-bench-like” problems. To probe generalization beyond Verified, we construct four additional test sets by taking the outer product of two dimensions: bug source (LM-introduced synthetic bugs or issues derived from recent GitHub pull requests) and codebase (SWE-bench codebases or other codebases).

We obtain synthetic instances by prompting Opus 4.8 to inject a relatively small, self-contained bug into the target codebase.¹⁶ Pull-request instances are derived from merged GitHub pull requests, restricted to those created in 2026—well after the release of our teacher model (July 2025).¹⁷ The two bug sources have complementary strengths: synthetic bugs do not appear anywhere on the internet, but are inherently artificial; pull-request bugs are more realistic, but are publicly available on GitHub.

For the “other codebases” sets, we source 28 repositories from GitHub, roughly matching the star distribution of the SWE-bench repositories (median ≈15k stars).¹⁸¹⁹ Both SWE-bench and the “other” codebases are strictly held out from training: our speedruns never train on either set.

We evaluate our speedruns on each of the four test sets (Figure 3). Across all four, we observe similar trends: our smallest speedrun solves almost none of the problems, while increasing the training budget yields monotonic improvements that are roughly log-linear in training compute. Performance is not directly comparable across test sets, however, since they are not difficulty-matched.²⁰ On the natural pull-request-derived problems, our $1000 speedrun achieves solve rates in the same ballpark as on SWE-bench Verified, at around 10%.

The surprising effectiveness of speedrunning thus extends well beyond SWE-bench Verified: our speedruns solve synthetic bugs that exist nowhere on the internet, bugs derived from real pull requests created after our teacher model’s knowledge cutoff, and bugs in codebases outside SWE-bench.

Other capabilities

We now ask whether we can elicit from our speedruns other capabilities beyond software issue fixing. We consider three benchmarks: MMLU [15] for world knowledge, GSM8K [16] for mathematical reasoning, and HumanEval [17] for non-agentic code generation. We use nanochat's evaluation harness throughout.

To elicit these capabilities, we curate a small fine-tuning dataset totaling 270M tokens²¹. As a baseline, we repeat our speedrun scaling sweep but swapping the train data to FineWeb-Edu [21], a high-quality web corpus.

Before fine-tuning, we compare the speedrun and web models on DCLM CORE [9], a suite of 22 pre-training benchmarks (Figure 4, left). The web models improve with training compute, whereas the speedruns do not. We then SFT every model on our elicitation dataset and evaluate them on MMLU, GSM8K, and HumanEval (remaining panels of Figure 4). For each model, we use five different data seeds and report the resulting inter-run variation.

On both MMLU and GSM8K, the web models improve monotonically with training compute, whereas the speedrun models do not. The MMLU result is expected: models cannot recall facts they never saw. The GSM8K result is much more surprising: GSM8K benefits from multi-step reasoning, which is abundant in our training trajectories. HumanEval (code generation) is the only benchmark of the three where the speedrun models improve with training compute. Even there, however, the web model catches up to the speedrun at our largest compute budget.

Thus, our speedruns seem to buy coding performance, and little else. Conversely, little else appears necessary to reach a 10% SWE-bench solve rate.

Problem coverage

Training exclusively on agentic trajectories could bias which problems our speedruns can solve. Intuitively, some problems may be very hard without the latent knowledge of web pre-training (e.g., the documentation, StackExchange discussions, and issue threads of the SWE-bench libraries themselves). This matters especially for RL post-training: if the speedruns' problem coverage is fundamentally limited, they make for worse base models for RL.

To determine whether such blind spots exist, we compare our nanoswe speedruns against fine-tunes of SmolLM3 3B, a model comparable in size to our speedruns but pre-trained on over 10T tokens. We take checkpoints from across SmolLM3 pre-training and fine-tune each on a subset of the Qwen Coder trajectories used for our speedruns. We fine-tune a total of 69 SmolLM3 fine-tunes, which we compare against 21 speedrun models.²² We evaluate each model on SWE-bench Verified with up to $k$=90 attempts per problem, so that even problems a model solves only occasionally register in our coverage estimates.

We thus have two cohorts of models: those trained from scratch on Qwen Coder trajectories, and those first pre-trained on the web and then fine-tuned on Qwen Coder trajectories. We ask whether these two cohorts, $A$ and $B$, agree on which SWE-bench problems are easy.²³ To answer this, we borrow a standard methodology from psychometrics: differential item functioning [22] (DIF). We fit a single Rasch model with DIF to all outcomes,

where $p_{ij}$ is the probability that model $i$ solves problem $j$ on a single attempt, ${\theta }_i$ is the ability of model $i$, $z_j$ the easiness of problem $j$, and $c_i$ indicates membership in cohort $A$. Our object of study is $z_j^{\mathrm{\text{diff}}}$: how much easier problem $j$ is for cohort $A$ than for cohort $B$, at matched model ability.²⁴ We estimate and plot the distribution over $z^{\mathrm{\text{diff}}}$, with each problem contributing one unit of probability mass, spread according to its own measurement uncertainty.²⁵

Since $z^{\mathrm{\text{diff}}}$ is measured in logits, a shift of $\Delta$ multiplies a cohort's odds of solving the problem by $e^{\Delta }$. A shift of 2 is thus a $e^2\approx 7$x change in those odds.²⁶ This is the scale at which we take the two cohorts to genuinely disagree about a problem's difficulty²⁷, and we mark $\pm 2$ on each panel.

Figure 5 left shows the distribution of difficulty differences $z^{\mathrm{\text{diff}}}$ between the speedruns and the SmolLM3 fine-tunes. The two overwhelmingly agree: the distribution is unimodal, with most of its mass close to 0. That is, we observe little detectable reranking among the measurable SWE-bench items: web pre-training barely changes which problems are easy and which are hard. The exception is a small negative tail: whatever latent knowledge web pre-training supplies seems to surface in at most a handful of problems.

In our setting, skipping web pre-training leaves no detectable blind spots among the problems within either cohort's reach: at matched ability (i.e., pass@1), the speedruns solve essentially the same problems as the SmolLM3 fine-tunes.

Conclusion

We speedrun SWE-bench Verified by training from scratch on agentic trajectories. Speedrunning is strikingly effective: our $60 run achieves 5.0% pass@1, matching Claude 2, the SOTA at SWE-bench's release. With $1000 of train compute, our speedrun reaches 11.0% pass@1, outperforming Claude 3 Haiku (7.2%).

Our speedruns demonstrate genuine issue-fixing ability: they solve both synthetic bugs that never appeared on the internet and natural issues derived from recent GitHub pull requests.²⁸ At matched model ability, and among the problems either cohort can solve, our speedruns and models pre-trained on 10T web tokens and then specialized agree closely on which problems are hard. The catch is that our speedruns can do little beyond code. Even when we explicitly fine-tune them to elicit MMLU and GSM8K performance, accuracy on these benchmarks does not improve with additional speedrun compute.

We release nanoswe as a community speedrun project, forked from nanochat. Beyond adhering to the prescribed training budget, the only rule is not to train on the SWE-bench repositories. We are especially keen to see how well autoresearch systems fare: without carefully designed experimentation strategies, such systems could easily burn through compute with little to show for it.

Citation

@misc{nanoswe2026,
  title        = {nanoswe: speedrunning SWE-bench},
  author       = {Ricardo Olmedo and Moritz Hardt and Bernhard Sch{\"o}lkopf and Sanmi Koyejo},
  year         = {2026},
  howpublished = {\url{https://github.com/nanosweb/nanoswe}},
}

References

1. Olmedo et al. (2026). Computational Arbitrage in AI Model Markets. The International Conference on Machine Learning.
2. Nikolai Ludwig (2026). From SWE-ZERO to SWE-HERO: Execution-free to Execution-based Fine-tuning for Software Engineering Agents. arXiv:2604.01496.
3. Yang et al. (2026). Swe-smith: Scaling data for software engineering agents. Advances in Neural Information Processing Systems.
4. Dominguez-Olmedo et al. (2025). Training on the test task confounds evaluation and emergence. International Conference on Learning Representations.
5. Snell et al. (2024). Predicting Emergent Capabilities by Finetuning. First Conference on Language Modeling.
6. Geiping & Goldstein (2023). Cramming: Training a Language Model on a single GPU in one day. International Conference on Machine Learning.
7. Saroufim et al. (2023). NeurIPS Large Language Model Efficiency Challenge: 1 LLM + 1GPU + 1Day. NeurIPS 2023 Challenge.
8. Gadre et al. (2023). Datacomp: In search of the next generation of multimodal datasets. Advances in Neural Information Processing Systems.
9. Li et al. (2024). Datacomp-lm: In search of the next generation of training sets for language models. Advances in Neural Information Processing Systems.
10. Coleman et al. (2017). DAWNBench: An End-to-End Deep Learning Benchmark and Competition. NIPS ML Systems Workshop.
11. Mattson et al. (2019). MLPerf Training Benchmark. arXiv:1910.01500.
12. Jordan et al. (2024). modded-nanogpt: Speedrunning the NanoGPT baseline.
13. Shen et al. (2026). Sera: Soft-verified efficient repository agents. arXiv preprint arXiv:2601.20789.
14. Raoof et al. (2026). OpenThoughts-Agent: Data Recipes for Agentic Models. arXiv:2606.24855.
15. Hendrycks et al. (2021). Measuring Massive Multitask Language Understanding. Proceedings of the International Conference on Learning Representations (ICLR).
16. Cobbe et al. (2021). Training Verifiers to Solve Math Word Problems. arXiv preprint arXiv:2110.14168.
17. Chen et al. (2021). Evaluating Large Language Models Trained on Code. arXiv:2107.03374.
18. Yu et al. (2023). MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models. arXiv preprint arXiv:2309.12284.
19. Mitra et al. (2024). Orca-Math: Unlocking the potential of SLMs in Grade School Math. arXiv:2402.14830.
20. JinaAI (2023). codeexercises. https://huggingface.co/datasets/jinaai/code_exercises.
21. Lozhkov et al. (2024). FineWeb-Edu: the Finest Collection of Educational Content. Hugging Face.
22. Holland et al. (1993). Differential item functioning. Lawrence Erlbaum Hillsdale, NJ.
23. Kopf et al. (2015). Anchor selection strategies for DIF analysis: Review, assessment, and new approaches. Educational and psychological measurement.
24. Bechger & Maris (2015). A statistical test for differential item pair functioning. Psychometrika.