We ran Perseus, our code-retrieval service, through DeepSWE, datacurve's 113-task software-engineering benchmark scored pass@1 by a held-out test suite, with exactly one variable flipped per comparison: is Perseus available to the agent, or not. Same model, same harness, same tasks, same grader. The result is not a single percentage but a relationship. The size of the lift is set by how much each agent was bottlenecked on finding the code, and you can read that off the bare scores before you run anything.
With Perseus, gpt-5.5 on mini-swe reaches 72.4%, past the 70% frontier model on the public board. On codex the lift is +18 points, 45.1% to 61.1%, thirty matched wins against twelve losses. On claude-code with opus-4.8 it is +4 points while spending 29% fewer tokens. On mini-swe, a harness that already walks to the right file on its own, Perseus moves nothing measurable. One rule produces all three outcomes, and the place Perseus does nothing is the place the benchmark accidentally removed the problem it solves. The rest of this post is that rule.
What we measured, and why the codex delta is the one to trust
DeepSWE is 113 real bug-fix tasks. Each ships with a held-out test suite split into fail-to-pass tests (red to green if the bug is fixed) and pass-to-pass tests (the regression guard). A task counts as solved only when every f2p test goes green and every p2p test stays green; there is no partial credit. We score Perseus the way the leaderboard scores a model, identical grader and identical tasks, and toggle Perseus off versus on. Every delta in this post is that toggle on a fixed model and harness, over the strict intersection of tasks both runs completed. No cross-run, cross-model, or hand-picked numbers.
Two facts make the codex result the load-bearing one. First, our plumbing reproduces the board: bare gpt-5.5 on mini-swe scored 68.1% in our harness against the official 67%, within a point, so a Perseus-versus-bare delta measures Perseus and not our harness. Second, we caught a measurement artifact before trusting any of it. The Perseus protocol instructs the agent to query with --json; the models largely ignored that flag and queried in prose, and the label that marks Perseus's surfaced pinning-test renders only in JSON. Our first instrumentation read came back near-zero for some arms, which looked like Perseus was inert. It was a rendering gap, not a capability gap. The same query in prose and in JSON returns byte-identical hits: same files, same ranks. We forced JSON at the harness shim and confirmed the forcing was outcome-neutral (net −1 pass, roughly 80% ties), so it makes the signal visible without moving outcomes. The final run produced 5,045 retrieval labels with zero auth-fallbacks.
The grading failures got the same treatment. Where a verifier timed out on a huge test suite (the agent had solved the task, the grading process wedged), we re-ran that one task at 4× verifier time rather than scoring it zero. Two tasks needed it. Where a comparison below runs on fewer than 113 tasks, it is the matched intersection of both runs, nothing more.
The numbers
At one trial per task, a raw pass-rate carries a few points of run-to-run noise. So we report Perseus-versus-bare on the same tasks, which cancels both the difficulty mix and the noise in one subtraction. Three matched comparisons follow; for orientation, the bare leaderboard reads gpt-5.5 67%, opus 59%, top model fable-5 70%.
Three matched wins
The codex row is the clean read of the mechanism: thirty rescues, twelve regressions, +18 net on the full 113. The mini-swe row puts gpt-5.5 at 72.4%, over the 70% frontier model on the board. The claude-code row is +4 points bought at 29% fewer tokens, which the cost section unpacks. One wrinkle is the thesis in miniature: on our full-113 mini-swe runs the gpt delta shrinks toward the noise floor, +2. The win is large where localization is the wall and small where it is not, so its magnitude depends on the task mix and the harness, exactly what the mechanism predicts. The codex +18 is the sharpest read of it, and the next two sections are why.
Why mini-swe at 68% beats codex at 45% bare, with the same model
The surprise deserves precision, because the naive reading ("codex is the weaker harness") is wrong. In its native setup OpenAI reports Codex CLI around 77% on SWE-bench Verified, and mini-swe is the Princeton/Stanford SWE-bench team's hundred-line agent that clears 74% on the same benchmark, in production at Meta and NVIDIA. Both are strong. The gap between them on these tasks is the harness acting as a lever on the same model: independent testing has measured a single model swinging ~16 points from the harness alone, one report putting Opus at 77% in one harness and 93% in another on identical tasks.
On these tasks, with gpt-5.5, the lean harness fit the shape of the work. mini-swe puts almost nothing between the model and the shell, so nearly every turn lands on reading, editing, or testing. On a bounded task with a known finish line that directness is correct, and it is the whole Princeton thesis. codex carries more agent-loop structure and tooling ceremony between the model and the file. That machinery earns its keep on sprawling open-ended work, but on a bounded bug with gpt-5.5 it spent proportionally more turns orienting and fewer turns editing. The performance a heavier harness leaves on the table by orienting inefficiently is the performance Perseus hands back, and that turns-spent-orienting figure is the variable everything else in this post moves.
Why codex benefits and mini-swe does not: the localization-deficit mechanism
You describe code by intent and Perseus returns ranked path:line locations in about two seconds. It replaces a loop (walk the tree, guess a grep, read the wrong file, guess again) with a single ranked answer. So the question is never "is the answer good," it is "what was the agent spending turns on before the answer existed." codex is the agent that was spending the most.
Perseus reallocates turns, it doesn't add capability
Take a rescue and trace it. The task fastapi-implicit-head-options asks the agent to add implicit HEAD and OPTIONS handling to FastAPI. That change lives wherever FastAPI decides which HTTP methods a route owns, in a codebase the agent has never read. Bare codex never cracked it, and the reason is structural, not a reasoning failure. You cannot grep for the place that owns method dispatch until you know the token it is named by, and you do not know that token because you have never seen the file. That circular dependency between knowing where the code is and being able to search for it is the localization deficit, and string search cannot break it. The Perseus-equipped agent broke it by asking in intent ("where does FastAPI generate path operations and method order," "what owns GET/HEAD/OPTIONS"), phrasings that name the behavior, not the symbol. Perseus ranked the files by meaning, the agent opened the right one, and the fix shipped. Retrieval by meaning does not need the token.
Run the same mechanism on each harness and the +18 and the wash fall out of one rule. codex was bleeding turns on orientation, so when Perseus deletes the orient loop those turns convert directly into fixes. mini-swe already localizes well, which is why it scores in the high sixties bare; its bottleneck was never finding the file. On a task it would have localized in one turn, the ranked answer confirms what the model already had, so no turns are saved or lost. Perseus pays off in proportion to how much localization is the agent's bottleneck: codex is bottlenecked on it (low bare, big lift), mini-swe is not (high bare, no effect). The benefit tracks the deficit, so you can tell in advance which agents Perseus will transform and which it will leave alone, without running the benchmark.
The confirming evidence: the value is concentrated on the hard tasks
If the rule is "Perseus helps in proportion to the localization deficit," it must help most on the tasks that were hardest for the bare agent and little (or even hurt) on the ones bare already aced. So split the 113 codex tasks by what bare codex did with each.
The win lives on the hard tasks
On the 62 tasks bare codex failed (the hard ones, where finding the code was the wall), Perseus rescued 30, a 48% solve rate on bugs the bare agent could not touch at all. On the 51 tasks bare codex solved (the easy ones, where it already had the answer), Perseus regressed 12; on about a quarter of them the mandatory query-and-verify detour pulled the agent off a line it already had. Net +30 − 12 = +18. The win is a localization bet that pays heavily where finding the code was the wall and costs a little where it was not, and on codex most of the benchmark is hard (62 of 113), so the bet lands far in the positive.
That same split is the mini-swe wash, viewed from the other harness. Perseus always rescues-hard and regresses-easy; only the net changes, and the net is set by the hard/easy ratio, which is set by how well the harness already localizes. codex is a weak localizer here, so the benchmark is mostly hard for it and the rescues dominate. mini-swe localizes well, so the identical tasks are mostly easy for it: the rescues are few, the small easy-task cost cancels them, and the arms land level. The rescue is the same behavior on both harnesses meeting a different proportion of hard tasks.
What it costs
The cost delta is the localization deficit read in tokens instead of pass-rate, and it points at the same agents the benefit does. What Perseus does to the bill tracks how the bare harness fails on a miss. A heavy harness churns on a miss (re-reading, re-grepping, re-orienting down dead ends), and Perseus replaces that churn with one ranked answer. So on claude-code, the heaviest harness here, Perseus cut the bill while raising the score: −29% tokens and −19% agent steps per task (15.7M versus 22.2M tokens, 62 versus 77 steps) for +4 points.
codex bails on a miss instead of churning, so Perseus spends to finish the fix: about 2.1× tokens per task, but only ~1.5× per solved task (21.4M versus 10.2M median tokens, +110%), and those are fixes bare never landed. mini-swe succeeds cheaply on its own, so the query-and-verify protocol is pure overhead: +35%, a median $6.42 against $4.77 per task, for a wash. Where finding the code was the bottleneck Perseus pays for itself, and on the worst-churning harness it is cheaper and better at once. Where it was not, you pay overhead for nothing, so do not run it there.
The benefit tracks the deficit
Where the benchmark undersells Perseus, and why the wash will not hold
The mini-swe wash is the most important result in this post, because of why it happens. A DeepSWE task is the easiest possible case for localization: one repo, one specified bug, a held-out test suite that announces when you are done. The DeepSWE repos run a median of 425 files, 77% of them under a thousand, small enough that a lean agent reads enough of the tree in a few turns to localize without searching, and once it has, the file is already found. It never had to search at all. So "mini-swe localizes well" really means "the repo fits in the context window," and a repo that fits poses no localization problem to anyone. The wash is not evidence that retrieval is unnecessary. It is evidence that this benchmark removed the problem retrieval solves.
That property does not survive contact with real engineering. Localization is the inverse of how much of the relevant code an agent can hold in context at once. On a bounded benchmark repo that fraction is effectively all of it, so the deficit is near zero and Perseus has nothing to convert. Real codebases run one to three orders of magnitude larger (a single service is one to five thousand files, a monorepo tens of thousands into the millions), multi-repo, and unfamiliar; the fraction any agent can hold is a sliver, and the deficit is large for every agent regardless of how lean it is. A hundred-line harness has no special exemption from that ceiling. Push the repo past the context window and mini-swe becomes as localization-bound as anything else, which is the regime where Perseus helps. The benchmark did not find that Perseus is unnecessary for lean agents. It found that the deficit is small when the repo is small, which is true and unsurprising, because the repos people ship in are not small.
So DeepSWE measures Perseus on its weakest terrain, the case where localization is nearly free for everyone, and even there it turns a 45% codex agent into 61% and wins on two of three harnesses. The mini-swe no-gain is the floor of Perseus's value, and that floor is a repo small enough to read whole. The regime Perseus is built for is the one a bounded benchmark cannot stage, where the hard questions ("where is this behavior implemented, what will my change break, where are the tests that pin it") eat hours, not turns. Three properties carry it there. It searches by meaning, not by string, so it answers "where is auth enforced on the query route" without the token you do not yet have. It goes past find to impact: perseus impact <symbol> returns the callers, the tests that pin them, and the cross-boundary consumers a change breaks, the blast radius before the edit rather than red CI after. And it indexes your live working tree, uncommitted edits and all, across repos, so the answer lines up with the files in front of you. Each of those is dead weight when the repo fits in context, and load-bearing the moment it does not.
Methods, for the record
Benchmark: DeepSWE v1.1, 113 tasks, pass@1, one trial per task; a task is solved only when every fail-to-pass test goes green and every pass-to-pass test stays green. Configs: gpt-5.5 on codex and mini-swe, opus-4.8 on claude-code and mini-swe, xhigh reasoning throughout; the agent queried production Perseus over HTTP (an MCTS planner on gemma3-4b, an SFR code embedder, a BGE merge-point reranker). Comparison: every delta is Perseus on-versus-off, same model and harness, on the strict intersection of completed tasks; bare gpt-5.5 mini-swe scored 68.1% against the public 67%, calibrating the harness to the board. Cost: mini-swe from the harness's own instance_cost (median, to resist runaway-task outliers); claude-code from per-message token usage and agent-step counts; we deliberately do not sum raw cumulative total_tokens, because prompt caching overcounts it by roughly 6×. Hygiene: --json forced at the harness shim so retrieval labels render uniformly across models, verified outcome-neutral (net −1, ~80% ties); two graders that timed out on huge suites were re-run at 4× verifier time rather than scored 0.
Caveats
The mini-swe result is a no-result, not a null we can stand on: pass@1 at one trial gives a few points of run-to-run swing and the mini Perseus-versus-bare gap sits inside it, so we report it as no measurable effect, and a pass@k run would tighten it. The opus read we stand on is claude-code, where a matched bare exists: Perseus + opus-4.8 + claude-code beats bare claude-code by +4 (59.2% versus 54.9% on the same 71 tasks) at 29% lower cost. opus + mini-swe at xhigh samples lower (55.8%) and we do not lean on it. The opus-low arm under-engaged, skipping the query on roughly 40% of tasks, so it measures a lazy agent with Perseus available rather than Perseus. And the clean deltas come from two model families across three harnesses; the mechanism predicts the same shape more broadly, but more models would strengthen it.
Bottom line
Perseus is a localization engine, and an agent's appetite for it is the inverse of how well it already localizes. codex needed it and went 45% to 61%; claude-code needed it and got +4 points at 29% lower cost; gpt-5.5 with Perseus tops the board at 72.4%. mini-swe never had a localization problem to solve on a repo small enough to read whole, so it rode along unchanged, and that no-gain is the benchmark's easiest terrain, not Perseus's limit. Point Perseus at the agents losing turns to "where is the code," and it pays for itself in turns saved, sometimes in dollars too, and more so the larger the codebase gets: the further from a benchmark and the closer to the code you actually ship.