# The loop that found its own bug *Kagerou · 18 Oct 2026* *Two models agree on what a vague function should do by trading test cases. A third model tunes the prompts. In May I ran this for $29.52, and the most useful thing it produced was a bug report about my own harness.* ## The protocol Hand two coding agents the same underspecified docstring: `chunk(xs, n)`, `percentile(xs, p)`, `format_duration(seconds)`. Each writes an implementation. They will disagree, and the disagreements are the interesting part, because the docstring never said what `chunk` does with a short last chunk. The witness-set protocol settles this with examples. Each writer proposes *witnesses*: input and expected-output pairs. Each implementation is run against the other writer's witnesses. Where both agree, the witness is frozen and becomes part of the shared spec. Where they disagree, a third model, the resolver, reads both sides and picks one; its decision is frozen too. The writers revise against the frozen set and go another round, up to three. A problem *converges* when both implementations pass every frozen witness. The writers and the resolver were Claude Sonnet 4.6, called through `claude -p`. The benchmark was 32 underspecified Python functions, later 37, each with a hidden reference implementation and hidden test inputs the agents never see. That gives two numbers per problem: did the writers converge, and is what they converged on correct? ## The loop on top Around the protocol I put an optimizer loop in the style of Karpathy's autoresearch. A fourth Sonnet 4.6 agent reads the spec and the logs, edits the editable files (prompt templates and hyperparameters, never the benchmark or the judge), runs the invariant tests, reruns the benchmark, and keeps the change only if a composite score improves on the training problems without falling more than 0.05 on six held-out problems: `M = 1.0 × convergence + 2.0 × correctness − 0.1 × Mtok − 0.05 × mean rounds` Two changes were accepted before both background runs ran out of Claude Code usage quota. **Iteration 1** looked at the two training problems that never converged, `identity` and `mean`. In the trace for `identity`, both writers proposed the same witness, `identity([1, 2, 3]) == [1, 2, 3]`, and both implementations were correct. The sandbox splats a list input into positional arguments, so the witness ran as `identity(1, 2, 3)` and raised `TypeError`. Each writer saw the other's witness fail, the resolver ruled the obvious way three times, and the frozen witness kept failing for the same reason. The writer prompt had never explained the calling convention. The loop added seven lines to it describing the argument-wrapping rule, with an example. Training convergence went from 0.923 to 1.000, and M from 2.871 to 2.950. **Iteration 2** came after I added five deliberately ambiguous problems (`indent_text`, `compress_rle`, `merge_intervals`, `balanced_brackets`, `format_duration`), where the writers sometimes agreed on the wrong reading. The loop told writers that at least half their witnesses should discriminate between two plausible interpretations, and told the resolver to prefer the more compact format when the docstring is silent. Correctness rose from 0.947 to 0.979. Convergence fell from 0.968 to 0.935: `chunk` and `percentile`, which had converged before, stopped converging. M rose by 0.031, so the rule accepted it. ## The bug During iteration 2's validation, the held-out score came back as 0.00 against a baseline of 2.95. It looked like catastrophic overfitting, and the acceptance rule would have reverted it, which would have been the correct decision for the wrong reason. Nothing had overfit. Midway through the run, `claude -p` had started failing on quota. The resulting `CLIAgentError` was not the exception type the per-problem handler caught, so it propagated up to the benchmark runner, whose outer `except Exception: continue` skipped the problem without writing a trace. Six held-out problems with no trace records produced a convergence rate of zero over an empty set. The fix was one change: a crashed problem now gets a trace with status `RUNTIME_ERROR`, so it counts as a failure in the log and in the metric. Re-reading the logs for this post, I found the same bug had already touched iteration 1. Its accepted run has 27 trace records for 32 problems. `factorial`, `parse_int_lenient`, `flatten_one`, `gcd_pair` and `is_prime` are missing; all five had converged in the baseline, and three of them were held-out problems. The two problems iteration 1 fixed, `identity` and `mean`, are genuinely fixed. But the "clean win" was accepted on an incomplete set, with half the held-out problems absent, and the harness never said so. That is the part I would tell anyone building one of these loops. An optimizer that accepts or reverts on a number is only as honest as the code that computes the number, and a silent `continue` turns infrastructure failures into apparent model behavior. Here the loop happened to surface it, because a zero is loud. A partial drop, like iteration 1's, is quiet. ## Three things the run showed **Writers can get stuck where the resolver cannot help.** In the revalidation, `chunk` had no cross-evaluation disagreements in rounds 1 and 2 and still failed to converge. That only happens if a writer keeps violating a frozen witness, choosing an interpretation the resolver has already ruled against. On `percentile`, four witnesses were contested in every round, with the writers trading sides. The protocol has no escalation after repeated deadlock. **The weights hide the trade.** With correctness weighted twice as heavily as convergence, iteration 2's gain on the ambiguous problems paid for losing two working ones. The rule said yes. A person reading the diff might well have said no. That is the ordinary Goodhart problem, and it shows up as soon as the loop has two things to balance. **Prompt advice has no scope.** The "discriminate between interpretations" instruction was aimed at the five ambiguous problems and applied to all 37. On `chunk` and `percentile` it manufactured edge cases the writers then disagreed about. One prompt per agent cannot carry per-category guidance. ## Limits Two accepted iterations, one seed (42), one run per configuration. A change of +0.031 is inside the noise of a single stochastic run; iteration 2 should be re-scored over five seeds before anyone believes it. The sandbox is a subprocess with a timeout, adequate for stdlib-only toy functions and nothing adversarial. The outer agent is itself a model, and I reviewed its diffs after it committed them, not before. The code, spec and per-iteration logs are in `witness_protocol/`; a full benchmark run takes about five minutes and about $8.