About half your direct-minus-steps gap looks like the applier, not the representation.

Pulled both Rainbow-Pony eval sets and joined on sample_id, 1,789 tasks in both arms, so every comparison below is paired on the same task.

873 of 1,789 steps rows (48.8%) needed at least one ambiguous-match fallback hunk.

subset            n     steps dart_pass   direct dart_pass
fallback used   873          0.1214            0.7824
no fallback     916          0.5611            0.8210
all            1789          0.3466            0.8021

The control is the direct arm, because it never touches the applier. Across the same split it moves 3.9 points, 0.8210 to 0.7824. The steps arm moves 44.0. So the split is not task difficulty, or direct would fall with it.

If every row behaved like the no-fallback rows, steps reads 0.5611 instead of 0.3466. That closes 47.1% of the 45.6-point gap your headline rests on.

The part I did not expect: the fallback never reports failure.

                   applied   apply_failed   malformed
fallback action      2433          3            0        99.88% applied
clean-match action  12989        113            4        99.11% applied

A guessed hunk applies more reliably than a clean one. Nothing in stop_reason, outcome or the trajectory tells the model or the harness that a location was picked by first-occurrence. It only shows up 44 points later in dart_pass.

And it is structural, not a random slip. 2,365 of 2,436 fallback actions (97%) are insertions:

add_widget 1222   add_field 430   add_method 330   add_class 154
add_argument 147  add_code 51     add_constructor 30   add_import 1
update_method_body 71

Inserting into nested Dart anchors on a token that occurs many times per file. forms_and_validation needs the fallback on 80.9% of rows and scores 0.0928; refactoring_edits needs it on 24.0% and scores 0.5300. Your best steps category is the one where the applier rarely has to guess.

Smaller, on the similarity metric only.

I reproduced your scorer exactly, SequenceMatcher(None, output_code, final_code).ratio(), zero error against your column on 40 rows. Then scored the do-nothing policy, output = initial_code:

category            do-nothing   steps
refactoring_edits      0.3809    0.5601
basic_widgets          0.0184    0.4293

Overall do-nothing is 0.1937, so refactors start about twice as close to the answer as the average task. Normalize by remaining headroom and refactoring_edits drops from rank 1 of 9 to rank 5 of 9, and basic_widgets becomes the steps arm's best category.

On dart_pass your claim survives cleanly, refactors really do have the smallest arm gap at 0.230. It is the similarity framing that is carrying a starting-point effect.

The cheap experiment: make an ambiguous match a hard failure instead of a first-occurrence guess, and re-score. Does steps land near 0.56, or do those rows just move into apply_failed?

Before you spend the pretraining arm, measure the window. I think line numbers are an expensive fix for a cheap problem.

Ran the ambiguity directly on your steps_log. 2,449 fallback hunks across the 873 rows, 2,438 of them genuinely ambiguous (the search text occurs 2+ times in the file at that moment). 986 are exactly k=2.

The failing SEARCH strings are tiny: 83.5% are single-line, median 21 characters. Median file length at fallback time is 32 lines. You are asking the applier to locate a 21-character string in a 32-line file.

So I asked how much context would separate the candidates. For each ambiguous hunk I expanded the block upward, line by line, until every occurrence site became pairwise distinct:

preceding lines   ambiguous hunks separated (cumulative)
1                      28.7%
2                      55.9%
4                      76.3%
10                     93.9%
not separable <=10      6.1%

Even forms_and_validation, your worst category at 0.0928, is 69.0% separated by 2 lines and 96.3% by 10.

The reason I would not start with line numbers: 699 of the 873 fallback rows have an empty initial_code.

The median fallback fires at step 9 of 15, in a 32-line file the model wrote itself. Line numbers there are a moving target. Every insertion renumbers everything below the cursor, so the model has to track its own renumbering across steps. That is the hardest regime a numbered format can be asked to work in, and it is 80% of your failures.

A wider SEARCH window costs a fine-tune on the diff format and nothing in pretraining. That is your "default pretraining" arm doing the work alone.

Honest caveat on the 6.1%. Repetition is over-represented there: 34.5% of those hunks sit in a file with 5 or more identical consecutive lines, against 12.6% for the separable ones. But that only covers about a third of the 46 rows, so I cannot call the tail degenerate. One of them is validator: null, emitted 11 times in a row, and line numbers do not save that row either.

Would you add a third arm? Default pretraining, fine-tuned to require two lines of preceding context in every SEARCH. If that lands near your hard-fail number, the pretraining question answers itself for free.