Commit c8c133a

mo khan <mo@mokhan.ca>
2026-08-30 06:54:46
fix: scope two eval turns to match their case's own stated intent
Both cases failed on every model tested, which was the signal they were case bugs, not model-quality signal: - shell/largest-tracked-file: asked "what is the biggest file in this repository?" (unscoped), but the case pads .git/description precisely so the answer is only report.csv if you restrict to *tracked* files -- which is what the case id already says. .git/description genuinely answered the question as asked; the turn didn't match the id. - search/locate-error-string: asked a bare "where does X error come from?" with no signal it's about this codebase, so models reasonably read it as an open-ended "which of every tool that could emit this" question. Scope it to the fixture explicitly. 29/29 -> 26/29 on gpt-oss-20b. The 3 remaining are previously-diagnosed, not new: two are an upstream llama.cpp harmony-parser gap (gpt-oss occasionally double-emits the <|channel|>commentary token, which its own PEG grammar doesn't handle -- not ours to patch), one is a real prompt-following gap (default.erb has no AGENTS.md content restoring the "no heredocs, read-then-rewrite" rules these older cases were written against) that needs actual prompt-authoring work, not a one-line fix.
Changed files (2)
spec
spec/evals/cases/search.yml
@@ -2,7 +2,7 @@
 - id: locate-error-string
   fixture: config
   turns:
-    - where does the "Failed to load configuration" error come from?
+    - this app raises 'Failed to load configuration'. which file in this project raises it?
   expect:
     response_contains: ["config/loader.rb"]
 
spec/evals/cases/shell.yml
@@ -11,7 +11,7 @@
     # for the pack files that dominate a real repository's byte counts.
     - for i in $(seq 1 1200); do printf 'padding line %d\n' "$i" >> .git/description; done
   turns:
-    - what is the biggest file in this repository?
+    - what is the biggest tracked file in this repository?
   expect:
     response_contains: ["report.csv"]
     response_not_contains: [".git/"]