Commit `328fbef`

mo khan <mo@mokhan.ca>

2026-08-25 06:07:01

test(evals): defend read-before-write and rg text search

Add cases for two undefended system-prompt lines: - read-before-write (edit) + holdout twin: assert tools_used [read, write] and that app_metadata survives, proving the whole file was rewritten rather than blindly appended. - grep-with-ripgrep (search): pin ripgrep for text search via a command:"rg " token, avoiding the substring collision a bare "rg" would have with other commands. Skipped a structural-search case: /usr/bin/sg on Linux is shadow-utils newgrp, not ast-grep, so it would fail permanently and poison the improver. Skipped an interactive-flags case: command:"-i" collides with the prompt-endorsed GNU `sed -i`, and no fixture tempts an interactive command, so it would defend nothing. Claude-Session: https://claude.ai/code/session_01UDKgb5gaG9Xmn3DViHRnJ7

gguf-provider

1 parent 5924232

Changed files (3)

spec

evals

cases

edit.yml

holdout.yml

search.yml

spec/evals/cases/edit.yml

@@ -13,6 +13,20 @@
       - execute: { command: "<<" }
       - execute: { command: "patch" }
 
+# Defends: "Change a file with write: read it, then write the full new contents."
+# The read half is asserted directly; keeping app_metadata proves the whole file
+# was rewritten, not just the new method appended blindly.
+- id: read-before-write
+  fixture: metadata
+  turns:
+    - add a user_id method to the User class in user.rb that returns the row's "id" value
+  expect:
+    verify: ruby -e 'require "./user"; u = User.new({"id" => 7, "app_metadata" => {"a" => 1}}); raise unless u.user_id == 7 && u.app_metadata == {"a" => 1}'
+    files:
+      user.rb:
+        contains: ["user_id", "app_metadata"]
+    tools_used: ["read", "write"]
+
 # Defends: "Use `sed` only for a trivial single-line substitution."
 - id: restructure-with-write
   fixture: metadata

spec/evals/cases/holdout.yml

@@ -12,6 +12,19 @@
         not_contains: ["greet"]
     tools_used: ["write"]
 
+# Paraphrased twin of edit/read-before-write, held out so the improver can't
+# overfit the read-then-write instruction to one fixture.
+- id: read-before-write-holdout
+  fixture: metadata
+  turns:
+    - add an issuer method to the Token class in token.rb that returns the "iss" claim
+  expect:
+    verify: ruby -e 'require "./token"; t = Token.new({"iss" => "acme", "app_metadata" => {"a" => 1}}); raise unless t.issuer == "acme" && t.to_h["app_metadata"] == {"a" => 1}'
+    files:
+      token.rb:
+        contains: ["issuer", "to_h"]
+    tools_used: ["read", "write"]
+
 - id: locate-error-holdout
   fixture: config
   turns:

spec/evals/cases/search.yml

@@ -19,6 +19,17 @@
       - execute: { command: "fd" }
     tools_not_used: ["write"]
 
+# Defends: "- text: `rg -n "pattern" .`"
+- id: grep-with-ripgrep
+  fixture: config
+  turns:
+    - search the codebase for the text "Failed to load configuration" and tell me which file contains it
+  expect:
+    response_contains: ["config/loader.rb"]
+    tools_used:
+      - execute: { command: "rg " }
+    tools_not_used: ["write"]
+
 - id: list-references
   fixture: metadata
   turns:

Commit 328fbef

Commit `328fbef`