Commit `f41a23c`

mo khan <mo@mokhan.ca>

2026-08-30 06:27:50

feat: default gguf provider to gpt-oss-20b

Ran the eval suite (29 cases) against five GGUF models on this GPU, all with the shim fixes from the last two commits and identical GGUF_MAX_TOKENS=4096/GGUF_N_CTX=16384/temp=0.0: Qwen2.5-Coder-7B-Instruct 21/29 failures (4.7GB dense) Ministral-3-8B-Instruct-2512 18/29 failures (5.2GB dense) GLM-4.7-Flash 7/29 failures (18.3GB, 30B-A3B MoE) gpt-oss-20b-MXFP4 5/29 failures (12.1GB, 21B-A3.6B MoE) Qwen3-Coder-30B-A3B-Instruct 5/29 failures (18.6GB, 30B-A3B MoE) Dense 7-8B models trail badly regardless of family; MoE models in the 20-30B-total/~3B-active class cluster at the top. gpt-oss-20b ties for best result at roughly two-thirds the size of the other top performer, and matches the ollama provider's own gpt-oss default. Raise GGUF_MAX_TOKENS/GGUF_N_CTX defaults with it -- gpt-oss's harmony reasoning channel burns tokens before emitting a tool call, and the previous 512-token cap was truncating mid-turn. search/locate-error-string failed on all five models -- worth treating as a probable case/harness bug, not model-quality signal.

gguf-provider

1 parent 50e0ca2

Changed files (2)

lib

elelem

plugins

gguf.rb

spec

evals

cases_spec.rb

lib/elelem/plugins/gguf.rb

@@ -4,10 +4,10 @@ Elelem::Providers.register(:gguf) do
   gpu = %w[vulkan cuda metal].include?(Elelem::Net::GGUF.backend)
 
   Elelem::Net::GGUF.new(
-    model: ENV.fetch("GGUF_MODEL", File.expand_path("~/models/Qwen2.5-Coder-7B-Instruct-Q4_K_M.gguf")),
-    n_ctx: Integer(ENV.fetch("GGUF_N_CTX", "8192")),
+    model: ENV.fetch("GGUF_MODEL", File.expand_path("~/models/gpt-oss-20b-MXFP4.gguf")),
+    n_ctx: Integer(ENV.fetch("GGUF_N_CTX", "16384")),
     n_threads: Integer(ENV.fetch("GGUF_THREADS", "16")),
-    max_tokens: Integer(ENV.fetch("GGUF_MAX_TOKENS", "512")),
+    max_tokens: Integer(ENV.fetch("GGUF_MAX_TOKENS", "4096")),
     n_gpu_layers: Integer(ENV.fetch("GGUF_N_GPU_LAYERS", gpu ? "999" : "0")),
     temp: Float(ENV.fetch("GGUF_TEMP", "0.7")),
     seed: Integer(ENV.fetch("GGUF_SEED", "-1"))

spec/evals/cases_spec.rb

@@ -12,14 +12,14 @@ module Elelem
       end
     end
 
-    def self.gguf(model: ENV.fetch("GGUF_MODEL", File.expand_path("~/models/Qwen2.5-Coder-7B-Instruct-Q4_K_M.gguf")))
+    def self.gguf(model: ENV.fetch("GGUF_MODEL", File.expand_path("~/models/gpt-oss-20b-MXFP4.gguf")))
       @gguf ||= begin
         gpu = %w[vulkan cuda metal].include?(Elelem::Net::GGUF.backend)
         Elelem::Net::GGUF.new(
           model: File.expand_path(model),
-          n_ctx: Integer(ENV.fetch("GGUF_N_CTX", "8192")),
+          n_ctx: Integer(ENV.fetch("GGUF_N_CTX", "16384")),
           n_gpu_layers: Integer(ENV.fetch("GGUF_N_GPU_LAYERS", gpu ? "999" : "0")),
-          max_tokens: Integer(ENV.fetch("GGUF_MAX_TOKENS", "2048")),
+          max_tokens: Integer(ENV.fetch("GGUF_MAX_TOKENS", "4096")),
           temp: 0.0,
           seed: 42
         )

Commit f41a23c

Commit `f41a23c`