Commit `63d20bb`

mo khan <mo@mokhan.ca>

2026-08-27 22:11:47

feat(gguf): opt-in Vulkan GPU offload

- extconf.rb: ELELEM_GGML_VULKAN=ON builds the ggml-vulkan backend (CPU stays the default so `gem install` needs no GPU toolchain) - thread n_gpu_layers through Net::GGUF + GGUF_N_GPU_LAYERS env (0 = CPU, 999 = offload all) Verified on AMD Radeon 8060S (RADV GFX1151): full layer offload, plain reply and tool call both work in-process. Suite 316/0. Build deps (Fedora): vulkan-headers vulkan-loader-devel glslc glslang spirv-tools spirv-headers-devel. Claude-Session: https://claude.ai/code/session_01UDKgb5gaG9Xmn3DViHRnJ7

gguf-provider

1 parent 67ccf82

Changed files (3)

ext

elelem_llama

extconf.rb

lib

elelem

net

gguf.rb

plugins

gguf.rb

ext/elelem_llama/extconf.rb

@@ -13,6 +13,12 @@ VENDOR    = File.join(GEM_ROOT, "vendor", "llama.cpp")
 BUILD_DIR = File.join(EXT_DIR, "build")
 NATIVE    = File.join(GEM_ROOT, "lib", "elelem", "native")
 
+# GPU offload is opt-in: ELELEM_GGML_VULKAN=ON builds the Vulkan backend (needs
+# vulkan-headers, vulkan-loader-devel, and glslc). CPU-only stays the default so a
+# bare `gem install` never requires GPU toolchain. Toggling this flips a cached
+# CMake var, so switching modes wants a clean build dir (rm -rf ext/*/build).
+VULKAN = %w[1 on true yes].include?(ENV["ELELEM_GGML_VULKAN"].to_s.strip.downcase)
+
 def run(*cmd)
   warn "elelem: + #{cmd.join(' ')}"
   system(*cmd) || abort("elelem: build step failed: #{cmd.join(' ')}")
@@ -32,7 +38,7 @@ run("cmake", "-S", VENDOR, "-B", BUILD_DIR,
     # libllama finds its ggml siblings there -- not in this throwaway build dir.
     "-DCMAKE_BUILD_WITH_INSTALL_RPATH=ON",
     "-DCMAKE_INSTALL_RPATH=$ORIGIN",
-    "-DGGML_VULKAN=OFF",
+    "-DGGML_VULKAN=#{VULKAN ? 'ON' : 'OFF'}",
     "-DLLAMA_CURL=OFF",
     "-DLLAMA_BUILD_COMMON=ON", # common_chat: chat templates + tool-call parsing
     "-DLLAMA_BUILD_TESTS=OFF",
@@ -67,4 +73,4 @@ File.write(File.join(EXT_DIR, "Makefile"), <<~MAKE)
   .PHONY: all clean install
 MAKE
 
-warn "elelem: built llama.cpp + shim into #{NATIVE}"
+warn "elelem: built llama.cpp + shim into #{NATIVE} (#{VULKAN ? 'Vulkan/GPU' : 'CPU'})"

lib/elelem/net/gguf.rb

@@ -30,9 +30,9 @@ module Elelem
         end
       end
 
-      def initialize(model_path:, n_ctx: 4096, n_threads: 16, max_tokens: 512)
+      def initialize(model_path:, n_ctx: 4096, n_threads: 16, max_tokens: 512, n_gpu_layers: 0)
         @max_tokens = max_tokens
-        @handle = self.class.functions[:open].call(model_path, 0, n_ctx, n_threads) # 0 = CPU
+        @handle = self.class.functions[:open].call(model_path, n_gpu_layers, n_ctx, n_threads)
         raise "gguf: failed to load model at #{model_path}" if @handle.null?
       end

lib/elelem/plugins/gguf.rb

@@ -6,6 +6,7 @@ Elelem::Providers.register(:gguf) do
     model_path: ENV.fetch("GGUF_MODEL", File.expand_path("~/models/Qwen2.5-Coder-7B-Instruct-Q4_K_M.gguf")),
     n_ctx: Integer(ENV.fetch("GGUF_N_CTX", "8192")), # room for system prompt + tool results
     n_threads: Integer(ENV.fetch("GGUF_THREADS", "16")),
-    max_tokens: Integer(ENV.fetch("GGUF_MAX_TOKENS", "512"))
+    max_tokens: Integer(ENV.fetch("GGUF_MAX_TOKENS", "512")),
+    n_gpu_layers: Integer(ENV.fetch("GGUF_N_GPU_LAYERS", "0")) # 0 = CPU; 999 = offload all layers
   )
 end

Commit 63d20bb

Commit `63d20bb`