elelem-ggml

An elelem provider plugin backed directly by ggml-org/ggml, the tensor library llama.cpp itself is built on – vendored as a git submodule at vendor/ggml, same pattern as elelem-llama’s vendor/llama.cpp.

Status: Qwen2 inference works end to end

The C extension (ext/elelem/ggml/elelem.c) loads a GGUF file directly via gguf.h, builds a Qwen2 transformer graph by hand (RMSNorm, GQA attention with NEOX RoPE, SwiGLU FFN), and runs greedy generation with a persistent per-layer KV cache sized off context_length (or an explicit n_ctx). Prefill computes hidden states for the whole prompt but only projects the final position through the output layer; decode reuses cached K/V and only appends the new token. ggml itself has no tokenizer, so BPE tokenization and the ChatML template live in Ruby (lib/elelem/ggml/tokenizer.rb, lib/elelem/ggml/chat_ml.rb), parsing vocab and merges straight out of the GGUF metadata.

This is a minimal, readable reference implementation of transformer inference on ggml – CPU-only, dense Qwen2-family models only, greedy decoding only, no tool-calling. That scope is deliberate: the whole point is a forward pass small enough to read start to finish and know exactly what’s happening to your tokens. For GPU backends, real sampling, tool-calling, or broad architecture support, use elelem-llama instead.

Attention bias is optional and the output projection falls back to the tied token embedding when absent, so the loader also loads other dense architectures (e.g. Llama-family GGUFs) without crashing. It doesn’t decode them correctly: RoPE ordering is architecture-dependent (Qwen2 needs NEOX, Llama-family needs NORMAL) and there’s no GGUF metadata key that says which – llama.cpp derives it from the architecture name, which would mean hardcoding a per-architecture branch. That’s the line this gem doesn’t cross, so only Qwen2-family models produce correct output. Verified by direct experiment; see spec/elelem/ggml/provider_spec.rb.

See spec/ for what’s currently verified.

Setup

git submodule update --init --recursive
bundle install
bundle exec rake compile

Usage

Set ELELEM_GGUF_MODEL=/path/to/model.gguf, or place a .gguf file (or a default.gguf symlink, if you keep more than one) in ~/.agents/models/.

export ELELEM_GGUF_MODEL=/path/to/model.gguf
ruby -Ilib exe/elelem-ggml ask "hi"