A lean, fully-tested LLM inference server built on llama.cpp — designed for the hardware you already have: shared vCPUs, free tiers, 2-core ARM boxes.
Reame is not the first inference server. It's the first one that treats cheap CPU
hardware as a first-class citizen instead of a fallback. Its thesis is simple:
On a CPU, never compute the same thing twice.
What Reame is for
Reame is built for narrow, repetitive AI workloads over your own data, on
hardware you already pay for — the case where the answer lives in the
context you provide, not in the model's general knowledge. That is exactly
where a small model matches a frontier one (we measured 100% accuracy on
long-context extraction with a 7B on a free 2-core ARM box) and where
Reame's memory makes request #100 cost a fraction of request #1.
What Reame is NOT for — said plainly, because trust is built here: a
general-purpose ChatGPT replacement (frontier reasoning and broad knowledge
need frontier parameter counts), agentic coding assistants, or creative
long-form writing at scale. If your task needs a 100B-class brain, buy one;
if it needs your documents processed privately, forever, at zero marginal
cost — that's a realm you can own.
🗂️ Persistent shared-prefix KV cache — prompt prefixes are snapshotted to disk
(zstd, checksummed, LRU-budgeted) and reused across different prompts, restarts
and processes. A system prompt is paid for once, by the first user.
📜 Palimpsest: the server remembers what it generated — every completed
generation feeds an on-disk n-gram archive; future requests draft from it
at zero cost. Domain workloads repeat themselves — let them pay off.
🎭 Il Suggeritore: grammar as a draft source — constrained decoding uses
structure to forbid tokens; Reame inverts it and uses structure to
propose them. List numbering, bullets and format tokens are speculated
for free on content nobody has ever generated before.
🔮 Self-regulating speculative decoding — a small draft model or zero-cost
n-gram lookup proposes tokens; the target verifies them in one batched pass.
Reame measures whether speculation pays on your hardware and switches it
off by itself when it doesn't.
🏛️ The Conclave: consensus as a quality knob — --best-of N generates N
candidate answers to the same prompt in one interleaved batch (one prefill,
cloned into the others via KV copy; every weight read shared) and elects the
winner by majority on the final result. The moment an absolute majority
agrees, the stragglers are stopped. Honestly measured: it squeezes roughly
one extra correct answer per quiz out of the model you already run — it
does not make a 1.5B out-reason a 3B (consensus fixes variance, not bias).
👥 Interleaved multi-user serving — N concurrent generations advance together
inside single multi-sequence batches, sharing every read of the model weights
(the cost that dominates memory-bound CPU decoding).
🌐 OpenAI-compatible REST API — /v1/completions, /v1/chat/completions,
SSE streaming, sessions, bearer auth, metrics. Point any OpenAI client at it.
⚡ Zero-config CLI — reame run qwen2.5-1.5b downloads the model once,
autoconfigures threads/KV/cache for the host and drops into a chat (or
--serve). No config file until you want one.
🧪 210 isolated test cases — every layer is mockable and tested without a
model; correctness of the multi-sequence, speculative and KV-clone paths is
pinned against real models in integration tests.
Measured, not promised
Every number below was produced by the shipped binary on the hardware named —
including the negative results that shaped the design.
Three negative results that matter. A 30B-class MoE on the maxed free tier answered the same extraction questions perfectly — and ten times slower than a 7B-A1B that also scored 100%: when the answer lives in the context, extra parameters buy nothing (MoE prefill touches nearly every expert, so the 3B-active discount vanishes on document reading). Use 30B-class models for hard reasoning in background batches, not for serving. On heavily oversubscribed shared vCPUs a draft
model runs as slowly as its target, so speculation is counter-productive there —
Reame detects this and disables it at runtime. And the Conclave does not
close the gap to a model twice the size on hard reasoning: majority voting
corrects random slips, not systematic misunderstanding — we measured a 1.5B ×5
land between the 1.5B and a 3B, never above the 3B. Benchmarks that only show
wins are advertising; these are engineering.
How it works
Shared-prefix disk cache. Prompts are split into fixed token blocks; a chain
hash keys a KV snapshot at every block boundary. A different prompt that shares
a prefix restores the longest cached boundary and decodes only its own tail.
Unlike GPU-resident prefix caches, snapshots live on NVMe: they survive restarts.
Self-regulating speculation. Classic Leviathan/Chen acceptance (the rejected
token is resampled from the residual distribution, so the output distribution is
exactly the target's), with two CPU-first twists: the draft source can be free
n-gram lookup mined from the prompt itself — ideal for extraction and rewrite
workloads — and a feedback controller adapts the draft length and turns
speculation off when measured acceptance or draft economics go negative.
The Conclave. --best-of N submits N attempts at the same prompt to the
interleaved scheduler: attempt 0 is the untouched anchor (greedy stays greedy),
the explorers shift seed and heat up. The scheduler notices the identical
prompts and clones the prompt KV instead of prefilling N times (copy the
donor's cache, decode only the last prompt token — argmax-verified equal to a
full prefill). Election is an exact-majority vote on each candidate's final
number, with a Jaccard text-medoid fallback for prose; the moment a majority
exists the remaining candidates are stopped mid-generation, and the CLI reports
CONCLAVE consensus=k/N so a caller can escalate only when the conclave split.
Use it as a quality knob: more accuracy from the model your hardware can
afford, paid in idle interleaved compute rather than a bigger model's RAM.
Quick start
run resolves a catalog name (or any local GGUF path), downloads to
~/.reame/models on first use and picks threads, KV quantization and cache
directory for the host. A config file is only needed when you want control.
Install
Homebrew (macOS / Linux):
Prebuilt binaries — Linux x64/arm64 and macOS arm64 on the
releases page
(runtime dependency: libzstd).
npm (npx reame): planned — binaries are already built per platform.
Build from source
Dependencies: CMake ≥ 3.16, a C++17 compiler, and for the server Boost (headers),
nlohmann-json and zstd:
Configuration highlights
API
A note on energy
Reame's footprint is watt-scale, not kilowatt-scale: it targets machines that
already exist and are already powered on — no new silicon is racked to serve your
model. We don't claim better joules-per-token than a saturated datacenter GPU —
we claim you don't need one.
Status & scope
Reame is young and deliberately opinionated and focused: CPU-only serving,
one model per process, correctness pinned by tests at every layer. Not goals:
GPU offload, training, model management UX. The llama.cpp submodule is pinned to
a known-good commit and bumped deliberately.
Documentation in Italian: docs/README.it.md.
Why Reame and not Ollama?
The laptop story is the same one command: reame run qwen2.5-1.5b downloads,
autoconfigures and chats — nothing to learn. From there the two projects
diverge: Ollama optimizes for running many models casually; Reame optimizes
for serving one workload seriously on hardware that costs nothing. The
difference is one sentence:
Ollama runs models. Reame remembers having run them.
General-purpose servers treat every request as brand new: compute, discard,
repeat. On a GPU that's fine — compute is cheap. On a cheap CPU, compute is
the most expensive thing you have, and throwing it away is the cardinal sin.
Everything in Reame attacks that: the disk prefix cache, the generation
archive, the grammar prompter, self-regulating speculation, interleaved
multi-user batches, the Conclave. None of it exists in Ollama.
The practical consequence: a Reame server gets faster the longer it runs.
The hundredth request costs a fraction of the first — the system prompt was
paid once, similar answers draft themselves from the archive, structure is
speculated for free. No other server has that property.
Support
Reame is free, MIT-licensed and built on nights and free-tier hardware. If it
saves you API bills or GPU rent, consider sponsoring
the work — sponsorships fund the roadmap: ARCA (the shared memory daemon),
warm-ahead prefill, and first-class MoE serving.
Acknowledgments
Reame stands on the shoulders of llama.cpp
(all tensor kernels; MIT). The disk-first cache thesis was inspired by
antirez's DwarfStar4 line of thinking; the speculative pipeline by DeepSeek's
DSpark work and the Leviathan/Chen speculative sampling theorem; archive
drafting is a shipped, persistent take on retrieval-based speculation (REST);
form drafting inverts grammar-constrained decoding. Ideas are cited, numbers
are ours.
License
MIT. Built on the shoulders of llama.cpp (MIT).