TL;DR: Since December 2025, llama-server can host every GGUF on your disk behind one OpenAI-compatible endpoint — models load on demand, each in its own process, and the least-recently-used one gets evicted when you hit the limit. The catch: the default limit is 4 simultaneous models, which no 24GB card survives, and two of the flags that control eviction have real, documented failure modes. This guide sets it up correctly on a single-GPU box.

What you’ll be able to do:

• Serve every model in a folder from http://localhost:8080/v1/chat/completions, switching between them with nothing but the "model" field in the request
• Write a --models-preset INI file that gives each model its own context size, GPU offload, and draft model — with one model preloaded at boot
• Set --models-max and --sleep-idle-seconds so a 24GB card never silently spills a second model into system RAM

Honest take: Router mode removes the single biggest reason home labs picked Ollama over raw llama.cpp — model switching without restarts. If you’re comfortable with a config file, switch; just set --models-max 1 on the command line (not in the INI) on any single-GPU box.

What router mode actually is

For most of llama.cpp’s life, llama-server had a one-model-per-process rule. You launched it with -m model.gguf, and serving a second model meant a second server on a second port — or a proxy like llama-swap juggling processes for you. The ggml team shipped router mode in December 2025, announced December 11, and it has matured over eight months of builds since. Everything below was verified against the official server README as of build b10256 (August 4, 2026).

The design is a supervisor pattern. Start llama-server without a -m flag and it becomes a router: a lightweight proxy that owns the port, discovers the GGUF files available to it, and spawns each requested model as its own child process. Requests carry a "model" field — the same field every OpenAI-compatible client already sends — and the router forwards each request to the matching child, starting it first if it isn’t running.

Because each model is a separate process, a crash in one model doesn’t take down the others or the router itself. That’s a real difference from the old single-process days, where one bad allocation killed everything on the port. If you’ve ever had a llama.cpp build issue take out an evening, you’ll appreciate the isolation.

Three ways to tell the router what models exist:

1. Cache discovery (zero config): with no arguments at all, it scans your llama.cpp cache (LLAMA_CACHE, default ~/.cache/llama.cpp) — every model you’ve ever pulled with -hf user/repo:tag shows up.
2. --models-dir /path: point it at a folder of GGUF files. Single-file models sit loose; multi-shard and multimodal models (with their mmproj file) each get a subfolder.
3. --models-preset config.ini: a preset file with per-model settings. This is the one you want — more below.

Five-minute setup on a single-GPU box

Assume a 24GB card and a models folder. Start the router with a hard cap of one loaded model (the reason for --models-max 1 gets its own section below):

llama-server --models-dir ~/models --models-max 1 --port 8080

List what the router can see:

curl -s http://localhost:8080/models | python3 -m json.tool

{
  "data": [
    {
      "id": "Qwen3.6-35B-A3B-Q4_K_M",
      "path": "/home/you/models/Qwen3.6-35B-A3B-Q4_K_M.gguf",
      "status": { "value": "unloaded" }
    },
    {
      "id": "gemma-4-26b-a4b-qat-Q4_0",
      "path": "/home/you/models/gemma-4-26b-a4b-qat-Q4_0.gguf",
      "status": { "value": "unloaded" }
    }
  ]
}

Model statuses cycle through unloaded, loading, loaded, sleeping, downloading, and failed. Now send a normal chat request — the router loads the model on first use (autoload is on by default):

curl -s http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "gemma-4-26b-a4b-qat-Q4_0", "messages": [{"role": "user", "content": "hello"}]}'

The first request pays the load time (a ~15GB Q4 model from a fast NVMe takes a few seconds; from SATA, considerably longer). Every request after that is instant routing. Ask for the other model and — because of --models-max 1 — the router unloads Gemma, loads Qwen, and answers. You can also drive it explicitly:

curl -X POST http://localhost:8080/models/load -d '{"model": "Qwen3.6-35B-A3B-Q4_K_M"}'
# {"success": true}
curl -X POST http://localhost:8080/models/unload -d '{"model": "Qwen3.6-35B-A3B-Q4_K_M"}'

The built-in web UI at http://localhost:8080 picked up a model dropdown in the same release — select a model there and it loads on demand, which makes the browser chat behave like Ollama or LM Studio without either installed. There’s also GET /models/sse, a server-sent-events stream of live status changes, if you’re building a dashboard on top.

Point any OpenAI-compatible client at the endpoint — Open WebUI, Continue.dev, or a coding agent — and every model in the folder is selectable from the client’s own model picker, because they all read GET /v1/models.

The preset file: per-model settings done right

--models-dir treats every model identically: same context size, same GPU offload, inherited from the router’s command line. Real model libraries don’t work that way — your 35B wants all layers on the GPU and a long context; your 8B utility model needs neither. The preset INI fixes this:

version = 1

[*]
n-gpu-layers = 99
c = 16384

[qwen-coder]
model = /home/you/models/Qwen3.6-35B-A3B-Q4_K_M.gguf
load-on-startup = true

[gemma-chat]
model = /home/you/models/gemma-4-26b-a4b-qat-Q4_0.gguf
c = 32768

[llama-draft]
model = /home/you/models/Llama-3.2-1B-Q4_K_M.gguf
c = 4096

llama-server --models-preset ~/models/presets.ini --models-max 1 --port 8080

Section names become the model IDs your clients see ("model": "qwen-coder"). The [*] section sets global defaults; per-model sections override it; command-line flags override everything. Two options exist only in presets: load-on-startup = true (preload a model at boot, so your daily driver never pays a cold start) and stop-timeout (seconds to wait before force-killing a child process on unload; default 10).

One flag-name warning: option names inside the INI are the long CLI names without dashes (c, n-gpu-layers, chat-template, model-draft). That last one matters — you can attach a speculative-decoding draft model per entry in the preset, which single-process llama-server made awkward to manage across model switches. Known gap: sampler settings currently can’t be set per-model in the preset — issue #23460 documents the samplers key being ignored, so keep sampling parameters in the client request. And don’t be surprised by a phantom default entry in GET /models when using a preset — that’s issue #22364, cosmetic but confusing.

The VRAM math the defaults get wrong

Here’s the part that generates the GitHub issues. --models-max defaults to 4 simultaneously loaded models, with LRU eviction when the limit is hit. Four models makes sense on a Mac Studio with 128GB of unified memory. On a discrete GPU it’s a trap, because the router counts models, not gigabytes — it has no idea what fits.

Q4_K_M weights alone, using sizes verified in our VRAM model guide:

Loadout	Weights on disk	24GB card with default --models-max 4
Llama 3.1 8B	~5 GB	Fine
Gemma 4 26B-A4B QAT + 8B	~15 + ~5 GB	Tight — KV cache for both eats the rest
Qwen3.6-35B-A3B + anything	~22 GB +	Second model spills or fails to load
Four mid-size models	40 GB+	Never fits — eviction won’t save you

When the second model doesn’t fit, you don’t always get a clean refusal. Depending on OS and backend you get a load error, a CUDA out-of-memory crash in that child process, or — worst — a silent spill into system RAM that craters your tokens per second while nvidia-smi still shows the GPU busy. On Windows, issue #19627 (February 2026, build 7941) reported exactly that pattern: with --models-max 1 set, rapidly switching models left the old model in memory and stacked new ones into RAM instead of evicting.

So on any single consumer GPU: set --models-max 1, or 2 if your two daily models genuinely fit together. Let LRU eviction do the swapping. The swap costs you a model-load each time you change (seconds from NVMe), which is the same price Ollama charges for the same convenience.

Real problem, real fix: the flag that only works on the command line

The most instructive router-mode failure came up in discussion #18939 (January 19, 2026). A user with two large models and limited VRAM set models-max = 1 in their preset INI, requested the second model, and got a load failure instead of an eviction — the router never unloaded the first model.

The fix: --models-max belongs on the command line, not in the INI. The INI configures models; router-level behavior comes from router-level flags. From the thread: “I have --models-max 1 added to the parent’s process, not to .ini file. And everything works, as expected. Only one model stays loaded.” Same thread, second lesson: the router does not unload idle models by default — a loaded model sits in VRAM forever until evicted or explicitly unloaded. If you want Ollama-style idle timeout, opt in:

llama-server --models-preset ~/models/presets.ini --models-max 1 --sleep-idle-seconds 600

That unloads any model after 10 idle minutes (the model shows as sleeping in /models). If you instead want your main model pinned 24/7, leave the flag off and use load-on-startup — the two behaviors compose nicely.

One more caveat for multi-user boxes: issue #20137 (March 2026, build b8189 on an M4 Max) demonstrated that under concurrent requests for different models, the limit check races — five simultaneous requests against --models-max 2 loaded 3+ models and peaked at 18+ GB of memory instead of the expected ~6 GB, and the count never settled back down. The issue was closed as stale rather than fixed, so treat --models-max as advisory under concurrency: if several people hit your server with different model names at once, preload the models you intend to serve and start the router with --no-models-autoload so a stray request can’t trigger a surprise load. (Autoload can also be controlled per request with ?autoload=false.)

Router mode vs Ollama vs llama-swap

	llama.cpp router mode	Ollama	llama-swap
Model switching	"model" field, on-demand child process	"model" field, on-demand	"model" field, proxy restarts backends
Default resident limit	4 models (--models-max)	3 × GPU count (details)	1 per group (YAML)
Idle unload	Off by default; --sleep-idle-seconds	On by default; 5-minute keep_alive	Configurable TTL
Engines	llama.cpp only	Own llama.cpp/MLX runtimes	Any OpenAI-compatible server (vLLM, etc.)
Config	INI preset + flags	Modelfiles + env vars	YAML

The takeaway from the comparison angle: router mode gives llama.cpp users Ollama’s headline convenience while keeping llama.cpp’s raw flag control and zero abstraction over GGUF files you manage yourself. llama-swap isn’t obsolete — it still earns its place when you’re mixing engines (a vLLM instance next to llama.cpp) — but for a pure llama.cpp home lab, the built-in router replaces it, as community write-ups like NemoClaw’s lab notes also concluded.

Note the idle-unload defaults are opposite: Ollama unloads after 5 minutes unless told otherwise (the classic cold-start complaint), while router mode keeps models pinned unless told otherwise. If your complaint with Ollama was reload latency, router mode’s default is what you wanted all along.

Hardware notes

Router mode changes nothing about the per-model VRAM math — one loaded 35B MoE still wants ~22GB plus KV cache — but it rewards disk speed more than single-model serving did, because model swaps become a routine event rather than a restart. If you’re swapping between two or three models all day on --models-max 1, the read speed of your drive is the swap time: a ~15GB Q4 model loads in roughly 2–3 seconds from a 7,000 MB/s-class NVMe versus ~30 seconds from a 550 MB/s SATA SSD (size ÷ sequential read, our arithmetic — drive details here).

A used RTX 3090 remains the sweet spot for this setup — 24GB fits any one of the strong mid-2026 open models at Q4, and router mode makes “any one at a time” a perfectly good way to live. Used prices ranged roughly $1,050–$1,254 in July 2026 per Best Value GPU’s tracking, up about 30% from winter — the supply squeeze hasn’t spared anything. No GPU yet, or want to test a multi-model workflow bigger than your card? A RunPod instance running llama-server --models-dir gives you the identical setup on an A100 for about $1.39/hour (Community Cloud, as of July 2026) before you commit to hardware.

For the models themselves, our open-source leaderboard covers what’s worth putting in the folder; if a brand-new GGUF refuses to load with an architecture error, that’s a stale-runtime problem, not a router problem — router mode ships with llama.cpp itself, so keeping one updated binary keeps every model current. The FOSS side of self-hosting stacks cleanly on top: one router endpoint behind a reverse proxy serves the whole house.

FAQ

Do I need a special build for router mode? No. It ships in standard llama-server binaries since December 2025. Update to a current release (b10256 as of August 4, 2026) and start the server without -m.

Does switching models lose my conversation? No — conversation state lives in your client. What you lose is the evicted model’s KV cache, so switching back mid-conversation re-processes the prompt. Frequent A/B switching between two models goes much better with both resident (--models-max 2) if they fit.

Can the router download models for me? Yes. POST /models with a Hugging Face model name triggers a non-blocking download into the cache, and -hf user/repo:tag at launch pre-registers cached models. DELETE /models?model=... removes one.

How is this different from just running two llama-server instances? One port, one endpoint, shared LRU memory management, and crash isolation per model. Two manual instances mean two ports, client-side switching, and you doing the VRAM arithmetic on every launch.

Does router mode work with multimodal models? Yes — put the model GGUF and its mmproj file in the same subfolder under --models-dir, and the router picks both up. With -hf downloads, the mmproj comes along automatically unless you pass --no-mmproj.

Sources

• llama-server README (router mode, presets, endpoints) — ggml-org/llama.cpp, GitHub
• New in llama.cpp: Model Management — Hugging Face blog, ggml-org
• Discussion #18939: Model router never unloads a model automatically — ggml-org/llama.cpp
• Issue #20137: —models-max not enforced under concurrent requests — ggml-org/llama.cpp
• Issue #19627: Router mode keeps loading more models into memory — ggml-org/llama.cpp
• Issue #22364: —models-preset creates an unexpected “default” model entry — ggml-org/llama.cpp
• Issue #23460: Unable to pass samplers to models-preset in router mode — ggml-org/llama.cpp
• llama.cpp releases (b10256, August 4, 2026) — GitHub
• llama-swap: transparent model-swapping proxy — mostlygeek, GitHub
• Router Mode in llama.cpp: Finally, a Native Alternative to Ollama’s Model Switching — Banandre
• RTX 3090 used price tracking — Best Value GPU
• A100 PCIe rental pricing — RunPod

Last updated August 4, 2026. Prices and specs change; verify current rates before purchasing.