TL;DR: When Ollama’s answers ignore your document, forget your instructions, or respond to only the end of a long prompt, the model isn’t dumb — Ollama silently threw away the front of your input because it exceeded the loaded context window. The server logs it (truncating input prompt) but the client never sees a warning. Check the real window with ollama ps, then raise it with num_ctx, OLLAMA_CONTEXT_LENGTH, or a derived model — and budget the VRAM the bigger window costs.

What you’ll be able to do:

• Confirm in under a minute whether truncation is your actual problem, using the CONTEXT column in ollama ps and one server-log line
• Raise the context window at the right layer — per session, per request, server-wide, or baked into the model — instead of guessing which knob is being ignored
• Keep a 32k–64k window inside a 24GB card using flash attention and K/V cache quantization, and know when no setting will save you

Honest take: This is the most common “my local model is stupid” complaint that has nothing to do with the model. Before you swap models, re-download weights, or blame quantization, spend sixty seconds checking what context window Ollama actually loaded. It is very often not the one you think you set.

The symptom: right answers to the wrong half of your prompt

The failure mode is distinctive once you know it. You paste a long document and ask a question about section 1 — the model answers as if section 1 doesn’t exist. Your RAG pipeline stuffs 8,000 tokens of retrieved chunks into the prompt and the model hallucinates instead of citing them. A coding agent sends its system prompt plus a repository map, and the model behaves like it never received the instructions at the top.

Ollama doesn’t return an error for any of this. The only place the truth appears is the server log:

level=WARN source=runner.go msg="truncating input prompt" limit=4096 prompt=8123 keep=5 new=4096

Reading that line: the loaded context window is 4,096 tokens (limit), your input was 8,123 tokens (prompt), Ollama kept the first 5 tokens (keep — typically the BOS/system-prompt head), and it trimmed everything else from the front until the last chunk fit (new). The model genuinely never saw roughly half of what you sent. Users have reported this exact behavior for years — a 17,624-token prompt cut to 2,048, RAG pipelines truncated at the default limit, and CLI runs silently trimmed — and the design hasn’t changed: truncate and proceed, don’t fail.

Where to find the log:

• Linux (systemd): journalctl -u ollama -f | grep -i truncat
• macOS: ~/.ollama/logs/server.log
• Windows: %LOCALAPPDATA%\Ollama\server.log (open with notepad or tail in PowerShell with Get-Content -Wait)

If that line shows up when your “dumb model” moment happens, you’ve found the whole problem. Everything below is the fix.

Why the window is smaller than you think

Every model ships with a maximum context it was trained for — ollama show llama3.1 will report something like context length 131072. But Ollama does not load the model at its maximum, because the context window costs VRAM (the K/V cache grows linearly with it). Instead it applies a default, and that default has changed enough times to make half the advice on the internet wrong:

Ollama era	Default context window
Older releases	2,048 tokens
Later releases	4,096 tokens
Since v0.15.5 (VRAM-tiered)	scales with detected VRAM, see below

The current behavior, per the official context-length docs, picks the default from your GPU:

Detected VRAM	Default context
Under 24 GiB	4,096 (4k)
24–48 GiB	32,768 (32k)
48 GiB or more	262,144 (256k)

Two consequences of that table are worth spelling out. First: on the cards most home labs actually run — an 8GB RTX 4060, a 16GB RTX 5060 Ti, a 12GB RTX 3060 — the default is still 4k tokens, which a single long email thread or one retrieved PDF chapter blows past. Second: on a 24GB card the new 32k default can swing the other way and exhaust VRAM you were counting on, because the K/V cache for 32k is allocated up front — and in early builds of the tiered feature it was multiplied again by OLLAMA_NUM_PARALLEL before that was fixed. If your GPU started spilling into system RAM right after an Ollama update, the new defaults are a prime suspect — that spillover failure mode is its own article: shared GPU memory slowdown fix.

Step 1: confirm what you’re actually running

Don’t set anything yet. First look at what’s loaded. Modern Ollama prints the live window in ollama ps:

$ ollama ps
NAME              ID              SIZE      PROCESSOR    CONTEXT    UNTIL
gemma4:latest     c6eb396dbd59    9.6 GB    100% GPU     131072     2 minutes from now

The CONTEXT column is the number that matters — it’s the window in effect for the running instance, after every default, environment variable, Modelfile parameter, and per-request option has been resolved. If it says 4096 while your prompts are 12,000 tokens, no amount of prompt engineering will help.

Two companion checks:

$ ollama show llama3.1
  ...
  context length    131072

That’s the model’s maximum — the ceiling you’re allowed to raise the window to. And the server log line from earlier is your ground truth for whether truncation fired on a specific request.

Step 2: raise the window at the right layer

There are four places to set the context window, and they override each other. Highest priority wins: an explicit per-request num_ctx beats the Modelfile, the Modelfile’s PARAMETER num_ctx beats the environment variable, and the environment variable beats the VRAM-tiered default. Most “I set it but nothing changed” reports are someone setting a lower-priority knob while a higher-priority one is pinned.

Per session (interactive CLI) — good for a quick test:

$ ollama run llama3.1
>>> /set parameter num_ctx 32768
Set parameter 'num_ctx' to '32768'

Per request (native API) — for your own scripts:

curl http://localhost:11434/api/generate -d '{
  "model": "llama3.1",
  "prompt": "…your long prompt…",
  "options": { "num_ctx": 32768 }
}'

This works on /api/generate and /api/chat. It does not work on the OpenAI-compatible endpoint — that trap gets its own section below.

Server-wide (environment variable) — the right fix for “every model I run should default higher”:

• Linux (systemd): sudo systemctl edit ollama, add Environment="OLLAMA_CONTEXT_LENGTH=16384" under [Service], then sudo systemctl restart ollama
• Windows: setx OLLAMA_CONTEXT_LENGTH 16384, then quit and restart Ollama from the tray
• macOS: launchctl setenv OLLAMA_CONTEXT_LENGTH 16384 and restart the app
• The desktop app also exposes a context-length setting in its UI on current builds — same knob, no terminal

Baked into a model (Modelfile) — the only fix that survives everything, and the one to use when a third-party tool talks to Ollama:

cat > Modelfile << 'EOF'
FROM llama3.1
PARAMETER num_ctx 32768
EOF
ollama create llama3.1-32k -f Modelfile

Then point your tool at llama3.1-32k. The derived model costs no extra disk for the weights (it references the same blobs) and carries its window wherever it’s used.

The OpenAI-compatibility trap (why your coding tool truncates)

Here’s the version of this bug that eats the most debugging hours in 2026: your editor plugin, agent framework, or RAG app speaks the OpenAI API format to Ollama’s /v1/chat/completions endpoint. You configured a 32k context window in the tool. The tool dutifully sends 20,000 tokens. Ollama truncates to its default anyway.

That’s because the OpenAI-compatible endpoint has no supported way to pass num_ctx — the OpenAI schema has no such field, requests for one have been open since 2024, and a PR to add it hasn’t landed. The tool’s “context window” setting usually only controls how much the client is willing to send, not what the server loads. Downstream projects keep rediscovering this — it’s a recurring bug report in agent gateways and IDE integrations alike.

The fix is always one of the two server-side layers: set OLLAMA_CONTEXT_LENGTH for everything, or (better) create a derived model with PARAMETER num_ctx baked in and select that model in your tool. We hit exactly this while wiring up Claude Code against Ollama, and it applies equally to Continue.dev, Open WebUI’s OpenAI-mode connections, and every LangChain/LlamaIndex app pointed at /v1. For the coding-tool side of this setup our sister site aicoderscope.com covers local backends in more depth.

Step 3: pay the VRAM bill (or shrink it)

Context isn’t free. The K/V cache — the per-token memory the model keeps for attention — grows linearly with the window and is allocated when the model loads. This is why Ollama’s tiered defaults exist at all, and why “just set 128k everywhere” ends in CUDA out-of-memory errors or silent spillover into slow system RAM.

You have two levers that shrink the bill dramatically, and they were built for exactly this. Enabling flash attention plus quantizing the K/V cache to 8-bit roughly halves context memory versus the default f16, per the feature’s author; q4_0 cuts it to about a third with a small, sometimes-noticeable quality cost. One multi-GPU user report in the implementing PR puts real numbers on it: the same model and window needed 40,960 MiB of K/V cache at f16 and 21,760 MiB at q8_0 — an 18.7GB saving from two environment variables:

OLLAMA_FLASH_ATTENTION=1
OLLAMA_KV_CACHE_TYPE=q8_0

Two caveats. The cache type is ignored unless flash attention is on — set one without the other and nothing changes. And on architectures where flash attention isn’t supported, Ollama silently falls back to f16, so verify the effect by watching VRAM in nvidia-smi before and after, not by trusting the variable.

If the window you need still doesn’t fit: a used 24GB RTX 3090 remains the cheapest way to buy K/V-cache headroom (our long-running case for it), spilling cache to fast storage is sometimes viable (NVMe KV-cache offloading), and for a one-off job that genuinely needs 128k+ context, an hour on a rented 48GB–80GB card via RunPod costs less than any hardware answer.

Which fix, when

Your situation	The right layer
Quick test in ollama run	/set parameter num_ctx 32768
Your own script on /api/generate or /api/chat	options.num_ctx per request
Everything on this machine should default higher	OLLAMA_CONTEXT_LENGTH env var
Third-party tool / OpenAI-format endpoint	Derived model: PARAMETER num_ctx + ollama create
Window fits the model but not your VRAM	OLLAMA_FLASH_ATTENTION=1 + OLLAMA_KV_CACHE_TYPE=q8_0
Nothing fits	Bigger card, NVMe offload, or rent the hours

And one boundary worth stating: if the model’s own maximum (ollama show) is 8k, no setting gives you 32k — raising num_ctx past the trained window makes output quality fall apart rather than erroring. If you’re seeing degeneration and repetition instead of forgetting, that’s a different failure with its own fixes: repetition and gibberish output.

FAQ

How do I know truncation happened on a specific request? Only the server log says so. Tail it while you reproduce: journalctl -u ollama -f on Linux, server.log in ~/.ollama/logs (macOS) or %LOCALAPPDATA%\Ollama (Windows), and look for truncating input prompt. The response itself carries no warning, and clients aren’t told.

Why does Ollama truncate from the front instead of erroring? Design choice inherited from its chat-first origins: in a rolling conversation, dropping the oldest turns and keeping the newest is reasonable. For single-shot prompts with instructions at the top, it’s the worst possible behavior — the instructions are exactly what gets cut. There’s no setting to make it error instead; your protection is loading a window big enough.

Does a bigger num_ctx slow generation down? The allocation itself mostly costs memory, not speed — but actually filling a long context slows prompt processing, and attention over long contexts reduces tokens/sec on consumer cards. If throughput matters, size the window to what you use, not the maximum. More tuning in Ollama slow? How to get more tokens per second.

I set OLLAMA_CONTEXT_LENGTH but ollama ps still shows the old number. Three usual causes: the Ollama server wasn’t restarted after the change (the tray app and ollama serve read the environment at startup); the model was still loaded with the old window (ollama stop <model> forces a reload); or the model’s Modelfile pins PARAMETER num_ctx, which outranks the environment variable — check with ollama show <model> --modelfile.

Is this the same as the “model requires more system memory” error? No — that one fires when the weights plus cache don’t fit and Ollama refuses to load at all, and it has its own fix path. Truncation is quieter: everything loads fine, and your prompt pays the price instead.

Recommended Gear

Products linked in this guide, for the context-window headroom problem specifically:

• RTX 3090 (used, 24GB) — still the price/VRAM king for holding a 32k–64k K/V cache alongside a 30B-class model
• RTX 5060 Ti 16GB — the budget pick, but remember it lands in the under-24GB tier: 4k default context until you raise it
• RTX 4060 — workable for 8B models at modest windows with q8_0 cache quantization

Sources

• Context length — Ollama official docs
• truncating input prompt limit=2048 prompt=17624 — ollama/ollama #7907
• ollama run silently truncating prompt — ollama/ollama #8099
• RAG pipeline truncated at default limit — ollama/ollama #10368
• Tiered context length can exhaust VRAM — ollama/ollama #14116
• Account for OLLAMA_NUM_PARALLEL in VRAM-based default context — ollama/ollama #14120
• allow num_ctx in the OpenAI API compatibility — ollama/ollama #5356
• support num_ctx for /v1/chat/completions — ollama/ollama #6137
• K/V Context Quantisation PR (with user VRAM reports) — ollama/ollama #6279
• Bringing K/V Context Quantisation to Ollama — smcleod.net
• Clarify supported architectures for Flash Attention / KV quantization — ollama/ollama #13337
• OpenAI compatibility — Ollama official docs

Last updated August 3, 2026. Ollama’s defaults have changed several times — verify against ollama ps on your own install before trusting any table, including ours.