TL;DR: Qwen3.5-9B at Q4_K_M (5.68GB file, ~7GB VRAM at 8K context) is the best all-round model for an 8GB card in August 2026 — roughly 38 tok/s, image input included. Gemma 4 12B QAT (~6.6GB) is the writing-quality alternative. The real enemy on 8GB isn’t the model file, it’s the KV cache: keep context at 4K–8K or the card silently spills to system RAM and speed craters.

	Qwen3.5-9B Q4_K_M	Gemma 4 12B QAT Q4	Qwen3.5-4B Q4_K_M
Best for	Coding, chat, everyday work, vision	Writing quality, instruction following	Speed, agent loops, long context
Download size	5.68GB	~6.6GB	2.74GB
Comfortable context on 8GB	8K	4K–8K (16K possible, tight)	32K+
Speed on an RTX 4060-class card	~38 tok/s	~20+ tok/s	Fastest of the three
The catch	Tight fit — watch background VRAM use	Slowest pick; KV cache binds early	Noticeably dumber on hard tasks

Honest take: run Qwen3.5-9B as your daily driver and keep Qwen3.5-4B pulled for anything that needs a big context window. And if you’re shopping for a GPU right now rather than using one you own — don’t buy 8GB at all; a used RTX 3060 12GB costs less than a new RTX 4060 and sidesteps this entire article.

The 8GB tier is still where most home-lab GPUs live — RTX 3060 Ti, 3070, 4060, 5060, and a pile of laptop cards. Our evergreen VRAM tier guide covers the whole ladder; this is the August 2026 snapshot for the 8GB rung specifically, because the model lineup that fits it has changed twice this year and most advice floating around still recommends 2024-era 7B models.

The real VRAM budget on an “8GB” card

You don’t get 8GB. On a desktop with a single GPU, Windows and the desktop compositor hold several hundred MB before you load anything; browsers with hardware acceleration grab more. Plan around 7 to 7.5GB usable, less if you game or run YouTube on the same card.

Three things have to fit inside that:

1. Model weights — the GGUF file size, near enough
2. KV cache — grows linearly with context length, and it’s the thing everyone forgets
3. Compute buffers — a few hundred MB of scratch space

That’s why a 5.68GB model file needs about 7GB of VRAM at 8K context, per InsiderLLM’s measurements of Qwen3.5-9B. The 1.3GB gap is cache and buffers, not waste. It’s also why “the file fits, so it’ll run” is the most common wrong assumption in this tier.

The picks, by what you actually do

All-rounder and coding: Qwen3.5-9B

The consensus pick, and for once the consensus is right. The Q4_K_M GGUF is 5.68GB from Unsloth, it needs ~7GB VRAM with 8K context, and InsiderLLM measured roughly 38 tok/s on an 8GB GPU — comfortably past reading speed, fast enough for Cline-style agent use. It scores 82.5 on MMLU-Pro and takes image input natively, which no other model in this size class does well.

ollama run qwen3.5:9b
>>> /show info
  Model
    architecture        qwen3.5
    parameters          9.2B
    quantization        Q4_K_M

Verify it’s actually on the GPU before judging speed — ollama ps should show 100% GPU in the PROCESSOR column. Anything else means CPU spillover, covered below.

For coding assistance specifically, this is the model we’d wire into a local backend for Cline or Continue.dev on an 8GB card. It won’t replace a frontier API on multi-file architecture work — see the honest limits piece — but for boilerplate, refactors, and autocomplete it’s genuinely usable.

Writing and instruction-following: Gemma 4 12B QAT

Google’s QAT (quantization-aware training) checkpoints changed what fits here. The 12B at QAT Q4 runs in about 6.6GB of VRAM per Unsloth’s Gemma 4 QAT documentation — a 12B-class model on an 8GB card with near-original quality, because the quantization loss was trained out rather than bolted on. Community testing puts it at 20+ tok/s on an RTX 4060 with multi-token prediction enabled, and one documented setup ran it at 16K context on an 8GB GPU — though at that point the KV cache, not the weights, is the binding constraint.

It’s the slowest of our three picks, but Gemma’s prose and instruction-following are noticeably better than anything else that fits. If your workload is drafting, summarizing, or chat rather than code, this is the pick. Full background on the QAT release is in our Gemma 4 QAT guide; the even smaller E4B (~5GB) and E2B (~3GB) QAT variants exist if you need more headroom.

Speed and context: Qwen3.5-4B

The Q4_K_M file is 2.74GB, which leaves 4GB+ free on an 8GB card. That’s the whole argument: you can run 32K context — enough to actually hold a codebase file set or a long document — without the card breaking a sweat, and generation is the fastest of the three picks. Quality drops accordingly on hard reasoning, but for RAG answers over your own documents, summarization, and quick agent tool-calls, a 4B with room to think beats a 9B that’s been squeezed to 4K context.

What won’t fit (stop trying)

The saddest GitHub issues are people forcing these onto 8GB. The current 24GB-tier favorites are simply out of reach:

Model	Q4-class footprint	Minimum comfortable card
Qwen3.6-27B	~16.8GB	24GB (details)
Gemma 4 26B-A4B QAT	~15GB	16GB
GPT-OSS 20B	~13GB+	16GB (details)
Qwen3.6-35B-A3B	~22GB	24GB

Partial offload technically works — Ollama will split any of these across GPU and CPU — but on an 8GB card you’re offloading more than half the model and decode speed lands in single digits. If you want those models occasionally, rent: a RunPod A100 runs any of them full-speed for under $1.50/hour, which buys a lot of hours before an upgrade pays off.

The context trap: why your fast model suddenly isn’t

The problem you’ll actually hit, and the fix.

An 8B-class model’s KV cache costs roughly 0.125MB per token at FP16 — our derived figure from the architecture, consistent with the measured ~1.3GB overhead at 8K above. Push context to 32K and the cache alone wants ~4GB, on top of a 5.68GB model, on a card with ~7.5GB usable. It does not fit. What happens next depends on your setup, and both outcomes are silent:

• Ollama spills to CPU. The model loads, works, and generates at 4–8 tok/s instead of 38. ollama ps shows something like 54%/46% CPU/GPU in the PROCESSOR column instead of 100% GPU. Full diagnosis in Ollama not using GPU.
• Windows spills to shared GPU memory. The NVIDIA driver quietly pages VRAM to system RAM over PCIe and everything crawls — the sysmem fallback problem, invisible to ollama ps because the driver hides it.

Three fixes, in order:

1. Set context deliberately. 4096 is the safe default on 8GB; 8192 works with the 9B if the desktop is light. Ollama’s own defaults are VRAM-tiered now, but app settings and Modelfiles can override them upward without you noticing — the num_ctx priority order explains who wins.
2. Quantize the KV cache. OLLAMA_FLASH_ATTENTION=1 plus OLLAMA_KV_CACHE_TYPE=q8_0 (set on the service, not your shell) roughly halves cache VRAM at near-zero quality cost — that’s 8K context for the price of 4K. Details in the tokens-per-second tuning guide.
3. Drop a quant tier before dropping the model. Q3_K_M of the 9B still beats Q4 of a 4B on most tasks — the quantization quality-loss numbers show where the cliff actually is (Q3, not Q4).

If you’re seeing outright CUDA out of memory errors instead of silent slowdown, that’s a different failure with its own fix list.

Should you even buy an 8GB card in August 2026?

For new purchases: no. The math stopped working this year, and the memory-price surge made it worse in a counterintuitive way — new cards with modern memory got expensive while used 8–12GB cards barely moved.

Card	VRAM	Price (Aug 2026)	Verdict
RTX 3060 12GB used	12GB	~$289 avg, $278–$300 fair range (343 listings)	The budget buy. 50% more VRAM than a new 4060, for less money
RTX 4060 new	8GB	~$339	Fine GPU, wrong VRAM. Skip for AI
RTX 3070 used	8GB	~$228	Cheapest way in if 8GB is a hard budget ceiling
RTX 5060 Ti 16GB new	16GB	~$589 (launched at $429 MSRP)	The real upgrade — unlocks the 26B/QAT tier

The 5060 Ti 16GB briefly dipped below its $429 MSRP in late 2025; the 2026 memory squeeze pushed it to ~$589 new. Even at that price it’s the sensible new-card floor for local AI — we benchmarked what 16GB unlocks in the 8GB vs 16GB comparison, and why the 8GB RTX 5060 disappoints for AI specifically in the RTX 5060 review. One step further up, the used RTX 3090 at 24GB remains the value king if the budget stretches past $1,000.

But — and this is the point of this article — if an 8GB card is what’s in the machine right now, you don’t need to spend anything. Qwen3.5-9B on a three-year-old RTX 3070 is a better local assistant than anything that existed at any price in 2024.

Setup that works, start to finish

Tested against Ollama v0.32.6 (August 4, 2026):

# the daily driver
ollama pull qwen3.5:9b

# the long-context sidekick
ollama pull qwen3.5:4b

# confirm full GPU residency after first run
ollama ps
NAME           ID          SIZE    PROCESSOR    UNTIL
qwen3.5:9b     a1b2c3d4    6.8 GB  100% GPU     4 minutes from now

If PROCESSOR shows anything other than 100% GPU, lower the context or apply the KV-cache quantization above before blaming the model. On the open-source tooling side, aifoss.dev covers llama.cpp-based alternatives to Ollama if you want tighter control over offload behavior.

FAQ

Is a 7B/8B model from 2024–2025 (Llama 3.1 8B, Mistral 7B) still worth running? Only if a specific fine-tune you need lives on that base. Qwen3.5-9B outperforms them across the board at the same VRAM cost, and the older models lack current tool-calling formats.

Can I run a 14B model on 8GB? At Q3 with 4K context and KV quantization — barely, and partial offload will likely still kick in. It’s the worst spot in the lineup: you give up speed and context to gain less quality than the jump to a 12B QAT provides. Gemma 4 12B QAT is the better “biggest thing that fits.”

What about image generation on 8GB? Different budget entirely — SD 1.5 and SDXL run, FLUX needs aggressive offload. Start with the ComfyUI Windows setup guide.

Does more system RAM help an 8GB GPU? It prevents crashes when spillover happens, but it doesn’t prevent the slowdown — PCIe transfers are an order of magnitude slower than VRAM. 32GB of system RAM is plenty for this tier.

My laptop has an 8GB RTX 4070 — same advice? Same picks, one adjustment: laptop cards throttle harder, so expect 20–30% lower tok/s than the desktop figures here, and set the Windows power mode to performance before benchmarking.

Recommended Gear

• RTX 3060 12GB — the used-market budget pick: more VRAM than a new 4060, for less money
• RTX 5060 Ti 16GB — the sensible new-card floor for local AI in 2026
• RTX 3070 — cheapest usable entry to this tier on the used market
• RTX 4060 — fine for gaming-first builds that dabble in AI

Sources

• Qwen3.5-9B-Q4_K_M.gguf (5.68GB) — Unsloth on Hugging Face
• Qwen3.5-4B-Q4_K_M.gguf (2.74GB) — Unsloth on Hugging Face
• Best 8GB GPU Model: How to Set Up Qwen 3.5 9B — InsiderLLM
• What Can You Actually Run on 8GB VRAM? — InsiderLLM
• Gemma 4 QAT — Unsloth Documentation
• Gemma 4 QAT 12B runs at 16k context on 8GB VRAM — Open Weight LLM (Substack)
• Gemma 4 12B on an 8GB GPU — Coding Nexus (Medium)
• RTX 3060 12GB Used GPU Price & Fair Asking Range — ResalePrices
• RTX 4060 Price Tracker US, Aug 2026 — Best Value GPU
• RTX 3070 Price Tracker US — Best Value GPU
• RTX 5060 Ti 16GB Price Tracker US — Best Value GPU
• NVIDIA RTX 5060 Ti 16GB drops below MSRP to $419 — Wccftech
• Ollama Release Notes, v0.32.6 — releases.sh

Last updated August 5, 2026. Prices and specs change; verify current rates before purchasing.