Last updated:
The best GPU for local LLMs is the one with the most VRAM you can afford. In 2026, the RTX 5090 (32 GB) is the fastest single consumer card, a used RTX 3090 (24 GB) is the best value, and the RTX 5060 Ti 16 GB is the best budget pick. VRAM decides which models you can run; memory bandwidth decides how fast they answer.
Best GPUs for local LLMs at a glance
| GPU | VRAM | Memory bandwidth | Largest model at 4-bit (approx.) | Best for |
|---|---|---|---|---|
| RTX 5090 | 32 GB | 1,792 GB/s | 32B with long context | Fastest single card |
| RTX 4090 | 24 GB | 1,008 GB/s | 32B with short context | High-end, used market |
| RTX 3090 (used) | 24 GB | 936 GB/s | 32B with short context | Best value for 24 GB |
| RX 7900 XTX | 24 GB | 960 GB/s | 32B with short context | AMD on Linux |
| RTX 5080 | 16 GB | 960 GB/s | 14B–20B | Fast 16 GB card |
| RTX 5070 Ti | 16 GB | 896 GB/s | 14B–20B | Balanced gaming + AI |
| RX 9070 XT | 16 GB | 640 GB/s | 14B–20B | AMD 16 GB option |
| RTX 5060 Ti 16 GB | 16 GB | 448 GB/s | 14B | Best budget pick |
| Intel Arc B580 | 12 GB | 456 GB/s | 8B–12B | Cheapest entry point |
Bandwidth figures are manufacturer specifications. Prices change quickly in 2026 because of the memory shortage, so check our GPU price tracker before you buy.
How much VRAM do you need for a local LLM?
A simple rule: at 4-bit quantization (such as Q4_K_M in llama.cpp, Ollama and LM Studio), a model needs about 0.6 GB of VRAM per billion parameters, plus 1–4 GB for context (the KV cache). At 8-bit, it needs roughly double that.
| Model size | 4-bit (Q4) | 8-bit (Q8) | Minimum GPU |
|---|---|---|---|
| 3B–4B | 3–4 GB | 5–6 GB | 8 GB |
| 7B–8B | 5–6 GB | 9–10 GB | 8 GB (Q4), 12 GB (Q8) |
| 12B–14B | 8–10 GB | 15–16 GB | 12–16 GB |
| 24B–32B | 15–20 GB | 28–34 GB | 24 GB |
| 70B | 40–43 GB | 70–75 GB | 2 × 24 GB or 48 GB+ |
Longer context windows need more memory. If a model barely fits, it will run out of memory when you paste long documents. For more on VRAM in general, read how much VRAM do I need.
Best GPU for local LLMs by budget
Best overall: RTX 5090 (32 GB)
The RTX 5090 has 32 GB of GDDR7 and the highest memory bandwidth of any consumer GPU, so it runs 32B models with room for long context and generates tokens very quickly. It also handles image and video generation well. The downsides are its price, its 575 W power draw and its size.
Best value: used RTX 3090 (24 GB)
The RTX 3090 is still a favourite for local AI because 24 GB of VRAM is enough for 24B–32B models at 4-bit, and used cards cost much less than any new 24 GB card. Two RTX 3090s give 48 GB, which is enough for 70B models; llama.cpp and Ollama can split a model across both cards without NVLink. Read our used GPU buying checklist before you buy one.
Best mid-range: RTX 5070 Ti or RTX 5080 (16 GB)
Both cards have 16 GB of fast GDDR7, which is ideal for 8B–14B models at high quality and some 20B models. They are a good choice if you also game, since they are strong 1440p and 4K cards.
Best budget: RTX 5060 Ti 16 GB
The 16 GB RTX 5060 Ti is the cheapest new NVIDIA card with 16 GB. It is slower than the cards above because of its narrower memory bus, but it runs the same models. Avoid the 8 GB version for AI.
AMD options: RX 7900 XTX and RX 9070 XT
AMD cards offer more VRAM per dollar. ROCm support in Ollama, LM Studio and llama.cpp has improved a lot, and Vulkan backends work on Windows too, but setup is still easiest on Linux and some tools remain NVIDIA-first because of CUDA.
Intel Arc B580 (12 GB)
The Arc B580 is the cheapest new card with 12 GB. It runs 7B–8B models well through llama.cpp and IPEX-LLM, but software support is less mature than CUDA.
NVIDIA vs AMD vs Apple for local AI
- NVIDIA: the easiest option. CUDA works with almost every AI tool, including PyTorch, vLLM, llama.cpp and ComfyUI. See how to check if PyTorch sees your GPU.
- AMD: more VRAM for the money, but ROCm setup takes more effort.
- Apple Silicon Macs and unified-memory PCs: Macs with 64–128 GB+ of unified memory, AMD Ryzen AI Max systems and NVIDIA DGX Spark can load 70B models that no single gaming GPU can hold, but they generate tokens more slowly than a high-end GPU.
What matters most when choosing a GPU for LLMs
- VRAM capacity: decides which models fit. Never pick a faster card with less VRAM if it cannot hold your model.
- Memory bandwidth: decides tokens per second, because generating each token reads the whole model from memory.
- Software support: CUDA is still the safest choice.
- Power and cooling: AI workloads keep the GPU at full load for long periods. Watch your GPU temperatures and make sure your PSU is large enough.
Should you rent a cloud GPU instead?
If you need 70B+ models only now and then, or you want to fine-tune models, renting an H100 or A100 by the hour is often cheaper than buying hardware. Compare options in our GPU rental guide.
FAQ
Is 8 GB VRAM enough for a local LLM?
Yes, for 7B–8B models at 4-bit quantization with short context. For larger models or long documents, get 12–16 GB or more.
Can I run a 70B model on one GPU?
Not on a single consumer gaming card at good quality. A 70B model at 4-bit needs about 40–43 GB, so you need two 24 GB cards, a 48 GB+ workstation card or a unified-memory system.
Is the RTX 4090 still good for AI in 2026?
Yes. Its 24 GB of VRAM and high bandwidth make it one of the best cards for local LLMs, especially on the used market.
Do I need NVLink to use two GPUs?
No. llama.cpp, Ollama and LM Studio split models across GPUs over PCIe without NVLink.
Does system RAM help run LLMs?
Yes, tools can offload part of a model to system RAM, but those layers run much slower than on the GPU.
Related reading: see our guide on using an external GPU with a laptop.
