Best GPU for local LLMs and AI (2026)

Most GPU buyer's guides are written around gaming benchmarks. This one isn't. Running a large language model locally is a different workload from rasterising triangles: the GPU streams model weights from memory one forward pass at a time, and the metrics that decide your experience are fundamentally different from what matters in a game. The number you should care about isn't TFLOPS — it's VRAM capacity and memory bandwidth.

This guide is built on a research-led synthesis of published vendor specifications, independent community benchmarks (llama.cpp tokens-per-second results, Ollama compatibility tables, and the collective output of forums like r/LocalLLaMA), and published buyer's guidance from sources including RunPod, Houtini, Spheron, BIZON, and Compute-Market. We do not run our own inference benchmarks — there are already thorough results across a wide range of quantisation levels published by the community and specialist sites, and duplicating them would add noise rather than signal. Where we cite a performance figure, it traces to an independent source; where we could not independently verify a number, we say so.

This guide was produced with AI assistance as part of Setup Quarterly's research workflow. Every vendor specification is cross-checked against primary sources before publication.

The picks at a glance

Why VRAM and bandwidth decide it — not TFLOPS

Token generation is a memory-bandwidth-bound workload. For each token the model generates, the GPU reads the entire set of model weights from VRAM and performs a relatively modest amount of arithmetic on them. The arithmetic is cheap; the memory read is the bottleneck. This is structurally different from a shader-heavy game scene or a dense matrix multiplication in a training run, where raw compute dominates.

Two numbers govern local LLM performance at the hardware level:

• VRAM capacity determines which models fit entirely on the GPU. A 7B model at 4-bit quantisation (Q4) weighs roughly 4–5 GB; a 70B model at Q4 weighs roughly 40–45 GB. If the model doesn't fit in VRAM, the overflow offloads to system RAM — and tokens-per-second drops by a factor of 10 or more, because RAM-to-GPU bandwidth is a fraction of on-card VRAM bandwidth. Fitting the model entirely in VRAM is the single most important hardware constraint you can address at purchase time.
• Memory bandwidth (GB/s) determines how fast the GPU streams those weights during generation, once the model fits in VRAM. Bandwidth is the primary determinant of tokens-per-second speed. The RTX 5090's 1,792 GB/s versus the RTX 4090's 1,008 GB/s is a roughly 78% bandwidth advantage — which translates into proportionally faster generation at equivalent model sizes, all else equal.

Raw compute (TFLOPS) matters for training and fine-tuning, where the GPU performs forward and backward passes through the full model repeatedly. For inference alone — running a local coding assistant, a chat model, or a local image model — bandwidth and VRAM are what you feel in daily use. A well-documented pattern in the local-LLM community is that an older card with more VRAM (say, a used RTX 3090 at 24 GB) consistently outperforms a newer, faster card with less VRAM (say, an RTX 4070 Ti at 12 GB) at any model size above the smaller card's ceiling, despite being slower on every traditional benchmark.

The tiered picks

Best overall consumer: NVIDIA GeForce RTX 5090

★★★★★ ★★★★★ 4.8/5 — Setup Quarterly editorial score (AI-inference suitability)

Check current RTX 5090 price on Amazon →

VRAM: 32 GB GDDR7 — Bandwidth: 1,792 GB/s (512-bit bus)

For: anyone who wants the maximum single-card local LLM capability available on the consumer market in 2026, without stepping up to professional hardware.

The RTX 5090 is the only consumer GPU shipping in 2026 with 32 GB of VRAM. That ceiling means it is the only GeForce card that can hold a 32B model at Q4 quantisation entirely on-GPU with comfortable headroom, and — paired with a second RTX 5090, with the model's layers split across both cards over PCIe — a consumer dual-GPU path to 70B Q4 without professional workstation hardware (consumer RTX 40- and 50-series cards have no NVLink, so PCIe layer-splitting is the standard multi-GPU inference path). Its 1,792 GB/s memory bandwidth, a roughly 78% step above the RTX 4090's 1,008 GB/s, translates directly into faster token generation at every model size that fits within VRAM; community llama.cpp results at equivalent Q4 model sizes generally show proportional generation speed differences consistent with the bandwidth ratio.

We cover the RTX 5090 in detail — including gaming, creator, power delivery, and street pricing — in our best graphics cards of 2026 guide. For AI workloads specifically, the value story is simpler: 32 GB is the ceiling, and the case for the RTX 5090 over the RTX 4090 hinges entirely on whether your workloads exceed ~30B parameters at Q4. Street pricing currently runs $2,400–3,000+ above NVIDIA's $1,999 MSRP due to GDDR7 supply constraints. If you're not sure whether you need 32 GB, the used RTX 4090 at 24 GB is the more considered buy.

The RTX 50 SUPER refresh is also worth watching if you're not in a rush — though its high-VRAM refresh has slipped to a rumored early 2027, and may be delayed further or cancelled amid a GDDR7 memory shortage.

24 GB class — serious local LLMs: NVIDIA GeForce RTX 4090

★★★★★ ★★★★★ 4.6/5 — Setup Quarterly editorial score (AI-inference suitability)

Check current RTX 4090 price on Amazon →

VRAM: 24 GB GDDR6X — Bandwidth: 1,008 GB/s (384-bit bus)

For: local LLM users who need a 7B–30B model range at Q4 with comfortable headroom, and who may also use the card for gaming, Stable Diffusion, or creative workloads.

The RTX 4090 remains the fastest 24 GB consumer card available in 2026, and 24 GB is a well-validated sweet spot for the current landscape of open-weight models. A single RTX 4090 fits Llama 3.1 8B, Mistral 7B, Code Llama 34B at Q4, and Mixtral 8×7B at Q4 entirely on-GPU without any offloading. The Ada Lovelace architecture has mature llama.cpp CUDA acceleration support, and frameworks including Ollama, LM Studio, and Jan work out of the box.

On the used market, the RTX 4090 tends to sit in the $1,100–1,400 range — overlapping with new RTX 5080 pricing. As a pure AI inference purchase, the right choice between them depends on whether you specifically need 24 GB: the RTX 5080 ships with 16 GB GDDR7 at 960 GB/s, which is faster per dollar at model sizes that fit within 16 GB, but it hits a hard ceiling at roughly 13B–20B Q4 where the 4090 keeps going. If 16 GB serves your current models, the 5080 is the better buy; if 24 GB is the target, read the RTX 3090 entry below before committing to new-market 4090 pricing.

24 GB value floor: NVIDIA GeForce RTX 3090 (used)

★★★★★ ★★★★★ 4.4/5 — Setup Quarterly editorial score (AI-inference suitability)

Check current RTX 3090 price on Amazon →

VRAM: 24 GB GDDR6X — Bandwidth: 936 GB/s (384-bit bus)

For: budget-conscious local LLM users who want the 24 GB VRAM ceiling without paying the RTX 4090 premium.

The RTX 3090 is the value anchor of the 24 GB tier. It carries the same 24 GB GDDR6X VRAM as the RTX 4090 — meaning the model-size ceiling is identical — at a street price that has settled below $1,000 on the used market (commonly $700–900 as of mid-2026, per community tracking on r/hardwareswap and completed eBay sales). The bandwidth deficit versus the 4090 (936 GB/s vs 1,008 GB/s) is real but modest: at equivalent Q4 model sizes, community llama.cpp results suggest roughly 10–15% slower token generation — a reasonable trade for a $400–600 purchase-price gap.

The honest caveats: the used GPU market carries real risks — no warranty, unknown compute hours, and potential mining-card exposure. If you buy a used RTX 3090 for LLM inference, test VRAM stability with a full-model load immediately after purchase. The Ampere architecture also lacks some of the newer CUDA kernels used by ExLlamaV2 and AWQ quantisation formats, though llama.cpp and Ollama support is solid. For the majority of local LLM users whose workloads sit below 30B parameters at Q4, a used RTX 3090 at sub-$1,000 is the most cost-efficient path to the 24 GB tier, and 24 GB is the most important single spec you can buy at a reasonable price in 2026.

Professional / workstation: NVIDIA RTX PRO 6000 Blackwell

★★★★★ ★★★★★ 4.9/5 — Setup Quarterly editorial score (AI-inference suitability)

Check current RTX PRO 6000 price on Amazon →

VRAM: 96 GB GDDR7 ECC — Bandwidth: ~1,920 GB/s (vendor-stated specification)

For: researchers, ML practitioners, and production teams running 70B models at high precision, or the largest currently-available open-weight models at Q4 on a single card.

The RTX PRO 6000 Blackwell is the professional-tier answer to the VRAM question. With 96 GB of GDDR7 ECC, it can hold a 70B model comfortably at Q8 or large-Q4 quantisation, and every currently-shipping open-weight model at Q4, including mixture-of-experts architectures in the ~109B effective-parameter range. The ECC (Error-Correcting Code) memory is relevant for production workloads where silent bit errors under sustained load could corrupt model outputs — less critical for casual inference, important for any production deployment or research that relies on reproducibility.

This is professional-grade hardware at professional-grade pricing. For most local LLM hobbyists and developers, the 32 GB RTX 5090 is the more appropriate ceiling. The RTX PRO 6000 Blackwell makes sense when 32 GB isn't enough, when ECC reliability is required, or when you're running a single-machine inference server for production workloads at 70B and above.

GPU comparison: VRAM, bandwidth, and model-size fit

For the broader GPU field — including cards not covered here, relative performance, and value-per-dollar rankings — see our 2026 GPU comparison and benchmark hierarchy.

VRAM → model-size rule of thumb

These are rough guidelines, not guarantees. Actual VRAM requirements vary by quantisation level, context window size, batch size, and framework overhead (llama.cpp, vLLM, Ollama all have slightly different memory footprints). The figures below are consistent with the ranges cited across llama.cpp documentation, RunPod's AI training GPU guide, Houtini's local-LLM hardware guide, and Spheron's NVIDIA-for-LLM overview.

• 8 GB: 7B at Q4 (barely; context window constrained to a few thousand tokens). No meaningful headroom above 7B at Q4.
• 12 GB: 7B at Q4 comfortably; 13B at Q4 at the limit. Good for light local inference on a budget; the ceiling matters at 13B+.
• 16 GB: 7B–13B at Q4 comfortably; 20B at Q4 possible with context constraints. Covers a broad swathe of available assistant and coding models.
• 24 GB: 7B–30B at Q4 comfortably; 34B at Q4 at the limit. The sweet spot for the current open-weight landscape — handles most widely-used models in a single card.
• 32 GB: 7B–34B at Q4 with headroom; 70B at Q4 is not possible on a single card (40–45 GB needed at Q4).
• 48 GB: the 70B doorway at Q4. Reachable via two 24 GB cards with the model's layers split over PCIe, or a dedicated 48 GB workstation card such as the NVIDIA RTX 6000 Ada.
• 96 GB: every currently-shipping open-weight model at Q4, including MoE architectures; 70B at Q8 or near-FP16.

Context window is the often-overlooked second variable. Running a 7B model at a 128K-token context window requires substantially more VRAM than the same model at a 4K context window, because the KV cache scales with context length. Framework implementations (llama.cpp, vLLM, Ollama) handle this differently. If you plan to use long-context models or large system prompts, add a meaningful overhead margin to the figures above and consult the model's quantisation notes for context-specific estimates.

Enterprise / datacenter cards: for reference

Three NVIDIA datacenter cards are worth knowing about for context, though they are not consumer purchases. They define the upper end of what "enough VRAM" looks like at scale, and contextualise why 96 GB is meaningful for a professional workstation card:

• NVIDIA H200 (141 GB HBM3e) — the current standard for cloud inference deployments; runs 70B at full FP16 with batching headroom.
• NVIDIA B200 (192 GB HBM3e) — NVIDIA's Blackwell datacenter flagship for cloud AI inference; runs 405B+ models at quantised precision.
• NVIDIA B300 (288 GB HBM3e) — the top of the Blackwell stack; designed for the largest models in single-card configurations.

These cards are accessible via cloud rental (RunPod, Lambda Labs, AWS) or direct purchase at six-figure prices — not relevant to a local inference build. They're listed here because understanding the datacenter tier helps calibrate how far the 96 GB RTX PRO 6000 Blackwell extends the consumer-accessible range.

Frequently asked questions

How much VRAM do I need to run a 70B model locally?

At 4-bit quantisation (Q4_K_M), a 70B model requires roughly 40–45 GB of VRAM. That puts the 70B tier squarely in the 48 GB+ range — two 24 GB cards with the model's layers split across both over PCIe, a dedicated 48 GB workstation card (such as the NVIDIA RTX 6000 Ada), or a single 96 GB card like the RTX PRO 6000 Blackwell. At full FP16 precision, a 70B model needs approximately 140 GB, which is datacenter territory (NVIDIA H200 or B200). For most local use cases, Q4 quantisation is the practical answer: you accept a modest quality trade-off in exchange for fitting the model on affordable hardware.

What is the best budget GPU for local LLMs?

A used NVIDIA RTX 3090 (24 GB GDDR6X) is the strongest budget pick for serious local LLM work. It shares the same 24 GB VRAM ceiling as the RTX 4090 — meaning the model-size range is identical — and has been available below $1,000 on the used market in 2026. The bandwidth deficit versus the RTX 4090 (936 GB/s vs 1,008 GB/s) results in roughly 10–15% slower token generation, based on community llama.cpp results at equivalent Q4 model sizes — a worthwhile trade for the price gap. Below 24 GB, the next realistic option is 16 GB (RTX 4080/5080), which handles 7B–13B Q4 comfortably but will hit a wall at 30B+.

Is the RTX 5090 good for AI and local LLMs?

Yes — the RTX 5090 is the best single consumer card for local LLM inference in 2026. Its 32 GB of GDDR7 at 1,792 GB/s lets it hold a 32B model entirely on-GPU at Q4, and two RTX 5090s — with the model's layers split across both cards over PCIe (consumer 40/50-series cards have no NVLink) — can handle 70B Q4 on consumer hardware. The bandwidth advantage over the RTX 4090 (1,792 vs 1,008 GB/s) translates directly to proportionally faster token generation at equivalent model sizes. The caveat is cost: street pricing runs $2,400–3,000+ above the $1,999 MSRP, driven by GDDR7 supply constraints. The case for the RTX 5090 over a used RTX 4090 for AI depends entirely on whether you need models larger than ~30B at Q4 — if not, the 4090 is the better value.

Do I need an NVIDIA GPU for local AI? Does CUDA matter?

CUDA still matters in practice, though its importance is shrinking. Most LLM inference frameworks — llama.cpp, Ollama, LM Studio — now ship mature Vulkan and ROCm backends that run on AMD and Intel Arc GPUs. For pure text generation with Ollama or LM Studio, a 24 GB AMD RX 7900 XTX is a genuine CUDA alternative. Where CUDA remains essential: fine-tuning and training (PyTorch GPU acceleration is significantly more mature on CUDA), and certain quantisation kernels (ExLlamaV2, AWQ) that ship CUDA-only at this writing. For inference-only use cases in 2026, AMD's ROCm stack is a viable path if you are comfortable with somewhat more setup and occasional driver friction. Intel Arc is usable for smaller models via Vulkan but is not a primary recommendation for serious local LLM work.

The bottom line: GPUs you can buy now

For most local LLM users, the decision simplifies quickly: buy the most VRAM you can afford, sized to the models you actually run. A used RTX 3090 at sub-$1,000 gives you 24 GB and handles the overwhelming majority of available open-weight models at Q4 — it's the most cost-efficient path to the 24 GB tier. The RTX 4090 adds bandwidth and new-card reliability at a higher price. The RTX 5090 raises the ceiling to 32 GB and delivers meaningfully faster token generation — the right buy only when you know you need 32B+ models on a single card. The RTX PRO 6000 Blackwell at 96 GB is for researchers and teams running 70B at high precision or MoE architectures in production.

For the full picture of what these cards do outside AI workloads — gaming, creative work, current pricing, and the complete field — see our best graphics cards of 2026 guide. The RTX 50 SUPER refresh could shift the value calculus in the 16–24 GB tier if it ships — but its timeline has slipped to a rumored 2027, and it may be cancelled. We track RTX 6090 rumors separately — that is a late-2027 story at the earliest, and it doesn't help you build a local inference machine today.

Related buyer's guides

Buyer's guide / Headphones

The best gaming headphones in 2026

Six headsets for six players: a planar-magnetic flagship, the best wireless value, an esports low-latency pick, an open-back audio champion, the wired value benchmark, and the lightest budget wireless option.

6 picks →

Buyer's guide / Mice

The best mouse in 2026

Six picks across gaming, productivity and ergonomics: a 48.9 g esports flagship, an innovative haptic-click mouse, the budget standout, the productivity king, a vertical ergonomic mouse, and a small-hands all-rounder.

6 picks →

Buyer's guide / Keyboards

The best gaming keyboards in 2026

Five picks for gamers: the Wooting 80HE Hall-effect flagship, a premium wireless magnetic board, a dependable mainstream mechanical, a $40 magnetic-switch surprise, and the high-polling wireless option — plus an AULA RGB family showcase.

5 picks →

Buyer's guide / Keyboards

The best keyboards for programming in 2026

Five picks for developers: a premium aluminium custom, a value board with the same firmware, a portable low-profile, a $40 budget surprise, and the quietest option for shared offices.

5 picks →

Buyer's guide / AIO coolers

The best AIO liquid CPU coolers in 2026

Five 360 mm AIOs for five different builders: the consensus value+performance winner, the LCD display for creators, the iCUE-Link cable cleanup, the RGB showpiece, and the budget cooler that beats coolers twice its price.

5 picks →

Buyer's guide / Standing desks

The best standing desks for developers and creators in 2026

Five picks for five workstations: the most-customizable premium desk, the price-to-performance value pick, the eco-conscious bamboo classic, the four-leg desk for triple-monitor builds, and the one that goes from box to built in fifteen minutes.

5 picks →

Buyer's guide / GPUs

The best GPUs in 2026

Five picks for five buyers: a halo card, a creator workstation, the last reasonable high-end value pick, the best 4K-per-dollar AMD card, and the budget GPU that finally has enough VRAM.

5 picks →