How to self-host an open LLM in 2026: hardware and cost
A practical 2026 guide to running open-weight LLMs yourself: the VRAM math, which serving framework to use, what fits one GPU vs a server, and when self-hosting actually beats an API.
목차
You can self-host an open-weight LLM in 2026, but the model you can actually run depends almost entirely on VRAM. The frontier open models now rival closed APIs on quality — and if you’ve read our open vs cloud comparison, you know they cost a fraction to call. The catch: the best of them are data-center-sized. This guide covers the VRAM math, the serving stack, what fits which hardware, and whether running it yourself saves money.
- ~0.5 GB/B
- VRAM at 4-bit, per billion params~2 GB/B at FP16 (RunPod)
- 24 GB
- runs a strong 32B model at 4-bitone RTX 4090-class GPU
- $0.34/hr
- to rent that 24 GB GPURTX 4090, RunPod community
- vLLM
- the production serving defaultTGI was archived in 2026
Numbers below are as of July 2026 from primary sources — model cards, framework docs and cloud pricing pages. I’ve flagged anything unverified.
How much VRAM do you actually need?
Start with one rule of thumb: multiply the parameter count by the bytes per weight, then add 15-20% for the CUDA context, activations and KV cache (RunPod). Bytes per weight is 2 at FP16, 1 at FP8 or INT8, and 0.5 at 4-bit. Long context adds KV cache on top of that.
That single formula tells you what fits.
| Model size | 4-bit (~0.5 GB/B) | FP8 (~1 GB/B) | FP16 (~2 GB/B) |
|---|---|---|---|
| 8B | ~5 GB | ~9 GB | ~18 GB |
| 14B | ~8 GB | ~16 GB | ~31 GB |
| 32B | ~18 GB | ~35 GB | ~69 GB |
| 70B | ~40 GB | ~77 GB | ~154 GB |
FP8 is close to lossless; 4-bit costs measurable quality on math and reasoning, which is a fair trade when it’s the difference between fitting one GPU or not (SGLang docs).
The mixture-of-experts (MoE) models break this table, because their total parameters are enormous even though only a slice activates per token. DeepSeek V4-Pro is 1.6T total (49B active) and weighs about 862 GB in its native FP4+FP8 format — 8x H200 or a multi-node H100 cluster, and it won’t fit 8x H100 even at 4-bit (DeepSeek V4-Pro card). GLM-5 is 744B, Kimi K2 is 1T. Those are data-center models. Don’t plan to run them on a workstation.
Which serving framework?
Pick the framework by how many users you serve, not by hype.
| Framework | Best for | Note |
|---|---|---|
| vLLM | Production, concurrent serving | PagedAttention + continuous batching; ~19x Ollama under load (Red Hat) |
| SGLang | Agents, shared prefixes, structured output | RadixAttention; ~29% higher throughput on those workloads |
| Ollama | Local, single-user, five-minute setup | GGUF; no continuous batching |
| llama.cpp | Offline, CPU+GPU hybrid | GGUF; runs where CUDA can’t |
One deployment note that trips people up: Hugging Face archived its Text Generation Inference (TGI) repository on March 21, 2026, and now points users to vLLM, SGLang or llama.cpp (HF TGI repo). If a 2025 tutorial tells you to start with TGI, skip it. For most teams the answer is Ollama to prototype on a laptop, then vLLM once real traffic arrives.
Which model, by hardware tier?
What you can self-host splits cleanly into three tiers. Every model below is verified from its Hugging Face card.
| Tier | Hardware | Strong picks (license) |
|---|---|---|
| Consumer | one 24 GB GPU | Qwen3-32B at 4-bit ~18 GB (Apache-2.0); gpt-oss-20b ~16 GB (Apache-2.0); Gemma 3 27B, multimodal (Gemma license) |
| Workstation | 48-96 GB | MiniMax-M2, 230B/10B active (Modified MIT); gpt-oss-120b on one 80 GB H100 (Apache-2.0) |
| Server | multi-GPU | GLM-4.6 357B (MIT); Qwen3-Coder-480B (Apache-2.0); Kimi-K2 1T/32B (Modified MIT); DeepSeek-V3.2-Exp 685B (MIT) |
The workstation tier is where the value hides. MiniMax-M2 is 230B on paper but activates only 10B parameters per token, so it serves cheaply while scoring 69.4 on SWE-bench Verified (MiniMax-M2 card). OpenAI’s gpt-oss-120b fits a single 80 GB card and, by OpenAI’s own model card, matches or beats o4-mini on several evals (gpt-oss card).
Which model, by job?
Hardware aside, the right model depends on the task.
- Coding and agents: GLM-5 (744B, MIT) leads open coders at roughly 77.8% SWE-bench Verified if you can run it. Below that, Qwen3-Coder-480B (69.6%), MiniMax-M2 (69.4%), Kimi-K2 (69.2%) and GLM-4.6 (68.0%) sit close together (llm-stats). All of them do noticeably better inside a real agent framework than in raw chat.
- General chat and reasoning: DeepSeek-V3.2-Exp posts MMLU-Pro 85.0 and AIME 2025 89.3, and its Sparse Attention makes long context cheap (DeepSeek-V3.2 card). GLM-4.6 and gpt-oss-120b are strong all-rounders too.
- Long context: Qwen3-Coder-480B handles 262K natively and up to 1M with YaRN; DeepSeek-V3.2 keeps 128K cheap; Kimi K2 offers 256K.
- Multilingual: Qwen3 was trained across 119 languages (Qwen), the broadest open family. For a single-GPU multilingual model with vision, Gemma 3 27B covers 140+ languages — under Google’s more restrictive Gemma license.
Does self-hosting actually save money?
Sometimes. Renting a GPU is cheap by the hour — on RunPod, an RTX 4090 24 GB runs $0.34/hr on community cloud, an A100 80 GB $1.19, an H100 80 GB $1.99 (RunPod pricing).
Source: RunPod (July 2026)
| RTX 4090 24GB | 0.34 USD/hr |
|---|---|
| L40S 48GB | 0.79 USD/hr |
| A100 80GB | 1.19 USD/hr |
| H100 80GB | 1.99 USD/hr |
The trap is utilization. Those hourly rates only turn into a low cost-per-token if the GPU stays busy. At full saturation a 17B-class MoE serves for roughly $0.16-0.19 per million tokens on one A100 or H100, which undercuts almost any API. But an idle GPU still bills every hour, so at low or bursty volume you’re paying for silence — and a cheap open API like DeepSeek’s (about $0.14 in / $0.28 out per million tokens for its Flash tier) wins outright.
So the break-even is about steady volume, not headline price. If you can keep a GPU near full load, self-hosting is cheaper. If you can’t, rent nothing and call an API. The honest reason most teams self-host below that line isn’t cost at all — it’s that the data can’t leave their network, or they need a fine-tuned model, or they want latency they control.
The bottom line
Match the model to your VRAM first, then pick the framework. One 24 GB card running a 32B model at 4-bit, served by vLLM, covers a surprising amount of real work — and it’s about $0.34 an hour to try. Reach for the multi-GPU frontier models only when the task genuinely needs them, and self-host over an API only when the GPU stays busy or the data has to stay home.
Figures are as of July 2026 from primary sources — model cards, framework docs and RunPod pricing. VRAM estimates for the largest MoE models vary between community sources; verify against the model’s own Hugging Face checkpoint before you provision hardware.
FAQ
- What GPU do I need to run a good open LLM locally?
- A single 24 GB card (an RTX 4090-class GPU) runs a strong 24-32B model at 4-bit quantization. The frontier mixture-of-experts models are far larger and need multiple data-center GPUs.
- Should I use vLLM or Ollama?
- Ollama for a five-minute local, single-user setup. vLLM for production or concurrent serving — it delivers far higher throughput under load through continuous batching.
- Is self-hosting actually cheaper than an API?
- Only at high, steady volume with the GPU near full utilization. Below that break-even a cheap open-weight API is cheaper. Privacy, data sovereignty and latency are the non-cost reasons to self-host anyway.
- Which open model is best for coding if I self-host?
- Among self-hostable models in mid-2026, GLM-5 leads if you can run 744B parameters; otherwise Qwen3-Coder-480B, MiniMax-M2, Kimi-K2 and GLM-4.6 all cluster around 68-70% on SWE-bench Verified.
