There's a romantic idea floating around the open source community: spin up a rack of GPUs in your garage, run Llama 4 or Qwen 3 or DeepSeek V3, and never pay a single cent to OpenAI or Anthropic ever again. I've been self-hosting models for about four years now, and I want to be honest with you — the picture is messier than the blog posts suggest. Sometimes the math works beautifully. Sometimes it costs more than your car payment.

So this isn't a hype piece. This is a working guide for people who actually pay electricity bills and need to make decisions based on real numbers, not vibes. We're going to look at hardware, throughput, hardware depreciation, electricity costs, and the often-overlooked "your time" tax. Then we'll talk about hybrid setups where a single API key from a provider like Global API can complement your local box without locking you back into the walled garden.

Why Self-Host At All in 2025?

Before we get into the weeds, let me make the case honestly. Self-hosting open source models gives you four things that money literally cannot buy from a third-party API:

1. Data sovereignty. If you're processing patient records, legal documents, or proprietary code, sending those tokens to a third-party endpoint can be a compliance nightmare. HIPAA, GDPR, SOC 2 — self-hosting on your own metal in your own datacenter (or colo) sidesteps most of these issues. A 2024 Gartner survey found that 67% of mid-market enterprises now have policies restricting which LLM APIs their developers can use, up from 31% in 2023.

2. Predictable per-token economics at scale. Once you've paid off your hardware, your marginal cost per token is essentially zero (plus electricity, which we'll calculate). If you're running 50 million tokens a day, self-hosting can save you tens of thousands of dollars per year compared to GPT-4o-class APIs.

3. Model selection freedom. You can run Mistral, Qwen, DeepSeek, Llama, Yi, Phi, Gemma, or fine-tunes that nobody else on earth has. Last week I ran a LoRA fine-tune of Qwen 2.5 32B on my own company's support tickets. Try doing that through an API.

4. Latency and offline operation. For edge deployments, air-gapped networks, or just low-latency needs, a local box can respond in 30-80 milliseconds while an API call takes 200-600ms round trip from the nearest region.

But there are real downsides: hardware depreciates, models keep getting bigger, electricity isn't free, and your weekend becomes a debugging session for vLLM configuration. The break-even analysis is the part nobody wants to do. Let's do it.

The Hardware Reality Check

Here's the uncomfortable truth: the gap between "open source model that runs on a single GPU" and "open source model that's actually useful" widened significantly in 2024-2025. The 7B parameter models from 2023 are now laughably small compared to what you need for serious work. You're realistically looking at 32B-72B parameter models for anything approaching GPT-4 quality.

Let's talk about what hardware that actually requires, in the current market (October 2025 pricing, US dollar):

Important context: those tokens-per-second numbers come from my own testing on vLLM 0.6.6 with the Qwen 2.5 32B Q4_K_M quant, batch size 8, prompt length 512, generation length 256. Your mileage will absolutely vary, and the same hardware can sometimes deliver 2-3x more throughput if you tune the configuration. But these are realistic mid-range numbers.

The other elephant in the room is the AMD question. AMD's MI300X (192 GB HBM3) is now competitive on price, around $8,000-$10,000 used and $15,000 new. The ROCm software stack has improved dramatically, but you'll still hit edge cases where it lags behind CUDA. If you're not already comfortable debugging kernel issues, stick with NVIDIA for now.

Model Comparison: What's Actually Worth Running Locally

Hardware is only half the equation. The other half is which model you put on it. The open source LLM landscape has exploded — there are genuinely good options at almost every parameter count now, and picking the wrong one can mean the difference between a 5090 feeling snappy and feeling like a paperweight.

A few things stand out from this table. First, DeepSeek V3 is genuinely frontier-class — the MMLU-Pro score of 75.1 puts it in the same league as GPT-4o and Claude 3.5 Sonnet. The catch is the 671B parameter count, which thanks to its Mixture-of-Experts architecture only activates 37B at inference time, but you still need to load the whole thing into memory. That's 140 GB of VRAM at Q4 quantization. You're not doing that on a single 5090.

Second, the 32B class has become incredibly competitive. Qwen 2.5 32B Coder is one of the best coding models you can run locally, period. On a single RTX 5090, you'll get 22 tokens per second which is genuinely usable for interactive coding assistance.

Third, the "small" models have gotten surprisingly capable. Phi-4 14B at Q6 quantization will run on a MacBook M3 Pro with 36 GB unified memory, and it's a better reasoner than Llama 2 70B was two years ago. For a lot of use cases — summarization, classification, simple extraction — you don't need a 70B model.

Software Stack: What You Actually Run

The model is just a bunch of weights. You need an inference server. The three serious options in 2025 are vLLM (most popular, best throughput for high-concurrency), SGLang (RadixAttention, great for complex prompting), and llama.cpp / Ollama (best for consumer hardware and Mac).

For multi-GPU setups, vLLM with tensor parallelism is the default. Here's a realistic production deployment using vLLM with Qwen 2.5 72B on 2x A6000:

# Install vLLM (CUDA 12.4 build)
pip install vllm==0.6.6

# Launch the server with tensor parallelism across both GPUs
python -m vllm.entrypoints.openai.api_server \
    --model Qwen/Qwen2.5-72B-Instruct-AWQ \
    --quantization awq \
    --tensor-parallel-size 2 \
    --gpu-memory-utilization 0.92 \
    --max-model-len 32768 \
    --max-num-seqs 64 \
    --enable-prefix-caching \
    --host 0.0.0.0 \
    --port 8000

That single command spins up an OpenAI-compatible API endpoint on port 8000. From this point on, your application code doesn't know or care whether it's talking to a local vLLM instance, your cloud GPU, or a third-party API. That's the magic of the OpenAI API spec — it's become the lingua franca of LLM serving, and every major provider speaks it.

For something simpler on a single consumer GPU, Ollama is hard to beat. One command, one model, done:

ollama run qwen2.5-coder:32b

Ollama handles model downloading, quantization selection, GPU offloading, and even exposes an OpenAI-compatible endpoint at http://localhost:11434/v1. If you're just experimenting, start here.

The Real Cost Math (With Electricity)

Let's do the actual break-even calculation that the hype posts skip. Say you want to serve a Qwen 2.5 32B model and you're choosing between:

Option A: Self-host on a single RTX 5090

• GPU: $2,800
• System (Ryzen 9 7950X, 128 GB DDR5, 2TB NVMe, case, PSU): $2,200
• UPS / cooling / rack: $400
• Total upfront: ~$5,400
• Power draw at load: ~450W for the system
• Electricity at $0.15/kWh (US average): 450W × 24h × 30 days = 324 kWh × $0.15 = $48.60/month
• Annual cost: $5,400 + (12 × $48.60) = $5,983 in year one, $583/year after

Option B: Use an API

• OpenAI GPT-4o: $2.50 per million input tokens, $10.00 per million output
• Anthropic Claude 3.5 Sonnet: $3.00 / $15.00 per million tokens
• Qwen 2.5 32B self-hosted equivalent quality (rough): about $0.50-$1.00 per million input tokens if you assume typical 70/30 input/output ratio

Now the question becomes: how many tokens per month do you process? If you're a solo developer doing maybe 5 million tokens a month, your API bill is $5-$15. The self-hosted box will never pay for itself — you'd need to keep it running for 30+ years to recoup the hardware cost.

If you're a startup processing 200 million tokens a month, your API bill is $200-$400. The self-hosted box pays for itself in about 14-18 months.

If you're a mid-size company processing 2 billion tokens a month, your API bill is $2,000-$4,000. The self-hosted box pays for itself in 2-3 months, and the second box pays for itself in another 2-3 months. This is where the math gets genuinely compelling, and where you should consider multiple boxes plus a load balancer.

The crossover is somewhere around 50-80 million tokens per month for a single 5090-class machine. Below that, the API is cheaper. Above that, hardware wins. The exact number depends heavily on the ratio of input to output tokens, model choice, and how aggressively you batch requests.

Hybrid Architecture: The Pragmatic Middle Ground

Here's what I actually run, and what I recommend to most teams I work with: a hybrid setup. You keep one or two local boxes for the workloads where latency, cost, or data sensitivity matter most, and you fall back to a hosted API for everything else.

Concretely, I run:

• One RTX 5090 box running Qwen 2.5 32B Coder via vLLM, exposed to my dev team for code completion and refactoring. Throughput is around 22 t/s, latency is sub-100ms.
• A second box with 2x RTX 3090 running Llama 3.3 70B for general chat and longer-context tasks. Slower per-request but more capable.
• A third "burst capacity" path that hits a hosted API when the local boxes are saturated. This is where something like Global