LLMs & AI Infrastructure

The Cloud AI Problem

Every time you ask ChatGPT a question, type a prompt into Claude, or use Google's Gemini, your words travel across the internet to a datacenter you do not control. Someone else's servers process your thoughts. Someone else's policies decide what you can and cannot ask.

This is convenient. It is also a compromise.

Three problems define the cloud AI experience: privacy, cost, and freedom.

The Privacy Problem

When you send a message to a cloud AI, you are sharing that data with the provider. Your business plans, personal conversations, code, medical questions — all of it passes through servers owned by OpenAI, Google, or Anthropic. Even with privacy policies, your data exists on infrastructure you cannot audit.

For individuals, this might be acceptable. For businesses handling sensitive data — medical records, legal documents, proprietary code — it can be a dealbreaker.

The Cost Problem

Cloud AI is not free, and it adds up fast.

For a single user, $20/month feels reasonable. For a team of 50 engineers using AI-assisted coding, the bill becomes $1,000-5,000/month in API costs alone.^[est.]

The Freedom Problem

Cloud providers can change their models, raise prices, add content filters, or discontinue services at any time. You have no control over model availability, response quality, or censorship policies.

When OpenAI deprecated GPT-3.5-turbo, applications that depended on it broke. When providers add new safety filters, workflows that previously worked stop functioning.

Self-hosted AI eliminates all three problems. Your data never leaves your hardware. The cost is fixed (electricity + hardware amortization). And the models you download today will work identically forever — no one can change them remotely.

The Self-Hosted Alternative

Running your own AI means downloading open-weight language models and serving them on your own GPUs. The software stack looks like this:

Real-World Analogy: Cloud AI is like renting a car every day. Self-hosted AI is like buying your own car — higher upfront cost, but it is always in your driveway, no one reads your GPS history, and it works even when the internet is down.

Language Models: The Core Idea

A Large Language Model is a program that predicts the next word in a sequence. That is it. Every seemingly intelligent conversation, every piece of generated code, every creative story — all of it emerges from a system that is extraordinarily good at one task: given some text, predict what comes next.

When you type "The capital of France is", the model predicts "Paris" because it has seen millions of examples where those words are followed by "Paris." But this simple mechanism, scaled to billions of parameters and trillions of training examples, produces behavior that looks remarkably like understanding.

Neural Networks: Layers of Math That Learn

Under the hood, an LLM is a neural network — a mathematical structure loosely inspired by the brain. A neural network consists of layers of artificial "neurons," each connected to neurons in the next layer.

Each connection between neurons has a weight — a number that controls how strongly one neuron influences the next. A model with 32 billion parameters has 32 billion of these weights. During training, each weight is adjusted slightly to make the model's predictions better.

Real-World Analogy: Think of a neural network as a massive telephone switchboard. Each connection has a dial (weight) that controls how strong the signal is. During training, the AI turns these billions of dials slightly, getting better at routing information from question to answer. During inference (when you chat), the dials are locked in place — the AI just uses what it learned.

Neurons, Weights, and Activation Functions

Each neuron performs three simple operations:

1. Multiply each input by its weight
2. Sum all the weighted inputs together
3. Apply an activation function to decide whether to "fire"

The activation function introduces non-linearity — without it, stacking layers would be mathematically equivalent to a single layer. Individually, these operations are trivially simple. The power comes from scale — 32 billion weights organized into hundreds of layers create emergent capabilities that no individual weight explains.

Training vs Inference: Learning vs Using

These are the two fundamental phases of any AI model's life:

Training is the learning phase. The model reads enormous amounts of text (trillions of tokens) and adjusts its weights to minimize prediction errors. Training a frontier model costs millions of dollars in GPU compute.

Inference is the using phase. The weights are frozen. You send a prompt, the model processes it through its layers, and it generates a response.

How Training Actually Works: A Simplified Example

Imagine the model sees: "The cat sat on the ___"

1. The model predicts "table" with 60% confidence
2. The correct answer was "mat"
3. The error (loss) is calculated: the model was wrong
4. Through backpropagation, every weight in the network is adjusted slightly to make "mat" more likely next time
5. This happens billions of times across trillions of training examples

After seeing enough examples, the model learns not just facts but patterns, reasoning strategies, and even coding conventions.

The Problem with Older Architectures

Before 2017, the dominant architecture for processing text was the Recurrent Neural Network (RNN). These models processed text one word at a time, from left to right, carrying a "hidden state" that summarized everything they had seen so far.

This had a fatal flaw: by the time the model reached the 500th word, information about the 1st word was severely degraded. Long documents turned into mush.

Self-Attention: Looking at Everything Simultaneously

In 2017, a research paper titled "Attention Is All You Need" introduced the transformer architecture. Its key innovation was self-attention — the ability for every word in a sequence to directly attend to every other word, regardless of distance.

Key, Query, and Value: The Attention Mechanism

Self-attention works through three learned transformations of each word's representation:

• Query (Q): "What am I looking for?"
• Key (K): "What do I contain?"
• Value (V): "What information should I share?"

Real-World Analogy: Imagine you are in a library doing research. Your Query is your research question. You scan the Key (title) of every book on the shelf simultaneously. The books with matching keys get high attention scores. Then you read the Value (content) of the highest-scoring books.

Multi-Head Attention: Multiple Perspectives

Multi-head attention runs multiple attention mechanisms in parallel, each with different learned weights. A model with 64 attention heads examines the text from 64 different perspectives simultaneously.

Real-World Analogy: Think of reading a mystery novel. Old AI read one word at a time, forgetting earlier clues by the end. Transformers read the entire book at once, with "spotlight teams" (attention heads). One team tracks the suspect, another tracks the murder weapon, another tracks the timeline. They all share notes to solve the mystery together.

Positional Encoding: Knowing Word Order

Since self-attention processes all words simultaneously, it has no inherent sense of order. Positional encoding adds a unique mathematical signature to each position in the sequence. Modern models use Rotary Position Embeddings (RoPE), which encode relative distances between tokens rather than absolute positions.

The AI Timeline

2017

"Attention Is All You Need"

Transformers invented at Google — the architecture that powers all modern AI.

2020

GPT-3 (175B parameters)

The scale breakthrough — in-context learning emerges for the first time.

2022

ChatGPT — AI Goes Mainstream

Transformers meet 100 million users in 2 months.¹ The world changes.

2023-2024

The Open-Source Explosion

Llama, Mistral, Qwen release competitive models. MoE scales to 685B (DeepSeek V3.2) on consumer budgets.

2025

Reasoning Models Change the Game

DeepSeek-R1, Qwen-QwQ, and Llama 4 launch with MoE architecture. Chain-of-thought reasoning emerges.

2026

Specialists Beat Generalists

Open-weight models close the gap with proprietary (80.2% vs 80.9% on SWE-bench).² Model selection shifts from "biggest" to "best specialist for the task." Agentic coding agents go mainstream.

What Are Tokens?

LLMs do not process words. They process tokens — subword pieces that balance vocabulary size with representational efficiency.

"Hello, world!"        → ["Hello", ",", " world", "!"]        = 4 tokens
"def fibonacci(n):"    → ["def", " fib", "onacci", "(n", "):"] = 5 tokens
"Supercalifragilistic" → ["Super", "cal", "ifrag", "ilistic"]  = 4 tokens
"192.168.1.20"         → ["192", ".", "168", ".", "1", ".", "20"] = 7 tokens

A typical English word averages about 1.3 tokens. Code tends to be more token-dense than prose.

Context Windows: How Much the AI Remembers

The context window is the total amount of text the model can process in a single interaction — your input prompt plus the model's response. Everything outside the context window simply does not exist to the model.

Real-World Analogy: The context window is like a desk. A 4K context window is a school desk — you can fit a few pages. A 128K context window is a conference table — you can spread out entire codebases. A 1M context window is a warehouse floor.

The KV Cache: Memory for Attention

When the model generates text, it needs to compute attention between the new token and every previous token. The Key-Value (KV) cache stores the Key and Value matrices for all previous tokens. When generating the next token, the model only needs to compute the new token's Query and compare it against cached Keys — no re-computation needed.

Temperature and Sampling: Controlling Creativity

Temperature scales the probability distribution:

• Temperature 0.0: Always picks the highest-probability token (deterministic)
• Temperature 0.7: Moderate randomness (good default for most tasks)
• Temperature 1.5: High randomness (creative, sometimes incoherent)

Prompt: "The programmer wrote a function to"

Temperature 0.0 → "calculate the sum of two numbers"     (predictable)
Temperature 0.7 → "parse JSON data from the API response" (balanced)
Temperature 1.5 → "transcend the boundaries of recursion"  (creative)

For coding tasks, low temperature (0.1-0.3) produces more reliable, deterministic output. For creative writing, higher temperature (0.7-1.0) adds variety.

What Are Parameters?

Parameters are the learnable weights in a neural network — the billions of “dials” we discussed in Section 2. When someone says “Qwen2.5-Coder-32B,” the “32B” means 32 billion parameters.

More parameters means more capacity to store knowledge and patterns. But it also means more VRAM required, slower inference, and diminishing returns at extreme scale.

Parameter Count vs Intelligence

Here is the counterintuitive truth: more parameters does not always mean smarter.

A 32B coding specialist might score 92.7% on HumanEval (a coding benchmark). A model with far more parameters, like GPT-4o, scores roughly the same on coding tasks. How can a smaller model match a larger one? Specialization.

A 32B model trained exclusively on high-quality code outperforms a 70B model trained on general internet text — at coding tasks. The specialist beats the generalist at its specialty.

Dense vs Mixture of Experts (MoE)

In a dense model, every parameter is active for every token. A 70B dense model processes every token through all 70 billion parameters.

In a Mixture of Experts (MoE) model, only a fraction of parameters are active per token. A router network decides which “experts” (subsets of the model) handle each token.

MoE achieves the knowledge capacity of a large model with the compute cost of a smaller one. Mixtral 8x7B has 47B total parameters but only activates 13B per token, giving it the speed of a 13B model with the knowledge of a much larger one.

The MoE Tradeoff

Scaling Laws and Diminishing Returns

Research by OpenAI and others has established scaling laws: predictable relationships between model size, training data, compute budget, and performance.

The key finding: doubling model size does not double performance. Each doubling yields progressively smaller improvements. Going from 7B to 13B is a massive jump. Going from 70B to 140B is a modest one.

This is why the industry is shifting focus from “make it bigger” to “make it smarter”:

• Better training data (quality over quantity)
• Specialized training (code, math, reasoning)
• Architectural innovations (MoE, efficient attention)
• Post-training optimization (RLHF, DPO, reasoning chains)

What Is Quantization?

Neural network weights are numbers. In full precision, each weight is stored as a 32-bit floating-point number (FP32), using 4 bytes of storage. Quantization reduces this precision — storing weights in fewer bits — to shrink the model and speed up inference.

A 32B parameter model in FP32 requires:

32,000,000,000 parameters × 4 bytes = 128 GB

That does not fit on any single consumer GPU. Quantization is what makes large models accessible.

The Precision Ladder

AWQ: Activation-Aware Weight Quantization

AWQ is a state-of-the-art 4-bit quantization method developed by MIT researchers. It works by observing which weights matter most during actual model inference (the “activation-aware” part) and protecting those weights from aggressive quantization.

How AWQ works:

1. Run the model on calibration data (real prompts and responses)
2. Identify which weights produce the largest activations (most important weights)
3. Scale important weights up before quantization (protecting them from precision loss)
4. Quantize all weights to 4-bit
5. During inference, the scaling is reversed mathematically

The result: 75% reduction in model size with minimal quality degradation.

AWQ Quality Preservation

Zero degradation on HumanEval. The coding benchmark scores are identical because AWQ preserves the weights that matter for code generation.

Quantization Format Comparison

When to use which:

• AWQ + vLLM: Best choice for NVIDIA GPU production serving. This is what most dual-GPU setups should use.
• GGUF + llama.cpp: Best for Apple Silicon (M-series Macs) or CPU-only servers. Also good for mixed CPU+GPU inference.
• EXL2: Best for single-GPU setups with ExLlamaV2. Offers per-layer bit allocation for maximum quality at a target size.
• GPTQ: Legacy format, still widely available. Use when AWQ is not available for your model.

Marlin Kernel Acceleration

Marlin is an NVIDIA-optimized CUDA kernel specifically designed for 4-bit quantized inference. It is not a quantization method — it is a speed optimization for already-quantized models.

Standard 4-bit inference: the GPU must unpack 4-bit weights to 16-bit, compute, then handle the results. Marlin performs the computation directly on 4-bit data using specialized CUDA instructions, eliminating the unpack step.

Critical point: Marlin changes speed, not quality. The model produces identical outputs whether using Marlin or standard kernels. It is purely a computational optimization.

vLLM automatically uses Marlin kernels when serving AWQ-quantized models on supported NVIDIA GPUs.

What Is an Inference Engine?

An inference engine is the software that loads a model into GPU memory and serves it via an API. Think of it as the “runtime” — just like Python is the runtime for Python scripts, vLLM is the runtime for language models.

The inference engine handles:

• Loading model weights into GPU VRAM
• Processing incoming prompts (tokenization, attention computation)
• Managing the KV cache across multiple concurrent requests
• Returning generated text via an API endpoint

vLLM: The Production Standard

vLLM (pronounced “v-L-L-M”) is the most widely used inference engine for GPU-based deployments. Created at UC Berkeley, it introduced two key innovations that make it dramatically faster than alternatives.

PagedAttention: Traditional inference engines allocate a contiguous block of GPU memory for each request’s KV cache. If the request might need up to 128K tokens, the engine reserves memory for 128K tokens — even if the actual conversation only uses 2K tokens. This wastes enormous amounts of VRAM.

PagedAttention borrows the concept of virtual memory from operating systems. Instead of contiguous allocation, it stores KV cache entries in fixed-size “pages” scattered across GPU memory. Pages are allocated on demand and freed immediately when no longer needed.

Continuous Batching: Instead of waiting for all requests in a batch to finish before starting new ones, vLLM adds new requests to the running batch as old ones complete. This keeps GPU utilization high even with variable-length requests.

vLLM Performance

vLLM is designed for production serving — multiple concurrent users with guaranteed low latency. Ollama is designed for single-user simplicity.

Tensor Parallelism: Splitting Across GPUs

When a model is too large for a single GPU, tensor parallelism (TP) splits the model across multiple GPUs. Each GPU holds a portion of each layer and they communicate to produce the final result.

Interconnect Bandwidth

For inference, PCIe 5.0 bandwidth is sufficient — the bottleneck is VRAM, not interconnect speed. For training, NVLink’s 14x higher bandwidth becomes essential.

vLLM Sleep Mode: Instant Model Switching

A breakthrough feature in vLLM 0.8+ is sleep mode, which enables near-instant switching between different models on the same GPU.

# Put current model to sleep (offload to CPU RAM)
POST /v1/sleep
{"level": "l1"}   # L1 = weights in CPU RAM (fast reload)

# Wake up and reload
POST /v1/wake

This is a game-changer for setups with limited VRAM that need to switch between models — for example, switching between a coding model during work hours and a general chat model in the evening.

Engine Comparison

Major Open-Weight Model Families

The open-weight model ecosystem has exploded since Meta released Llama 2 in 2023. Dozens of families now compete across coding, reasoning, and general intelligence. Here are the most significant families as of early 2026:

Qwen (Alibaba Cloud, China)

The Qwen family from Alibaba’s research lab has become a dominant force in open-weight AI. Qwen2.5 and Qwen3 models consistently top benchmarks across coding, math, and general intelligence.

• Qwen2.5-Coder-32B-Instruct: A strong code generation specialist. Trained on 5.5 trillion code tokens. Scores 92.7% on HumanEval. 128K context window fits entire codebases. Best suited for code completion and generation tasks rather than agentic tool-use workflows.
• Qwen3-32B: General-purpose powerhouse with hybrid thinking modes (can switch between fast response and deep reasoning).
• Qwen3-Coder-Next: 80B MoE model (only ~3B active per token) with 262K context. Explicitly trained for agentic coding with tool-use and recovery behaviors.
• Qwen3-Coder-30B-A3B: Mid-range agentic coding model (30B total, 3B active). Strong tool-calling capability with smaller VRAM footprint.

License: Apache 2.0 (fully permissive, commercial use allowed, no restrictions).

DeepSeek (DeepSeek AI, China)

DeepSeek made headlines with V3 and the R1 reasoning model, both trained at a fraction of typical costs.

• DeepSeek-R1-Distill-Qwen-32B: A 32B model distilled from the 671B R1 reasoning model. Inherits deep chain-of-thought reasoning. Beats OpenAI’s o1-mini on math and logic benchmarks.^[3]
• DeepSeek V3.2: 685B MoE (37B active per token). Scores 73.1% on SWE-bench Verified and 74.2% on Aider Polyglot — the highest agentic coding scores among open-weight models.^[4]
• DeepSeek V3.2-Speciale: Variant optimized for specialized tasks with enhanced reasoning.

License: MIT (fully permissive).

Llama (Meta, USA)

Meta’s Llama family established the open-weight movement. Llama 3.3 and 4.0 represent the latest generations.

• Llama 3.3 70B: The general-purpose workhorse. Strong across all tasks, well-supported by every inference engine. Reliable tool-calling support with the llama3_json parser in vLLM.
• Llama 4 Scout (109B MoE): 109B total parameters, ~17B active. Latest-generation model with improved reasoning and native function calling support.
• Llama 4 Maverick (400B MoE): 400B total parameters, ~17B active. Flagship model requiring multi-GPU or aggressive quantization for consumer deployment.

License: Llama Community License (free for commercial use under 700M monthly active users).

Mistral (Mistral AI, France)

• Mixtral 8x7B: 47B total parameters, 13B active. The original “cheap but capable” MoE model.
• Mistral Large (2): 123B dense model, competitive with GPT-4.
• Mistral Small 3.1 24B: Efficient model for resource-constrained setups.

License: Apache 2.0.

Yi (01.AI, China)

• Yi-Coder-9B: Good coding quality for its size. Fits on a single 12GB GPU.
• Yi-34B: Strong general model at a moderate size.

License: Apache 2.0.

GLM (Zhipu AI, China)

Zhipu AI’s GLM family focuses on bilingual Chinese/English capability and has emerged as a strong coding contender.

• GLM-4.7: 120B+ parameters with strong coding and reasoning. Scores 94.2% on HumanEval.
• GLM-4.7-Flash: Lighter variant with thinking and tool-calling capabilities. Scores 59.2% on SWE-bench Verified.
• GLM-5: 744B MoE (40B active). Scores 77.8% on SWE-bench — one of the highest among open models.^[5]

License: MIT.

MiniMax (MiniMax AI, China)

• MiniMax M2.5: Scores 80.2% on SWE-bench Verified — the highest among all open-weight models, within 0.7 points of Claude Opus 4.5.^[2]
• MiniMax PRISM: Official uncensored variant for unrestricted use cases.

License: MIT.

Kimi (Moonshot AI, China)

• Kimi-Dev-72B: Strong development-focused model with function-calling support.
• Kimi K2 / K2.5: Latest generation. K2.5 scores 76.8% on SWE-bench Verified.

License: Modified MIT (commercial use allowed; attribution required above 100M users).

GPT-OSS (OpenAI, USA)

OpenAI’s first open-weight releases, focused on accessibility and transparency.

• gpt-oss-20b: Compact model requiring only 16GB VRAM in its native MXFP4 format. Designed for reliable tool calling.
• gpt-oss-120b: Larger variant requiring ~80GB VRAM. Stronger coding capability.

License: Apache 2.0.

IBM Granite (IBM, USA)

• Granite 3.3 8B / 34B: Optimized for enterprise deployment, SQL generation, and business analytics.
• Granite 4.0: Latest generation with improved code generation.
• Trained exclusively on license-permissible data — the safest choice for IP-sensitive deployments.

License: Apache 2.0.

Microsoft Phi-4 (Microsoft, USA)

• Phi-4 (14B): Matches much larger models on reasoning benchmarks despite its compact size. Only ~7GB VRAM with AWQ quantization.
• Phi-4-mini (3.8B): Ultra-compact with built-in function calling support.

License: MIT.

The Comprehensive Model Comparison

Commercial Model Comparison

The self-hosted models match or exceed commercial APIs on coding tasks, with zero per-token costs and full data privacy. The tradeoff is upfront hardware investment and maintenance responsibility.

Understanding Licenses

Open-weight vs open-source: “Open-weight” means the model weights are publicly available. “Open-source” means the weights, training code, AND training data are all available. Most “open” models are actually open-weight — the training process is proprietary.

How to Choose: The Decision Framework

Step 1: Define Your Task

• Code generation (autocomplete, scaffolding) → Qwen2.5-Coder-32B or Yi-Coder-9B
• Agentic coding (tool use, file editing, debugging) → Qwen3-Coder-Next or Qwen3-Coder-30B-A3B
• Reasoning/Math → DeepSeek-R1-Distill-32B (chain-of-thought reasoning)
• General chat → Qwen3-32B, Llama 4 Scout, or Llama 3.3 70B
• Lightweight/fast → Phi-4 or Yi-Coder-9B (5-7GB VRAM)
• Enterprise (IP-sensitive) → IBM Granite (license-permissible training data)

Step 2: Check VRAM Budget

• Single 12GB GPU → Yi-Coder-9B, Phi-4, or smaller
• Single 24GB GPU → Qwen2.5-Coder-32B-AWQ, Qwen3-Coder-30B-A3B-AWQ, DeepSeek-R1-32B-AWQ
• Dual 32GB GPUs → Qwen3-Coder-Next-AWQ (80B MoE), Llama 3.3 70B-AWQ, Llama 4 Scout-AWQ (all TP=2)
• 80GB+ (H100) → Up to 120B FP16, 400B+ in AWQ

Step 3: Consider Context Needs

• Short conversations (< 4K) → Any model
• Code files (8K-32K) → Need 32K+ context model
• Full codebases (32K-128K) → Qwen2.5-Coder, Qwen3-32B, Llama (128K native)
• Massive repositories (128K-262K) → Qwen3-Coder-Next (262K context)
• Ultra-long documents (1M+) → MiniMax, Kimi (API), Gemini API

GPU Architecture Fundamentals

A GPU (Graphics Processing Unit) was originally designed to render video game graphics — thousands of simple calculations in parallel. It turns out this same architecture is perfect for neural network inference, which also requires massive parallelism.

Modern AI GPUs contain three types of processing units:

• CUDA Cores / Stream Processors: General-purpose parallel processors. Handle the basic matrix multiplications that drive neural network computation. Thousands per GPU.
• Tensor Cores / Matrix Accelerators: Specialized units that perform matrix multiply-and-accumulate operations in a single clock cycle. 4-16x faster than CUDA cores for AI workloads. This is where actual inference math happens.
• VRAM (Video RAM): High-bandwidth memory attached directly to the GPU. This is where model weights, the KV cache, and intermediate computations live. VRAM is the single most important specification for AI inference.

Why VRAM Is the Bottleneck

For AI inference, the processing pipeline is:

1. Load model weights from VRAM into compute units
2. Load input tokens
3. Compute attention and feed-forward layers
4. Store KV cache entries back to VRAM
5. Output next token

The bottleneck is almost always step 1 — moving data from VRAM to compute units. This is called being memory-bandwidth-bound. More VRAM means you can load larger models. Faster VRAM bandwidth means you can feed the compute units faster.

Consumer Tier: Gaming GPUs for AI

Key insights for consumer GPUs:

• RTX 4090 (24GB): Runs most 7B-13B models in FP16, 32B models in AWQ/4-bit. The workhorse of the hobbyist AI community.
• RTX 5090 (32GB): Runs 32B models in AWQ comfortably with room for large KV caches. Two in tandem (64GB total) can run 70B AWQ models.
• Multi-GPU: Consumer GPUs communicate via PCIe 5.0 (64 GB/s). Fast enough for inference but limits training efficiency.
• Limitation: No NVLink support. Consumer motherboards typically support 2 GPUs maximum for AI workloads.

Prosumer Tier: Apple Silicon and AMD

Apple Silicon (M-series)

Apple Silicon’s unique advantage is unified memory — the CPU and GPU share the same memory pool. A Mac Studio M3 Ultra with 192GB can load a 70B model in FP16 without quantization. No NVIDIA GPU at any consumer price point can do this.

The downside: Apple’s GPU architecture is optimized for different workloads than NVIDIA’s Tensor Cores. Token-for-token, NVIDIA GPUs are faster, but Apple Silicon can load larger models.

AMD MI300X

The MI300X is AMD’s datacenter GPU with a massive 192GB of HBM3 memory. It can run 70B models in full FP16 on a single card. However, software support (ROCm) lags behind NVIDIA’s CUDA ecosystem.

Datacenter Tier: The NVIDIA AI Factory

Key datacenter-only features:

• HBM (High Bandwidth Memory): Stacked memory chips with 3-8x the bandwidth of consumer GDDR. An H100’s 3,350 GB/s feeds data to Tensor Cores fast enough to keep them fully utilized.
• NVLink: A direct GPU-to-GPU interconnect with 14x the bandwidth of PCIe 5.0. Makes tensor parallelism across multiple GPUs nearly as efficient as a single larger GPU.
• NVSwitch: A fabric switch connecting up to 576 GPUs in a single domain with full bisection bandwidth. The GB200 NVL72 rack operates as a single logical machine with 13.5TB of aggregate GPU memory.

The Interconnect Hierarchy

• PCIe 5.0 (consumer): Fine for 2-GPU tensor parallelism on inference. The ~2% overhead is negligible.
• NVLink (datacenter): Essential for 4-8 GPU tensor parallelism and training.
• NVSwitch (mega-scale): Required for 72+ GPU configurations.

VRAM Budgeting: The Math That Matters

Every GPU deployment needs a VRAM budget:

Total VRAM Needed = Model Weights + KV Cache + Activation Memory + CUDA Overhead

Where:
  Model Weights = Parameters × Bytes_per_Parameter
    - FP16: params × 2 bytes
    - INT4/AWQ: params × 0.5 bytes

  KV Cache = 2 × num_layers × num_heads × head_dim × max_context × batch_size × 2 bytes

  Activation Memory ≈ 1-2 GB (varies by batch size)

  CUDA Overhead ≈ 0.5-1 GB (driver, context, buffers)

Example: Qwen3-Coder-Next-AWQ on Dual 32GB GPUs (TP=2)

Model Weights (AWQ 4-bit): 80B × 0.5 bytes = 40 GB
  (Note: 80B total params, but ALL must be loaded even though only 3B active)
  Per GPU (TP=2): 40 / 2 = 20 GB

KV Cache (32K context, 1 user): ~2 GB
  Per GPU: 1 GB

Activation Memory: ~1 GB per GPU
CUDA Overhead: ~0.5 GB per GPU

Total per GPU: 20 + 1 + 1 + 0.5 = 22.5 GB
Available per GPU: 32 GB
Remaining per GPU: 9.5 GB  ← Fits with headroom

Example: Llama 3.3 70B-AWQ on Dual 32GB GPUs (TP=2)

Model Weights (AWQ 4-bit): 70B × 0.5 bytes = 35 GB
  Per GPU (TP=2): 35 / 2 = 17.5 GB

KV Cache (8K context, 1 user): ~2.8 GB
  Per GPU: 1.4 GB

Activation Memory: ~1.5 GB per GPU
CUDA Overhead: ~0.5 GB per GPU

Total per GPU: 17.5 + 1.4 + 1.5 + 0.5 = 20.9 GB
Available per GPU: 32 GB
Remaining per GPU: 11.1 GB  ← Fits, less headroom

Power and Cooling at Scale

At the consumer level, power is manageable. At the datacenter level, power and cooling become the dominant operating costs — often exceeding the hardware amortization cost.

The Attention Problem

The transformer’s self-attention mechanism has a fundamental limitation: it scales quadratically with context length. Processing a 128K token context requires 128K × 128K = 16 billion attention computations. Doubling the context to 256K quadruples this to 64 billion.

This O(n²) scaling is why long-context inference is so expensive and why researchers are exploring alternatives.

Flash Attention: Making Attention Faster (Today)

Flash Attention (by Tri Dao, Stanford) does not change the math of attention — it changes how the computation is organized in GPU memory. By reordering memory access patterns to maximize GPU cache utilization, Flash Attention computes exact attention 2-4x faster with 5-20x less memory overhead.

Flash Attention is already integrated into vLLM and is the default for all modern inference engines.

Gated DeltaNet: Beyond Transformers

Gated DeltaNet is a linear attention architecture that achieves O(n) scaling — doubling context length only doubles cost, not quadruples it.

Gated DeltaNet is still in the research phase, but early results show quality approaching transformers on standard benchmarks. If it works at scale, it could enable million-token context windows on consumer hardware.

YaRN: Extending Context Without Retraining

YaRN (Yet another RoPE extension method) allows models trained with short context windows to be extended to longer contexts without full retraining. It modifies the positional encoding (RoPE) to handle positions beyond the original training range.

A model trained with 4K context can be extended to 32K or even 128K using YaRN, with some quality degradation at extreme extensions. This is valuable because training long-context models is extremely expensive.

Speculative Decoding: Two Models, One Speed

Speculative decoding uses a small, fast model (the “draft” model) to predict multiple tokens ahead, then verifies them with the large, accurate model in a single forward pass.

The key insight: verifying multiple tokens in parallel is nearly as fast as generating one token. If the draft model’s predictions are mostly correct, the total throughput improves 2-3x.

Model Distillation: Teaching Small Models to Think Big

Distillation trains a small “student” model to mimic a large “teacher” model. The student learns not just the correct answers but the teacher’s probability distributions — capturing nuanced knowledge that would require a much larger dataset to learn directly.

Notable example: DeepSeek-R1-Distill-Qwen-32B is a 32B model distilled from the 671B DeepSeek-R1. It inherits the 671B model’s deep reasoning abilities despite being 21x smaller. This is why it beats OpenAI’s o1-mini on reasoning benchmarks.^[3]

LoRA and QLoRA: Customizing Models

LoRA (Low-Rank Adaptation) allows you to fine-tune a model by training only a tiny fraction of its parameters. Instead of modifying all 32 billion weights, LoRA adds small trainable matrices (typically 0.1-1% of total parameters) that adjust the model’s behavior.

QLoRA combines LoRA with 4-bit quantization, enabling fine-tuning on a single consumer GPU:

Use cases for fine-tuning:

• Adapting a general model to your coding style
• Teaching a model domain-specific terminology
• Improving performance on a narrow task (e.g., SQL generation)

RAG: Giving AI Access to Your Documents

Retrieval-Augmented Generation (RAG) solves a fundamental limitation: LLMs only know what was in their training data. They cannot access your private documents, recent emails, or local files.

RAG vs Fine-Tuning vs Prompt Engineering

Tool Use and Function Calling

Modern LLMs can use external tools through function calling. Instead of only generating text, the model can output structured requests to execute code, search the web, query databases, or call APIs.

How Function Calling Works

When a model supports tool use, it follows a four-step process:

1. Detect when a tool is needed — the model recognizes that the user’s request requires external action
2. Generate structured tool calls — instead of text, the model outputs JSON matching defined tool schemas
3. Wait for tool results — the external system executes the tool and returns results
4. Incorporate results — the model reads the tool output and crafts its final response

User: "What is the latest version of Python?"

LLM thinks: "I need current information. Let me search."

LLM outputs: {"tool": "search", "query": "latest Python version 2026"}

System executes: SearXNG search → returns results

LLM receives: "Python 3.13.2 released January 2026"

LLM responds: "The latest stable version of Python is 3.13.2,
              released in January 2026."

Models with Native Tool Use Support

Not all models are trained for function calling. Tool-use capability requires specific training. Models designed for tool use include:

• Qwen3-Coder-Next / Qwen3-Coder-30B-A3B: Trained specifically for agentic coding workflows with recovery behaviors
• Llama 4 Scout/Maverick: Native function calling support
• IBM Granite: Strong enterprise function-calling benchmarks
• Salesforce xLAM: Purpose-built for multi-turn tool use
• OpenAI gpt-oss: Designed for reliable agentic tasks

Agentic Coding: Beyond Code Completion

Agentic coding is the next evolution beyond code completion. Instead of generating code snippets in a chat window, agentic systems use tool calls to read project structures, create and modify files, run tests, execute shell commands, debug iteratively, and use version control.

User: "Fix the authentication bug in the login system"

Agent → tool_call: read_file("src/auth/login.py")
Agent → reads file, identifies the issue
Agent → tool_call: write_file("src/auth/login.py", fixed_code)
Agent → tool_call: run_command("pytest tests/test_auth.py -v")
Agent → reads test results: all pass
Agent → responds: "Fixed! The issue was missing input validation on line 47."

Tool Use vs RAG: When to Use Which

Use RAG when the answer is in documents you control. Use tool calling when the answer requires real-time data or computation.

Level 1: Single GPU, Single User

Typical setup: Ollama on a single RTX 4090 running a 32B AWQ model. Perfect for personal use — chatting, coding assistance, document analysis. Setup time: 30 minutes.

Limitations: One user at a time. Cannot run models larger than VRAM allows. No redundancy.

Level 2: Multi-GPU, Single Node

Benefits over Level 1:

• Run 70B models (AWQ) that do not fit on one GPU
• Model-switching strategies (sleep mode, multiple compose files)
• Can serve 5-10 concurrent users with vLLM

Model-switching pattern: When you need different models for different tasks, use mode-switching scripts. With vLLM sleep mode, switching takes 4-13 seconds instead of 50-160 seconds.

Level 3: Multi-GPU, Multi-Node

Pipeline parallelism has higher latency than tensor parallelism (network round-trips between nodes), but it is the only way to serve models that do not fit in a single node’s GPU memory.

Level 4: Kubernetes + vLLM Autoscaling

How it works:

1. A Kubernetes Horizontal Pod Autoscaler monitors GPU utilization
2. When demand exceeds capacity, new vLLM pods are launched
3. Each pod attaches to a GPU node and loads the model from shared storage
4. A load balancer distributes requests across all active pods
5. When demand drops, pods are scaled down to save resources

Self-Hosted vs API: The Cost Analysis

The crossover point for H100-class hardware is approximately 5-10 million tokens per day. Below that, API is cheaper. Above that, self-hosting wins and the savings compound rapidly with scale.

For consumer hardware, the crossover is much lower:

The consumer GPU advantage: hardware is a one-time purchase ($4,000-5,000 for dual RTX 5090 setup), and ongoing costs are just electricity. Even at minimal usage, self-hosting is cheaper after 3-4 months.

Data Sovereignty and Compliance

For regulated industries, self-hosting is not just about cost — it is about legal requirements:

• HIPAA (Healthcare): Patient data cannot leave your infrastructure without a BAA. Most AI API providers do not offer BAAs.
• GDPR (EU): Requires data to be processed within the EU. Self-hosting on EU-located servers guarantees compliance.
• ITAR (Defense): Certain technical data cannot leave the United States. Self-hosted AI on air-gapped networks is the only option.
• Financial Regulations: Many banks and funds require all data processing to occur on audited infrastructure.

Key Takeaways

After completing this lesson, you understand:

• LLMs are transformer-based neural networks that predict the next token, with intelligence emerging from scale and specialization
• Self-attention allows every token to attend to every other token, enabling understanding across long contexts
• The KV cache trades VRAM for speed, and grows linearly with context length
• AWQ quantization reduces model size by 75% with minimal quality loss, enabling large models on consumer hardware
• vLLM’s PagedAttention and continuous batching make production-grade serving possible on personal GPUs
• Model selection is about matching the right specialist to the right task, not choosing the biggest model
• GPU hardware ranges from $400 consumer cards to $3M datacenter racks, with VRAM being the critical bottleneck at every tier
• Emerging technologies (Flash Attention, speculative decoding, linear attention) will continue making AI more accessible
• Self-hosting becomes cost-effective at surprisingly low usage levels on consumer hardware
• Architect your applications with OpenAI-compatible APIs to freely switch between cloud and self-hosted

Resources and Further Reading

Official Documentation:

• vLLM: docs.vllm.ai
• Hugging Face Model Hub: huggingface.co/models
• NVIDIA CUDA Toolkit: developer.nvidia.com/cuda-toolkit

Foundational Papers:

• “Attention Is All You Need” (Vaswani et al., 2017) — The transformer paper
• “Language Models are Few-Shot Learners” (Brown et al., 2020) — GPT-3 / scaling laws
• “AWQ: Activation-aware Weight Quantization” (Lin et al., 2023) — Quantization method
• “Efficient Memory Management for LLM Serving with PagedAttention” (Kwon et al., 2023) — vLLM

Community Resources:

• r/LocalLLaMA (Reddit) — Self-hosted AI community
• Hugging Face Open LLM Leaderboard — Model benchmarks
• LMSys Chatbot Arena — Blind model comparisons

Next Lesson Preview

Continue to Lesson 05: Coding LLMs & Agentic AI to dive deep into coding benchmarks, the 22 model families landscape, agentic tool calling, vLLM parser matching, censorship analysis, and how to select and deploy the right coding model for any scenario.