The AI Infrastructure Atlas

Predict whether a model fits and how many users it can serve, so every later lab has a number to check against.

1. Download the model's config file. Any open model works; this path uses Qwen2.5-7B-Instruct throughout so the numbers stay comparable.
   Copy

   curl -LO https://huggingface.co/Qwen/Qwen2.5-7B-Instruct/resolve/main/config.json

2. Save this script as fit.py. It uses only the Python standard library.
   Copy

   # fit.py: will the model fit, and how many users can share the GPU?
   # usage: python3 fit.py config.json <params, billions> <bytes per weight> <GPU GB> <context>
   import json, sys
   cfg = json.load(open(sys.argv[1]))
   params_b, bpw, gpu_gb, ctx = float(sys.argv[2]), float(sys.argv[3]), float(sys.argv[4]), int(sys.argv[5])
   layers = cfg["num_hidden_layers"]
   kv_heads = cfg.get("num_key_value_heads", cfg["num_attention_heads"])
   head_dim = cfg.get("head_dim") or cfg["hidden_size"] // cfg["num_attention_heads"]
   kv_per_token = 2 * layers * kv_heads * head_dim * 2          # K and V, 16-bit cache
   weights = params_b * 1e9 * bpw
   budget = max(gpu_gb * 1e9 * 0.9 - weights - 1.5e9, 0)       # 10% headroom, ~1.5 GB runtime
   print(f"weights {weights/1e9:.1f} GB | KV cache {kv_per_token/1024:.0f} KiB per token")
   print(f"KV budget {budget/1e9:.1f} GB = {int(budget // kv_per_token):,} tokens "
         f"= {int(budget // (kv_per_token * ctx))} users at {ctx:,} tokens each")

3. Predict a 24 GB GPU in 16-bit weights and in 4-bit weights, at 8,192 tokens of context.
   Copy

   python3 fit.py config.json 7.6 2 24 8192     # BF16
   python3 fit.py config.json 7.6 0.5 24 8192   # 4-bit

Done when You have a written prediction: weights in GB, KV cache per token (about 56 KiB for this model), and users per GPU. Labs 4 and 7 check it.

Compare with Fanout 03.05–03.06 Calculate model, runtime and KV memory · Pro lab: Hardware and model fit planner

Serve an open model on your laptop behind the same OpenAI-compatible API you'll use on GPUs later.

1. Install llama.cpp and start its server. The -hf flag downloads the quantized model from Hugging Face.
   Copy

   brew install llama.cpp
   llama-server -hf bartowski/Qwen2.5-7B-Instruct-GGUF:Q4_K_M --port 8080

2. Call it like any hosted API.
   Copy

   curl -s localhost:8080/v1/chat/completions -H 'Content-Type: application/json' \
     -d '{"messages":[{"role":"user","content":"What is a KV cache?"}]}'

3. Read the server log after each request: it prints prompt-eval speed (prefill) and eval speed (decode) in tokens per second. Watch memory in Activity Monitor.

Done when The model answers over HTTP, and you've noted prefill tok/s, decode tok/s and memory used.

Compare with Fanout 01 Run Your First LLM

Measure the two phases yourself: long prompts make the first token slow, while decode speed hardly changes.

1. Save this as ttft.py: it streams one request and times every token.
   Copy

   # ttft.py: time to first token and inter-token latency on any OpenAI-compatible server
   # usage: MODEL=<name> python3 ttft.py <url> [prompt-file]
   import json, os, sys, time, urllib.request
   url = sys.argv[1] if len(sys.argv) > 1 else "http://localhost:8080/v1/chat/completions"
   prompt = open(sys.argv[2]).read() if len(sys.argv) > 2 else "Explain the KV cache in three sentences."
   body = json.dumps({"model": os.environ.get("MODEL", "local"), "max_tokens": 200, "stream": True,
                      "messages": [{"role": "user", "content": prompt}]}).encode()
   req = urllib.request.Request(url, body, {"Content-Type": "application/json"})
   t0, stamps = time.perf_counter(), []
   with urllib.request.urlopen(req) as resp:
       for raw in resp:
           line = raw.decode().strip()
           if not line.startswith("data:") or line.endswith("[DONE]"):
               continue
           choices = json.loads(line[5:]).get("choices") or [{}]
           if choices[0].get("delta", {}).get("content"):
               stamps.append(time.perf_counter())
   gaps = [b - a for a, b in zip(stamps, stamps[1:])]
   print(f"TTFT {1000 * (stamps[0] - t0):.0f} ms | {len(stamps)} chunks | "
         f"mean ITL {1000 * sum(gaps) / max(len(gaps), 1):.1f} ms | "
         f"decode {len(gaps) / max(stamps[-1] - stamps[0], 1e-9):.1f} tok/s")

2. Run it with a short prompt, then with a long one (paste about 3,000 words into long.txt).
   Copy

   python3 ttft.py
   python3 ttft.py http://localhost:8080/v1/chat/completions long.txt

3. Cross-check with llama-bench, pointing -m at the GGUF that llama-server downloaded (under ~/Library/Caches/llama.cpp). It reports prompt processing (pp) and token generation (tg) speeds separately.
   Copy

   llama-bench -m <path-to>.gguf -p 512,4096 -n 128

Done when A two-row table: short vs long prompt, with time to first token and decode tok/s. You can explain why time to first token grew and decode speed didn't.

Compare with Fanout 02 Understand One Token · 03 Prefill, Decode, KV Cache & Memory · 04.01 TTFT, TPOT, ITL

Feel the quantization trade-off on hardware you own before paying for GPUs.

1. Write 20 questions from your real domain (for a bank: policy, product and regulation questions) with expected answers, in questions.txt.
2. Serve the same model at 8-bit, then at 4-bit, and ask all 20 questions each time.
   Copy

   llama-server -hf bartowski/Qwen2.5-7B-Instruct-GGUF:Q8_0 --port 8080
   llama-server -hf bartowski/Qwen2.5-7B-Instruct-GGUF:Q4_K_M --port 8080

3. Record memory, decode tok/s and how many answers are still correct.

Done when A table with three rows (8-bit, 4-bit, your verdict). This 20-question set becomes your quality check for every later lab.

Compare with Fanout 11 Quantization & KV Compression · 17.04 llama.cpp & edge runtimes

Serve the same model with vLLM on an NVIDIA GPU and compare its memory numbers with your Lab 0 prediction.

1. On the GPU machine, install vLLM and serve the model.
   Copy

   pip install vllm
   vllm serve Qwen/Qwen2.5-7B-Instruct --max-model-len 8192 --gpu-memory-utilization 0.90

2. In the startup log, find the GPU KV cache size (in tokens) and the maximum concurrency at 8,192 tokens. Compare both with fit.py.
3. Run ttft.py against it, and keep nvidia-smi open in a second terminal.
   Copy

   MODEL=Qwen/Qwen2.5-7B-Instruct python3 ttft.py http://localhost:8000/v1/chat/completions long.txt
   watch -n 1 nvidia-smi

Done when Your Lab 0 prediction and vLLM's reported KV capacity are within about 20% of each other, and you can say where the difference went.

Compare with Fanout 06 Move to a Real Inference Engine · 09.02 Tune GPU memory & KV capacity

Find the load where latency stops meeting your target. This is the number that sizes the hardware.

1. Set a target first, for example: p99 time to first token under 1 s and p99 inter-token latency under 60 ms.
2. Sweep the request rate with vLLM's built-in benchmark, saving each run.
   Copy

   for R in 1 2 4 8 16 32; do
     vllm bench serve --model Qwen/Qwen2.5-7B-Instruct --dataset-name random \
       --random-input-len 1024 --random-output-len 256 --num-prompts 300 --request-rate $R \
       --percentile-metrics ttft,tpot,itl --metric-percentiles 50,99 --save-result
   done

3. Plot request rate against p99 latency and throughput. Mark the knee where p99 crosses your target.

Done when A latency–throughput chart with the knee marked, and the highest request rate that still meets your target.

Compare with Fanout 04 Benchmark Inference Correctly · 05 Break One GPU · Pro lab: Reproducible benchmark laboratory

Change one engine setting at a time and see what it does to the curve from Lab 5.

1. Prefix caching: send the same long prompt twice. The second time to first token should collapse. Then restart with caching off and repeat.
   Copy

   python3 ttft.py http://localhost:8000/v1/chat/completions long.txt   # run twice
   vllm serve Qwen/Qwen2.5-7B-Instruct --no-enable-prefix-caching          # then run twice again

2. Chunked prefill: rerun one Lab 5 sweep with long prompts (--random-input-len 4096) at two token budgets, and compare p99 inter-token latency.
   Copy

   vllm serve Qwen/Qwen2.5-7B-Instruct --max-num-batched-tokens 2048
   vllm serve Qwen/Qwen2.5-7B-Instruct --max-num-batched-tokens 8192

3. Write one sentence per setting: what it trades, and which workload (short chat or long-document RAG) prefers which value.

Done when Before-and-after numbers for prefix caching and for the batch token budget, each with a one-line explanation.

Compare with Fanout 07 Continuous Batching & Scheduling · 09 Optimize One GPU · 10 Prefill & Prefix Optimization

Repeat Lab 3 on the GPU, now measuring capacity as well as quality.

1. Serve a 4-bit AWQ checkpoint, then the original model with FP8 weights and an FP8 KV cache (FP8 needs an Ada or Hopper GPU, such as L4, L40S or H100).
   Copy

   vllm serve Qwen/Qwen2.5-7B-Instruct-AWQ
   vllm serve Qwen/Qwen2.5-7B-Instruct --quantization fp8 --kv-cache-dtype fp8

2. For each: note the KV cache size from the log, rerun the request rate at your Lab 5 knee, and ask your 20 questions.

Done when A table of BF16 vs AWQ vs FP8 showing KV tokens, throughput at the target, and correct answers out of 20, with the format you'd ship.

Compare with Fanout 11 Quantization & KV Compression (AWQ, GPTQ, FP8, KV quantization, quality regression)

Serve a model too big for one GPU, and measure what the split costs.

1. Check how the GPUs are linked: NVLink (NV#) or PCIe (PIX, PHB, SYS).
   Copy

   nvidia-smi topo -m

2. Serve a 32B model across both GPUs with tensor parallelism.
   Copy

   vllm serve Qwen/Qwen2.5-32B-Instruct-AWQ --tensor-parallel-size 2 --max-model-len 8192

3. Rerun the Lab 5 sweep and your 20 questions. Compare with the 7B on one GPU.

Done when You can state what the bigger model buys in correct answers and what it costs in latency, throughput and GPUs.

Compare with Fanout 18 Multi-GPU & Distributed Parallelism

Add the enterprise layer: one endpoint, routing between a small and a big model, guardrails, and metrics for audits and incidents.

1. Run both models (the 7B on port 8000, the 32B on 8001) behind a LiteLLM proxy.
   Copy

   # litellm.yaml
   model_list:
     - model_name: small
       litellm_params: {model: hosted_vllm/Qwen/Qwen2.5-7B-Instruct, api_base: http://localhost:8000/v1}
     - model_name: big
       litellm_params: {model: hosted_vllm/Qwen/Qwen2.5-32B-Instruct-AWQ, api_base: http://localhost:8001/v1}

2. Start it, then add a PII-masking guardrail (LiteLLM integrates Microsoft Presidio) and test that an account number in a prompt never reaches the model.
   Copy

   pip install 'litellm[proxy]'
   litellm --config litellm.yaml --port 4000

3. Scrape vLLM's /metrics endpoint and NVIDIA's dcgm-exporter (GPU telemetry) with Prometheus, then import the example Grafana dashboard from the vLLM repository. Watch the waiting-requests and KV-cache-usage gauges during a Lab 5 sweep.

Done when A dashboard showing time to first token, waiting requests, KV-cache usage and GPU utilization live, and a test showing PII is masked before it reaches the model.

Compare with Fanout 23.01–23.03 Load balancing & cache-aware routing · 24 Observability, Reliability & Security. Fanout has no guardrail or PII step.

Run the model server as a cluster service that schedules GPUs, slices them, and scales on queue length.

1. Install the NVIDIA GPU Operator with Helm.
   Copy

   helm repo add nvidia https://helm.ngc.nvidia.com/nvidia && helm repo update
   helm install --wait gpu-operator -n gpu-operator --create-namespace nvidia/gpu-operator

2. Deploy vLLM as a Deployment requesting nvidia.com/gpu: 1, with a readiness probe on /health.
3. On an A100 or H100 node, enable MIG through the operator's node label (for example all-1g.10gb), then fit a small model into one slice.
4. Autoscale replicas with KEDA on the Prometheus query for waiting requests, then rerun a Lab 5 sweep and watch new pods arrive.

Done when Two teams share one physical GPU through MIG slices, and replicas scale up under load and back down when idle.

Compare with Fanout 23.04–23.07 Autoscaling, Ray Serve, Kubernetes & KServe. MIG isn't in Fanout's lesson list.

Turn every lab into the one-page decision a bank's management actually needs.

1. Cost per million output tokens = GPU cost per hour ÷ (output tokens per second at your target × 3,600) × 1,000,000. For example, a $2/hour GPU sustaining 1,500 tok/s costs about $0.37 per million tokens.
2. For owned hardware, use purchase price ÷ (36 months × 730 hours) plus power and hosting as the hourly cost.
3. Write the memo: workload and target, chosen model and format, measured curve, GPUs needed with failure headroom, cost per million tokens vs a cloud API at equal quality, and the risks.

Done when A memo you could hand to a CTO, with every number traceable to one of your labs.

Compare with Fanout 25 Inference Economics & Capacity Planning · 30 Final Inference Engineering Capstone

Measure when guessing tokens ahead helps, and when it hurts.

1. Enable n-gram speculation, then compare single-user and high-load runs of Lab 5.
   Copy

   vllm serve Qwen/Qwen2.5-7B-Instruct \
     --speculative-config '{"method": "ngram", "num_speculative_tokens": 4, "prompt_lookup_max": 4}'

Done when You can show the concurrency level where the speedup disappears.

Compare with Fanout 16 Speculative Decoding

Write a softmax kernel in Triton, beat or match PyTorch, and read the profile.

1. Follow the official Triton fused-softmax tutorial, then benchmark it against torch.softmax.
2. Profile both with Nsight Systems and explain the difference in memory traffic.
   Copy

   nsys profile -o softmax python3 softmax.py

Done when A chart of your kernel vs PyTorch across row sizes, and one paragraph explaining the result from the profile.

Compare with Fanout 13 Triton & Kernel Engineering · 12 Profiling & GPU Performance