Skip to content
LeetLLM
Tracks
Advanced33 lessons

Inference and Serving

Find out why a model keeps people waiting, where the hardware spends its time, and which changes improve the experience.

Engineers responsible for GPU performance, latency, cost, local deployment, model gateways, and serving reliability.

You can measure kernels and serving systems, diagnose bottlenecks, and choose the optimization that fits each workload.

  1. 1Inference: TTFT, TPS & KV CacheA blank chat window and a busy GPU can both be right: prefill and decode hit different limits, and the KV cache decides how many sequences fit.Inference & Production ScaleHard47 min
  2. 2Multi-Query & Grouped-Query AttentionEight query heads don't need eight key and value histories. Sharing them changes how much cache a long conversation stores.Inference & Production ScaleHard54 min
  3. 3KV Cache & PagedAttentionPaging packs growing KV state into reusable blocks, but a full pool, a shared prefix, and a slow first token are three different problems.Inference & Production ScaleHard50 min
  4. 4Prefix Caching and Prompt CachingReuse a shared guide's KV state across requests only when the token prefix, the stored entry, and the tenant all match.Inference & Production ScaleHard35 min
  5. 5FlashAttention & Memory EfficiencyAt batch 8, 32 heads, and 8,192 tokens, FP16 scores need 32 GiB. FlashAttention computes the same attention without storing that matrix.Inference & Production ScaleHard59 min
  6. 6Continuous Batching & SchedulingA finished one-line completion can keep its GPU seat while a long plan runs. Refill that seat without letting the next prompt stall every live stream.Inference & Production ScaleHard48 min
  7. 7Scaling LLM InferenceSize a serving replica from memory bandwidth and KV residency, then pick the batch, precision, and speculation the latency target can afford.Inference & Production ScaleHard48 min
  8. 8Model Parallelism for LLM InferenceWhen a 70 GB weight file exceeds one GPU, sharding buys memory and spends it on traffic between devices.Inference & Production ScaleHard44 min
  9. 9Accelerator Architecture Field GuideA chip with twice the peak arithmetic can still leave decode almost unchanged. Follow one projection and see what to remeasure on NVIDIA, AMD, TPU, Trainium, and Apple silicon.Inference & Production ScaleHard46 min
  10. 10Correct GPU Profiling and BenchmarkingA 1.8x kernel speedup is credible when outputs match, timing covers completed GPU work, and another engineer can rerun the receipt.Inference & Production ScaleHard46 min
  11. 11GPU Kernel Performance Engineering LabA 40% faster transpose can still be wrong when the output is stale NaNs. Change one mechanism, then let correctness decide whether the timer counts.Inference & Production ScaleHard34 min
  12. 12GPU Parallel Primitives LabEight independent values are easy. Their sum is not. Follow the dependencies that make reduction, scan, and online softmax safe on a GPU.Inference & Production ScaleHard38 min
  13. 13GPU Compilers and Kernel LanguagesA correct kernel can still stall on its first call or fail because no image matches the GPU. The evidence is in the compiler artifacts.Inference & Production ScaleHard46 min
  14. 14GEMM From Naive CUDA to Tensor CoresThe same output can be numerically right and still come from the wrong addresses. Follow one product from a scalar thread to a Tensor Core tile.Inference & Production ScaleHard43 min
  15. 15Low-Precision GPU ProgrammingA scaled FP8 round trip can turn 500 into 10 with no NaN. The scale, the accumulator, and the kernel contract decide whether that loss stays visible.Inference & Production ScaleHard37 min
  16. 16GPU Runtime Execution LabWhen short CUDA kernels finish before the host submits the next one, streams, events, and graphs close that gap only if completion and addresses stay correct.Inference & Production ScaleHard32 min
  17. 17AI-Generated Kernel EngineeringA 1.4x speedup on one H100 shape can fail the next row width. Promote that generated kernel only after the contract and the hidden checks agree.Inference & Production ScaleHard39 min
  18. 18Model Quantization: GPTQ, AWQ & GGUFFour-bit codes can shrink a 54 GB weight payload toward 14 GB. Learn how quantization methods, tensor encodings, and runtime support determine the result.Inference & Production ScaleHard36 min
  19. 19Local LLM DeploymentA 7.6 GB model can load on a 24 GB GPU and still run out of memory once cache, workspace, and long prompts share the card.Inference & Production ScaleHard35 min
  20. 20SLM Specialization & Edge DeploymentFit a small model to an offline device job, then check whether quality, heat, privacy, and updates still hold after the weights leave the server.Inference & Production ScaleHard50 min
  21. 21Speculative DecodingGuess several tokens with a cheap proposer, then accept or correct them so the target's distribution stays the same in exact arithmetic.Inference & Production ScaleHard41 min
  22. 22Long Context Window ManagementA year's traces can fit the advertised window and still hide the note that matters, or cost more cache than the weights.Inference & Production ScaleHard52 min
  23. 23Mixture of Experts ArchitectureEight experts can mean two feed-forward calls per token and a full expert pool still sitting in memory.Inference & Production ScaleHard52 min
  24. 24Reasoning & Test-Time ComputeSpend extra inference compute on a hard prompt only when another sample, a revision, or a check beats the latency and cache cost.Inference & Production ScaleHard54 min
  25. 25LLM Cost Engineering & Token EconomicsPolicyOps forecasts USD 717.18 against a USD 750 budget, but a stored answer and a prefix read are not the same savings.Applied LLM EngineeringMedium34 min
  26. 26Model Gateways, Routing, and FallbacksThe healthy backup cannot return the cited answer access-R900 needs. A faster reply would still be the wrong route.Applied LLM EngineeringMedium32 min
  27. 27LLM Observability & MonitoringSix responses return HTTP 200 and still include two unsupported approvals. Latency, cost, and traces matter only once that claim is visible.Applied LLM EngineeringMedium32 min
  28. 28Model Versioning & DeploymentAn engineer reverts the prompt in Git, but the image and the evidence have already moved. The release ID has to name the combination that is actually answering.Applied LLM EngineeringMedium33 min
  29. 29GPU Serving & AutoscalingAt 9:00 a release freeze sends 100 incident summaries at one warm GPU. Queue age, not utilization, decides whether to add replicas.Inference & Production ScaleHard66 min
  30. 30GPU Failure OperationsA CUDA out-of-memory alert names one failed allocation, not whether to restart the replica, quarantine the GPU, or drain the node.Inference & Production ScaleHard36 min
  31. 31Distributed Inference Data PlaneAn 8,192-token prompt moves 2.50 GiB of KV state, not a small routing note. See why a correct-looking copy can still be unsafe to reuse.Inference & Production ScaleHard29 min
  32. 32Serving Benchmark EngineeringThe server finishes more requests per second, but the first word is slower and some fast answers are invalid. Keep the offered load, the client's timing, and every failure.Inference & Production ScaleHard31 min
  33. 33A/B Testing for LLMsTake one docs-assistant prompt duel from a golden-set rubric to a live resolution-rate test with sticky routing and registered guardrails.Inference & Production ScaleHard76 min