LeetLLM
Tracks
Advanced21 lessons

Inference and Serving

Understand serving bottlenecks from TTFT and KV cache through batching, quantization, model parallelism, and autoscaling.

Engineers responsible for latency, cost, local deployment, model gateways, and GPU serving reliability.

You can reason about model fit, slow responses, and which serving technique fixes each bottleneck.

  1. 1Inference: TTFT, TPS & KV CacheUnderstand the two-phase inference process (prefill vs decode), derive the KV cache memory formula, and learn production optimizations like chunked prefill and prefill/decode disaggregation.Inference & Production ScaleHard29 min
  2. 2Multi-Query & Grouped-Query AttentionCompare MHA, MQA, and GQA architectures, calculate their KV cache footprint, and reason about memory-limited serving tradeoffs.Inference & Production ScaleHard35 min
  3. 3KV Cache & PagedAttentionCalculate KV cache capacity, trace paged block allocation, and separate memory packing from prefix reuse and scheduling tradeoffs.Inference & Production ScaleHard34 min
  4. 4Prefix Caching and Prompt CachingStructure exact reusable prefixes, validate cache hits from usage fields, and enforce invalidation and tenant-isolation boundaries.Inference & Production ScaleHard19 min
  5. 5FlashAttention & Memory EfficiencyUnderstand how FlashAttention cuts auxiliary attention memory from O(n²) to O(n) with tiling and online softmax, and analyze its IO complexity.Inference & Production ScaleHard32 min
  6. 6Continuous Batching & SchedulingUnderstand how LLM schedulers use continuous batching, chunked prefill, and prefill-decode disaggregation to improve throughput without violating TTFT, TPOT, or inter-token latency targets.Inference & Production ScaleHard32 min
  7. 7Scaling LLM InferenceExplains why decode-heavy LLM serving is often memory-bound and how KV-cache design, batching, PagedAttention, and speculative decoding improve scale.Inference & Production ScaleHard40 min
  8. 8Model Parallelism for LLM InferenceLearn tensor parallelism, pipeline parallelism, context parallelism, and how multi-GPU serving trades memory capacity for communication overhead.Inference & Production ScaleHard22 min
  9. 9Model Quantization: GPTQ, AWQ & GGUFUnderstand how GPTQ, AWQ, and GGUF trade off accuracy, memory footprint, and portability when serving LLMs on GPUs or local hardware.Inference & Production ScaleHard33 min
  10. 10Local LLM DeploymentPlan local LLM deployment with model size, quantization, pruning and sparsity trade-offs, Docker packaging, runtime choice, and hardware budgets.Inference & Production ScaleHard22 min
  11. 11SLM Specialization & Edge DeploymentDistill large teachers into compact SLMs using MobileLLM architectures and Phi-style data recipes. Compile and run them on-device with MLC LLM, ONNX Runtime, Core ML, and ExecuTorch while respecting power, thermal, and strict privacy constraints.Inference & Production ScaleHard26 min
  12. 12Speculative DecodingReduce LLM inter-token latency by pairing cheap drafting with target-model verification. Learn the rejection-sampling proof, speedup model, method choices, and production rollout gates.Inference & Production ScaleHard32 min
  13. 13Long Context Window ManagementMaster long-context LLM engineering: KV-cache math, prefill-vs-decode bottlenecks, RoPE scaling, lost-in-the-middle behavior, and long-context vs. RAG trade-offs.Inference & Production ScaleHard34 min
  14. 14Mixture of Experts ArchitectureMaster MoE routing, load balancing, and dense-vs-sparse serving tradeoffs from Mixtral through GLM-5.2 and DeepSeek V4 Flash 0731.Inference & Production ScaleHard35 min
  15. 15Reasoning & Test-Time ComputeUnderstand how reasoning models trade extra inference compute for better answers, and what that means for search, verifiers, KV cache pressure, and routing.Inference & Production ScaleHard38 min
  16. 16LLM Cost Engineering & Token EconomicsBuild an auditable LLM cost ledger from usage traces, cache decisions, output contracts, offline batch work, and release budget gates.Applied LLM EngineeringMedium16 min
  17. 17Model Gateways, Routing, and FallbacksTurn an audited cost contract into a model gateway that preserves data, schema, review, and budget requirements across routing and fallback.Applied LLM EngineeringMedium16 min
  18. 18LLM Observability & MonitoringTurn claim-level answer traces into production metrics, actionable alerts, privacy-safe debugging records, and reproducible incident evidence.Applied LLM EngineeringMedium18 min
  19. 19Model Versioning & DeploymentTurn an evaluated LLM change into an immutable release bundle, promote it through measured traffic, and roll back without losing lineage.Applied LLM EngineeringMedium19 min
  20. 20GPU Serving & AutoscalingMaster the design of GPU serving infrastructure for LLMs with autoscaling, continuous batching, and cost optimization.Inference & Production ScaleHard50 min
  21. 21A/B Testing for LLMsMaster the design of an A/B testing framework for LLM-powered features, including traffic routing, metric selection, sample sizing, and automated guardrails.Inference & Production ScaleHard43 min