Skip to content
LeetLLM
Tracks
Advanced33 lessons

Inference and Serving

Understand accelerator and kernel behavior through serving bottlenecks, distributed data planes, benchmarking, and autoscaling.

Engineers responsible for GPU performance, latency, cost, local deployment, model gateways, and serving reliability.

You can measure kernels and serving systems, diagnose bottlenecks, and choose the optimization that fits each workload.

  1. 1Inference: TTFT, TPS & KV CacheMap prefill vs decode bottlenecks, measure TTFT and decode cadence, and size KV cache so concurrent sequences fit on one GPU.Inference & Production ScaleHard42 min
  2. 2Multi-Query & Grouped-Query AttentionCompare MHA, MQA, and GQA architectures, calculate their KV cache footprint, and reason about memory-limited serving tradeoffs.Inference & Production ScaleHard49 min
  3. 3KV Cache & PagedAttentionCalculate KV cache capacity, trace paged block allocation, and separate memory packing from prefix reuse and scheduling tradeoffs.Inference & Production ScaleHard46 min
  4. 4Prefix Caching and Prompt CachingStructure exact reusable prefixes, validate cache hits from usage fields, and enforce invalidation and tenant-isolation boundaries.Inference & Production ScaleHard29 min
  5. 5FlashAttention & Memory EfficiencyUnderstand how FlashAttention cuts auxiliary attention memory from O(n²) to O(n) with tiling and online softmax, and analyze its IO complexity.Inference & Production ScaleHard49 min
  6. 6Continuous Batching & SchedulingTrace iteration-level slot reuse, budget prefill and decode work, and evaluate chunking or phase separation against TTFT, TPOT, and inter-token latency targets.Inference & Production ScaleHard43 min
  7. 7Scaling LLM InferenceSize LLM serving from HBM bandwidth and KV residency: pick a throughput-latency-cost operating point, then compose batching, paging, speculation, and precision as capacity levers.Inference & Production ScaleHard42 min
  8. 8Model Parallelism for LLM InferenceLearn tensor parallelism, pipeline parallelism, context parallelism, and how multi-GPU serving trades memory capacity for communication overhead.Inference & Production ScaleHard34 min
  9. 9Accelerator Architecture Field GuideCompare accelerator execution, memory, interconnect, and programming models through one LLM workload, then choose what must be remeasured when moving between NVIDIA, AMD, TPU, Trainium, and Apple silicon.Inference & Production ScaleHard37 min
  10. 10Correct GPU Profiling and BenchmarkingBuild trustworthy GPU performance evidence: check kernel correctness, choose system or kernel profiling scope, control benchmark state, and preserve a reproducible receipt before claiming a speedup.Inference & Production ScaleHard40 min
  11. 11GPU Kernel Performance Engineering LabOptimize one CUDA matrix-transpose kernel through a repeatable evidence loop. Diagnose coalescing, shared-memory bank conflicts, occupancy limits, correctness failures, and the point where a reduction or scan is the right primitive.Inference & Production ScaleHard29 min
  12. 12GPU Parallel Primitives LabTrace reduction, prefix scan and online softmax through concrete values, then check CUDA indexing and compare custom collectives with CUB.Inference & Production ScaleHard30 min
  13. 13GPU Compilers and Kernel LanguagesTrace a GPU kernel from CUDA C++ or a tile language through compiler IR, PTX, cubin, and native instructions, then choose an authoring model by control, portability, and inspectable evidence.Inference & Production ScaleHard37 min
  14. 14GEMM From Naive CUDA to Tensor CoresBuild one matrix multiplication through coalesced scalar CUDA, shared-memory tiling, register blocking, Tensor Core MMA, and the boundary where async copies and CUTLASS take over.Inference & Production ScaleHard35 min
  15. 15Low-Precision GPU ProgrammingBuild numerically sound FP16, BF16, TF32, FP8, and MXFP8 GPU paths by making scale, accumulation, layout, and kernel contracts explicit.Inference & Production ScaleHard28 min
  16. 16GPU Runtime Execution LabMaster asynchronous CUDA submission with streams, events, graph capture, replay, updates, runtime dispatch, and correctness-first benchmark receipts.Inference & Production ScaleHard25 min
  17. 17AI-Generated Kernel EngineeringTurn generated GPU code into a promotable library kernel through explicit operator contracts, hidden correctness tests, sanitizer gates, fair benchmarks, reproducible receipts, and controlled rollout.Inference & Production ScaleHard30 min
  18. 18Model Quantization: GPTQ, AWQ & GGUFPack INT4 codes, follow GPTQ error compensation, and search AWQ channel scales. Budget GGUF tensors and test quality separately from storage and speed.Inference & Production ScaleHard32 min
  19. 19Local LLM DeploymentBuild local model memory budgets, check GPU placement, distinguish offload from paging, choose a runtime, and verify latency and private-data boundaries.Inference & Production ScaleHard27 min
  20. 20SLM Specialization & Edge DeploymentChoose and adapt a small model for a device job: compare training signals and compact architectures, inspect runtime placement, and evaluate quality, memory, sustained speed, energy, privacy, and update recovery.Inference & Production ScaleHard41 min
  21. 21Speculative DecodingPropose tokens cheaply, verify with the target, and correct rejections without changing its distribution in exact arithmetic. Practice cache dependencies, tree masks, and performance decisions under load.Inference & Production ScaleHard34 min
  22. 22Long Context Window ManagementMaster long-context LLM engineering: KV-cache math, prefill-vs-decode bottlenecks, RoPE scaling, lost-in-the-middle behavior, and long-context vs. RAG trade-offs.Inference & Production ScaleHard45 min
  23. 23Mixture of Experts ArchitectureTrace selected-expert computation, test routing and balancing assumptions, and account for stored weights, cache state, and dispatch from Mixtral to current hybrid MoE models.Inference & Production ScaleHard44 min
  24. 24Reasoning & Test-Time ComputeChoose between more samples, revision, and guided search. Test verifier failures, distinguish effort from hard limits, and account for reasoning tokens and shared cache state.Inference & Production ScaleHard45 min
  25. 25LLM Cost Engineering & Token EconomicsBuild an auditable LLM cost ledger from usage traces, cache decisions, output contracts, offline batch work, and release budget gates.Applied LLM EngineeringMedium33 min
  26. 26Model Gateways, Routing, and FallbacksCompile route requirements, reject incompatible fallbacks, and test shared deadlines and spending reservations in a local gateway lab.Applied LLM EngineeringMedium30 min
  27. 27LLM Observability & MonitoringConnect answer-quality checks to latency, cost, traces, and actionable alerts without confusing missing evidence with a healthy service.Applied LLM EngineeringMedium31 min
  28. 28Model Versioning & DeploymentTurn an evaluated LLM change into an immutable release bundle, promote it through measured traffic, and roll back without losing lineage.Applied LLM EngineeringMedium31 min
  29. 29GPU Serving & AutoscalingServe one incident-summary request through continuous batching, paged KV, and an autoscaler driven by queue, cache, and TTFT pressure.Inference & Production ScaleHard61 min
  30. 30GPU Failure OperationsDiagnose one failing GPU replica, contain the smallest safe scope, preserve evidence, and re-admit hardware through explicit gates.Inference & Production ScaleHard26 min
  31. 31Distributed Inference Data PlaneFollow an 8K prompt through rank collectives and KV handoff, calculate transfer and memory budgets, and test why expired buffers aren't immediately reusable.Inference & Production ScaleHard22 min
  32. 32Serving Benchmark EngineeringBuild a reproducible serving benchmark with a frozen arrival trace, explicit streaming timing boundaries, complete failure accounting, and SLO-qualified capacity estimates.Inference & Production ScaleHard22 min
  33. 33A/B Testing for LLMsTake one docs-assistant prompt duel from a golden-set rubric to a live resolution-rate test with sticky routing and registered guardrails.Inference & Production ScaleHard59 min