LeetLLM
Tracks
Advanced45 lessons

AI Research Scientist

Learn to turn open questions into reproducible studies across model training, reinforcement learning, agents, and evaluation.

Aspiring research scientists and research engineers who need rigorous experimental judgment, implementation depth, and public evidence.

You can frame a falsifiable question, reproduce a baseline, run controlled ablations, evaluate agent and model behavior, and publish a defensible research artifact.

  1. 1Linear Algebra for MLFind hidden directions in a support-incident matrix with SVD, then use rank, PCA, truncation, and condition numbers without losing sight of what the numbers mean.Math & StatisticsEasy15 min
  2. 2Adam, Momentum, SchedulersTrace SGD, momentum, RMSProp, Adam, AdamW, schedules, and gradient clipping on a 100-to-1 loss valley. Learn what each optimizer buffer measures and how to validate a training choice.Math & StatisticsEasy17 min
  3. 3Probability for Machine LearningUse one API abuse-risk detector to learn events, random variables, distributions, conditional probability, independence, Bayes rule, and base-rate mistakes.Math & StatisticsEasy18 min
  4. 4Statistics and UncertaintyTurn 16 abusive flags out of 100 reviews into a rate, a Wilson interval, and a sampling-bias check, then refuse to treat a four-point calibration gap as proof.Math & StatisticsEasy14 min
  5. 5Distributions and SamplingMatch binary outcomes, routes, tool-call counts, and latency to first distributions, then reject a simulation that doesn't fit the traces.Math & StatisticsEasy19 min
  6. 6Hypothesis Tests, Intervals, and pass@kCompare a code-generation model with paired evidence, uncertainty for lift, and pass@k under a fixed sampling budget.Math & StatisticsEasy15 min
  7. 7Validation and LeakageSplit access-review requests by time and user, block post-decision fields, fit preprocessing on training rows only, and treat public LLM benchmarks as contamination-prone.ML Algorithms & EvaluationMedium19 min
  8. 8Experiment Design and A/B TestingDesign a trustworthy online experiment for an incident-assistant change: randomize incidents, measure useful outcomes, quantify uncertainty, and reject false wins.ML Algorithms & EvaluationMedium22 min
  9. 9Reinforcement Learning BasicsTurn the earlier one-shot access-review label into an MDP. Compute discounted returns and Bellman backups, run value iteration and Q-learning, watch abandonment reverse a policy, and connect REINFORCE to LLM post-training.ML Algorithms & EvaluationMedium16 min
  10. 10Dataset Pipelines and Data QualityBuild versioned AI datasets with schema gates, grouped splits, contamination checks, and auditable receipts.ML Algorithms & EvaluationMedium18 min
  11. 11Experiment Tracking with MLflow and W&BTurn a live LLM regression into a reproducible candidate decision by logging inputs, metrics, artifacts, and promotion evidence.Applied LLM EngineeringMedium19 min
  12. 12PyTorch Training LoopsBuild a PyTorch classifier from raw logits through autograd, validation, and reloadable checkpoints.ML Algorithms & EvaluationMedium19 min
  13. 13CUDA for ML TrainingFollow one access-ticket batch from CPU memory into CUDA kernels. Learn thread and memory hierarchy, safe device placement, honest timing, and first-line diagnosis for setup, OOM, and throughput failures.Computing FoundationsEasy15 min
  14. 14The Transformer Architecture End-to-EndTrace function returns the through masked attention, a decoder block, and next-token logits with readable NumPy and PyTorch code.Preparation & PrerequisitesEasy15 min
  15. 15Language Modeling & Next TokensLearn how next-token prediction becomes a trainable language model, from bigram counts and neural n-grams to causal Transformer generation and KV-cache serving.Preparation & PrerequisitesEasy23 min
  16. 16Scaled Dot-Product AttentionBuild scaled dot-product attention from a token sequence: Q/K/V routing, variance scaling, masks, multi-head shapes, KV-cache cost, and FlashAttention.Transformer Deep DivesHard39 min
  17. 17Layer Normalization: Pre-LN vs Post-LNUnderstand LayerNorm mechanics, Pre-LN versus Post-LN placement, RMSNorm simplification, gradient stability, and hybrid normalization layouts for deep transformers.Transformer Deep DivesHard30 min
  18. 18Inference: TTFT, TPS & KV CacheMap prefill vs decode bottlenecks, measure TTFT and decode cadence, and size KV cache so concurrent sequences fit on one GPU.Inference & Production ScaleHard29 min
  19. 19FlashAttention & Memory EfficiencyUnderstand how FlashAttention cuts auxiliary attention memory from O(n²) to O(n) with tiling and online softmax, and analyze its IO complexity.Inference & Production ScaleHard34 min
  20. 20Scaling Laws & Compute-Optimal TrainingLearn how Kaplan, Chinchilla, and inference-aware fits split a training budget across parameters and tokens, and when a smaller over-trained model wins on lifetime cost.Advanced Training & AdaptationHard35 min
  21. 21Pre-training Data at ScaleUnderstand how web-scale pre-training data is extracted, filtered, deduplicated, mixed, tokenized, and packed into training-ready shards, including decontamination, late-stage annealing, and synthetic-data tradeoffs.Advanced Training & AdaptationHard35 min
  22. 22Build GPT from Scratch LabBuild and train a tiny GPT end to end on Shakespeare: tokenize with GPT-style subwords, remap active token IDs, run causal self-attention, track validation loss, save a checkpoint, and sample text.Advanced Training & AdaptationHard23 min
  23. 23JAX for PyTorch ResearchersRead and modify JAX research code after the PyTorch GPT lab by making state, randomness, transformations, compilation, and timing explicit.Advanced Training & AdaptationHard26 min
  24. 24Instruction Tuning & Chat TemplatesTeach a base language model to answer as an assistant: curate grounded SFT rows, serialize chat turns exactly, choose loss targets, pack safely, and detect serving-time template drift.Core LLM FoundationsMedium20 min
  25. 25Continued Pretraining for Domain ShiftLearn when to keep the causal language-modeling objective and continue pretraining on domain text instead of jumping straight to SFT, and how to evaluate the trade-off against forgetting, cost, and downstream gain.Advanced Training & AdaptationHard24 min
  26. 26LLM-as-a-Judge EvaluationAdd calibrated soft judgments to a RAG evaluation trace without letting an LLM override deterministic evidence gates.Applied LLM EngineeringMedium19 min
  27. 27Synthetic Data PipelinesBuild post-training synthetic data as a gated pipeline: Self-Instruct, Evol-Instruct, grounded execution, calibrated judges, preference pairs, diversity, decontamination, and versioned shards.Advanced Training & AdaptationHard29 min
  28. 28Supervised Fine-Tuning PipelineRun supervised fine-tuning as a real training system: choose the learning objective before the update surface, verify response-token loss and packing, track the real batch budget, save resumable checkpoints, and export on held-out behavior.Advanced Training & AdaptationHard26 min
  29. 29Mixed Precision TrainingChoose FP16 or BF16 for an SFT run by measuring range, update resolution, memory, and held-out policy quality instead of assuming faster math is an upgrade.Advanced Training & AdaptationHard23 min
  30. 30Distributed Training: FSDP & ZeROUnderstand ZeRO stages, current FSDP2 fully_shard guidance, mixed-precision recipes that don't share one byte count, and when native PyTorch or DeepSpeed is the right choice.Advanced Training & AdaptationHard42 min
  31. 31Training Run OperationsTreat a training job as a resumable system: distinguish continue vs initialize vs export, save sharded state that can survive preemption, keep global batch and learning-rate scaling honest, and choose full SFT, LoRA, QLoRA, continued pretraining, or distillation from data, domain shift, and GPU budget.Advanced Training & AdaptationHard24 min
  32. 32Data Labeling and Human FeedbackBuild a trustworthy human-feedback data flywheel: redact traces, write rubrics, measure agreement, select useful examples, prevent leakage, and promote versioned datasets.Applied LLM EngineeringMedium18 min
  33. 33Reward Modeling from Preference DataTrain reward models as a first-class post-training stage: validate chosen/rejected pairs and splits, fit a scalar reward head with Bradley-Terry loss, audit generalization, and decide when explicit rewards are worth the extra complexity.Advanced Training & AdaptationHard22 min
  34. 34RLHF & DPO AlignmentTurn a reward model into an aligned policy: run PPO-style RLHF with a KL budget, or skip the extra judge and train DPO on the same preference pairs, then catch reward hacking and likelihood displacement.Advanced Training & AdaptationHard26 min
  35. 35Constitutional AI & Red TeamingUnderstand how Constitutional AI reduces reliance on repeated human preference labeling through AI critique and ranking, and how automated red teaming stress-tests those safeguards.Advanced Training & AdaptationHard30 min
  36. 36RLVR & Verifiable RewardsUnderstand RLVR, a post-training approach that uses programmatic verification instead of learned human-preference rewards to improve checked outcomes in math, code, and other contract-driven tasks.Advanced Training & AdaptationHard39 min
  37. 37Function Calling & Tool UseBuild a safe tool-calling runtime that validates model requests, executes controlled actions, feeds observations back, and evaluates complete workflows.Applied LLM EngineeringMedium19 min
  38. 38ReAct & Plan-and-ExecuteCompare ReAct for tightly coupled tool use with Plan-and-Execute for longer workflows with explicit planning and replanning.Advanced Agents & RetrievalHard41 min
  39. 39Evaluating AI AgentsEvaluate model-promotion agent runs by final state, observable trace, safety gates, cost, and repeatability, then map private tests to public benchmarks.Applied LLM EngineeringMedium21 min
  40. 40LLM Benchmarks & LimitationsBuild an evaluation suite for a policy-answering LLM: score evidence use, understand public benchmark contracts, control judge bias, and make release decisions from private tests.Core LLM FoundationsMedium25 min
  41. 41KV Cache & PagedAttentionCalculate KV cache capacity, trace paged block allocation, and separate memory packing from prefix reuse and scheduling tradeoffs.Inference & Production ScaleHard34 min
  42. 42Continuous Batching & SchedulingUnderstand how LLM schedulers use continuous batching, chunked prefill, and prefill-decode disaggregation to improve throughput without violating TTFT, TPOT, or inter-token latency targets.Inference & Production ScaleHard28 min
  43. 43A/B Testing for LLMsTake one docs-assistant prompt duel from a golden-set rubric to a live resolution-rate test with sticky routing and registered guardrails.Inference & Production ScaleHard49 min
  44. 44Mechanistic InterpretabilityLearn how sparse autoencoders decompose transformer activations into candidate interpretable features, support circuit tracing, and enable controlled activation-steering experiments.Transformer Deep DivesHard33 min
  45. 45Capstone: Reproducible ML StudyTurn one research paper into a falsifiable, public-safe ML study with paired experiments, uncertainty, reproducible artifacts, and a defensible report.Portfolio CapstonesHard35 min