LeetLLM
Tracks
Advanced44 lessons

AI Research Scientist

Learn to turn open questions into reproducible studies across model training, reinforcement learning, agents, and evaluation.

Aspiring research scientists and research engineers who need rigorous experimental judgment, implementation depth, and public evidence.

You can frame a falsifiable question, reproduce a baseline, run controlled ablations, evaluate agent and model behavior, and publish a defensible research artifact.

  1. 1Linear Algebra for MLFind hidden directions in a support-incident matrix with SVD, then use rank, PCA, truncation, and condition numbers without losing sight of what the numbers mean.Math & StatisticsEasy13 min
  2. 2Adam, Momentum, SchedulersTrace SGD, momentum, Adam, AdamW, schedules, and gradient clipping on one uneven loss surface. Learn what each optimizer buffer measures and how to validate a training choice.Math & StatisticsEasy14 min
  3. 3Probability for Machine LearningUse one API abuse-risk detector to learn events, priors, conditional probability, independence, Bayes rule, and base-rate mistakes.Math & StatisticsEasy15 min
  4. 4Statistics and UncertaintyEstimate abuse risk in a flagged review queue from finite labels, using bootstrap intuition, score intervals, sampling bias checks, and calibrated reporting.Math & StatisticsEasy11 min
  5. 5Distributions and SamplingModel an incident assistant with binary outcomes, request routes, tool-call counts, and tail latency, then challenge each simulation before trusting it.Math & StatisticsEasy12 min
  6. 6Hypothesis Tests, Intervals, and pass@kCompare a code-generation model with paired evidence, uncertainty for lift, and pass@k under a fixed sampling budget.Math & StatisticsEasy13 min
  7. 7Validation and LeakageMake model and policy claims honestly: define the decision moment, split access-review episodes by time and user, expose feature and preprocessing leakage, and audit LLM evaluation contamination.ML Algorithms & EvaluationMedium15 min
  8. 8Experiment Design and A/B TestingDesign a trustworthy online experiment for an AI support change: randomize customers, measure useful outcomes, quantify uncertainty, and reject false wins.ML Algorithms & EvaluationMedium20 min
  9. 9Reinforcement Learning BasicsLearn reinforcement learning through the access-review workflow from earlier lessons. Define an MDP, compute discounted returns and Bellman backups, implement value iteration and Q-learning, model abandonment risk, and connect policy gradients to LLM post-training.ML Algorithms & EvaluationMedium14 min
  10. 10Dataset Pipelines and Data QualityBuild versioned AI datasets with schema gates, grouped splits, contamination checks, and auditable receipts.ML Algorithms & EvaluationMedium16 min
  11. 11Experiment Tracking with MLflow and W&BTurn a live LLM regression into a reproducible candidate decision by logging inputs, metrics, artifacts, and promotion evidence.Applied LLM EngineeringMedium17 min
  12. 12PyTorch Training LoopsBuild a PyTorch classifier from raw logits through autograd, validation, and reloadable checkpoints.ML Algorithms & EvaluationMedium17 min
  13. 13CUDA for ML TrainingBuild beginner-first CUDA intuition for model training: CPU vs GPU roles, host-device copies, asynchronous execution, PyTorch device placement, and first-line debugging of OOM and performance issues.Computing FoundationsEasy14 min
  14. 14The Transformer Architecture End-to-EndTrace an incident note through masked attention, a decoder block, and next-token logits with readable NumPy and PyTorch code.Preparation & PrerequisitesEasy13 min
  15. 15Language Modeling & Next TokensLearn how next-token prediction becomes a trainable language model, from bigram counts and neural n-grams to causal Transformer generation and KV-cache serving.Preparation & PrerequisitesEasy21 min
  16. 16Scaled Dot-Product AttentionLearn scaled dot-product attention from first principles, including Q/K/V routing, variance scaling, masks, multi-head shapes, KV-cache costs, and FlashAttention.Transformer Deep DivesHard41 min
  17. 17Layer Normalization: Pre-LN vs Post-LNUnderstand LayerNorm mechanics, Pre-LN versus Post-LN placement, RMSNorm simplification, gradient stability, and hybrid normalization layouts for deep transformers.Transformer Deep DivesHard29 min
  18. 18Inference: TTFT, TPS & KV CacheUnderstand the two-phase inference process (prefill vs decode), derive the KV cache memory formula, and learn production optimizations like chunked prefill and prefill/decode disaggregation.Inference & Production ScaleHard29 min
  19. 19FlashAttention & Memory EfficiencyUnderstand how FlashAttention cuts auxiliary attention memory from O(n²) to O(n) with tiling and online softmax, and analyze its IO complexity.Inference & Production ScaleHard32 min
  20. 20Scaling Laws & Compute-Optimal TrainingLearn the empirical power laws governing LLM performance, from Kaplan's parameter-heavy frontier through Chinchilla-optimal ratios to modern inference-aware training strategies.Advanced Training & AdaptationHard36 min
  21. 21Pre-training Data at ScaleUnderstand how web-scale pre-training data is extracted, filtered, deduplicated, mixed, tokenized, and packed into training-ready shards, including decontamination, late-stage annealing, and synthetic-data tradeoffs.Advanced Training & AdaptationHard35 min
  22. 22Build GPT from Scratch LabBuild and train a tiny GPT end to end on Shakespeare: tokenize with GPT-style subwords, remap active token IDs, run causal self-attention, track validation loss, save a checkpoint, and sample text.Advanced Training & AdaptationHard22 min
  23. 23JAX for PyTorch ResearchersRead and modify JAX research code from a PyTorch foundation by making state, randomness, transformations, compilation, and timing explicit.Advanced Training & AdaptationHard23 min
  24. 24Instruction Tuning & Chat TemplatesTeach a base language model to answer as an assistant: curate grounded SFT rows, serialize chat turns exactly, choose loss targets, pack safely, and detect serving-time template drift.Core LLM FoundationsMedium18 min
  25. 25Continued Pretraining for Domain ShiftLearn when to keep the causal language-modeling objective and continue pretraining on domain text instead of jumping straight to SFT, and how to evaluate the trade-off against forgetting, cost, and downstream gain.Advanced Training & AdaptationHard19 min
  26. 26LLM-as-a-Judge EvaluationAdd calibrated soft judgments to a RAG evaluation trace without letting an LLM override deterministic evidence gates.Applied LLM EngineeringMedium18 min
  27. 27Synthetic Data PipelinesBuild synthetic post-training data pipelines with Self-Instruct, Evol-Instruct, calibrated judge signals, verifiers, preference pairs, diversity checks, and decontamination.Advanced Training & AdaptationHard25 min
  28. 28Supervised Fine-Tuning PipelineRun supervised fine-tuning as a real training system: choose the learning objective before the update surface, verify response-token loss and packing, track the real batch budget, save resumable checkpoints, and export on held-out behavior.Advanced Training & AdaptationHard23 min
  29. 29Mixed Precision TrainingMeasure how FP16 and BF16 affect training range, update precision, memory, and release evidence before enabling faster low-precision compute.Advanced Training & AdaptationHard22 min
  30. 30Distributed Training: FSDP & ZeROUnderstand ZeRO stages, current FSDP1 vs FSDP2 guidance, and when native PyTorch or DeepSpeed is the right choice for large-model training.Advanced Training & AdaptationHard40 min
  31. 31Data Labeling and Human FeedbackBuild a trustworthy human-feedback data flywheel: redact traces, write rubrics, measure agreement, select useful examples, prevent leakage, and promote versioned datasets.Applied LLM EngineeringMedium16 min
  32. 32Reward Modeling from Preference DataTrain reward models as a first-class post-training stage: validate chosen/rejected pairs and splits, fit a scalar reward head with Bradley-Terry loss, audit generalization, and decide when explicit rewards are worth the extra complexity.Advanced Training & AdaptationHard18 min
  33. 33RLHF & DPO AlignmentUnderstand the RLHF pipeline and DPO, including reward modeling, PPO mechanics, and the trade-offs between iterative reinforcement learning and direct preference optimization.Advanced Training & AdaptationHard33 min
  34. 34Constitutional AI & Red TeamingUnderstand how Constitutional AI reduces reliance on repeated human preference labeling through AI critique and ranking, and how automated red teaming stress-tests those safeguards.Advanced Training & AdaptationHard30 min
  35. 35RLVR & Verifiable RewardsUnderstand RLVR, a post-training approach that uses programmatic verification instead of learned human-preference rewards to improve checked outcomes in math, code, and other contract-driven tasks.Advanced Training & AdaptationHard37 min
  36. 36Function Calling & Tool UseBuild a safe tool-calling runtime that validates model requests, executes controlled actions, feeds observations back, and evaluates complete workflows.Applied LLM EngineeringMedium16 min
  37. 37ReAct & Plan-and-ExecuteCompare ReAct for tightly coupled tool use with Plan-and-Execute for longer workflows with explicit planning and replanning.Advanced Agents & RetrievalHard34 min
  38. 38Evaluating AI AgentsEvaluate model-promotion agent runs by final state, observable trace, safety gates, cost, and repeatability, then map private tests to public benchmarks.Applied LLM EngineeringMedium18 min
  39. 39LLM Benchmarks & LimitationsBuild an evaluation suite for a policy-answering LLM: score evidence use, understand public benchmark contracts, control judge bias, and make release decisions from private tests.Core LLM FoundationsMedium21 min
  40. 40KV Cache & PagedAttentionCalculate KV cache capacity, trace paged block allocation, and separate memory packing from prefix reuse and scheduling tradeoffs.Inference & Production ScaleHard34 min
  41. 41Continuous Batching & SchedulingUnderstand how LLM schedulers use continuous batching, chunked prefill, and prefill-decode disaggregation to improve throughput without violating TTFT, TPOT, or inter-token latency targets.Inference & Production ScaleHard32 min
  42. 42A/B Testing for LLMsMaster the design of an A/B testing framework for LLM-powered features, including traffic routing, metric selection, sample sizing, and automated guardrails.Inference & Production ScaleHard43 min
  43. 43Mechanistic InterpretabilityLearn how sparse autoencoders decompose transformer activations into candidate interpretable features, support circuit tracing, and enable controlled activation-steering experiments.Transformer Deep DivesHard31 min
  44. 44Capstone: Reproducible ML StudyTurn one research paper into a falsifiable, public-safe ML study with paired experiments, uncertainty, reproducible artifacts, and a defensible report.Portfolio CapstonesHard31 min