LeetLLM
Tracks
Intermediate25 lessons

Production ML Systems

Build predictive ML systems from validated data and baselines through feature pipelines, deployment, monitoring, and rollback.

ML engineers, data scientists, and backend engineers shipping predictive systems beyond notebooks.

You can build a reproducible training pipeline, preserve batch and online feature parity, and operate a monitored model release.

  1. 1Python for AI EngineeringLearn Python as the first AI engineering loop: read JSONL rows, validate fields, compute exact-match accuracy, and harden the scorer with pytest, prompt snapshots, leakage checks, seeded runs, and a CI gate.Computing FoundationsEasy14 min
  2. 2NumPy and Tensor ShapesLearn NumPy shape reasoning from first principles: name axes, predict indexing and broadcasting, reduce safely, distinguish reshape from transpose, and add shape guards.Computing FoundationsEasy15 min
  3. 3SQL and Data ModelingTurn an in-memory support retriever into durable SQL tables. Create rows and keys, query with parameters and joins, enforce permissions, roll back failed work, deduplicate retries, inspect indexes, and see where pgvector fits.Computing FoundationsEasy13 min
  4. 4Probability for Machine LearningUse one API abuse-risk detector to learn events, priors, conditional probability, independence, Bayes rule, and base-rate mistakes.Math & StatisticsEasy15 min
  5. 5Statistics and UncertaintyEstimate abuse risk in a flagged review queue from finite labels, using bootstrap intuition, score intervals, sampling bias checks, and calibrated reporting.Math & StatisticsEasy11 min
  6. 6Linear Regression from ScratchFit key-rotation assistant latency by hand, implement least squares and gradient descent in NumPy, then test failure cases and held-out behavior.ML Algorithms & EvaluationMedium19 min
  7. 7Logistic Regression and MetricsRoute access-change requests with logistic regression from scratch: derive sigmoid and log loss, fit NumPy weights, select a cost-aware threshold on validation data, audit ranking and calibration, then compare with scikit-learn.ML Algorithms & EvaluationMedium22 min
  8. 8Decision Trees, Forests, and BoostingModel access-change review with decision trees from scratch: compute impurity, test a non-perfect stump on held-out cases, compare forests and boosting, and audit feature explanations.ML Algorithms & EvaluationMedium21 min
  9. 9Validation and LeakageMake model and policy claims honestly: define the decision moment, split access-review episodes by time and user, expose feature and preprocessing leakage, and audit LLM evaluation contamination.ML Algorithms & EvaluationMedium15 min
  10. 10Dataset Pipelines and Data QualityBuild versioned AI datasets with schema gates, grouped splits, contamination checks, and auditable receipts.ML Algorithms & EvaluationMedium16 min
  11. 11Feature Engineering for Production MLTurn training-job events into stable prediction inputs while preventing leakage and training-serving mismatch.Production ML SystemsMedium9 min
  12. 12Batch and Streaming Feature PipelinesBuild point-in-time training-run features from events and preserve the same meaning in online serving.Production ML SystemsMedium12 min
  13. 13Gradient Boosted Trees in ProductionTrain a boosted SLA-risk baseline from tabular features, evaluate slices, and package deployment evidence.Production ML SystemsMedium12 min
  14. 14Ranking and Recommendation SystemsRank documents for a developer using candidate retrieval, relevance metrics, and feedback-loop safeguards.Production ML SystemsMedium15 min
  15. 15Forecasting and Anomaly DetectionForecast batch-job demand with time-aware evaluation and turn large forecast errors into reviewable operational alerts.Production ML SystemsMedium14 min
  16. 16CNNs from ScratchTrace a CNN over a cracked equipment-panel photo patch: shared kernels, feature-map shapes, pooling, padding failures, and a NumPy-to-PyTorch forward pass.Preparation & PrerequisitesEasy10 min
  17. 17Monitoring Predictive ModelsMonitor predictive models from feature freshness through delayed labels, then gate retraining, promotion, and rollback.Production ML SystemsMedium16 min
  18. 18Experiment Design and A/B TestingDesign a trustworthy online experiment for an AI support change: randomize customers, measure useful outcomes, quantify uncertainty, and reject false wins.ML Algorithms & EvaluationMedium20 min
  19. 19Experiment Tracking with MLflow and W&BTurn a live LLM regression into a reproducible candidate decision by logging inputs, metrics, artifacts, and promotion evidence.Applied LLM EngineeringMedium17 min
  20. 20Model Versioning & DeploymentTurn an evaluated LLM change into an immutable release bundle, promote it through measured traffic, and roll back without losing lineage.Applied LLM EngineeringMedium19 min
  21. 21Capstone: Delivery ETA PredictionShip a delivery-delay warning service with as-of features, versioned policy gates, baseline evidence, and monitored fallback.Portfolio CapstonesHard13 min
  22. 22Capstone: Product RankingShip a marketplace ranking candidate with eligible retrieval, separate recall and NDCG gates, replayable exposure rows, and an A/B-ready rollback receipt.Portfolio CapstonesHard14 min
  23. 23Capstone: Demand ForecastingShip a demand forecast and capacity-alert artifact with rolling backtests, alert review, and retraining policy.Portfolio CapstonesHard14 min
  24. 24Capstone: Image Damage ClassifierShip a damaged-package photo triage service with quality checks, slice evaluation, serving bundles, and review monitoring.Portfolio CapstonesHard16 min
  25. 25Capstone: Production ML PipelineAssemble predictive ML artifacts into validated training, registry promotion, canary monitoring, and rollback.Portfolio CapstonesHard21 min