LeetLLM
Tracks
Intermediate25 lessons

Production ML Systems

Build predictive ML systems from validated data and baselines through feature pipelines, deployment, monitoring, and rollback.

ML engineers, data scientists, and backend engineers shipping predictive systems beyond notebooks.

You can build a reproducible training pipeline, preserve batch and online feature parity, and operate a monitored model release.

  1. 1Python for AI EngineeringRebuild the access-request scorer in Python: inspect one JSONL row, validate fields, print the 0.667 receipt, keep a 2/3 count floor, and cover those contracts with pytest.Computing FoundationsEasy14 min
  2. 2NumPy and Tensor ShapesLearn NumPy shape reasoning from first principles: name axes, predict indexing and broadcasting, reduce safely, distinguish reshape from transpose, and add shape guards.Computing FoundationsEasy18 min
  3. 3SQL and Data ModelingTurn an in-memory support retriever into durable SQL tables. Model versioned sources, query safely, preserve permissions and lineage, and inspect the paths that keep retrieval correct as data changes.Computing FoundationsEasy20 min
  4. 4Probability for Machine LearningUse one API abuse-risk detector to learn events, random variables, distributions, conditional probability, independence, Bayes rule, and base-rate mistakes.Math & StatisticsEasy18 min
  5. 5Statistics and UncertaintyTurn 16 abusive flags out of 100 reviews into a rate, a Wilson interval, and a sampling-bias check, then refuse to treat a four-point calibration gap as proof.Math & StatisticsEasy14 min
  6. 6Linear Regression from ScratchFit key-rotation assistant latency by hand, implement least squares and gradient descent in NumPy, then test failure cases and held-out behavior.ML Algorithms & EvaluationMedium20 min
  7. 7Logistic Regression and MetricsRoute access-change requests with logistic regression from scratch: derive sigmoid and log loss, fit NumPy weights, select a cost-aware threshold on validation data, audit ranking and calibration, then compare with scikit-learn.ML Algorithms & EvaluationMedium25 min
  8. 8Decision Trees, Forests, and BoostingRoute the same access-change requests with axis-aligned rules: compute Gini, fit a stump, watch extra depth memorize R3, average bootstrap trees, boost residuals, and audit MDI versus SHAP.ML Algorithms & EvaluationMedium25 min
  9. 9Validation and LeakageSplit access-review requests by time and user, block post-decision fields, fit preprocessing on training rows only, and treat public LLM benchmarks as contamination-prone.ML Algorithms & EvaluationMedium19 min
  10. 10Dataset Pipelines and Data QualityBuild versioned AI datasets with schema gates, grouped splits, contamination checks, and auditable receipts.ML Algorithms & EvaluationMedium18 min
  11. 11Feature Engineering for Production MLTurn training-job events into stable prediction inputs while preventing leakage and training-serving mismatch.Production ML SystemsMedium14 min
  12. 12Batch and Streaming Feature PipelinesReplay job-SLA features at decision time from events, then keep that same meaning in online serving.Production ML SystemsMedium14 min
  13. 13Gradient Boosted Trees in ProductionTrain a boosted SLA-risk baseline from tabular features, evaluate slices, and package deployment evidence.Production ML SystemsMedium14 min
  14. 14Ranking and Recommendation SystemsRank documents for a developer using candidate retrieval, relevance metrics, and feedback-loop safeguards.Production ML SystemsMedium18 min
  15. 15Forecasting and Anomaly DetectionForecast batch-job demand with time-aware evaluation and turn large forecast errors into reviewable operational alerts.Production ML SystemsMedium16 min
  16. 16CNNs from ScratchTrace a CNN over a 4 by 4 screenshot crop: shared kernels, feature-map shapes, pooling winners, padding artifacts, and a matching NumPy-to-PyTorch forward pass.Preparation & PrerequisitesEasy13 min
  17. 17Monitoring Predictive ModelsMonitor predictive models from feature freshness through delayed labels, then gate retraining, promotion, and rollback.Production ML SystemsMedium19 min
  18. 18Experiment Design and A/B TestingDesign a trustworthy online experiment for an incident-assistant change: randomize incidents, measure useful outcomes, quantify uncertainty, and reject false wins.ML Algorithms & EvaluationMedium22 min
  19. 19Experiment Tracking with MLflow and W&BTurn a live LLM regression into a reproducible candidate decision by logging inputs, metrics, artifacts, and promotion evidence.Applied LLM EngineeringMedium19 min
  20. 20Model Versioning & DeploymentTurn an evaluated LLM change into an immutable release bundle, promote it through measured traffic, and roll back without losing lineage.Applied LLM EngineeringMedium19 min
  21. 21Capstone: Delivery ETA PredictionShip a delivery-delay warning service with as-of features, versioned policy gates, baseline evidence, and monitored fallback.Portfolio CapstonesHard17 min
  22. 22Capstone: Product RankingShip a marketplace ranking candidate with eligible retrieval, separate recall and NDCG gates, replayable exposure rows, and an A/B-ready rollback receipt.Portfolio CapstonesHard17 min
  23. 23Capstone: Demand ForecastingShip a demand forecast and capacity-alert artifact with rolling backtests, alert review, and retraining policy.Portfolio CapstonesHard17 min
  24. 24Capstone: Image Damage ClassifierShip a damaged-package photo triage service with quality checks, slice evaluation, serving bundles, and review monitoring.Portfolio CapstonesHard20 min
  25. 25Capstone: Production ML PipelineTurn four shipped models into one receipt-bound promotion path: offline gates, canary windows, alias movement, and rollback.Portfolio CapstonesHard24 min