Skip to content
LeetLLM
Tracks
Intermediate25 lessons

Production ML Systems

Build predictive ML systems from validated data and baselines through feature pipelines, deployment, monitoring, and rollback.

ML engineers, data scientists, and backend engineers shipping predictive systems beyond notebooks.

You can build a reproducible training pipeline, preserve batch and online feature parity, and operate a monitored model release.

  1. 1Python for AI EngineeringLearn Python by building a small evaluator: read JSONL, validate fields, count correct predictions, and test the difference between bad input and a wrong answer.Computing FoundationsEasy20 min
  2. 2NumPy and Tensor ShapesLearn NumPy shape reasoning from first principles: name axes, predict indexing and broadcasting, reduce safely, distinguish reshape from transpose, and add shape guards.Computing FoundationsEasy22 min
  3. 3SQL and Data ModelingTurn an in-memory support retriever into durable SQL tables. Model versioned sources, query safely, preserve permissions and lineage, and inspect the paths that keep retrieval correct as data changes.Computing FoundationsEasy27 min
  4. 4Probability for Machine LearningUse one API abuse-risk detector to learn events, random variables, distributions, conditional probability, independence, Bayes rule, and base-rate mistakes.Math & StatisticsEasy23 min
  5. 5Statistics and UncertaintyEstimate a review queue's abuse rate with bootstrap and Wilson intervals, distinguish sampling variation from bias, and assess calibration without overstating the evidence.Math & StatisticsEasy21 min
  6. 6Linear Regression from ScratchFit assistant latency with a line, derive OLS and gradient descent from first principles, and expose hidden non-linearities through residual diagnostics.ML Algorithms & EvaluationMedium35 min
  7. 7Logistic Regression and MetricsRoute access-change requests with logistic regression from scratch: derive sigmoid and log loss, fit NumPy weights, select a cost-aware threshold on validation data, audit ranking and calibration, then compare with scikit-learn.ML Algorithms & EvaluationMedium48 min
  8. 8Decision Trees, Forests, and BoostingRoute the same access-change requests with axis-aligned rules: compute Gini, fit a stump, watch extra depth memorize R3, average bootstrap trees, boost residuals, and audit MDI versus SHAP.ML Algorithms & EvaluationMedium41 min
  9. 9Validation and LeakageSplit access-review requests by time and user, block post-decision fields, fit preprocessing on training rows only, and treat public LLM benchmarks as contamination-prone.ML Algorithms & EvaluationMedium32 min
  10. 10Dataset Pipelines and Data QualityBuild versioned AI datasets with schema gates, grouped splits, contamination checks, and auditable receipts.ML Algorithms & EvaluationMedium30 min
  11. 11Feature Engineering for Production MLTurn training-job events into stable prediction inputs while preventing leakage and training-serving mismatch.Production ML SystemsMedium26 min
  12. 12Batch and Streaming Feature PipelinesReplay job-SLA features at decision time from events, then keep that same meaning in online serving.Production ML SystemsMedium27 min
  13. 13Gradient Boosted Trees in ProductionTrain an SLA-risk booster, select rounds and thresholds on later jobs, challenge it with cheap baselines, and record what the release evidence actually supports.Production ML SystemsMedium30 min
  14. 14Ranking and Recommendation SystemsRank documents for a developer using candidate retrieval, relevance metrics, and feedback-loop safeguards.Production ML SystemsMedium38 min
  15. 15Forecasting and Anomaly DetectionForecast batch-job demand with time-aware evaluation and turn large forecast errors into reviewable operational alerts.Production ML SystemsMedium36 min
  16. 16CNNs from ScratchTrace a CNN over a 4 by 4 crop: shared kernels, feature-map shapes, pooling winners, padding artifacts, and a matching NumPy-to-PyTorch forward pass.Preparation & PrerequisitesEasy19 min
  17. 17Monitoring Predictive ModelsMonitor predictive models from feature freshness through delayed labels, then gate retraining, promotion, and rollback.Production ML SystemsMedium32 min
  18. 18Experiment Design and A/B TestingDesign a trustworthy online experiment for an incident-assistant change: randomize incidents, measure useful outcomes, quantify uncertainty, and reject false wins.ML Algorithms & EvaluationMedium41 min
  19. 19Experiment Tracking with MLflow and W&BRecord the inputs, results, artifacts, and limits behind a candidate decision, then verify them with local MLflow and W&B examples.Applied LLM EngineeringMedium31 min
  20. 20Model Versioning & DeploymentTurn an evaluated LLM change into an immutable release bundle, promote it through measured traffic, and roll back without losing lineage.Applied LLM EngineeringMedium32 min
  21. 21Capstone: Delivery ETA PredictionCompare as-of arrival estimates and warning policies using delayed labels, meaningful baselines, and honest uncertainty.Portfolio CapstonesHard35 min
  22. 22Capstone: Product RankingFit a small ranking model, expose misleading metrics, and audit eligibility, delayed labels, and click bias before planning an experiment.Portfolio CapstonesHard41 min
  23. 23Capstone: Demand ForecastingFit rolling demand forecasts, compare capacity signals with later errors, and catch misleading backtest evidence.Portfolio CapstonesHard24 min
  24. 24Capstone: Image Damage ClassifierTest photo review routes, expose misleading quality scores, and plan a real-image classifier with case-level evaluation.Portfolio CapstonesHard37 min
  25. 25Capstone: Production ML PipelineBuild a point-in-time training and reload path, then test offline gates, simulated canary decisions, promotion, and rollback.Portfolio CapstonesHard46 min