Linear Regression from Scratch

Linear regression is the simplest useful model for predicting a number from features. Building it from scratch teaches features, weights, residuals, and optimization without neural-network complexity. This chapter starts from zero and builds toward the concrete job skill: Implement linear regression with NumPy, compare closed-form least squares with gradient descent, and inspect residuals. ^[1][2][3]

Step map

Start here

Start with a line. The model says prediction = intercept + slope * feature. With more features, the line becomes a weighted sum, but the habit is the same: multiply inputs by weights, compare to targets, and reduce error.

Read this chapter once for the idea, then run the demo and change one value. For Linear Regression from Scratch, progress means you can name the input, explain the operation, and say what result would prove the idea worked.

By the end, you should be able to explain Linear Regression from Scratch with a worked example, not a library name. Keep one runnable file and one short note with the result you expected before you ran it.

Why this chapter matters

Linear Regression from Scratch matters because later LLM work assumes this habit already exists. You will use it when you inspect data, debug model behavior, compare evaluations, or explain why a result should be trusted.

The job skill here is: Implement linear regression with NumPy, compare closed-form least squares with gradient descent, and inspect residuals. Treat the snippet as lab equipment: run it, change one input, and write down what changed before you move on.

Beginner mental model

Imagine predicting a score from one numeric feature. You add a column of ones for the intercept, solve for weights, then inspect residuals to see what the model missed.

A useful beginner checklist for Linear Regression from Scratch:

1. What object enters the system?
2. What transformation happens to it?
3. What evidence says the result is correct?

Keep the answer concrete. If you can't point to the value, shape, row, metric, or test that proves the point, the Linear Regression from Scratch concept is still fuzzy.

Vocabulary in plain English

• feature: input column used by the model.
• weight: number the model learns for a feature.
• least squares: choosing weights that minimize squared prediction errors.
• residual: actual value minus predicted value.
• gradient descent: iterative method that nudges weights downhill on loss.
• regularization: penalty that discourages fragile or overly large weights.
• baseline: simple comparison model, such as always predicting the mean.

Use these definitions while reading the demo. Each term should map to a variable, an assertion, or a decision you could explain in review.

Build it

Start with the smallest version that can run from a terminal. The goal for this Linear Regression from Scratch demo is visibility: one file, one output, and no hidden notebook state.

python
1import numpy as np
2
3X = np.c_[np.ones(5), [1, 2, 3, 4, 5]]
4y = np.array([2.0, 3.1, 3.9, 5.2, 6.1])
5w = np.linalg.solve(X.T @ X, X.T @ y)
6pred = X @ w
7print(w, y - pred)

Read the code in this order:

• np.c_[np.ones(5), ...] creates an intercept column plus one feature column.
• np.linalg.solve(X.T @ X, X.T @ y) solves the least-squares normal equation.
• pred = X @ w turns features and weights into predictions.
• y - pred shows residuals, which tell you where the line misses.

After it runs, make three small edits. Add a normal-case test, add an edge-case test, then log the intermediate value a beginner would most likely misunderstand. That turns Linear Regression from Scratch from a reading exercise into an engineering exercise.

For Linear Regression from Scratch, a strong submission includes a runnable command, one test file, and notes for any assumptions. If data, randomness, training, or evaluation appears, save the split rule, seed, config, and metric definition.

Beginner failure case

A beginner may stop at low average error and never check whether residuals show a pattern the model can't represent.

For Linear Regression from Scratch, make the failure visible before adding the fix. Write the symptom in plain English, then add the smallest guard that would catch it next time.

Good guards for Linear Regression from Scratch are concrete: assertions, fixture rows, duplicate checks, seed control, metric intervals, or release checks. Pick the guard that makes the hidden assumption executable.

Practice ladder

1. Run the snippet exactly as written and save the output.
2. Change one input value and predict the output before running it again.
3. Add one assertion that would catch a beginner mistake.
4. Add a naive baseline that predicts y.mean(), then compare its mean squared error with the learned line.
5. Write a two-line README: one command to run the demo, one command to run the test.

Keep this ladder small. Linear Regression from Scratch should feel runnable before it feels impressive. The capstones later reuse the same habit at product scale.

Production check

Always compare against a naive baseline, plot residuals, record feature transforms, and test the training code on a tiny dataset with known weights.

A production check for Linear Regression from Scratch is proof another engineer can trust the result. At foundation level that means a reproducible command and tests. At capstone level it also means a design note, eval evidence, cost or latency notes, and rollback criteria.

Before moving on, answer four Linear Regression from Scratch questions: What input does this accept? What output or metric proves it worked? What failure would fool you? What test catches that failure?

What to ship

Ship a small Linear Regression from Scratch folder with code, tests, and notes. Make it boring to run: install dependencies, run tests, run the demo. That boring path is what makes the artifact useful in a portfolio.

Linear Regression from Scratch feeds later LLM engineering work directly. Retrieval, fine-tuning, agents, evals, and serving all depend on small foundations like this being clear before systems get large.