Simple Linear Regression from Scratch: Least Squares Derivation and Python Implementation

Why Implement Linear Regression from Scratch

In any machine learning learning path, Simple Linear Regression is typically the first algorithm you encounter. Thanks to highly abstracted libraries like scikit-learn, we can build a model in just a few lines of code by calling LinearRegression().fit(). But this convenience comes with a cost: many learners understand what to do without understanding why — they have no grasp of the underlying math, error evaluation, or statistical testing behind the model.

Recently, a developer shared a hands-on project in the community that implements simple linear regression entirely from scratch, without relying on sklearn at all. The value of this project isn't in the complexity of the code — it's in the way it systematically walks through the complete chain from intuitive understanding to statistical inference. This article breaks down the core concepts you need to master when implementing linear regression from scratch.

The Mathematical Essence of Linear Regression: From Intuition to Formula

The Basic Assumption of Linear Regression

The core idea behind simple linear regression is straightforward: we assume that an approximate linear relationship exists between the independent variable X and the dependent variable Y:

Y ≈ β0 + β1·X

Here, β0 is the intercept and β1 is the slope. The entire modeling process boils down to finding the optimal β0 and β1 that make this line fit all data points as well as possible.

Solving for Regression Coefficients with Least Squares

How do we define "as well as possible"? This is where the Residual Sum of Squares (RSS) comes in. A residual is the difference between each actual value and its predicted value. We want to minimize the sum of all squared residuals. By taking the partial derivatives of RSS with respect to β0 and β1, then setting them to zero, we can derive a closed-form solution:

β1 = Σ(xi - x̄)(yi - ȳ) / Σ(xi - x̄)²
β0 = ȳ - β1·x̄

These two formulas are the foundation of linear regression. In a from-scratch implementation, you simply iterate over the data to compute the mean, covariance, and variance — no machine learning library required. This step gives you a genuine understanding of what "training a model" actually means: it's not a black box, but a set of explicit mathematical operations.

Statistical Evaluation of Model Reliability

Calculating β0 and β1 and making predictions only gets you halfway through the job. What truly separates someone who deeply understands the material from someone who just calls library functions is the ability to statistically evaluate model reliability.

Standard Error and Confidence Intervals

The β0 and β1 we calculate are only estimates based on sample data — they carry their own uncertainty. Standard Error (SE) measures the degree of variability in those estimates. Using standard error, we can construct a 95% Confidence Interval:

β1 ± 2·SE(β1)

The interpretation: if we repeatedly sampled and built models, the true β1 would fall within this interval roughly 95% of the time. A narrower interval means we have more confidence in our coefficient estimate.

Hypothesis Testing and p-values: Is the Coefficient Actually Meaningful?

A critical statistical question is: does a real relationship exist between X and Y, or is it just a coincidence caused by random noise? This requires hypothesis testing.

We set the null hypothesis H0: β1 = 0 (i.e., X has no effect on Y). We then compute the t-statistic:

t = β1 / SE(β1)

From this, we derive the corresponding p-value using the t-distribution. If the p-value is small enough (typically below 0.05), we have sufficient evidence to reject the null hypothesis and conclude that a statistically significant linear relationship exists between X and Y. This step is one that many sklearn-only users completely skip — yet it's precisely what determines whether a model has any real-world meaning.

Two Core Metrics for Measuring Goodness of Fit

Residual Standard Error (RSE)

Residual Standard Error (RSE) measures the average prediction deviation of the model — the typical magnitude by which predictions deviate from actual values. The smaller the RSE, the better the model fits the data. Since RSE is expressed in the same units as Y, it is highly interpretable.

R² (Coefficient of Determination)

R² is the most commonly used metric for evaluating goodness of fit, ranging from 0 to 1:

R² = 1 - RSS/TSS

where TSS is the Total Sum of Squares. R² represents the proportion of the total variance in the dependent variable that the model explains. An R² closer to 1 indicates stronger explanatory power. That said, a high R² doesn't always mean a good model — in complex scenarios, residual analysis and other methods should be used together for a comprehensive assessment.

Key Takeaways from Implementing Linear Regression from Scratch

Compared to simply calling sklearn, building linear regression from scratch delivers several irreplaceable benefits:

• Understanding the mathematical essence: Manually deriving the least squares coefficient formulas clarifies how a model actually "learns";
• Developing statistical thinking: Standard error, confidence intervals, hypothesis testing, and p-values are core tools for judging model credibility — not optional decorations;
• Building an evaluation framework: Knowing how to use RSE and R² to assess model performance from different angles;
• Improving debugging ability: When a model behaves unexpectedly, someone who understands the underlying principles can quickly pinpoint the problem — while someone who only knows how to call APIs is often left helpless.

Conclusion

In an era dominated by large models and automated tools, writing a simple linear regression from scratch might seem "outdated" — but it's actually one of the best ways to build a solid machine learning foundation. It reminds us that genuine technical ability comes from a deep understanding of principles, not just fluency with APIs.

If you're learning machine learning, consider starting with exactly this kind of exercise — no sklearn, just walk through intuition, mathematical derivation, prediction, error evaluation, and statistical testing end to end. When you can implement and explain every formula by hand, your understanding of linear regression — and machine learning as a whole — will reach an entirely new level.