Understanding Gradient Boosting from Scratch: Core Principles Explained

The Core Problem Gradient Boosting Solves

The fundamental goal of machine learning is to find a function f(x) that accurately describes the relationship between inputs X and outputs y. If your data traces out a clean sine wave, you can simply write y = sin(x). But real-world data is far more complex — it's rarely possible to capture it all with a single, simple analytical expression.

One intuitive approach is to use high-degree polynomial regression to approximate complex curves. But this leads to the well-known Runge's phenomenon: as the polynomial degree increases, fitting errors can actually explode at the edges of the interval. In other words, throwing a single, enormously complex model at the problem often does more harm than good.

Gradient Boosting takes the opposite approach — instead of trying to find the entire complex function all at once, it builds up incrementally: starting from a rough initial model, it continuously adds small models, each one specifically designed to correct the mistakes left behind by all the previous models.

Runge's phenomenon was discovered by German mathematician Carl Runge in 1901. He observed that when using equally spaced nodes to interpolate the function f(x) = 1/(1+25x²), increasing the polynomial degree actually caused increasingly violent oscillations at the interval boundaries — errors went up, not down. This reveals a counterintuitive truth: greater model complexity doesn't always translate to greater accuracy. In machine learning terms, this corresponds to overfitting — the model has "memorized" the noise in the training data and generalizes poorly to new data. Gradient Boosting sidesteps this trap structurally, by decomposing the complex problem into a sum of weak learners and using a learning rate to constrain how large each step can be.

Additive Modeling: Decomposing a Big Function into a Sum of Small Ones

The key to understanding Gradient Boosting is embracing one idea: a complex function can be approximated by the sum of many small models.

A useful intuitive example: suppose a seemingly complex curve actually has the following true relationship:

y = sin(x) + x

This is simply the combination of two simple functions:

y = f1(x) + f2(x)   →  f1 = sin(x),  f2 = x

This idea of "summing several small models to obtain a large model" is known as Additive Modeling.

A common misconception is worth clearing up here: these small models don't necessarily correspond to some "component" of the true underlying function. In Gradient Boosting, each new model isn't learning sin(x) or x itself — it's learning the portion of error that the current overall model has yet to fit. Their job is to "fill in the gaps," not to "reconstruct the truth."

The Gradient Boosting Training Process (fit)

Bringing the principle down to code, the Gradient Boosting training procedure breaks into a few clear steps.

Step 1: Build the initial model f0(x). The simplest starting point is to use the mean of y as the initial prediction for all samples:

prediction = np.full(len(X_train), np.mean(y_train))

At this point, "Model 0" knows nothing — it simply outputs a constant: the mean of y.

Step 2: Compute the residuals. The residual is the difference between the true value and the current prediction — it represents "where the model is still getting things wrong":

residual = y - prediction

Step 3: Train a weak learner to fit the residuals. A common approach is to use a shallow decision tree (with tunable depth). The inputs are X_train, but the target values are no longer y — they're the residuals computed in the previous step. This tree's job is to learn "how to make up for the current errors."

Step 4: Update the predictions. Scale the tree's output by the learning rate and add it to the current prediction:

prediction = prediction + learning_rate * model.predict(X_train)

At the same time, save each trained tree for use when predicting on new data later.

The learning rate is a critical regularization mechanism. Multiplying each tree's contribution by a small coefficient (e.g., 0.1) causes the model to learn more slowly and more robustly, which reduces overfitting. Repeat Steps 2 through 4 to continuously generate f1(x), f2(x), f3(x)… — each new tree correcting the errors that remain after all previous trees are combined.

The word "gradient" in Gradient Boosting carries real meaning — understanding it reveals the algorithm's deeper logic. When using mean squared error (MSE) as the loss function, the negative gradient of the loss with respect to the predictions is exactly equal to the residual y - prediction. So fitting the residuals is mathematically equivalent to updating the model in the direction of steepest descent of the loss function — an analog of gradient descent in function space. When a different loss function is used (such as MAE, log loss, etc.), the target value for each round is no longer a simple difference but rather the negative gradient of the corresponding loss. This gives Gradient Boosting tremendous flexibility: by swapping the loss function, the same framework can handle regression, classification, ranking, and more. Engineering implementations like XGBoost and LightGBM are built on exactly this foundation.

The Prediction Process (predict)

Making predictions on new, unseen data follows logic that mirrors training exactly, and is quite concise:

1. Initialize the prediction array with the mean of the training y values (length equals the number of test samples) — this corresponds to the initial model f0(x).
2. Iterate through each tree saved during training and run predictions on X_test.
3. Multiply each tree's output by the learning rate and add it to the current prediction.
4. Return the final accumulated result.

prediction = np.full(len(X_test), np.mean(y_train))
for tree in trees:
    prediction += learning_rate * tree.predict(X_test)
return prediction

The whole process can be summarized as: starting from a mean baseline, a sequence of small trees progressively chips away at the error.

Recap: What Is Gradient Boosting Actually Doing?

Putting all the steps together, the full picture of Gradient Boosting becomes clear:

• f0(x): The initial model — simply the mean of y.
• f1(x): A tree trained on X and the residuals from f0, then prediction = prediction + lr * f1.
• f2(x): Similarly, fits the residuals remaining after f0 + f1.
• f3(x): Continues correcting the errors left by the accumulated model so far…

Each new model stands on the shoulders of all previous models and focuses on tackling the remaining hard cases. The final strong model is the initial mean plus the sum of all weak learners' contributions (each scaled by the learning rate).

This is the essence of Gradient Boosting as an Ensemble Technique: rather than straining to train one all-powerful model, it lets many simple models collaborate and progressively converge on the answer. It sidesteps the oscillation problems of high-degree polynomials and maintains generalization ability through mechanisms like the learning rate — which is precisely why Gradient Boosting and its derivatives (such as XGBoost and LightGBM) remain dominant in real-world applications.

Gradient Boosting and Random Forest — both ensemble methods — are fundamentally different, though beginners often confuse them. Random Forest uses a Bagging strategy: multiple deep trees are trained in parallel, each independently on a bootstrap sample of the training data, and their results are combined by voting or averaging. The goal is to reduce variance. Gradient Boosting uses a Boosting strategy: trees have a strict sequential dependency — each new tree can only begin training after the previous one is complete — and the goal is to progressively reduce bias. Because of this serial dependency, Gradient Boosting is typically slower to train than Random Forest, but it tends to achieve higher predictive accuracy on structured/tabular data. This is the core reason it dominates data competitions like Kaggle.