Implementing XGBoost Regression from Scratch: 5 Days of Hand-Coding, Accuracy on Par with sklearn

A developer spent 5 days implementing the XGBoost algorithm for regression from scratch. On day 54 of the project, he released the result: the hand-written version achieves accuracy almost identical to sklearn's XGBoost, with the only gap being speed—sklearn's implementation is still faster.

Why Implement XGBoost from Scratch?

For machine learning practitioners, implementing classic algorithms from scratch is the best way to deeply understand their internal mechanisms. As a representative of gradient boosted trees, XGBoost involves multiple complex components such as decision tree construction, gradient optimization, regularization, and parallelization. The developer's project "Building Machine Learning Algorithms from Scratch" has reached day 54, and he has likely implemented other algorithms before; XGBoost is one of the more challenging ones.

The value of implementing from scratch is that you no longer treat XGBoost as a black box—you clearly know how the trees grow in each iteration, how the loss function is computed, and how the regularization term affects model complexity. This understanding is crucial for hyperparameter tuning and solving real-world problems.

Implementation Difficulty and Key Points

The core of XGBoost lies in the gradient boosting framework: each round trains a new decision tree to fit the residuals of the previous round (more precisely, the negative gradient of the loss function). Unlike ordinary gradient boosting, XGBoost adds a regularization term to the objective function, controlling the number of leaf nodes and their weights to prevent overfitting. In addition, XGBoost uses second-order Taylor expansion to approximate the loss function, utilizing both first-order and second-order derivative information to make split point selection more precise.

When implementing from scratch, the following key points need to be handled:

• Tree construction: Support custom splitting criteria, typically using a greedy algorithm to traverse features and split points to find the split that maximizes gain.
• Gradient computation: For regression tasks, squared loss is commonly used; its first derivative is the predicted value minus the true value, and its second derivative is the constant 1.
• Regularization: Add penalty terms for the number of leaf nodes and weights in the gain calculation.
• Learning rate: Multiply each tree's contribution by the learning rate to control the step size of model updates.
• Missing value handling: XGBoost by default assigns missing values to the direction with greater gain after splitting.

The developer did not disclose specific implementation details, but it can be inferred that he followed XGBoost's original paper or similar tutorials.

Second-order Taylor expansion is the key mathematical tool that distinguishes XGBoost from traditional gradient boosted trees. Traditional GBDT only uses the first derivative (gradient) of the loss function to fit residuals, while XGBoost performs a second-order Taylor expansion of the loss function around the current prediction, using both the first derivative (gradient) and second derivative (Hessian) to approximate the objective function. The direct benefit is that the split gain calculation can be written in a closed form that depends only on sample gradients and Hessian values, without needing to know the specific expression of the loss function—so the same framework can accommodate squared loss, logistic loss, and other loss functions. For squared loss in regression tasks, the second derivative is the constant 1, and XGBoost's gain formula degenerates to a form similar to ordinary GBDT, but the framework itself remains general. Understanding this makes it clear why XGBoost remains efficient with custom loss functions.

Approximate splitting algorithm is an important means for XGBoost to handle large-scale data. Exact greedy splitting requires traversing all possible split points for each feature; when feature values are continuous and the sample size is huge, the computational cost becomes unacceptable. XGBoost uses a weighted quantile sketch to discretize continuous features into a finite set of candidate split points, and computes gains only on these candidates. This greatly reduces computation while ensuring split quality close to exact greedy. In addition, XGBoost supports column-wise parallelism: when searching for the best split point, computations for different features can be assigned to different threads, because the split point search for each feature is independent. These engineering optimizations are one reason why a hand-written pure Python version struggles to reproduce the speed advantage.

The Trade-off Between Accuracy and Performance

The most noteworthy result is that the hand-written implementation's accuracy is almost identical to sklearn's XGBoost, but it is slower. This is entirely expected—sklearn's XGBoost calls a highly optimized C++ library (such as the official xgboost library) underneath, leveraging engineering techniques like parallel computation, cache optimization, and approximate splitting algorithms. A pure Python (or NumPy) implementation cannot achieve the same efficiency, especially on large datasets.

However, consistent accuracy shows that the algorithm logic is implemented correctly. For learning purposes, speed is not the primary consideration. The developer's result confirms that as long as the core mathematics of gradient boosting is correctly implemented, predictive capability comparable to industrial-grade libraries can be achieved.

Implications for Machine Learning Learners

This project provides a reference path for self-learners: consolidate understanding by implementing algorithms from scratch. The day 54 milestone shows that sustained small-step accumulation can lead to substantial results. If you want to try it too, it is recommended to start with simple algorithms (such as linear regression and decision trees) and gradually move to ensemble methods. For XGBoost, you can first implement basic gradient boosting, then add regularization and second-order derivative optimization.

Of course, in practical work, mature libraries (such as xgboost, LightGBM, and CatBoost) are still recommended, as they offer better speed and stability. But the experience of implementing from scratch will make you more proficient when using these libraries.

Possible Next Directions

The developer did not state his next plans, but based on the project name, he may continue implementing other algorithms such as neural networks and support vector machines. He may also optimize the existing implementation, for example by introducing parallelization or approximate algorithms to speed it up. Either way, this kind of from-scratch building practice is worth following.