Building a CNN from Scratch in C++: Should You Keep Polishing or Move to Something New?

A Real Dilemma Faced by a Beginner Engineer

Recently in Reddit's machine learning community, a first-year computer engineering student shared his project and asked a question that almost every beginner encounters: When a project already works and achieves decent results, should I keep digging deeper or move on to something new?

This student didn't use off-the-shelf frameworks like PyTorch or TensorFlow. Instead, he chose to build a handwritten doodle classifier (Doodle Guesser) from scratch in C++17. The project includes a complete convolutional neural network (CNN) architecture, backpropagation, gradient checking, stochastic gradient descent (SGD), and other core modules, ultimately achieving approximately 94% accuracy on the validation set.

For a first-year student, this is remarkably solid engineering practice. But what's truly worth discussing is the meta-question he raised about "learning rhythm" — which actually conceals a key decision in ML learning paths that's often overlooked.

Why Building a CNN from Scratch Is Inherently Valuable

Core Principles of Convolutional Neural Networks

Before diving into the discussion, it's important to understand why CNNs matter so much. A convolutional neural network is a deep learning model specifically designed to process data with grid-like structure (such as images). Its core idea originates from biological visual systems — extracting features through local receptive fields, then progressively combining them into higher-level abstract representations. The basic components of a CNN include convolutional layers (extracting local features through learnable filters), pooling layers (reducing spatial dimensions and enhancing translation invariance), and fully connected layers (performing final classification). Compared to traditional fully connected networks, CNNs leverage parameter sharing and local connectivity to dramatically reduce the number of parameters, making it possible to train large-scale image recognition models. LeNet-5, proposed by Yann LeCun in 1998, is the classic CNN prototype and remains a standard teaching example for handwritten digit recognition.

First-Principles Understanding Beyond Frameworks

Today, the first line of code for the vast majority of ML beginners is import torch or import tensorflow. The high-level abstraction of frameworks allows you to train a model in just a few dozen lines of code, but it also leaves many concepts at the level of "knowing the name without understanding the principle."

By choosing to hand-write a CNN in C++, this student had to personally handle the following problems:

• Memory layout and loop implementation of convolution operations: No nn.Conv2d to handle tensor dimensions for you
• Chain rule derivation in backpropagation: Manually deriving the gradient formula for each layer
• Gradient checking: Using numerical methods to verify whether analytical gradients are correct — a debugging step that framework users almost never encounter
• SGD optimizer update logic: Implementing parameter updates by hand rather than calling optimizer.step()

Regarding backpropagation and gradient checking, their technical significance deserves further explanation. Backpropagation is the core algorithm for training neural networks — essentially a systematic application of the chain rule on a computation graph. Starting from the loss function, it computes the partial derivative of each parameter with respect to the final loss, layer by layer in reverse. Manually implementing backpropagation requires deriving gradient formulas separately for each layer type (convolution, fully connected, activation functions, etc.) — complexity that framework auto-differentiation hides. Gradient checking is a debugging technique: by applying a small perturbation ε (typically 1e-5 to 1e-7) to each parameter, you approximate the gradient using the numerical difference [f(x+ε) - f(x-ε)] / 2ε, then compare it against the analytical gradient. If the relative error between the two exceeds 1e-5, it usually indicates a bug in the backpropagation implementation. This process is computationally expensive and is used only for debugging, not training.

Being able to independently accomplish all of this and achieve 94% accuracy demonstrates that he has built a first-principles understanding of neural network internals far beyond his peers. This understanding will become an invaluable intuitive foundation when learning more complex models (like Transformers) in the future.

The Engineering Value of a C++ Implementation

Implementing in C++ rather than Python additionally trains skills in memory management, performance optimization, and data structure design. Specifically, C++17 introduces modern features like structured bindings, std::optional, and parallel algorithms, but choosing C++ for machine learning algorithms means the developer must directly manage memory allocation and deallocation, design efficient data layouts (e.g., row-major vs. column-major tensor storage), and handle resource lifecycles under the RAII pattern. These low-level details, completely hidden in Python/NumPy, are precisely the key to understanding how high-performance computing frameworks (such as PyTorch's ATen backend or TensorFlow's XLA compiler) work under the hood. In fact, the core computational kernels of virtually all mainstream deep learning frameworks are written in C/C++, with Python serving only as the upper-level interface.

For a computer engineering student, this dual training in "algorithms + systems" is precisely what many purely application-focused ML learners lack.

Keep Polishing or Move On? A Practical Decision Framework

The Core Question: Evaluating Marginal Learning Returns

The most effective way to answer "continue or move on" isn't to look at how "complete" the project is, but to evaluate marginal learning returns — how much genuinely new knowledge can you gain by spending another month on this project?

You can categorize potential improvements into two types:

Category One: Repetitive Polishing (Low Marginal Returns)

• Adding more classification categories
• Repeatedly tuning hyperparameters to improve accuracy by 1-2%
• Beautifying the output interface

While this type of work can improve the project's "completeness," it offers limited cognitive growth. If you're just trying to make numbers look better, it's easy to fall into the trap the original poster worried about — "spending months endlessly polishing the same beginner project."

Category Two: Introducing New Concepts (High Marginal Returns)

• Data Augmentation: Involves image transformations and random sampling strategies. Data augmentation is a technique that artificially expands the training set by applying random transformations (such as rotation, flipping, scaling, cropping, color jittering, etc.) to training samples. Its theoretical basis is regularization — by increasing the diversity of training data, the model is forced to learn more robust feature representations rather than memorizing specific patterns in training samples. For a handwritten doodle classification task, common augmentation strategies include small-angle rotation (simulating writing angle variations), elastic deformation (simulating natural pen stroke jitter), and random erasing (enhancing robustness to partial occlusion). Data augmentation is virtually a zero-cost performance boost and is especially critical when data is limited.

• Deeper Network Architectures: Introduces new problems like vanishing gradients, batch normalization (BatchNorm), and residual connections. When network depth increases, gradients during backpropagation shrink exponentially (vanishing gradients) or explode (exploding gradients) through layer-by-layer multiplication, making deep networks difficult to train. Batch Normalization (proposed in 2015) stabilizes gradient distributions by normalizing activations at each layer. Residual Connections (proposed in 2015 with ResNet) allow gradients to flow directly to shallow layers through "skip connections," making it possible to train networks with hundreds or even thousands of layers. Kaiming He, the proposer of ResNet, won the CVPR Best Paper Award for this work, and the architecture remains a foundational backbone network in computer vision to this day.

• Different Optimizers: From SGD to Momentum and Adam, understanding adaptive learning rates. Stochastic gradient descent is the most basic optimization algorithm, but it's extremely sensitive to learning rate and converges slowly at saddle points and narrow valleys in the loss surface. To address these issues, researchers successively proposed Momentum (introducing exponential moving averages of historical gradients to accelerate convergence), RMSProp (adaptively scaling each parameter's learning rate), and Adam (combining momentum with adaptive learning rates). Understanding the evolution from SGD to Adam helps learners build intuition about the optimization landscape — why certain hyperparameters work, why training sometimes diverges or oscillates.

• Performance Optimization: Accelerating convolution computation with SIMD or multithreading

Each of these improvements brings substantial new knowledge. If you want to continue with the project, prioritize directions that "force" you to learn new concepts, rather than simply piling on more data or tuning parameters.

A Simple Decision Signal

To determine whether a project has "taught you enough," here's a practical self-check:

When facing the next improvement, if you already "know how to do it and just haven't done it yet," then the learning value of continuing is declining. If you're facing a new problem where you "have no idea where to start," that indicates there's still space worth exploring.

Practical Advice for ML Beginners

Tip 1: Turn the Project into a Presentable, Complete Work

Regardless of whether you ultimately dig deeper, first spend a small amount of time polishing the project to a presentable state: write a clear README, explain the architecture design, and include accuracy curves and implementation details. For a first-year student, a GitHub project titled "CNN built from scratch achieving 94% accuracy" sends an extremely strong signal in internship applications and technical discussions. The ROI of this effort is exceptionally high.

Tip 2: Use a New Project to Continue the Same Learning Thread

Rather than iterating endlessly on the same doodle classifier, start a new project that naturally builds on existing knowledge while introducing new dimensions. For example:

• Use the same from-scratch C++ approach to implement a simple RNN or Transformer to understand sequence models
• Try implementing an automatic differentiation engine (similar to micrograd) to fully abstract gradient computation. Automatic Differentiation is a third approach between symbolic and numerical differentiation — it records the computation process to build a computation graph, then precisely and efficiently computes gradients on that graph. Andrej Karpathy's micrograd is a teaching project of roughly 100 lines of code that implements scalar-level forward computation and backpropagation. PyTorch's autograd module is essentially an industrial-scale extension of micrograd's ideas, supporting tensor operations, dynamic computation graphs, and GPU acceleration. Understanding the core mechanism of automatic differentiation — how to traverse the computation graph via topological sort, how to accumulate multi-path gradients — is the key cognitive leap from "hand-writing gradients for a single model" to "building a general-purpose deep learning framework."
• Switch to PyTorch and compare the differences between "hand-written" and "framework-based," understanding exactly what the framework does for you

This avoids spinning in circles while maintaining continuity rather than making too large a leap.

Tip 3: Beware the Perfectionism Trap

A common misconception among beginners is mistaking "polishing an existing project" for "making learning progress." In fact, breadth-first exploration in early learning stages is often more valuable than depth-first polishing — exposure to different types of models and problems helps you find your true area of interest faster. Once you've identified your direction, it's never too late to invest deeply.

Conclusion

This first-year student's dilemma actually reflects a universal wisdom in ML learning: A project's value lies not in how perfect it is, but in what it taught you. A CNN built from scratch that achieves 94% accuracy has already fulfilled its most important mission — letting the author truly "see through" the internal mechanisms of neural networks.

Going forward, whether continuing to explore data augmentation and deeper networks, or pivoting to entirely new model types, either choice is correct as long as it follows the principle of "maximizing marginal learning returns." For beginners, maintaining forward momentum and continuously engaging with new concepts matters far more than pursuing perfection on a single project.