Building Computer Vision Models from Scratch: Book Recommendations Worth Reading

Many developers first truly understood how each layer inside a language model works through Sebastian Raschka's Build a LLM from Scratch series. Now, a group of readers with the same aspirations has begun asking in communities: is there an equivalent book for computer vision? Not just calling torchvision.models.resnet50() and calling it a day, but building CNNs, ViTs, and Swin Transformers line by line.

Behind this question lies a deeper need: CV learning has long remained at the "calling libraries" level. Data augmentation, pretrained weights, transfer learning — these tools are so convenient that they actually block the path to understanding the essence of models. The reason Raschka's writing style is so popular is precisely that he forces readers to confront every matrix multiplication and attention head.

Why the CV Field Lacks "From Scratch" Books

There is an objective condition behind the success of from-scratch LLM books: the Transformer architecture itself is extremely unified. Once you understand the attention mechanism, the differences between GPT, LLaMA, and Mistral come down to stacking patterns and training strategies.

But the model lineage in computer vision is far more diverse. LeNet, AlexNet, VGG, ResNet, DenseNet, EfficientNet, then Vision Transformer, Swin, ConvNeXt — each architecture solves a specific flaw exposed by the previous one. Residual connections solve vanishing gradients, depthwise separable convolutions compress parameter counts, window attention reduces ViT's quadratic complexity. Weaving these threads together clearly is an order of magnitude harder than explaining LLMs.

This is also why classic CV books tend toward a "survey-style" approach rather than Raschka's "hand-hold you through building one" style.

Depthwise Separable Convolution is a core building block introduced by the MobileNet series to significantly compress computation while maintaining accuracy. It splits standard convolution into two steps: first, apply a convolution independently to each input channel (depthwise), then use a 1×1 convolution to recombine the channels (pointwise). Standard convolution costs roughly (D_k^2 \cdot M \cdot N \cdot D_f^2), while depthwise separable convolution costs roughly (D_k^2 \cdot M \cdot D_f^2 + M \cdot N \cdot D_f^2), typically reducing to one-eighth or one-ninth of the original with 3×3 convolutions. Understanding this decomposition is key to grasping why MobileNet and EfficientNet can land on mobile devices, and it's also the place where channel dimensions are most easily gotten wrong when "building from scratch."

Window Attention is Swin Transformer's answer to ViT's quadratic complexity problem with global attention. In ViT, every patch computes attention with all other patches, giving (O(N^2)) complexity for sequence length (N). Swin partitions the image into non-overlapping local windows and performs self-attention only within windows, reducing complexity to linear; Shifted Windows then allow adjacent windows to exchange information, compensating for the loss of global modeling caused by locality. This hierarchical design of "local attention + shifting" is a typical example of what the article calls "each architecture solves a specific flaw exposed by the previous one."

CV Books Closest to Raschka's Style

Programming Computer Vision with Python — A Hands-On Entry Point

Jan Erik Solem's book uses Python and NumPy, starting from the most basic image processing and covering topics like feature extraction, image registration, and 3D reconstruction. Its strength is that the code is extensive and low-level enough that readers must implement classic algorithms like SIFT themselves. The downside is that it was published relatively early and completely misses Transformer-era CV models.

If you're already familiar with PyTorch fundamentals, you can skip the first few chapters and start directly from the feature engineering section.

SIFT (Scale-Invariant Feature Transform) is a classic local feature algorithm proposed by David Lowe in 1999 and refined in 2004, and it's one of the representative algorithms this book has readers implement by hand. Its pipeline roughly includes: constructing a Difference-of-Gaussians pyramid to detect scale-space extrema, filtering out low-contrast and edge-response points, assigning a dominant orientation to each keypoint to achieve rotation invariance, and finally generating a 128-dimensional gradient histogram descriptor. SIFT is classic because it simultaneously possesses scale, rotation, and brightness invariance, making it a long-standing staple in image stitching, 3D reconstruction, and object retrieval. Implementing it by hand gives an intuitive understanding of what "features" actually are, and lets you appreciate why, before deep learning emerged, so much CV research effort went into hand-designing invariances — which is precisely where CNN's value lies in replacing manual design with learning.

Deep Learning for Vision Systems — Balancing Understanding and Implementation

Mohamed Elgendy's book does a solid job of implementing classic CV models with PyTorch. It starts from the perceptron and gradually moves to CNNs, object detection (YOLO, Faster R-CNN), image segmentation, and finally touches on vision Transformers. Every chapter has runnable code examples.

Compared to Raschka's book, it falls somewhat short in the meticulousness of mathematical derivations, but it wins on breadth of coverage and offers fairly thorough discussion of "why each architecture is designed this way."

Faster R-CNN and YOLO represent the two main technical routes in object detection, and understanding their divergence helps in reading the evolution of subsequent architectures. Faster R-CNN is a two-stage detector: it first uses a Region Proposal Network to generate candidate boxes, then performs classification and regression on each candidate box — high accuracy but slow. YOLO, on the other hand, unifies detection into a single regression problem, dividing the image into a grid where each cell directly predicts bounding boxes and classes in one forward pass — extremely fast. The difference is essentially a trade-off between "locate then judge" and "one step to done," which also explains why the YOLO series is more popular in real-time scenarios while two-stage methods still hold their ground on accuracy-sensitive academic benchmarks.

Dive into Deep Learning — Free and Continuously Updated

Although D2L by Mu Li's team isn't specifically a CV book, its CV chapters (including CNN, modern convolutional networks, and computer vision applications) are of high quality, and the code is PyTorch-based and can be run line by line. Most importantly: it is continuously updated, and ViT-related chapters have been added.

For readers who want to "build from scratch," D2L's interactive learning experience may be closer to Raschka's philosophy than any paper book.

Supplementary Books from a Single Source

In that Reddit discussion thread, a user mentioned Modern Computer Vision with PyTorch (published by Packt), which covers a broad range of topics from basic CNNs to GANs and diffusion models, with abundant code examples. Others recommended Fast.ai's course as an alternative — it's not a book, but its "top-down" teaching method (run a complete project first, then dig into the underlying layers) suits people with different learning styles.

It should be noted that these recommendations come from community users' personal experience rather than systematic evaluation. Readers can make their own choices based on their learning preferences.

How to Learn CV the "From Scratch" Way

Regardless of which book you choose, the effective learning path is roughly the same:

1. First implement the simplest CNN, get it running on MNIST or CIFAR-10, and understand the actual effects of convolution kernels, stride, and pooling.
2. Reproduce ResNet's residual block, comparing training convergence with and without skip connections — this experiment is more convincing than any written description.
3. Manually implement multi-head self-attention, then build a mini-ViT on a small-scale dataset. This step will expose many dimension-matching problems you'd "never encounter when calling libraries."
4. Read the source code — every model in PyTorch's torchvision.models has a corresponding implementation file, and reading it line by line is more direct than any tutorial.

This path is in the same vein as Raschka's methodology: first get the code running, then understand why it's written this way, and finally try modifying the design and observing the effects.

Multi-Head Self-Attention is the core operation of Transformers and ViTs, and manual implementation usually gets stuck on several dimension details. Given input (X \in \mathbb{R}^{N \times d}), first obtain Query, Key, and Value through three linear transformations; then split into (h) heads, each with dimension (d/h), each computing (\text{softmax}(QK^\top/\sqrt{d/h})V); finally concatenate the outputs of all heads and project back to (d) dimensions. In CV, (N) is the number of patches (e.g., (N=196) when a 224×224 image is split into 16×16 patches), and positional encoding must be added separately because attention itself is order-insensitive. Clarifying these tensor shapes is the most effective step from "calling models" to "understanding models."

Final Thoughts

Strictly speaking, there is currently no CV book that fully matches Raschka's LLM from-scratch series. The diversity of CV model architectures makes this inherently harder. But Deep Learning for Vision Systems and D2L's CV chapters can be used in combination, and together with cross-referencing PyTorch's official source code, they are enough to build a sufficiently deep understanding.

The more convenient the tools, the scarcer the understanding. Those willing to spend time building CV models from scratch gain not just knowledge, but also the confidence that "I know how it ticks" when facing a new architecture.

One-line Summary

There's no CV equivalent to Raschka's from-scratch book, but D2L plus several hands-on books can serve as a substitute.

Paragraph Summary

Starting from a community question, the article explores why computer vision lacks a "from scratch" book like Raschka's Build a LLM from Scratch. The core reason is that CV's model lineage is far more diverse than LLMs — from LeNet to ResNet, ViT, and Swin, each architecture patches a specific flaw of its predecessor, making a coherent narrative harder. The article recommends several resources closest to this philosophy: Programming Computer Vision with Python is low-level but doesn't cover Transformers, Deep Learning for Vision Systems balances understanding and implementation, and Mu Li's D2L is free, continuously updated with ViT chapters, and the most interactive. The community also mentions Modern Computer Vision with PyTorch and Fast.ai's course as supplements. Finally, it offers a learning path: implement a simple CNN, reproduce ResNet's residual block, hand-write multi-head attention to build a mini-ViT, and read PyTorch source code line by line.