Free & Open-Source: A New Book on Systematically Speeding Up ML Models, from Silicon to Agents

Machine learning engineers often fall into a common trap: assuming that reducing a model's floating point operations (FLOPs) will automatically make it run faster. One ML performance engineer spent several months writing a free, open-source book dedicated to dismantling this misconception. It's called How to Make Your Model Fast: A Systems View of Efficient Machine Learning, from Silicon to Agents.

Core Argument: Identify the Bottleneck Before You Optimize

The most valuable claim in this book is simple but counterintuitive — reducing FLOPs doesn't necessarily make your model faster. Before you start optimizing anything, you need to understand what's actually constraining your system. The author's methodology revolves around a few key questions:

• What's the theoretical maximum speed for this model on this hardware?
• Is my workload compute bound, bandwidth bound, memory bound, or system bound?
• Which optimization approach can actually raise the performance ceiling?
• In my current scenario, is quantization, pruning, or kernel optimization even worth doing?
• What happens when you apply the same reasoning to serving infrastructure and agent systems?

The core of this framework is building intuition: given a model and a piece of hardware, you should be able to reason about where the performance ceiling lies and which optimization lever is worth pulling. That's far more practical than blindly applying optimization tricks.

Distinguishing between the four bound types is critical for real engineering decisions. Compute Bound means the hardware's compute units are fully saturated — this is when increasing batch size or optimizing numerical precision (e.g., switching from FP32 to BF16) actually helps. Bandwidth Bound means data movement can't keep up with computation, which is common in scenarios where weights are large but compute is light (like single-step LLM decoding) — quantization to reduce weight size is the right tool here. Memory Bound typically refers to insufficient VRAM, which limits batch size or model size, requiring offloading or model parallelism. System Bound means non-GPU factors like CPU scheduling, PCIe transfers, or Python interpreter overhead become the bottleneck — solutions involve system-level approaches like async execution and reducing synchronization points. Each bound type calls for a completely different optimization strategy, and confusing them is the root cause of a huge amount of wasted effort.

A Complete Technical Stack from Roofline to Agents

The book is structured bottom-up. It begins with Roofline analysis and hardware fundamentals — the classic model for identifying which type of bound you're hitting. By comparing a workload's arithmetic intensity against the hardware's compute-to-bandwidth ratio, you can immediately see where your workload is bottlenecked.

From there, the content builds upward layer by layer, covering:

• Kernels: Optimizing low-level compute units
• Compilers: Unlocking performance through compilation-level techniques
• Quantization and Pruning: The two major approaches to model compression
• Vision models and on-device LLMs: Real-world deployment considerations for specific use cases
• Robotics: A domain with extreme real-time requirements
• Profiling and Serving
• Agents: Extending the systems performance mindset to modern agent architectures

This span from silicon to agents covers the full stack — from hardware fundamentals to application-level systems — making it a rare and comprehensive map for anyone who wants to understand ML performance engineering systematically.

The Roofline model, introduced by Williams et al. in 2009, is a foundational tool in performance analysis. Its core idea is to express the performance ceiling of any computation as the minimum of two "roofs": peak hardware throughput (FLOP/s) and the product of memory bandwidth and arithmetic intensity. Arithmetic intensity is defined as the number of floating-point operations performed per byte of memory accessed (FLOP/Byte). When an operation's arithmetic intensity falls below the hardware's "ridge point" (peak compute divided by peak bandwidth), it is bandwidth bound; otherwise, it is compute bound. Matrix multiplication, for example, is typically compute bound during batched inference but almost always bandwidth bound during batch-size-1 autoregressive decoding — which is precisely why reducing parameter count (and thus FLOPs) barely helps in the latter case, while reducing model weight size through quantization is the correct approach. Mastering Roofline analysis is the first step from "optimizing by gut feeling" to "making evidence-based decisions."

Why This Systems Perspective Matters

A large portion of model optimization work today stays at the surface level — swap in a smaller model, throw on quantization — without any real ability to identify where the system bottleneck actually is. The result is often wasted engineering effort with no meaningful performance gain, because the optimization never touched the real constraint.

The author's motivation is straightforward: this is the book he wished had existed when he was starting out in ML performance engineering. That kind of origin story — born from real pain points — typically means the content is grounded in genuine engineering problems rather than abstract theory. For practitioners in inference optimization, compilers, edge AI, and performance engineering, this book offers a transferable analytical framework, not just a collection of disconnected tips.

Completely Free and Open Source

The entire book is free and open source, hosted on GitHub: usamahz/make-your-model-fast. The author is actively soliciting feedback and contributions, particularly from readers working in ML systems, inference, compilers, edge AI, or performance engineering.

For engineers stuck in the "I optimized it but it didn't get faster" trap, the book's "diagnose first, then treat" approach is well worth reading. Its goal isn't to build proficiency in any single technique — it's to develop judgment: knowing which tool fits which situation, and which optimizations simply aren't worth pursuing.