The PyTorch Mac Memory Illusion: Why Process Footprint Plummets After .to("cpu")

Anyone running PyTorch on a Mac may have run into a counterintuitive phenomenon: after moving a tensor from the MPS device to the CPU, Activity Monitor's reported memory usage drops to nearly zero almost instantly, and the MPS driver counter falls in lockstep—yet the data is still there, and nothing was actually freed.

This phenomenon comes from a minimal reproduction in a diffusers discussion, using a 4 GiB tensor to simulate model weights (torch 2.14.0, macOS 26.5.2, M5 Pro 48 GB). The code reads the process's phys_footprint (the physical footprint shown in Activity Monitor's Memory column) via libproc, and simultaneously records torch.mps.driver_allocated_memory(). For readers unfamiliar with the low-level APIs, here's the original author's source screenshot as a quick visual.

Reproduction Results: The Numbers Don't Lie

Three runs produced identical output:

start          footprint  0.1 GiB, mps driver  0.0 GiB
on mps         footprint  4.2 GiB, mps driver  4.0 GiB
moved to cpu   footprint  0.1 GiB, mps driver  0.0 GiB
cpu copy read  footprint  4.2 GiB, mps driver  0.0 GiB

• Right after moving to CPU, both counters dropped back close to zero, yet w still held the full 4 GiB of data intact.
• Only after the CPU read through the copy page by page did the footprint climb back to 4.2 GiB.

In other words, if you sample memory between those two steps, you get the illusion of "memory freed." The author repeatedly emphasizes in the original experiments: the copy was there the whole time, and the memory it occupies never shrank—only the way the OS accounts for it changed.

The Hardware-Level Root Cause

Why does .to("cpu") not actually free anything at first? The answer lies in the hardware architecture. On Apple Silicon, the CPU and GPU share the same pool of physical memory (unified memory architecture).

This means: on a PC, moving weights from GPU to CPU frees VRAM for the next model; on a Mac, the so-called "on CPU" weights actually sit right in the same unified RAM that the GPU also needs, so memory pressure doesn't drop at all.

For a cross-device operation like PyTorch's, the copy from MPS to CPU just relocates within unified memory. The underlying physical pages may be marked compressible or already swapped out, but until the next access, the process footprint indeed shows as near zero.

What phys_footprint is: phys_footprint is a process-level memory metric that macOS exposes via libproc, and it's exactly what Activity Monitor's Memory column displays. It isn't the process's total virtual memory, but an estimate of "physical pages actually resident and counted toward system memory pressure," including anonymous mappings, some dirty pages, and compressed pages, while subtracting pages that have been swapped out. So under a unified memory architecture, when pages are marked compressible or have entered swap, phys_footprint drops rapidly—even though the tensor object is still valid. That's why it briefly hits zero after the MPS→CPU copy and climbs back when the CPU reads page by page: pages are only re-counted in the footprint once they're touched again.

Unified Memory Architecture (UMA): Apple Silicon connects the CPU, GPU, and other coprocessors to the same physical RAM, shared over a high-bandwidth on-chip interconnect, with no separate VRAM pool. This is fundamentally different from traditional PC dGPU architectures: those have dedicated video memory, and operations like cudaMemcpy move data between VRAM and system memory, so the VRAM really is freed after offloading. On UMA, "moving a tensor to CPU" only changes the access path and page state—the physical memory stays in the same pool, so it does almost nothing to relieve overall memory pressure. MPS's driver_allocated_memory counter reflects how much the MPS driver currently claims from the system; it's a separate ledger from the process phys_footprint, which is why the two numbers in the original article fall in sync yet don't represent actual freeing.

What This Means for Model CPU Offload

diffusers's enable_model_cpu_offload() strategy is built precisely on the idea of "keep the model on CPU, only move it to GPU at runtime." The author ran Qwen-Image-2.1 on the same Mac at 1024×1024 resolution for a real test (see the trace and write-up), and the results are worth being wary of:

• Process footprint never exceeded 18.5 GB at peak—it looks like a 48 GB Mac has plenty of headroom.
• But swap grew by 3.5 GB during denoising, and by the VAE decode stage it had grown 9.6 GB.
• Ultimately the memory guard proactively terminated the run before the image was written out.

The author nails the key point: plotting only the footprint curve tells you "this run fits," while swap is actually creeping up in the background. This echoes the 4 GiB reproduction from the start—footprint can stay low even while the weights are still in memory.

How to Check Your Own Runs

The author's advice is: watch swap, not just footprint.

sysctl vm.swapusage             # compare the "used" field before and after
sysctl kern.memorystatus_level  # system-wide free memory percentage

Two decision rules:

1. As long as swap grows while a task runs, no matter how good the footprint looks, it means it doesn't fit.
2. The system-wide free memory percentage reflects global pressure; looking only at process memory easily leads to misjudgment.

The author wrote a small tool for this, stageload (pip install stageload), in which MemoryMeter records both metrics every 0.5 seconds, and stageload guard can terminate a command when swap exceeds budget:

stageload guard --wait-free 40 --swap-budget 8G -- python generate.py

The Way to Actually Free Memory

For pipelines that run multiple models in sequence, the author's approach isn't to park models on CPU, but to "load at stage start, release at stage end":

• stageload's release(module) moves all parameters and buffers to PyTorch's meta device, then triggers garbage collection and empties the MPS cache.
• Tensors on meta have no storage, so even if other code still holds a reference to the module, the memory can truly be returned.

The author used this approach to load Qwen-Image-2.1: the text encoder (16.3 GB) and the transformer (13.3 GB) are loaded one at a time. On the same Mac, the peak dropped to 19.0 GB, and swap didn't grow on either run. For the full configuration, see the article "How to run Qwen-Image-2.1 on a 48 GB Mac without swap."

Boundaries Not Yet Verified

The author also honestly lists what wasn't measured:

• System-wide free memory during the example runs: other tasks on this Mac cause fluctuations of several GiB between runs, so no clean numbers could be obtained.
• Whether del + torch.mps.empty_cache() frees as thoroughly and how fast it is compared with migrating to meta—no comparison was done.
• Other macOS / torch versions, and Macs with more memory—not covered.

These open questions also remind iOS/macOS developers: when profiling on a unified memory architecture, process-level footprint is an unreliable single point of measurement; swap and system-wide free memory must be observed together. The author's combination of reproduction + tool + boundary disclosure provides a reusable observational baseline for those who follow in doing model offload on Mac.

How the meta device works: PyTorch's meta is a kind of "bodiless" device—tensors keep only metadata like shape, dtype, and stride, and allocate no underlying storage. After moving a module's nn.Parameters and buffers all .to('meta'), these objects still exist and references remain valid, but their memory footprint is zero. Then call gc.collect() to clear intermediate references, and use torch.mps.empty_cache() to reclaim the MPS cache, to truly return memory to the system. Compared with del module, the key advantage of the meta approach is that it can complete the release without breaking external references, making it suited for safely "unloading" a spent stage in a pipeline without triggering dangling-reference errors. Note that a migrated module can no longer run forward computation unless you reload the parameters with .to(device).

Background Notes

What swap and memorystatus_level mean: the used field reported by sysctl vm.swapusage is the cumulative number of bytes the system has written to swap; its growth means physical memory is maxed out and the kernel has begun paging out to disk. kern.memorystatus_level is the system's measure of "free memory percentage," and macOS's memory guard (jetsam) uses it to decide whether to terminate processes. Only together do they reflect real pressure: if you look only at process footprint, pages being swapped out make the curve look light, while swap growth exposes the actual tightness. This is why the author suggests comparing these two metrics before and after a task, and using whether swap grows as the hard criterion for "fits / doesn't fit."

One-line Summary

On Mac unified memory, MPS→CPU moves are a measurement illusion; judge memory by swap, not footprint.

Paragraph Summary

Starting from a minimal reproduction, the article reveals a counterintuitive phenomenon on Apple Silicon: after moving a 4 GiB tensor from MPS to CPU, the process footprint and MPS driver counter drop to nearly zero at the same time, yet the data is still in memory, and the footprint only climbs back once the CPU reads page by page. The root cause is unified memory architecture—CPU and GPU share the same pool of physical RAM, so "moving to CPU" frees nothing; only the OS's accounting method changes. Based on this, the author points out a trap in diffusers's enable_model_cpu_offload() on Mac: when running Qwen-Image-2.1, footprint never exceeded 18.5 GB, but swap had grown to 9.6 GB and triggered the memory guard. The right approach is to watch swap and system free memory, or use stageload to migrate model parameters to the meta device and empty the MPS cache, achieving genuine memory return.