Phase-HDC: Memory-Free Training by Replacing Optimizer History with Gradient Thresholds

When Training Consumes More Memory Than Storage

Here's a counterintuitive fact: training a compact model often requires far more memory than storing the model itself. The culprit is the optimizer — mainstream optimizers like Adam maintain their own "memory" by tracking gradient momentum from past updates. For typical models, this overhead is acceptable. But for one particular class of models, the cost becomes intolerable.

A new paper on arXiv, Phase-HDC: Replacing Optimizer History with Gradient Thresholds in Discrete Phase Learning, focuses on hyperdimensional classifiers (HDC). These models store learnable parameters as low-bit angular values, which the paper calls "phase memory." The problem: when the parameters themselves occupy only a handful of bits, Adam's per-parameter gradient history can consume several times more memory than the model itself. This leads the researchers to ask a sharp question — can we train a model while storing nothing beyond the model itself?

Hyperdimensional classifiers (HDC) are a class of machine learning models inspired by associative memory in the brain. Their core idea is to encode data as extremely high-dimensional (typically thousands to tens of thousands of dimensions) binary or low-bit vectors, then perform classification or retrieval in that high-dimensional space. Compared to neural networks, HDC inference is computationally simpler (mainly vector addition and similarity queries), making them a natural fit for resource-constrained embedded systems. Their parameters are stored as low-bit angular values — "phase memory" — and this extreme compression is precisely what makes the float32 momentum terms that Adam maintains per parameter so bloated relative to the model itself. For example, if a parameter needs only 2–4 bits of storage, but Adam requires two float32 momentum terms (64 bits total) per parameter, the optimizer state alone can be more than ten times larger than the parameters it tracks.

The Core Rule Behind Phase-HDC

Phase-HDC's answer is surprisingly simple. For each stored angular parameter, it rotates by at most one step per update, in the direction opposite the sign of the current gradient — and only when the gradient is large enough. In other words, it replaces the optimizer's dependence on historical momentum with a single threshold test applied to the current gradient.

This rule isn't arbitrary. The paper proves it is the exact optimal solution to a first-order loss model in which every parameter change carries a fixed cost. This means "only update when it's worth it, and move just one small step at a time" is a mathematically optimal strategy, not an empirical heuristic.

Why a Threshold Can Replace Momentum

Instrumented training experiments in the paper reveal the underlying mechanism. Once parameters must lie on a discrete grid, Adam's momentum is primarily doing one thing: deciding whether a given parameter should move at all. That decision can be made entirely by a threshold on the current gradient — no history required. More critically, when momentum itself is coarsely quantized, this "move or not" judgment breaks down for rare inputs in the data — which explains why low-bit Adam collapses in certain scenarios.

Adam's momentum mechanism (first- and second-moment estimation) was designed to smooth gradient noise and adaptively scale learning rates in continuous parameter spaces. But in a discrete parameter space — where a phase parameter can only occupy a finite set of angular grid points — updates are binary: either jump one grid step or don't. The smooth gradient signal that momentum provides in continuous space becomes redundant; the effective decision logic degenerates into a simple threshold on update magnitude. Low-bit Adam fails on rare inputs because their gradients are weaker, and once momentum is coarsely quantized, these faint signals get swamped or clipped, leaving the corresponding parameters permanently frozen. Phase-HDC's threshold mechanism operates directly on raw gradients, sidestepping the information loss introduced by quantized momentum.

The Memory–Accuracy Tradeoff

Phase-HDC's value is most clearly seen in storage efficiency. In controlled experiments that swap only the update rule while keeping everything else identical, it matches 6-bit momentum Adam's accuracy at one-third the storage.

Testing across eleven datasets spanning image, tabular, and text domains yields a fuller picture:

• 16–23× reduction in storage versus standard float32 Adam
• 4–6× reduction versus 8-bit Adam
• An average accuracy penalty of roughly 5 percentage points relative to float32 Adam

The comparison against 8-bit Adam is particularly striking: across six of the eleven datasets, Phase-HDC actually achieves higher accuracy, including on a byte-level text prediction task where 8-bit Adam outright collapses. This corroborates the mechanistic explanation above — coarse quantization corrupts update decisions for rare inputs, while Phase-HDC's threshold rule naturally avoids this failure mode.

How to Interpret This Work

At its core, Phase-HDC is an exercise in subtraction: rather than compressing or quantizing the optimizer's history, it directly questions whether that history is necessary at all — and provides a theoretical answer that holds in the discrete phase learning setting.

One important caveat: the paper explicitly notes that the storage savings are measured at the logical state level, not as directly observed hardware memory. In other words, these numbers reflect the amount of state the algorithm needs to maintain in principle; real-world hardware gains still require further validation. Additionally, an average 5-point accuracy loss is not trivial for accuracy-sensitive applications — Phase-HDC is best suited for edge and embedded scenarios where memory is severely constrained and HDC models are already the architecture of choice.

For researchers working on efficient training and on-device AI, this work offers an interesting perspective: optimizer history is not a given. In specific discrete parameter spaces, a simple gradient threshold may be all you need.