[KongchangAI]
· 1 min read· 839 words

New Breakthrough in Spiking Transformer Quantization: DAQ Achieves 4-bit Weights and Membrane Variables

New Breakthrough in Spiking Transformer Quantization: DAQ Achieves 4-bit Weights and Membrane Variables

DAQ calibrates quantizers to align neuron firing decisions, enabling efficient 4-bit compression and energy reduction for Spiking Transformers.

Spiking Transformers combine attention mechanisms with sparse spike communication, offering great potential for edge computing, but the storage and computation overhead of weights and membrane variables limits deployment. Conventional post-training quantization (PTQ) minimizes numerical error but overlooks the sensitivity of spiking neurons to quantization errors, especially near decision boundaries where firing patterns can be disrupted. The proposed Decision-Aligned Post-Training Quantization (DAQ) shifts the calibration objective from numerical error to firing decision alignment, analyzing error propagation at decision boundaries and over time steps to construct an objective function that keeps quantized membrane variables within the reference firing region. Experiments show DAQ is effective with 4-bit weights and membrane variables, achieving 58.8% AUC and 90.9% PR on FE108 with SDTrack while reducing energy by 66.2%, applicable to image classification, semantic segmentation, and event tracking.

Spiking Neural Networks (SNNs) have attracted significant attention in edge computing scenarios due to their sparse spike communication and low power consumption. Spiking Transformers further combine the representation learning capability of attention mechanisms, demonstrating strong potential. However, their weights and membrane variables still incur considerable storage and computation overhead. This article introduces the Decision-Aligned Post-Training Quantization (DAQ) method, which calibrates quantizers to align neuron firing decisions, achieving efficient compression and energy reduction with 4-bit weights and membrane variables.

DAQ method overview

Quantization Challenges of Spiking Transformers

Spiking Transformers combine attention-based representation learning with sparse spike communication, but the storage and computation overhead of weights and membrane variables remains significant. Post-training quantization (PTQ) is a practical approach to reduce overhead, but preserving neuron firing behavior under weight and membrane variable quantization remains challenging.

Conventional quantization methods typically focus on minimizing the mean squared error of weights or activations, but the firing decisions of spiking neurons are highly sensitive to quantization errors. Especially near the decision boundary, small quantization errors can cause neurons to erroneously fire or suppress spikes, thereby affecting information transmission across the entire network.

Membrane variables are core state quantities in spiking neurons (such as the LIF model), used to accumulate input current and determine whether to fire a spike. When the membrane potential exceeds the threshold, the neuron fires a spike and resets; otherwise, it remains at rest. Therefore, membrane variables carry the temporal integration history of the neuron, and their quantization errors directly affect firing timing and frequency. In Spiking Transformers, membrane variables are typically as important as weights, because query, key, and value computations in attention mechanisms all rely on spike trains, and small deviations in membrane variables propagate and accumulate over time steps, causing firing patterns to deviate.

Post-training quantization (PTQ) is a technique that converts floating-point weights and activations to low-bit integers after model training is complete, without requiring retraining or fine-tuning. Its typical pipeline includes: collecting calibration data, statistics of activation ranges, determining quantization parameters (scale factors and zero points), and then applying quantization. The advantage of PTQ lies in deployment convenience and low computational overhead, but the difficulty is that at low bit-widths, quantization errors can significantly affect model outputs, especially for nonlinear dynamical systems such as spiking neurons.

Core Idea of DAQ: Aligning Firing Decisions

The key innovation of DAQ is shifting the calibration objective of the quantizer from "minimizing numerical error" to "aligning firing decisions." Specifically, DAQ calibrates the quantizer so that the firing decisions of quantized neurons are as consistent as possible with those of the floating-point reference model.

The method analyzes how quantizer errors affect firing at the decision boundary, and how these errors propagate over time steps in recurrent neurons. Based on this analysis, DAQ constructs an objective function that encourages quantized membrane variables to remain within the reference firing region along the quantized model's own state trajectory. This alignment strategy effectively reduces the disruption of quantization to spike firing patterns.

The decision boundary in spiking neurons refers to the state where the membrane potential is exactly equal to the firing threshold. When the membrane potential is close to the threshold, a small quantization perturbation can cause the neuron to erroneously fire or suppress a spike. DAQ analyzes the effect of quantization errors at the decision boundary and constructs an objective function that keeps quantized membrane variables fluctuating within the reference firing region, thereby reducing false firings. This idea draws on boundary alignment strategies in classification tasks, but is specially designed for spike temporal dynamics.

Experimental Validation: Effectiveness on Multiple Tasks

The researchers conducted experiments on multiple Spiking Transformer architectures, covering image classification, semantic segmentation, and event-based tracking tasks. Experimental results show that DAQ remains effective with 4-bit weights and membrane variables.

On the FE108 dataset, DAQ combined with the SDTrack model achieved 58.8% AUC and 90.9% PR while reducing energy consumption by 66.2%. This result demonstrates DAQ's potential to significantly reduce energy consumption while maintaining task performance.

Significance and Outlook

DAQ provides a new approach for the practical deployment of Spiking Transformers. Through the decision-aligned PTQ method, Spiking Transformers can substantially compress models and reduce energy consumption while maintaining consistency in firing behavior. This is particularly important for resource-constrained edge devices and is expected to promote the application of spiking neural networks in real-time vision tasks.

In the future, the idea of DAQ may be extended to even lower bit-widths (such as 2-bit) or more complex spiking architectures, further exploring the interaction between quantization and spike dynamics.

Key Points

  • DAQ is a post-training quantization method for Spiking Transformers that calibrates quantizers to align neuron firing decisions.
  • The method is effective with 4-bit weights and membrane variables, achieving 58.8% AUC and 90.9% PR on the FE108 dataset with 66.2% energy reduction.
  • DAQ is applicable to various tasks including image classification, semantic segmentation, and event tracking.
  • The method analyzes the propagation of quantization errors at decision boundaries and over time steps, providing insights for quantization-aware spiking network design.
#Spiking Transformer#Post-Training Quantization#DAQ#Spiking Neural Networks#4-bit Quantization#Membrane Variables#Energy Efficiency
Share:

Related articles