Polynomial Approximation Accelerates LLM Transcendental Functions: Up to 8% Speedup on GB200

Why Transcendental Functions Are Becoming a New GPU Bottleneck

GPU architectural evolution isn't uniform. Matrix compute units, Special Function Units (SFUs), and memory pipelines scale at different rates — meaning that as hardware generations advance, kernel performance bottlenecks keep shifting. In other words, an operation that wasn't a problem on the previous GPU generation can become a drag on the next one.

A new arXiv paper (arXiv:2610.00049v1) zeroes in on exactly this phenomenon. The researchers note that FlashAttention-4 has already exposed this kind of imbalance within attention mechanisms on NVIDIA's Blackwell architecture. Given that the problem exists in attention, the authors push further: could other operations that rely on SFUs — like sigmoid, tanh, and SiLU, the activation functions ubiquitous in large language models — also be accelerated through smarter computation?

Replacing Native Transcendental Functions with Short Polynomials

The core idea is straightforward: approximate these transcendental functions using short polynomial programs instead of calling the GPU's native special function implementations. Polynomial evaluation can be done entirely with Fused Multiply-Add (FMA) instructions, sidestepping the SFU bottleneck altogether.

The polynomial programs the researchers designed fuse three key techniques: analytic symmetry (exploiting the function's mathematical symmetry to reduce computation), target-format rounding (rounding directly toward BF16/FP16 output formats to avoid precision waste), and packed arithmetic (batch-processing within consumer kernels). These programs are embedded directly into the kernels that use them rather than called as standalone functions, maximizing throughput gains.

On the implementation side, the authors tested both degree-3 and degree-4 BF16 polynomial variants, trading off accuracy against speed.

Transcendental functions are functions that cannot be expressed through a finite sequence of algebraic operations — exponential, logarithmic, and trigonometric functions all fall into this category. Activation functions like sigmoid, tanh, and SiLU fundamentally depend on such operations. On GPUs, the Special Function Unit (SFU) is the dedicated hardware circuit for handling these computations, typically using table lookups plus interpolation to deliver approximate results in just a few clock cycles. The number of SFUs on a GPU chip is far smaller than the number of standard CUDA cores or Tensor Cores, so when many threads simultaneously request transcendental function evaluations, the SFU easily becomes a throughput bottleneck.

Fused Multiply-Add (FMA) instructions are a different story: they perform a × b + c in a single instruction and are the most abundant, highest-throughput arithmetic operations on GPUs — the inner products in matrix multiplication also rely on them. Polynomial evaluation (e.g., Horner's method: ((a₃x + a₂)x + a₁)x + a₀) can be fully unrolled into a sequence of FMA instructions, shifting the load that was pressing on the SFU to the abundantly available FMA units. This is the core logic of the paper's approach.

Isolated Testing: The Raw Speedup of FMA Programs

Before integrating the approach into real models, the research team ran an isolated IEEE binary16 (FP16) performance sweep comparing native PyTorch evaluation against packed FMA programs. The tests covered two working-set scenarios: small datasets resident in the L2 cache, and large datasets resident in High Bandwidth Memory (HBM).

The results showed clear tiered differences:

• In L2-resident scenarios, the polynomial path delivered 1.19–2.19× speedup;
• In HBM-resident scenarios, the speedup range was 1.00–1.70×.

This difference is intuitive — when data is bottlenecked by memory bandwidth (the HBM scenario), the gains from compute optimization are partially offset by the memory bottleneck. In compute-intensive scenarios where data is already in the L2 cache, the benefit of reducing SFU calls is fully realized.

GB200 in Practice: End-to-End Gains Across Four Integrated Tasks

Isolated testing proves the potential; the real test lies in the full training pipeline. The researchers selected four representative integrated tasks on the NVIDIA GB200 platform, replacing native implementations with polynomial programs:

Training Throughput Improvements

• Dense SiLU: full training step throughput improvement of 2.7%;
• tanh-softcapped attention: improvement of 2.9%;
• Routed-expert SwiGLU: improvement of 8.0%, the highest gain among the four tasks;
• Sigmoid attention: attention forward pass improvement of 7.4%, but since this operation represents a limited fraction of the overall GPU step, the full GPU step improvement is only 0.3%.

This data conveys an important signal: the practical value of any speedup depends on how much of the overall computation the target operation accounts for. The local speedup on sigmoid attention is quite significant (7.4%), but because it isn't the dominant operation in the end-to-end pipeline, the global gain is diluted to near-negligible levels. The routed-expert SwiGLU achieves an 8% overall improvement precisely because the gated activation computations carry enough weight in Mixture-of-Experts (MoE) architectures.

SwiGLU (Swish-Gated Linear Unit) is a standard component in the feed-forward layers of today's leading large language models (such as LLaMA and Gemma). Its formula is SwiGLU(x, W, V) = Swish(xW) ⊙ (xV), where Swish(x) = x · sigmoid(x), also commonly written as SiLU. In Mixture-of-Experts (MoE) architectures, each token is routed to a small number of "expert" sub-networks, and each expert internally contains these gated activations. Because MoE models heavily use experts to pursue parameter efficiency, the number of gated activation calls grows linearly with the number of experts — making SwiGLU/SiLU computation far more prominent than in dense models. This explains why the "routed-expert SwiGLU" task achieves the highest end-to-end improvement of 8%: the SFU bottleneck is more pronounced in MoE scenarios, so the optimization gains are amplified accordingly.

Accuracy Cost: Does Model Behavior Suffer?

The other side of any speedup is always accuracy. Does replacing high-precision native functions with polynomial approximations hurt model quality? The researchers answer this through two types of experiments: open-weight ablations on identical checkpoints, and paired pretraining comparisons for each task.

At a training scale of nearly 100 billion tokens — a common benchmark — the final smoothed training loss difference (polynomial minus native) falls between -0.107 and +0.079. Note that this range includes both positive and negative values: in some tasks, the polynomial version actually achieves slightly lower loss than the native implementation. Overall, the differences stay within a quite narrow band.

This means that for most practical scenarios, the accuracy perturbation introduced by polynomial approximation is manageable and doesn't systematically degrade model performance. The conventional concern about "trading accuracy for speed" turns out to be less valid for this specific approach.

The accuracy of polynomial approximation is jointly determined by the approximation degree and the coefficient optimization method. The degree-3 and degree-4 polynomial coefficients in the paper are not chosen arbitrarily — they are derived by minimizing the maximum error in the target format (BF16/FP16) via minimax approximation, calibrated directly for rounding behavior after quantization to low precision. This differs from traditional Taylor or Chebyshev approximations, which target continuous real-number error; here, the discrete low-precision grid points are incorporated into the optimization to minimize error over values actually representable in BF16/FP16. One reason the training loss difference falls in such a narrow range (-0.107 to +0.079) is precisely this fine-grained, target-format-oriented design — the polynomials behave nearly identically to native SFU implementations at actual inference precision, rather than only performing well in higher-precision space.

Implications for LLM Engineering Practice

The significance of this work goes beyond a few percentage points. It offers a general methodology for dealing with GPU architectural imbalances: when one category of hardware unit (here, the SFU) becomes a bottleneck, offload the work to more abundant units (FMA/matrix pipelines), using mathematical forms — like polynomials — that execute efficiently on FMA.

As MoE architectures grow more popular and attention variants continue to multiply, the computational share of activation functions and gating operations will only increase. In this trend, kernel-level optimization targeting special function units becomes an easily overlooked but high-return front in the competition for training efficiency. An 8% end-to-end training speedup, for large models running thousands of GPUs over weeks-long training runs, translates to real, tangible cost savings.

That said, a sober perspective is warranted: these gains are highly dependent on the specific operation's share of the overall pipeline and on the hardware generation in question. The paper itself emphasizes that bottlenecks shift with hardware evolution, so these optimizations need to be re-evaluated for each new GPU generation — there's no set-it-and-forget-it solution.