New Breakthrough in Neural Operators: Learnable Fractional-Order Derivatives Make PDE Solving More Accurate

Neural operators have long faced a fundamental problem when solving partial differential equations (PDEs): the network needs to implicitly learn differentiation operations, which increases the learning burden. A recent study proposes a counterintuitive solution—using fractional-order derivative features, with the optimal order strictly smaller than the order of the PDE itself.

The Implicit Differentiation Dilemma of Neural Operators

Neural operators learn mappings between function spaces and can provide fast surrogate solutions for partial differential equations. Despite the proliferation of architectural designs, these methods typically share the same input representation: raw coordinate-value pairs $(x, u(x))$.

The problem is that the solution operator of a PDE inherently involves differentiation operations. When the network only receives coordinates and function values as input, it must implicitly learn differentiation through multiple layers of nonlinear transformations. This undoubtedly increases the network's burden, especially in high-dimensional or high-order PDE problems.

The research team analyzed mild solutions through Picard iteration and found that derivatives of the input function appear explicitly in the solution operator. This means that if derivative features are directly provided to the network, it should be able to reduce its implicit differentiation burden.

Neural Operator is a class of deep learning models that learn mappings between infinite-dimensional function spaces. Their goal is to approximate the solution operator of a PDE $\mathcal{G}: a \mapsto u$, i.e., mapping from coefficient functions or initial/boundary conditions to solution functions. Unlike traditional neural networks that learn finite-dimensional vector mappings, neural operators learn at the function space level, and therefore possess resolution independence—after training, they can generalize to different discrete grids. Representative architectures include FNO (Fourier Neural Operator, which performs global convolution in the frequency domain), DeepONet (Deep Operator Network, which employs a branch-trunk dual network structure), etc. Compared to traditional numerical solvers that require recomputation for each solve, neural operators can be trained offline once and perform fast online inference, providing orders-of-magnitude speedup for PDE surrogate modeling.

Picard iteration is a classical tool for analyzing mild solutions of ordinary and partial differential equations. Its basic idea is to rewrite the differential equation $\dot{u} = f(u)$ as the integral equation $u(t) = u_0 + \int_0^t f(u(s))ds$, then construct a sequence of solutions through iteration $u_{n+1} = u_0 + \int_0^t f(u_n(s))ds$. In the PDE context, in the mild solution form given by Duhamel's principle, the nonlinear term acts through an integral operator, and the integrand before integration often contains derivatives of the input function. It is precisely through this analysis that the research team discovered: in the solution operator's dependence on the input function, derivative terms appear explicitly, thus providing theoretical justification for supplying derivative features as network input.

Theoretical Proof and Unexpected Discovery

The research team theoretically proved this intuition. After providing derivative features, the approximation bound improves from $O(W^{-(s-m)/d})$ to $O(W^{-s/d})$. Here $W$ is the number of parameters, $s$ is the input regularity, $m$ is the PDE order, and $d$ is the spatial dimension. This result confirms that derivative features can indeed reduce learning difficulty.

However, the core finding is unexpected: the optimal derivative order $\beta^*$ is strictly smaller than the PDE order $m$. In other words, providing higher-order derivatives is not necessarily better.

This gap arises from the bias-variance tradeoff in spectral space, for which the research team provides a closed-form characterization. Learning $\beta$ from data actually achieves automatic spectral regularization—the network adaptively selects the derivative order most favorable for generalization.

Approximation bound is an important theoretical metric for measuring the expressive power of neural networks. In the text, $O(W^{-(s-m)/d})$ means: when the network has $W$ parameters, the input function belongs to a Sobolev space with regularity $s$, the PDE order is $m$, and the spatial dimension is $d$, the approximation error decays at a rate as the number of parameters increases. The exponent $(s-m)/d$ in this rate reflects the intrinsic difficulty of the problem: the higher the PDE order $m$ and the larger the dimension $d$, the smaller the exponent and the slower the decay, meaning more parameters are needed to achieve the same accuracy. After providing derivative features, the approximation bound improves to $O(W^{-s/d})$, which is equivalent to eliminating the penalty term introduced by $m$—this theoretically quantifies the value of derivative features.

Bias-variance tradeoff is a core concept in statistical learning: model bias refers to the systematic error when the hypothesis space cannot perfectly fit the true function, while variance refers to the model's sensitivity to perturbations in training data. In spectral space, higher-order derivative features can better capture the high-frequency components of the solution and reduce bias, but at the same time amplify noise and increase variance. Therefore, there exists an optimal derivative order $\beta^$ that minimizes the total error. The closed-form characterization provided by the research team shows that $\beta^$ is strictly smaller than the PDE order $m$, because excessive pursuit of higher-order derivatives leads to variance dominance, which in turn harms generalization.

∂-NO: Architecture-Agnostic Plug-and-Play Enhancement

Based on the above theory, the research team proposed $\partial$-NO (del-NO), a derivative-enhanced neural operator. Its core feature is architecture agnosticism: it can provide learnable fractional-order derivative features for any neural operator backbone network.

Experiments on multiple benchmark problems and different architectures show that $\partial$-NO can consistently improve prediction accuracy. More notably, the learned order reflects a modulated representation of known physical laws under the constraints of noise and limited samples—this means that what the network learns is not merely numerical fitting, but also captures physical structure.

The value of this method lies in its generality. Whether it is FNO, DeepONet, or other neural operator architectures, $\partial$-NO enhancement can be directly integrated without redesigning the network structure.

Fractional-order derivative is a generalization of integer-order derivatives, defined through the Fourier transform as $\mathcal{F}D^\alpha f = (i\xi)^\alpha \mathcal{F}f$, where $\alpha$ can be any real number. Compared with integer-order derivatives, fractional-order derivatives provide a continuously adjustable "differentiation strength," enabling a finer balance between low and high frequencies. In $\partial$-NO, the network learns a continuous order parameter $\beta$, applies fractional-order differentiation to the input function, and then feeds it into the backbone network. This parameterization allows derivative features to retain physical priors (differential structure) while also possessing data-driven flexibility (learnable order), avoiding the discrete limitations of manually selecting integer-order derivatives.

Automatic spectral regularization refers to the network automatically learning the optimal derivative order $\beta^*$ during training. This process is equivalent to applying different degrees of attenuation or enhancement to different frequency components in spectral space. Specifically, fractional-order differentiation corresponds in the frequency domain to multiplication by $(i\xi)^\beta$, whose magnitude is $|\xi|^\beta$. When $\beta$ is small, high-frequency components are relatively preserved; when $\beta$ is large, high frequencies are amplified. The process of learning $\beta$ is actually adjusting the weighting in spectral space, so that while fitting the data, the network suppresses noise-dominated high-frequency components, thereby acting as regularization. Unlike traditional regularization coefficients that require manual tuning, $\beta$ is automatically determined from data and is physically interpretable.

Implications for PDE Surrogate Modeling

This study conveys several important signals. Differential structure priors are worth explicitly encoding into network inputs, rather than relying entirely on implicit learning by the network. Fractional-order derivatives are more flexible than integer-order ones and can better balance bias and variance. Automatic spectral regularization provides a data-driven way to determine the optimal feature order, avoiding the difficulty of manual parameter tuning.

For practitioners engaged in interdisciplinary research between scientific computing and AI, this means that when designing PDE surrogate models, input feature engineering should be re-examined—derivative features may bring greater benefits than more complex architectural designs.

One-line Summary

Research proposes ∂-NO, using learnable fractional-order derivative features to enhance neural operators, with the optimal order unexpectedly smaller than the PDE order.

Paragraph Summary

Neural operators need to implicitly learn differentiation operations when solving partial differential equations, which increases the network burden. A recent study proves both theoretically and experimentally that explicitly providing derivative features to the network can significantly reduce learning difficulty: the approximation bound improves from O(W^{-(s-m)/d}) to O(W^{-s/d}). But the core finding is counterintuitive—the optimal derivative order β* is strictly smaller than the PDE order m, arising from the bias-variance tradeoff in spectral space, and learning β* achieves automatic spectral regularization. Based on this, the research team proposes ∂-NO, an architecture-agnostic plug-and-play enhancement method that can provide learnable fractional-order derivative features for any neural operator backbone, consistently improving accuracy on multiple benchmarks, while the learned order reflects a modulated representation of physical laws under noise and limited samples.