Backpropagation

The mathematical foundation of the backpropagation algorithm is the chain rule in calculus, which decomposes the gradient of a composite function into the product of local gradients at each layer, propagating the error signal layer by layer from the output layer to the input layer.

The backpropagation algorithm was popularized by Rumelhart, Hinton, and Williams in their 1986 paper, with the chain rule as its mathematical core.

Backpropagation is the mathematical application of the chain rule.

The core idea of backpropagation is to use the chain rule to decompose output error layer by layer, allowing the error signal to flow backward from the output layer to every weight parameter.

Backpropagation is essentially a dynamic programming algorithm for efficiently computing gradients, which reduces the computational complexity of gradient calculation by propagating error signals layer by layer from the output layer to the input layer and reusing intermediate computation results.

反向传播算法的本质是通过链式法则在计算图上高效计算标量损失对每个参数的偏导数

Backpropagation is nearly impossible in biological brains because it requires the use of symmetric weights in the forward and backward paths (the weight transport problem), and no known mechanism in real brains supports this.

Backpropagation is essentially a systematic application of the chain rule, which efficiently computes the partial derivative of each parameter with respect to the loss function by propagating error signals layer by layer from the output layer to the input layer.

The backpropagation implemented by VIDRAFT on the NPU is standard reverse-mode gradient computation, not a gradient-free alternative, but the technical mechanism has not been disclosed.

The hardware bottleneck of backpropagation is not compute power but memory access and precision; the activation values of each layer must be retained until gradient backpropagation is complete, and an FP32-precision gradient accumulation buffer is usually required.

Improper weight initialization causes signals to decay or explode layer by layer during forward propagation, and gradients to vanish or diverge during backpropagation.

In backpropagation, gradient values are often extremely small, and under low precision they easily underflow or are drowned out by rounding noise, requiring techniques such as loss scaling, an FP32 master weight copy or stochastic rounding to maintain numerical stability.

Inference-specific NPUs are generally considered capable only of forward computation, because their software stacks lack the computation graphs, gradient synchronization and optimizer states required for training. This is the industry's default conclusion.

Backpropagation requires a global, explicit objective function to define 'what is good', and requires the computational graph of the entire network to be differentiable.

Backpropagation differs significantly from how biological brains learn: no precise error backpropagation pathway has been found in the brain, and neurons cannot obtain global error information.

The key breakthrough of convolutional neural networks (CNNs) is treating the values in convolution kernels as learnable parameters, automatically optimized through backpropagation and large amounts of data, thereby learning feature extractors that are difficult for humans to design by hand.

A neural network consists of multiple layers of nodes, including an input layer, several hidden layers, and an output layer; depth refers to a large number of hidden layers, and it relies on the backpropagation algorithm to adjust weights.

Backpropagation and attention mechanisms are algorithmic breakthroughs that have driven the development of machine learning.

Backpropagation requires the network to have a clear layered structure and differentiable activation functions, which is difficult to map biologically onto the real learning mechanisms of neurons.

Deep networks using the Sigmoid activation function are prone to the vanishing gradient problem, making it difficult for early layers to learn.