Unverified50% confidenceFactExact time
The FFN feed-forward network module upscales vectors via the up layer (e.g., 4K→8K), passes them through an activation function (such as SwiGLU gated activation), and then reduces them back to the original dimension via the down layer.
1
Sources
50%
Confidence
Long-term
Relevance
9/11/2026
First Seen
Sources
Related Entities
Related Claims
VerifiedQuantization compresses neural network weights from high-precision floating-point numbers (such as FP32 and FP16) to low-bit integers (such as INT8 and INT4), with modern quantization schemes including GPTQ, AWQ, and GGUF's K-quant series61% similarUnverifiedFP8量化将模型权重从FP16压缩为8位浮点表示，在NVIDIA Hopper架构（H100/H200）上可获得原生加速，且相比INT8对精度损失更小60% similarUnverifiedThe MoE architecture replaces the FFN layer of a standard Transformer with N parallel expert sub-networks, and selects the Top-K experts (typically K=2) for computation each time via a gating routing network, achieving sparse activation.58% similarUnverifiedFP8 has two main sub-formats — E4M3 (4-bit exponent + 3-bit mantissa) and E5M2 (5-bit exponent + 2-bit mantissa) — optimized for forward propagation and backward propagation respectively.58% similarUnverified工程实践中建议将路由器层、注意力投影矩阵及门控网络保留在 BF16/FP16 精度，仅对专家 FFN 权重进行激进量化57% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/896505API
curl https://kongchang.com/api/v1/knowledge/claims/896505MCP
get_claim(id=896505)