#Kimi Delta Attention
11 related articles
·3 min
Kimi Delta Attention Explained: The Evolution from Standard Attention to Linear Attention
A deep dive into Kimi Delta Attention (KDA): tracing the evolution from quadratic Softmax attention through linear attention, Delta rules, and gated decay mechanisms, with insights on associative memory and hardware optimization.
Read more →
·3 min
Kimi Delta Attention Explained: The Evolution from Standard Attention to Linear Attention
Deep dive into Kimi Delta Attention (KDA): from standard Softmax attention's quadratic bottleneck through linear attention, Delta Rule, and gated decay mechanisms — the complete evolution explained.
Read more →
·3 min
Moonshot AI Open-Sources FlashKDA: A Deep Dive into High-Performance CUDA Kernels for KDA Attention
Moonshot AI open-sources FlashKDA, providing high-performance CUDA kernels for Kimi Delta Attention. Learn about its technical principles, performance gains, and value for long-context training and inference acceleration.
Read more →
·3 min
Moonshot AI Open-Sources FlashKDA: A Deep Dive into High-Performance CUDA Kernels for KDA Attention
Moonshot AI open-sources FlashKDA, providing high-performance CUDA kernels for Kimi Delta Attention. Explore its technical principles, performance gains, and value for long-context training and inference.
Read more →
·3 min
GitHub Daily · July 29: Voice AI and On-Device Training Take Flight as Open-Source Alternatives Rise
GitHub Trending July 29: Microsoft's VibeVoice leads voice AI open-source wave, MoonshotAI's FlashKDA CUDA kernel surges 25%, and open-source alternatives rise.
Read more →
·4 min
Kimi K3 Deep Dive: 2.8 Trillion Parameter Open-Source Model Closes the Gap with Closed-Source Frontier for the First Time
Moonshot AI releases Kimi K3 open-weight model with 2.8T parameters and 1M token context. Our deep dive covers coding, 3D dev, agent capabilities, and safety concerns.
Read more →
·3 min
Kimi K3 Released: How a 2.8 Trillion Parameter Open Model Reshapes AI Cost-Effectiveness
Moonshot AI unveils Kimi K3: a 2.8 trillion parameter, 1M context, natively multimodal open model. With KDA architecture and ultra-low cost, it rivals GPT-5.6 and Fable 5, redefining AI cost-effectiveness.
Read more →
·6 min
Hands-On with Qwen 3.8 Max: How an Open-Source Model Closes In on Top-Tier Closed Models
Qwen 3.8 Max has 2.4 trillion parameters and will be open-sourced. In KingBench testing it scored 81.25%, ranking second, beating Claude Opus 4.8 and trailing only Fable 5. A deep dive into its performance across 8 tests.
Read more →
·4 min
Cursor Composer2 Training Revealed: A Complete Guide to Distributed Reinforcement Learning Engineering
Deep dive into how Cursor trained Composer2: two-stage architecture, global distributed clusters, MOE numerical alignment, simulation anti-cheating, and more.
Read more →
Research·2 min
Technical Breakdown of Cursor Composer 2's Distributed RL Training
Deep dive into how Cursor trained Composer 2 via distributed RL, covering async pipelines, MoE numerical alignment, global weight sync, and more.
Read more →
Deep Dives·1 min
Qwen3.5 Deep Dive: Hybrid Attention Architecture Achieves 19x Speedup for Long Context
Deep analysis of Alibaba's open-source Qwen3.5 hybrid attention architecture, how Gated Delta Net achieves 19x speedup at 256K context, and multimodal results surpassing Gemini 3 Pro and GPT-5.2.
Read more →