#rotary position embedding

62 related articles

Aug 15, 2026·4 min

Muse Glimmer Local Testing: Meta's Open-Source 30B Multimodal Model Runs on a Single GPU

Meta releases Muse Glimmer, a 30B open-source multimodal model running on a single 24GB GPU. Tested at 233 tokens/sec with speculative decoding on RTX 5090, Apache 2.0 licensed with GGUF support.

Aug 3, 2026·3 min

Training a 1.3B Parameter LLM from Scratch: Complete Workflow and Core Challenges

A deep dive into the complete workflow of training a 1.3B parameter LLM from scratch, covering Transformer architecture design, data preparation, and distributed training optimization.

Aug 3, 2026·4 min

SELENE Open-Source Project: An Interactive Notebook Library for Systematically Learning AI and Machine Learning from Scratch

SELENE is an open-source AI learning resource built on Jupyter Notebooks, systematically covering ML, deep learning, Transformers, and LLMs with interactive code and math derivations for beginners.

Aug 1, 2026·2 min

AI Frontier Technology Accelerating: Decoding Industry Signals and Future Trends

Decoding signals like "frontiermogging" to analyze upcoming AI frontier model leaps, Agent automation deployment, and developer ecosystem expansion trends.

Jul 29, 2026·3 min

TokenTown: A Visual Approach to Understanding How LLMs Predict the Next Token

TokenTown is an open-source visualization project that intuitively presents the internal token prediction process of LLMs using a town metaphor. Learn its design philosophy and educational value.

Jul 28, 2026·3 min

Kimi K3 Reaches 2.5T Parameters, MiniMax M3 Pro Brewing Simultaneously: China's LLM Arms Race Escalates

Kimi K3 adopts new architecture with 2.5T parameters and 1M token context. MiniMax M3 Pro targets 2.7-3T. OpenAI surpasses 7M users, HY-OCR 1.5 achieves SOTA, Amap launches world model.

Jul 24, 2026·7 min

Getting Started with Claude Code: Five Dimensions That Show Why It Outperforms Ordinary AI

Claude Code isn't just a chat AI—it can directly read projects, modify code, and run commands. This article compares Claude Code with ordinary AI across five dimensions: interaction, context, execution, memory, and tool calling.

Jul 24, 2026·7 min

Getting Started with Claude Code: Five Dimensions to Understand Why It Outperforms Regular AI

Claude Code isn't just a chat AI—it can directly read projects, modify code, and run commands. This article compares Claude Code with regular AI across five dimensions to help you decide if it's worth trying.

Jul 23, 2026·7 min

Context Engineering: A Systematic Approach to Stop AI Agents from 'Talking Nonsense'

Context engineering is the core methodology for building efficient AI Agents, covering query enhancement, RAG retrieval, prompt design, memory management, and tool invocation. Master Write, Select, Compress, and Isolate to solve LLM hallucination at its root.

Jul 23, 2026·3 min

Alibaba's 2.4 Trillion Parameter Qwen Model: Separating Technical Substance from Marketing Hype

Alibaba's Qwen releases a 2.4T parameter MoE model claiming to be 'second only to Gemini 2.5.' We break down what's real—and what's just hype.

Jul 23, 2026·7 min

Qwen3 235B A22B in Practice: A 2.4-Trillion-Parameter Open Model Surpasses Claude Opus

Hands-on with Alibaba Tongyi Qianwen's strongest Qwen3: a 2.4-trillion-parameter open weight model scoring 81.25% on KingBench, ranking second and beating Claude Opus 4.8 with perfect scores in game dev, math, and agent tasks.

Jul 22, 2026·3 min

Qwen's Six-Generation Evolution: A Complete Breakdown from Architecture Innovations to the Reasoning Race

From Qwen1 to Qwen3-2507: a complete breakdown of six generations of evolution over two years — GQA, MoE, GRPO, dynamic thinking, and the core shift from architecture to reasoning.

Jul 21, 2026·2 min

Zhipu GLM New Model Coming Soon: Decoding the Founder's Teaser

Zhipu AI's Z.ai founder teases a new GLM model just one month after GLM 5.2. We decode the announcement, explore likely upgrade directions, and analyze China's LLM iteration pace.

Jul 20, 2026·3 min

Bernini Tile-Based Generation: An Engineered Approach to Video Super-Resolution in ComfyUI

Bernini is a ComfyUI video super-resolution node package using Tile Split/Select/Merge to solve seams, drift, and VRAM overflow. Benchmarked at 325s for 39 frames at 1920×1080.

Jul 20, 2026·4 min

Understanding Transformer From the Ground Up: How Language Models Predict the Next Word

From Tokenization and Embedding to the Attention mechanism, this article systematically breaks down how Transformer works — and how ChatGPT turns input text into next-token probabilities.

Jul 20, 2026·3 min

Building a Transformer from Scratch: A PyTorch Hands-On Guide for Self-Taught Developers

A self-taught developer with 2 years of Python experience implements Transformer from scratch using plain PyTorch, following the original 'Attention Is All You Need' paper with a two-phase approach: inference first, then full training.

Jul 18, 2026·3 min

AI Coding Agents Explosion: CodeWell Goes Open Source, Kimi K3 Launches, Claude Gets Browser Access

CodeWell open-sources a multi-model terminal coding agent, Kimi K3 launches with ultra-long context, MiniMax Code 2.0 rebuilds its Agent architecture, and Claude gets browser access. AI is accelerating from content generation to task execution.

Jul 15, 2026·2 min

Long Conversations Are Draining Your AI Credits: A Deep Dive into Token Billing

Why do long AI conversations drain your credits so fast? This article explains LLM stateless architecture, Token billing mechanics, and practical tips like sliding windows and summary compression to cut API costs.

Jul 13, 2026·3 min

Training a 100M-Parameter Model for Under $50: A Complete Record of Building SmoLLM from Scratch

An ML engineer trained SmoLLM, a 109M-parameter LLaMA-style model from scratch for under $50. Full breakdown of architecture, training pitfalls, instruction tuning, and real-world performance.

Jul 13, 2026·2 min

15-Year-Old Trains 200M MoE Language Model from Scratch: Full Architecture & Hands-On Breakdown

A 15-year-old trained Tiny-MoE, a 200M-parameter MoE language model from scratch using free Kaggle GPUs, featuring MLA attention, RoPE+YaRN, and native PyTorch.