#Token compression
705 related articles
·3 min
Maple-Preview: 20B Ternary MoE Model Achieves 120 tok/s On-Device Inference on iPhone
Maple-Preview achieves 120 tok/s inference of a 20B ternary MoE model on iPhone. We analyze ternary quantization, MoE sparse activation, and on-device inference challenges.
Read more →
·2 min
Creating Characters with Fake Words: Unveiling a Midjourney Semantic Imagination Experiment
Explore how random fake words drive Midjourney to generate unique characters. Learn how AI interprets nonexistent vocabulary and how this method helps creators break prompt dependency and spark original character design inspiration.
Read more →
·3 min
Quantprobe: An Open-Source Tool for Running 30B LLMs on 6GB GPUs
Quantprobe is an open-source memory optimization framework that enables 30B LLMs to run at 22 tokens/s on 6GB GPUs through per-layer quantization and intelligent CPU/GPU splitting.
Read more →
·3 min
Harness Engineering: The Key Technique for Unlocking Hidden AI Agent Capabilities
Explore how harness engineering dramatically improves AI Agent performance. From the Codex case study, learn how tool orchestration, context management, and execution environments become the core competitive battleground.
Read more →
·3 min
LangChain Managed DeepAgents: Hosted Agent Infrastructure So You Can Focus on Core Logic
LangChain launches Managed DeepAgents public beta, hosting evals, memory, OAuth, Slack integration, and sandbox infrastructure so developers can focus on Agent core logic.
Read more →
·4 min
SAM 3 Auto-Labeling in Practice: Six Lessons Where Preprocessing Matters More Than the Model
Practical lessons from building a SAM 3 auto-labeling pipeline: vision embedding reuse, resolution handling, prompt engineering, threshold sweeping, and more.
Read more →
·4 min
Running an 80B Large Model on Mac with Only 4.3GB of Memory: Extreme Compression Techniques Explained
Deep dive into how an 80B-parameter LLM runs on Mac with only 4.3GB memory, covering ultra-low-bit quantization, sparsity, memory mapping, and implications for privacy and edge AI.
Read more →
·3 min
Token Economics: The New Unit of Value Measurement in the AI Era
Deep dive into how Tokens evolved from a technical concept in LLMs to the core unit of measurement in the AI economy. Exploring Token consumption explosion, cost optimization, and Token economics.
Read more →
·4 min
SELENE Open-Source Project: An Interactive Notebook Library for Systematically Learning AI and Machine Learning from Scratch
SELENE is an open-source AI learning resource built on Jupyter Notebooks, systematically covering ML, deep learning, Transformers, and LLMs with interactive code and math derivations for beginners.
Read more →
·3 min
Running the 1.56TB Kimi K3 Model on 8GB RAM with Pure C
A developer built a pure C99 inference engine that runs the 1.56TB Kimi K3 model on 8GB RAM using MoE sparsity and NVMe on-demand loading—no GPU, 176KB binary.
Read more →
·3 min
Persistent State Machines: How INT4 Memory Cells Reshape LLM Attention Mechanisms
Exploring how persistent state machines with INT4-quantized memory cells reshape LLM attention, breaking KV Cache memory bottlenecks for long-context inference on edge devices and high-concurrency scenarios.
Read more →
·3 min
Why Local Agents Crash: A Real-World Report from 4,265 Actual Sessions
A developer spent a month testing 4,265 Claude Code/Codex sessions, revealing why local Agents crash on consumer hardware: tool lists consume 41% of cache, q4_0 quantization traps, and eviction strategy ceilings of only 11.88%.
Read more →
·3 min
Dual RTX 3060 Running DeepSeek V4 Flash: IQ2_M Quantized Inference Benchmarked at 3.5 tok/s
Benchmarking DeepSeek V4 Flash on dual RTX 3060 GPUs with 96GB RAM at IQ2_M quantization achieving 3.5 tokens/sec. Covers hardware choices, 2-bit quantization techniques, and local LLM deployment optimization.
Read more →
·3 min
GPT-5.6 Dominates Math but Stumbles on Puzzle Games? Two API Settings Unlock 3x Performance
GPT-5.6 Sol conquers frontier math but struggles on ARC-AGI-3 puzzles. The fix? Not a smarter model, but two API settings that tripled scores and cut token costs 6x.
Read more →
·3 min
Scanning 7.6PB of HuggingFace Data: A Hidden Key Crisis in AI Training Sets
A security audit of 7.6PB of HuggingFace training data uncovered massive API key and credential leaks. Analysis of risks, scanning challenges, and data supply chain security governance.
Read more →
·2 min
Running Local LLMs on a 24GB Mac Mini: Model Selection & Speed Optimization Guide
24GB Mac Mini too slow for local LLMs? Learn why 14B models struggle, get 3B-8B model recommendations for Home Assistant, and discover Ollama speed optimization tips.
Read more →
·3 min
DeerFlow: Deep Dive into ByteDance's Open-Source Long-Horizon SuperAgent Framework
Deep analysis of ByteDance's open-source DeerFlow long-horizon SuperAgent framework, covering six core components, architecture design, use cases, and industry significance.
Read more →
·2 min
DeepSeek V4 Flash Free Access: Complete Breakdown of the 0731 Version on InferX
InferX offers free access to DeepSeek V4 Flash (0731 version) with zero data retention and OpenAI-compatible API. Full breakdown of features, pricing, and developer value.
Read more →
·1 min
The Third Dimension of Pretraining: A New Paradigm Claiming 6.2x Sample Efficiency and 250x Inference Speedup
Harvard and UIUC propose a third axis of pretraining, claiming 6.2x sample efficiency and 250x inference speedup. Deep analysis of this new paradigm's implications and key caveats.
Read more →
·1 min
Predictive KV Replication: Breaking Through the Bursty Traffic Bottleneck in LLM Inference
Deep dive into predictive speculative KV replication: how anticipating bursty traffic and pre-replicating KV Cache can reduce LLM inference tail latency.
Read more →