#tokens per second

1405 related articles

May 22, 2026·2 min

Mercury 2: How Diffusion Architecture Achieves 1000 Tokens Per Second Inference Speed

Inception Labs releases Mercury 2, a diffusion-based language model achieving 1000+ tokens/sec with strong reasoning, challenging autoregressive architectures.

How Fast Is 10 Tokens Per Second from an LLM? An Open-Source Tool for Visualizing Token Generation Speed

Deep Dives

May 20, 2026·2 min

How Fast Is 10 Tokens Per Second from an LLM? An Open-Source Tool for Visualizing Token Generation Speed

An open-source tool lets you visualize LLM token generation speeds from 5-800 TPS, helping developers make rational choices about model selection and inference optimization.

Jul 30, 2026·3 min

LLM Cost Optimization in Practice: Balancing Savings with User Experience

Practical LLM cost optimization strategies covering Prompt trimming, context compression, and multi-model routing to cut Token costs while maintaining output quality at scale.

Jul 30, 2026·3 min

LLM Cost Optimization in Practice: Balancing Savings and User Experience

Practical strategies for LLM cost optimization: prompt trimming, context compression, multi-model routing, and more to cut token costs while maintaining output quality at scale.

Jul 30, 2026·3 min

Open Weight Models: The Strategic Game Between AI Democratization and American Competitiveness

Explore how open weight models achieve both global AI democratization and maintain U.S. competitiveness. Learn the differences between open weight, open source, and closed models, and their strategic impact.

Jul 30, 2026·3 min

Open Weight Models: The Strategic Game of AI Accessibility and American Competitiveness

Explore how open weight models simultaneously enable global AI accessibility and maintain U.S. competitiveness. Learn the differences between open weight, open source, and closed source models.

Jul 30, 2026·3 min

Local Merge Queues: Solving Code Conflicts for Parallel AI Programming Agents

Deep dive into how local merge queues solve code conflict challenges when multiple AI programming agents work in parallel, covering merge queue principles and multi-agent development trends.

Jul 30, 2026·3 min

Local Merge Queues: Solving Code Conflicts from Parallel AI Coding Agents

Deep dive into how local merge queues solve code conflict challenges when multiple AI coding agents work in parallel, covering merge queue principles and multi-agent development trends.

Jul 30, 2026·3 min

Building Your Own Email Archiving Tool: Breaking Free from Single-Provider Dependency and Reclaiming Data Sovereignty

Learn how to build your own email archiving tool using Maildir format and metadata separation to break free from Gmail/Outlook dependency and reclaim data sovereignty.

Jul 30, 2026·3 min

Is LLM Inference Really That Expensive? Breaking Down the Cost Reduction Logic of GPU Compute and Inference Frameworks

Analyzing real LLM inference costs: from B200 GPU compute gains, vLLM framework optimization to MTP multi-token prediction, explaining why serving costs are widely overestimated.

Jul 30, 2026·2 min

The Case Against Anthropic's Decline Is Weak: The Truth Behind 7300% ARR Growth

Anthropic faces decline narratives yet achieves 7300% ARR growth. This article analyzes the market logic behind this explosive growth and why data should trump narratives when evaluating AI companies.

Jul 30, 2026·2 min

Behind the Open-Source Model Frenzy: Who Will Provide Cheap Inference Services?

Open-source LLM weights don't equal low-cost access for developers. This article analyzes the inference service gap in open-source AI and how providers like Together AI and Groq are addressing it.

Jul 30, 2026·2 min

Behind the Open-Source Model Frenzy: Who Will Provide Cheap Inference Services?

Open-source LLM weights don't mean developers can use them cheaply. This article examines the inference service gap in open-source AI and how providers like Together AI and Groq are addressing it.

Jul 29, 2026·4 min

Infrastructure Architecture for Agent Applications: Four Core Patterns Explained

A deep dive into infrastructure architecture patterns for production-grade Agent applications, covering state persistence, sandbox isolation, LLM observability, and cost control.

Jul 29, 2026·4 min

Infrastructure Architecture for Agent Applications: Four Core Patterns Explained

Deep dive into infrastructure architecture patterns for production-grade Agent applications, covering state persistence, sandbox isolation, LLM observability, and cost control.

Jul 29, 2026·3 min

Self-Hosting Kimi K3: Is Spending 20% More on Hardware Worth a 20% Improvement in Task Performance?

Analysis of whether spending 20% more on hardware for self-hosting Kimi K3 to gain 20% task performance improvement is worthwhile, covering inference precision, VRAM optimization, and tiered deployment.

Jul 29, 2026·2 min

Real User Feedback: Claude Opus 5 vs Gemini 3.1 Pro — Does Speed Determine the Experience?

A Reddit user's hands-on comparison of Claude Opus 5 vs Gemini 3.1 Pro reveals that response speed and interaction fluidity may matter more than raw intelligence in choosing an LLM.

Jul 29, 2026·3 min

Local LLM Agent Lagging? Lowering num_gpu Is the Key

Local LLM crashing in Agent frameworks? The issue may be num_gpu set too high. Learn what num_gpu really controls (GPU layer offloading, not GPU count) and how to tune it for stable Agent performance.

Jul 29, 2026·3 min

Ollama Template Silently Overridden After GGUF Import: A Troubleshooting Guide for Custom TEMPLATE Being Ignored

Analysis of Ollama silently overriding custom TEMPLATE with GGUF-embedded templates after import. Learn to verify active templates via /api/show to prevent evaluation failures.

Jul 29, 2026·2 min

AI-Driven Persistent RPG Engine: A Full-Stack Practice Guide with React + Supabase

Deep dive into an AI persistent RPG game engine built with React SPA and Supabase, exploring how LLMs combine with modern web stacks for cross-session memory, dynamic narrative, and game state management.