#token throughput
450 related articles
·3 min
Claude's Invisible Watermark Technology Explained: How AI Text Tracing Works
Anthropic embeds invisible text watermarks in all Claude outputs for machine-readable tracing. Deep dive into the watermark technology, C2PA file provenance standard, and industry impact.
Read more →
·3 min
DeepSeek Plans Significant Price Hikes: Is the Era of Cheap AI Coming to an End?
DeepSeek plans significant API price hikes, signaling the end of ultra-cheap AI. We analyze the drivers, developer impact, and industry shift from price wars to rational pricing.
Read more →
·3 min
Practical Guide to Accelerating LLM Inference with Apple Silicon Virtual Machines
A deep dive into accelerating llama.cpp inference in macOS VMs using Apple Silicon's unified memory architecture, covering Metal backend configuration, memory allocation, and quantization optimization.
Read more →
·3 min
PagedAttention Explained: GPU Memory Optimization and Model Routing Techniques
Deep dive into how PagedAttention borrows OS paging to optimize GPU memory, eliminate KV Cache fragmentation, and how model routing enables intelligent scheduling for multi-model inference services.
Read more →
·3 min
Running Local LLMs on 24GB VRAM? Your Actual Usable Memory Is Less Than You Think
A detailed breakdown of actual usable VRAM when running local LLMs on 24GB GPUs. Covers the three memory buckets — model weights, KV cache, and runtime headroom — with structured planning methods.
Read more →
·2 min
A Go Developer's Guide to Gemini: Complete Model Family Breakdown
A comprehensive Gemini model family guide for Go developers, covering Pro vs Flash selection strategies, multimodal capabilities, official Go SDK integration, and token management practices.
Read more →
·3 min
Self-Hosting LLMs on Azure: A Complete Guide to vLLM + AKS + GPU Deployment
Complete guide to self-hosting LLMs on Azure Kubernetes Service with vLLM, covering GPU scheduling, NVIDIA GPU Operator setup, deployment troubleshooting, and cost control for production workloads.
Read more →
·2 min
Google SDK Code Leak Reveals Gemini 4 Flash: A New Lightweight Model on the Horizon?
Google SDK tokenizer code accidentally reveals a gemini-4-flash-preview identifier, suggesting a new Gemini 4 Flash model is in development. We analyze the leak details, Flash series positioning, and naming possibilities.
Read more →
·3 min
Soup CLI: An Open-Source Tool That Fine-Tunes 8-Billion Parameter LLMs on a 4GB GPU — A Deep Dive
Soup CLI is an open-source CLI tool that uses layer-by-layer streaming to fine-tune 8B parameter LLMs like Llama-3.1-8B on laptop GPUs with just 4GB VRAM.
Read more →
·4 min
Total Parameter Count Is Obsolete: Understanding the Two-Number Era of MoE Models
Mixture-of-Experts (MoE) makes single parameter counts obsolete. Learn the difference between total and active parameters, how MoE decouples knowledge capacity from inference cost, and why this matters for model selection.
Read more →
·2 min
Qwen3 Priced at Just $2/Million Input Tokens — The LLM Price War Intensifies
Alibaba's Qwen3 model priced at $2/million input tokens and $6 output, far below mainstream closed-source LLMs. Analysis of pricing logic, comparison with Claude, and the open vs closed-source debate.
Read more →
·3 min
llm-sketchkit: Solving High-Cardinality and Privacy Challenges in LLM Telemetry with Sketch Algorithms
Deep dive into how the open-source library llm-sketchkit uses HLL++, Bloom filters, MinHash and other probabilistic data structures to solve high-cardinality and privacy challenges in LLM telemetry.
Read more →
·4 min
How Databricks Cut AI Coding Costs by 70%: Strategies and Insights
Databricks cut AI coding tool costs by 70% through intelligent model routing, prompt caching, context optimization, and self-hosted open-source models. Learn actionable strategies for controlling LLM inference costs.
Read more →
·3 min
Software Optimization Makes a Single B200 Rival Dedicated Inference Chips
Analysis of how a single NVIDIA B200 GPU surpasses Groq LPU and approaches Cerebras performance through software optimization alone, covering CUDA kernels, TensorRT-LLM, and FP8 quantization.
Read more →
·3 min
Qwen 3.8 Max Tops Agentic Benchmark: Is the Tipping Point for Local Deployment Approaching?
Qwen 3.8 Max tops the Artificial Analysis Agentic Index ahead of Opus 5. Reddit debates the gap between benchmark scores and real-world agent performance, and what it means for local deployment.
Read more →
·4 min
Quantization Illustrated: A Deep Dive into Core Compression Techniques for Large Language Models
A deep dive into LLM quantization techniques covering symmetric/asymmetric quantization, PTQ, QAT, GPTQ, AWQ, and outlier solutions for efficient model deployment.
Read more →
·3 min
Deep Dive into vLLM Architecture: How PagedAttention Enables High-Throughput LLM Inference
Deep analysis of vLLM's high-throughput inference engine architecture, covering PagedAttention paging, KV Cache memory management, and continuous batching scheduling strategies.
Read more →
·2 min
AMD Acquires Taalas: What Does Etching AI Models Into Silicon Mean?
AMD acquires chip startup Taalas to etch AI models directly into silicon for extreme inference efficiency. We analyze the technology, tradeoffs, and AMD's differentiated AI strategy.
Read more →
·3 min
NVFP4 Dynamic Quantization in Practice: W4A4 Accelerated Deployment for the Full Gemma-4 Model Family
NVFP4 dynamic quantization covers all five Gemma-4 model sizes using W4A4 mixed-precision with calibrated FP8 KV Cache, dramatically reducing VRAM usage and deployment costs for efficient inference from edge to cloud.
Read more →
·3 min
How Open-Source Models Achieve Superior Retrieval Performance at 1% of GPT's Cost
Deep analysis of how open-source models match GPT-level retrieval performance at 1/100th the cost. Covers RAG cost optimization, embedding model fine-tuning, and deployment strategies.
Read more →