#high-throughput serving
40 related articles
·4 min
Building Container Networking From Scratch: A Deep Dive Into Linux Bridge and veth
A deep dive into the fundamentals of container networking. By manually building a Linux bridge, veth pair, and iptables NAT, fully reproduce Docker's default bridge mode to understand container connectivity, port mapping, and Kubernetes networking.
Read more →
·6 min
DeepSeek Builds Its Own AI Chips: The Strategic Logic and Challenges Behind Compute Independence
DeepSeek is entering AI chip development, targeting compute autonomy. This article analyzes its motivations, software-hardware synergy, chip R&D challenges, and impact on China's AI vertical integration.
Read more →
·5 min
GPT-5.6 Released: Outperforms Rivals but Halted by Government—Cybersecurity Capabilities Trigger Regulatory Red Line
OpenAI releases GPT-5.6 (Sol/Terra/Luna), beating Anthropic on Terminal Bench at ~40% lower cost. But its cybersecurity capabilities hit danger thresholds, limiting access to trusted partners at government request.
Read more →
Local Coding Agents in Practice: A Com…
·6 min
Local Coding Agents in Practice: A Complete Guide to Replacing Your Claude Subscription with Open-Source Models
An in-depth look at local coding agents—core concepts, advantages, and real challenges. Compare against Claude Code and learn to build a zero-subscription, private AI coding workflow with open-weight models.
Read more →
·6 min
Grok 4.5 Coding Benchmark: A Deep Dive into Performance, Speed, and Cost
xAI releases Grok 4.5, purpose-built for coding agents. 80 TPS speed, $2/M input tokens, SWE Bench Pro score of 64.7, and 4.2x better token efficiency than Opus 4.8. A deep hands-on review.
Read more →
·6 min
A Deep Dive into the vLLM Inference Framework: Core Principles of Throughput Optimization and Interview Guide
An in-depth analysis of the vLLM inference framework's core principles: from the meaning of throughput (tokens/s), to the bottlenecks of autoregressive generation, to KV Cache, PagedAttention, and continuous batching.
Read more →
·4 min
Proactive AI Agent Monitoring: Stop Fighting Fires, Catch Bugs Before Your Customers Do
A real case study: team builds AI Agent "Oogway" to auto-patrol after every job, investigate anomalies, create tickets, and update a knowledge Wiki — catching bugs before customers do.
Read more →
·6 min
Databricks Agent Framework in Practice: A Guide to ChatAgent Wrapping and Production Deployment
A deep dive into Databricks Agent Framework (Mosaic AI): unify LangGraph/OpenAI agents via ChatAgent, log & evaluate with MLflow, version with Unity Catalog, and deploy Model Serving Endpoints for production AI agents.
Read more →
·7 min
HuggingFace Open-Sources ml-intern: An AI Engineer That Reads Papers and Trains Models Autonomously
Hugging Face's open-source ml-intern autonomously reads papers, writes training scripts, and finetunes LLMs, deeply integrating the HF ecosystem and smolagents. Explore its features and impact on ML careers.
Read more →
·1 min
Unsloth v0.1.47-beta Released: 2x Faster LLM Fine-Tuning with 70% Less VRAM
Unsloth v0.1.47-beta is out. This 67.9k-star open-source framework fine-tunes Llama, Mistral, and Qwen 2x faster with 70% less VRAM on consumer GPUs.
Read more →
·6 min
Anthropic Launches Life Sciences Hackathon with $100K to Empower AI-Driven Research
Anthropic launches a global life sciences hackathon with Gladstone Institutes, offering $100K in API credits to advance AI-driven scientific research using Claude Science and Claude Code.
Read more →
·3 min
RuView: Camera-Free Spatial Awareness and Vital Sign Monitoring Using Ordinary WiFi Signals
RuView is a Rust-based open-source project that uses commercial WiFi CSI signals for real-time spatial awareness, presence detection, and vital sign monitoring — no cameras needed. 76,500+ GitHub stars.
Read more →
·2 min
Baseten Raises $1.5 Billion: Why AI Inference Infrastructure Has Become a Capital Darling
AI inference startup Baseten is raising $1.5B at a $130B valuation. We analyze why inference infrastructure is booming, the competitive landscape, and what this mega-round signals.
Read more →
·2 min
MiniMax M3 Launches on Fireworks: 512K Context and MSA Sparse Attention Explained
MiniMax M3 launches on Fireworks with 512K context and multimodal input. MSA sparse attention delivers 9x prefill and 15x decode speedups. Deep dive into architecture, pricing, and open-model competition.
Read more →
·3 min
vLLM Deep Dive: How PagedAttention Enables High-Throughput LLM Inference
Deep dive into vLLM's core technologies for high-throughput LLM inference, including PagedAttention memory management, continuous batching, distributed deployment, and comparisons with TensorRT-LLM.
Read more →
·1 min
Gemini 3.5 Flash Early Experience: Balancing Speed and Capability
Developers share early hands-on experience with Google's Gemini 3.5 Flash model, showing fast speed, strong coding ability, and self-correction. A deep dive into its performance, testing, and positioning.
Read more →
Industry Insights·3 min
Deep Dive into Three Major LLM Career Paths: Requirements, Tech Stacks, and Career Prospects
Deep analysis of three core LLM roles—Application Engineer, Development Engineer, and Algorithm Engineer—covering technical requirements, salary thresholds, and career prospects including RAG, fine-tuning, and inference deployment.
Read more →
Industry Insights·2 min
Meta Partners with AWS: Bringing in Tens of Millions of Graviton Cores to Expand AI Infrastructure
Meta partners with AWS to add tens of millions of Graviton cores for AI inference, diversifying its infrastructure to support Meta AI and Agentic experiences for billions of users.
Read more →
Industry Insights·2 min
NVIDIA Blackwell Sets New STAC-AI Records for Financial LLM Inference
NVIDIA Blackwell GPU sets new LLM inference records in STAC-AI financial benchmark. Explore Blackwell architecture advantages, TensorRT-LLM co-optimization, and LLM applications in trading and risk management.
Read more →
Tutorials·5 min
The Complete Guide to Ollama: Run DeepSeek and Other LLMs Locally with a Single Command
Ollama is an open-source tool with 170K GitHub Stars that lets you run DeepSeek, Qwen, Kimi-K2.5 and other LLMs locally with one command. Learn about its model ecosystem, advantages, and use cases.
Read more →