#inference deployment

1954 related articles

Jul 29, 2026·3 min

Production-Grade LLM Inference Deployment: A Practical K8s Handbook

Complete guide to deploying production-grade LLM inference on Kubernetes, covering GPU scheduling, vLLM engine selection, autoscaling, observability, and cost optimization.

Jul 28, 2026·3 min

ncnn Vulkan Cross-Platform Edge Inference in Practice: Solving the GPU Fragmentation Deployment Challenge

Facing GPU fragmentation on edge devices, the PostSlate team used ncnn's Vulkan backend for cross-platform ML inference, achieving 10× speedup on RTX 4070 with half the model size and zero runtime installation.

Jul 27, 2026·3 min

DeepSeek-OCR Deployment & Fine-Tuning in Practice: From vLLM Inference to Unsloth Fine-Tuning

Complete guide to DeepSeek-OCR from vLLM inference deployment and Unsloth model loading to fine-tuning, covering cloud server setup, GPU selection, and code examples — all on a single 4090 GPU.

Jul 24, 2026·6 min

AI Application Engineer Interview Guide: Core Topics in Model Quantization and Inference Deployment

A systematic review of must-know topics for AI Application Engineer interviews: PTQ/QAT quantization, operator fusion, inference pipelines, latency/throughput analysis, and edge deployment of detection/segmentation/BEV models.

Jul 24, 2026·6 min

AI Application Engineer Interview Guide: Core Topics on Model Quantization and Inference Deployment

A systematic guide to must-know AI application engineer interview topics: PTQ/QAT quantization, operator fusion, inference pipelines, latency/throughput analysis, and edge deployment of detection/segmentation/BEV models.

Jul 22, 2026·2 min

Agent A1 Open Source: 35B MoE Local Inference Speed Test & Deployment Guide

Agent A1 is a 35B MoE open-source model from Shanghai's Intern Science, activating only 3B params. Hits 95 tok/s on M4 Mac Studio. Supports 256K context, tool calling, and free local deployment via Ollama.

Jul 17, 2026·2 min

vLLM Inference Framework Deep Dive: A Practical Guide to High-Concurrency DeepSeek Deployment

A deep dive into vLLM's five core features — KV cache management, continuous batching, and distributed inference — plus a full setup guide for deploying DeepSeek on a cloud server.

Java Local LLM Inference: Low-Latency …

Jul 16, 2026·2 min

Java Local LLM Inference: Low-Latency LLM Deployment with Panama FFM

Learn how Java and OpenJDK Panama FFM API enable local LLM inference. Explore the technical foundations, JVM ecosystem benefits, and low-latency AI deployment in enterprise Java systems.

Jul 14, 2026·2 min

Production LLM Deployment Guide: Model Selection, GPU Configuration, and Inference Engine Breakdown

A comprehensive decision framework for production LLM deployment: model selection (open-source vs. API), GPU VRAM configuration, and inference engine comparison including vLLM and TensorRT-LLM.

Jul 13, 2026·2 min

Distributed AI Systems: A Practical Engineering Guide to Training, Inference, and Production Deployment

A deep dive into distributed AI systems engineering: data/model/tensor parallelism for training, KV cache, quantization, elastic scaling for inference, and cloud deployment with Kubernetes, Ray, and DeepSpeed.

Jul 13, 2026·3 min

Distributed AI Systems in Practice: A Complete Guide to Training, Inference, and Production Deployment

A deep dive into Distributed AI Systems: a new book distilling 10 years of AI engineering experience covering distributed training, inference optimization, and production model serving.

Jun 15, 2026·3 min

Firebase AI Logic Deep Dive: Hybrid Inference, Prompt Templates & Multi-Layer Security Deployment

Deep dive into Firebase AI Logic: server-side prompt templates to prevent leakage, Cloud Function triggers, four-layer security defense, AI monitoring with context caching for cost control, and cross-platform hybrid inference.

Jun 4, 2026·4 min

Google Hybrid Inference Comes to iOS: A Complete Guide to On-Device AI Cross-Platform Deployment

Google Hybrid Inference now supports iOS, adds Gemma 4 on Android, and Chrome local Web inference nears GA. A deep dive into hybrid inference technology, cross-platform advantages, and developer opportunities.

Jun 4, 2026·4 min

Google Hybrid Inference Comes to iOS: A Complete Guide to On-Device AI Cross-Platform Deployment

Google Hybrid Inference officially supports iOS, adds Gemma 4 on Android, and Chrome local Web inference nears GA. A deep dive into hybrid inference technology, cross-platform advantages, and developer opportunities.

ONNX Model Deployment in Practice: A Complete Guide from PyTorch Export to Cross-Framework Inference

Tutorials

May 22, 2026·2 min

ONNX Model Deployment in Practice: A Complete Guide from PyTorch Export to Cross-Framework Inference

Complete guide to ONNX neural network exchange format: PyTorch and TensorFlow model export, ONNX Runtime inference, MNIST classifier deployment, and HuggingFace model downloads with runnable code examples.

AI Model Deployment Pipeline Friction: How TensorRT Systematically Eliminates Inference Optimization Bottlenecks

Deep Dives

May 17, 2026·3 min

AI Model Deployment Pipeline Friction: How TensorRT Systematically Eliminates Inference Optimization Bottlenecks

Deep dive into pipeline friction in AI model deployment from training to production, covering TensorRT automated optimization, ONNX export, and Triton Inference Server best practices.

Aug 3, 2026·3 min

Synthesizing 190° Fisheye Driving Videos: The Geometric Consistency Challenge for ADAS Training Data

Exploring how to synthesize 190° fisheye driving videos based on camera calibration parameters, analyzing how geometric consistency impacts ADAS perception model training, and the opportunities and domain gap challenges of synthetic data in surround view systems.

Aug 3, 2026·3 min

Token Economics: The New Unit of Value Measurement in the AI Era

Deep dive into how Tokens evolved from a technical concept in LLMs to the core unit of measurement in the AI economy. Exploring Token consumption explosion, cost optimization, and Token economics.

Aug 3, 2026·3 min

Training a 1.3B Parameter LLM from Scratch: Complete Workflow and Core Challenges

A deep dive into the complete workflow of training a 1.3B parameter LLM from scratch, covering Transformer architecture design, data preparation, and distributed training optimization.

Aug 3, 2026·2 min

Ollama Max Account Banned with 403 Error: Paying Users Face a Service Trust Crisis

An Ollama Max paid user's account was banned with a 403 error just two weeks after subscribing, with no support response in three days. Analysis of causes and practical appeal solutions.