#CUDA inference setup

62 related articles

Jul 29, 2026·3 min

Production-Grade LLM Inference Deployment: A Practical K8s Handbook

Complete guide to deploying production-grade LLM inference on Kubernetes, covering GPU scheduling, vLLM engine selection, autoscaling, observability, and cost optimization.

Jul 28, 2026·3 min

Monocular RGB Human Mesh Reconstruction: Real-Time Motion Capture at 18 FPS with Rust + CUDA

Deep dive into a real-time 3D human mesh reconstruction project using a single RGB camera, built with Rust, Candle, and CUDA, achieving 55ms/frame on RTX 5080. Exploring its architecture, Metal porting plans, and applications in VTuber, AR/VR, and sports analysis.

Jul 28, 2026·3 min

Monocular RGB Human Mesh Reconstruction: Real-Time Motion Capture at 18 FPS with Rust + CUDA

A deep dive into a real-time 3D human mesh reconstruction project using a single RGB camera, built with Rust, Candle, and CUDA, achieving 55ms/frame on an RTX 5080.

Jul 27, 2026·2 min

RX 9060 XT vs RTX 5060 Ti: Which GPU to Pick for Local AI?

RX 9060 XT vs RTX 5060 Ti — both 16GB VRAM, but which is better for local AI? We compare CUDA ecosystem, ROCm compatibility, LLM inference, and real-world usability.

Jul 24, 2026·6 min

The AI Industrial Age Arrives: Compute Wars, Chip Reconstruction, and the GPT-5.6 Cheating Scandal Explained

Five key AI industry trends: Doubao surpasses 180 trillion daily calls, OpenAI's in-house AI chip, NVIDIA's $3-4 trillion compute forecast, China catching up, and the GPT-5.6 cheating scandal.

Jul 24, 2026·4 min

Dify Local Deployment Guide: One-Click Private AI Platform with Docker

Step-by-step guide to deploying Dify AI platform locally with Docker. Covers Linux, Windows, macOS setup, docker compose launch, and first-time initialization in under 30 minutes.

Jul 23, 2026·3 min

InvokeAI In-Depth Review: Can It Replace SD WebUI and ComfyUI?

A hands-on InvokeAI review covering installation, VRAM optimization, infinite canvas, node workflows, and model management — with an objective comparison to SD WebUI and ComfyUI.

Jul 23, 2026·7 min

The Complete AI Character LoRA Training Workflow: Hands-On Multi-Character Consistency Generation with Local Deployment

A detailed guide to a complete local AI character generation workflow: from the five golden rules of LoRA training and automated ComfyUI dataset construction to hands-on comparisons of Crea2, Ideogram4, and Wan for multi-character same-frame interaction—all running free on personal hardware.

Jul 21, 2026·3 min

Jetson Edge AI Development: Setting Up a VS Code + Claude Code Workstation

Set up an AI coding workstation on NVIDIA Jetson using VS Code + Claude Code. Covers JetPack NVIDIA Skills, token budget management, headless SSH dev, and coding agent best practices.

Jul 20, 2026·3 min

lx: 72 Local CLI Tools That Bring Ollama Models Into Your Terminal

lx is a set of 72 single-purpose CLI tools running on local Ollama models — no API key, fully offline. Supports git commit generation, log debugging, and more. Rust binaries with <15ms cold start; 7–8B models work great.

Jul 20, 2026·3 min

Bernini Tile-Based Generation: An Engineered Approach to Video Super-Resolution in ComfyUI

Bernini is a ComfyUI video super-resolution node package using Tile Split/Select/Merge to solve seams, drift, and VRAM overflow. Benchmarked at 325s for 39 frames at 1920×1080.

Jul 19, 2026·4 min

AMD Ryzen AI Halo Real-World Test: Can a Non-Programmer Use Local AI to Write Code?

A non-programmer tests AMD Ryzen AI Halo by deploying local AI models to tackle a real dev task. After testing Ollama and Qwen3, the verdict: AI amplifies developers, it doesn't replace them.

Jul 19, 2026·3 min

The Complete Ollama Guide: The Simplest Way to Run LLMs Locally

Ollama has 9M active developers and 170K GitHub stars. This guide covers its features, architecture, business model, and use cases for running LLMs locally.

Bonsai Open Source Project Deep Dive: …

Jul 18, 2026·2 min

Bonsai Open Source Project Deep Dive: A Fresh Approach to Edge AI with Lightweight Machine Learning

Bonsai is a Shell-based lightweight ML open source project that gained 196 GitHub stars in one day. This deep dive covers its TinyML positioning, edge AI use cases, and value for embedded AI developers.

Jul 17, 2026·3 min

Cherry Studio + MCP: A Practical Guide to Building AI Agents with Local Deployment

Learn how to build an automated AI agent using Cherry Studio, MCP protocol, and locally deployed models — covering DeepSeek integration, web scraping, and private knowledge base setup.

Jul 17, 2026·2 min

vLLM Inference Framework Deep Dive: A Practical Guide to High-Concurrency DeepSeek Deployment

A deep dive into vLLM's five core features — KV cache management, continuous batching, and distributed inference — plus a full setup guide for deploying DeepSeek on a cloud server.

Jul 17, 2026·3 min

Complete Guide to Deploying LLMs Locally on RobotCore Mini: From Ollama Installation to Web Interface Setup

A complete guide to deploying LLMs locally on RobotCore Mini using Ollama — covering model pulling, CLI verification, Web backend setup, and LAN access.

Jul 17, 2026·3 min

Cherry Studio + MCP + Ollama: The Complete Guide to Building a Local AI Agent and Private Knowledge Base

A hands-on guide to building a local AI agent and private knowledge base using Cherry Studio, MCP, and Ollama — with web scraping, report generation, and terminal control.

Jul 16, 2026·3 min

Bonsai-27B Extreme Quantization: Running 1-bit LLMs on Consumer Hardware

Bonsai-27B supports binary/ternary extreme quantization for 27B LLMs running on 8GB VRAM. Covers llama.cpp upstream progress, RTX 4060 benchmarks (30 t/s), and real-world limitations.

Jul 16, 2026·3 min

GPUStack in Action: Complete Guide to Day 0 Deployment of DeepSeek-V4-Flash

Deploy DeepSeek-V4-Flash DSpark on 8× H20-141G using GPUStack's SGLang backend on Day 0. Full walkthrough of Web UI config, parameter tuning, and 200 tokens/sec benchmark results.