#CUDA inference setup
86 related articles
Bonsai Open Source Project Deep Dive: …
·2 min
Bonsai Open Source Project Deep Dive: A Fresh Approach to Edge AI with Lightweight Machine Learning
Bonsai is a Shell-based lightweight ML open source project that gained 196 GitHub stars in one day. This deep dive covers its TinyML positioning, edge AI use cases, and value for embedded AI developers.
Read more →
·3 min
Cherry Studio + MCP: A Practical Guide to Building AI Agents with Local Deployment
Learn how to build an automated AI agent using Cherry Studio, MCP protocol, and locally deployed models — covering DeepSeek integration, web scraping, and private knowledge base setup.
Read more →
·2 min
vLLM Inference Framework Deep Dive: A Practical Guide to High-Concurrency DeepSeek Deployment
A deep dive into vLLM's five core features — KV cache management, continuous batching, and distributed inference — plus a full setup guide for deploying DeepSeek on a cloud server.
Read more →
·3 min
Complete Guide to Deploying LLMs Locally on RobotCore Mini: From Ollama Installation to Web Interface Setup
A complete guide to deploying LLMs locally on RobotCore Mini using Ollama — covering model pulling, CLI verification, Web backend setup, and LAN access.
Read more →
·3 min
Cherry Studio + MCP + Ollama: The Complete Guide to Building a Local AI Agent and Private Knowledge Base
A hands-on guide to building a local AI agent and private knowledge base using Cherry Studio, MCP, and Ollama — with web scraping, report generation, and terminal control.
Read more →
·3 min
Bonsai-27B Extreme Quantization: Running 1-bit LLMs on Consumer Hardware
Bonsai-27B supports binary/ternary extreme quantization for 27B LLMs running on 8GB VRAM. Covers llama.cpp upstream progress, RTX 4060 benchmarks (30 t/s), and real-world limitations.
Read more →
·3 min
GPUStack in Action: Complete Guide to Day 0 Deployment of DeepSeek-V4-Flash
Deploy DeepSeek-V4-Flash DSpark on 8× H20-141G using GPUStack's SGLang backend on Day 0. Full walkthrough of Web UI config, parameter tuning, and 200 tokens/sec benchmark results.
Read more →
·3 min
Qwen3.5 Uncensored Version Local Deployment: Run a 35B Model on Just 6GB VRAM
Run Qwen3 35B-A3B uncensored locally on just 6GB VRAM using llama.cpp and GGUF quantization. Full deployment guide, code generation tests, and Agent integration walkthrough.
Read more →
·2 min
ExLlamaV3 v1.0.0 Official Release: Faster Inference, Lossless KV Quantization, and Streamlined Dependencies
ExLlamaV3 v1.0.0 releases with lossless KV cache quantization via kernel fusion, removal of flash-attention-2/xformers, major GEMM/GEMV gains, and broader tensor parallelism support.
Read more →
·2 min
Ollama Complete Guide: Running Open-Source LLMs Locally
Want to run open-source LLMs like DeepSeek locally for free? Ollama is the easiest solution. This guide covers what Ollama is, its core advantages, and key use cases.
Read more →
·4 min
Running Qwen3.6 27B Locally on a Single RTX 3090: A Complete Guide to Cloud-Rivaling Performance
One used RTX 3090, one 16.8GB GGUF file, and Qwen3.6 27B runs locally offline. SWE-bench score of 77 rivals Claude Sonnet. MTP boosts speed to 59 tok/s. Full local AI coding assistant deployment guide.
Read more →
Building an eGPU for Local LLM on a $1…
·2 min
Building an eGPU for Local LLM on a $1,000 Budget: A Practical Buying and Setup Guide
Building an eGPU for local LLMs on a $1,000 budget? This guide covers GPU selection, dock costs, RTX 3090 vs 3060 value, and top tools like Ollama and llama.cpp.
Read more →
·2 min
Getting Started with Ollama: A Powerful Tool for Deploying Open-Source LLMs Locally
Ollama is a free, open-source LLM management platform that lets you deploy open-source models like DeepSeek locally with one click. It supports macOS, Windows, Linux, and Docker, with both API and CLI modes to build private AI apps at zero cost.
Read more →
·4 min
AMD Ryzen AI Halo Hands-On: Can x86 Architecture Challenge NVIDIA's DGX Spark?
In-depth review of the AMD Ryzen AI Halo mini AI box: powered by the Ryzen AI Max Plus 395 (Strix Halo) chip with 128GB unified memory, priced at $4,000. Compared against NVIDIA's DGX Spark across token generation, prefill speed, and x86 advantages.
Read more →
·4 min
AI Coding Tool Data Leakage Risks: A Security Guide for Protecting Quantitative Strategies
AI coding tools carry cloud data transmission risks, exposing quantitative trading strategies to leakage. This article analyzes AI tool data security and offers protection strategies.
Read more →
·5 min
Getting Started with LangChain: LLM Invocation, Agent Practice, and Environment Setup Guide
Starting from the three limitations of LLMs, this guide systematically explains LangChain's core positioning, environment setup, API key prep, model init, and the message system. Learn init_chat_model and AIMessage/HumanMessage/SystemMessage.
Read more →
·3 min
ComfyUI-INT4-Fast: Running Flux Models Smoothly on 6GB GPUs
ComfyUI-INT4-Fast brings W4A4 quantized inference to ComfyUI. RTX 3060 (6GB VRAM) generates 1024×1024 images in 17s. Per-layer mixed-precision routing balances speed and quality for Flux models.
Read more →
·5 min
Why Did Ollama Rise So Fast? Decoding the Success of a Local LLM Tool
How did Ollama grow from a niche open-source project into developers' default choice for running local LLMs? This article breaks down its rise across product design, technical strategy, and ecosystem building.
Read more →
·5 min
Real-World Comparison of 6 Open-Source LLMs for On-Premise Deployment: A Selection Guide to Avoid Pitfalls
A hands-on comparison of 6 open-source LLMs (DeepSeek, Qwen3, Zhipu GLM, Kimi K2, MiniMax M3, Tencent Hunyuan 3) for on-premise deployment—covering hardware cost, inference efficiency, and deployment difficulty.
Read more →
·4 min
Complete Guide to Dify Local Deployment: Build a Private AI Workflow Platform from Scratch
Step-by-step guide to deploying Dify locally: Docker setup, Docker Compose installation, source code configuration, .env file setup, and container startup for Windows, macOS, and Linux.
Read more →