#quantization GGUF
305 related articles
Tutorials·3 min
AnythingLLM Installation & Configuration Guide: Building a Local Knowledge Base with API Integration
Complete guide to AnythingLLM local knowledge base setup: installation tips, Ollama model configuration, document vectorization, recall optimization, and API integration.
Read more →
Product Reviews·3 min
Hertzman: A Free, No-Install Local LLM Deployment Tool Review
Detailed review of Hertzman local inference engine covering one-click deployment, smart hardware recommendations, OpenAI-compatible API, and performance comparison with LM Studio.
Read more →
Tutorials·2 min
Complete Guide to Configuring Local DeepSeek Model in PyCharm for AI-Assisted Programming
Learn how to configure a local DeepSeek model in PyCharm via Ollama for free, privacy-safe AI-assisted programming. Includes installation steps, plugin setup, usage tips, and hardware recommendations.
Read more →
Product Reviews·4 min
OpenHuman Deep Dive: A Context-First Open-Source Personal AI Agent
Deep dive into OpenHuman open-source AI Agent: context-first architecture, Rust+React hybrid, Memory Tree system, Token Juice compression, and multi-model routing.
Read more →
Tutorials·2 min
pnpm Monorepo Full-Stack AI Engineering in Practice: Building a Multimodal Conversation System
Learn how to build a full-stack multimodal AI conversation system using pnpm Monorepo architecture, covering local model integration, image understanding, and streaming chat.
Read more →
Tutorials·3 min
Practical Guide to Building Multi-Agent Collaborative Applications with CrewAI + FastAPI
Learn how to build a multi-Agent collaborative system with CrewAI and FastAPI. Covers Agent, Task, Crew concepts, GPT/Tongyi Qianwen/Ollama integration, with complete code examples and model comparisons.
Read more →
Tutorials·2 min
PyCharm AI Assistant Deep Dive: Local Completion, Edit Mode & Practical Tips
Explore PyCharm AI Assistant's new features: free local AI completion, cloud-powered generation, Chat & Edit modes, and context management tips for Python developers.
Read more →
Product Reviews·3 min
Running Qwen3.6-27B Locally on Mac: 4 Solutions Benchmarked
Benchmarking 4 solutions for running Qwen3.6-27B locally on Mac: GGUF, MLX Diflash, and MTP-LX. MTP-LX 4bit leads at 43.6 tok/s with solid coding, writing, and reasoning quality.
Read more →
Tutorials·3 min
Decoding LLM Naming Conventions: Parameter Counts, Quantization Formats & VRAM Requirements Quick Reference
Decode LLM naming conventions, understand 32B parameters & AWQ/GGUF quantization formats, with 4-bit VRAM estimation formulas, MOE model pitfalls, and model selection by GPU tier.
Read more →
Tutorials·3 min
Running AI Models on a P106 Mining GPU: Build a Local AI Workstation for Under $10
Build a local AI workstation with a P106 mining GPU for under $10. Run Live Portrait and other AI models locally with full privacy, zero marginal cost, and incredible value.
Read more →
Tutorials·1 min
Ollama Local LLM Deployment Tutorial: Complete Guide from Installation to API Integration
Complete guide to deploying open-source LLMs locally with Ollama, covering installation, model selection, quantization strategies, Python API integration, and performance optimization tips.
Read more →
Product Reviews·1 min
QwenCoder Local Deployment Review: Can It Replace Paid AI Coding Assistants?
Hands-on review of QwenCoder 80B deployed locally, compared to Gemini and Claude. Covers hardware setup, LM Studio deployment, and real coding test results to help you decide if local models can save on AI subscriptions.
Read more →
Tutorials·2 min
DeepSeek Local Deployment Tutorial: One-Click Install & Run with Ollama
Learn how to deploy DeepSeek R1 locally using Ollama with this step-by-step guide covering installation, model selection, hardware requirements, and advanced features like GUI and knowledge bases.
Read more →
Tutorials·2 min
How to Tell If Your GPU Is Out of VRAM When Running Local LLMs: A Complete Guide to VRAM Monitoring
How to tell if your GPU is out of VRAM when running local LLMs. Learn the difference between dedicated and shared GPU memory, monitor VRAM overflow via Task Manager, and use quantization and context length control to avoid OOM.
Read more →
Product Reviews·1 min
DeepSeek V4 vs Qwen3.6 Real-World Testing: In-Depth Comparison Across 8 Categories and 150 Scenarios
BenchLocal real-world testing of DeepSeek V4 Pro, V4 Flash vs Qwen3.6 27B across 8 categories and 85 scenarios. V4 Pro leads by 6% but stumbles on math reasoning. Qwen3.6 Q6 rivals V4 Pro in agent tasks.
Read more →
Product Reviews·1 min
Qwen3.6-27B Open-Source Model Review: Flagship-Level Code and Multimodal Capabilities with Just 27B Parameters
Deep dive into Alibaba's Qwen3.6-27B: a 27B dense model delivering flagship-level code generation and multimodal capabilities on a single GPU with INT4 quantization.
Read more →
Tutorials·2 min
Hermes + Qwen3.6 Local Deployment Tutorial: Build Your Private AI Agent at Zero Cost
Step-by-step tutorial to deploy Hermes Agent with Qwen3.6 open-source LLM locally. Covers WSL setup, model download, Telegram bot integration for a zero-cost private AI Agent.
Read more →
Tutorials·2 min
vLLM and SGLang Local Deployment Tutorial: A Practical Guide to 3-8x Performance Gains
Complete guide to deploying vLLM and SGLang locally. Compare performance vs LM Studio, deploy in 3 steps with Docker + AI assistant. Covers SGLang vs vLLM selection, 5090 VRAM optimization, and Cherry Studio integration.
Read more →
Tech Frontiers·1 min
Qwen3.6 MTP Acceleration Benchmarked: Single GPU Inference Hits 220 token/s
Qwen3.6 experimental MTP-GGUF benchmarked: single GPU pushes 35B-A3B model to 220 token/s, 1.4x faster with zero accuracy loss. Covers MTP principles, optimal Draft Tokens strategy, and RTX 5090 results.
Read more →
Industry Insights·2 min
Enterprise LLM Selection Guide: In-Depth Comparison of Llama 3.1, Qwen 2.5, and DeepSeek
How should enterprises choose open-source LLMs? This guide compares Llama 3.1, Qwen 2.5, DeepSeek, and Mistral across model capabilities, hardware requirements, and business scenarios.
Read more →