#quantized model inference
354 related articles
Tech Frontiers·1 min
Gemini 3.5 Flash Released: How Google Balances AI Model Speed and Capability
Google releases Gemini 3.5 Flash, optimizing the balance between speed and capability. Analysis of Flash series evolution, comparisons with GPT-4o mini, and practical value for developers.
Read more →
Product Reviews·1 min
DeepSeek V4 vs Qwen3.6 Real-World Testing: In-Depth Comparison Across 8 Categories and 150 Scenarios
BenchLocal real-world testing of DeepSeek V4 Pro, V4 Flash vs Qwen3.6 27B across 8 categories and 85 scenarios. V4 Pro leads by 6% but stumbles on math reasoning. Qwen3.6 Q6 rivals V4 Pro in agent tasks.
Read more →
Product Reviews·1 min
Qwen3.6 Quantized Local Deployment Benchmark: Which Is Best — NVFP4, APEX, Q4, or Q6?
Benchmarking 7-8 Qwen3.6 quantized models across 8 dimensions including tool calling, CLI ops, and bug fixing. Comparing NVFP4, APEX, Q4, Q6 with rankings and recommendations.
Read more →
Tech Frontiers·1 min
Qwen3.6 MTP Acceleration Benchmarked: Single GPU Inference Hits 220 token/s
Qwen3.6 experimental MTP-GGUF benchmarked: single GPU pushes 35B-A3B model to 220 token/s, 1.4x faster with zero accuracy loss. Covers MTP principles, optimal Draft Tokens strategy, and RTX 5090 results.
Read more →
Tutorials·1 min
DeepSeek + RAGFlow Local Deployment AI Knowledge Base Tutorial (2025 Latest)
Step-by-step guide to locally deploy a personal AI knowledge base with DeepSeek + RAGFlow + Ollama. Covers RAG principles, Ollama setup, Docker deployment, and knowledge base optimization.
Read more →
Tech Frontiers·4 min
Step 3.5 Flash Deep Dive: How a 196B Parameter Open-Source Model Beat Gemini
Deep dive into StepFun's Step 3.5 Flash: 196B parameter MoE model activating only 11B, 350 tokens/sec coding speed, 256K context window, local deployment ready. How it beats Gemini 3 Flash.
Read more →
Tutorials·4 min
NVIDIA Model Optimizer Post-Training Quantization (PTQ) Practical Guide
A deep dive into NVIDIA Model Optimizer's PTQ workflow, covering INT8/INT4 quantization principles, calibration methods, RTX GPU optimization, and best practices for deploying quantized LLMs on consumer GPUs.
Read more →
Deep Dives·2 min
The MARVIS Project: A Complete Analysis of Embedded AI Agents Empowering Autonomous Space Exploration
Deep dive into how the MARVIS project deploys LLM agents on spacecraft, covering agent architecture, edge hardware token performance benchmarks, expert evaluations, and space AI benchmark planning.
Read more →
Tutorials·2 min
Beginner's Guide to AI Large Language Models: GPU Requirements & Core Tech Stack Explained
2025 complete guide to AI LLMs: local deployment GPU/VRAM requirements (RTX 4090/24GB) and core tech stack including Prompt Engineering, Agents, MCP, LangGraph, and WorkFlow orchestration.
Read more →
Tutorials·3 min
Grammar-Constrained Decoding: A Practical Guide to Generating Reliable Bash Code with Small Models
Deep dive into Grammar-Constrained Decoding (GCD) technology: applying Bash syntax constraints during inference to dramatically improve small language models' code generation correctness and executability for AI Agent edge deployment.
Read more →
Product Reviews·3 min
AnythingLLM: A Deep Dive into the All-in-One Local AI Knowledge Base Tool
Deep dive into AnythingLLM, a privacy-first, zero-config local AI productivity platform. Supports RAG document chat, multi-model integration, knowledge bases, and AI Agents with nearly 60K GitHub stars.
Read more →
Product Reviews·4 min
Hugging Face Transformers: Deep Dive into the 160K-Star Open Source AI Model Framework
Deep dive into Hugging Face Transformers: architecture, multimodal support, ecosystem, and trends. Learn how this 160K-Star project became essential for AI developers.
Read more →
Tutorials·3 min
Connect Personal WeChat to AI: Screenshot + OCR Solution for Auto-Reply in 1 Hour
Low-risk personal WeChat AI integration via screenshot + OCR + hotkey simulation. Includes three approach comparisons, Ollama local Qwen vision model deployment, and solutions for infinite loops and cursor flicker issues.
Read more →
Product Reviews·3 min
Sim Studio: Deep Dive into the Open-Source AI Agent Workflow Platform with Nearly 10K Stars
Deep analysis of open-source AI workflow platform Sim Studio with nearly 10K GitHub Stars. Apache 2.0 licensed, supports full local deployment and Ollama local LLM integration. Compared with Dify and n8n.
Read more →
Tech Frontiers·4 min
Claude's Quota Slashed by 20x — Can GLM 4.6 Be the Best Alternative?
Anthropic slashes Claude Max Opus quotas by 20x, driving developer exodus. GLM 4.6 enters with 1/10th API pricing, 200K context window & open weights. Full comparison inside.
Read more →
Product Reviews·3 min
VS Code AI Toolkit 2.0 Deep Dive: Agent Builder + Free Access to GPT-5
Deep dive into VS Code AI Toolkit 2.0 major update, covering Agent Builder, MCP tool integration, batch testing, model evaluation, and a complete guide to using GPT-5 and Claude for free via GitHub Models.
Read more →
Tutorials·3 min
Ollama + OpenCode Local AI Coding: A Complete Zero-Cost Alternative to Cursor
Tutorial: Deploy Qwen3 Coder locally via Ollama with OpenCode for zero-cost AI coding. Covers setup, code generation, auto-debugging, and hardware recommendations.
Read more →
Product Reviews·3 min
GPT-OSS 120B Local Deployment Review: Comprehensive Comparison with O4 Mini on Code Generation and Reasoning
Hands-on review of OpenAI's first open-weight reasoning model GPT-OSS 120B, locally deployed via Ollama, benchmarked against O4 Mini on code generation, reasoning, and UI design.
Read more →
Tutorials·3 min
GPT-OSS Open-Source Model Local Deployment Tutorial: Real-World Performance and VRAM Usage Analysis
Complete guide to deploying OpenAI's open-source GPT-OSS model locally with Ollama. Real-world testing of the 20B version on RTX 4090 covering Chinese comprehension, logical reasoning, and VRAM usage analysis under MoE architecture.
Read more →
Tutorials·3 min
LocalAI Tutorial: Deploy LLMs Locally Without a GPU Using Docker
Complete LocalAI deployment tutorial: run nearly 1,000 open-source LLMs locally without a GPU. One-click Docker setup, OpenAI API compatible, supports chat, image generation, and voice — fully private.
Read more →