#on-device inference

150 related articles

Jul 21, 2026·3 min

27B Parameter LLM on an iPhone? A Deep Dive into On-Device Full-Parameter AI

Startup Prismo claims to compress a 27B-parameter model to 4GB for full local inference on iPhone 17 Pro. We break down the compression tech, compare it to Apple's MoE approach, and assess real-world limits.

Jul 21, 2026·3 min

Qwen3 Deep Dive: A Complete Look at the Open-Source Hybrid Reasoning Model

Qwen3 deep dive: 6 Dense & MoE models from 0.6B to 235B, world's first open-source hybrid reasoning model, matching Gemini 2.5 Pro. Complete guide for developers and enterprises.

Moonshine: A Low-Latency Speech Engine…

Jul 21, 2026·2 min

Moonshine: A Low-Latency Speech Engine and Open-Source Powerhouse for Building Voice Agents

Moonshine is an open-source, C++-based low-latency speech engine combining STT, intent recognition, and TTS for building voice agents. 9,400+ GitHub Stars.

Jul 21, 2026·3 min

Claude Opus 5 Launch Imminent, Alibaba Qwen Officially Integrated into Apple Intelligence

Claude Opus 5 launches next week; Alibaba Qwen integrates into Apple Intelligence for Chinese users; 27B on-device model compressed to 3.8GB; open-source models narrow gap to closed-source by 3.3%.

Local AI Models vs. Cloud: This Tech J…

Jul 21, 2026·2 min

Local AI Models vs. Cloud: This Tech Judgment Call Is Testing Your Thinking

"Your opinion on local AI is an IQ test" — this viral claim reveals the cognitive divide between local and cloud AI deployment, from data sovereignty to TCO.

Jul 20, 2026·3 min

LightMem-Ego: The AI Memory Assistant That Remembers the Details of Your Daily Life

LightMem-Ego is an AI life assistant with long-term memory — tracking object locations, meetings, and life events over time. A deep dive into its technology and real-world challenges.

Jul 20, 2026·3 min

DeepSeek and Zhipu Race to Build Custom AI Chips: China's Path to AI Compute Independence

DeepSeek seeks $7B for custom AI inference chips; Zhipu AI explores ASIC. Deep dive into China's AI compute independence strategy, multimodal generation, agents, and hardware trends.

Jul 20, 2026·3 min

Running a 27B Model on iPhone: How 1-bit Quantization Achieves Extreme Compression

PrismML's Bonsai compresses a 27B model from 54GB to 3.9GB, running at ~11 tokens/sec on iPhone. A deep dive into QAT, knowledge distillation, and speculative decoding.

Jul 20, 2026·2 min

On-Device LLMs, Custom Silicon, and 100K-GPU Clusters: AI Industry Enters Full-Spectrum Competition

Apple's 1-bit compression fits 27B models on iPhone, Meta builds custom chip Iris, China's 100K-GPU cluster goes live, Samsung enters AI PC — a deep dive into AI's new full-spectrum competition.

Jul 20, 2026·3 min

Apple vs. OpenAI: The 400-Employee Exodus and the Battle for AI Hardware Dominance

Apple sues OpenAI over 400 former employees allegedly stealing trade secrets. A deep dive into AI talent wars, hardware competition, HBM memory chips, and chip localization.

AirLLM: How a 4GB GPU Can Run a 70B Mo…

Jul 19, 2026·2 min

AirLLM: How a 4GB GPU Can Run a 70B Model — Core Principles and Use Cases

AirLLM is an open-source project that uses layer-by-layer inference to run 70B LLMs on a single 4GB GPU. Learn how it works, its tradeoffs, and ideal use cases.

Jul 19, 2026·3 min

Google Gemma 4 On-Device Model: Optimized for Pixel 10 TPU — Offline Multimodal Capabilities Tested and Analyzed

Google's Gemma 4 E2B for TPU runs offline on Pixel 10's Tensor G5 chip, enabling local AI chat, image recognition, and audio transcription. We break down the features and real-world test results.

Jul 19, 2026·3 min

GPT-5.6 Launches, Siri Gets a Major Overhaul: Your Complete AI Industry Roundup

OpenAI launches GPT-5.6, Apple rebuilds Siri, China's AI companionship regulations take effect, Google labels AI images — your one-stop global AI industry roundup.

Block Low-Rank Compression: A Guide to…

Jul 19, 2026·2 min

Block Low-Rank Compression: A Guide to Accelerating Large Model Inference on Memory-Constrained GPUs

Learn how Block Low-Rank (BLR) decomposition compresses large model memory usage and accelerates GPU inference, including CUDA kernel optimization and combination with quantization and pruning.

The Guardian Angels Framework: How LLM…

Jul 18, 2026·3 min

The Guardian Angels Framework: How LLM Personalization Can Achieve Both Efficiency and Data Security

The Guardian Angels framework shows how LLM personalization can achieve both productivity and data security through local deployment, differential privacy, and tiered permissions.

Jul 18, 2026·1 min

Apple's Vibe Coding Showcase: What Did College Students Build with Mac + AI?

Apple hosted a college AI showcase in Guangzhou where students used Mac and AI tools to build a smart anti-sedentary chair, AR robotic arm, dental simulator, and more. A deep dive into Vibe Coding and Apple's AI hardware strategy.

Jul 18, 2026·4 min

GPT-5.6 + Codex Tested: 3 Prompts to Edit a 15-Minute Video, Efficiency Doubled

OpenAI's GPT-5.6 and Codex integration tested: 3 prompts to edit a 15-min video in just 20 minutes. Full breakdown of performance, Token costs, GPT Work automation, and Chat Card real-world results.

Jul 17, 2026·3 min

Chrome Built-in AI in Action: Six APIs for Building a Smart Blog Editor

Deep dive into Chrome Built-in AI's six core APIs — Summarizer, Prompt API, Translator and more — applied to a real blog editor. Local, private, no server needed.

Bonsai 27B: The First 1-bit LLM That R…

Jul 17, 2026·3 min

Bonsai 27B: The First 1-bit LLM That Runs on Your Phone — A New Milestone for On-Device AI

Bonsai 27B is the first 27B-parameter LLM that runs on smartphones via 1-bit quantization, compressing to 3–4GB. We break down the tech, privacy benefits, and community debate.

Jul 17, 2026·3 min

The AI Privacy Debate: Why Local Models and Open-Source Frameworks Are Essential

When cloud AI privacy terms hide data-sharing risks, local model deployment and open-source frameworks offer developers a path to true data control. Analyzing xAI, OpenAI, GLM, Kimi, and Qwen.