On-Device AI
A technical paradigm in which AI models are run locally on end-user devices—such as smartphones or PCs—for inference, without relying on cloud servers. This approach offers advantages in low latency and privacy protection.
Core Facts
Choosing Apple hardware to run AI locks one into the Apple ecosystem, incurring the cost of software adaptation and insufficient toolchain completeness.
Local AI (on-device AI) runs model inference directly on end-user devices, offering advantages such as privacy protection, low latency, offline availability, and cost reduction.
On-device AI refers to AI models running on the user's local device, rather than uploading data to cloud servers for processing.
Timeline (last 90 days)
For on-device AI, the model is often the least flexible part of the system; the image pipeline around it is where cheap optimization hides.
On-device AI, inference optimization, and compiler technology may become the main battleground for community discussion in the next phase.
The speaker hopes to exchange ideas with developers around four directions: on-device AI, efficient inference, voice agents, and machine learning compilers.
The more realistic form of on-device deployment will likely be running 30B-scale models on 32 GB of memory, combined with sparse attention and retrieval augmentation.
Running AI models on-device requires more RAM, which is driving up demand for memory chips across the entire industry.
On-device AI demand is driving the standard memory configuration of flagship models to jump from 4 GB to 8 GB or even 12 GB.
Placing the model as close to the user as possible simultaneously optimizes three dimensions: latency, privacy, and cost.
The fact that models run entirely locally in the browser means that user input never leaves the device, which is the privacy advantage of on-device models.
Running deepfake detection on-device means voice data does not need to be uploaded to a remote server, and call content can be processed locally on the device, providing a privacy advantage.
Deploying models on mobile phones typically requires compressing the model to within tens of MB, and is subject to constraints from processor computing power and battery life.
25 more timeline events
All Facts (20)
Choosing Apple hardware to run AI locks one into the Apple ecosystem, incurring the cost of software adaptation and insufficient toolchain completeness.
90%VerifiedLocal AI (on-device AI) runs model inference directly on end-user devices, offering advantages such as privacy protection, low latency, offline availability, and cost reduction.
85%VerifiedOn-device AI refers to AI models running on the user's local device, rather than uploading data to cloud servers for processing.
75%VerifiedThe core appeal of local AI lies in privacy, low latency, and offline availability.
70%VerifiedProcessing more AI tasks locally on-device helps enable better privacy protection, as data does not need to be uploaded to servers.
70%VerifiedOn-device AI keeps data local, protecting user privacy while reducing the latency and cost of cloud-based inference.
70%VerifiedOn-device AI offloads inference tasks to the local device, providing near-real-time response speeds while protecting user privacy.
65%VerifiedFor tasks such as complex reasoning, long document understanding, and high-quality image generation, on-device AI still requires invoking cloud-side Pro or Ultra-level models through a hybrid edge-cloud architecture.
65%Verified边缘AI（Edge AI）是将人工智能计算从云端迁移到本地设备（手机、传感器、嵌入式系统）上运行的技术范式
65%UnverifiedFor on-device AI, the model is often the least flexible part of the system; the image pipeline around it is where cheap optimization hides.
50%UnverifiedThe speaker hopes to exchange ideas with developers around four directions: on-device AI, efficient inference, voice agents, and machine learning compilers.
50%UnverifiedOn-device AI demand is driving the standard memory configuration of flagship models to jump from 4 GB to 8 GB or even 12 GB.
50%UnverifiedRunning AI models on-device requires more RAM, which is driving up demand for memory chips across the entire industry.
50%UnverifiedPlacing the model as close to the user as possible simultaneously optimizes three dimensions: latency, privacy, and cost.
50%UnverifiedThe fact that models run entirely locally in the browser means that user input never leaves the device, which is the privacy advantage of on-device models.
50%UnverifiedDeploying models on mobile phones typically requires compressing the model to within tens of MB, and is subject to constraints from processor computing power and battery life.
50%UnverifiedRunning deepfake detection on-device means voice data does not need to be uploaded to a remote server, and call content can be processed locally on the device, providing a privacy advantage.
50%UnverifiedThe fundamental reason for the response quality gap between on-device and cloud models lies in parameter scale and quantization compression; on-device models undergo aggressive quantization (such as INT4/INT8) and distillation, resulting in degraded instruction-following capability.
50%UnverifiedHybrid computing is not a concept unique to Perplexity; the collaboration between on-device AI and cloud AI is becoming an industry trend.
50%UnverifiedWhen on-device AI runs, data does not leave the device, which protects privacy, reduces latency, and eliminates the need for a network connection.
50%