NVMe SSD
A solid-state drive using the NVMe protocol, characterized by high capacity, relatively low cost, and fast read/write speeds. In AI inference, NVMe SSDs are used to extend model storage capacity as an alternative to expensive GPU memory and system RAM.
Core Facts
Timeline (last 90 days)
Janas-style streaming loading schemes depend heavily on NVMe, and on SATA drives the random read latency is so high that inference speed drops to an unusable level.
The approach of separating compute from storage—using DAS with mechanical hard drives for cold data and NVMe SSDs for hot data and the system drive—controls costs while preserving upgrade flexibility.
The random read latency of modern NVMe SSDs has dropped to under 100 microseconds, which is a key prerequisite for the Solid series' boldness in replacing Redis with a database.
In a storage-compute separation architecture, data access requires crossing network boundaries, and even within the same availability zone, round-trip network latency is one to two orders of magnitude higher than that of local NVMe SSDs.
128GB DDR5 memory can fully load all weights of a 70B-class model, but at a speed of approximately 2–5 tokens/s.
All Facts (6)
Mainstream NVMe SSDs can achieve sequential read speeds of 3500–7000 MB/s.
75%Verified128GB DDR5 memory can fully load all weights of a 70B-class model, but at a speed of approximately 2–5 tokens/s.
65%UnverifiedJanas-style streaming loading schemes depend heavily on NVMe, and on SATA drives the random read latency is so high that inference speed drops to an unusable level.
50%UnverifiedThe approach of separating compute from storage—using DAS with mechanical hard drives for cold data and NVMe SSDs for hot data and the system drive—controls costs while preserving upgrade flexibility.
50%UnverifiedThe random read latency of modern NVMe SSDs has dropped to under 100 microseconds, which is a key prerequisite for the Solid series' boldness in replacing Redis with a database.
50%UnverifiedIn a storage-compute separation architecture, data access requires crossing network boundaries, and even within the same availability zone, round-trip network latency is one to two orders of magnitude higher than that of local NVMe SSDs.
50%