NVMe
A high-speed solid-state storage interface protocol that provides low-latency, high-throughput data access, commonly used for local high-speed disks and training data caches.
Timeline (last 90 days)
After fixing the non-compute overhead (configuring persistent storage, migrating data to local NVMe, caching the base model and tokenizer, and increasing num_workers), the single-run time dropped from 9.5 hours to 2 hours 40 minutes, nearly a 4x speedup compared with the RTX 4070's 11 hours.
After NVMe SSDs became fast enough to respond at the microsecond level, the storage industry adopted polling instead of interrupts, having the CPU actively and frequently check the completion queue, sacrificing idle-time energy efficiency for determinism and throughput at low latency.
The open-source tool galahad-kv saves the KV states of each approximately 16,000-token chunk to an encrypted local NVMe disk, then reloads them byte-accurately, completely skipping recomputation.
Writing to memory is a one-time cost, as the first processing of text still requires full KV state computation; the KV states for 50 million tokens occupy several terabytes of local NVMe disk space.
For NVMe SSDs, each unit in the Data Units Written field of the smartctl output corresponds to 1,000 512-byte sectors, or approximately 512 KB.
NVMe is a communication protocol designed specifically for flash storage that communicates directly with the CPU via the PCIe bus, with latency as low as the microsecond level.
TRIM is a command in the ATA instruction set, corresponding to Deallocate in the NVMe protocol, allowing the operating system to proactively inform the SSD which logical blocks are no longer in use so that the controller can perform garbage collection.
现代存储金字塔中GPU显存(HBM)带宽可达数TB/s但消费级容量通常8-24GB，CPU内存(DDR5)带宽约50-100GB/s，NVMe固态硬盘带宽约5-14GB/s
All Facts (8)
After fixing the non-compute overhead (configuring persistent storage, migrating data to local NVMe, caching the base model and tokenizer, and increasing num_workers), the single-run time dropped from 9.5 hours to 2 hours 40 minutes, nearly a 4x speedup compared with the RTX 4070's 11 hours.
50%UnverifiedAfter NVMe SSDs became fast enough to respond at the microsecond level, the storage industry adopted polling instead of interrupts, having the CPU actively and frequently check the completion queue, sacrificing idle-time energy efficiency for determinism and throughput at low latency.
50%UnverifiedThe open-source tool galahad-kv saves the KV states of each approximately 16,000-token chunk to an encrypted local NVMe disk, then reloads them byte-accurately, completely skipping recomputation.
50%UnverifiedWriting to memory is a one-time cost, as the first processing of text still requires full KV state computation; the KV states for 50 million tokens occupy several terabytes of local NVMe disk space.
50%UnverifiedFor NVMe SSDs, each unit in the Data Units Written field of the smartctl output corresponds to 1,000 512-byte sectors, or approximately 512 KB.
50%UnverifiedNVMe is a communication protocol designed specifically for flash storage that communicates directly with the CPU via the PCIe bus, with latency as low as the microsecond level.
50%UnverifiedTRIM is a command in the ATA instruction set, corresponding to Deallocate in the NVMe protocol, allowing the operating system to proactively inform the SSD which logical blocks are no longer in use so that the controller can perform garbage collection.
50%Unverified现代存储金字塔中GPU显存(HBM)带宽可达数TB/s但消费级容量通常8-24GB，CPU内存(DDR5)带宽约50-100GB/s，NVMe固态硬盘带宽约5-14GB/s
50%