Llama-3.1-70B
A large-scale open-source language model released by Meta with 70 billion parameters, intended for high-performance inference tasks.
Timeline (last 90 days)
An 8192-token prompt on a 70B model produces 2.68 GB of KV cache in BF16.
Under the above assumptions, an 8192-token prompt's transfer time is about 30% of prefill time on a 100GbE link, about 15% at 200Gb, and about 7% at 400Gb.
Lucebox splits the draft model and the target model to run on different hardware; in the official example, Llama-3.2-1B runs as the draft model on a second GPU or the CPU, and Llama-3.1-70B runs as the target model on the main card.
All Facts (3)
An 8192-token prompt on a 70B model produces 2.68 GB of KV cache in BF16.
50%UnverifiedUnder the above assumptions, an 8192-token prompt's transfer time is about 30% of prefill time on a 100GbE link, about 15% at 200Gb, and about 7% at 400Gb.
50%UnverifiedLucebox splits the draft model and the target model to run on different hardware; in the official example, Llama-3.2-1B runs as the draft model on a second GPU or the CPU, and Llama-3.1-70B runs as the target model on the main card.
50%