M2 Max
A high-performance system-on-chip (SoC) from Apple with a unified memory architecture, commonly used in MacBook Pro and Mac Studio.
Timeline (last 90 days)
On an M2 Max, using Llama-3.1-8B-Instruct (Q4_K_M quantization), Magnitude's throughput is 38.7 tok/s, and Ollama's is 18.2 tok/s.
llama.cpp 0.3.0 leverages the unified memory feature via the --n-gpu-layers parameter, and on an M2 Max a 7B model can offload 35 layers.
The Mac Studio is equipped with M-series chips, such as the M2 Max, M2 Ultra, and subsequent updated versions.
All Facts (3)
The Mac Studio is equipped with M-series chips, such as the M2 Max, M2 Ultra, and subsequent updated versions.
60%UnverifiedOn an M2 Max, using Llama-3.1-8B-Instruct (Q4_K_M quantization), Magnitude's throughput is 38.7 tok/s, and Ollama's is 18.2 tok/s.
50%Unverifiedllama.cpp 0.3.0 leverages the unified memory feature via the --n-gpu-layers parameter, and on an M2 Max a 7B model can offload 35 layers.
50%