bonsai-8b
An 8-billion-parameter ternary 1-bit large language model developed by PrismML. It can run on a CPU with only about 2 GB of memory, requires no GPU, and is suited to low-power local inference scenarios.
Timeline (last 90 days)
BitRun v0.2 can run PrismML's 8-billion-parameter model bonsai-8b entirely in CPU memory, using about 2.08 GB of memory, without needing a discrete GPU, a cloud API, or compilation.
Running the bonsai-8b model on a MacBook Pro (M4 Pro CPU, 12 cores, 24 GB memory), the generation speed is 21.0 tok/s, active power is 11.7 W, and energy per token is 0.886 J.
All Facts (2)
BitRun v0.2 can run PrismML's 8-billion-parameter model bonsai-8b entirely in CPU memory, using about 2.08 GB of memory, without needing a discrete GPU, a cloud API, or compilation.
50%UnverifiedRunning the bonsai-8b model on a MacBook Pro (M4 Pro CPU, 12 cores, 24 GB memory), the generation speed is 21.0 tok/s, active power is 11.7 W, and energy per token is 0.886 J.
50%