Conceptoffloading
Weight Offloading
A technique that moves some layers or experts of a model from GPU memory to slower storage tiers such as system memory or disk, swapping them in on demand at runtime. It reduces GPU memory usage at the cost of inference speed.
Timeline (last 90 days)
Oct 23
Weight offloading refers to moving some of the model's layers or experts from VRAM to slower storage tiers such as memory or NVMe drives, swapping them in on demand at runtime, which sacrifices inference speed.
Unverified50%