Diffbot
An AI and knowledge graph technology company that provides web data extraction and knowledge graph construction services. It has also contributed engineering work on optimizing MoE model inference.
Timeline (last 90 days)
Diffbot's approach moves 4,608 of the 15,360 routed experts, plus lookup tables (Engram tables int4 and embedding), entirely to host memory, with the GPU reading them over PCIe via UVA, reducing the footprint per card to 72.4 GiB.
The working solution relies on a combination: DeepSeek provides the weights, coolbho3k does the 3-bit quantization, turboderp wrote the EXL3 format and the ExLlamaV3 inference library, Victor Cruz wrote the vllm-exl3 bridge layer, Diffbot designed the per-expert split scheme, startup flow and kernels, and InferenceKey did the integration and measurement.
All Facts (2)
Diffbot's approach moves 4,608 of the 15,360 routed experts, plus lookup tables (Engram tables int4 and embedding), entirely to host memory, with the GPU reading them over PCIe via UVA, reducing the footprint per card to 72.4 GiB.
50%UnverifiedThe working solution relies on a combination: DeepSeek provides the weights, coolbho3k does the 3-bit quantization, turboderp wrote the EXL3 format and the ExLlamaV3 inference library, Victor Cruz wrote the vllm-exl3 bridge layer, Diffbot designed the per-expert split scheme, startup flow and kernels, and InferenceKey did the integration and measurement.
50%