Dragonfly 2.5 Deep Dive: How P2P Technology Accelerates AI Model Distribution

Dragonfly: From Image Distribution to AI Model Acceleration

In large-scale cluster environments, pulling images or models is a pain point nearly every ops team has encountered: when hundreds or thousands of nodes simultaneously fetch the same image from the origin, the source bandwidth gets saturated instantly. Some nodes finish quickly while the majority are stuck in a long-tail wait. Dragonfly was built to solve exactly this problem — it's a P2P-based file distribution system that now accelerates distribution of OCI images, AI models, and more.

According to Broadcom engineer and Dragonfly Maintainer Chengyu Zhang, the project graduated from CNCF last year, signaling it has reached production-ready maturity. Community contributors come from companies including Ant Group, Alibaba Cloud, and ByteDance. The companion sub-project Nydus, based on the RAFS-format filesystem, focuses on accelerating container cold starts — compressing startup time from minutes down to seconds, which is especially valuable for AI post-training, Agent, and similar scenarios.

Three Core Use Cases

Dragonfly and Nydus cover three typical scenarios. For image acceleration, there are three combinations: using Dragonfly alone for container image distribution (ideal for large-scale cluster pulls), pairing Dragonfly with Nydus for both large-scale distribution and fast cold starts, or deploying Nydus standalone when only cold-start performance matters. File distribution is Dragonfly's original strength, supporting HTTP, HDFS, and S3-compatible storage like S3, OBS, and OSS. AI workloads represent its newest frontier — both training and inference benefit from faster distribution.

Architecture Breakdown: Manager, Scheduler, and Peers

Dragonfly's component model is straightforward. The Manager is analogous to K8s' API Server, handling request validation and authentication. The Scheduler is the system's "brain" — comparable to the K8s scheduler, but instead of scheduling pods, it schedules pieces (Dragonfly's unit of download granularity), deciding which nodes each piece should be fetched from. Seed Peers are typically high-performance machines responsible for pulling from the origin, functioning as in-cluster CDN nodes, while other Peer nodes pull from them and exchange data with each other.

Downloads happen in three scenarios: on the first download with no cluster cache, the scheduler directs a Peer to pull from the origin; on subsequent downloads, if the local Peer has a cache hit, it serves the data directly; if the local node doesn't have it but other Peers do, the file can be downloaded in parallel from multiple Peers, avoiding an origin fetch.

The core idea behind P2P (peer-to-peer) file distribution is that every downloading node also acts as an uploading node. Files are split into small chunks (pieces), and nodes exchange different chunks with each other, spreading bandwidth load from a single origin across the entire cluster. In Dragonfly, a piece is the smallest scheduling unit, typically around 4MB by default. When deciding "where to fetch a given piece," the scheduler weighs factors like how many nodes hold that piece, each node's available upload bandwidth, and network topology (e.g., same datacenter). The goal is to keep traffic local, only falling back to the origin when the cluster has no cache at all. This design means the Nth node to download often finishes faster than the 1st, since earlier nodes have already accumulated pieces that can be shared.

Key Updates in Version 2.5

Version 2.5 brings a series of improvements focused on usability and stability. Most notably, the dfget command now supports downloading models directly from Hugging Face and ModelScope — users simply specify the model URL, and Dragonfly automatically reads the file list (weights, config files) and pulls everything to the cluster.

Dragonfly Injector is a new project launched in the second half of the year, focused on improving the user experience. Previously, users had to manually download CLI tools or mount Unix sockets by hand. Injector automates this via a sidecar: it automatically injects the required binaries, configs, and socket. It works through K8s' webhook admission mechanism — when a Pod annotation containing dragonfly-injector=true is detected, an init container is automatically injected to copy dfget and other tools into a shared volume and mount the Unix socket, making Dragonfly accessible inside the container via 127.0.0.1. Enabling it is as simple as toggling injector enable at install time.

Blocklist addresses real-world emergency mitigation needs. When hotspot data or cluster overload triggers a "thundering herd" effect, you can identify the bandwidth-hogging tasks and configure interception — GRPC calls return permission denied, HTTP calls return 403 Forbidden. This also serves security use cases, such as blocking downloads of files with known vulnerabilities or supply-chain compromises. In addition to Console and API configuration, a GitOps-friendly ConfigMap-based dynamic configuration option is also available.

Adaptive Rate Limiting fills the gap left by static rate limits. Traditional fixed upload/download/proxy rate values can't respond to runtime conditions, but adaptive rate limiting automatically adjusts based on memory, CPU, and other resource metrics, better reflecting actual system load.

Additionally, a new dfcontrol tool — similar to kubectl — lets you list tasks on a node, delete local cache, and pre-warm files or images to the cluster. At the proxy level, a new proxyAllRegistry switch allows Dragonfly to automatically infer the upstream registry from containerd's ns parameter, eliminating the need to configure hosts.toml for each individual registry. Under the hood, large file transfers, scheduling coordination (parent selector, piece collector), and gRPC parameters have all been tuned.

K8s' webhook admission mechanism (admission controllers) is an extension hook in the API Server: when the cluster receives a request to create or modify a resource, it first forwards the request to a registered webhook service, where external logic decides whether to allow (Validating) or modify (Mutating) the request before proceeding. Dragonfly Injector uses a Mutating Admission Webhook: it intercepts Pod creation requests before the Pod is actually scheduled to a node, checks the annotations, and if dragonfly-injector=true is found, automatically appends an init container and volume definitions to the Pod spec. This approach is completely non-intrusive to application containers — developers don't need to change their images or deployment scripts. Just add a label, and Dragonfly's full toolchain is ready when the Pod starts.

Distributing Large AI Models with OCI Image Conventions

As models like Kimi K2 exceed 2TB in size, inference engines serving those models must download all weights before they can serve traffic — making single-point downloads painfully slow. Dragonfly's approach is to reuse the mature software delivery paradigm.

In traditional workflows, developers commit code, triggering CI/CD: build a binary, package it as an OCI image, push to a registry, then distribute via P2P to the cluster. The AI model workflow is nearly identical — just replace "build binary" with "bundle model weights and configs into an OCI image." The key benefit: any registry that supports container images can store model images, and you immediately gain image scanning, signing, versioning, and access control capabilities.

When packaging, each safetensors file becomes its own layer, and config files each get their own layer. Tools like ModelKit can generate the OCI image (newer Docker versions also support this). Using Harbor as an example, model images gain full capabilities: versioned control (v1/v2/v3), lifecycle management, cross-datacenter replication, RBAC access control, signing for supply-chain protection, and audit logging.

On the deployment side, Dragonfly enables data-compute separation. The OCI Image Volume mechanism introduced in K8s 1.31 decouples inference engines (like vLLM) from models — in the Pod's volumes, you reference the model via the image field's reference, and K8s has containerd or CRI-O download and mount it at startup. This feature became a stable, default-on feature in K8s 1.33 (note: the original talk mentioned "1.36" verbally; readers should refer to official docs for the actual release timeline). For older clusters that can't upgrade, Dragonfly also provides a Model CSI Driver option, deployable with a single command. Combined with Harbor's pre-warming policies, pushing a model can automatically trigger distribution to all nodes, significantly reducing first-start latency.

OCI (Open Container Initiative) is a container standardization body co-founded by Docker, CoreOS, and others, responsible for the Image Spec and Runtime Spec. An OCI image is essentially a stack of tar archives, each layer representing a filesystem change, with layers linked by SHA256 digest chains — making content-addressable deduplication a built-in property. Packing AI model weights as individual layers means that when a model updates from v1 to v2, only the layers with changed weight files need to be re-downloaded; unchanged layers are served from local cache. This is critical for incremental updates to multi-hundred-GB models. safetensors is the safe tensor serialization format introduced by Hugging Face. Compared to pickle-based .bin files, it contains no executable code, making it inherently resistant to supply-chain attacks — and it has become the de facto standard for model distribution.

Snapshotter for Agent Workloads and a Lighter Deployment Option

For AI Agent scenarios, the new Snapshotter project enables snapshot and restore of Agent sandboxes. It provides a Go SDK that snapshots files inside a Sandbox and stores them in the Dragonfly cluster, then leverages P2P to accelerate restoring those snapshots across multiple nodes. It exposes just two APIs externally — Snapshot and Restore — while managing metadata, storage, and GC internally. Snapshot data flows two ways: metadata goes to an OCI Registry, and actual data goes into the P2P network. During restore, the local CAS is checked first; on a miss, data is fetched and reconstructed via P2P.

Another significant update is the lightweight Dragonfly deployment mode. The previous deployment required multiple components including Manager, MySQL, and Redis. The lightweight version removes heavy dependencies like databases, keeping only the Scheduler, Seed Client, and Client, with Clients using K8s Service mechanisms to auto-discover their corresponding Scheduler. This dramatically reduces deployment complexity, making it suitable for resource-constrained clusters or edge nodes — a single helm install is all it takes. The tradeoff is reduced functionality; for the full feature set, Redis and Manager are still required.

CAS (Content-Addressable Storage) is a mechanism that uses the hash of data content as its unique identifier for storage and retrieval. Unlike path-based addressing (locating files by name and directory), CAS enables natural deduplication — files with identical content, regardless of which snapshot they came from, are stored only once at the storage layer. When Dragonfly Snapshotter checks the local CAS during a Restore, it's verifying whether the target node already has the same data blocks. A cache hit means direct reuse, significantly reducing P2P network transfer. This is especially valuable for AI Agent scenarios: large numbers of similar Sandbox environments (e.g., different Agents fine-tuned on the same base image) will have very high duplication rates at the snapshot level, and CAS compresses actual storage and transfer costs to the theoretical minimum.

What's Coming in Version 2.6

Version 2.6 is planned for release on December 31st. The roadmap spans several areas: Manager will continue CPU/memory optimization and UI improvements; the Scheduler will keep refining its scheduling algorithm to push bandwidth utilization even further; the Client will explore bandwidth-aware scheduling and RDMA support. On the AI side, optimizations targeting model and dataset distribution (such as Direct IO writes and Buffer IO reads) are planned, along with a new Python SDK (including a Snapshot SDK), with exploration into integration with Agent Sandboxes and environment engineering. On the quality front, AI-assisted troubleshooting capabilities are being considered.

From its origins in image distribution to its current focus on AI models and Agent workloads, Dragonfly demonstrates how a graduated CNCF project can continue evolving in response to new demands — using mature P2P and OCI ecosystem capabilities to tackle the distribution bottlenecks of the large model era.