ProtoDiffusion: Compressing Multi-Layer Image Generation with Prototype Tokens

ProtoDiffusion compresses multi-layer design representations with a shared prototype plus one vector per layer, natively generating layer structures.
The article introduces ProtoDiffusion and its companion autoencoder ProtoVAE, which natively generate multi-layer RGBA designs in a diffusion model. Existing approaches allocate a full-canvas spatial latent grid per layer, requiring N×H×W tokens. ProtoVAE shares one spatial prototype across layers and keeps one candidate vector per layer, cutting tokens to HW+N while retaining 40.43 dB PSNR. ProtoDiffusion jointly denoises the prototype and a fixed set of optionally empty candidate slots, needing no layer count input, with bounding boxes as optional layout guidance, supporting both image and text conditioning.
The Core Challenge of Multi-Layer Design Generation: Representation Grows Linearly with Layer Count
The essence of Graphic design is the stacking of RGBA layers—each layer is independently selectable, movable, and reusable, which is exactly the foundation of a designer's workflow. But in generative models, representing a design as "multi-layer" is expensive: existing approaches allocate a full-canvas or region-sized spatial latent grid to each layer, so the representation volume grows linearly with the spatial stacking range of layers. An N-layer design occupies N×H×W tokens, and the more layers and the larger the canvas, the heavier the compute and memory pressure.
A new study from RSS, ProtoDiffusion, proposes another route: instead of storing a grid per layer, all layers share a single spatial prototype, with each layer retaining only one candidate vector. This design compresses the representation of an N-layer design to HW+N tokens (before backbone-specific packing), a difference of orders of magnitude compared with the NHW of the full-grid approach.

Spatial latent grid refers to the 2D feature map obtained after an autoencoder compresses an image, where each position in the spatial dimension corresponds to a region of the original image (e.g., 8×8 or 16×16 pixels), and each position carries a channel vector. The token count is H×W (after flattening by patch). For multi-layer designs, storing one grid per layer amounts to retaining a complete canvas feature for each layer, so N layers stack to N×H×W. This is the key to understanding the paper's motivation: the value of design layers lies in "independent editability," but if a generative model copies this structure directly, it converts editing granularity directly into memory and attention compute costs, because Transformer attention complexity is superlinear in token count. The compression idea in this paper is precisely to remove the redundant assumption that "every layer has an independent spatial structure."
ProtoVAE: Shared Prototype + Per-Layer Candidate Vectors
This representation comes from the companion autoencoder ProtoVAE. It encodes a design into a shared spatial prototype plus one candidate vector per layer; each candidate vector conditions the same shared decoder to reconstruct the corresponding RGBA layer. The shared spatial structure is carried by the prototype, while differences between layers are expressed by lightweight candidate vectors—essentially separating "what is common across layers" from "what is unique to each layer" into separate models.
The key benefit of this separation is scalability. Since the representation grows only by one fixed-length vector per layer rather than an entire spatial grid, ProtoVAE still achieves 40.43 dB PSNR on layer cutouts, showing that this compact representation does not sacrifice reconstruction fidelity. For downstream generation tasks, this means the model can handle all layers simultaneously in a latent space that does not expand significantly.
PSNR (Peak Signal-to-Noise Ratio) is a common metric for reconstruction quality, measured in dB, where a higher value indicates a smaller per-pixel error between the reconstructed and reference images. It is computed as the square of the maximum possible pixel value (usually 1 or 255) divided by the mean squared error, then log-transformed, so every ~3 dB increase roughly corresponds to halving the error. In autoencoder literature, above 30 dB is generally considered visually good reconstruction, and the 35–40 dB range approaches high quality. The reported 40.43 dB is measured under the layer cutout setting, meaning that the compact representation of a shared prototype plus one vector per layer still preserves layer details (including boundaries and alpha edges). This is the prerequisite for the representation to support downstream generation—if reconstruction itself loses too much, the latent distribution learned by the diffusion model would also be distorted.
Joint Denoising: Generating Prototype and Candidate Slots Together
ProtoDiffusion performs the diffusion process in this compact latent space: it jointly denoises the shared prototype and a fixed number of candidate slots, conditioned on either images or text, with bounding boxes as optional layout guidance.
Several design details are worth noting:
- Slots can be empty: the model decides on its own which candidate slots are empty, so the number of layers does not need to be provided as input—the model determines it itself.
- Layout can be inferred: even when no bounding boxes are provided, the model can infer the layer layout; boxes serve only as optional constraints, not required inputs.
- One architecture, two tasks: the same model supports both image-to-design and text-to-design, with no need to design separate networks for the two directions.
This "fixed slots + can-be-empty" mechanism turns the problem of variable layer counts into a denoising problem over a fixed slot set, avoiding the trouble of designing different network structures for different layer counts.
Joint denoising in diffusion models means processing multiple sets of latent variables simultaneously in the same denoising network, allowing them to perceive each other within the same step. Unlike "generate independently then concatenate," joint denoising allows the prototype and candidate slots to exchange information through attention, thereby maintaining spatial alignment between layers during generation. Candidate slots are a preset number of learnable slots, each responsible for carrying the candidate vector of one layer; designing slots to be optionally empty is equivalent to letting the model output a judgment of "whether this slot is valid," so variable layer counts are absorbed into a fixed-size tensor without padding or bucketed training. This is also why the architecture can support both image and text conditions: the condition only affects the denoising input, while the structure of the slot set remains unchanged.
Measured Performance: Reconstruction, Layout, and Text Generation
On the LayerD benchmark, the RGB L1 score for image-to-design is 0.040 with bounding boxes and 0.057 without—the benefit of layout guidance is clearly visible, but even without boxes the model still produces usable results. With box guidance, alpha soft IoU is 0.850, comparable to the strongest fine-tuned baseline (0.871), indicating that the compressed representation does not noticeably lose points on the detail of layer transparency.
The text-to-design comparison better illustrates the architectural advantage. On merged images, ProtoDiffusion achieves an FID of 66.41 versus 86.84 for FLUX.2 [klein]; the text accuracy score reaches 0.860; and it is 19.1× faster than the same-backbone "generate-then-decompose" cascade approach. It should be emphasized that the baseline in this comparison is a cascaded pipeline—generate the full image first, then decompose layers—whereas ProtoDiffusion generates directly in the multi-layer representation, eliminating the decomposition step, which is the main source of the speed gap.

FID (Fréchet Inception Distance) measures the distance between the generated and real distributions: features are extracted with an Inception network, Gaussian distributions are fitted to the two feature sets, and the Fréchet distance between them is computed—lower is better. It penalizes both poor quality and insufficient diversity, but it is sensitive to the feature extractor and sample size, so absolute values across papers should not be compared directly; only relative comparisons under the same evaluation setting are meaningful. The 66.41 vs. 86.84 comparison in the paper should be interpreted under the LayerD evaluation protocol. Alpha soft IoU is a soft intersection-over-union for the layer transparency channel, measuring the overlap between the predicted alpha mask and the ground truth, ranging from 0 to 1; closer to 1 means more accurate restoration of layer boundaries and transparent regions, and it reflects "whether a layer can be cleanly selected" better than RGB metrics.
Why This Route Is Worth Watching
Compressing the layer representation from "one grid per layer" to "shared prototype + one token per layer" fundamentally changes the complexity structure of the multi-layer generation problem. Existing cascaded approaches (generate the full image → decompose layers) accumulate errors across two stages, whereas ProtoDiffusion jointly models all layers within a single model, with spatial alignment between layers naturally constrained by the shared prototype.
Of course, the boundaries of this method also come from its assumptions: a shared prototype means all layers share the same spatial structure prior, and for designs whose layers differ greatly in spatial distribution, whether this sharing remains sufficient requires more scenario validation beyond the reported metrics. A fixed number of candidate slots also means that designs with very many layers need to rely on slot reuse or expansion strategies.
From an engineering perspective, the 19.1× speedup and lower representation overhead bring native generation of multi-layer designs one step closer to practical production workflows—what designers receive is not a flat image that needs manual splitting, but a layer structure that can be directly selected, moved, and reused.
One-line Summary
ProtoDiffusion compresses multi-layer design representations with a shared prototype plus one vector per layer, natively generating layer structures.
Paragraph Summary
The article introduces ProtoDiffusion and its companion autoencoder ProtoVAE, which natively generate multi-layer RGBA designs within a diffusion model. Existing approaches allocate a full-canvas spatial latent grid per layer, requiring N×H×W tokens for an N-layer design and growing linearly with layer count and canvas size. ProtoVAE instead shares one spatial prototype across all layers and keeps only one candidate vector per layer, reducing token count to HW+N while still achieving 40.43 dB PSNR on layer cutout reconstruction. ProtoDiffusion jointly denoises the prototype and a fixed number of optionally empty candidate slots in this compact latent space, requiring no layer count as input, with bounding boxes as optional layout guidance, and the same architecture supports both image- and text-conditioned generation.
Related articles

Making Marketing Videos with Claude Code Is More Efficient Than AI Video Generation
Generate marketing videos programmatically with Claude Code—more flexible and cheaper than AI video generators. Reference a style, generate a storyboard, output content as code, and make edits instantly.

Open-Weight Models Are Falling Behind: Three Structural Reasons
A veteran developer uses his platform's usage data and public benchmark experiments to argue why open-weight models are falling behind closed-source frontier models.

Is AI Becoming a Climate Disaster? A Deep Dive into the Carbon Cost of Compute Expansion
AI compute expansion is driving dual energy pressure from training and inference, while data centers face challenges in power, cooling, and water. This article examines AI's climate impact and possible sustainable paths for the industry.