PPMem: Building Persistent 3D Memory for Video World Models with Plane Primitives

To generate and explore large-scale environments, video world models must maintain a persistent 3D memory of what has already been generated. How to represent this memory as compactly as possible is a key bottleneck in current research. PPMem, from academia, offers an ingenious answer: using a set of oriented plane primitives as persistent 3D memory. Each primitive requires only 7 parameters, yet can capture the dominant structural elements of indoor and outdoor scenes—such as walls, floors, facades, and ceilings.

Why Plane Primitives?

Previous 3D-aware world models mostly rely on per-pixel point clouds to maintain scene consistency. While versatile, point clouds incur storage costs that grow linearly with the number of frames and resolution—often hundreds of MB or more—and lack explicit geometric semantics. Plane primitives naturally align with the statistical regularities of man-made environments: most indoor and outdoor scenes consist of a small number of large planar surfaces. With about 100,000 primitives and only ~5MB, a complete scene can be represented—three orders of magnitude more compact than per-pixel point clouds.

PPMem's cleverness lies in not treating plane primitives as a post-processing artifact, but rather having the video diffusion transformer jointly predict plane geometry during generation. Specifically, a plane prediction head and a Depth Token Fusion (DTF) module are added to the diffusion backbone. Each autoregressive video chunk outputs plane geometry parameters alongside the generated RGB frames; the two are merged into persistent plane memory, which then conditions subsequent video chunks via rendered depth. This means the memory is not a static asset built once, but adaptively refined as generation progresses.

Diffusion Transformer (DiT) is one of the mainstream backbones for current video generation models. It replaces U-Net with a Transformer as the denoising network, splitting video frames into patch sequences and modeling attention along the temporal dimension. PPMem attaches a plane prediction head in parallel to the DiT backbone, meaning that during the denoising of each autoregressive video chunk, the same set of features is used both to reconstruct RGB and to regress plane parameters (such as plane normal, distance from origin, confidence, etc.). The benefit of this multi-task joint prediction is that geometric signals can back-constrain appearance generation, preventing geometrically self-contradictory frames; the cost is the need for additional supervision signals and loss terms to balance the two tasks.

Autoregressive video chunk refers to the model generating video chunk by chunk in temporal order, feeding each generated chunk (or its memory representation) as a condition into the next. PPMem's persistent memory is updated in a rolling fashion within this loop: planes predicted for a new chunk are merged and refined with historical planes, then fed back as rendered depth to the next chunk. This mechanism requires the memory to be incrementally updatable rather than a static product of offline reconstruction.

Four Key Properties of Persistent Memory

The plane memory constructed by PPMem has four practical properties that constitute its core difference from existing methods:

• Compactness: About 100K primitives, 5MB to represent the entire scene—three orders of magnitude smaller than per-pixel point clouds, making memory overhead for long video generation manageable.
• Explicit geometry: The memory carries metric depth, surface normals, and exportable planar surfaces rather than implicit features, and can be directly used for downstream geometric tasks.
• Cross-view consistency: Planes are jointly predicted by a shared backbone, so the same structure naturally aligns across different views, avoiding jitter from independent per-frame estimation.
• Adaptivity: The merging process continuously refines memory as generation proceeds; early coarse planes are corrected in later chunks.

These four properties are not independent but mutually supportive: because geometry is explicit, cross-view consistency can be effectively constrained during merging; because memory is sufficiently compact, long-sequence autoregressive generation does not become unstable due to context bloat.

Metric depth refers to depth expressed in real-world physical units (meters), rather than affine or scale-invariant depth that only carries relative ordering. For world models, metric depth is a prerequisite for interactivity and navigation—only by knowing whether a wall is 3 meters or 30 meters away can an agent make reasonable movement decisions. PPMem's plane primitives inherently carry orientation and distance to the origin, naturally forming metric geometry, so their depth output can be directly used in downstream tasks without additional scale alignment steps.

Surface normals and exportable planar geometry describe the interface form of the memory. Each plane primitive is defined by a few parameters such as normal direction and offset; mesh vertices, depth maps, and normal maps can all be analytically rendered from this set of parameters. This "parametric → differentiable rendering" pipeline allows geometric supervision to be backpropagated end-to-end, and enables cross-view consistency constraints to be applied directly in parameter space rather than expensive matching in pixel space.

Experimental Performance: DL3DV and RealEstate10K

On the DL3DV and RealEstate10K datasets, PPMem outperforms previous 3D-grounded world models in video quality while jointly outputting metric depth, surface normals, and exportable planar geometry. More noteworthy is long-range consistency: under large viewpoint changes, PPMem maintains coherence over hundreds of frames of generation—a challenging metric for models relying on autoregressive chunk-by-chunk generation.

From a technical perspective, PPMem represents a shift in world model memory design—from "implicit feature caching" to "structured geometric memory." Point clouds and implicit neural representations each have their applicable boundaries, and plane primitives, by selectively sacrificing fine-grained representation of non-planar objects, gain order-of-magnitude storage savings and a more interpretable geometric interface. For video world models that need long-term exploration and repeated revisits to the same region, this trade-off is reasonable.

DL3DV and RealEstate10K are the two most commonly used benchmarks for evaluating 3D perception and view synthesis. DL3DV provides multi-view videos of large-scale real indoor and outdoor scenes with wide viewpoint spans and complex lighting; RealEstate10K comes from real estate videos and features continuous camera trajectories in indoor corridors and rooms. Together they cover two types of challenges—"large viewpoint changes" and "long-trajectory continuous generation"—and are therefore often used to test whether a world model can maintain geometric consistency when revisiting the same region, rather than merely generating a single good-looking frame.

Limitations and Outlook

The plane assumption also brings clear boundaries: for scenes dominated by non-planar elements such as vegetation, cloth, and fluids, the expressive power of plane primitives is limited and may require hybrid use with point clouds or Gaussian representations. In addition, the 7-parameter single-plane representation still has room for improvement in fitting accuracy for curved surfaces and thin structures.

PPMem's value lies more in providing a clear idea—world model memory need not be a pixel-level faithful copy, but can be a structured abstraction aligned with scene statistics. Along this line, how to design more general primitive combinations and how to enable incremental memory updates during interactive exploration are directions worth pursuing. The project homepage is now public, and code and more visualization results can be further consulted.

One-line Summary

PPMem uses about 100,000 7-parameter plane primitives as persistent 3D memory for video world models, three orders of magnitude more compact than point clouds.

Paragraph Summary

PPMem proposes using oriented plane primitives as persistent 3D memory for video world models. Each primitive requires only 7 parameters; about 100,000 primitives and 5MB can represent a complete scene—three orders of magnitude more compact than per-pixel point clouds. Its key design is to have the video diffusion transformer jointly predict plane geometry while generating RGB: through a new plane prediction head and Depth Token Fusion module, plane parameters output by each autoregressive video chunk are merged into persistent memory, which then conditions subsequent chunks via rendered depth, allowing memory to adaptively refine as generation progresses. The memory has four properties: compactness, explicit geometry, cross-view consistency, and adaptivity. On DL3DV and RealEstate10K, PPMem achieves better video quality than existing 3D-grounded world models and maintains coherence over hundreds of frames under large viewpoint changes. The authors also note the expressive boundaries of the plane assumption for non-planar scenes such as vegetation, cloth, and fluids.