Tencent IntelliScene: Multi-Agent Reasoning Drives Game Scene Layout

When players immerse themselves in an open world, they typically see only the carefully constructed tip of the iceberg. Beneath the surface, scene artists must repeatedly weigh engine performance, asset reuse, and narrative consistency. The Tencent Game AI team showcased the IntelliScene project on Bilibili—an exploration of using multi-agent systems and large language model clusters to automate game scene layout. From the pure text reasoning of 1.0 to the visual guidance of 2.0 and the hierarchical placement architecture of 3.0, the project attempts to make AI understand "why an object should be placed there."

The Iceberg Dilemma of Game Scene Construction

Game scene construction follows the iceberg theory: players see the surface of the scene, while behind the scenes lies a complex workflow of collaboration among designers, artists, and programmers—design planning, virtual preview, asset preparation, scene assembly, narrative details, and organization and testing. In open worlds, large areas need to be filled with decorative objects, and this repetitive labor consumes a great deal of artists' time. The team's goal is not to replace humans, but to let AI handle repetitive filling and free up artists' time for core creativity.

Traditional procedural content generation (PCG) is fast but constrained by rules, and players can easily notice patterns after playing for a while; purely data-driven models can learn from 3D data but cannot understand the narrative logic behind layout. IntelliScene chooses a third path: adding a reasoning framework to AI so that agents can divide labor, justify decisions, and use tools like a human team.

IntelliScene 1.0: Multi-Agent Clusters and Slow Thinking

The core of 1.0 is "multi-agent + slow thinking." Human designers follow a slow-thinking process: understand requirements, brainstorm elements, reason about functional circulation and atmosphere, and arrange combinations. The team broke this process down into steps that agents can execute—understand the request, brainstorm the required objects, define attributes such as size and material, decide the relationships between objects, select placement rules, and use optimization tools to compute final positions.

The biggest challenge came from LLMs' natural weakness in precise geometric computation. The team initially tried to have the model directly output coordinates and rotation angles, but the results were unreliable. The adjusted approach lets the LLM do what it is good at: outputting semantic constraints such as "keep a certain distance to the left of B" or "place against the wall," which are then converted into precise coordinates by an optimization solver. Another challenge was hierarchical layout—books on a cabinet, decorations on the books—where outputting all objects at once is unreliable. The team treated this as a recursive problem: first solve the main object, then repeat the layout process within its available internal space, gradually refining and controlling error.

1.0 works in three areas: generating story-rich local scenes based on existing assets, automatically annotating assets for rapid retrieval, and using physics simulation to place assets naturally and believably. In the actual demo, users can box-select an area or click an object and enter "This is a run-down alley; help me pile up new garbage and clutter," and the agent filters relevant objects from a library of tens of thousands of assets, combining narrative to determine type, quantity, density, and orientation. Users can also rearrange locally and keep satisfactory parts while continuing to edit. The team used 1.0 to quickly generate about 110 test scenes and also accumulated high-quality scene data with reasoning, forming a data flywheel.

1.0's limitations are equally obvious: narrative is acceptable for small-scale scenes, but understanding of space and physics is insufficient, geometric precision is low, and text interaction limits workflow intuitiveness. These experiences directly drove the upgrade to 2.0.

Slow Thinking originates from the dual-system theory proposed by Daniel Kahneman in Thinking, Fast and Slow: System 1 corresponds to fast, intuitive, automatic "fast thinking," while System 2 corresponds to slow, deliberate "slow thinking" that requires working memory. In the AI context, "fast thinking" usually refers to one-step reasoning in which a large model directly generates an answer—fast response but lacking global planning; "slow thinking" refers to models explicitly decomposing tasks, reasoning step by step, calling external tools, and verifying intermediate results, represented by work such as Chain-of-Thought (CoT), Tree-of-Thought (ToT), and ReAct-style agent loops. IntelliScene 1.0 borrows this concept to let LLM agents simulate the complete process of a human designer: "understand requirements—brainstorm—reason about attributes—decide relationships—select rules—solve positions," trading more reasoning steps for more reasonable design decisions. The cost is that inference latency and invocation cost are significantly higher than single-pass generation, so the team outsources precise geometric computation to an optimization solver, leaving "slow thinking" responsible only for the semantic-level decisions it excels at.

IntelliScene 2.0: Vision-Knowledge-Driven VRM Agents

The core goal of 2.0 is to use vision-language model (VLM) agents and visual knowledge to solve two major problems: significantly improving aesthetic quality and layout rationality, and achieving more precise control and repair at the image level. Input becomes text, concept art, and a predefined asset library, and output is a 3D scene that matches both the concept art and the text.

The workflow mimics professional artists: first use a fine-tuned Flux-like image model to generate guide images that match the style of the asset library, equivalent to concept art; then perform object detection and segmentation on the image and build a scene graph; finally map to the asset library and run continuous layout optimization. To this end, the team built an asset library of about 500 categories and more than 2,000 high-quality 3D models, each with annotations such as English name, dimensions, description, surrounding space, and whether it can be placed or contained, automatically generated by VLM agents using Blender on Linux. At the same time, they built a dataset of 20 types and 147 high-quality 3D scenes, averaging about 47 models per scene, emphasizing narrative, aesthetic value, and object detail.

Image model fine-tuning borrows from the DreamBooth idea but extends it to full-scene multi-object alignment. By introducing special asset category IDs, hierarchical optimization, and loss constraints, it improves asset retrieval accuracy while avoiding overfitting and maintaining output diversity.

Image-to-3D conversion is performed by the scene construction agent: semantic parsing and detection/segmentation (including gap filling), geometric 3D reconstruction, scene graph construction, asset retrieval, 6D pose matching, and global layout optimization. Pose matching uses cross-screening plus Grounding DINOv2 feature matching, then geometric enhancement with 3D oriented bounding boxes; global optimization takes scene graph relations as hard constraints, uses simulated annealing to search the overall solution, and adds lightweight physics so objects settle and stack naturally.

The team invited 100 art students to evaluate the scenes and asked external senior game artists to review fully textured 3D scenes. The recovery rate for key objects exceeded 90%, and the related paper has been published in C-Graphics. In a small efficiency test with a real game team, the system improved placement speed while maintaining quality.

6D Pose refers to the complete rigid-body transformation of an object in 3D space, including 3 translational degrees of freedom (x, y, z) and 3 rotational degrees of freedom (usually represented by Euler angles or quaternions), for a total of six parameters. In contrast, predicting only bounding boxes or segmentation masks in 2D images gives only planar position and scale, and cannot determine an object's orientation and depth in the 3D world. This is one of the fundamental reasons why many "image-in, scene-out" methods produce objects that float, intersect, or face unreasonable directions. In IntelliScene 2.0, 6D pose matching is responsible for aligning retrieved assets to the 3D scene reconstructed from the image; 3.0 further adopts cross-modal hierarchical pose prediction, refining from coarse position to fine rotation level by level, and internalizes this capability into dedicated model weights. Game engines (Unity, Unreal) ultimately need exactly the 6D transformation matrix of each asset, so 6D pose accuracy directly determines whether layout results can be deployed directly into the engine without manual rework.

IntelliScene 3.0: Hierarchical Placement Architecture and Dedicated Models

3.0 splits scene asset generation and placement into two layers: the top layer is the design brain of a VLM agent cluster, which understands the current scene, breaks down steps in combination with story goals, divides regions, and translates complex designs into executable instructions; the bottom layer is a placement domain-specific model focused on physical and spatial execution, generating precise 6D poses, avoiding collisions, and ensuring assets are natural and stable in the 3D engine. The two layers collaborate through an interaction mechanism similar to MCP, forming a closed loop of planning, execution, and feedback.

To address three common limitations of existing methods—input accepts only 2D images, output lacks complete 6D poses, and weak spatial reasoning leads to floating and collisions—3.0 proposes three key designs: 3D-aware input encoding, which uses encoded point clouds and metadata of scenes and assets as model input; cross-modal hierarchical pose prediction, maintaining efficiency while improving accuracy from coarse to fine; and training on high-quality placement datasets to internalize spatial reasoning and asset planning into model weights.

Comparison results show that 3.0 can correctly place objects on the floor and choose more reasonable orientations in scenes such as bathrooms, offices, and kitchens, is more stable in complex rooms, and has a clear speed advantage: when the number of assets increases from 5 to 20, the VLM agent approach takes more than 200 seconds, while 3.0 stays under 25 seconds.

MCP (Model Context Protocol) is a standardized protocol open-sourced by Anthropic at the end of 2024, used to let large model applications connect to external tools, data sources, and services in a unified way, often compared to the "USB-C interface for AI applications." Its core is to define a context exchange format between clients and servers, enabling agents to discover tools, call tools, and receive structured return results without writing a separate adapter layer for each tool. When IntelliScene 3.0 mentions that the two layers use an "MCP-like interaction mechanism," it means that the top-layer VLM design agent and the bottom-layer placement-specific model collaborate through an agreed message and tool-calling interface: the upper layer issues region division and semantic placement instructions, the lower layer returns 6D poses and collision detection results, and the upper layer then revises the plan based on feedback, forming a closed loop of plan—execute—feedback. This approach decouples "the model that thinks" from "the model that executes," allowing each to iterate and be replaced independently.

From Scene Layout to AI-Native Games

The team summarized the core lessons from three years of work: LLM agents plus slow thinking can decompose design tasks, but control and precision are limited; visual guidance combined with high-quality scene data, vision models, and physics solvers can achieve more accurate scene generation; multi-agent collaboration is effective for complex tasks; and continuously collecting high-quality scene data is the long-term foundation. Future directions include more open scene generators, deep-research-style design workflows, and end-to-end game content generation. The speaker also introduced the game S.I.World as an AI-native game producer, demonstrating an attempt to let players generate a playable RPG with a single sentence—scene layout automation may be just the prelude to this larger transformation.

One-line Summary

Tencent IntelliScene automates game scene layout in three stages using multi-agent and large model clusters.

Paragraph Summary

The IntelliScene project showcased by the Tencent Game AI team attempts to automate game scene layout using multi-agent systems and large language model clusters, freeing artists' time from repetitive filling work. Version 1.0 uses LLM multi-agents plus "slow thinking" to decompose the design process, letting the model output semantic constraints rather than coordinates, which are then converted into precise positions by an optimization solver, and handles hierarchical layout recursively; 2.0 introduces fine-tuned image models to generate concept art as visual guidance, combining scene graph construction, asset retrieval, and 6D pose matching, with a key object recovery rate exceeding 90%; 3.0 splits design understanding and placement execution into two layers, and through 3D-aware input encoding and hierarchical pose prediction, still stays within 25 seconds when the number of assets increases to 20. The team also built an asset library of 500 categories and 2,000+ models and a dataset of 147 high-quality 3D scenes to support continuous iteration of the data flywheel.