GeoThinker-F: A New Framework That Frees Multimodal LLMs from External 3D Encoders for Spatial Intelligence

The Bottleneck of Spatial Intelligence: External 3D Encoders Slow Down Inference

Multimodal large language models (MLLMs) have become quite mature at understanding 2D information such as images and text. But to truly understand three-dimensional space—judging object distances, predicting motion trajectories, and reasoning about occlusion relationships—they often need an additional 3D geometry encoder attached. This extra inference path introduces noticeable latency and compute overhead: every time a spatial query is processed, the model must run the encoder network one more time.

According to a research summary from the RSS source, GeoThinker-$F$ is a solution proposed specifically for this pain point. Its core idea is not to make the MLLM accurately reproduce the features of some 3D encoder, but to let the model learn to "internally" predict the spatial representations it needs. In other words, the model no longer relies on an external expert to provide geometric priors in real time; instead, it generates this information itself within its internal hidden states.

3D geometry encoders are a class of neural network modules specialized in extracting three-dimensional spatial information from images, point clouds, or videos. Common forms include depth estimation networks based on RGB-D input (such as Depth Anything), point cloud processing networks (such as PointNet++), and neural radiance field (NeRF) encoders based on multi-view geometry. They are usually pretrained separately on large-scale 3D datasets and output dense spatial feature maps or voxel representations for downstream models to use.

A typical approach to attaching such encoders in an MLLM is to inject the geometric features output by the 3D encoder into the LLM's hidden states through a projection layer or cross-attention. The cost of this "plug-in" architecture is that the encoder must be fully run on every forward pass, and the parameter count and computation of a 3D encoder are often comparable to or even larger than the LLM backbone itself. More critically, the encoder's output is a fixed geometric prior that cannot be dynamically adjusted according to the query content, resulting in wasted compute. GeoThinker-$F$ attempts to solve precisely this architectural redundancy.

Two-Stage Decoupled Training: Learn to Predict First, Then Learn to Fuse

In the design of GeoThinker-$F$, the most noteworthy aspect is its training strategy. The entire training process is split into two stages rather than being jointly optimized end to end.

Stage One: Freeze the Backbone and Learn Latent Frame Prediction

In the first stage, the MLLM backbone remains frozen, and only lightweight Latent Frame Prediction (LFP) heads are trained. These prediction heads receive two supervision signals simultaneously: one is the spatial features provided by an offline 3D geometry expert, and the other is the semantic target of the next frame.

The key insight here is that the MLLM's intermediate semantic representations themselves contain enough information to predict useful representations of the current geometry as well as future semantic changes. The model does not need to exactly copy the output of the 3D encoder; it only needs to learn predictions that are "useful for downstream reasoning."

Latent Frame Prediction (LFP) draws on the "world model" research tradition in self-supervised learning, such as the Dreamer series and JEPA (Joint Embedding Predictive Architecture). Its core idea is: rather than predicting future frames at the pixel level, predict future abstract representations in latent space—this avoids interference from irrelevant details while forcing the model to learn high-level semantics useful for decision-making.

In GeoThinker-$F$, the LFP head is designed as a lightweight module that takes the hidden state of a certain intermediate layer of the MLLM as input and outputs two prediction targets: first, the feature vector of the offline 3D expert at the corresponding location; second, the semantic representation of the next frame. The first target teaches the model to "infer back" geometric information from semantic features, while the second ensures that the predicted spatial representation contains temporal dynamics. Under the joint constraints of the two supervision signals, what the LFP head learns is not a simple imitation of the 3D encoder, but a compressed representation oriented toward downstream tasks. The difference between this design and knowledge distillation is that distillation aligns output distributions, whereas LFP aligns intermediate representations and allows the model to encode spatial information in whatever form is convenient for it.

Stage Two: Remove the Expert and Jointly Optimize the Fusion Module

The second stage is even bolder—it directly removes the external 3D expert, lets the model rely on the predictive ability learned in the first stage to generate spatial priors on its own, and uses a dedicated Spatial-Grounded Temporal (SGT) module to selectively integrate these self-predicted priors.

This "learn first, then wean off" strategy means that the inference stage does not need to load any external 3D encoder at all, fundamentally eliminating the extra computational path.

The role of the Spatial-Grounded Temporal (SGT) module is, after removing the external 3D expert, to selectively integrate the "self-predicted spatial priors" generated by the LFP head with the current semantic representations of the MLLM backbone. "Spatial grounding" means allowing each token representation of the language model to establish a correspondence with a specific spatial location; "temporal fusion" emphasizes that the module needs to handle cross-frame spatial consistency at the same time.

The implementation of the SGT module typically involves gating mechanisms or cross-attention: the model needs to determine whether the current query requires invoking the self-predicted spatial priors, and to what extent it should trust these priors. This selective integration is crucial—because the predictions of the LFP head are not always accurate, and blind injection could introduce noise. The gating design allows the model to dynamically switch between pure semantic reasoning and spatially enhanced reasoning. From an architectural perspective, the parameter count of the SGT module is far smaller than that of a complete 3D encoder, which is one of the main sources of the 3.6x speedup. This two-stage strategy of "first relying on an external expert, then running independently" can also be viewed as a self-distillation paradigm for intermediate representations.

Performance: 3.6x Speedup Without Sacrificing Accuracy

According to the data reported in the research summary, GeoThinker-$F$ maintains competitive performance on multiple spatial intelligence benchmarks while achieving significant efficiency gains:

• Inference speed improved by up to 3.6x
• Latency reduced from 1.45 seconds to 0.40 seconds
• Runtime completely free of dependence on 3D geometry encoders

This result means that spatial reasoning capability no longer has to come at the cost of high inference expense. For application scenarios requiring real-time response—such as robot navigation, AR/VR interaction, and autonomous driving perception—the value of this kind of efficiency improvement is especially prominent.

Implications for Spatial AI Architecture Design

The idea of "internally predicted world representations" proposed by GeoThinker-$F$ points to a broader architectural question: do multimodal models really need dedicated external encoders for every perceptual modality?

The traditional approach tends to be "add what is missing"—if spatial understanding is insufficient, add a 3D encoder; if temporal understanding is insufficient, add a video encoder. But the experiments of GeoThinker-$F$ suggest that the intermediate representations of an MLLM already contain sufficiently rich semantic information, and usable spatial priors can be extracted through lightweight prediction heads. If this idea is validated and can be generalized to other perceptual dimensions, it will greatly simplify the architectural complexity of multimodal systems.

Another point worth considering is the "depend first, then become independent" paradigm in the two-stage training. This strategy lets the model first learn to predict under the guidance of an external expert, and then cut off the dependency and run independently. In essence, it is a variant of knowledge distillation, but the object being distilled is not the final output—it is "predictive capability" itself.

Knowledge distillation was proposed by Hinton et al. in 2015 and originally referred to training a small student model using the soft labels of a large teacher model. Its core assumption is that the probability distribution output by the teacher contains richer inter-class similarity information than hard labels. Since then, the method has evolved into many variants, including feature distillation (aligning intermediate-layer representations), relational distillation (aligning relationships between samples), and self-distillation (knowledge transfer between different layers of the same model or different training stages).

The difference between the two-stage training of GeoThinker-$F$ and classical distillation is noteworthy: in traditional distillation, the teacher model exists throughout training, whereas here the teacher (the 3D expert) is directly removed in the second stage; traditional distillation aligns outputs or features themselves, whereas here it aligns "predictive capability"—what the model learns is how to generate spatial representations, not the specific values of the representations. This idea of "capability distillation" is gradually attracting attention in embodied intelligence and world model research, because it is closer to the way humans learn spatial sense: not by memorizing the exact coordinates of every object, but by acquiring an internal mechanism for inferring spatial relationships.

Limitations and Points Still to Be Verified

One additional note: the currently public information comes only from the research summary and lacks the experimental details found in a full paper. For example, key information such as the specific architecture design of the LFP head, the implementation of the SGT fusion module, which specific benchmarks were used for evaluation, and which baseline methods were compared against has not yet been disclosed. In addition, under what hardware conditions and input scales the 3.6x speedup was measured is also worth further confirmation.

For researchers focused on spatial intelligence and efficient multimodal inference, the core contribution of this work—replacing external encoders with internal prediction—provides a valuable direction, but its generalization ability and practical deployment effects still require more experiments to verify.

One-line Summary

GeoThinker-F replaces external 3D encoders with internal prediction, achieving a 3.6x inference speedup without sacrificing accuracy.

Paragraph Summary

To address the high latency caused by relying on external 3D encoders for spatial reasoning in multimodal large models, GeoThinker-F proposes letting the model internally predict the spatial representations it needs. Its core is a two-stage decoupled training approach: in the first stage, the MLLM backbone is frozen and a lightweight latent frame prediction head is trained using signals from an offline 3D expert and the next frame's semantic target; in the second stage, the external expert is removed and a spatial-grounded temporal fusion module is jointly optimized, so that inference no longer depends on a 3D encoder at all. Reported experiments show up to a 3.6x improvement in inference speed, with latency reduced from 1.45 seconds to 0.40 seconds, while maintaining competitive performance on multiple spatial intelligence benchmarks. This work points toward simplifying multimodal architectures: MLLM intermediate representations already contain enough information to extract spatial priors, so there is no need to equip every perceptual dimension with a dedicated external encoder.