OctLLM
A 3D large language model approach that represents three-dimensional geometry through explicit octree occupancy token sequences and introduces 3D capabilities via independent parameter branches without disrupting pre-trained language pathways, balancing parameter efficiency, language capability retention, and 3D expressiveness.
Timeline (last 90 days)
The arXiv paper 'Octrees as an Explicit 3D Language' proposes OctLLM, targeting two pain points of 3D large models: compressed geometric information and fine-tuning that damages language capabilities.
OctLLM replaces compressed implicit representations with explicit octree occupancy token sequences, and introduces 3D capabilities via independent parameter branches without disrupting the pretrained language pathway.
OctLLM designed a Sparse Octree (S-Octree) that randomly clears nodes at the second-to-last layer and omits their descendants, retaining shape contours to shorten the sequence.
OctLLM routes mesh tokens to independent trainable branches in a subset of Transformer blocks, while text and image tokens remain in the frozen vision-language pathway, with the two data streams interacting through a shared self-attention mechanism.
OctLLM achieves a new state of the art (SOTA) within the category of unified multimodal large models with significantly fewer training parameters.
Compared to ShapeLLM-Omni, OctLLM reduces the FID metric for image-to-3D generation by 17.4%.
Compared to ShapeLLM-Omni, OctLLM improves render-grounded captioning by 28.7 points.
As a newly released preprint, OctLLM's generalization capability on larger-scale data and more complex 3D tasks remains to be validated by follow-up work and community reproduction.
All Facts (8)
The arXiv paper 'Octrees as an Explicit 3D Language' proposes OctLLM, targeting two pain points of 3D large models: compressed geometric information and fine-tuning that damages language capabilities.
50%UnverifiedOctLLM replaces compressed implicit representations with explicit octree occupancy token sequences, and introduces 3D capabilities via independent parameter branches without disrupting the pretrained language pathway.
50%UnverifiedOctLLM designed a Sparse Octree (S-Octree) that randomly clears nodes at the second-to-last layer and omits their descendants, retaining shape contours to shorten the sequence.
50%UnverifiedOctLLM routes mesh tokens to independent trainable branches in a subset of Transformer blocks, while text and image tokens remain in the frozen vision-language pathway, with the two data streams interacting through a shared self-attention mechanism.
50%UnverifiedOctLLM achieves a new state of the art (SOTA) within the category of unified multimodal large models with significantly fewer training parameters.
50%UnverifiedCompared to ShapeLLM-Omni, OctLLM reduces the FID metric for image-to-3D generation by 17.4%.
50%UnverifiedCompared to ShapeLLM-Omni, OctLLM improves render-grounded captioning by 28.7 points.
50%UnverifiedAs a newly released preprint, OctLLM's generalization capability on larger-scale data and more complex 3D tasks remains to be validated by follow-up work and community reproduction.
50%