RT-2
A Robotic Transformer model developed by Google DeepMind that unifies visual-language understanding and robotic arm control within a single neural network, functioning as an embodied agent capable of interpreting natural language instructions and translating them into physical actions.
Core Facts
VLA models integrate visual perception, language understanding, and robot action planning into a unified model, typically using a pretrained large vision-language model as the backbone, with representative works including Google's RT-2 and OpenVLA.
Google DeepMind's RT-2, Tesla's Optimus, and Figure AI are representative explorations in the direction of Embodied AI.
DeepMind developed the VLA (Vision-Language-Action model) system, which treats the robot's physical actions on equal footing with visual tokens and language tokens.
Timeline (last 90 days)
Typical representatives of action models are vision-language-action models (VLA), such as RT-2 and OpenVLA, which usually use a pretrained multimodal large model as the backbone and, after fine-tuning on robot data, directly output joint positions, end-effector trajectories or discrete action tokens.
RT-2 is jointly trained on internet-scale image-text data and robot demonstration data, giving the model a certain degree of semantic generalization and commonsense reasoning ability.
In the field of robot foundation models, NVIDIA's GR00T series already exists, and in the open-source community, VLA models such as RT-2 and OpenVLA are iterating rapidly.
Representative VLA works include RT-2, OpenVLA, π0, etc.
Representative VLA architectures include Google's RT-2 and Physical Intelligence's π0; the core challenge of VLA lies in the fact that VLM autoregressive decoding speed is lower than the 10–50 Hz real-time frequency required by the robot control loop.
Google DeepMind's RT-2 and π0 (pi-zero) are representative VLA implementations.
RT-2 fine-tunes vision-language large models such as PaLM-E into controllers that output robot actions, demonstrating zero-shot transfer capabilities across tasks.
Representative works in VLA models include Google's RT-2, OpenVLA, and π0.
Google's accumulation in 3D world understanding stems from its long-term robotics research (such as the RT-2 series) and its experience in 3D mapping with Google Maps.
RT-2, PaLM-E, and OpenVLA are robot control frameworks based on pretrained models.
20 more timeline events
All Facts (20)
VLA models integrate visual perception, language understanding, and robot action planning into a unified model, typically using a pretrained large vision-language model as the backbone, with representative works including Google's RT-2 and OpenVLA.
75%VerifiedGoogle DeepMind's RT-2, Tesla's Optimus, and Figure AI are representative explorations in the direction of Embodied AI.
65%VerifiedDeepMind developed the VLA (Vision-Language-Action model) system, which treats the robot's physical actions on equal footing with visual tokens and language tokens.
65%VerifiedRepresentative works in the field of embodied intelligence include Stanford University's Mobile ALOHA, Google's RT-2, and the humanoid robot project developed through OpenAI's collaboration with Figure.
65%UnverifiedGoogle DeepMind's RT series includes RT-1, RT-2, and RT-X: RT-1 trains foundational manipulation capabilities, RT-2 links VLMs with robot actions, and RT-X achieves stronger generalization through cross-platform data sharing.
70%UnverifiedRT-2 (Robotics Transformer 2) directly uses a vision-language model to generate robot control instructions end-to-end.
70%UnverifiedRT-2, released by Google DeepMind in 2023, was the first to validate the feasibility of fine-tuning robot control on a 55-billion-parameter vision-language model, with generalization capability nearly doubling compared to its predecessor.
60%UnverifiedGoogle DeepMind的RT-2首次将视觉-语言模型直接用于生成机器人动作指令
60%UnverifiedTypical representatives of action models are vision-language-action models (VLA), such as RT-2 and OpenVLA, which usually use a pretrained multimodal large model as the backbone and, after fine-tuning on robot data, directly output joint positions, end-effector trajectories or discrete action tokens.
50%UnverifiedIn the field of robot foundation models, NVIDIA's GR00T series already exists, and in the open-source community, VLA models such as RT-2 and OpenVLA are iterating rapidly.
50%UnverifiedRT-2 is jointly trained on internet-scale image-text data and robot demonstration data, giving the model a certain degree of semantic generalization and commonsense reasoning ability.
50%UnverifiedRepresentative VLA works include RT-2, OpenVLA, π0, etc.
50%UnverifiedRepresentative VLA architectures include Google's RT-2 and Physical Intelligence's π0; the core challenge of VLA lies in the fact that VLM autoregressive decoding speed is lower than the 10–50 Hz real-time frequency required by the robot control loop.
50%UnverifiedRT-2 fine-tunes vision-language large models such as PaLM-E into controllers that output robot actions, demonstrating zero-shot transfer capabilities across tasks.
50%UnverifiedGoogle DeepMind's RT-2 and π0 (pi-zero) are representative VLA implementations.
50%UnverifiedRepresentative works in VLA models include Google's RT-2, OpenVLA, and π0.
50%UnverifiedGoogle's accumulation in 3D world understanding stems from its long-term robotics research (such as the RT-2 series) and its experience in 3D mapping with Google Maps.
50%UnverifiedRT-2, PaLM-E, and OpenVLA are robot control frameworks based on pretrained models.
50%UnverifiedCommonly used technical approaches in the industry for controlling robotic arms with large models include direct prompt engineering, Code-as-Action, and specially trained vision-action models (such as RT-2 and π0)
50%UnverifiedOpenVLA, π0, RT-2, and RDT-1B all belong to the "monolithic" VLA model category.
50%