Kyutai REKI 2.4B: 32 Register Tokens Redefine VLA Robot Policies

Robot manipulation models have long been trapped in a dilemma: either they look only at the current frame with no temporal memory, oscillating back and forth between two poses under occlusion or symmetric scenes, or they stuff multi-frame history videos directly into the vision encoder, trading in memory explosions and unacceptable latency. French lab Kyutai, together with Inria and École des Ponts, has released REKI 2.4B, which implements short-term memory inside the action expert using 32 learnable register tokens—solving the problem. The weights are now available for download on Hugging Face.

Why Physical AI Needs Smarter Memory

Purely digital conversational assistants tolerate hundreds of milliseconds of latency, but industrial robotic arms and humanoid arms require closed-loop control above 40 Hz—no more than 25 milliseconds per cycle. This physical constraint shuts most vision-language-action (VLA) architectures out the door.

Memoryless models act solely on the instantaneous frame, falling into perceptual aliasing under occlusion: when the robot hovers above a bowl about to grasp it, and when it has just put the bowl down, the camera sees nearly identical images, so the policy loops endlessly between advancing and retreating. Meanwhile, concatenating visual tokens from frames t-2, t-1, and t into a 2.2B-parameter VLM backbone triples attention overhead, sending both memory and latency out of control. REKI's choice is to put memory in the lightest component of all.

Perceptual aliasing is a classic problem in robotics and reinforcement learning: when two different physical states are nearly indistinguishable under the current observation, a policy relying only on instantaneous observations will produce contradictory or even looping actions. Typical examples include the similar images of a bowl on the table versus the tabletop left behind after the bowl is picked up, the left and right poses of symmetric objects, and partial views before and after occlusion. Corresponding to the framework of partially observable Markov decision processes (POMDPs), perceptual aliasing essentially shows that a single-frame observation is not a sufficient statistic, and historical information or belief states must be introduced. The "advance–retreat" oscillation common in robot manipulation is precisely the policy repeatedly switching between two high-confidence but mutually contradictory poses, and is also one of the most noticeable failure modes of memoryless VLAs in real deployment.

Architecture Dissection: 2.2B Backbone + 200M Action Expert

REKI employs strict module separation: a 2.2B-parameter vision-language backbone handles static perception, while a 200M-parameter action expert based on Flow Matching generates continuous trajectories. Total parameters: 2.4B.

The memory mechanism design is the key to the whole. The model maintains 32 learned register tokens covering the last 3 seconds of stereo observations, but they live only inside the action expert, reading the history registers of seconds t-1 and t-2 through causal self-attention. The cost is about 2 MiB of GPU memory per frame and a mere 0.1 millisecond increment in the inference loop.

Even more elegant is the attention mask constraint: registers never read action tokens. In Flow Matching's 10-step Euler integration, action tokens are noisy and constantly changing with the time step; if registers depended on them, they would have to be recomputed 10 times per frame. REKI lets registers only cross-attend to the current frame's vision and text representations, so they can be computed once before the Euler loop begins, and during the loop the action tokens query ready-made registers from the cache.

Pretraining Data: Entirely from Public Repositories

REKI's pretraining is built entirely on public datasets, totaling 1 billion frames of visual data and 12.2 million vision-language multimodal samples, trained for 200,000 steps on just 32 H100s. The data composition is as follows:

• Real robot teleoperation (639.8 million frames): from DROID, BridgeData V2, and AgiBot World, covering deformable object grasping, sub-millimeter precision insertion, and articulated manipulation on cluttered tabletops.
• Deterministic physics simulation (193.9 million frames): generated in physics engines with randomized textures, lighting, and contact friction, to defend against out-of-distribution shifts.
• Human first-person manipulation videos (170.9 million frames): from EgoDex and EgoVerse, letting the vision backbone absorb priors on human motion intent and hand-eye coordination before mechanical coupling.

One engineering decision particularly emphasized: during pretraining, each dataset's native control convention is preserved, rather than forcibly unifying end-effector Cartesian deltas and joint-angle spaces through remapping. Each robot morphology enters the action expert through a dedicated state MLP plus adaptive normalization layers (AdaRMSNorm), thereby maintaining clean transfer when fine-tuning on real hardware like the UR5—converging with just 50 demonstrations and 1 hour of training (3000 steps).

Flow Matching is a class of generative modeling methods whose goal is to learn a vector field that continuously transforms a simple prior distribution (usually Gaussian noise) into the target action distribution. During training, noise and real actions are interpolated, and the network regresses the corresponding velocity field; during inference, starting from noise, a numerical integrator (such as the Euler method) advances step by step along the learned velocity field to obtain the final action sequence. Compared with diffusion policies, Flow Matching can often generate high-quality samples with fewer integration steps; REKI uses 10-step Euler integration. This explains the statement that "action tokens are noisy and change with the step count during integration": if registers depended on action tokens, they would have to be recomputed repeatedly across the 10 integration steps; otherwise, they need only be computed once before the loop.

Register tokens were originally proposed in vision Transformers to provide the model with a set of "scratch workspace" tokens that do not directly correspond to input patches, absorbing global information and reducing high-norm artifacts in attention maps. REKI transfers this idea to the action expert: the 32 learnable tokens are not part of the observation but internal memory slots of the model, chaining the last 3 seconds of history temporally through causal self-attention, then cross-attending to the current frame's vision and text representations. Since they belong to no input modality, their number and update scheme can be designed independently of the backbone scale—which is also why sinking memory into the 200M action expert remains effective.

AdaRMSNorm (Adaptive RMSNorm) is a conditional variant of RMSNorm: activations are first normalized by root mean square, then affine-transformed using scale and shift parameters generated by a small MLP from an external conditioning signal (here, the robot morphology's state vector), functioning similarly to adaptive layer normalization (AdaIN) in generative models. In REKI, each robot morphology is injected into the action expert through a dedicated state MLP plus AdaRMSNorm, enabling the same set of pretrained weights to adapt to different kinematic structures such as UR5, Franka Panda, and ALOHA without forcibly unifying the representation of end-effector deltas and joint-angle spaces.

Benchmarks: Smaller Body, Stronger Generalization

REKI goes head-to-head with contemporary mainstream policies on LIBERO-Plus, the four LIBERO suites, RoboTwin 2.0, and RoboCasa365, against opponents including Qwen-RobotManip (4.7B), Physical Intelligence π0.5 (3.3B), AMAP ABot-M0 (4.6B), and RLDX-1.

Three conclusions are worth noting:

• Leading out-of-distribution generalization: achieves 90.4% on LIBERO-Plus, surpassing the 4.7B Qwen-RobotManip (89.0%) and π0.5 (85.7%). Reportedly, public-data-only pretraining alone contributes +13.8 percentage points on Clean2Rand and +30.3 percentage points in heavily cluttered scenes.
• Eliminating pose oscillation: oscillation episode rate on RoboTwin drops from 18.6% to 4.1%, and RoboCasa365 improves by +5.4 percentage points.
• Leading cycle time: compiled with torch.compile at BF16 precision with two 256x256 RGB inputs (top and wrist views), producing a full action chunk takes only 21.8 milliseconds on a single H100—35.5% faster than π0.5 (33.8 ms) and nearly twice the speed of ABot-M0 (46.1 ms).

A Reference Frame Beyond the Data: The Deployment Reality of Hierarchical Physical AI

The original article also presents a comparison framework worth noting: in real industrial deployment, high-level semantic planning and low-level visuomotor control are often split into two layers. The former (cited in the article as GPT-6 Astra, Claude Fable 5.1, DeepSeek^Ad 4.1, and Gemini 3.8 Flash Cyber) decomposes tasks, diagnoses anomalies, and reorders collision constraints at 1–2 Hz; the latter (REKI) translates subgoals into joint velocities and torque tensors at about 45 Hz. The comparative data for such hierarchical stacks is industry evaluation cited in the original article and has not been independently verified by third parties; readers may treat it as a deployment-approach reference rather than a settled conclusion.

Side-by-Side Comparison of Memory Approaches

Memory Approach	Extra GPU Memory	Per-Chunk Latency Impact	RoboTwin Oscillation Rate	Can It Support >40 Hz
REKI: 32 registers inside action expert	~2 MiB/frame	+0.1 ms	4.1%	Yes (~45.8 Hz on a single H100)
Single-frame, memoryless policy	0	~21.7 ms	18.6%	Risky
Multi-frame history injected into VLM backbone	>1400 MiB	+46.8 ms	5.8%	Not feasible

This table explains why sinking memory into the 200M action expert is a worthwhile trade: it buys stability close to the multi-frame approach while paying only latency close to the memoryless approach.

Deployment Path: LeRobot gRPC and MCAP Data Curation

The original article provides a three-step process for deploying REKI checkpoints (kyutai/reki-2b-base, reki-2b-libero, reki-2b-robotwin) onto UR5, Franka Panda, or ALOHA dual-arm platforms:

1. Data curation: use Voxel51 3.0 to ingest synchronized teleoperation logs in ROS 2 / MCAP format (top and wrist camera streams, jointstates topics), and filter for 50 collision-free, hesitation-free demonstrations by temporal segment embedding.
2. Rapid fine-tuning: run 3000 fine-tuning steps while preserving the robot's native control convention, with a proprioception MLP adapting to local kinematics without damaging the 1-billion-frame pretrained weights.
3. Asynchronous inference serving: compile the computation graph with torch.compile (reduce-overhead mode) at BF16, call policy.reset() at the start of each episode to clear the 32 registers, then run action-chunk prediction at 45 Hz over a low-latency gRPC channel.

The original article also includes a complete Python asynchronous controller implementation, including a register ring buffer, AdaRMSNorm state conditioning, 10-step Euler ODE integration, and trajectory stability monitoring, which can serve as an engineering scaffold reference.

License and Applicable Boundaries

The six official checkpoints are released on Hugging Face at BF16 precision under the CC BY-NC-SA 4.0 license: usable for academic research, technical evaluation, scientific reproduction, and non-commercial prototyping; commercial deployment requires separate authorization.

For teams building robot policies, REKI's inspiration lies not in any single impressive number, but in the division of labor it demonstrates: keep the heavy things light, and put memory where it is closest to the action. When the hardware budget is only a few dozen GPUs and data can only come from public repositories, this design choice may be more decisive than stacking parameters.

One-line Summary

Kyutai releases REKI, a 2.4B robot manipulation model that uses 32 register tokens to achieve lightweight short-term memory inside the action expert.

Paragraph Summary

The French lab Kyutai, together with Inria and École des Ponts, has released REKI, a 2.4B-parameter vision-language-action model aimed at resolving the dilemma in robot manipulation between "memoryless perceptual aliasing oscillation" and "multi-frame history injected into the VLM causing memory and latency explosions." REKI uses a modular design of a 2.2B VLM backbone plus a 200M Flow Matching action expert, placing 32 learnable register tokens inside the action expert to cover the last 3 seconds of observations, with only about 2 MiB of GPU memory and 0.1 millisecond latency overhead per frame. It achieves 90.4% success on LIBERO-Plus, cuts the RoboTwin oscillation episode rate from 18.6% to 4.1%, and performs action-chunk inference in just 21.8 milliseconds on a single H100, meeting closed-loop control above 40 Hz. Pretraining uses entirely 1 billion frames of public data, and the weights are openly available on Hugging Face under the CC BY-NC-SA 4.0 license.