Carmack × Sutton: Teaching AI Robots to Play Atari in the Real World

A Heavyweight Duo Behind One Paper

A paper titled Physical Atari: A Robust and Accessible Platform for Real-time Reinforcement Learning on Robots has sparked widespread discussion in the AI community. The attention it's drawn comes not only from its technical content, but from its origin: Keen Technologies — a new AI research team co-founded by legendary game programmer John Carmack and reinforcement learning pioneer Richard Sutton.

For anyone familiar with AI history, this pairing is nothing short of dream-team material. John Carmack is the core developer behind DOOM and Quake, co-founder of id Software, and renowned for his extreme engineering precision and low-level optimization wizardry. He pioneered ray-casting rendering on consumer hardware, invented the fast inverse square root algorithm, and has long championed code minimalism — accomplishing the most complex tasks with the fewest layers of abstraction. After leaving Meta in 2022, Carmack stated in interviews that he believes the core obstacle to AGI is algorithms, not compute, and that he favors a "small but sharp" experimental approach.

Richard Sutton, meanwhile, is a textbook-level figure in reinforcement learning. His co-authored book Reinforcement Learning: An Introduction (with Andrew Barto) is the classic entry point for countless researchers, and earned him the Turing Award. The two joining forces in AI research is, in itself, a landmark moment for the field.

From Simulators to the Real World

Why Atari Still Matters

Since DeepMind released DQN (Deep Q-Network) in 2013, Atari games have served as the standard proving ground for reinforcement learning. DQN was a milestone for deep RL — the first to combine deep convolutional neural networks with Q-learning, surpassing human-level performance across 49 Atari games and demonstrating that a single algorithm could learn vastly different tasks. Its key innovations — Experience Replay and Target Networks — effectively addressed the training instability that arises when combining deep learning with RL. Since then, a string of major algorithms including Rainbow, PPO, and A3C have all been validated on this "universal exam," building a decade's worth of research momentum.

The classic Arcade Learning Environment (ALE) allows researchers to test learning algorithms in a unified, reproducible setting. But virtually all of these experiments run inside simulators — the AI reads game memory states or rendered frames directly, and action commands take effect instantaneously as digital signals.

The core breakthrough in this paper is bringing Atari into the physical world. "Physical Atari" means placing a robot in a real physical environment, where it observes a real screen through a camera and manipulates a real game controller with a mechanical device — all in real time. This is fundamentally different from a pure software simulator.

The Challenges of the Real World

Moving from simulation to physics may seem like just adding a physical "wrapper," but it introduces a host of problems that have long been deliberately sidestepped:

• Visual noise and latency: Camera-captured images are subject to lighting changes, blur, and glare — far from the pixel-perfect frames of a game renderer.
• Real-time constraints: The physical world won't wait for an algorithm to finish computing. The agent must complete a full perception–decision–execution loop within a strict time window.
• Execution uncertainty: Mechanical controller operation involves physical delays and errors, meaning actions may not take effect precisely as intended.

These are the genuine obstacles in real-world robot learning — and exactly what simulation environments have long failed to reflect. This is known as the Reality Gap, one of the most central challenges in robot learning: policies trained in simulators often degrade significantly on real robots, due to factors like physics engine accuracy, sensor noise, and actuator hysteresis. To bridge this gap, the research community has developed methods like Domain Randomization and Adaptive Transfer, but the cost and reproducibility barriers remain high. Physical Atari's potential contribution lies in offering a relatively affordable, reproducible physical benchmark that allows more researchers to validate algorithms directly in the real world. The paper's emphasis on being both "Robust" and "Accessible" reflects exactly this intent.

The Deeper Significance of Real-Time Reinforcement Learning

Sutton's Long-Held Vision

This work aligns closely with the research philosophy Richard Sutton has championed for years. Sutton has consistently placed "experience" at the center of intelligence, arguing that agents should learn in real time through continuous interaction with their environment — not through batch training on massive offline datasets. His recently promoted "Alberta Plan" is a roadmap for AGI built around continual learning as its core principle.

Unlike the dominant pretraining-and-fine-tuning paradigm, the Alberta Plan emphasizes that agents should continuously update themselves through ongoing environmental interaction, rather than relying on a one-time training pass over a static dataset. This vision directly challenges the technical trajectory of large language models, arguing that true intelligence requires online, incremental accumulation of experience. The primary technical challenge in continual learning is Catastrophic Forgetting — neural networks tend to overwrite previously learned knowledge when acquiring new tasks. Recent work in this space includes Elastic Weight Consolidation (EWC) and Progressive Neural Networks, but a significant gap remains before reaching biological levels of continual learning.

Deploying reinforcement learning on physical robots in real time is a direct embodiment of this philosophy. It forces algorithms to confront the irreversibility of time — you can't pause the world to think, and you can't replay the same second ten thousand times for training.

Engineering and Theory, Deeply Combined

This also explains why Carmack's involvement is so critical. Real-time robot learning places extreme demands on systems engineering — low-latency data pipelines, hardware synchronization, system robustness, and a host of other low-level concerns. The extreme optimization skills Carmack has honed over decades in real-time graphics and game engines are precisely what fills the engineering gaps that purely academic teams often face. His commitment to code minimalism resonates with Sutton's preference for simple, online learning — giving Keen Technologies a distinctly counter-mainstream flavor in today's large-model-dominated AI landscape. The deep fusion of theoretical insight and engineering capability may be exactly the differentiated path this team is trying to forge.

One Piece of the AGI Puzzle

John Carmack has publicly stated on multiple occasions that he left VR to go all-in on AGI research, and that he believes AGI may arrive sooner than most people expect — and require far less code than commonly assumed. He favors starting from simple, verifiable experimental environments and building general learning capabilities incrementally, rather than immediately pursuing the large language model approach.

This path aligns closely with the research paradigm of physical embodiment. Embodied cognition theory holds that genuine intelligence cannot exist apart from the interaction between a body and its physical environment — intelligence is not merely information processing in the brain, but a complete closed loop of perception, action, and environmental feedback. In AI, Embodied AI has attracted significant interest in recent years, with representative work including Stanford's Mobile ALOHA and Google's RT-2, though the barrier to entry for these systems remains high. Physical Atari transplants the most classic purely-software benchmark into a physical robot platform — essentially creating a lower-cost, more reproducible standardized entry point for embodied reinforcement learning.

From this perspective, choosing "Physical Atari" as the team's debut paper is symbolically deliberate: it extends the most iconic benchmark in reinforcement learning history while pushing it toward the physically embodied scenarios that real intelligence may ultimately require. The goal isn't to break score records in a game — it's to test whether an agent can learn autonomously in a real, real-time, uncertainty-filled world.

A Team Worth Watching

It's worth noting that the specific technical details and performance data from this paper are still awaiting broader disclosure and peer review, and community discussion so far has largely centered on the symbolic significance of its research direction. As the debut work from a new team, its true value may only become clear through the series of work that follows.

That said, when a Turing Award winner's theoretical insight meets a legendary engineer's implementation ability, the combination alone is enough to make Keen Technologies' next moves highly anticipated. The AI field has never lacked grand narratives, but teams that can steadily push forward the mission of "making machines learn in the real world" are rare. This paper on Physical Atari may well be a signal worth tracking for the long term.

Key Takeaways