Carmack and Sutton's First Paper: How Physical Atari Challenges Real-Time Reinforcement Learning

A Reinforcement Learning Experiment That Crosses the Screen

Keen Technologies, the AI company co-founded by John Carmack and Richard Sutton, has released its first research paper: Physical Atari: A Robust and Accessible Platform for Real-time Reinforcement Learning on Robots. The significance of this paper lies not only in its technical contributions, but also in what it represents — the formal beginning of a new chapter for two legendary figures in the field.

Carmack is the creator of game engines behind Doom and Quake, and a central force in the technical evolution of id Software and Oculus VR. Sutton is one of the founding fathers of Reinforcement Learning (RL) — a machine learning paradigm rooted in operant conditioning from behavioral psychology, where an agent learns to optimize decision-making through continuous interaction with an environment and feedback via reward signals, grounded mathematically in Markov Decision Processes (MDPs). Sutton's co-authored textbook with Andrew Barto, Reinforcement Learning: An Introduction, laid the theoretical foundation for the entire field and remains essential reading for researchers worldwide. Both were jointly awarded the Turing Award in 2024. When the embodiment of "extreme engineering" and "theoretical depth" join forces, the first problem they choose to tackle sends a clear signal.

Why "Physical Atari"?

The Atari game platform — formally known as the Arcade Learning Environment (ALE) — has played an enormously important role in the history of deep reinforcement learning. Introduced by Marc Bellemare and colleagues in 2013, ALE packages dozens of classic Atari 2600 games into a unified benchmark for reinforcement learning research. Its key design principle: use only raw screen pixels as input, game score as reward, and provide no prior knowledge whatsoever. This "minimal assumptions" setup made it an ideal tool for evaluating general-purpose learning ability. DeepMind's DQN (Deep Q-Network), published on the cover of Nature in 2015, was the first to demonstrate that a single architecture could surpass human-level performance across multiple Atari games — igniting the deep reinforcement learning wave. Yet nearly all of these achievements took place in purely simulated, software environments.

"Physical Atari" takes this classic benchmark out of the virtual world and places it in the real, physical one: a robot observes a real game screen with a camera and operates a real controller with a robotic arm, completing gameplay under real-world latency, noise, and mechanical error. The elegance of this design is that it preserves Atari's value as a standardized, comparable benchmark while introducing all the complexity of the physical world.

Bridging the Sim-to-Real Gap

The famous "sim-to-real gap" — the chasm between simulated environments and the real world — is one of the most persistent unsolved challenges in robot learning. In simulation, physics engines offer precise and repeatable state transitions, training can run hundreds of times faster than real time, and there's no risk of hardware wear. But three major categories of discrepancy emerge in the real world: perceptual differences (sensor noise, lighting variation, resolution limits), dynamics differences (motor friction, inertia error, material deformation), and temporal differences (control latency, asynchronous communication). Specifically:

• Camera feeds introduce latency and noise;
• Robotic arms take time to execute actions and carry inherent error;
• Lighting, reflections, and vibration all disrupt perception;
• The agent must make decisions in real time, with no ability to pause and wait for computation.

Existing mitigation strategies include Domain Randomization and System Identification, but neither fundamentally eliminates this gap. These are precisely the barriers that prevent many reinforcement learning results from transferring to real robots. Keen Technologies takes a more direct approach — learn directly on real physical systems and sidestep the transfer problem at its source.

Core Claims: Robustness and Reproducibility

Two keywords in the paper's title — "Robust" and "Accessible" — clearly reveal the research team's priorities.

Robustness means the system must operate reliably under a wide range of real-world disturbances — a prerequisite for moving from a demonstration to a dependable system. Accessibility reflects Sutton's longstanding commitment to scientific rigor: a platform that other researchers can build and validate at low cost is far more valuable in the long run than a one-time impressive demo.

This aligns closely with Carmack's engineering philosophy. He has expressed in multiple venues his dissatisfaction with the AI research culture's tendency to prioritize results over engineering, advocating instead for solid, controllable, and iteratively improvable experimental infrastructure. Using Atari — a task "everyone understands" — as the entry point also lowers the barrier for the research community to engage with and critically evaluate the work.

A Path Toward Artificial General Intelligence

Carmack has publicly stated on multiple occasions that he believes the path to Artificial General Intelligence (AGI) may be more "mundane" than most people expect — not necessarily reliant on massive language datasets and enormous models, but possibly emerging from agents that learn through continuous real-time interaction with the real world, much like animals and infants. This view resonates deeply with Sutton's position.

In his widely discussed essay The Bitter Lesson, Sutton reviewed 70 years of AI research history and put forward a provocative central argument: nearly every AI method that relied on human domain knowledge was ultimately surpassed by general methods that leveraged increased computational scale — chess, Go, speech recognition, and image recognition all bear this out. He argues researchers should focus their energy on general learning methods that scale with compute, rather than hardcoding human intuition into systems. The essay sparked years of debate across the AI community, while reinforcing Sutton's own long-term commitment to learning from experience. Real-world, real-time interaction is the purest source of "experience."

It's worth noting that the Embodied Intelligence emphasized here has deep theoretical roots. Embodied cognition theory holds that intelligence must be grounded in the physical interaction between body and environment — that the body's morphology, sensorimotor loops, and physical constraints are themselves constituents of cognitive ability. In the context of real-time online learning, the agent must simultaneously perceive, decide, and update parameters within a continuously flowing stream of time, with no recourse to offline batch processing. This stands in stark contrast to the "train then infer" paradigm of large language models, and demands far stricter requirements on computational efficiency, memory management, and algorithmic stability — precisely the domain where Carmack's decades of low-level systems engineering experience can shine.

Seen from this angle, "Physical Atari" is not merely a project about robots playing games — it is a minimal testbed for studying real-time, online, embodied reinforcement learning. The core question it seeks to answer: can an agent, operating in real continuous time and relying solely on perception and reward signals, gradually master complex tasks?

Significance and What to Watch For

For the broader AI community, the symbolic significance of this first paper may be as important as its concrete technical contributions. It signals that Keen Technologies has not followed the large language model wave, but has instead committed to a differentiated path focused on embodied intelligence and reinforcement learning.

Several directions are worth watching closely:

• Whether the platform will be open-sourced, truly delivering on the promise of "Accessible";
• What levels of sample efficiency real-time learning can achieve under physical constraints;
• Whether this framework can scale beyond 2D games like Atari to more complex real-world manipulation tasks.

Regardless of the eventual outcomes, the Carmack–Sutton pairing itself warrants sustained attention. One is an engineering master who has pushed complex systems to their absolute limits; the other is a theoretical titan who defined an entire research field. The direction they are jointly betting on may well provide an important counterpoint to the mainstream for the next phase of AI research.

Key Takeaways