Amazon AGI Lab Open-Sources Perception Agents: Teaching AI to Read the Screen

While most AI Agents still rely on API interfaces to complete tasks, Amazon AGI Lab has chosen a path closer to how humans work—letting agents directly "see" the screen. At the AI Engineer conference, the team not only laid out the design philosophy behind Perception Agents, but also open-sourced two core tools and demonstrated a complete closed loop from voice command to code change on stage.

Perception Agents: Computer Use Beyond "Blind Clicking"

Emil Bazel, head of Amazon AGI Lab, positions the team as "a frontier research and product team combining startup speed with Amazon scale," focused on building knowledge-work agents that can perceive, reason, and act across many environments.

Traditional computer use capabilities let agents click buttons on screen and fill out forms, essentially treating the visual interface as an operable coordinate plane. But Emil points out that this is only half the story: "We want the agent to understand what it sees on screen, just like a human does." He offers a vivid example: give anyone a late-1990s Hotmail or Yahoo mailbox interface, and even if they've never used it, they can intuitively figure out how to send an email—this ability to understand interfaces across specific products is exactly what perception agents aim for.

The value of this capability lies in unlocking a large number of work scenarios that "have a screen but no API." A great deal of knowledge work is still done by humans through browsers and desktop software, and perception agents allow automation to extend into these domains.

Nova Act and Two Open-Source Primitives: Annotation and Verification

According to Emil, the team's previously released Amazon Nova Act is a first-party service built on AWS that lets developers drive browser workflows with natural language—filling out forms, searching, extracting data, comparison shopping—with support for human-in-the-loop and tool calling. The console can replay an agent's complete trace, including the bounding boxes of the screen elements it focused on.

The focus of this workshop was two already open-sourced perception agent primitives:

Visual Annotation

This is a Chrome extension that lets users circle, click, or drop pins directly on a web page to annotate change requests. After annotating, clicking export generates a JSON file containing all the change descriptions plus a prompt, which can be handed directly to a coding agent for execution. Emil emphasizes that this tool is independent of model inference, produces fully deterministic output, and is not bound to any model.

Visual Verification

This is a "skill" that can be installed into coding agents such as Claude or Kiro. The team writes design specs and user flows as Gherkin files, and the verification skill drives the browser to check item by item whether the site matches the design's colors, font sizes, and interaction flows, then generates a report. On first run, the tool does a shallow crawl of the site and automatically generates user flows as the baseline for subsequent verification.

Gherkin is a domain-specific language used for behavior-driven development (BDD), describing software behavior with a natural-language-style Given/When/Then structure. For example, "Given the user is on the login page, When they enter correct credentials, Then they should be redirected to the home page." Its value lies in letting non-technical people read and write test scenarios, while also being executable automatically by tools like Cucumber. The Amazon team uses it for visual verification, in effect translating "design specs and user flows" into a machine-readable set of assertions, giving the verification skill a clear basis for judgment rather than relying on the model's subjective assessment of whether a page "looks right."

MCP (Model Context Protocol) is an open protocol proposed by Anthropic for letting AI applications connect to external tools and data sources in a standardized way, often likened to "the USB-C port for AI applications." The Jam MCP mentioned in the article follows this protocol, packaging debugging information such as browser console logs and network requests into tools that an agent can call, then forwarding them to Nova. Understanding MCP helps grasp the extension logic of this workflow: the perception agent itself handles "reading the screen," while MCP is responsible for bringing observable data from beyond the screen into the model—the two are complementary.

From Feedback to Merge: A Workflow Compressed to Hours

Workshop presenter Shruti Arora uses a front-end engineer role-play to tie together the value of the whole toolset. In the traditional workflow, design review, Slack feedback collection, and item-by-item code changes often take weeks; after bringing in a coding agent, this shortens to days. And if feedback can be consumed directly by the agent—annotation completing automatically triggers code changes, which then automatically run visual verification—the cycle can be compressed to "done by the time you're back from a coffee break."

According to the live demo, the full chain requires bridging an "Agent Bridge" between the Chrome extension and the coding agent, and modifying the extension's manifest and content scripts to support local requests and an "Apply changes" button. In the demo, the presenter changes the publish date to the current date and clicks apply; the coding agent then automatically takes over the modification, after which Nova Act's logs show the model's reasoning process and screen decision points in real time, ultimately producing a verification report.

The workshop offers two paths: an easy mode that walks through the two primitives step by step, and an intermediate mode that asks participants to chain the two primitives into a complete closed loop themselves.

When Wearables Join In: Voice Becomes Code Changes

Amazon's Bee wearable provides another input method for this workflow. Team member Giovanni demonstrates connecting Bee's conversation summaries to the perception agent: in a simulated design discussion, saying "change the title to red" or "swap the brain emoji for a robot" causes the conversation transcript and key points to appear in a side panel; clicking apply triggers the site modification and automatic verification.

On-site participants also showed other extension ideas: a Wharton student combined Bee with a self-built GTD (Getting Things Done) project management tool called Sidekick, hoping to automatically file ideas from meetings into the corresponding project; another attendee from a financial company tried using Jam MCP to capture console logs and network requests, forwarding them through an agent to Nova, so that non-technical colleagues could describe interface change requests in natural language.

Reliability: An Agent That Works 50% of the Time Is Effectively Unusable

In his opening, Emil quotes a customer: "If your agent only works 50% of the time, it effectively doesn't work at all." The reason is straightforward—users either don't dare run it, or spend all their time debugging, and the benefits are offset entirely.

To this end, the team invests heavily in reinforcement learning training in simulated environments, with training scenarios drawn from real knowledge-work use cases from external customers and Amazon's internal teams. This also explains the lab's dual role: on one hand, helping early adopters put early models into production; on the other, turning customer feedback into evaluation sets that feed back to the research team for model training.

Human-in-the-loop refers to preserving the ability for humans to confirm, correct, or take over at key points in an agent's execution flow. It is especially important in perception agent scenarios: because the agent directly operates a real interface, a single misclick can have irreversible consequences. Nova Act's support for human-in-the-loop and tool calling is essentially a safety valve between the benefits of automation and the risk of operation. This is also a concrete mechanism behind the team's emphasis on reliability—not pursuing full automation, but having humans step in at necessary points, thereby improving overall task success rates and trustworthiness.

A New Interface for Human-AI Collaboration

The impromptu ideation session at the end of the workshop selected several application concepts, including a "pixel judge," an end-to-end product testing agent, and a tool that lets humans and perception agents collaborate in real time on Canvas to design UIs.

Shruti's core takeaway in her summary: the way humans and agents collaborate is changing, and the visual layer of collaboration tooling is the missing piece. The two primitives open-sourced by the perception agent are precisely the starting point for filling that gap—the code is now open on GitHub, and the accompanying Nova Act SDK and the team's blog post on perception agents have also been published.

For teams looking to bring AI into front-end workflows, this toolset offers a paradigm worth watching: don't make the agent guess at the interface—let it share the same visual understanding layer as humans.

One-line Summary

Amazon AGI Lab open-sources visual annotation and verification primitives, letting agents read the screen and close the loop on code changes.

Paragraph Summary

At the AI Engineer conference, Amazon AGI Lab laid out the design philosophy behind perception agents: letting agents understand screen content like humans do, rather than merely clicking coordinates, thereby unlocking many knowledge-work scenarios that have a screen but no API. The team open-sourced two core primitives—a browser-extension visual annotation tool and a visual verification skill that can be installed into coding agents—both of which do not depend on model inference and produce deterministic results. Together with Nova Act and coding agents, they can compress the cycle from front-end design feedback to code merge from weeks to hours. The Bee wearable further demonstrates a new interaction where voice conversations directly trigger interface changes. The team emphasizes that reliability is a first-class citizen, and an agent that works 50% of the time is effectively unusable.