Agentic Engineering: How AI Agents Are Reshaping Physics Simulation and Robotics Development

At an interactive demonstration during SIGGRAPH, NVIDIA technical experts teamed up with a real-time voice AI model named Lexi to showcase a key trend that is reshaping software development: Agentic Engineering.

Agentic Engineering is a systematic methodology that transforms large language models (LLMs) from passive Q&A tools into proactive executors. Unlike traditional software engineering where humans write every line of code, the core of Agentic Engineering is: humans define goals, constraints, and acceptance criteria, while AI Agents autonomously plan execution paths, invoke tools, and iterate corrections until requirements are met. The rise of this concept stems from the convergence of multiple breakthroughs in the second half of 2024—reasoning models gave Agents multi-step planning capabilities, standardized function calling enabled Agents to reliably operate external tools, and long context windows allowed Agents to handle entire codebase-level tasks within a single session.

This demonstration wasn't merely theoretical—through live demos of a real-time web factory viewer and physics simulation applications, it proved that AI Agents can already help engineers build complex 3D world simulations and robotics applications.

This article distills the core insights from the demonstration, analyzes the paradigm shift from "vibe coding" to "agentic engineering," and explores how NVIDIA's Omniverse libraries empower AI Agents for physical AI development.

From Autocomplete to Long-Running Agents: 12 Months of Accelerating Evolution

The presenter summed up the past year-plus of AI programming evolution in one sentence: "It feels like time has been stretched by AI." This acceleration is exponential, not linear.

Looking back at this journey, AI programming has gone through several distinct phases:

• Reasoning: Agents began "thinking" for longer periods before answering. This breakthrough was marked by OpenAI's o1 model (released September 2024). Traditional LLMs begin generating output tokens immediately upon receiving a prompt, whereas reasoning models first conduct internal Chain-of-Thought processing, decomposing complex problems into sub-steps for sequential derivation. This mechanism dramatically improved accuracy on tasks like mathematical proofs, code debugging, and multi-step planning. For AI programming, reasoning capability means Agents no longer just pattern-match to complete code—they can understand code's logical intent, foresee potential bugs, and select optimal approaches from multiple possible implementation paths.

• Tool Use: Agents learned to invoke Bash, CLI, and even 3D tools, enabling them to truly "operate" rather than merely "answer." The technical foundation is the Function Calling protocol—models output structured tool invocation instructions (including function names and parameters), which are executed by an external runtime that returns results to the model. Anthropic's Model Context Protocol (MCP) further standardized this process, providing an open protocol for any tool to interface with Agents in a unified manner. In 3D simulation scenarios, tool use means Agents can directly call scene construction APIs, run physics simulations, and read sensor outputs, forming a complete perception-decision-execution loop.

• Harness (Loop Execution): Frameworks like Claude Code and Codex enable Agents to enter long-cycle loops of "reason → use tools → observe output → continue."

• Multi-Agent Explosion: The past six months have seen multi-agent, multi-day, long-horizon collaborations between powerful models, as well as "ambient models" like NemoClaw that remain continuously online and available for conversation. The core idea of multi-agent systems is to decompose complex tasks among multiple specialized Agents for collaborative completion, similar to role division in software teams. NemoClaw is NVIDIA's continuously online AI assistant built on the Nemotron series of large models. It maintains long-term context, tracks project state across multiple conversations, and responds to developer needs at any time—like a technical partner who's always on call.

Lexi summarized the transition she's most excited about: "Models no longer just answer—they operate. The real shift is from demos to reliable workflows." This statement captures the essence—AI is evolving from "impressive demos" into "dependable productivity tools."

A Fundamental Shift in Workload

From the text editor era of burying yourself in files and agonizing over details, to today's high-level control, the core of work has shifted. The presenter emphasized a key concept: Agents bear the burden of "typing" while humans handle "direction".

Like a good manager, you need to give agents the right instructions, constraints, and goals. Agents don't build "for you"—they build "with you." You remain a partner in the process at all times.

Vibe Coding vs. Agentic Engineering: Trade-offs Between Two Paradigms

Both terms were coined by Andrej Karpathy, and the presenter specifically emphasized that "neither is better"—they suit different scenarios.

Vibe Coding

Vibe coding emphasizes "going with the flow." You provide minimal prompts yet receive massive output—the experience feels magical. But the cost is weak control—the agent makes all decisions and you can't precisely steer. It's suited for exploratory, rapid-iteration scenarios.

Agentic Engineering

Agentic Engineering is a true engineering discipline. You need to:

• Plan and reason about what you want to achieve
• Gather data, write specifications (Specs)
• Provide hard rules (e.g., "must maintain 60 FPS," "load time under 5 seconds," "write tests")

This workflow is controllable, repeatable, and deliverable—just like writing code. While it requires more upfront investment, the payoff is reliability and engineering quality.

The Core: Plan-Change-Test-Verify Loop

The demonstration repeatedly emphasized the importance of the "Verify" step. Many people complain that agents claim to be "done" but things don't actually work—the problem is often that they weren't asked to verify.

Once verification is incorporated into the workflow, agents will self-check whether their work actually runs. If it fails, the agent returns to the planning stage, identifies the problem, and resolves it. This means that by the time results reach you, they've already been tested, are runnable, and functional. This lets you stay in a high-level guiding role without having to hunt down why something broke.

Three Levels of Autonomy

The presenter outlined levels of autonomous agent operation:

1. Prompt: e.g., "Add a button for me"—completes in 10 minutes, quick return to the loop.
2. Spec: Write well-defined PRD documents, gather requirements, set completion gates—agents can run for hours.
3. Long-running: Provide extensive specs and hard rules, build large applications—can run for up to four days.

As Jensen Huang said at Computex: "AI agents are revolutionizing software development." And this revolution is extending into physical AI.

Omniverse Libraries: A Physical AI Toolkit Built for AI Agents

To enable agents to operate in the physical world and simulation environments, specialized tools are needed. NVIDIA's Omniverse libraries are a toolkit designed specifically for AI Agents. The demonstration highlighted five core libraries:

• OV Stage: Brings USD and 3D content into an accelerated space for running simulations. USD (Universal Scene Description) was originally developed by Pixar Animation Studios to solve scene data composition and reuse problems in large-scale collaborative production. It's essentially a description language and file format system for 3D scenes, supporting non-destructive layering, referencing, and variants. In industrial simulation, USD provides a unified scene graph representation that allows different software (CAD, renderers, physics engines, robot control systems) to collaborate on the same data model. NVIDIA positions USD as the HTML of the physical AI era—a universal language for describing 3D worlds. SIM-ready USD refers to assets annotated with physical properties (mass, friction coefficients, joint constraints, etc.) that can be used directly for physics simulation rather than just visual rendering.

• OV RTX: Handles sensor rendering—not just beautiful visuals, but also non-visual sensors like LiDAR and radar.

• OV Physics: Brings GPU compute to physics simulation, modeling complex physical interactions for robots and large factories.

• OV Stream: Decouples client and server rendering/computation, solving local compute limitations through low-latency streaming.

• OV UI: A lightweight, reusable UI library—an evolution of Omni UI from OmniverseKit—supporting cross-platform USD workflows.

Skills: The Agent's "Playbook"

The key design element of these libraries is Skills. Think of Skills as "documentation for agents"—they help agents quickly understand how to optimally use APIs, follow correct patterns, and achieve best performance.

Lexi explained: "Skills are the agent's playbook. They transform raw capabilities into repeatable behaviors, with correct patterns, constraints, and performance expectations built in. This means less trial and error, faster convergence, and safer, more consistent results."

Higher-level Skills (such as "Omniverse Real-time Viewer Skill") teach agents to combine multiple libraries. There are also material agents, physics agents, and others that can assign materials to CAD assets through visual inspection, and automatically assign and test physical properties.

New SIGGRAPH Announcements

Several important updates were announced at this SIGGRAPH:

• Omniverse Libraries in Blender: Integrating physically accurate SIM-ready technology directly into existing applications as a reference workflow implementation.

• NemoClaw on DGX Station: These workstations can locally run near-frontier open-source models like Nemotron 3 Ultra. DGX Station is NVIDIA's desktop-class AI supercomputer for developers and research teams. The latest DGX Spark features the GB10 super chip, shrinking data center-grade AI compute to desktop size. Its core value is enabling developers to run hundred-billion-parameter models locally without relying on cloud APIs. This is crucial for enterprise applications involving sensitive data and real-time interaction scenarios requiring low latency. Local deployment also eliminates API call costs and network latency, allowing Agents to iterate at high frequency without token budget constraints. The presenter offered a brilliant analogy: "These computers come with an engineer built in—the engineer in the hardware knows how to use the hardware."

• Ecosystem integration with partners including Side Effects (Houdini), PTC (Onshape), and others.

Live Demo: From Spec to Real-Time Streaming Application

The demonstration's highlight was two real running applications. The presenter repeatedly emphasized: "These aren't mockups—these are real applications running on the box on the desk."

Web Factory Viewer

The first application was a large factory viewer running in a browser. The entire pipeline loads a SIM-ready large factory USD scene from DGX Spark, organizes it through OV Stage, renders sensors via OV RTX, wraps it with OV UI, and finally streams it to the browser via OV Stream with low latency—all buffers passed within the GPU.

The presenter also shared an important practice: the Design Oracle. Before writing any server-side code, use Claude AI Design or AI agents in Figma to define the application's look and feel. This approach is inspired by the Test-Driven Development (TDD) philosophy of "write tests before code." In traditional development, UI design often isn't validated until late in development, causing extensive rework. The Design Oracle front-loads this process—using AI design tools to rapidly generate visual prototypes and interaction specifications. This prototype becomes the "golden standard" reference during Agent construction. The Agent can compare its output against the Design Oracle, enabling automated UI acceptance testing. This approach moves design decisions from the expensive engineering phase to the cheap prototyping phase, dramatically reducing iteration costs. Modifications at this stage are extremely fast since they're unburdened by testing and engineering frameworks, ultimately becoming the "golden standard image" for construction.

The Importance of Reports

When long-running agents produce a hundred thousand lines of code overnight, how do humans understand what happened? The presenter suggested: Have agents write their own HTML-formatted reports, including timelines, images, and themed designs. This is much more readable than Markdown. He emphasized: "If you're asked what happened in the application and you answer 'I don't know, my agent did it,' that's not engineering."

Robotics Desktop Application

The second demo was a robotics simulation application using OV Physics, streaming from a DGX box in London. The robot arm runs real policy-driven robot control, not game engine keyframe animation. Unlike traditional game engines where animators manually set poses at each time point via keyframe animation, policy-driven robot control uses neural network policies trained through reinforcement learning or imitation learning to drive robot behavior. The policy network takes sensor observations as input and outputs joint torques or target positions, forming real-time closed-loop control. The advantage of this approach is adaptability—robots can handle perturbations and variations never seen during training, rather than rigidly repeating preset action sequences. After validating these policies in NVIDIA's simulation environment, they can be directly deployed to real robot hardware (sim-to-real transfer).

Interestingly, during the construction of this application, the agent proactively exceeded expectations. The presenter hadn't originally asked for a 3D scene to be built, but the agent inferred the user needed one, independently searched the web to find a Franka robot model, and constructed conveyor belt USD files—the entire asset was created autonomously.

Conclusion: Agentic Engineering Is a Discipline, Not a Novelty

As Rev Lebaredian, NVIDIA's VP of Omniverse and Simulation Technology, said: "The next era of AI is physical intelligence—robots and systems that understand and operate in the physical world."

For developers looking to get started with agentic engineering, the action path is clear and practical:

1. Get an agent (Claude Code, Codex, NemoClaw—any will do)
2. Clone samples and have the agent download and learn the Omniverse libraries
3. Start small, but start now—because this train is moving fast

Finally, Lexi's summary deserves to be remembered by every developer: "The faster this field moves, the more agentic engineering becomes a discipline, not a novelty. Give agents tools, verify relentlessly, and maintain a firm grip on human intent and control."

Agents won't do the work for you, but they'll help you realize your vision. And learning to "steer" them is the new requirement this physical AI era places on every engineer, researcher, and student.