Artificial Intelligence

Developers Build Simulations With Frontier AI Agents and Omniverse

Developers are combining frontier AI models with specialized software libraries to turn simulation ideas into working applications for testing scenarios, investigating failures, and refining designs.

Radwan AbdulhadiEditor and technology writer

Published: October 9, 2026 · Updated: October 9, 2026 · 4 min read

Developers Build Simulations With Frontier AI Agents and Omniverse — Image: NVIDIA Blog

Turning Simulation Ideas Into Working Applications

Turning a simulation idea into a working application requires assembling assets, connecting physics and rendering, and checking that scenes behave as intended. Developers are increasingly combining frontier AI models with NVIDIA Omniverse libraries to help carry out this work, building applications designed for exploring scenarios, investigating failures, and improving designs. Through natural-language instructions, developers direct AI agents, review results, and guide necessary changes while utilizing GPU-accelerated physics, rendering, and sensor simulation capabilities.

Humanoid Warehouse Simulators and Autonomous Driving Workflows

In warehouse automation planning, developers require interactive simulation environments to explore task behavior and evaluate workflows. Frank DeLise, an Omniverse product manager at NVIDIA, used the Astra model to turn a warehouse and humanoid robot into an interactive simulator featuring first- and third-person views. DeLise directed Astra to connect libraries for physics, scene updates, rendering, and the user interface. He also used Astra alongside SimReady components to establish the physical scene in simulation, with the AI agent generating the necessary animation and application code.

In autonomous vehicle development, changes to scenes, sensors, or driving models can significantly affect simulation outcomes. Doyub Kim, a simulation technology team manager at NVIDIA, asked Astra to build a reusable simulation environment based on San Francisco’s Market Street. Kim directed the agent to map out the workflow and connect asset creation, traffic, sensor simulation, and driving models in stages to track how downstream behavior responds to modifications.

Sensor Validation and Digital Twin Creation

To test robots and autonomous vehicles accurately, developers need to understand how closely simulated sensors match real-world data. Ashley Reid, working on RTX sensor validation at NVIDIA, directed Astra and Claude Fable 5 agents to compare camera and raw LiDAR outputs with recorded information. Over a three-day iterative workflow guided by Reid, the agents measured differences, modified OpenUSD scenes, and addressed missing objects or geometry based on specific camera and LiDAR metrics.

Robotics Testing, Computer-Aided Design, and Space Exploration

Teaching robots new movements involves verifying whether actions function under physical constraints. Tae Kim, who leads NVIDIA Omniverse engineering and product, utilized sports videos and natural-language prompts to guide Astra in building Robo Olympics, an experimental project testing simulated humanoids performing sports movements. The project incorporated physics simulation, calculations accelerated by open-source frameworks, and rendering for virtual-camera images to provide actionable feedback on robot timing and control.

In another engineering application, Jens Jebens, a senior product manager for OpenUSD at NVIDIA, directed Astra to model a car suspension in computer-aided design software and configure it within simulation tools. The agent measured available space and designed a custom wrench to help the robot reach suspension bolts, successfully demonstrating component removal in simulation and linking tooling decisions to policy training.

Bringing complex 3D models into browser-based environments also benefited from AI agent workflows. Nic Johns, an engineering director at NVIDIA, prompted Astra to assemble NASA assets into an OpenUSD International Space Station model equipped with live telemetry. Johns built the application through initial prompts and subsequently adjusted scene parameters to display Earth's daytime side, utilizing Blender for asset preparation and Omniverse libraries for runtime rendering and streaming.

Reconstructing Captured Rooms for Interaction Testing

Digitally reconstructed rooms require editable objects and accurate physical behaviors before developers can test interactions effectively. Chirag Majithia from the Isaac engineering applications team directed Astra to transform stereo camera captures into an editable OpenUSD studio environment. The workflow combined reconstruction tools with user reviews for object placement, allowing AI content agents to configure how objects move and interact during collision and contact tests.