GPT-6 Astra Full Benchmark: From Ray Tracing to Auto-Completing Pokémon

The AI Search channel put GPT-6 Astra — billed as the latest flagship model — through a rigorous gauntlet of tests. From physics simulation and game development to computer control and music creation, the model's agent capabilities were genuinely impressive — though it stumbled on a few classic challenges. This article summarizes the key findings and presents both the strengths and weaknesses honestly.

Agentic Coding: Writing a Ray-Traced Physics Simulation from Scratch

The first test went straight at the model's understanding of physics and ray tracing. The prompt asked it to simulate a bullet piercing a water balloon — the balloon bursting, water spraying and deforming, then falling — with support for freeze-frame viewing from any angle, plus sliders for bullet speed, balloon size, lighting, and gravity. The key constraint: no three.js or any external web libraries allowed. Everything had to be hand-coded from scratch.

What's most worth noting here isn't the model itself, but a reusable prompting technique: introducing an independent "reviewer agent." After each iteration, the reviewer agent takes screenshots from different angles and scores the result on realism and physical accuracy (0–10). Only a score of 8 or above passes; otherwise, it generates a prioritized list of issues for the builder agent to address, looping for a set number of rounds.

The actual run took 32 minutes. The first review scored 4.7, the second 6.0, the third 6.5 — never reaching 8, and ultimately stopping when the round limit was hit. Despite the low scores, the final output convincingly showed a bullet penetrating a water mass, with water spraying and falling, and supported adjustable lighting intensity and direction. This "reviewer–builder" dual-agent loop is the core methodology used throughout the entire evaluation.

A Playable Unreal Engine 3D Game in 4 Hours

The harder challenge was game development. The prompt asked it to build a third-person procedural 3D game in Unreal Engine, using a SketchFab character model, automatically finding and mapping walk, run, and jump animations from Mixamo, generating a procedurally created ancient Chinese imperial palace as the scene, and producing assets via Blender MCP.

The model's full workflow was quite complex: logging in and downloading assets using the Playwright extension, connecting to Blender via the MCP protocol to generate assets, procedurally rendering in Unreal Engine, and finally using the reviewer agent to score screenshots from multiple angles (with a 3A-quality threshold of 8.5). The first round alone took 1 hour and 26 minutes, earning scores of just 5.8 and 6.3 — still below target. After multiple follow-up rounds addressing character animation, camera effects, and jump mechanics, the cumulative time reached about 4 hours. The final product was a genuinely playable game where the character could leap across rooftops in sequence and collect icons — though flaws like misaligned building floors and lanterns remained. Overall, the level of completion was impressive.

---

Ray tracing is a rendering technique that simulates photons traveling from a light source through reflections and refractions until they reach the camera. Traditional real-time rendering takes shortcuts with rasterization, while ray tracing calculates the true light path per pixel, faithfully reproducing soft shadows, caustics, subsurface scattering, and other physical phenomena at enormous computational cost. The requirement to use "no external libraries" meant the model had to hand-write everything from vector math and ray-sphere intersection detection to Lambertian diffuse reflection models — a dual test of mathematical reasoning and code synthesis. The automated scoring by a reviewer agent is essentially replacing the human feedback loop (the manual review step in RLHF) with another language model, allowing iteration to continue unsupervised. This "LLM-as-Judge" paradigm has become increasingly common in automated evaluation, though it carries the risk of bias from the reviewer model itself.

---

Computer Control: Painting Stroke by Stroke and Playing Piano

The biggest official selling point of GPT-6 Astra is its "computer use" capability — the ability to directly control browsers and desktop applications through clicking, dragging, and form-filling. The evaluation designed a series of progressively harder tasks.

The most basic was opening Photopea (an online Photoshop alternative), creating a canvas, and copying a painting. Rather than cheating with code, it genuinely drew stroke by stroke with the cursor, finishing in about 20 minutes. The advanced version had it visit a sprite animation website and draw frame-by-frame running, jumping, sword-swinging, and slashing animations for a reference character, then export them as a GIF — taking about 40 minutes.

The most striking test was the piano challenge: open an online virtual piano, compose a fast-paced Chopin-style solo piece, then "perform" it live by pressing keys in sequence with accurate timing. The difficulty lies in the time dimension — key timing must be precise. The model spent a few minutes exploring the recording interface, then played out a genuinely convincing, fast-paced classical solo in about 7 minutes. Completing a time-sequenced task on a real interface like this demonstrates real maturity in agent control.

Music, Video, and 3D Reconstruction

The evaluation also tested breadth of content creation. It was asked to produce a Europop EDM track in Waveform; the first version was basic, but iterated noticeably after feedback. Acting as a "director," it used Higgsfield MCP to orchestrate a video generation model and produce a 30-second product ad — completing it in 20 minutes, with results deemed better than competing models given the same prompt. Another highlight: feeding it all the photos from a Japanese luxury ryokan listing and having it reconstruct the entire building and interior in Blender MCP, then create a virtual tour with a camera path. After 1 hour and 12 minutes, it generated thousands of elements in the scene — not a perfect replica, but the pool, sofa, and living room layout were quite close to the original.

---

Direct control of computer interfaces relies on a "Computer Use" interface that typically perceives the current screen state via screenshots, then outputs mouse coordinates and keyboard events to execute actions — forming a "perceive–plan–act" loop. Unlike API calls or code generation, this approach doesn't require the target application to provide any structured interface, theoretically enabling control of any software a human can use. The piano test is particularly compelling because it introduces a time dimension — the model must not only know which key to press, but trigger click events at the correct moment in time according to the beat. This demands temporal planning at the execution level, not just static UI interaction. MCP (Model Context Protocol) is an open protocol proposed by Anthropic that allows AI models to connect to external tools like Blender and Higgsfield through standardized interfaces — essentially giving the model a set of pluggable "tool arms."

---

Where It Failed: Finding the Frog and Identifying Brain Tumors

The evaluation didn't shy away from the model's shortcomings. In the classic "find the frog" test, the model ran for nearly 7 minutes without successfully circling the hidden frog, even misidentifying it as a rattlesnake; retrying with another image multiple times also failed. This reveals clear blind spots in fine-grained visual search.

In the brain tumor identification test, out of six brain scan images of different types, the model only answered one correctly (a meningioma in the top-left), getting the other five wrong. The reviewer honestly noted that while one out of six was the "highest score" among models tested, it's nowhere near reliable for medical applications. The decision to present these failure cases honestly makes the evaluation far more valuable as a reference.

Benchmark Results and Landmark Achievements

Based on self-reported official data, GPT-6 Astra stands out on several leaderboards:

• Frontier Math: approximately 98%, nearly perfect; Frontier Math (Erdős Benchmark): the only model with a score above 0%
• Terminal Bench 4 (terminal agent coding), OS World, Agents Last Exam, and Automation Bench: all ranked first across agent and computer-use categories; officially described as "the world's best computer-use model"
• ARC-AGI-3: over 60% (other frontier models under 10%); with an external framework, approaching 100%
• iBench (visual capability): 95%, while other models score under 50%; also leads in 3D reconstruction and music transcription

Landmark achievements are equally notable: it's the first general-purpose AI to complete Portal, which requires 3D spatial reasoning and precise control; and it completed Pokémon FireRed in just 18 hours and 12 minutes, far faster than other models.

That said, the evaluation remained objective — on the "AI Analysis Index" it ranks second behind the Claude series, and on LiveBench it even falls to third, below some Claude models. Its real advantage is cost-effectiveness: task costs are far lower than expensive Claude models, and its hallucination rate is noticeably lower than Claude Opus 5 and the previous GPT-5.6.

---

ARC-AGI (Abstraction and Reasoning Corpus) was designed by François Chollet to test fluid intelligence that can't be solved by rote memorization: each problem provides a small number of visual pattern examples, asking the model to infer the rule and predict a new output. The first generation of ARC was considered a threshold humans could easily clear but AI struggled with; ARC-AGI-3 further increases rule complexity and out-of-distribution difficulty, leaving most frontier models scoring under 10%. Frontier Math was built by a team of mathematicians and includes research-level problems that take hours even to verify. The Erdős benchmark is a subset designed for extreme difficulty, named after Hungarian mathematician Paul Erdős, symbolizing challenges near the frontier of human mathematical research. High scores on these two benchmarks are more reflective of a model's true ceiling in open-ended reasoning than traditional benchmarks like MMLU or GSM8K.

---

Access and Final Thoughts

According to the evaluation, the model is now available across all paid ChatGPT plans (Plus and Pro). In the web interface, it's accessible through the "Work" tab, and can also be invoked in the desktop app (formerly Codex). The reviewer uses a 5X Pro plan; a roughly one-hour agentic coding task typically consumes 5–10% of the weekly quota.

Taken together, this evaluation shows GPT-6 Astra delivering significant leaps in agentic coding, computer control, long-horizon planning, and mathematical reasoning. The multi-agent self-evaluation workflow is also genuinely inspiring. However, it's worth noting that all data in this article comes from a single creator's hands-on demo, including a large amount of self-reported benchmarks and subjective judgments, and some conclusions (such as product naming and release status) cannot be cross-verified from independent sources. Readers should exercise caution and await further objective evaluations.