Benchmarks measure more than the model
A benchmark score can look like a clean verdict on an AI model. It isn’t. The software and tools wrapped around the model can change its performance dramatically, which makes the setup behind a score part of the story.
That distinction sits at the heart of the argument over artificial general intelligence, or AGI: a system able to match or exceed human performance across essentially any cognitive task, rather than only the tasks it was trained to handle. Nvidia CEO Jensen Huang said AGI had arrived after OpenAI released GPT-6 Astra on Sept. 3. In a post on X, he also said “400K GPUs” were “coming online next.” OpenAI didn’t call Astra AGI; the company emphasized its speed and versatility.
I’m an AI engineer building ChatMakr, a platform for creating multi-agent chatbots. After a year of watching new releases get framed as the model that changes everything, I’d be cautious about treating a striking score—or a sweeping label—as proof of general intelligence.
Astra’s scores are striking, but they don’t settle the AGI question
OpenAI presents Astra as an operator that can browse the web, write code, run multi-step workflows, model in Blender and handle cybersecurity work with little supervision. Its reported results are eye-catching: 97.6% on FrontierMath Tier 4, 99.9% on ARC-AGI-3 and 100% on ExploitBench, compared with 78.5% for GPT-5.6 Sol on that last benchmark. Those gains deserve attention. They don’t, by themselves, show that a system can handle essentially any cognitive task.
That’s a distinction OpenAI itself appears to respect: the company stopped short of calling Astra AGI. Researchers have also challenged the term. Wharton professor Ethan Mollick describes current systems as “jagged AGI”—superhuman in some narrow areas, unreliable in others. There’s no agreed test for AGI, so the declaration remains a judgment, not a settled measurement. Augmented Mind and Manolo Remiddi put the practical test this way: “AGI is the point where you can trust the output without checking. We are not there.”
Similar scores can mask very different tools
For people building with AI, the comparison between Astra and Anthropic’s Claude Fable 5.1 is more useful than the AGI debate. The models post near-matching scores on several coding and reasoning benchmarks, yet engineers report different results in practice. Fable tends to produce production-ready code with fewer tokens; Astra’s strengths show up more in computer use and multi-step automation.
The failure pattern matters, too. Snorkel AI’s testing found that Fable can lose valid work partway through a task, rather than simply running out of budget. For a team shipping software, that distinction may matter more than a leaderboard position. A score says how a system performed under particular conditions; it doesn’t tell you how well it will fit your workflow.
The harness can move the score
Those conditions include the “harness”: the prompts, tools, memory and error-recovery systems that shape how a model works. MindStudio offers a useful analogy: “A model is the engine. The harness is the car. Most benchmarks test the car.” Change the harness and the same model can behave very differently. Cursor’s research found a model scoring 46% accuracy with one harness and 80% with another, without changing the model.
Astra’s ARC-AGI-3 results make the point unusually clearly. With a standard harness, it scored 62.7%, roughly level with its predecessor. With a “Provider Adapter” harness that preserved and reused prior reasoning instead of starting fresh at each step, it reached 99.9%, according to Augmented Mind and Manolo Remiddi. Same model weights; a 37-point difference tied to infrastructure. That’s a large enough swing to change how we should read claims of model progress. Leaderboards that don’t control for the harness may tell a compelling story, but the score alone can’t show which part of the system deserves the credit.
Cursor’s acquisition points to where competition is shifting
That’s why SpaceX’s $60 billion acquisition of Cursor may matter more than the headline suggests. Cursor builds the layer that determines how coding models are put to work. SpaceX is pairing that capability with its GPU fleet, a bet that control of the user-facing tools and workflows can be as valuable as control of the underlying model.
The dispute over access reinforces the point: OpenAI cut Cursor’s access to its models over trust concerns related to Musk’s companies. Model providers understand that the harness shapes the product people actually experience. If you’re choosing AI for a business, test the full system on your own work, ask how it fails and find out what the vendor has built around the model. Astra’s benchmark numbers are impressive. They still can’t make that decision for you.
