Nvidia has moved its Vera Rubin NVL72 platform into full production with a new inference accelerator attached, and the target it named is unusually specific: AI agents. In an announcement dated 24 August 2026, the company said Vera Rubin NVL72 paired with Nvidia Groq 3 LPX is now shipping for production inference workloads, with performance claims aimed squarely at the token-generation bottleneck that slows multi-step autonomous systems.
The headline figure: 3,400 output tokens per second on 100,000-token long-context workloads running Gemma 4 31B, an open-source agentic model, in an Artificial Analysis benchmark — which Nvidia characterises as roughly four times faster than the nearest alternative platform.
Decode Latency Is the New Bottleneck
The engineering argument behind the launch is worth understanding, because it explains why inference hardware is diverging from training hardware.
When a model answers a single question, a few hundred milliseconds of decode latency is invisible to the user. When an agent reasons through a plan, calls a tool, reads the result, revises its approach and calls another tool, that same latency compounds across every step in the chain. A workflow with thirty sequential model calls inherits thirty times the delay. Long context makes it worse: agents accumulate transcripts, tool outputs and retrieved documents, so the working context grows as the task proceeds.
Nvidia's framing is that the industry has optimised for throughput on batch workloads while agentic systems need fast sequential token generation over large contexts — a different problem requiring different silicon. That is the gap Groq 3 LPX is positioned to fill, sitting alongside the Vera CPU and Rubin GPU inside the same rack-scale system.
The company was explicit that this is a systems play rather than a chip play, writing that the next era of AI inference "won't be defined by a single breakthrough chip" but by how every layer of the AI factory works together.
The Rest of the Stack
The August announcement bundled several components beyond the accelerator itself:
- Spectrum-X Multiplane networking, which Nvidia claims delivers 1.6x better AI networking performance, retains 90% of bandwidth during a single-plane failure, and recovers from hardware faults 11 times faster than software-based approaches.
- Scale-In infrastructure, designed to handle multi-tenant networking, storage access, security, provisioning and observability without consuming host compute — pushing infrastructure services closer to the AI system.
- NVLink Fusion, extending the interconnect fabric, with the overall architecture said to scale to 512,000 GPUs without requiring a third network tier.
Early adopters were named. Nebius is the first AI cloud to deploy Groq 3 LPX. CoreWeave is rolling out Spectrum-X Multiplane. And SpaceXAI is adopting Vera CPUs for agentic AI workloads.
Why It Matters
For AI research and deployment teams, the significance is economic before it is technical. Agentic architectures have been held back less by model capability than by cost-per-completed-task: a chain of twenty reasoning steps at long context is expensive to run and slow enough that users abandon it. Hardware that materially improves decode throughput on long-context workloads changes which agent designs are viable in production rather than in a demo.
It also signals where the compute market is heading. Training clusters bought the last two years of Nvidia's growth; inference for agents is being positioned to buy the next several. That distinction matters for anyone forecasting AI infrastructure spend, because inference demand scales with usage, not with model release cadence — a more durable, more recurring revenue base.
There is a competitive read too. By choosing an open-source agentic model for its headline benchmark rather than a proprietary flagship, Nvidia is aligning its performance story with the part of the ecosystem most likely to run on-premises and on rented cloud capacity — the buyers who care most about tokens per second per dollar.
Read the Benchmarks Carefully
Some caution is warranted. Vendor-published benchmarks, even those run through third-party evaluators, are chosen to flatter the architecture under test. A 3,400 tokens-per-second figure on one open model at one context length does not translate cleanly to a mixture-of-experts frontier model, a different batch profile or a different serving stack. Real-world agent performance depends heavily on tool-call latency, retrieval systems and orchestration overhead — none of which the accelerator controls.
The infrastructure claims deserve similar scrutiny. Bandwidth retention during a single-plane failure and recovery speed are meaningful reliability metrics, but they describe behaviour under a specific failure mode, not overall cluster availability.
The honest summary is that Nvidia has shipped a system explicitly designed for the workload shape that agentic AI produces, backed by named cloud adopters, with performance claims that will now be tested in the open by customers running their own models.
What to Watch Next
Three things will indicate whether the agentic-inference thesis holds. First, whether independent evaluations reproduce the long-context decode advantage on frontier-scale mixture-of-experts models. Second, whether inference pricing at Nebius, CoreWeave and other adopters falls in a way customers can actually observe. Third, whether competing accelerator vendors reorganise their roadmaps around decode latency rather than raw training FLOPs.
If they do, the defining spec of the next hardware cycle will be how fast a machine can think out loud.
