While NVIDIA's recent earnings call was met with the usual fanfare, celebrating record-breaking revenue and the seemingly unstoppable march of AI, a subtle shift in language may be hinting at a looming challenge. Hidden within Jensen Huang's enthusiastic pronouncements about AI factories and trillion-parameter models lies a potential roadblock: the growing reliance on inference and the potential for an "inference gap."

For years, the narrative around NVIDIA's success has focused on the insatiable demand for their high-performance GPUs for AI training. Building ever-larger, more complex models has fueled a seemingly endless hunger for compute power, and NVIDIA has been more than happy to supply the fuel. But now, as Huang himself admits, inference – the process of actually using those trained models – is rapidly becoming a dominant force, accounting for an estimated 40% of their data center revenue.

This isn't inherently bad. Inference is where the rubber meets the road in AI, translating theoretical potential into real-world applications. But the rise of generative AI, with its ability to create complex outputs like text, images, and even video, introduces a new wrinkle. As Huang puts it, "The inference part of our business has grown tremendously. We estimate about 40%... but the estimate is probably understated."

He goes on to highlight the vast and growing range of applications relying on inference: recommender systems for e-commerce, content creation tools like Adobe Firefly, AI-powered search, and of course, the seemingly ubiquitous ChatGPT. Each interaction, each query, each generated image or video, adds to the inference burden on NVIDIA's GPUs.

The question then becomes, can NVIDIA keep up? While their supply chain is improving, and they're continually innovating with new architectures like Blackwell, the sheer scale of the inference demand they're predicting is staggering. Huang uses phrases like "demand will continue to be stronger than our supply provides," "demand is greater than supply," and "no way we can reasonably keep up on demand in the short term" when discussing both Hopper and Blackwell.

This suggests a potential "inference gap," where the rapidly growing need for inference outpaces even NVIDIA's impressive ability to increase supply. If this gap widens, it could create a bottleneck in the AI ecosystem. Developers may struggle to deploy their applications, companies might face delays in implementing AI solutions, and ultimately, the breakneck pace of innovation in the field could slow down.

Here's where the hypothesis gets interesting. Let's assume NVIDIA's estimate of inference accounting for 40% of data center revenue is, as Huang suggests, understated. If we bump that number to even 50%, it represents a significant portion of their current $18.4 billion quarterly data center revenue. That's $9.2 billion tied directly to the ability to keep up with inference demand.

Now, consider the implications of a supply-demand mismatch. Could it force companies to prioritize certain inference workloads over others? Will access to NVIDIA's latest GPUs become a competitive advantage, potentially favoring larger players with deeper pockets? Could it even stifle the growth of smaller, innovative AI startups that rely heavily on efficient and readily available inference?

Projected Growth of Inference Demand

The chart below depicts the potential growth of inference demand based on the provided data. This is a hypothetical representation for illustrative purposes.

The answers to these questions remain to be seen. However, the subtle shift in NVIDIA's language from focusing solely on training to acknowledging the growing importance, and potential challenge, of inference should serve as a wake-up call. The "inference gap" may be the next battleground in the AI revolution, and NVIDIA's ability to navigate this challenge will be crucial for both their own success and the future of the industry as a whole.

"Fun Fact: Did you know that NVIDIA's journey began not with AI, but with gaming? Their first product, the NV1, was a multimedia accelerator card released in 1995."