AI Infrastructure Is Expanding Faster Than Companies Can Prove Its Value

The race to build artificial intelligence infrastructure is entering a more demanding phase. For the past several years, companies have focused on obtaining powerful GPUs, deploying advanced models, and expanding their AI capabilities. Now, a less comfortable question is emerging. How much of that expensive computing power is actually producing useful business results?

A September survey by VentureBeat Intelligence offers a striking answer. Among 68 qualified respondents whose organizations were running AI in production and reported GPU utilization, 81% said their average utilization was 50% or less. Yet many organizations are preparing to invest in new infrastructure anyway.

The figures do not prove that every underused GPU represents wasted money. Some spare capacity is necessary for sudden demand, reliability, and future growth. But the findings expose a larger challenge for enterprise AI. Companies can invest heavily in infrastructure without having a clear picture of whether that investment is delivering enough value.

As Nvidia promotes its vision of AI factories, the next competitive advantage may depend less on how much computing power a company owns and more on how effectively it turns that power into completed work.

The AI infrastructure race meets a reality check

At its October 21 GTC Berlin keynote, Nvidia CEO Jensen Huang urged companies to become producers of AI rather than simply consumers. His argument reflects an important shift in the market. Businesses increasingly want AI systems that can perform useful work inside their own operations, rather than merely provide answers through a chatbot.

That ambition requires more than powerful processors. It requires infrastructure that connects models with company data, business applications, security controls, and the systems employees already use.

The market is responding accordingly. HPE is emphasizing governed APIs and Model Context Protocol (MCP) integration. VAST has announced an Agent Sandbox Foundation Stack with Nvidia, designed to preserve agent state, data access, and audit trails as computing resources scale up or down. Nebius has highlighted GPU confidential computing, which can help protect data and model weights during processing and allow customers to verify an environment before releasing encryption keys.

These developments point toward a broader definition of AI infrastructure. The goal is no longer simply to run a model. It is to build an environment in which AI can perform work securely, consistently, and at an economically defensible cost.

That distinction matters because buying more computing capacity does not automatically solve the problems that prevent AI applications from succeeding.

The numbers reveal an investment gap

The September VB Pulse survey included 102 qualified respondents from organizations with at least 100 employees, including consultants responding on behalf of their main clients. The results were self-reported by a self-selected group, so they should not be treated as a census of the entire AI industry.

Even with that limitation, the findings reveal a notable tension.

Of the 68 respondents whose organizations had AI in production and supplied GPU utilization figures, 55 reported average utilization of 50% or less. That represents 81% of this group.

More than seven in ten of those 55 respondents—71%—planned to adopt a new, additional, or replacement AI infrastructure platform within 12 months.

The wider investment picture is equally revealing. Across all 102 respondents, 63% planned to adopt a new, additional, or replacement platform within a year. Yet only 10% reported an infrastructure return-on-investment figure that had been reviewed and approved by their finance department.

Among respondents planning to adopt a platform, 91% lacked a finance-reviewed infrastructure ROI figure. The corresponding share among those without plans to change platforms was 89%. The report found no statistically significant difference between these groups.

In other words, companies were making infrastructure decisions without a finance-validated understanding of the returns from their existing investments.

Operational satisfaction did not resolve that uncertainty. Respondents whose primary platform was in production gave their platforms an average satisfaction score of 4.0 out of 5.0. Ease of implementation averaged 3.9, while value for money averaged 3.6.

These scores suggest that infrastructure can be reasonably easy to deploy and satisfactory to operate while its overall business value remains unclear.

A system can work as designed and still fail to justify its total cost.

Why more computing power may not solve the problem

Low GPU utilization can have several explanations. Workloads may be uneven, systems may be poorly scheduled, or organizations may have purchased capacity in anticipation of demand that has not yet arrived. The survey does not establish which explanation applies to each company.

The practical lesson is that businesses need to identify the actual bottleneck before buying additional infrastructure.

Consider an AI agent that helps employees investigate customer cases. It might use a powerful model but repeatedly retrieve irrelevant information, lose track of earlier decisions, or attempt the same operation several times. Human employees may then need to correct its output.

Adding more GPU capacity will not necessarily fix any of those problems.

A better retrieval system could give the agent more relevant information. A smaller model might handle routine steps, leaving a more capable model for difficult decisions. Improved task-state management could prevent unnecessary repetition. Clear escalation rules could send uncertain cases to a human before they become expensive mistakes.

Each of these changes could improve the economics of the application without requiring the company to expand its computing resources.

The survey reinforces this point through its infrastructure-buying priorities. Integration with the existing cloud and data stack was selected by 48% of respondents, performance by 42%, and GPU availability by 35%. The difference between integration and performance was not statistically significant, according to the report.

The message is straightforward. Businesses care about compute, but they also need it to work with the systems and information that make their operations function.

Memory is becoming an infrastructure decision

As AI agents take on longer assignments, memory becomes more important. However, the term describes several different technical challenges, and confusing them can lead companies toward the wrong solution.

Institutional memory is the organization's durable knowledge: policies, records, operating procedures, and previous decisions. Agent task memory is the information required to continue a particular assignment, including what has happened, what remains unfinished, and what earlier actions produced. Inference memory concerns the resources and cached computation used while a model serves requests.

These layers serve different purposes.

An agent may need to retrieve an internal policy without receiving permission to access unrelated records. It may need to remember the progress of a long-running task without treating a temporary conclusion as an approved company policy. Meanwhile, the infrastructure serving its model may benefit from caching repeated computation.

For example, Anthropic's context-engineering guidance describes structured notes stored outside a model's context window as one way to preserve information during extended work. Separately, vLLM's automatic prefix caching can reuse computation when requests share a prompt prefix. That can reduce prompt-processing work, although it does not reduce the time required to generate new tokens.

The distinction is economically important. Saving information, retrieving information, and reusing computation are not interchangeable operations. Each addresses a different source of cost or operational difficulty.

In the September survey, Nvidia and Dell were the most frequently selected inference-memory options for use or evaluation, at 41% and 39%, respectively. The difference was not statistically significant. Open-source key-value cache tooling and model-level efficiency techniques each received 21% of selections. Another 16% of respondents had not addressed inference-memory limitations or were unaware of them.

These results suggest that memory optimization remains an area where some companies are experimenting while others may not yet have assessed the problem.

Smaller models could improve the economics of enterprise AI

The industry often treats model capability as the central measure of progress. Yet a business application does not necessarily need the most capable model for every operation.

A smaller language model may be sufficient for a bounded task, such as classifying a routine request or extracting structured information. More complex work may require a larger model, a specialized tool, or human judgment.

The right choice depends on the required quality and the complete workflow.

A useful evaluation must consider more than the price of a million tokens. Token consumption, retrieval, tool calls, repeated attempts, and human intervention all influence the final cost. A low-cost model that regularly produces incorrect results may be more expensive overall than a stronger model that completes the task reliably on its first attempt.

The September report found that 23% of respondents selected cost per million tokens as an infrastructure-buying factor, up from 11% in August. The report described the difference between the two monthly samples as statistically significant. However, the samples were different groups of respondents, so the result should not be interpreted as a direct measurement of how individual companies changed their priorities.

Token pricing is becoming more visible in purchasing decisions. But token price alone cannot establish whether an AI system is economical.

The more useful question is how much it costs to complete a task successfully at the required quality.

That measure should account for every meaningful step, including information retrieval, model inference, retries, external tools, and human review. A customer-support case closed incorrectly should not count as a successful completion. Likewise, the cost of repairing a defective code change should be included when evaluating an AI coding assistant.

The denominator matters as much as the price.

The business value lies in completed work

For companies building AI products, this shift creates a more practical way to connect technical performance with commercial value.

Imagine a business using an AI agent to review incoming customer requests. The company could measure the cost of processing each request, but that number would be incomplete if some cases required substantial human correction or were incorrectly classified.

A better assessment would compare the cost of successfully resolved cases against an agreed baseline. The same principle applies to software development, research, internal knowledge retrieval, and other enterprise workflows.

This does not mean every task must be fully automated. In some situations, the most valuable outcome is an agent that prepares information for an employee rather than making the final decision. Human involvement should be measured against the quality and business value of the outcome, not treated automatically as a failure.

This is also where infrastructure decisions become business decisions.

The September survey found a significant difference in priorities by seniority. Among vice presidents and directors, 52% selected total cost of ownership as a buying factor, compared with 4% of individual contributors. Among individual contributors, integration was the most frequently selected consideration, at 58%.

These findings describe different concerns rather than necessarily conflicting objectives. Leaders need to understand the financial consequences of architecture. Technical teams need to understand how their decisions affect integration, performance, reliability, and cost.

A shared measure—cost per successfully completed task—can help connect those perspectives.

It also gives businesses a way to evaluate whether changing a model, improving retrieval, adjusting caching, or redesigning task memory creates enough additional value to justify the change.

AI infrastructure is becoming a platform strategy

The movement toward AI factories reflects a broader ambition than deploying individual models. Companies want a reusable foundation for running multiple AI applications, connecting them to internal information, and managing their operations.

That is a platform strategy.

The advantage of a platform is that common capabilities can support several applications. Security controls, data access, monitoring, and infrastructure management do not need to be reinvented for every workflow. But a platform is valuable only when its shared capabilities reduce complexity or improve outcomes enough to justify the investment.

This creates an important distinction between infrastructure scale and infrastructure usefulness.

A company might operate a large GPU environment while leaving its most important business workflows poorly integrated. Another company might use a more modest setup but achieve better results by selecting models carefully, maintaining reliable task state, and limiting unnecessary computation.

Neither outcome can be judged from GPU utilization alone. High utilization can indicate efficient use of resources, but it can also reflect congestion or insufficient capacity. Low utilization can indicate waste, but it may also be intentional protection against demand spikes.

Utilization is a diagnostic measure, not a complete business verdict.

The same caution applies to platform satisfaction. A positive user experience is useful, but it does not replace a financial assessment of the value delivered.

Competition will increasingly depend on integration and trust

As enterprise AI infrastructure expands, the competitive question is becoming more complex than which provider offers the most computing power.

Integration determines whether models can work with existing applications and data. Memory determines whether agents can preserve useful progress and retrieve the right information. Security determines whether organizations can use sensitive data under appropriate controls. Observability determines whether teams can understand what their systems are doing and why costs are rising.

The initiatives described by HPE, VAST, Nvidia, and Nebius illustrate different parts of this challenge. Governed APIs and MCP integration address connections between systems. Agent-state preservation and audit trails address continuity and accountability. Confidential computing addresses protection of data and model weights during processing.

These approaches are not identical products, and the survey does not establish that one provider is superior to another. Instead, they demonstrate how the infrastructure market is broadening beyond raw compute.

For buyers, this means vendor comparisons should be based on the needs of actual workloads. A platform that excels at raw performance may not be the best choice for a workflow dominated by retrieval delays, inconsistent task state, or complex security requirements.

The strongest architecture is the one that addresses the real constraint rather than the most visible specification.

The risks of investing before proving returns

The first risk is financial. Companies may commit to additional infrastructure before they understand the economics of their existing systems. The survey's ROI findings make this concern difficult to ignore, although they do not establish that planned investments will ultimately fail.

The second risk is measuring the wrong outcome. A model that processes more tokens or responds more quickly may still create little value if it fails to complete the required task correctly.

The third risk involves information governance. Institutional knowledge must remain current and accessible only under the appropriate permissions. Task memory must preserve progress without accidentally turning provisional conclusions into authoritative records. Systems must also be evaluated for reliability and monitored after deployment.

The fourth risk is architectural complexity. An application can involve models, retrieval systems, caches, external tools, storage, and human review. Improving one component may shift costs or create new failure points elsewhere. That is why testing individual components in isolation is not enough.

The fifth risk is drawing conclusions from incomplete data. The survey relies on self-reported responses from a self-selected sample. Its figures are useful signals, but they do not represent every enterprise or establish the cause of low utilization.

A sound investment decision requires more than a market trend or a vendor's performance claims. It requires evidence from the company's own workloads.

What businesses should do before buying more infrastructure

The starting point should be one specific AI workflow, not an abstract ambition to become an AI-first company.

First, the product owner should define successful completion. That means specifying acceptable quality, the conditions under which a task must be escalated, and how much human intervention is reasonable.

Second, the data team should identify the authoritative sources of information and establish the access rules governing them. An agent cannot produce reliable results if it retrieves outdated or irrelevant material, or if it cannot distinguish trusted information from temporary task notes.

Third, platform engineering should measure the application's actual behavior. Relevant metrics include latency, GPU utilization, caching performance, retrieval activity, retries, and total operating cost. These measurements should be tied to the chosen workflow rather than collected simply because the infrastructure makes them available.

Fourth, finance should agree on a baseline and validate the value created. The comparison should account for the cost of operating the AI system alongside the benefits it delivers, using a consistent definition of success.

Finally, teams should change one major architectural component at a time and measure the effect on cost per successfully completed task. They should test model selection, retrieval, task-state handling, and escalation together against representative work, repeating the evaluation when important components change.

This process can reveal whether the business needs additional GPUs, better scheduling, a different model, improved memory, or a more reliable path to enterprise data. It can also establish when the existing infrastructure is already sufficient.

The next AI advantage will be measured in outcomes

The idea of an AI factory captures an important direction for the technology industry. Businesses want to produce useful AI-driven work, not simply consume model outputs. But the factory analogy also raises a basic economic question: What is being produced, at what cost, and with what level of quality?

The September survey suggests that many organizations are still working toward clear answers. GPU utilization is low for a large share of the production respondents who reported figures, while plans for additional infrastructure remain widespread. At the same time, relatively few respondents reported returns validated by finance.

That gap should encourage better measurement rather than an automatic retreat from AI investment.

More powerful models will expand what agents can attempt. Yet the business value of those models will depend on how well they connect to data, preserve task progress, select appropriate tools, and recover from errors. Infrastructure must support that entire process.

For business leaders, the priority is to establish evidence before expanding capacity. For technical teams, it is to identify the bottleneck and measure the effect of architectural changes. For finance, it is to connect operating costs to a clear and agreed definition of business value.

The future of enterprise AI will not be determined by GPU ownership alone. It will be determined by which organizations can turn computing power into reliable work—and demonstrate that the work is worth its cost.

AI Infrastructure Is Expanding Faster Than Companies Can Prove Its Value

AI infrastructure, GPU utilization, Nvidia AI factories, enterprise AI strategy, AI infrastructure costs, AI return on investment, AI agent memory, inference optimization, small language models, AI task completion, enterprise AI platforms, GPU efficiency, AI operating costs, AI infrastructure strategy