AI Inference: The Compute Layer Powering Everything

The Dictionary

There’s a tendency in this business to conflate terms, so we’ll be defining our view of the terminology before diving into our content.

Inference occurs in perpetuity at whatever scale the end product demands. Gartner projects that inference will account for 55% of AI-optimized cloud infrastructure spending in 2026 and more than 65% by 2029. As AI applications move into production, inference is becoming the dominant AI computing workload and an increasingly important driver of where value accrues across the ecosystem.¹

This post looks at where that market is heading: the multi-model, increasingly open world that inference has to serve, the hardware racing to make it cheaper, the emerging split between video generation and video understanding, and the push to move decision-making out of centralized data centers and onto the edge.

The End of the Single-Model Bet

For four years, the central question in AI was which lab would win the model war. That question has, for practical purposes, been answered: Anthropic and OpenAI lead the frontier, both reportedly preparing for what could be trillion-dollar IPOs. But the answer to that question turned out to matter less than expected because almost nobody actually runs on one model.

In practice, almost no enterprise commits to a single model anymore. Most run several in production at once, and the number tends to grow over time rather than shrink. Reasoning, coding, latency, and multimodal grounding tend to each have a different leaderboard winner at any given moment, so picking one model for everything means accepting losses on three or four dimensions to win on one. Teams are picking by task instead, and managing that sprawl by keeping several models running reliably in production has become one of the biggest infrastructure headaches today.

Cost and Sovereignty Are Reinforcing the Shift

Cost is accelerating this trend. Token spend is turning into a line item that finance teams want actively managed, not a sunk cost buried in engineering budgets. Open-source models, increasingly led by Chinese labs like DeepSeek, Qwen, and Kimi, now make up a meaningful share of inference traffic, and their mere existence puts a soft ceiling on what frontier labs can charge for comparable performance.

Sovereignty is adding another reason to distribute workloads. When the U.S. government temporarily restricted foreign access to Anthropic’s Fable 5 and Mythos models in June 2026 over export-control concerns, allied governments and regulated enterprises that had built workflows on those models lost access overnight, even though it was restored within weeks. The episode reinforced a trend already underway: enterprises and governments increasingly treat control over where a model runs, where its data travels, and whose infrastructure it depends on as an operational requirement rather than a preference.

Open Weight vs. Open Source

It is worth being precise about what “open” means here, too. Almost every model described as open source on social media is actually only open weight. An open-weight release hands over the finished model (“the weights”) so anyone can download it and run it on their own infrastructure. A genuinely open-source release goes further, including the training code and a description of the training data: everything a skilled team would need to rebuild an equivalent model from scratch. By that standard, models like Llama, DeepSeek, Qwen, Kimi, and Mistral are open weight, not open source. Truly open-source efforts like OLMo, EleutherAI, and BLOOM tend to come from research-driven projects rather than commercial labs, precisely because publishing the full recipe is expensive and gives away the part of the work that is hardest to reproduce.

The distinction matters commercially because the incentives to release open weight, without going all the way to open source, are so varied. Some labs use it to undercut a leader they can’t beat on price; some, like Meta, use it to commoditize a complement to their core business so they’re never dependent on a frontier lab’s pricing; others treat a smaller open release as a funnel toward a paid API for their strongest model. Whatever the motive, the effect on the inference layer is the same: a growing supply of capable, freely downloadable models that anyone can host. Inference companies, like Together AI, Fireworks, and Baseten, have built much of their business on exactly this shift, running efficient, multi-tenant infrastructure so enterprises can capture the cost and control benefits of open models without operating a single GPU themselves.

We’re also in the midst of a live policy fight. On July 24, 2026, NVIDIA CEO Jensen Huang used his first-ever post on X to share a joint industry letter, Open Weights and American AI Leadership, urging Washington to avoid premature restrictions on downloadable models and framing a plural, open ecosystem as core to the country’s competitive position. The timing followed Moonshot AI’s release of a 2.8-trillion-parameter open-weight model days earlier that reached near-frontier benchmark performance, which revived a policy debate over restricting access to Chinese models and distillation techniques used to train challenger models. Whatever one makes of politics, it’s a useful data point: enough of today’s infrastructure incumbents (e.g. Baseten, Fireworks, Modal, Mistral, Nebius, Cohere, Microsoft, Meta, OpenAI, Google) now see commercial upside in an open ecosystem that they’re willing to lobby publicly to keep it that way.

The consequence for the inference layer is pretty straightforward: it has to serve a plural, shifting mix of models across different regulatory and geographic environments, at a cost that keeps on falling. That is the demand side of the inference story. The rest of this post is about the supply side: what it takes to serve that demand physically, visually, and geographically.

#1: Compute & Chips

Closing the Physical Gap

Inference has a hardware problem before it has a software problem. Only a sliver of the global population has access to top-tier frontier models today, and the bottleneck is physical: power, advanced chip manufacturing capacity, and memory remain constrained while demand sits orders of magnitude higher.

Most inference today still runs on general-purpose GPUs, which were built for flexibility across AI workloads rather than high-volume inference. A new wave of AI-specific chips, including those from our portfolio company Groq, is optimized exclusively for serving models, prioritizing throughput, latency, and efficiency over training performance. For example, Etched combines a low-voltage chip design with a unified memory architecture that lets processors draw from a shared pool, improving utilization and bandwidth. Next-generation entrants MatX and Fractile remain earlier in their paths to market. Together, these companies are targeting the “memory wall,” where moving data between memory and processors becomes slower than the computation itself. As language models generate responses one token at a time and agents perform increasingly long reasoning loops, memory bandwidth, rather than raw compute, is emerging as the primary constraint.

The economics are moving fast enough to reshape the market underneath these bets. Some research points to a roughly 100x decline in the cost of LLM inference over two years, even as AI adoption and token consumption continued to climb rapidly. As AI adoption grows, compute demand is shifting from training models to running them at scale. This shift has fueled a new generation of serverless inference clouds and GPU providers. Companies such as Together AI, Lambda Labs, Modal, and Fal, several of which are in our portfolio, are built around the idea that customers want different models, in different regions, at different price points, and many have raised or been valued in the billions over the past year.²

#2: Video Inference

Generation and Understanding Diverge

Text was the first modality to get an inference stack. Video is next, and it’s splitting into two distinct businesses that are easy to conflate but increasingly compete for different budgets.

1. Video generation is the more visible of the two, capturing the flashiest headlines and enormous capital. Models like Runway, Kling, Veo, and Sora turn text or images into new footage. But the more durable opportunity may sit one layer down, in infrastructure designed to serve video and image models quickly and cheaply. Fal.ai provides a serverless layer beneath much of this activity, offering API access to 600+ generative media models. MachGen goes narrower: it optimizes inference specifically for open diffusion models, image and video generators like Wan, LTX, and HiDream, cutting latency roughly 6x on the same weights and output quality. That’s enough to turn MachGen’s target workloads (e.g. real-time avatars, in-game generation, and personalized ad creative) from batch jobs into live product features.
2. Video understanding is the quieter, less-funded counterpart, solving the opposite problem: making existing video searchable and usable. Video already represents the vast majority of global internet traffic, yet most remains unindexed across surveillance archives, sports footage, and enterprise media libraries. Innovation is emerging across the stack: Twelve Labs is advancing multimodal models that jointly interpret motion, speech, audio, and spatial context; General Intuition is developing spatial-temporal models that understand how events unfold over time; and Vamana is making raw video computable by compressing, structuring, and preparing massive unlabeled datasets for AI. As enterprises look to unlock the value of their existing video archives, understanding is becoming foundational infrastructure for the next generation of AI applications.

#3 Inference at the Edge

From Content Delivery to Decision-Making

The “edge”, or computing infrastructure located closer to users and devices rather than in centralized data centers, was originally about delivering content faster: Akamai’s pitch in 1999 was to cache popular web pages regionally so they loaded more quickly. AI inference is turning that same physical layer (e.g. regional data centers, telecom networks, on-site servers) into something categorically different. Instead of simply storing content closer to users, edge infrastructure can now run AI models near the devices generating the data, allowing them to respond in real time without sending everything to a distant data center.

As inference grows to more than half of AI compute by 2030, according to McKinsey, demand for the infrastructure needed to run these workloads closer to users and devices will grow alongside it. IDC projects worldwide edge computing spending to rise nearly 14% annually between 2025 and 2028, with edge AI among the fastest-growing workloads driving that investment.³

The reason is latency. Voice agents, coding assistants, fraud detection, robotics, and factory automation all need low time-to-first-token and consistent response times that a distant, centralized data center can’t reliably guarantee. While training is concentrating in gigawatt-scale facilities where power is cheapest, inference is moving in the opposite direction: out toward factories, vehicles, hospitals, and devices, deployed across many smaller, distributed sites optimized for continuity and latency sensitivity rather than raw throughput.

Looking Ahead

Model quality is converging toward a plural, multi-vendor equilibrium rather than a single winner. What differentiates the winners from here is almost entirely a question of inference: who can serve the right model, on the right silicon, at the right place, cheaply and reliably enough that the economics work at scale. The companies that treat inference as an afterthought to model quality will find themselves out-executed by companies that treat it as the product.

We see the durable value accruing to the layers that are hardest to replicate and closest to production-critical workloads: purpose-built silicon, sovereignty-grade serving, and the video- and edge-native infrastructure that the current generation of frontier labs has the least structural incentive to build themselves.

1. Gartner, “Gartner Says AI-Optimized IaaS Is Poised to Become the Next Growth Engine for AI Infrastructure,” October 15, 2025
2. Nestor Maslej, “AI Index 2025: State of AI in 10 Charts,” Stanford Institute for Human-Centered Artificial Intelligence, April 7, 2025
3. Phillip Marangella, “How AI Inference Sends Decision-Making to the Edge,” Forbes, July 2, 2026

This communication is from Alumni Ventures, a for-profit venture capital company that is not affiliated with or endorsed by any school. It is not personalized advice, and AV only provides advice to its client funds. This communication is neither an offer to sell, nor a solicitation of an offer to purchase, any security. Such offers are made only pursuant to the formal offering documents for the fund(s) concerned, and describe significant risks and other material information that should be carefully considered before investing. For additional information, please see here. Achievement of investment objectives, including any amount of investment return, cannot be guaranteed. Co-investors are shown for illustrative purposes only, do not reflect all organizations with which AV co-invests, and do not necessarily indicate future co-investors. Example portfolio companies shown are not available to future investors, except potentially in the case of follow-on investments. Venture capital investing involves substantial risk, including risk of loss of all capital invested. Diversification cannot prevent investment loss; it is a strategy to mitigate investment risk. This communication includes forward-looking statements, generally consisting of any statement pertaining to any issue other than historical fact, including without limitation predictions, financial projections, the anticipated results of the execution of any plan or strategy, the expectation or belief of the speaker, or other events or circumstances to exist in the future. Forward-looking statements are not representations of actual fact, depend on certain assumptions that may not be realized, and are not guaranteed to occur. Any forward-looking statements included in this communication speak only as of the date of the communication. AV and its affiliates disclaim any obligation to update, amend, or alter such forward-looking statements, whether due to subsequent events, new information, or otherwise.