AI running costs are decided at inference

A model is trained once, but running it costs money with every answer.

For a small business owner, AI running costs often look simple on the invoice: a monthly fee or a price per amount of text used. Behind that price, however, sits a whole chain that starts every time a customer service assistant answers a question or a tool summarises an email. This is called inference.

Y Combinator’s YC Paper Club episode Why Inference Matters 12x More Than Training deals with exactly this. According to the episode description, for every compute operation used to train a model, around six are spent each year running it. The description also states that solving inference has created more than $220 billion in market value over the past decade. The figures are YC’s own and have not been separately checked here.

Every answer passes through many layers

According to the description, inference is much more than calling a model. Every generated piece of text, or token, has to pass through routing, caching, scheduling, compute kernels, memory and hardware. At the same time, speed, cost and quality have to be balanced.

The episode features researchers and builders from Together AI, Ollama, Luminal, SGLang and RadixArk. At 12:14 in the programme, Dan Fu walks through the lifecycle of an inference request. According to the description, the episode shows how a token travels from an API request to the GPU and back.

Business owners do not need to know how to tune these layers. It is still worth knowing they exist, because they explain why two services that look the same on the surface can cost different amounts and respond at different speeds. We have written earlier about what to ask about AI costs before believing in cheaper computing.

The same model is not always the same service

According to the description, the episode explores why the same open model can behave differently depending on who runs it. The title of the 42:18 segment sums it up: according to Jeff Morgan, not all open model tokens are the same.

I consider this the most practical lesson of the episode for a small business. When a provider says it uses a well-known open model, the model’s name does not tell the whole story. I would suggest comparing services with your own examples. Give two services the same ten typical customer questions and compare the quality of the answers, the response time and the price. In our earlier article on open AI models, we discussed how openness can give a business the freedom to switch suppliers. That freedom is only useful if you actually test the alternatives.

Compilers and caching make inference more efficient

According to the description, the final part of the episode focuses on new ways to make inference faster and considerably more efficient. At 1:04:31, Joe Fioti talks about compiling models through search instead of ready-made rules of thumb. At 1:30:30, Liangsheng Yin discusses SGLang and memory in the age of agents.

Agents are an interesting detail here. When AI handles a multi-step task, it makes several consecutive calls and needs earlier information again. Memory and cache management then has a direct effect on cost. We wrote earlier about why an AI agent needs a working environment and why cost is best calculated per finished piece of work.

What should business owners ask an AI service provider?

The episode is technical and long, but a few procurement questions can be drawn from it:

• What does one typical task, such as one customer service reply, cost in our use?
• How quickly does the service respond at peak times, and how does that show in the price?
• Which model does the service use, and who runs it?
• Does quality change if the service shifts load to a cheaper solution?

AI running costs should therefore be assessed over the whole lifecycle, not just on the price of getting started. The model is trained once, but your business pays for every answer for as long as the service is in use. When you measure price, speed and quality for your own way of using it, you get a better basis for your decision than the model’s name.

▶

Watch the original episode on YouTube: Y Combinator.

Sources

• Y Combinator: Why Inference Matters 12x More Than Training (YC Paper Club) — Based on YC’s episode description and timestamped programme, not a verified transcript.