Less than a year ago, our reporters kept hearing doubts about a group of startups called inference providers. Companies like Fireworks, Baseten and Together AI, which rent out Nvidia servers to app developers and help them customize open-source models, had grown quickly but seemed at risk of getting steamrolled by major cloud providers that could build these capabilities in-house. Those traditional cloud providers also have the advantage of owning the AI chips servers they rent out; inference firms, in contrast, generally rent the chips from those traditional providers and then turn around and rent them out to their customers. That dynamic has dragged down the gross profit margins of some inference providers in the past.
Less than a year ago, our reporters kept hearing doubts about a group of startups called inference providers. Companies like Fireworks, Baseten and Together AI, which rent out Nvidia servers to app developers and help them customize open-source models, had grown quickly but seemed at risk of getting steamrolled by major cloud providers that could build these capabilities in-house.
Those traditional cloud providers also have the advantage of owning the AI chips servers they rent out; inference firms, in contrast, generally rent the chips from those traditional providers and then turn around and rent them out to their customers. That dynamic has dragged down the gross profit margins of some inference providers in the past.
While the current margins of the inference startups couldn’t be learned, they clearly have customer demand and cash—including sometimes from heavyweight Nvidia—on their side. As Stephanie and Katie reported Tuesday, Baseten has been discussing raising $1 billion at a valuation of $11 billion, or more than double its last valuation. The Nvidia-backed company had even higher offers, which would value it up to $15 billion, but decided to stick with the investors who are backing the startup at the lower valuation.
Piquing investor interest is red-hot revenue growth at these startups, as the table below shows. Baseten’s revenue tripled in the three months ending in March, to $600 million, and is up 20 times year over year. Modal’s revenue has jumped five times from last fall, to more than $300 million. The biggest of the pack is Together. It was generating about $1 billion in annualized revenue when we broke the news in March that it was raising at a $8.5 billion post-money valuation.
But as rental prices for AI chips surge and traditional cloud providers like Microsoft save more chips for mega-customers like Anthropic, the margin squeeze on inference providers could become a bigger concern. Investors expect the inference startups to counter that pressure by expanding to other product offerings for which they can charge a premium.
Some of these startups have already started doing that. Baseten, for instance, has used tech from its acquisition of the startup Parsed to help its customers use their own data to help them fine-tune models, according to a person with direct knowledge of the company.
At Modal, some of its growth is related to its so-called sandbox product, which lets developers run AI agents and code without affecting the rest of their computer and codebase.
“The cloud providers are very good at offering capacity all over the world at a decent price,” said Modal’s founder and CEO Erik Bernhardsson on The Information’s new streaming show, TITV, Tuesday. “What they’re often not that good at is, in my opinion, delivering good developer experience and delivering a software product that makes it possible to iterate quickly.”
Bernhardsson said the company is now on the hunt to buy startups that can help it build that infrastructure layer, say in reinforcement learning, a technique that rewards a model for accomplishing certain goals and penalizes it for other behaviors.
Unlike Together, which has made moves to buy and run its own Nvidia servers, Model won’t use its new capital to buy chips: “Being pure software enables us to grow very quickly and move very fast,” Bernhardsson said.
Today on TITV
Watch reporter Ann Gehan discuss OpenAI’s evolving ad strategy.
Tickets are on sale now for The Information’s AI Agenda Live in San Francisco. Join top AI leaders for conversations on the future of AI, and secure early bird pricing while it lasts!
Save the Date: The Information returns to Napa Valley October 27-28 to convene senior women across tech, media, and finance. The event will feature two days of intimate, candid conversations with the leaders navigating today’s global shifts.
