6
AI
How do you estimate what an LLM feature will cost before launch?
Context
Finance wants a monthly cost estimate for our new AI assistant before we launch, and my honest answer is "somewhere between $2K and $40K".
What I know: expected daily active users, rough messages per session, our prompt size. What I don't know: how long conversations really get, how often tools are called, and how much prompt caching will save.
How do you produce a number you can defend? Shadow traffic, a pilot cohort, a spreadsheet model? What surprised you after launch?
Community discussion
4 comments
Have you made this decision in production? Share your reasoning.
Sign in to commentPut a hard budget cap and per-user rate limit in before launch. Our worst surprise was one customer's integration looping on the API overnight.
Track cost per successful outcome (resolved ticket, completed task), not cost per token. It is the only number finance and product both care about, and it tells you when a cheaper model is a false economy.
Model it per conversation, not per message: system prompt + retrieved context + history grows with every turn. Our turn-10 messages cost 6x our turn-1 messages until we started summarising history.
Then run a pilot with 2% of users for two weeks and fit the model to reality. Our spreadsheet was off by 2.3x; the pilot was off by 15%.
The per-conversation framing is the missing piece. Our estimate assumed every message was a turn-1 message.