Open AI API Cost Estimation for Chatbots

I found many blogs/articles estimating cost of using Open AI API, but most of them were based on certain estimated number of tokens per day/per month etc. This kind of estimation is difficult to be used in estimation for actual applications for example a chatbot etc, because different applications usage should be estimated in completely different ways. Even chatbots with similar number of users but different use cases for example travel app vs sales app vs marketing app will have different usage patterns. We should follow a better approach that helps to distinguish between their usage patterns and thus estimate their cost more reliably. I have discussed one such approach in this article for chatbots, but before that I will briefly explain OpenAI’s pricing.

The Open AI’s pricing rates can be found here. Below we can see rates for GPT-3.5 turbo (chatGPT). There are two variations of model based on context length, 4K and 16K. Context length includes input+output token lengths. To get some intuition, 4K tokens roughly means 3K words (roughly 100 tokens refer to 75 words). This paragraph is roughly 60 words and 80 tokens.

Open AI GPT-3.5 turbo cost

Let’s take the example of a travel app with 100 K registered users.

Assumptions :

1. On average, a user makes 20 queries per session with chatbot. For a travel app, users could be asking many back and forth queries about travel destinations. For other use cases like sales app, this could be less, say 10 per session.
2. For each of those 20 query-response, let’s take 1K tokens in total (input+output).
3. For our travel app, Input to output token ratio of 60:40. This can drastically differ based on the query/task. If somebody wants to ask a question from documents, it would be close to 90:10 but if the task is to generate a mail, this ratio could be 20:80. Most cases, input is likely to be larger as the conversation history keeps adding up as context to input.
4. 10 % of users are active on any day with on average 5 sessions per day.
5. Total number of registered users is 100 K

All these assumptions can be made more accurately if you have access to any kind of historical data.

Trend and Seasonality of users activity can also be considered, as some days will have more or less % of active users.

Let’s get to the final calculation.

If we assume that input to output ratio of 60:40, applicable rate will be $0.0017/1K tokens (0.6*$0.0015+0.4*$0.002 = $0.0017) for 4K context. This makes cost per user session $0.034.

Number of user sessions per day = (10 % of 100 K) * 5 = 50 K user sessions.

Total cost per day = 50 K user sessions * $0.034 = $ 1.7 K

Hope this helps in providing a thought process of making a better estimate of OpenAI API cost for your use cases. If you think so, please give a clap, encourages me to write more.