The real value of Large Language Models

Today, AI and Large Language Models (LLMs) like GPT-4 are making big waves in tech. Recent studies indicate that over 77% of companies are either using or exploring the use of AI, and LLMs are leading the way. LLMs are helping with everything, from content creation to customer support. But how do businesses know if they’re getting their money’s worth?

Usage of LLMs trend. Image taken from Databricks’ “2023 State of Data + AI” survey.

The expansive reach of LLMs

LLMs are more than just the latest tech trend. Businesses use them for a variety of tasks: marketers create engaging stories, customer service teams give fast, tailored answers, and they help with real-time language translation. They assist researchers in data analysis and are tools for coders, with 92% of U.S. developers relying on AI for programming.

Forbes reports that 73% of businesses are integrating or planning to use AI-powered chatbots. Moreover, 61% enhance their email strategies with AI, and 55% use it for customized product recommendations. With all these applications, it’s clear LLMs have taken a central role in modern business strategies.

Tangible and intangible returns

ROI (Return on Investment) is traditionally calculated using a basic formula:

ROI = (gain — cost) / cost x 100

Yet, in our modern world with technologies like LLMs deeply integrated into businesses, ROI isn’t just about this basic calculation. It’s crucial to consider both the direct, tangible benefits (hard ROI) and the less obvious, intangible ones (soft ROI).

Direct Benefits (Hard ROI) from LLMs:

• Efficiency: LLMs make processes smoother, reduce errors, and speed up decisions. This leads to cost savings in areas like data management and reduced manual work.
• User experience: LLMs’ 24/7 support and predictive features can boost user retention, website visits, and revenues. For example, a better user experience can increase sales and leads.
• Innovation: LLMs can open up new opportunities or enhance current offerings, attracting more supporters or subscribers.

Indirect Benefits (Soft ROI) from LLMs:

• Brand reputation: Adopting LLMs can improve a brand’s image, leading to benefits like more word-of-mouth recommendations or positive media coverage.
• Attracting talent: Being technologically advanced can attract high-quality employees, reduce staff turnover, all impacting the bottom line.
• Strategic position: Staying ahead technologically can give a competitive edge, build trust with stakeholders, and open doors to new collaborations.
• Investor and stakeholder trust: Adopting AI proactively can inspire confidence, leading to long-term financial rewards.

Metrics that were once seen as “soft” benefits, like social media engagement, are becoming quantifiable as technologies progress. For example, marketers can now assign dollar values to social media actions, turning them into “hard” ROI metrics. Similarly, as LLMs become more present in business functions, their indirect benefits will be easier to measure.

Let’s consider a retail company, that decided to integrate an LLM-based chatbot to enhance customer experience:

Direct Benefits (Hard ROI):

1. Efficiency: By reducing manual customer service representatives from 10 to 5, the company saves $250,000 annually (assuming $50,000 in annual cost per FTE).
2. User experience: Post-integration, the company saw a 20% increase in sales, equating to an additional $300,000 in annual revenue.
3. Innovation: By using LLMs to analyze customer feedback, they launched a new product line that generated an additional $100,000 in the first year.

Indirect Benefits (Soft ROI):

1. Brand reputation: Their LLM-driven positive reviews led to an estimated $50,000 in new customer acquisitions through word of mouth.
2. Attracting talent: With their technological advancement, they attracted top-tier talent leading to faster project completions, estimated to be worth $100,000.

Total Gain: $800,000

By weighing both direct and indirect returns, businesses can get a comprehensive view of their overall performance, helping them understand the true value of investments like LLMs. Let’s now see the costs involved.

The costs of LLM implementation

• Training and fine-tuning: Tailoring LLMs for specific business needs requires extensive training. This not only needs powerful computing but also expert knowledge and specialized data.
• Infrastructure: Maintaining efficient LLMs means investing in top-tier digital systems, which include advanced servers, fast network setups, and regular hardware upgrades. For perspective, the OpenAI’s GPT-3 model, holding 175B parameters, needs over 300GB just for storage and a 16GB memory GPU for smooth operation.
• Regular updates: To keep LLMs running, you’ll need regular updates. These cover technical improvements and adjustments to business changes.
• Licensing and collaborations: Using certain LLMs might mean paying licensing fees. Companies might also need to partner with others or buy specialized data to make the model more precise and relevant.
• Expertise: Using and looking after LLMs means hiring or training experts. For instance, a data scientist in the US earns, on average, over $150,000 annually.

For a detailed breakdown of AI-related costs, check our Understanding the Total Cost of OpenAI blog post.

The previously mentioned retail company’s expenditure for LLM integration are the following:

1. Fine-tuning: OpenAI states that fine-tuning a GPT-3 turbo model with 100,000 tokens for three epochs costs $2.40. For our retail company, which used 1,000,000 tokens and did it thirty epochs, the basic cost is about $240. However, real-world setups have added costs like testing prompts, domain-specific expertise, tech changes, and unexpected expenses. With all this in mind, the company set aside $10,000 for the first year.
2. Infrastructure: Server costs and required systems amounted to $50,000.
3. Regular updates: $50,000 annually.
4. Licensing and collaborations: Initial licensing and data acquisition costs totaled $150,000.
5. Expertise: They hired two AI specialists at $150,000 each per year, totaling $300,000.

Total Cost: $560,000

So using the previously mentioned formula: ROI= (gain-cost)/cost x 100, they will have a ROI of around 42%.

Assessing the true worth of LLMs

As AI becomes more popular, tools like GPT-4 are changing how businesses work. For companies, it’s not just about using LLMs, but knowing their real benefits. When we try to see if these tools are worth the cost, we shouldn’t just look at the money saved or earned. We should also think about the long-term benefits, new opportunities they might bring, and how they affect a company’s reputation and customer relationships.