(Part 1) Generative AI for Business Leaders: Lessons from the Trenches

Preface:
After nearly two years of implementing Generative AI solutions in Enterprise environments I’ve learned a thing or two about the practical realities of GenAI adoption. The gotchas and some golden nuggets of wisdom from hard earned experience. Throughout this series I’ll do my best to share these lessons in a format suitable for business leaders with little technical experience in the space. My goal is to cut through the jargon and complexity to sharpen your intuition and empower your decision making for this once in a generation technology. Alright, let’s get to it…

LLMs(Large Language Models) are a ‘commodity’.

Lebron James, Nahendra Modi and Brad Pitt all use the same version of ChatGPT as you and I.

Countless companies connected to OpenAI GPT3.5/4/4-o over the past year to create internal demos. Guess what? They’re all using the same model as everyone else.

The model itself is not what’s going to create unique value for you. The opportunity lies in the unique context(information) you provide the model.

“Context is a frame that surrounds an event and provides resources for its appropriate interpretation.” — Wikipedia

This “blank state” is both a blessing and a curse. It means you have the power to generate anything but it also means you run the risk of generating anything(things you don’t want).

When building Generative AI solutions, this means that much of our time is spent on transforming the user’s prompt into context that will result in the best output by the LLM.

The Fine-tuning Fallacy

Unless you’re training a model from scratch(you’re not and you shouldn’t), this means you really only have one way to customize anything outside of the context — fine tune an existing model.

While fine tuning gets a lot of press, it’s not practical for 95% of applications. If you’re just starting your journey, please don’t even entertain this discussion.

So that leaves you with one lever to create unique value — the context you send to the model. While this may sound like a simple problem, it couldn’t be further from the truth(if you want to build something high quality that can scale).

Context Crafting: An Art & Science

We have 4 main strategies for crafting the context that goes into the LLM.

1. Prompt Engineering(non-technical) → crafting context for the model in the UI only(think: ChatGPT text input above).
2. Retrieval Augmented Generation(RAG) → bringing data to the model that it was never trained on.
3. Prompt Engineering(technical) → using backend engineering to craft context for the model based on a user prompt.
4. Agents → an AI system that leverages the capabilities of an LLM to perform complex tasks autonomously or semi-autonomously.

Instead of speaking in the abstract, I’ll do my best to keep an example of a personalized travel itinerary planner as the thru-line when we cover each of these strategies. Each new strategy will be introduced to improve the existing system in an effort to improve the quality of the generated travel itinerary.

At a high level, we will cover:

1. Non-Technical Prompt Engineering

• What it is: Crafting effective prompts within the user interface.
• Business Impact: Enables non-technical teams to improve AI interactions.
• Example: Creating templates for users to input travel preferences to ensure a higher level of consistency.
• When to use: This is our starting point, ideal for quickly improving user interactions without significant technical investment.

2. Retrieval Augmented Generation (RAG)

• What it is: Enhancing AI responses with your organization’s proprietary data.
• Business Impact: Leverages your unique information assets to create differentiated AI solutions.
• Example: Incorporating a database of travel destinations, user reviews, and seasonal information to enhance the AI’s recommendations.
• When to use: As you accumulate more proprietary data and want to leverage it for more accurate and personalized itineraries.

3. Technical Prompt Engineering

• What it is: Using backend systems to dynamically craft and optimize AI prompts.
• Business Impact: Enables more sophisticated, context-aware AI interactions.
• Example: Breaking down a complex travel request (e.g., “Plan a multi-city European tour”) into a series of optimized queries for each aspect of the trip.
• When to use: When you need more sophisticated planning capabilities that exceed current models reasoning capabilities.

4. AI Agents

• What it is: Creating semi-autonomous AI systems that can perform complex tasks and dynamically access external systems based on objectives.
• Business Impact: Automates demanding processes, potentially transforming entire business operations.
• Example: An AI system that not only plans trips but also books flights, hotels, and activities based on user preferences, real-time pricing, and availability of data.
• When to use: For creating advanced, semi-autonomous systems that can handle complex, multi-step tasks with minimal human intervention.

**This is an overly simplified depiction but bear with me…

Throughout this 5-part series we’ll break down key techniques for implementing these 4 strategies along with practical use cases, challenges to look out for and guidance for getting started. These will be sequenced(generally speaking) from simplest to most complex to implement. Although the sequence may vary based on use case, this is the most common progression I’ve seen adopted by clients.

Part 1: Generative AI for business leaders(you are here)

Part 2: Frontend(UI) Prompt Engineering

• Frontend prompt templates → Predefined “fill in the blank” templates.
• Chain of Thought(CoT) → Breaking complex reasoning down into steps.
• Multi-shot → Providing examples to guide the model.

Part 3: RAG

• Metadata/Entity Extraction → Extract data types and relationships.
• Hybrid Retrieval → Combine keyword and semantic search.
• Re-ranking → Reorder results by relevance.
• Multi-step Retrieval → Refine queries iteratively.
• Graph Retrieval → Retrieve info using graph embeddings.

Part 4: Backend Prompt Engineering

• Backend routing to templates → Routing user queries to pre-defined templates on the backend.
• Prompt decomposition → Breaking down a complex prompt into multiple steps.
• Structuring Inputs & Outputs → Providing clear and specific formats for the input data and the expected output.

Part 5: Agents

• Structured Output Formatting → Implementing standardized output formats, such as JSON templates, to ensure consistency, reliability, and ease of integration.
• Dynamic Information Gathering → Equipping agents with the ability to access and utilize up-to-date information from defined sources.
• Task Decomposition and Planning → Breaking complex tasks into manageable steps and creating a structured plan for execution.
• Advanced Reasoning Framework → Problem-solving strategies like ReAct, Tree of Thoughts, and Reflexion to enhance an agent’s ability to reason.

My goal here is to cut through the jargon and complexity to provide an introduction for non-technical business leaders. There are MANY things I haven’t touched on, including evaluating your responses, ensuring security/compliance, common “tooling stacks” for each strategy etc. Let me know if you’re especially interested in any topic I haven’t touched on and I’ll prioritize my writing based on demand.

Next up: (Part 2) Practical Prompting: Strategies to drive ROI