Got a hundred ideas for generative AI but not sure where to begin? Embarking on a generative AI project can be both exciting and challenging.
Recently, we’ve worked with clients to apply generative AI to a number of impactful use cases. We’ve seen positive outcomes from applying the technology using design thinking and getting feedback early by getting prototypes in front of users.
That’s why we developed a guide that draws on our real-world experience in this space. It highlights the key factors we’ve found help smooth the path from Proof of Concept (PoC ) to production.
1. Understand what LLMs do best
Start by recognising the proven strengths of Large Language Models (LLMs) in order to examine the opportunities and pain points that they are best suited to address. This will increase the chances of success before exploring more novel or complex use cases.
LLMs are well suited to tasks like information extraction, summarisation, labelling, evaluating, and conversational interactions. They are commonly extended with search capabilities, allowing them to consider additional information when generating a response.
Be mindful that the LLM is one component of the solution:
- The reasoning abilities of LLMs allows us to extend their capabilities by allowing them to interact with external services and APIs.
- Acknowledge the role of traditional web solutions in augmenting LLMs and allocate the appropriate time for the development of extended or custom functionality.
Also understand the strengths of LLMs are not without tradeoffs. Incorporating a LLM may increase latency, reduce determinism, and add variable costs. The relative size of these trade offs will vary depending on the use case.
Above all, be open minded and solution agnostic. Consider whether non-generative AI solutions could solve your problem effectively.
2. Identify and prioritise pain points across the business
Get engagement across the business by giving departments the opportunity to identify problems in their domain. Conduct workshops to understand the problem space and brainstorm potential applications of the technology. Include people with first-hand experience of problems identified.
The goal of the workshops should be to choose targeted problems that match the strengths of LLMs, define clear problem statements, and establish benchmarks and success metrics.
Work with problem owners and domain experts to:
- Help shape and validate what good output looks like. This information is really important for evaluating the performance of the model.
- Understand risks so the PoC can be designed to explore techniques that address these concerns.
- Score use cases based on their Desirability, Viability, and Feasibility. Use the totals to help prioritise use cases and select a few problems to explore deeply.
3. Make sure you are data ready
At a high level, understand your organisation’s stance on data governance. Identify the data that each PoC will depend upon. Data sources may include document stores, databases, spreadsheets, support tickets, wikis or emails.
Consider how access to your data is currently restricted and ensure that your solution will not break these restrictions. Generally, it is best to start with use cases that avoid complex access requirements or sensitive data.
Get familiar with your data so that you can inform the PoC scope and design:
- If there are multiple data sources consider narrowing the scope to a single data source, or using a smaller subset of data from each.
- Unstructured data may have inconsistent content, layout and formatting (e.g. tables, images) which can distract or confuse an LLM, so factor in additional time for extra processing steps.
- Where possible, seek out source data so you don’t have to spend extra time getting it in the right format.
4. Form a cross-functional team
Generative AI PoCs are not a purely technical exercise. A cross-functional team will be able to deliver the breadth of skills needed to comprehensively assess Desirability, Viability and Feasibility. Include generative AI experts, software engineers, experience designers, product specialists and business domain experts.
Implementing an LLM solution requires considering interaction types, managing LLM latency for user experience, and ensuring answer verification by providing sources. These kinds of problems often benefit when approached with a broader set of skills and perspectives.
The team is also able to work more effectively in parallel. For example, experience design can work closely with domain experts, develop a deep understanding of a problem to share with the development team and inform the solution design.
5. Sequencing can matter
Start with PoCs that address foundational issues or validate basic assumptions before moving onto more complex PoCs, which may build upon the previous findings. To help sequence PoCs consider which data sources are readily available, whether any PoCs depend on the output of another, and whether one PoC is a simpler implementation of a common pattern. Examples of common patterns in generative AI applications include retrieval (searching across a knowledge base), conversational interaction, multi-step reasoning and external tool use.
6. Get in front of users and stakeholders to iterate and improve
Creating a simple prototype is an effective way to seek regular user feedback and maintain engagement. Getting the prototype in front of users provides a better understanding of the actual use, intended use and importantly helps identify edge cases. Assessing user satisfaction with the solution’s capabilities or limitations will help prioritise development efforts. Conduct regular user testing and feed findings into subsequent iterations of the prototype. Regularly assess the PoC results against success criteria.
7. Build in observability to manage risk
Utilise observability tools specifically designed for LLMs, to log and monitor the inputs and outputs of every interaction. By capturing this information during the PoC you can track progress, costs, hallucinations, grounding and the effectiveness of guardrails. Some observability tools also capture user ratings and comments, and store it in an organised and accessible manner. Stored interactions could be applied to fine tune a model at a later date.
What’s next? Start your generative AI success story
Taking the leap into generative AI can transform your organisation, but it requires a thoughtful approach to ensure success. The learnings in this guide will help you navigate the complexities of a Generative AI PoC, from initial ideation to full-scale production. If you’re ready to explore how generative AI can create value for your business, let’s discuss how DiUS can support your journey with our expertise and proven methodologies.
Get in touch today to get started on your path to AI-driven innovation.
