What We Learnt at AWS GenAI Deep Dive Days

Last week, we attended the Generative AI Deep Dive Days event organised by AWS. Over two days, we saw a series of talks, demos, and workshops covering various aspects of the Generative AI services that AWS offers. It gave us a chance to learn the theory behind AWS’s approach, understand the capabilities made possible by different solutions, and put things into practice during the workshop.

Here, we summarise our highlights and key findings from the event having come away feeling that while there is value to be gained in the short term, the technology still needs refining.

Generative AI at Different Levels of Abstraction

AWS has developed a number of generative AI services aimed at different levels of abstraction. This persona-focused approach means there’s something for everyone. Starting at the highest level of abstraction, there is Q for Business and Q for Developers — these are the typical co-pilot / assistant services.

If you’re looking for a more tailored solution, AWS offers Bedrock, a comprehensive platform for deploying and managing generative AI models. It is particularly useful for businesses concerned about data privacy, as AWS ensures that your training and validation data, as well as prompts and completions, are not used to train its models or shared with third parties.

Going another level deeper, there is Trainium for fine-tuning as well as pre-training, and Inferentia for running inference workloads.

Boosting Productivity with Q for Developers

Q for Developers is AWS’s co-pilot that can be used within AWS Console, your IDE, and AWS documentation. This tool can boost productivity by automating code suggestions, generating entire scripts, and citing relevant documentation. It also offers tailored capabilities such as security scans or Java 8 to 17 upgrades.

After spending some time experimenting with Q for Developers, we’ve noticed a few things.

• It can definitely increase your productivity if you already know what you’re doing.
• Code generation is often decent, requiring no or only minor tweaks when requirements are simple, and giving up when it gets more complex.
  Its access to AWS documentation comes in handy when trying to figure out AWS specific problems.

However, it is still important to know what ‘good code’ looks like to get the most out of Q for Developers. I wouldn’t trust it to generate whole scripts in a language I’m not familiar with.

Similarly, when the questions venture off the beaten path, Q often struggles to fail gracefully. It often cites unanswered questions on AWS forums as solutions that it makes up.

Bedrock: the Foundation of AI Solutions

AWS Bedrock serves as a one-stop shop for all your generative AI model management needs. It supports deploying agents and other AI solutions efficiently, handling infrastructure for you.

Bedrock contains a set of tools one might need to build with LLMs and other foundational models, including knowledge-bases for RAG, integration with Lambda Functions for agent workflows, and guardrails.

Guardrails is another feature that is often required for productionisation of Generative AI. The built-in capabilities are easy to manage, but often fall short of expectations.

For example, using PII guardrail — masking addresses, masks any location information. Asking a chatbot with such guardrails for a list of countries will return a masked list (with some exceptions where some countries slip through the filter).

Bedrock provides a set of tools that almost work out-of-the-box, but will require significant effort to make them production-ready.

Not One Model to Rule Them All

The two day event concluded with a talk from HuggingFace about their experience with helping enterprises use LLM and other Generative AI technologies. The main takeaway from the talk is that fine-tuning smaller models is more cost effective than using the biggest models.

Thanks to the long-standing relationship between AWS and HuggingFace, it is very easy to use models found on their extensive repositories, and fine-tuning them using SageMaker.

The standardised HuggingFace API makes for easy set-up (they even compile most popular models). The standardisation also enables easy switching of models. One thing to keep in mind is the overengineering of prompts — resulting in significant drop in performance when switching to a superior model (at least on paper).

Prompts should be tailored to the use case, but not over-optimised for a specific mode; as one thing we’ve learned over the last few years is that a more effective model is just around the corner.

Solutions Looking for a Problem?

The two days event was filled with incredible demonstrations of the technology. Yet, the one thing that was missing was examples of it being applied to more than co-pilots and Q&A assistants.

To answer this, Byron (from Mesh-AI) highlighted the importance of focusing on bounded problems to derive immediate value from generative AI. Instead of aiming for broad, all-encompassing solutions, businesses should target specific, well-defined problems where generative AI can make a significant impact.

This pragmatic approach ensures that AI implementations are not only effective but also deliver tangible benefits in the short term.

Conclusion

The AWS Generative AI Deep Dive Days provided an in-depth look at the powerful capabilities across AWS’s generative AI stack. From high-level co-pilot tools like Q for Business and Q for Developers to the comprehensive Bedrock platform for model management, AWS offers solutions tailored to various needs.

However, the technology is still rough around the edges, making it difficult to productionise within the Enterprise. For generative AI to deliver substantive value in the short term, the focus needs to shift from newest, biggest models to fine-tuned smaller models, and move away from fantastical use cases to more bounded problems.