My Journey towards the “Databricks Certified Generative AI Engineer Associate” Beta Exam

My Experiences from preparing for and successfully passing the New Databricks Generative AI (Beta) Exam

Last Update: 2024–12–23

𝗗𝗮𝘁𝗮𝗯𝗿𝗶𝗰𝗸𝘀 𝗖𝗲𝗿𝘁𝗶𝗳𝗶𝗲𝗱 𝗚𝗲𝗻𝗲𝗿𝗮𝘁𝗶𝘃𝗲 𝗔𝗜 𝗘𝗻𝗴𝗶𝗻𝗲𝗲𝗿 𝗔𝘀𝘀𝗼𝗰𝗶𝗮𝘁𝗲 — Axel Schwanke

It all started when I saw an email invitation to test for the new Databricks Certified Generative AI Engineer Associate exam in one of the Databricks webinars (I can’t remember which webinar) early April 2024.

Application for the Beta Exam

Since I was interested and had time (I’m currently on sabbatical) I sent an email indicating my interest. Shortly afterwards, I received a request to fill out a survey.

2024–04–09 Completing the Survey

Hello,

Thank you for your interest in our upcoming Beta Test of the Databricks Certified Generative AI Engineer Associate exam.

To be considered for participation in the Beta Test of this exam, you must complete and submit our survey before April 15, 2024, 6:00pm EDT/10:00pm UTC

Survey Link: XXX (no longer accepting responses)

As indicated in the survey, please read the attached Exam Guide PDF document to understand the scope of this exam when considering participation.

If selected, you will receive an email with further instructions and official beta dates.

Thank you,

Databricks Certification Team

Two weeks after completing the survey and answering questions about my experience with Databricks, I received an email with details of the beta exam.

2024–04–19 Beta Exam Acceptance Email

Dear Databricks Community Member,

The Databricks Certification Team is pleased to inform you that you have been accepted into the Beta Test of our new exam, Databricks Certified Generative AI Engineer Associate. Please carefully read the details in this email to successfully set up and complete your one, free exam attempt.

Reminder: This Beta Test exam is set at a maximum of 180 minutes with up to 135 multiple-choice questions and you will need to answer all presented items in one, uninterrupted session. Also, as this is a Beta Test of exam content, you may choose to add feedback notes to any item but this is optional.

Key Dates
Now that you have been accepted in this Beta Test, you must register for, schedule, and take the exam with our online exam delivery system within these timeframes:

Registration and Scheduling: Monday, April 24, 2024 — Friday, May 3, 2024

Exam Attempt: Tuesday, April 30, 2024 — Sunday, May 12, 2024

Prepare for your exam topics
Review the Exam Guide we sent you in the Survey invitation (Databricks Exam Guide — Generative AI Engineer Associate — BETA TEST.pdf) paying close attention to Recommended Preparation and Exam Outline.

Exam Guide (for Beta Exam)

The purpose of this exam guide is to give you an overview of the exam and what is covered on the exam to help you determine your exam readiness. This document will get updated anytime there are any changes to an exam (and when those changes will take effect on an exam) so that you can be prepared. This version covers the Beta Exam only and is subject to change at any time.

Audience Description

The Databricks Certified Generative AI Engineer Associate certification exam assesses an individual’s ability to design and implement LLM-enabled solutions using Databricks. This includes

• problem decomposition to break down complex requirements into manageable tasks as well as
• choosing appropriate models, tools, and approaches from the current generative AI landscape for developing comprehensive solutions.
• It also assesses Databricks-specific tools such as Vector Search for semantic similarity searches,
• Model Serving for deploying models and solutions,
• MLflow for managing solution lifecycle, and
• Unity Catalog for data governance.

Individuals who pass this exam can be expected to build and deploy performant RAG applications and LLM chains that take full advantage of Databricks and its toolset.

About the (Beta) Exam

• Number of items: 135 multiple-choice or multiple-selection questions, including standard items and extra beta-test items for the beta test only (instead of 45 questions)
• Time Limit: 180 minutes, for the beta test only (instead of 90 minutes)
• Registration fee: The single exam attempt for the beta test is free of charge (instead of $200)
• Prerequisite: None required; course attendance and six months of hands-on experience in Databricks is highly recommended. Also, see Recommended Preparation in this document.
• Validity: 2 years. Beta Note: Results will not be immediately available at exam attempt end but beta testers who are successful in their will receive the full credential at project end.
  Results can take 4–6 weeks.

Recommended Preparation

• All current Databricks Academy courses related to the Generative AI learner role
• Knowledge of current LLM’s and their capabilities
• Knowledge of prompt engineering, prompt generation, and evaluation
• Knowledge of current related online tools and services like LangChain, Hugging Face Transformers, etc.
• Working knowledge of Python and its libraries that support RAG application and LLM chain development
• Working knowledge of current APIs for data preparation, model chaining, etc.
• Relevant Databricks Documentation resources

Exam outline

Section 1: Design Applications

• Design a prompt that elicits a specifically formatted response
• Select model tasks to accomplish a given business requirement
• Select chain components for a desired model input and output
• Recognize intents to be delivered
• Define and order tools that gather knowledge or take actions for multi-stage reasoning

Section 2: Data Preparation

• Apply chunking strategy for a given document structure and model constraints
• Filter extraneous content in source documents that degrades quality of a RAG application
• Choose the appropriate Python package to extract document content from provided source data and format.
• Define operations and sequence to write given chunked text into Delta Lake tables in Unity Catalog
• Identify needed source documents that provide necessary knowledge and quality for a given RAG application
• Identify prompt/response pairs that align with a given model task
• Use tools and metrics to evaluate retrieval performance

Section 3: Application Development

• Create tools needed to extract data for a given data retrieval need
• Select Langchain/similar tools for use in a Generative AI application.
• Identify how prompt formats can change model outputs and results
• Extract features from qualitative responses to evaluate model performance, safety, and quality.
• Select chunking strategy based on model & retrieval evaluation
• Augment a prompt with additional context from a user’s input based on key fields, terms, and intents
• Create a prompt that adjusts an LLM’s response from a baseline to a desired output
• Implement LLM guardrails to prevent negative outcomes
• Write metaprompts that minimize hallucinations or leaking private data
• Build agent prompt templates exposing available functions
• Select the best LLM based on the attributes of the application to be developed
• Select an embedding model context length based on source documents, expected queries, and optimization strategy
• Select a model for from a model hub or marketplace for a task based on model metadata/model cards
• Select the best model for a given task based on common metrics generated in experiments

Section 4: Assembling and Deploying Applications

• Introduce custom code into a pyfunc model to preprocess prompts before they are sent to an LLM
• Control access to resources from model serving endpoints
• Code a simple chain according to requirements
• Code a simple chain using langchain
• Choose the basic elements needed to create a RAG application: model flavor, embedding model, retriever, dependencies, input examples, model signature
• Register the model to Unity Catalog using MLflow
• Sequence the steps needed to deploy an endpoint for a basic RAG application
• Create and query a Vector Search index
• Size the endpoint needed for serving a RAG application to the target user population
• Identify resources needed to serve features for a RAG application

Section 5: Governance

• Use masking techniques as guard rails to meet a performance objective
• Protect against malicious prompts that bypass system prompt guardrails
• Recommend an alternative for problematic text mitigation in a data source feeding a RAG application
• Use legal/licensing requirements for data sources to avoid legal risk

Section 6: Evaluation and Monitoring

• Select an LLM choice (size and architecture) based on a set of quantitative evaluation metrics
• Select key metrics to monitor for a specific LLM deployment scenario
• Evaluate model performance in a RAG application using MLflow
• Use inference logging to assess deployed RAG application performance
• Use Databricks features to control LLM costs for RAG applications

Preparation for the (Beta) Exam

Based on the exam guide I mainly used the following courses to prepare for the exam:

Generative AI Fundamentals

This short course introduces organizations to generative AI models, focusing on large language models (LLMs). It covers what generative AI is, how LLM applications work, and how Lakehouse AI can drive success. Additionally, it addresses essential considerations for AI adoption and evaluates potential risks and challenges associated with generative AI usage.

Generative AI Engineering with Databricks (2023)

This course is aimed at data scientists, machine learning engineers, and other data practitioners looking to build LLM-centric applications with the latest and most popular frameworks. In this course, you will build common LLM applications using Hugging Face, develop retrieval-augmented generation (RAG) applications, create multi-stage reasoning pipelines using LangChain, fine-tune LLMs for specific tasks, assess and address societal considerations of using LLMs, and learn how to deploy your models at scale leveraging LLMOps best practices.

By the end of this course, you will have built an end-to-end LLM workflow that is ready for production.

Update June 2024: Generative AI Engineering with Databricks (2024)

This course, updated June 2024, is the latest version of the Generative AI Engineering with Databricks course. It is aimed at data scientists, machine learning engineers, and other data practitioners looking to build generative AI applications with the latest and most popular frameworks and Databricks capabilities. Below, we describe each of the four, four-hour modules included in this course.

Generative AI Solution Development: This is your introduction to contextual generative AI solutions using the retrieval-augmented generation (RAG) method. First, you’ll be introduced to RAG architecture and the significance of contextual information using Mosaic AI Playground. Next, we’ll show you how to prepare data for generative AI solutions and connect this process with building a RAG architecture. Finally, you’ll explore concepts related to context embedding, vectors, vector databases, and the utilization of Mosaic AI Vector Search.

Generative AI Application Development: Ready for information and practical experience in building advanced LLM applications using multi-stage reasoning LLM chains and agents? In this module, you’ll first learn how to decompose a problem into its components and select the most suitable model for each step to enhance business use cases. Following this, we’ll show you how to construct a multi-stage reasoning chain utilizing LangChain and HuggingFace transformers. Finally, you’ll be introduced to agents and will design an autonomous agent using generative models on Databricks.

Generative AI Application Evaluation and Governance: This is your introduction to evaluating and governing generative AI systems. First, you’ll explore the meaning behind and motivation for building evaluation and governance/security systems. Next, we’ll connect evaluation and governance systems to the Databricks Data Intelligence Platform. Third, we’ll teach you about a variety of evaluation techniques for specific components and types of applications. Finally, the course will conclude with an analysis of evaluating entire AI systems with respect to performance and cost.

Generative AI Application Deployment and Monitoring: Ready to learn how to deploy, operationalize, and monitor generative deploying, operationalizing, and monitoring generative AI applications? This module will help you gain skills in the deployment of generative AI applications using tools like Model Serving. We’ll also cover how to operationalize generative AI applications following best practices and recommended architectures. Finally, we’ll discuss the idea of monitoring generative AI applications and their components using Lakehouse Monitoring.

Creating my first Generative AI Application

I already had my first experience with LLM development last year (2023): Based on the Databricks Solution Accelerator Enhancing Product Search With Large Language Models (LLMs) I did a feasibility study on “Large Language Models (LLM) for Property Search” in Sept. 2023 to:

• Investigating the use of open source LLMs in the real-estate sector
• Assessing the capabilities of LLMs
• Identifying the challenges of LLM development

Enhancing product search with large language models (LLMs)

The Databricks Solution Accelerator includes several notebooks to

1. set up a basic search model using only the product descriptions
2. fine-tune the model using labeled search results
3. package both models for deployment using Databricks model serving

Development steps of the LLM model for basic property search

1. Prepare the property descriptions
2. Download a pre-trained, open source LLM model
3. Use the model to convert the property descriptions into embeddings
4. Configure the model to use these embeddings as the knowledge base to which it directs its search
5. Deploy the model as a micro-service that can be easily integrated into applications — using Databricks Model Serving

Model Development and Serving

My Experience with LLM development based on the Solution Accelerator

• Getting started with the development was straightforward, requiring only the download of the notebooks from the Databricks solution accelerator.
• The adaptation of the Solution Accelerator for the real estate search proved to be very simple — apart from the property data preparation
• It was also easy to adapt the development process to a German LLM
• The management of multiple model versions and the transfer of selected versions to production is very efficiently supported by Databricks Model Serving
• The service provided is able to respond to search queries within ten to one hundred milliseconds — ready for production
• It took me about 4 weeks to get it all done, with most of the time spent on the first step: “preparing the real estate product data”

Recommendation: Don’t hesitate to gain your first real-world experience in LLM development with a Databricks LLM Solution Accelerator. For me, it was the best preparation for the Generative AI exam.

Databricks Solution Accelerators - Deliver Data & AI Value Faster - Databricks

Additional recommendations from Lara Rachidi (Solution Architect at Databricks)

Here’s how you can prepare for the exam:

• Review the exam preparation guide, which includes 5 sample questions
• Register for the exam

For additional preparation, check out the videos with Maria Zervou and the ones by Youssef Mrini such as:

• Vector Lab — Databricks Blog Posts Highlights — April 2024
• Vector Lab — Transformers explained in 5 minutes
• Getting Started with Databricks Vector Search
• Getting started with Databricks Foundation model APIs and External Models

We’ll be posting more videos on ML topics in the coming weeks… Stay tuned and make sure to subscribe to the NextGenLakehouse and VectorLab Youtube channels.

Here’s a summary of some of the topics you need to study (non-exhaustive):

• Filtering for RAG applications and ensuring quality
• Chunking strategies
• Using LLM as a judge
• Choosing the right LLM (size and architecture)
• Selecting your embedding model
• Using langchain and chain components
• Creating and querying a Vector Search index
• Serving features for a RAG application
• Prompt engineering and formats
• Python packages for document content extraction
• Evaluating retrieval performance and response quality
• Assessing model performance in a RAG application using MLflow
• Working with LLM Agents
• Controlling access to resources from model serving endpoints
• Registering models to Unity Catalog using MLflow
• Using masking techniques
• Using inference logging to assess deployed RAG application performance
• Leveraging Databricks features to manage LLM costs for RAG applications

I’d also recommend taking a look at dbdemos.ai, particularly this demo: Deploy Your LLM Chatbot With Retrieval Augmented Generation (RAG), DBRX Instruct Foundation Models and Vector Search. It includes notebooks on how to use LLM as a judge for evaluation, how to use inference tables:

Deploy Your LLM Chatbot With Retrieval Augmented Generation (RAG), Foundation Models and Vector…

Additional tips from Priyanka Mane (Data & AI Specialist at Accenture)

Over the past few months, I have delved into exploring Generative AI and RAG app development. This certification has been instrumental in covering the key aspects of building GenAI applications.

𝐊𝐞𝐲 𝐭𝐚𝐤𝐞𝐚𝐰𝐚𝐲𝐬 :

𝐏𝐫𝐨𝐦𝐩𝐭 𝐞𝐧𝐠𝐢𝐧𝐞𝐞𝐫𝐢𝐧𝐠- System and User prompt, Managing hallucinations, few shot prompting
𝐃𝐚𝐭𝐚 𝐏𝐫𝐞𝐩𝐚𝐫𝐚𝐭𝐢𝐨𝐧 -semi/unstructured data preparation, parsing, chunking strategy (identifying optimal chunking size),selecting embedding models ,tokenization
𝐃𝐞𝐬𝐢𝐠𝐧𝐢𝐧𝐠 𝐆𝐞𝐧𝐀𝐈 𝐀𝐩𝐩𝐥𝐢𝐜𝐚𝐭𝐢𝐨𝐧𝐬 -RAG-based applications, LLM chains like langchain, multi-stage reasoning applications
𝐀𝐩𝐩𝐥𝐢𝐜𝐚𝐭𝐢𝐨𝐧 𝐃𝐞𝐯𝐞𝐥𝐨𝐩𝐦𝐞𝐧𝐭 -model selection, retrieval mechanisms, vector stores, feature stores, in-context learning, generation mechanisms
𝐀𝐬𝐬𝐞𝐦𝐛𝐥𝐢𝐧𝐠 𝐚𝐧𝐝 𝐃𝐞𝐩𝐥𝐨𝐲𝐢𝐧𝐠 𝐀𝐩𝐩𝐬 -Compound AI systems, AI Agents, multi-stage reasoning, Agent Reasoning techniques like ReAct, multi-agent collaboration, model serving, vector search endpoints, LLMOps
𝐆𝐨𝐯𝐞𝐫𝐧𝐚𝐧𝐜𝐞 -masking, PII handling, access control, Unity Catalog governance model, legal implications on using Dataset, using LLamaguard for identifying ethical prompts and responses.
𝐄𝐯𝐚𝐥𝐮𝐚𝐭𝐢𝐨𝐧 𝐚𝐧𝐝 𝐌𝐨𝐧𝐢𝐭𝐨𝐫𝐢𝐧𝐠 -bias and oversight, model metrics like perplexity, toxicity, and cost/performance metrics, context metrics such as context precision, context relevance, context recall, response based metrics such as answer similarity, answer correctness, Benchmark evaluation, using LLM as a judge, inferencing, lakehouse monitoring

𝐓𝐢𝐩𝐬 𝐟𝐨𝐫 𝐩𝐚𝐬𝐬𝐢𝐧𝐠 𝐭𝐡𝐞 𝐞𝐱𝐚𝐦:

• Take all current Databricks Academy courses related to the Generative AI specifically, Generative AI Engineering with Databricks
• Review the Databricks documentation on Databricks tools and libraries

Tips from Dipendu Chanda: Databricks Generative AI Engineer Associate Certification: Study Guide Part 1

Part 1 of this guide primarily focuses on design applications, data preparation, and effective deployment strategies for AI-driven projects.

Design Applications

1. Prompt Design and Engineering: This section underscores the importance of crafting precise and clear prompts tailored for specific tasks. Techniques such as zero-shot and few-shot prompting, using delimiters, and structured outputs are emphasized. Effective prompt design involves avoiding bias and hallucinations by including instructions like “Do not make things up.”
2. Task Selection and Model Mapping: Identifying business objectives and breaking them down into manageable tasks are critical. This involves choosing appropriate AI models based on requirements (e.g., MPT, GPT) and considering interactions between tasks. Frameworks like LangChain are recommended for multi-stage reasoning workflows.
3. Chain Components: Key components include prompts, retrievers, tools, and language models (LLMs). Frameworks such as LangChain, LLamaIndex, and OpenAI agents are utilized for building efficient processing chains. Integrating these components with databases and external APIs enhances the functionality of generative AI applications.

Data Preparation Techniques

1. Chunking Strategies: Text chunking is vital for managing large documents within the model’s context window. Techniques include context-aware and fixed-size chunking, which help maintain context across different document types. Advanced methods like windowed summarization ensure comprehensive coverage.
2. Data Cleaning and Quality: Filtering extraneous content and preprocessing data improve the quality of Retrieval-Augmented Generation (RAG) applications. Tools like PyPDF and OCR methods are necessary for extracting data from various file formats.

Application Development and Deployment

1. Using LangChain: LangChain simplifies the creation and management of chains that involve data retrieval, transformation, and model inference. This ensures scalable, modular AI solutions.
2. Evaluation and Optimization: Continuous performance monitoring, using tools like MLflow, is essential for maintaining accuracy and optimizing AI workflows. Implementing guardrails and prompt enhancements minimizes risks like data leakage and hallucinations.

By mastering these key areas, professionals can effectively deploy and manage generative AI solutions on the Databricks platform, ensuring high-quality, reliable outcomes.

Databricks Generative AI Engineer Associate Certification: Study Guide Part 1

2024–04–30 Exam

The exam was designed to be completed within 180 minutes, allowing sufficient time to carefully answer each of the 135 multiple-choice questions. Despite the comprehensive nature of the exam, I was able to complete it in about 145 minutes.

The questions were varied and covered different aspects of generative AI, including:
- Design Applications
- Data Preparation
- Application Development
- Assembling and Deploying Apps
- Governance
- Evaluation and Monitoring

2024–05–31 Received Information on passing the Exam

On a very rainy morning, May, 31, 2024, I received the email informing me that I had successfully passed the exam:

Dear Databricks Certification Beta Tester,

We are pleased to inform you that you have successfully passed the Databricks Certified Generative AI Engineer Associate exam that you took during our recent Beta Test. You are now a certified Databricks Generative AI Engineer Associate and one of the first people to achieve this new certification.

You will receive a notification from our credentialing system soon with the details of your newly-awarded certification. We encourage you to share your success in your networks by linking it to your social media accounts. More information about this is located here.

Our team would like to thank you for your input and effort during the Beta Test. We do appreciate your effort and are excited for your success.

Sincerely,

Databricks Certification Team

Databricks Certified Generative AI Engineer Associate

Databricks Certified Generative AI Engineer Associate

The Databricks Certified Generative AI Engineer Associate certification exam assesses an individual’s ability to design and implement LLM-enabled solutions using Databricks. This includes problem decomposition to break down complex requirements into manageable tasks as well as choosing appropriate models, tools, and approaches from the current generative AI landscape for developing comprehensive solutions. It also assesses Databricks-specific tools such as Vector Search for semantic similarity searches, Model Serving for deploying models and solutions, MLflow for managing solution lifecycle, and Unity Catalog for data governance. Individuals who pass this exam can be expected to build and deploy performant RAG applications and LLM chains that take full advantage of Databricks and its toolset.

Conclusion

My journey from receiving the email invitation to passing the Databricks Certified Generative AI Engineer Associate exam was an enriching experience. The extensive preparation materials, including Databricks Academy courses, solution accelerators and expert recommendations, provided a solid foundation for success. This certification is not only a testament to one’s skills in generative AI, but also a gateway to innovative opportunities in the field.

I encourage everyone to take the Generative AI exam and utilize the many preparation resources available. Whether you are relatively new to the field of AI or already have experience, this exam offers a comprehensive assessment of your skills and a chance to be part of the groundbreaking advances in AI technology.

Take the challenge, prepare diligently and become part of the growing community of certified Databricks Generative AI Engineer Associates.

References

Databricks Certified Generative AI Engineer Associate | Databricks

Establish your Generative AI expertise with the latest Databricks certification

Generative AI Engineering Pathway

Databricks Certified Generative AI Engineer Associate Guide (pdf)

Generative AI Engineering with Databricks

The Big Book of Generative AI

If you liked the post, please clap and follow me on Medium and LinkedIn!