LLM TWIN COURSE: BUILDING YOUR PRODUCTION-READY AI REPLICA

The Role of Feature Stores in Fine-Tuning LLMs

From raw data to instruction dataset

→ the 6nd out of 12 lessons of the LLM Twin free course

What is your LLM Twin? It is an AI character that writes like yourself by incorporating your style, personality and voice into an LLM.

Image by DALL-E

Why is this course different?

By finishing the “LLM Twin: Building Your Production-Ready AI Replica” free course, you will learn how to design, train, and deploy a production-ready LLM twin of yourself powered by LLMs, vector DBs, and LLMOps good practices.

Why should you care? 🫵

→ No more isolated scripts or Notebooks! Learn production ML by building and deploying an end-to-end production-grade LLM system.

What will you learn to build by the end of this course?

You will learn how to architect and build a real-world LLM system from start to finish — from data collection to deployment.

You will also learn to leverage MLOps best practices, such as experiment trackers, model registries, prompt monitoring, and versioning.

The end goal? Build and deploy your own LLM twin.

What is an LLM Twin? It is an AI character that learns to write like somebody by incorporating its style and personality into an LLM.

The architecture of the LLM twin is split into 4 Python microservices:

1. the data collection pipeline: crawl your digital data from various social media platforms. Clean, normalize and load the data to a NoSQL DB through a series of ETL pipelines. Send database changes to a queue using the CDC pattern. (deployed on AWS)
2. the feature pipeline: consume messages from a queue through a Bytewax streaming pipeline. Every message will be cleaned, chunked, embedded (using Superlinked), and loaded into a Qdrant vector DB in real-time. (deployed on AWS)
3. the training pipeline: create a custom dataset based on your digital data. Fine-tune an LLM using QLoRA. Use Comet ML’s experiment tracker to monitor the experiments. Evaluate and save the best model to Comet’s model registry. (deployed on Qwak)
4. the inference pipeline: load and quantize the fine-tuned LLM from Comet’s model registry. Deploy it as a REST API. Enhance the prompts using RAG. Generate content using your LLM twin. Monitor the LLM using Comet’s prompt monitoring dashboard (deployed on Qwak)

LLM twin system architecture [Image by the Author]

Along the 4 microservices, you will learn to integrate 3 serverless tools:

• Comet ML as your ML Platform;
• Qdrant as your vector DB;
• Qwak as your ML infrastructure;

Who is this for?

Audience: MLE, DE, DS, or SWE who want to learn to engineer production-ready LLM systems using LLMOps sound principles.

Level: intermediate

Prerequisites: basic knowledge of Python, ML, and the cloud

How will you learn?

The course contains 10 hands-on written lessons and the open-source code you can access on GitHub, showing how to build an end-to-end LLM system.

Also, it includes 2 bonus lessons on how to improve the RAG system.

You can read everything at your own pace.

→ To get the most out of this course, we encourage you to clone and run the repository while you cover the lessons.

Costs?

The articles and code are completely free. They will always remain free.

But if you plan to run the code while reading it, you must know that we use several cloud tools that might generate additional costs.

The cloud computing platforms (AWS, Qwak) have a pay-as-you-go pricing plan. Qwak offers a few hours of free computing. Thus, we did our best to keep costs to a minimum.

For the other serverless tools (Qdrant, Comet), we will stick to their freemium version, which is free of charge.

Meet your teachers!

The course is created under the Decoding ML umbrella by:

• Paul Iusztin | Senior ML & MLOps Engineer
• Alex Vesa | Senior AI Engineer
• Alex Razvant | Senior ML & MLOps Engineer

🔗 Check out the code on GitHub [1] and support us with a ⭐️

Lessons

→ Quick overview of each lesson of the LLM Twin free course.

The course is split into 12 lessons. Every Medium article will be its own lesson:

1. An End-to-End Framework for Production-Ready LLM Systems by Building Your LLM Twin
2. The Importance of Data Pipelines in the Era of Generative AI
3. Change Data Capture: Enabling Event-Driven Architectures
4. SOTA Python Streaming Pipelines for Fine-tuning LLMs and RAG — in Real-Time!
5. The 4 Advanced RAG Algorithms You Must Know to Implement
6. The Role of Feature Stores in Fine-Tuning LLMs
7. How to fine-tune LLMs on custom datasets at Scale using Qwak and CometML
8. Best Practices When Evaluating Fine-Tuned LLMs
9. Architect scalable and cost-effective LLM & RAG inference pipelines
10. How to evaluate your RAG pipeline using the RAGAs Framework
11. [Bonus] Build a scalable RAG ingestion pipeline using 74.3% less code
12. [Bonus] Refactoring the 4 Advanced RAG Algorithms using Superlinked… WIP

To better understand the course’s goal, technical details, and system design → Check out Lesson 1

Let’s start with Lesson 6 ↓↓↓

Lesson 6: Prepare Finetuning Dataset

Large language models (LLMs) have changed how we interact with machines. These powerful models have a remarkable understanding of human language, enabling them to translate text, write different kinds of creative content formats, and answer your questions in an informative way.

But how do we take these LLMs and make them even better?

The answer lies in fine-tuning.

Fine-tuning is the process of taking a pre-trained LLM and adapting it to a specific task or domain.

One important aspect of fine-tuning is dataset preparation.

Remember the quote from 2018: “garbage in, garbage out.”

The quality of your dataset directly impacts how well your fine-tuned model will perform.

Why Data Matters?

Let’s explore why a well-prepared, high-quality dataset is essential for successful LLM fine-tuning:

• Specificity is Key: LLMs like Mistral are trained on massive amounts of general text data. This gives them a broad understanding of language, but it doesn’t always align with the specific task you want the model to perform. A carefully curated dataset helps the model understand the nuances of your domain, vocabulary, and the types of outputs you expect.
• Contextual Learning: High-quality datasets offer rich context that the LLM can use to learn patterns and relationships between words within your domain. This context enables the model to generate more relevant and accurate responses for your specific application.
• Avoiding Bias: Unbalanced or poorly curated datasets can introduce biases into the LLM, impacting its performance and leading to unfair or undesirable results. A well-prepared dataset helps to mitigate these risks.

Today, we will learn how to generate a custom dataset for our specific task: content generation.

Understanding the Data Types

Our data consists of two primary types: posts and articles. Each type serves a different purpose and is structured to accommodate specific needs:

• Posts: Typically shorter and more dynamic, posts are often user-generated content from social platforms or forums. They are characterized by varied formats and informal language, capturing real-time user interactions and opinions.
• Articles: These are more structured and content-rich, usually sourced from news outlets or blogs. Articles provide in-depth analysis or reporting and are formatted to include headings, subheadings, and multiple paragraphs, offering comprehensive information on specific topics.
• Code: sSourced from repositories like GitHub, this data type encompasses scripts and programming snippets crucial for LLMs to learn and understand technical language

Both data types require careful handling during insertion to preserve their integrity and ensure they are stored correctly for further processing and analysis in MongoDB. This includes managing formatting issues and ensuring data consistency across the database.

Finetuning Dataset Generation Process [Image by the Author]

Table of Contents

1. System Design and Data Flow
2. Insert Data into MongoDB
3. Qdrant VectorDB — storing cleaned data
4. Generate Finetuning Dataset
5. CometML — data versioning

🔗 Check out the code on GitHub [1] and support us with a ⭐️

1. System Design and Data Flow

In a real-world application, data is not manually inserted into MongoDB but flows through a well-structured Data Collection Pipeline. This section explains how the data is integrated into the system.

In Lesson 2 of LLM Twin Course we explained the importance of data pipelines in an AI project:

• Data Collection: Initially, data is automatically collected from various sources such as social media platforms, news feeds, and other digital channels. This automated process ensures a continuous stream of fresh data that feeds into the pipeline.

In Lesson 4 of LLM Twin Course we learnt how to build a real time feature pipeline which:

• leverage Change Data Capture (CDC)
• consume data from a RabbitMQ queue
• use a Bytewax streaming engine to process data streams used for fine-tuning LLMs & RAG
• load fine-tuning & RAG data to a Qdrant vector DB

We also emphasize the importance of having a feature store like Qdrant, in a production environment:

• Data Cleaning: Once collected, the data undergoes a cleaning process to remove inconsistencies, correct formatting issues, and prepare it for storage. This step is crucial to ensure that the data stored in MongoDB is of high quality and ready for analysis.
• Role of Qdrant: In the broader system architecture, Qdrant acts as a feature store. After the data is stored in MongoDB, it is further processed and the relevant features are extracted and stored in Qdrant. This makes the data readily accessible for machine learning models and other analytical tasks, enhancing the system’s efficiency.

Test this module

• Manual Insertion for Testing: For the purposes of this lesson, we manually insert data into MongoDB to simulate how cleaned data is stored. This step helps us understand and test the dataset creation process in a controlled environment.
• Manual Retrieval from Qdrant: When testing or developing features, data can be manually retrieved from Qdrant. This step involves querying the feature store to fetch specific feature vectors, which allows you to use them in specific tasks like finetuning.

2. Insert Data into MongoDB

Before starting any logic on how to generate a finetuning dataset for our LLM Twin, we need data in MongoDB.

Be sure, that your local setup it’s ready for the entire code.

Check out the README of this module .

In insert_data_mongo.py we already prepare articles and posts, crawled from our platforms.

Downloading the Dataset

Utilize the download_dataset function from the provided script to fetch data files from Google Drive. This function checks if the data directory exists and downloads the necessary JSON files if it's empty. Each file represents a different type of content—articles, posts, and repositories—essential for diverse finetuning.

With the data downloaded, use the following functions to insert content into MongoDB:

• Insert Posts: The insert_posts function reads post data from JSON files and stores them in the database using the PostDocument model. Handle exceptions gracefully to ensure all posts are tried, and log the number of successfully inserted posts.
• Insert Articles: Similarly, the insert_articles function processes article data. It involves reading the files, potentially fixing JSON formatting issues, and saving the data using the ArticleDocument model.

3. Qdrant VectorDB — storing & retrieve cleaned data

To fully understand the data flow, please review the other two lessons that detail our use of the CDC pattern and Bytewax pipelines

↓↓↓

Change Data Capture: Enabling Event-Driven Architectures

The output of the Bytewax pipelines are sent to Qdrant VectorDB.

↓↓↓

SOTA Python Streaming Pipelines for Fine-tuning LLMs and RAG — in Real-Time!

Qdrant can be utilized via its Docker implementation or through its managed cloud service. This flexibility allows you to choose an environment that best suits your project’s needs.

Environment Variables

To configure your environment for Qdrant, set the following variables:

Docker Variables

• QDRANT_HOST: The hostname or IP address where your Qdrant server is running.
• QDRANT_PORT: The port on which Qdrant listens, typically 6333 for Docker setups.

Qdrant Cloud Variables

• QDRANT_CLOUD_URL: The URL for accessing Qdrant Cloud services.
• QDRANT_APIKEY: The API key for authenticating with Qdrant Cloud.

Please check this article to learn how to obtain these variables:

Additionally, you can control the connection mode (Cloud or Docker) using a setting in your configuration file. More details can be found in db/qdrant.py :

USE_QDRANT_CLOUD: True  # Set to False to use Docker setup

If the data pipeline succeeded you should see in Qdrant cloud 3 collections:cleaned_articles, cleaned_posts, cleaned_repositories

Qdrant Collections [Image by the author]

Retrieve Cleaned Data

Once you have a cleaned dataset, the next step is to utilize it effectively. Qdrant serves as a robust feature store, where data specifically prepared for machine learning applications is stored. In our case, this data is used for finetuning tasks.

Importance of a Feature Store

A feature store plays a critical role in machine learning workflows by:

• Centralizing Data Management: It centralizes feature data, making it accessible and reusable across multiple machine learning models and projects.
• Ensuring Consistency: Consistent feature calculation ensures that the same data preprocessing steps are applied in both training and prediction phases, reducing errors.
• Improving Efficiency: By storing pre-computed features, it significantly speeds up the experimentation process, allowing for rapid testing of different models.
• Scaling with Ease: As projects grow, a feature store can manage scaling of data operations efficiently, supporting larger datasets and more complex feature engineering tasks.

By leveraging a feature store like Qdrant, teams can enhance the reproducibility and scalability of their machine learning projects.

We will use the client from from db/qdrant module:

from qdrant_client import QdrantClient
from qdrant_client.http.exceptions import UnexpectedResponse
from qdrant_client.http.models import Batch, Distance, VectorParams

import logger_utils
from settings import settings

logger = logger_utils.get_logger(__name__)


class QdrantDatabaseConnector:
    _instance: QdrantClient = None

    def __init__(self):
        if self._instance is None:
            try:
                if settings.USE_QDRANT_CLOUD:
                    self._instance = QdrantClient(
                        url=settings.QDRANT_CLOUD_URL,
                        api_key=settings.QDRANT_APIKEY,
                    )
                else:
                    self._instance = QdrantClient(
                        host=settings.QDRANT_DATABASE_HOST,
                        port=settings.QDRANT_DATABASE_PORT,
                    )

            except UnexpectedResponse:
                logger.exception(
                    "Couldn't connect to the database.",
                    host=settings.QDRANT_DATABASE_HOST,
                    port=settings.QDRANT_DATABASE_PORT,
                )

                raise

    def get_collection(self, collection_name: str):
        return self._instance.get_collection(collection_name=collection_name)

    def create_non_vector_collection(self, collection_name: str):
        self._instance.create_collection(collection_name=collection_name, vectors_config={})

    def create_vector_collection(self, collection_name: str):
        self._instance.create_collection(
            collection_name=collection_name,
            vectors_config=VectorParams(size=settings.EMBEDDING_SIZE, distance=Distance.COSINE),
        )

    def write_data(self, collection_name: str, points: Batch):
        try:
            self._instance.upsert(collection_name=collection_name, points=points)
        except Exception:
            logger.exception("An error occurred while inserting data.")

            raise

    def scroll(self, collection_name: str, limit: int):
        return self._instance.scroll(collection_name=collection_name, limit=limit)

    def close(self):
        if self._instance:
            self._instance.close()

            logger.info("Connected to database has been closed.")


connection = QdrantDatabaseConnector()

To easily fetch data from a Qdrant database, you can utilize the Python function fetch_all_cleaned_content. This function demonstrates how to efficiently retrieve a list of cleaned content from a specified collection:

from db.qdrant import connection as client
    
def fetch_all_cleaned_content(self, collection_name: str) -> list:
        all_cleaned_contents = []

        scroll_response = client.scroll(collection_name=collection_name, limit=10000)
        points = scroll_response[0]

        for point in points:
            cleaned_content = point.payload["cleaned_content"]
            if cleaned_content:
                all_cleaned_contents.append(cleaned_content)

        return all_cleaned_contents

1. Initialize the Scroll: Start by scrolling through the database using the scroll method, which can fetch large amounts of data. Specify the collection name and set a limit to manage data volume effectively.
2. Access Points: Retrieve points from the initial scroll response. Each point contains payloads where the actual data is stored.
3. Extract Data: Loop through the points to extract the cleaned_content from each payload. This content is checked for existence before being added to the results list.
4. Return Results: The function returns a list containing all the cleaned content extracted from the database, making the process straightforward and manageable.

4.Generate Finetuning Dataset

• The Challenge: Manually creating a dataset for fine-tuning a language model like Mistral-7B can be time-consuming and prone to errors.
• The Solution: Instruction Datasets Instruction datasets offer an efficient way to guide a language model toward a specific task like news classification.
• Methods: While instruction datasets can be built manually or derived from existing sources, we’ll leverage a powerful LLM like OpenAI’s GPT 3.5-turbo due to our time and budget constraints.

Using Qdrant Data to Craft Instructions

Let’s analyze the sample data point from Qdrant to demonstrate how we can derive instructions for generating our instruction dataset:

Data Point:

{
  "author_id": "2",
  "cleaned_content": "Do you want to learn to build hands-on LLM systems using good LLMOps practices? A new Medium series is coming up for the Hands-on LLMs course\n.\nBy finishing the Hands-On LLMs free course, you will learn how to use the 3-pipeline architecture & LLMOps good practices to design, build, and deploy a real-time financial advisor powered by LLMs & vector DBs.\nWe will primarily focus on the engineering & MLOps aspects.\nThus, by the end of this series, you will know how to build & deploy a real ML system, not some isolated code in Notebooks.\nThere are 3 components you will learn to build during the course:\n- a real-time streaming pipeline\n- a fine-tuning pipeline\n- an inference pipeline\n.\nWe have already released the code and video lessons of the Hands-on LLM course.\nBut we are excited to announce an 8-lesson Medium series that will dive deep into the code and explain everything step-by-step.\nWe have already released the first lesson of the series  \nThe LLMs kit: Build a production-ready real-time financial advisor system using streaming pipelines, RAG, and LLMOps:  \n[URL]\n  In Lesson 1, you will learn how to design a financial assistant using the 3-pipeline architecture (also known as the FTI architecture), powered by:\n- LLMs\n- vector DBs\n- a streaming engine\n- LLMOps\n.\n  The rest of the articles will be released by the end of January 2024.\nFollow us on Medium's Decoding ML publication to get notified when we publish the other lessons:  \n[URL]\nhashtag\n#\nmachinelearning\nhashtag\n#\nmlops\nhashtag\n#\ndatascience",
  "platform": "linkedin",
  "type": "posts"
},
{
  "author_id": "2",
  "cleaned_content": "RAG systems are far from perfect   This free course teaches you how to improve your RAG system.\nI recently finished the Advanced Retrieval for AI with Chroma free course from\nDeepLearning.AI\nIf you are into RAG, I find it among the most valuable learning sources.\nThe course already assumes you know what RAG is.\nIts primary focus is to show you all the current issues of RAG and why it is far from perfect.\nAfterward, it shows you the latest SoTA techniques to improve your RAG system, such as:\n- query expansion\n- cross-encoder re-ranking\n- embedding adaptors\nI am not affiliated with\nDeepLearning.AI\n(I wouldn't mind though).\nThis is a great course you should take if you are into RAG systems.\nThe good news is that it is free and takes only 1 hour.\nCheck it out  \n  Advanced Retrieval for AI with Chroma:\n[URL]\nhashtag\n#\nmachinelearning\nhashtag\n#\nmlops\nhashtag\n#\ndatascience\n.\n  Follow me for daily lessons about ML engineering and MLOps.[URL]",
  "image": null,
  "platform": "linkedin",
  "type": "posts"
}

Process:

Deriving Instructions: We can use these insights to craft instructions for GPT 3.5-turbo:

• Instruction 1: “Write a LinkedIn post promoting a new educational course on building LLM systems with a focus on LLMOps. Use relevant hashtags and a tone that is both informative and engaging.”
• Instruction 2: “Write a LinkedIn post explaining the benefits of using LLMs and vector databases in real-time financial advising applications. Highlight the importance of LLMOps for successful deployment.”

Generating the Dataset with GPT 3.5-turbo

• Feeding Instructions: We would input the content derived from our Qdrant data points, along with others similarly created, into GPT 3.5-turbo.
• LLM Outputs: GPT 3.5-turbo would then generate instruction for every LinkedIn-style text content.

Result: This process would yield a dataset of instruction-output pairs designed to fine-tune Mistral-7B for the content generation.

Practical Example

In this section, we will demonstrate a practical application of the methods discussed above using a batch of content derived from a variety of posts.

The example will simulate the process of creating a training dataset for an LLM, using the format we’ve outlined.

Imagine that we want to go from this ↓

{
  "author_id": "2",
  "cleaned_content": "Do you want to learn to build hands-on LLM systems using good LLMOps practices? A new Medium series is coming up for the Hands-on LLMs course\n.\nBy finishing the Hands-On LLMs free course, you will learn how to use the 3-pipeline architecture & LLMOps good practices to design, build, and deploy a real-time financial advisor powered by LLMs & vector DBs.\nWe will primarily focus on the engineering & MLOps aspects.\nThus, by the end of this series, you will know how to build & deploy a real ML system, not some isolated code in Notebooks.\nThere are 3 components you will learn to build during the course:\n- a real-time streaming pipeline\n- a fine-tuning pipeline\n- an inference pipeline\n.\nWe have already released the code and video lessons of the Hands-on LLM course.\nBut we are excited to announce an 8-lesson Medium series that will dive deep into the code and explain everything step-by-step.\nWe have already released the first lesson of the series  \nThe LLMs kit: Build a production-ready real-time financial advisor system using streaming pipelines, RAG, and LLMOps:  \n[URL]\n  In Lesson 1, you will learn how to design a financial assistant using the 3-pipeline architecture (also known as the FTI architecture), powered by:\n- LLMs\n- vector DBs\n- a streaming engine\n- LLMOps\n.\n  The rest of the articles will be released by the end of January 2024.\nFollow us on Medium's Decoding ML publication to get notified when we publish the other lessons:  \n[URL]\nhashtag\n#\nmachinelearning\nhashtag\n#\nmlops\nhashtag\n#\ndatascience",
},
{
  "author_id": "2",
  "cleaned_content": "RAG systems are far from perfect   This free course teaches you how to improve your RAG system.\nI recently finished the Advanced Retrieval for AI with Chroma free course from\nDeepLearning.AI\nIf you are into RAG, I find it among the most valuable learning sources.\nThe course already assumes you know what RAG is.\nIts primary focus is to show you all the current issues of RAG and why it is far from perfect.\nAfterward, it shows you the latest SoTA techniques to improve your RAG system, such as:\n- query expansion\n- cross-encoder re-ranking\n- embedding adaptors\nI am not affiliated with\nDeepLearning.AI\n(I wouldn't mind though).\nThis is a great course you should take if you are into RAG systems.\nThe good news is that it is free and takes only 1 hour.\nCheck it out  \n  Advanced Retrieval for AI with Chroma:\n[URL]\nhashtag\n#\nmachinelearning\nhashtag\n#\nmlops\nhashtag\n#\ndatascience\n.\n  Follow me for daily lessons about ML engineering and MLOps.[URL]",
}

to this ↓

[
  {
    "instruction": "Share the announcement of the upcoming Medium series on building hands-on LLM systems using good LLMOps practices, focusing on the 3-pipeline architecture and real-time financial advisor development. Follow the Decoding ML publication on Medium for notifications on future lessons.",
    "content": "Do you want to learn to build hands-on LLM systems using good LLMOps practices? A new Medium series is coming up for the Hands-on LLMs course\n.\nBy finishing the Hands-On LLMs free course, you will learn how to use the 3-pipeline architecture & LLMOps good practices to design, build, and deploy a real-time financial advisor powered by LLMs & vector DBs.\nWe will primarily focus on the engineering & MLOps aspects.\nThus, by the end of this series, you will know how to build & deploy a real ML system, not some isolated code in Notebooks.\nThere are 3 components you will learn to build during the course:\n- a real-time streaming pipeline\n- a fine-tuning pipeline\n- an inference pipeline\n.\nWe have already released the code and video lessons of the Hands-on LLM course.\nBut we are excited to announce an 8-lesson Medium series that will dive deep into the code and explain everything step-by-step.\nWe have already released the first lesson of the series  \nThe LLMs kit: Build a production-ready real-time financial advisor system using streaming pipelines, RAG, and LLMOps:  \n[URL]\n  In Lesson 1, you will learn how to design a financial assistant using the 3-pipeline architecture (also known as the FTI architecture), powered by:\n- LLMs\n- vector DBs\n- a streaming engine\n- LLMOps\n.\n  The rest of the articles will be released by the end of January 2024.\nFollow us on Medium's Decoding ML publication to get notified when we publish the other lessons:  \n[URL]\nhashtag\n#\nmachinelearning\nhashtag\n#\nmlops\nhashtag\n#\ndatascience"
  },
  {
    "instruction": "Promote the free course 'Advanced Retrieval for AI with Chroma' from DeepLearning.AI that aims to improve RAG systems and takes only 1 hour to complete. Share the course link and encourage followers to check it out for the latest techniques in query expansion, cross-encoder re-ranking, and embedding adaptors.",
    "content": "RAG systems are far from perfect   This free course teaches you how to improve your RAG system.\nI recently finished the Advanced Retrieval for AI with Chroma free course from\nDeepLearning.AI\nIf you are into RAG, I find it among the most valuable learning sources.\nThe course already assumes you know what RAG is.\nIts primary focus is to show you all the current issues of RAG and why it is far from perfect.\nAfterward, it shows you the latest SoTA techniques to improve your RAG system, such as:\n- query expansion\n- cross-encoder re-ranking\n- embedding adaptors\nI am not affiliated with\nDeepLearning.AI\n(I wouldn't mind though).\nThis is a great course you should take if you are into RAG systems.\nThe good news is that it is free and takes only 1 hour.\nCheck it out  \n  Advanced Retrieval for AI with Chroma:\n[URL]\nhashtag\n#\nmachinelearning\nhashtag\n#\nmlops\nhashtag\n#\ndatascience\n.\n  Follow me for daily lessons about ML engineering and MLOps.[URL]"
  },

We are enhancing the LLM’s capability to generate specific instructions for each data point, aiming to impregnate the model with specialized knowledge in content creation.

Step 1: Set Up Data Points

First, let’s define a sample set of data points. These represent different content pieces from which we want to generate LinkedIn post instructions.

{
  "author_id": "2",
  "cleaned_content": "Do you want to learn to build hands-on LLM systems using good LLMOps practices? A new Medium series is coming up for the Hands-on LLMs course\n.\nBy finishing the Hands-On LLMs free course, you will learn how to use the 3-pipeline architecture & LLMOps good practices to design, build, and deploy a real-time financial advisor powered by LLMs & vector DBs.\nWe will primarily focus on the engineering & MLOps aspects.\nThus, by the end of this series, you will know how to build & deploy a real ML system, not some isolated code in Notebooks.\nThere are 3 components you will learn to build during the course:\n- a real-time streaming pipeline\n- a fine-tuning pipeline\n- an inference pipeline\n.\nWe have already released the code and video lessons of the Hands-on LLM course.\nBut we are excited to announce an 8-lesson Medium series that will dive deep into the code and explain everything step-by-step.\nWe have already released the first lesson of the series  \nThe LLMs kit: Build a production-ready real-time financial advisor system using streaming pipelines, RAG, and LLMOps:  \n[URL]\n  In Lesson 1, you will learn how to design a financial assistant using the 3-pipeline architecture (also known as the FTI architecture), powered by:\n- LLMs\n- vector DBs\n- a streaming engine\n- LLMOps\n.\n  The rest of the articles will be released by the end of January 2024.\nFollow us on Medium's Decoding ML publication to get notified when we publish the other lessons:  \n[URL]\nhashtag\n#\nmachinelearning\nhashtag\n#\nmlops\nhashtag\n#\ndatascience",
  "platform": "linkedin",
  "type": "posts"
},
{
  "author_id": "2",
  "cleaned_content": "RAG systems are far from perfect   This free course teaches you how to improve your RAG system.\nI recently finished the Advanced Retrieval for AI with Chroma free course from\nDeepLearning.AI\nIf you are into RAG, I find it among the most valuable learning sources.\nThe course already assumes you know what RAG is.\nIts primary focus is to show you all the current issues of RAG and why it is far from perfect.\nAfterward, it shows you the latest SoTA techniques to improve your RAG system, such as:\n- query expansion\n- cross-encoder re-ranking\n- embedding adaptors\nI am not affiliated with\nDeepLearning.AI\n(I wouldn't mind though).\nThis is a great course you should take if you are into RAG systems.\nThe good news is that it is free and takes only 1 hour.\nCheck it out  \n  Advanced Retrieval for AI with Chroma:\n[URL]\nhashtag\n#\nmachinelearning\nhashtag\n#\nmlops\nhashtag\n#\ndatascience\n.\n  Follow me for daily lessons about ML engineering and MLOps.[URL]",
  "image": null,
  "platform": "linkedin",
  "type": "posts"
}

Step 2: Utilize the DataFormatter Class

We’ll use the DataFormatter class to format these data points into a structured prompt for the LLM. Here’s how you would use the class to prepare the content:

data_type = "posts"
USER_PROMPT = (
    f"I will give you batches of contents of {data_type}. Please generate me exactly 1 instruction for each of them. The {data_type} text "
    f"for which you have to generate the instructions is under Content number x lines. Please structure the answer in json format,"
    f"ready to be loaded by json.loads(), a list of objects only with fields called instruction and content. For the content field, copy the number of the content only!."
    f"Please do not add any extra characters and make sure it is a list with objects in valid json format!\n"
)


class DataFormatter:
    @classmethod
    def format_data(cls, data_points: list, is_example: bool, start_index: int) -> str:
        text = ""
        for index, data_point in enumerate(data_points):
            if not is_example:
                text += f"Content number {start_index + index }\n"
            text += str(data_point) + "\n"
        return text

    @classmethod
    def format_batch(cls, context_msg: str, data_points: list, start_index: int) -> str:
        delimiter_msg = context_msg
        delimiter_msg += cls.format_data(data_points, False, start_index)
        return delimiter_msg

    @classmethod
    def format_prompt(cls, inference_posts: list, start_index: int):
        initial_prompt = USER_PROMPT
        initial_prompt += f"You must generate exactly a list of {len(inference_posts)} json objects, using the contents provided under CONTENTS FOR GENERATION\n"
        initial_prompt += cls.format_batch(
            "\nCONTENTS FOR GENERATION: \n", inference_posts, start_index
        )
        return initial_prompt

Output of the format_prompt function:

I will give you batches of contents of posts. Please generate me exactly 1 instruction for each of them. The posts text for which you have to generate the instructions is under Content number x lines. Please structure the answer in json format,ready to be loaded by json.loads(), a list of objects only with fields called instruction and content. For the content field, copy the number of the content only!.Please do not add any extra characters and make sure it is a list with objects in valid json format!
You must generate exactly a list of 1 json objects, using the contents provided under CONTENTS FOR GENERATION

CONTENTS FOR GENERATION: 
Content number 0
Do you want to learn to build hands-on LLM systems using good LLMOps practices? A new Medium series is coming up for the Hands-on LLMs course
.
By finishing the Hands-On LLMs free course, you will learn how to use the 3-pipeline architecture & LLMOps good practices to design, build, and deploy a real-time financial advisor powered by LLMs & vector DBs.
We will primarily focus on the engineering & MLOps aspects.
Thus, by the end of this series, you will know how to build & deploy a real ML system, not some isolated code in Notebooks.
There are 3 components you will learn to build during the course:
- a real-time streaming pipeline
- a fine-tuning pipeline
- an inference pipeline

Step 3: Automate finetuning data generation using the DatasetGenerator Class

To automate the generation of finetuning data, we desgined the DatasetGenerator class. This class is designed to streamline the process from fetching data to logging the training data into Comet ML.

The DatasetGenerator class is initialized with three components: a file handler for managing file I/O, an API communicator for interacting with the LLM, and a data formatter for preparing data:

class DatasetGenerator:
    def __init__(self, file_handler, api_communicator, data_formatter):
        self.file_handler = file_handler
        self.api_communicator = api_communicator
        self.data_formatter = data_formatter

Generating Finetuning Data

The generate_training_data method from DatasetGenerator class handles the full lifecycle of data generation:

def generate_training_data(self, collection_name: str, batch_size: int = 1):
    all_contents = self.fetch_all_cleaned_content(collection_name)
    response = []
    for i in range(0, len(all_contents), batch_size):
        batch = all_contents[i : i + batch_size]
        initial_prompt = self.data_formatter.format_prompt(batch, i)
        response += self.api_communicator.send_prompt(initial_prompt)
        for j in range(i, i + batch_size):
            response[j]["content"] = all_contents[j]

    self.push_to_comet(response, collection_name)

Fetch Content

The fetch_all_cleaned_content method retrieves all relevant content from a specified collection and prepares it for processing:

def fetch_all_cleaned_content(self, collection_name: str) -> list:
    all_cleaned_contents = []
    scroll_response = client.scroll(collection_name=collection_name, limit=10000)
    points = scroll_response[0]
    for point in points:
        cleaned_content = point.payload["cleaned_content"]
        if cleaned_content:
            all_cleaned_contents.append(cleaned_content)
    return all_cleaned_contents

4. CometML — data versioning

In this section, we focus on a critical aspect of Machine Learning Operations (MLOps) — data versioning.

We’ll specifically look at how to implement this using Comet ML, a platform that facilitates experiment management and reproducibility in machine learning projects.

CometML is a cloud-based platform that provides tools for tracking, comparing, explaining, and optimizing experiments and models in machine learning. CometML helps data scientists and teams to better manage and collaborate on machine learning experiments.

Why Use CometML?

• Artifacts: Leverages artifact management to capture, version, and manage data snapshots and models, which helps maintain data integrity and trace experiment lineage effectively.
• Experiment Tracking: CometML automatically tracks your code, experiments, and results, allowing you to compare between different runs and configurations visually.
• Model Optimization: It offers tools to compare different models side by side, analyze hyperparameters, and track model performance across various metrics.
• Collaboration and Sharing: Share findings and models with colleagues or the ML community, enhancing team collaboration and knowledge transfer.
• Reproducibility: By logging every detail of the experiment setup, CometML ensures experiments are reproducible, making it easier to debug and iterate.

Maybe you’re asking why not to choose MLFlow for example [2]:

• CometML excels in user interface design, providing a clean, intuitive experience for tracking experiments and models.
• It offers robust collaboration tools, making it easier for teams to work together on machine learning projects.
• CometML provides comprehensive security features that help protect data and models, an important consideration for enterprises.
• It has superior scalability, supporting larger datasets and more complex model training scenarios.
• The platform allows for more detailed tracking and analysis of experiments compared to MLflow.

CometML Variables

When integrating CometML into your projects, you’ll need to set up several environment variables to manage the authentication and configuration:

• COMET_API_KEY: Your unique API key that authenticates your interactions with the CometML API.
• COMET_PROJECT: The project name under which your experiments will be logged.
• COMET_WORKSPACE: The workspace name that organizes various projects and experiments.

Obtaining CometML Variables

To access and set up the necessary CometML variables for your project, follow these steps:

1. Create an Account or Log In:

• Visit CometML’s website and log in if you already have an account, or sign up if you’re a new user.

2. Create a New Project:

• Once logged in, navigate to your dashboard. Here, you can create a new project by clicking on “New Project” and entering the relevant details for your project.

3. Access API Key:

• After creating your project, you will need to obtain your API key. Navigate to your account settings by clicking on your profile at the top right corner. Select ‘API Keys’ from the menu, and you’ll see an option to generate or copy your existing API key.

4. Set Environment Variables:

• Add the obtained COMET_API_KEY to your environment variables, along with the COMET_PROJECT and COMET_WORKSPACE names you have set up.

The Importance of Data Versioning in MLOps

Data versioning is the practice of keeping a record of multiple versions of datasets used in training machine learning models. This practice is essential for several reasons:

1. Reproducibility: It ensures that experiments can be reproduced using the exact same data, which is crucial for validating and comparing machine learning models.
2. Model Diagnostics and Auditing: If a model’s performance changes unexpectedly, data versioning allows teams to revert to previous data states to identify issues.
3. Collaboration and Experimentation: Teams can experiment with different data versions to see how changes affect model performance without losing the original data setups.
4. Regulatory Compliance: In many industries, keeping track of data modifications and training environments is required for compliance with regulations.

Comet ML’s Artifacts

• Version Control: Artifacts in Comet ML are versioned, allowing you to track changes and iterate on datasets and models efficiently.
• Immutability: Once created, artifacts are immutable, ensuring that data integrity is maintained throughout the lifecycle of your projects.
• Metadata and Tagging: You can enhance artifacts with metadata and tags, making them easier to search and organize within Comet ML.
• Alias Management: Artifacts can be assigned aliases to simplify references to versions, streamlining workflow and reference.
• External Storage: Supports integration with external storage solutions like Amazon S3, enabling scalable and secure data management.

The provided push_to_comet function is a key part of this process.

def push_to_comet(self, data: list, collection_name: str):
        try:
            logging.info(f"Starting to push data to Comet: {collection_name}")

            # Assuming the settings module has been properly configured with the required attributes
            experiment = Experiment(
                api_key=settings.COMET_API_KEY,
                project_name=settings.COMET_PROJECT,
                workspace=settings.COMET_WORKSPACE,
            )

            file_name = f"{collection_name}.json"
            logging.info(f"Writing data to file: {file_name}")

            with open(file_name, "w") as f:
                json.dump(data, f)

            logging.info("Data written to file successfully")

            artifact = Artifact(collection_name)
            artifact.add(file_name)
            logging.info(f"Artifact created and file added: {file_name}")

            experiment.log_artifact(artifact)
            experiment.end()
            logging.info("Data pushed to Comet successfully and experiment ended")

        except Exception as e:
            logging.error(f"Failed to push data to Comet: {e}", exc_info=True)

Breakdown of Function Components

• Experiment Initialization: An experiment is created using the project settings. This ties all actions, like logging artifacts, to a specific experimental run.
• Data Saving: Data is saved locally as a JSON file. This file format is versatile and widely used, making it a good choice for data interchange.
• Artifact Creation and Logging: An artifact is a versioned object in Comet ML that can be associated with an experiment. By logging artifacts, you keep a record of all data versions used throughout the project lifecycle.

After running the script that invokes the push_to_comet function, Comet ML will update with new data artifacts, each representing a different dataset version. This is a crucial step in ensuring that all your data versions are logged and traceable within your MLOps environment.

What to Expect in Comet ML

Here is what you should see in Comet ML after successfully executing the script:

1. Artifacts Section: Navigate to the “Artifacts” tab in your Comet ML dashboard.
2. List of Artifacts: You will see entries for each type of data you’ve processed and saved. For example, if you have cleaned and versioned articles and posts, they will appear as separate artifacts.
3. Artifact Versions: Each artifact can have multiple versions. Each time you run the script with a new or updated dataset, a new version of the respective artifact is created.

For example, in the screenshot below there are two artifacts listed:

• cleaned_articles: This artifact has 10 versions, indicating it has been updated or modified 10 times.
• cleaned_posts: This artifact shows 3 versions, showing fewer updates compared to the articles.

Each version is timestamped and stored with a unique ID, allowing you to track changes over time or revert to previous versions if necessary.

Comet ML data artifacts [Image by the author]

A final version of your dataset, such as cleaned_articles.json, will be ready for the fine-tuning task.

This JSON file will include pairs of instructions and contents.

The instructions are generated by the LLM, guiding the model to perform specific tasks or understand particular contexts, while the content comprises data crawled and processed through your pipeline.

This format ensures each dataset version is not only tracked but also structured to enhance machine learning model performance through targeted finetuning.

Here’s an example of what the final version of cleaned_articles.json might look like, ready for the fine-tuning task:

[
  {
    "instruction": "Design and build a production-ready feature pipeline using Bytewax as a stream engine, RabbitMQ queue, and Qdrant vector DB. Include steps for processing raw data, data snapshots, Pydantic models, and loading cleaned features to the vector DB. Deploy the pipeline to AWS and integrate with previous components.",
    "content": "SOTA Python Streaming Pipelines for Fine-tuning LLMs and RAG \\u2014 in Real-Time!Use a Python streaming engine to populate a feature store from 4+ data sourcesStreaming Pipelines for LLMs and RAG | Decoding MLOpen in appSign upSign inWriteSign upSign inMastodonLLM TWIN COURSE: BUILDING YOUR PRODUCTION-READY AI REPLICASOTA Python Streaming Pipelines for Fine-tuning LLMs and RAG  in Real-Time!Use a Python streaming engine to populate a feature store from 4+ data sourcesPaul IusztinFollowPublished inDecoding ML18 min read6 days ago698ListenShare  the 4th out of 11 lessons of the LLM Twin free courseWhat is your LLM Twin? It is an AI character that writes like yourself by incorporating your style, personality and voice into an LLM.Image by DALL-EWhy is this course different?By finishing the LLM Twin: Building Your Production-Ready AI Replica free course, you will learn how to design, train, and deploy a production-ready LLM twin of yourself powered by LLMs, vector DBs, and LLMOps good practices.Why should you care? \\U0001faf5  No more isolated scripts or Notebooks! Learn production ML by building and deploying an end-to-end production-grade LLM system.What will you learn to build by the end of this course?You will learn how to architect and build a real-world LLM system from start to finish  from data collection to deployment.You will also learn to leverage MLOps best practices, such as experiment trackers, model registries, prompt monitoring, and versioning.The end goal? Build and deploy your own LLM twin.The architecture of the LLM twin is split into 4 Python microservices:the data collection pipeline: crawl your digital data from various social media platforms. Clean, normalize and load the data to a NoSQL DB through a series of ETL pipelines. Send database changes to a queue using the CDC pattern. (deployed on AWS)the feature pipeline: consume messages from a queue through a Bytewax streaming pipeline. Every message will be cleaned, chunked, embedded (using Superlinked), and loaded into a Qdrant vector DB in real-time. (deployed on AWS)the training pipeline: create a custom dataset based on your digital data. Fine-tune an LLM using QLoRA. Use Comet MLs experiment tracker to monitor the experiments. Evaluate and save the best model to Comets model registry. (deployed on Qwak)the inference pipeline: load and quantize the fine-tuned LLM from Comets model registry. Deploy it as a REST API. Enhance the prompts using RAG. Generate content using your LLM twin. Monitor the LLM using Comets prompt monitoring dashboard. (deployed on Qwak)LLM twin system architecture [Image by the Author]Along the 4 microservices, you will learn to integrate 3 serverless tools:Comet ML as your ML Platform;Qdrant as your vector DB;Qwak as your ML infrastructure;Who is this for?Audience: MLE, DE, DS, or SWE who want to learn to engineer production-ready LLM systems using LLMOps good principles.Level: intermediatePrerequisites: basic knowledge of Python, ML, and the cloudHow will you learn?The course contains 11 hands-on written lessons and the open-source code you can access on GitHub.You can read everything at your own pace.  To get the most out of this course, we encourage you to clone and run the repository while you cover the lessons.Costs?The articles and code are completely free. They will always remain free.But if you plan to run the code while reading it, you have to know that we use several cloud tools that might generate additional costs.The cloud computing platforms (AWS, Qwak) have a pay-as-you-go pricing plan. Qwak offers a few hours of free computing. Thus, we did our best to keep costs to a minimum.For the other serverless tools (Qdrant, Comet), we will stick to their freemium version, which is free of charge.Meet your teachers!The course is created under the Decoding ML umbrella by:Paul Iusztin | Senior ML & MLOps EngineerAlex Vesa | Senior AI EngineerAlex Razvant | Senior ML & MLOps EngineerLessonsThe course is split into 11 lessons. Every Medium article will be its own lesson.An End-to-End Framework for Production-Ready LLM Systems by Building Your LLM TwinThe Importance of Data Pipelines in the Era of Generative AIChange Data Capture: Enabling Event-Driven ArchitecturesSOTA Python Streaming Pipelines for Fine-tuning LLMs and RAG  in Real-Time!Vector DB retrieval clients [Module 2] WIPTraining data preparation [Module 3] WIPFine-tuning LLM [Module 3] WIPLLM evaluation [Module 4] WIPQuantization [Module 5] WIPBuild the digital twin inference pipeline [Module 6] WIPDeploy the digital twin as a REST API [Module 6] WIP  Check out the code on GitHub [1] and support us with a Lets start with Lesson 4  Lesson 4: Python Streaming Pipelines for Fine-tuning LLMs and RAG  in Real-Time!In the 4th lesson, we will focus on the feature pipeline.The feature pipeline is the first pipeline presented in the 3 pipeline architecture: feature, training and inference pipelines.A feature pipeline is responsible for taking raw data as input, processing it into features, and storing it in a feature store, from which the training & inference pipelines will use it.The component is completely isolated from the training and inference code. All the communication is done through the feature store.To avoid repeating myself, if you are unfamiliar with the 3 pipeline architecture, check out Lesson 1 for a refresher.By the end of this article, you will learn to design and build a production-ready feature pipeline that:uses Bytewax as a stream engine to process data in real-time;ingests data from a RabbitMQ queue;uses SWE practices to process multiple data types: posts, articles, code;cleans, chunks, and embeds data for LLM fine-tuning and RAG;loads the features to a Qdrant vector DB.Note: In our use case, the feature pipeline is also a streaming pipeline, as we use a Bytewax streaming engine. Thus, we will use these words interchangeably.We will wrap up Lesson 4 by showing you how to deploy the feature pipeline to AWS and integrate it with the components from previous lessons: data collection pipeline, MongoDB, and CDC.In the 5th lesson, we will go through the vector DB retrieval client, where we will teach you how to query the vector DB and improve the accuracy of the results using advanced retrieval techniques.Excited? Lets get started!The architecture of the feature/streaming pipeline.Table of ContentsWhy are we doing this?System design of the feature pipelineThe Bytewax streaming flowPydantic data modelsLoad data to QdrantThe dispatcher layerPreprocessing steps: Clean, chunk, embedThe AWS infrastructureRun the code locallyDeploy the code to AWS & Run it from the cloudConclusion  Check out the code on GitHub [1] and support us with a 1. Why are we doing this?A quick reminder from previous lessonsTo give you some context, in Lesson 2, we crawl data from LinkedIn, Medium, and GitHub, normalize it, and load it to MongoDB.In Lesson 3, we are using CDC to listen to changes to the MongoDB database and emit events in a RabbitMQ queue based on any CRUD operation done on MongoDB.and here we are in Lesson 4, where we are building the feature pipeline that listens 24/7 to the RabbitMQ queue for new events to process and load them to a Qdrant vector DB.The problem we are solvingIn our LLM Twin use case, the feature pipeline constantly syncs the MongoDB warehouse with the Qdrant vector DB while processing the raw data into features.Important: In our use case, the Qdrant vector DB will be our feature store.Why we are solving itThe feature store will be the central point of access for all the features used within the training and inference pipelines.For consistency and simplicity, we will refer to different formats of our text data as features.  The training pipeline will use the feature store to create fine-tuning datasets for your LLM twin.  The inference pipeline will use the feature store for RAG.For reliable results (especially for RAG), the data from the vector DB must always be in sync with the data from the data warehouse.The question is, what is the best way to sync these 2?Other potential solutionsThe most common solution is probably to use a batch pipeline that constantly polls from the warehouse, computes a difference between the 2 databases, and updates the target database.The issue with this technique is that computing the difference between the 2 databases is extremely slow and costly.Another solution is to use a push technique using a webhook. Thus, on any CRUD change in the warehouse, you also update the source DB.The biggest issue here is that if the webhook fails, you have to implement complex recovery logic.Lesson 3 on CDC covers more of this.2. System design of the feature pipeline: our solutionOur solution is based on CDC, a queue, a streaming engine, and a vector DB:  CDC adds any change made to the Mongo DB to the queue (read more in Lesson 3).  the RabbitMQ queue stores all the events until they are processed.  The Bytewax streaming engine cleans, chunks, and embeds the data.  A streaming engine works naturally with a queue-based system.  The data is uploaded to a Qdrant vector DB on the flyWhy is this powerful?Here are 4 core reasons:The data is processed in real-time.Out-of-the-box recovery system: If the streaming pipeline fails to process a message will be added back to the queueLightweight: No need for any diffs between databases or batching too many recordsNo I/O bottlenecks on the source database  It solves all our problems!The architecture of the feature/streaming pipeline.How is the data stored?We store 2 snapshots of our data in the feature store. Here is why  Remember that we said that the training and inference pipeline will access the features only from the feature store, which, in our case, is the Qdrant vector DB?Well, if we had stored only the chunked & embedded version of the data, that would have been useful only for RAG but not for fine-tuning.Thus, we make an additional snapshot of the cleaned data, which will be used by the training pipeline.Afterward, we pass it down the streaming flow for chunking & embedding.How do we process multiple data types?How do you process multiple types of data in a single streaming pipeline without writing spaghetti code?Yes, that is for you, data scientists! Jokingam I?We have 3 data types: posts, articles, and code.Each data type (and its state) will be modeled using Pydantic models.To process them we will write a dispatcher layer, which will use a creational factory pattern [9] to instantiate a handler implemented for that specific data type (post, article, code) and operation (cleaning, chunking, embedding).The handler follows the strategy behavioral pattern [10].Intuitively, you can see the combination between the factory and strategy patterns as follows:Initially, we know we want to clean the data, but as we dont know the data type, we cant know how to do so.What we can do, is write the whole code around the cleaning code and abstract away the login under a Handler() interface (aka the strategy).When we get a data point, the factory class creates the right cleaning handler based on its type.Ultimately the handler is injected into the rest of the system and executed.By doing so, we can easily isolate the logic for a given data type & operation while leveraging polymorphism to avoid filling up the code with 1000x if else statements.We will dig into the implementation in future sections.Streaming over batchYou may ask why we need a streaming engine instead of implementing a batch job that polls the messages at a given frequency.That is a valid question.The thing is thatNowadays, using tools such as Bytewax makes implementing streaming pipelines a lot more frictionless than using their JVM alternatives.The key aspect of choosing a streaming vs. a batch design is real-time synchronization between your source and destination DBs.In our particular case, we will process social media data, which changes fast and irregularly.Also, for our digital twin, it is important to do RAG on up-to-date data. We dont want to have any delay between what happens in the real world and what your LLM twin sees.That being said choosing a streaming architecture seemed natural in our use case.3. The Bytewax streaming flowThe Bytewax flow is the central point of the streaming pipeline. It defines all the required steps, following the next simplified pattern: input -> processing -> output.As I come from the AI world, I like to see it as the graph of the streaming pipeline, where you use the input(), map(), and output() Bytewax functions to define your graph, which in the Bytewax world is called a flow.As you can see in the code snippet below, we ingest posts, articles or code messages from a RabbitMQ queue. After we clean, chunk and embed them. Ultimately, we load the cleaned and embedded data to a Qdrant vector DB, which in our LLM twin use case will represent the feature store of our system.To structure and validate the data, between each Bytewax step, we map and pass a different Pydantic model based on its current state: raw, cleaned, chunked, or embedded.Bytewax flow   GitHub Code We have a single streaming pipeline that processes everything.As we ingest multiple data types (posts, articles, or code snapshots), we have to process them differently.To do this the right way, we implemented a dispatcher layer that knows how to apply data-specific operations based on the type of message.More on this in the next sections  Why Bytewax?Bytewax is an open-source streaming processing framework that:- is built in Rust  for performance- has Python   bindings for leveraging its powerful ML ecosystem so, for all the Python fanatics out there, no more JVM headaches for you.Jokes aside, here is why Bytewax is so powerful  - Bytewax local setup is plug-and-play- can quickly be integrated into any Python project (you can go wild  even use it in Notebooks)- can easily be integrated with other Python packages (NumPy, PyTorch, HuggingFace, OpenCV, SkLearn, you name it)- out-of-the-box connectors for Kafka and local files, or you can quickly implement your ownWe used Bytewax to build the streaming pipeline for the LLM Twin course and loved it.To learn more about Bytewax, go and check them out. They are open source, so no strings attached   Bytewax [2] 4. Pydantic data modelsLets take a look at what our Pydantic models look like.First, we defined a set of base abstract models for using the same parent class across all our components.Pydantic base model structure   GitHub Code Afterward, we defined a hierarchy of Pydantic models for:all our data types: posts, articles, or codeall our states: raw, cleaned, chunked, and embeddedThis is how the set of classes for the posts will look like  Pydantic posts model structure   GitHub Code We repeated the same process for the articles and code model hierarchy.Check out the other data classes on our GitHub.Why is keeping our data in Pydantic models so powerful?There are 4 main criteria:every field has an enforced type: you are ensured the data types are going to be correctthe fields are automatically validated based on their type: for example, if the field is a string and you pass an int, it will through an errorthe data structure is clear and verbose: no more clandestine dicts that you never know what is in themyou make your data the first-class citizen of your program5. Load data to QdrantThe first step is to implement our custom Bytewax DynamicSink class  Qdrant DynamicSink   GitHub Code Next, for every type of operation we need (output cleaned or embedded data ) we have to subclass the StatelessSinkPartition Bytewax class (they also provide a stateful option   more in their docs)An instance of the class will run on every partition defined within the Bytewax deployment.In the course, we are using a single partition per worker. But, by adding more partitions (and workers), you can quickly scale your Bytewax pipeline horizontally.Qdrant worker partitions   GitHub Code Note that we used Qdrants Batch method to upload all the available points at once. By doing so, we reduce the latency on the network I/O side: more on that here [8] The RabbitMQ streaming input follows a similar pattern. Check it out here 6. The dispatcher layerNow that we have the Bytewax flow and all our data models.How do we map a raw data model to a cleaned data model?  All our domain logic is modeled by a set of Handler() classes.For example, this is how the handler used to map a PostsRawModel to a PostCleanedModel looks like  Handler hierarchy of classes   GitHub Code Check out the other handlers on our GitHub:  ChunkingDataHandler and EmbeddingDataHandlerIn the next sections, we will explore the exact cleaning, chunking and embedding logic.Now, to build our dispatcher, we need 2 last components:a factory class: instantiates the right handler based on the type of the eventa dispatcher class: the glue code that calls the factory class and handlerHere is what the cleaning dispatcher and factory look like  The dispatcher and factory classes   GitHub Code Check out the other dispatchers on our GitHub.By repeating the same logic, we will end up with the following set of dispatchers:RawDispatcher (no factory class required as the data is not processed)CleaningDispatcher (with a ChunkingHandlerFactory class)ChunkingDispatcher (with a ChunkingHandlerFactory class)EmbeddingDispatcher (with an EmbeddingHandlerFactory class)7. Preprocessing steps: Clean, chunk, embedHere we will focus on the concrete logic used to clean, chunk, and embed a data point.Note that this logic is wrapped by our handler to be integrated into our dispatcher layer using the Strategy behavioral pattern [10].We already described that in the previous section. Thus, we will directly jump into the actual logic here, which can be found in the utils module of our GitHub repository.Note: These steps are experimental. Thus, what we present here is just the first iteration of the system. In a real-world scenario, you would experiment with different cleaning, chunking or model versions to improve it on your data.CleaningThis is the main utility function used to clean the text for our posts, articles, and code.Out of simplicity, we used the same logic for all the data types, but after more investigation, you would probably need to adapt it to your specific needs.For example, your posts might start containing some weird characters, and you dont want to run the unbold_text() or unitalic_text() functions on your code data point as is completely redundant.Cleaning logic   GitHub Code Most of the functions above are from the unstructured [3] Python package. It is a great tool for quickly finding utilities to clean text data.  More examples of unstructured here [3] One key thing to notice is that at the cleaning step, we just want to remove all the weird, non-interpretable characters from the text.Also, we want to remove redundant data, such as extra whitespace or URLs, as they do not provide much value.These steps are critical for our tokenizer to understand and efficiently transform our string input into numbers that will be fed into the transformer models.Note that when using bigger models (transformers) + modern tokenization techniques, you dont need to standardize your dataset too much.For example, it is redundant to apply lemmatization or stemming, as the tokenizer knows how to split your input into a commonly used sequence of characters efficiently, and the transformers can pick up the nuances of the words.  What is important at the cleaning step is to throw out the noise.ChunkingWe are using Langchain to chunk our text.We use a 2 step strategy using Langchains RecursiveCharacterTextSplitter [4] and SentenceTransformersTokenTextSplitter [5]. As seen below  Chunking logic   GitHub Code Overlapping your chunks is a common pre-indexing RAG technique, which helps to cluster chunks from the same document semantically.Again, we are using the same chunking logic for all of our data types, but to get the most out of it, we would probably need to tweak the separators, chunk_size, and chunk_overlap parameters for our different use cases.But our dispatcher + handler architecture would easily allow us to configure the chunking step in future iterations.EmbeddingThe data preprocessing, aka the hard part is done.Now we just have to call an embedding model to create our vectors.Embedding logic   GitHub Code We used the all-MiniLm-L6-v2 [6] from the sentence-transformers library to embed our articles and posts: a lightweight embedding model that can easily run in real-time on a 2 vCPU machine.As the code data points contain more complex relationships and specific jargon to embed, we used a more powerful embedding model: hkunlp/instructor-xl [7].This embedding model is unique as it can be customized on the fly with instructions based on your particular data. This allows the embedding model to specialize on your data without fine-tuning, which is handy for embedding pieces of code.8. The AWS infrastructureIn Lesson 2, we covered how to deploy the data collection pipeline that is triggered by a link to Medium, Substack, LinkedIn or GitHub   crawls the given link   saves the crawled information to a MongoDB.In Lesson 3, we explained how to deploy the CDC components that emit events to a RabbitMQ queue based on any CRUD operation done to MongoDB.What is left is to deploy the Bytewax streaming pipeline and Qdrant vector DB.We will use Qdrants self-hosted option, which is easy to set up and scale.To test things out, they offer a Free Tier plan for up to a 1GB cluster, which is more than enough for our course.  We explained in our GitHub repository how to configure Qdrant.AWS infrastructure of the feature/streaming pipeline.The last piece of the puzzle is the Bytewax streaming pipeline.As we dont require a GPU and the streaming pipeline needs to run 24/7, we will deploy it to AWS Fargate, a cost-effective serverless solution from AWS.As a serverless solution, Fargate allows us to deploy our code quickly and scale it fast in case of high traffic.How do we deploy the streaming pipeline code to Fargate?Using GitHub Actions, we wrote a CD pipeline that builds a Docker image on every new commit made on the main branch.After, the Docker image is pushed to AWS ECR. Ultimately, Fargate pulls the latest version of the Docker image.This is a common CD pipeline to deploy your code to AWS services.Why not use lambda functions, as we did for the data pipeline?An AWS lambda function executes a function once and then closes down.This worked perfectly for the crawling logic, but it won't work for our streaming pipeline, which has to run 24/7.9. Run the code locallyTo quickly test things up, we wrote a docker-compose.yaml file to spin up the MongoDB, RabbitMQ queue and Qdrant vector db.You can spin up the Docker containers using our Makefile by running the following:make local-start-infraTo fully test the Bytewax streaming pipeline, you have to start the CDC component by running:make local-start-cdcUltimately, you start the streaming pipeline:make local-bytewaxTo simulate the data collection pipeline, mock it as follows:make local-insert-data-mongoThe README of our GitHub repository provides more details on how to run and set up everything.10. Deploy the code to AWS & Run it from the cloudThis article is already too long, so I wont go into the details of how to deploy the AWS infrastructure described above and test it out here.But to give you some insights, we have used Pulumi as our infrastructure as a code (IaC) tool, which will allow you to spin it quickly with a few commands.Also, I wont let you hang on to this one. We made a promise and  We prepared step-by-step instructions in the README of our GitHub repository on how to use Pulumni to spin up the infrastructure and test it out.ConclusionNow you know how to write streaming pipelines like a PRO!In Lesson 4, you learned how to:design a feature pipeline using the 3-pipeline architecturewrite a streaming pipeline using Bytewax as a streaming engineuse a dispatcher layer to write a modular and flexible application to process multiple types of data (posts, articles, code)load the cleaned and embedded data to Qdrantdeploy the streaming pipeline to AWS  This is only the ingestion part used for fine-tuning LLMs and RAG.In Lesson 5, you will learn how to write a retrieval client for the 3 data types using good SWE practices and improve the retrieval accuracy using advanced retrieval & post-retrieval techniques. See you there!  Check out the code on GitHub [1] and support us with a Enjoyed This Article?Join the Decoding ML Newsletter for battle-tested content on designing, coding, and deploying production-grade ML & MLOps systems. Every week. For FREE  Decoding ML Newsletter | Paul Iusztin | SubstackJoin for battle-tested content on designing, coding, and deploying production-grade ML & MLOps systems. Every week. Fordecodingml.substack.comReferencesLiterature[1] Your LLM Twin Course  GitHub Repository (2024), Decoding ML GitHub Organization[2] Bytewax, Bytewax Landing Page[3] Unstructured Cleaning Examples, Unstructured Documentation[4] Recursively split by character, LangChains Documentation[5] Split by tokens, LangChains Documentation[6] sentence-transformers/all-MiniLM-L6-v2, HuggingFace[7] hkunlp/instructor-xl, HuggingFace[8] Qdrant, Qdrant Documentation[9] Abstract Factory Pattern, Refactoring Guru[10] Strategy Pattern, Refactoring GuruImagesIf not otherwise stated, all images are created by the author.Sign up to discover human stories that deepen your understanding of the world.FreeDistraction-free reading. No ads.Organize your knowledge with lists and highlights.Tell your story. Find your audience.Sign up for freeMembershipAccess the best member-only stories.Support independent authors.Listen to audio narrations.Read offline.Join the Partner Program and earn for your writing.Try for $5/monthMl System DesignMachine LearningArtificial IntelligenceData ScienceSoftware Engineering698698FollowWritten by Paul Iusztin2.4K FollowersEditor for Decoding ML Senior ML Engineer | Helping machine learning engineers design and productionize ML systems. | Decoding ML Newsletter: [URL] from Paul Iusztin and Decoding MLPaul IusztininDecoding MLAn End-to-End Framework for Production-Ready LLM Systems by Building Your LLM TwinFrom data gathering to productionizing LLMs using LLMOps good practices.16 min readMar 16, 20241.5K10Razvant AlexandruinDecoding MLHow to build a Real-Time News Search Engine using Serverless Upstash Kafka and Vector DBA hands-on guide to implementing a live news aggregating streaming pipeline with Apache Kafka, Bytewax, and Upstash Vector Database.19 min readApr 13, 20246873Paul IusztininDecoding MLThe LLMs kit: Build a production-ready real-time financial advisor system using streamingLesson 1: LLM architecture system design using the 3-pipeline pattern12 min readJan 5, 2024383Paul IusztininDecoding MLA Real-time Retrieval System for RAG on Social Media DataUse a streaming engine to populate a vector DB in real-time. Improve RAG accuracy using rerank & UMAP.12 min readMar 30, 2024305See all from Paul IusztinSee all from Decoding MLRecommended from MediumVipra SinghBuilding LLM Applications: Serving LLMs (Part 9)Learn Large Language Models ( LLM ) through the lens of a Retrieval Augmented Generation ( RAG ) Application.49 min readApr 18, 20243741A B Vijay KumarMulti-Agent System\\u200a\\u200aCrew.AIMulti-Agent systems are LLM applications that are changing the automation landscape with intelligent bots.7 min readApr 18, 20241574ListsPredictive Modeling w/ Python20 stories1126 savesNatural Language Processing1402 stories899 savesPractical Guides to Machine Learning10 stories1352 savesdata science and AI40 stories137 savesFabio MatricardiinGenerative AILlama3 is out and you can run it on your Computer!After only 1 day from the release, here is how you can run even on your Laptop with CPU only the latest Meta-AI model.8 min read6 days ago1.2K13Gavin LiinAI AdvancesRun the strongest open-source LLM model: Llama3 70B with just a single 4GB GPU!The strongest open source LLM model Llama3 has been released, Here is how you can run Llama3 70B locally with just 4GB GPU, even on Macbook4 min read4 days ago7982Plaban NayakinThe AI ForumSemantic Chunking for RAGWhat is Chunking\\xa0?17 min read4 days ago1651Damian GilinTowards Data ScienceAdvanced Retriever Techniques to Improve Your RAGsMaster Advanced Information Retrieval: Cutting-edge Techniques to Optimize the Selection of Relevant Documents with Langchain to Create18 min readApr 17, 20245393See more recommendationsHelpStatusAboutCareersBlogPrivacyTermsText to speechTeams\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nTo make Medium work, we log user data. By using Medium, you agree to our Privacy Policy, including cookie policy."
  },
  {
    "instruction": "Focus on creating battle-tested content on building production-grade ML systems using good SWE and MLOps practices.",
    "content": "About Decoding MLThe hub for continuous learning on ML system design, ML engineering, MLOps, LLMs and computer vision.About Decoding ML. The hub for continuous learning on ML | by Paul Iusztin | Decoding ML | MediumOpen in appSign upSign inWriteSign upSign inMastodonAbout Decoding MLThe hub for continuous learning on ML system design, ML engineering, MLOps, LLMs and computer vision.Paul IusztinFollowPublished inDecoding ML2 min readFeb 19, 202491ListenShareDecoding ML is a publication that creates battle-tested content on building production-grade ML systems leveraging good SWE and MLOps practices.Our motto is More engineering, less F1 scores.Following Decoding ML, you will learn about the entire lifecycle of an ML system, from system design to deploying and monitoring.Decoding ML is the hub for continuous learning on:ML system designML engineeringMLOpsLarge language modelsComputer visionWe are all about end-to-end ML use cases that can directly be applied in the real world  no stories  just hands-on content.The minds behind Decoding ML  Alex Vesa (left), Paul Iusztin (middle) and Alexandru Razvant (right)The minds behind Decoding ML are Vesa Alexandru, Paul Iusztin and Razvant Alexandru.Our passion for constantly learning and engineering production-ready ML systems inevitably turned into Decoding ML.With our 10+ years of hands-on experience in the industry in:Data ScienceDeep LearningComputer VisionGenerative AIML InfrastructureMLOpsSoftware Engineeringwe decided it is about time to share it with the world!Why follow?Join Decoding ML for battle-tested content on designing, coding, and deploying production-grade ML & MLOps systems. Every week. For FREE.No more bedtime stories in Jupyter Notebooks.  DML is all about hands-on advice from our 10+ years of experience in AI.We are also on: Newsletter  GitHubSign up to discover human stories that deepen your understanding of the world.FreeDistraction-free reading. No ads.Organize your knowledge with lists and highlights.Tell your story. Find your audience.Sign up for freeMembershipAccess the best member-only stories.Support independent authors.Listen to audio narrations.Read offline.Join the Partner Program and earn for your writing.Try for $5/monthAbout9191FollowWritten by Paul Iusztin2.4K FollowersEditor for Decoding ML Senior ML Engineer | Helping machine learning engineers design and productionize ML systems. | Decoding ML Newsletter: [URL] from Paul Iusztin and Decoding MLPaul IusztininDecoding MLAn End-to-End Framework for Production-Ready LLM Systems by Building Your LLM TwinFrom data gathering to productionizing LLMs using LLMOps good practices.16 min readMar 16, 20241.5K10Razvant AlexandruinDecoding MLHow to build a Real-Time News Search Engine using Serverless Upstash Kafka and Vector DBA hands-on guide to implementing a live news aggregating streaming pipeline with Apache Kafka, Bytewax, and Upstash Vector Database.19 min readApr 13, 20246873Paul IusztininDecoding MLThe LLMs kit: Build a production-ready real-time financial advisor system using streamingLesson 1: LLM architecture system design using the 3-pipeline pattern12 min readJan 5, 2024383Paul IusztininDecoding MLA Real-time Retrieval System for RAG on Social Media DataUse a streaming engine to populate a vector DB in real-time. Improve RAG accuracy using rerank & UMAP.12 min readMar 30, 2024305See all from Paul IusztinSee all from Decoding MLRecommended from MediumQwakHow to Build an End-to-End ML Pipeline in 2024Learn to build an end-to-end ML pipeline and streamline your ML workflows in 2024, from data ingestion to model deployment and performance24 min readApr 7, 20242071Paul IusztininDecoding MLThe LLMs kit: Build a production-ready real-time financial advisor system using streamingLesson 1: LLM architecture system design using the 3-pipeline pattern12 min readJan 5, 2024383ListsMedium's Huge List of Publications Accepting Submissions285 stories2516 savesYesmine RouisinTheFork Engineering BlogA Guide to MLOps with Airflow and MLflowIntroduction11 min readNov 6, 2023319ManralaiMastering MLOps for FreeYour Go-To GitHub Repositories Collection4 min readJan 28, 20246463Vipra SinghBuilding LLM Applications: Evaluation (Part 8)Learn Large Language Models ( LLM ) through the lens of a Retrieval Augmented Generation ( RAG ) Application.47 min readApr 8, 20241891Ramazan OlmezEnd-to-End Machine Learning Project: Churn PredictionThe main objective of this article is to develop an end-to-end machine learning project. For a model to be truly useful, it needs to be18 min readFeb 22, 202487See more recommendationsHelpStatusAboutCareersBlogPrivacyTermsText to speechTeams\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nTo make Medium work, we log user data. By using Medium, you agree to our Privacy Policy, including cookie policy."
  },
  {
    "instruction": "Design and build a production-ready feature pipeline using Bytewax as a stream engine to process data in real-time, ingest data from a RabbitMQ queue, implement SWE practices to process multiple data types, clean, chunk, and embed data for LLM fine-tuning and RAG, and load features to a Qdrant vector DB.",
    "content": "SOTA Python Streaming Pipelines for Fine-tuning LLMs and RAG \\u2014 in Real-Time!Use a Python streaming engine to populate a feature store from 4+ data sourcesStreaming Pipelines for LLMs and RAG | Decoding MLOpen in appSign upSign inWriteSign upSign inMastodonLLM TWIN COURSE: BUILDING YOUR PRODUCTION-READY AI REPLICASOTA Python Streaming Pipelines for Fine-tuning LLMs and RAG  in Real-Time!Use a Python streaming engine to populate a feature store from 4+ data sourcesPaul IusztinFollowPublished inDecoding ML18 min read6 days ago698ListenShare  the 4th out of 11 lessons of the LLM Twin free courseWhat is your LLM Twin? It is an AI character that writes like yourself by incorporating your style, personality and voice into an LLM.Image by DALL-EWhy is this course different?By finishing the LLM Twin: Building Your Production-Ready AI Replica free course, you will learn how to design, train, and deploy a production-ready LLM twin of yourself powered by LLMs, vector DBs, and LLMOps good practices.Why should you care? \\U0001faf5  No more isolated scripts or Notebooks! Learn production ML by building and deploying an end-to-end production-grade LLM system.What will you learn to build by the end of this course?You will learn how to architect and build a real-world LLM system from start to finish  from data collection to deployment.You will also learn to leverage MLOps best practices, such as experiment trackers, model registries, prompt monitoring, and versioning.The end goal? Build and deploy your own LLM twin.The architecture of the LLM twin is split into 4 Python microservices:the data collection pipeline: crawl your digital data from various social media platforms. Clean, normalize and load the data to a NoSQL DB through a series of ETL pipelines. Send database changes to a queue using the CDC pattern. (deployed on AWS)the feature pipeline: consume messages from a queue through a Bytewax streaming pipeline. Every message will be cleaned, chunked, embedded (using Superlinked), and loaded into a Qdrant vector DB in real-time. (deployed on AWS)the training pipeline: create a custom dataset based on your digital data. Fine-tune an LLM using QLoRA. Use Comet MLs experiment tracker to monitor the experiments. Evaluate and save the best model to Comets model registry. (deployed on Qwak)the inference pipeline: load and quantize the fine-tuned LLM from Comets model registry. Deploy it as a REST API. Enhance the prompts using RAG. Generate content using your LLM twin. Monitor the LLM using Comets prompt monitoring dashboard. (deployed on Qwak)LLM twin system architecture [Image by the Author]Along the 4 microservices, you will learn to integrate 3 serverless tools:Comet ML as your ML Platform;Qdrant as your vector DB;Qwak as your ML infrastructure;Who is this for?Audience: MLE, DE, DS, or SWE who want to learn to engineer production-ready LLM systems using LLMOps good principles.Level: intermediatePrerequisites: basic knowledge of Python, ML, and the cloudHow will you learn?The course contains 11 hands-on written lessons and the open-source code you can access on GitHub.You can read everything at your own pace.  To get the most out of this course, we encourage you to clone and run the repository while you cover the lessons.Costs?The articles and code are completely free. They will always remain free.But if you plan to run the code while reading it, you have to know that we use several cloud tools that might generate additional costs.The cloud computing platforms (AWS, Qwak) have a pay-as-you-go pricing plan. Qwak offers a few hours of free computing. Thus, we did our best to keep costs to a minimum.For the other serverless tools (Qdrant, Comet), we will stick to their freemium version, which is free of charge.Meet your teachers!The course is created under the Decoding ML umbrella by:Paul Iusztin | Senior ML & MLOps EngineerAlex Vesa | Senior AI EngineerAlex Razvant | Senior ML & MLOps EngineerLessonsThe course is split into 11 lessons. Every Medium article will be its own lesson.An End-to-End Framework for Production-Ready LLM Systems by Building Your LLM TwinThe Importance of Data Pipelines in the Era of Generative AIChange Data Capture: Enabling Event-Driven ArchitecturesSOTA Python Streaming Pipelines for Fine-tuning LLMs and RAG  in Real-Time!Vector DB retrieval clients [Module 2] WIPTraining data preparation [Module 3] WIPFine-tuning LLM [Module 3] WIPLLM evaluation [Module 4] WIPQuantization [Module 5] WIPBuild the digital twin inference pipeline [Module 6] WIPDeploy the digital twin as a REST API [Module 6] WIP  Check out the code on GitHub [1] and support us with a Lets start with Lesson 4  Lesson 4: Python Streaming Pipelines for Fine-tuning LLMs and RAG  in Real-Time!In the 4th lesson, we will focus on the feature pipeline.The feature pipeline is the first pipeline presented in the 3 pipeline architecture: feature, training and inference pipelines.A feature pipeline is responsible for taking raw data as input, processing it into features, and storing it in a feature store, from which the training & inference pipelines will use it.The component is completely isolated from the training and inference code. All the communication is done through the feature store.To avoid repeating myself, if you are unfamiliar with the 3 pipeline architecture, check out Lesson 1 for a refresher.By the end of this article, you will learn to design and build a production-ready feature pipeline that:uses Bytewax as a stream engine to process data in real-time;ingests data from a RabbitMQ queue;uses SWE practices to process multiple data types: posts, articles, code;cleans, chunks, and embeds data for LLM fine-tuning and RAG;loads the features to a Qdrant vector DB.Note: In our use case, the feature pipeline is also a streaming pipeline, as we use a Bytewax streaming engine. Thus, we will use these words interchangeably.We will wrap up Lesson 4 by showing you how to deploy the feature pipeline to AWS and integrate it with the components from previous lessons: data collection pipeline, MongoDB, and CDC.In the 5th lesson, we will go through the vector DB retrieval client, where we will teach you how to query the vector DB and improve the accuracy of the results using advanced retrieval techniques.Excited? Lets get started!The architecture of the feature/streaming pipeline.Table of ContentsWhy are we doing this?System design of the feature pipelineThe Bytewax streaming flowPydantic data modelsLoad data to QdrantThe dispatcher layerPreprocessing steps: Clean, chunk, embedThe AWS infrastructureRun the code locallyDeploy the code to AWS & Run it from the cloudConclusion  Check out the code on GitHub [1] and support us with a 1. Why are we doing this?A quick reminder from previous lessonsTo give you some context, in Lesson 2, we crawl data from LinkedIn, Medium, and GitHub, normalize it, and load it to MongoDB.In Lesson 3, we are using CDC to listen to changes to the MongoDB database and emit events in a RabbitMQ queue based on any CRUD operation done on MongoDB.and here we are in Lesson 4, where we are building the feature pipeline that listens 24/7 to the RabbitMQ queue for new events to process and load them to a Qdrant vector DB.The problem we are solvingIn our LLM Twin use case, the feature pipeline constantly syncs the MongoDB warehouse with the Qdrant vector DB while processing the raw data into features.Important: In our use case, the Qdrant vector DB will be our feature store.Why we are solving itThe feature store will be the central point of access for all the features used within the training and inference pipelines.For consistency and simplicity, we will refer to different formats of our text data as features.  The training pipeline will use the feature store to create fine-tuning datasets for your LLM twin.  The inference pipeline will use the feature store for RAG.For reliable results (especially for RAG), the data from the vector DB must always be in sync with the data from the data warehouse.The question is, what is the best way to sync these 2?Other potential solutionsThe most common solution is probably to use a batch pipeline that constantly polls from the warehouse, computes a difference between the 2 databases, and updates the target database.The issue with this technique is that computing the difference between the 2 databases is extremely slow and costly.Another solution is to use a push technique using a webhook. Thus, on any CRUD change in the warehouse, you also update the source DB.The biggest issue here is that if the webhook fails, you have to implement complex recovery logic.Lesson 3 on CDC covers more of this.2. System design of the feature pipeline: our solutionOur solution is based on CDC, a queue, a streaming engine, and a vector DB:  CDC adds any change made to the Mongo DB to the queue (read more in Lesson 3).  the RabbitMQ queue stores all the events until they are processed.  The Bytewax streaming engine cleans, chunks, and embeds the data.  A streaming engine works naturally with a queue-based system.  The data is uploaded to a Qdrant vector DB on the flyWhy is this powerful?Here are 4 core reasons:The data is processed in real-time.Out-of-the-box recovery system: If the streaming pipeline fails to process a message will be added back to the queueLightweight: No need for any diffs between databases or batching too many recordsNo I/O bottlenecks on the source database  It solves all our problems!The architecture of the feature/streaming pipeline.How is the data stored?We store 2 snapshots of our data in the feature store. Here is why  Remember that we said that the training and inference pipeline will access the features only from the feature store, which, in our case, is the Qdrant vector DB?Well, if we had stored only the chunked & embedded version of the data, that would have been useful only for RAG but not for fine-tuning.Thus, we make an additional snapshot of the cleaned data, which will be used by the training pipeline.Afterward, we pass it down the streaming flow for chunking & embedding.How do we process multiple data types?How do you process multiple types of data in a single streaming pipeline without writing spaghetti code?Yes, that is for you, data scientists! Jokingam I?We have 3 data types: posts, articles, and code.Each data type (and its state) will be modeled using Pydantic models.To process them we will write a dispatcher layer, which will use a creational factory pattern [9] to instantiate a handler implemented for that specific data type (post, article, code) and operation (cleaning, chunking, embedding).The handler follows the strategy behavioral pattern [10].Intuitively, you can see the combination between the factory and strategy patterns as follows:Initially, we know we want to clean the data, but as we dont know the data type, we cant know how to do so.What we can do, is write the whole code around the cleaning code and abstract away the login under a Handler() interface (aka the strategy).When we get a data point, the factory class creates the right cleaning handler based on its type.Ultimately the handler is injected into the rest of the system and executed.By doing so, we can easily isolate the logic for a given data type & operation while leveraging polymorphism to avoid filling up the code with 1000x if else statements.We will dig into the implementation in future sections.Streaming over batchYou may ask why we need a streaming engine instead of implementing a batch job that polls the messages at a given frequency.That is a valid question.The thing is thatNowadays, using tools such as Bytewax makes implementing streaming pipelines a lot more frictionless than using their JVM alternatives.The key aspect of choosing a streaming vs. a batch design is real-time synchronization between your source and destination DBs.In our particular case, we will process social media data, which changes fast and irregularly.Also, for our digital twin, it is important to do RAG on up-to-date data. We dont want to have any delay between what happens in the real world and what your LLM twin sees.That being said choosing a streaming architecture seemed natural in our use case.3. The Bytewax streaming flowThe Bytewax flow is the central point of the streaming pipeline. It defines all the required steps, following the next simplified pattern: input -> processing -> output.As I come from the AI world, I like to see it as the graph of the streaming pipeline, where you use the input(), map(), and output() Bytewax functions to define your graph, which in the Bytewax world is called a flow.As you can see in the code snippet below, we ingest posts, articles or code messages from a RabbitMQ queue. After we clean, chunk and embed them. Ultimately, we load the cleaned and embedded data to a Qdrant vector DB, which in our LLM twin use case will represent the feature store of our system.To structure and validate the data, between each Bytewax step, we map and pass a different Pydantic model based on its current state: raw, cleaned, chunked, or embedded.Bytewax flow   GitHub Code We have a single streaming pipeline that processes everything.As we ingest multiple data types (posts, articles, or code snapshots), we have to process them differently.To do this the right way, we implemented a dispatcher layer that knows how to apply data-specific operations based on the type of message.More on this in the next sections  Why Bytewax?Bytewax is an open-source streaming processing framework that:- is built in Rust  for performance- has Python   bindings for leveraging its powerful ML ecosystem so, for all the Python fanatics out there, no more JVM headaches for you.Jokes aside, here is why Bytewax is so powerful  - Bytewax local setup is plug-and-play- can quickly be integrated into any Python project (you can go wild  even use it in Notebooks)- can easily be integrated with other Python packages (NumPy, PyTorch, HuggingFace, OpenCV, SkLearn, you name it)- out-of-the-box connectors for Kafka and local files, or you can quickly implement your ownWe used Bytewax to build the streaming pipeline for the LLM Twin course and loved it.To learn more about Bytewax, go and check them out. They are open source, so no strings attached   Bytewax [2] 4. Pydantic data modelsLets take a look at what our Pydantic models look like.First, we defined a set of base abstract models for using the same parent class across all our components.Pydantic base model structure   GitHub Code Afterward, we defined a hierarchy of Pydantic models for:all our data types: posts, articles, or codeall our states: raw, cleaned, chunked, and embeddedThis is how the set of classes for the posts will look like  Pydantic posts model structure   GitHub Code We repeated the same process for the articles and code model hierarchy.Check out the other data classes on our GitHub.Why is keeping our data in Pydantic models so powerful?There are 4 main criteria:every field has an enforced type: you are ensured the data types are going to be correctthe fields are automatically validated based on their type: for example, if the field is a string and you pass an int, it will through an errorthe data structure is clear and verbose: no more clandestine dicts that you never know what is in themyou make your data the first-class citizen of your program5. Load data to QdrantThe first step is to implement our custom Bytewax DynamicSink class  Qdrant DynamicSink   GitHub Code Next, for every type of operation we need (output cleaned or embedded data ) we have to subclass the StatelessSinkPartition Bytewax class (they also provide a stateful option   more in their docs)An instance of the class will run on every partition defined within the Bytewax deployment.In the course, we are using a single partition per worker. But, by adding more partitions (and workers), you can quickly scale your Bytewax pipeline horizontally.Qdrant worker partitions   GitHub Code Note that we used Qdrants Batch method to upload all the available points at once. By doing so, we reduce the latency on the network I/O side: more on that here [8] The RabbitMQ streaming input follows a similar pattern. Check it out here 6. The dispatcher layerNow that we have the Bytewax flow and all our data models.How do we map a raw data model to a cleaned data model?  All our domain logic is modeled by a set of Handler() classes.For example, this is how the handler used to map a PostsRawModel to a PostCleanedModel looks like  Handler hierarchy of classes   GitHub Code Check out the other handlers on our GitHub:  ChunkingDataHandler and EmbeddingDataHandlerIn the next sections, we will explore the exact cleaning, chunking and embedding logic.Now, to build our dispatcher, we need 2 last components:a factory class: instantiates the right handler based on the type of the eventa dispatcher class: the glue code that calls the factory class and handlerHere is what the cleaning dispatcher and factory look like  The dispatcher and factory classes   GitHub Code Check out the other dispatchers on our GitHub.By repeating the same logic, we will end up with the following set of dispatchers:RawDispatcher (no factory class required as the data is not processed)CleaningDispatcher (with a ChunkingHandlerFactory class)ChunkingDispatcher (with a ChunkingHandlerFactory class)EmbeddingDispatcher (with an EmbeddingHandlerFactory class)7. Preprocessing steps: Clean, chunk, embedHere we will focus on the concrete logic used to clean, chunk, and embed a data point.Note that this logic is wrapped by our handler to be integrated into our dispatcher layer using the Strategy behavioral pattern [10].We already described that in the previous section. Thus, we will directly jump into the actual logic here, which can be found in the utils module of our GitHub repository.Note: These steps are experimental. Thus, what we present here is just the first iteration of the system. In a real-world scenario, you would experiment with different cleaning, chunking or model versions to improve it on your data.CleaningThis is the main utility function used to clean the text for our posts, articles, and code.Out of simplicity, we used the same logic for all the data types, but after more investigation, you would probably need to adapt it to your specific needs.For example, your posts might start containing some weird characters, and you dont want to run the unbold_text() or unitalic_text() functions on your code data point as is completely redundant.Cleaning logic   GitHub Code Most of the functions above are from the unstructured [3] Python package. It is a great tool for quickly finding utilities to clean text data.  More examples of unstructured here [3] One key thing to notice is that at the cleaning step, we just want to remove all the weird, non-interpretable characters from the text.Also, we want to remove redundant data, such as extra whitespace or URLs, as they do not provide much value.These steps are critical for our tokenizer to understand and efficiently transform our string input into numbers that will be fed into the transformer models.Note that when using bigger models (transformers) + modern tokenization techniques, you dont need to standardize your dataset too much.For example, it is redundant to apply lemmatization or stemming, as the tokenizer knows how to split your input into a commonly used sequence of characters efficiently, and the transformers can pick up the nuances of the words.  What is important at the cleaning step is to throw out the noise.ChunkingWe are using Langchain to chunk our text.We use a 2 step strategy using Langchains RecursiveCharacterTextSplitter [4] and SentenceTransformersTokenTextSplitter [5]. As seen below  Chunking logic   GitHub Code Overlapping your chunks is a common pre-indexing RAG technique, which helps to cluster chunks from the same document semantically.Again, we are using the same chunking logic for all of our data types, but to get the most out of it, we would probably need to tweak the separators, chunk_size, and chunk_overlap parameters for our different use cases.But our dispatcher + handler architecture would easily allow us to configure the chunking step in future iterations.EmbeddingThe data preprocessing, aka the hard part is done.Now we just have to call an embedding model to create our vectors.Embedding logic   GitHub Code We used the all-MiniLm-L6-v2 [6] from the sentence-transformers library to embed our articles and posts: a lightweight embedding model that can easily run in real-time on a 2 vCPU machine.As the code data points contain more complex relationships and specific jargon to embed, we used a more powerful embedding model: hkunlp/instructor-xl [7].This embedding model is unique as it can be customized on the fly with instructions based on your particular data. This allows the embedding model to specialize on your data without fine-tuning, which is handy for embedding pieces of code.8. The AWS infrastructureIn Lesson 2, we covered how to deploy the data collection pipeline that is triggered by a link to Medium, Substack, LinkedIn or GitHub   crawls the given link   saves the crawled information to a MongoDB.In Lesson 3, we explained how to deploy the CDC components that emit events to a RabbitMQ queue based on any CRUD operation done to MongoDB.What is left is to deploy the Bytewax streaming pipeline and Qdrant vector DB.We will use Qdrants self-hosted option, which is easy to set up and scale.To test things out, they offer a Free Tier plan for up to a 1GB cluster, which is more than enough for our course.  We explained in our GitHub repository how to configure Qdrant.AWS infrastructure of the feature/streaming pipeline.The last piece of the puzzle is the Bytewax streaming pipeline.As we dont require a GPU and the streaming pipeline needs to run 24/7, we will deploy it to AWS Fargate, a cost-effective serverless solution from AWS.As a serverless solution, Fargate allows us to deploy our code quickly and scale it fast in case of high traffic.How do we deploy the streaming pipeline code to Fargate?Using GitHub Actions, we wrote a CD pipeline that builds a Docker image on every new commit made on the main branch.After, the Docker image is pushed to AWS ECR. Ultimately, Fargate pulls the latest version of the Docker image.This is a common CD pipeline to deploy your code to AWS services.Why not use lambda functions, as we did for the data pipeline?An AWS lambda function executes a function once and then closes down.This worked perfectly for the crawling logic, but it won't work for our streaming pipeline, which has to run 24/7.9. Run the code locallyTo quickly test things up, we wrote a docker-compose.yaml file to spin up the MongoDB, RabbitMQ queue and Qdrant vector db.You can spin up the Docker containers using our Makefile by running the following:make local-start-infraTo fully test the Bytewax streaming pipeline, you have to start the CDC component by running:make local-start-cdcUltimately, you start the streaming pipeline:make local-bytewaxTo simulate the data collection pipeline, mock it as follows:make local-insert-data-mongoThe README of our GitHub repository provides more details on how to run and set up everything.10. Deploy the code to AWS & Run it from the cloudThis article is already too long, so I wont go into the details of how to deploy the AWS infrastructure described above and test it out here.But to give you some insights, we have used Pulumi as our infrastructure as a code (IaC) tool, which will allow you to spin it quickly with a few commands.Also, I wont let you hang on to this one. We made a promise and  We prepared step-by-step instructions in the README of our GitHub repository on how to use Pulumni to spin up the infrastructure and test it out.ConclusionNow you know how to write streaming pipelines like a PRO!In Lesson 4, you learned how to:design a feature pipeline using the 3-pipeline architecturewrite a streaming pipeline using Bytewax as a streaming engineuse a dispatcher layer to write a modular and flexible application to process multiple types of data (posts, articles, code)load the cleaned and embedded data to Qdrantdeploy the streaming pipeline to AWS  This is only the ingestion part used for fine-tuning LLMs and RAG.In Lesson 5, you will learn how to write a retrieval client for the 3 data types using good SWE practices and improve the retrieval accuracy using advanced retrieval & post-retrieval techniques. See you there!  Check out the code on GitHub [1] and support us with a Enjoyed This Article?Join the Decoding ML Newsletter for battle-tested content on designing, coding, and deploying production-grade ML & MLOps systems. Every week. For FREE  Decoding ML Newsletter | Paul Iusztin | SubstackJoin for battle-tested content on designing, coding, and deploying production-grade ML & MLOps systems. Every week. Fordecodingml.substack.comReferencesLiterature[1] Your LLM Twin Course  GitHub Repository (2024), Decoding ML GitHub Organization[2] Bytewax, Bytewax Landing Page[3] Unstructured Cleaning Examples, Unstructured Documentation[4] Recursively split by character, LangChains Documentation[5] Split by tokens, LangChains Documentation[6] sentence-transformers/all-MiniLM-L6-v2, HuggingFace[7] hkunlp/instructor-xl, HuggingFace[8] Qdrant, Qdrant Documentation[9] Abstract Factory Pattern, Refactoring Guru[10] Strategy Pattern, Refactoring GuruImagesIf not otherwise stated, all images are created by the author.Sign up to discover human stories that deepen your understanding of the world.FreeDistraction-free reading. No ads.Organize your knowledge with lists and highlights.Tell your story. Find your audience.Sign up for freeMembershipAccess the best member-only stories.Support independent authors.Listen to audio narrations.Read offline.Join the Partner Program and earn for your writing.Try for $5/monthMl System DesignMachine LearningArtificial IntelligenceData ScienceSoftware Engineering698698FollowWritten by Paul Iusztin2.4K FollowersEditor for Decoding ML Senior ML Engineer | Helping machine learning engineers design and productionize ML systems. | Decoding ML Newsletter: [URL] from Paul Iusztin and Decoding MLPaul IusztininDecoding MLAn End-to-End Framework for Production-Ready LLM Systems by Building Your LLM TwinFrom data gathering to productionizing LLMs using LLMOps good practices.16 min readMar 16, 20241.5K10Razvant AlexandruinDecoding MLHow to build a Real-Time News Search Engine using Serverless Upstash Kafka and Vector DBA hands-on guide to implementing a live news aggregating streaming pipeline with Apache Kafka, Bytewax, and Upstash Vector Database.19 min readApr 13, 20246873Paul IusztininDecoding MLThe LLMs kit: Build a production-ready real-time financial advisor system using streamingLesson 1: LLM architecture system design using the 3-pipeline pattern12 min readJan 5, 2024383Paul IusztininDecoding MLA Real-time Retrieval System for RAG on Social Media DataUse a streaming engine to populate a vector DB in real-time. Improve RAG accuracy using rerank & UMAP.12 min readMar 30, 2024305See all from Paul IusztinSee all from Decoding MLRecommended from MediumVipra SinghBuilding LLM Applications: Serving LLMs (Part 9)Learn Large Language Models ( LLM ) through the lens of a Retrieval Augmented Generation ( RAG ) Application.49 min readApr 18, 20243741A B Vijay KumarMulti-Agent System\\u200a\\u200aCrew.AIMulti-Agent systems are LLM applications that are changing the automation landscape with intelligent bots.7 min readApr 18, 20241574ListsPredictive Modeling w/ Python20 stories1126 savesNatural Language Processing1402 stories899 savesPractical Guides to Machine Learning10 stories1352 savesdata science and AI40 stories137 savesFabio MatricardiinGenerative AILlama3 is out and you can run it on your Computer!After only 1 day from the release, here is how you can run even on your Laptop with CPU only the latest Meta-AI model.8 min read6 days ago1.2K13Gavin LiinAI AdvancesRun the strongest open-source LLM model: Llama3 70B with just a single 4GB GPU!The strongest open source LLM model Llama3 has been released, Here is how you can run Llama3 70B locally with just 4GB GPU, even on Macbook4 min read4 days ago7982Plaban NayakinThe AI ForumSemantic Chunking for RAGWhat is Chunking\\xa0?17 min read4 days ago1651Damian GilinTowards Data ScienceAdvanced Retriever Techniques to Improve Your RAGsMaster Advanced Information Retrieval: Cutting-edge Techniques to Optimize the Selection of Relevant Documents with Langchain to Create18 min readApr 17, 20245393See more recommendationsHelpStatusAboutCareersBlogPrivacyTermsText to speechTeams\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nTo make Medium work, we log user data. By using Medium, you agree to our Privacy Policy, including cookie policy."
  },
  {
    "instruction": "Generate a publication that offers battle-tested content on building production-grade ML systems leveraging good SWE and MLOps practices. Follow for learning about the entire lifecycle of an ML system.",
    "content": "About Decoding MLThe hub for continuous learning on ML system design, ML engineering, MLOps, LLMs and computer vision.About Decoding ML. The hub for continuous learning on ML | by Paul Iusztin | Decoding ML | MediumOpen in appSign upSign inWriteSign upSign inMastodonAbout Decoding MLThe hub for continuous learning on ML system design, ML engineering, MLOps, LLMs and computer vision.Paul IusztinFollowPublished inDecoding ML2 min readFeb 19, 202491ListenShareDecoding ML is a publication that creates battle-tested content on building production-grade ML systems leveraging good SWE and MLOps practices.Our motto is More engineering, less F1 scores.Following Decoding ML, you will learn about the entire lifecycle of an ML system, from system design to deploying and monitoring.Decoding ML is the hub for continuous learning on:ML system designML engineeringMLOpsLarge language modelsComputer visionWe are all about end-to-end ML use cases that can directly be applied in the real world  no stories  just hands-on content.The minds behind Decoding ML  Alex Vesa (left), Paul Iusztin (middle) and Alexandru Razvant (right)The minds behind Decoding ML are Vesa Alexandru, Paul Iusztin and Razvant Alexandru.Our passion for constantly learning and engineering production-ready ML systems inevitably turned into Decoding ML.With our 10+ years of hands-on experience in the industry in:Data ScienceDeep LearningComputer VisionGenerative AIML InfrastructureMLOpsSoftware Engineeringwe decided it is about time to share it with the world!Why follow?Join Decoding ML for battle-tested content on designing, coding, and deploying production-grade ML & MLOps systems. Every week. For FREE.No more bedtime stories in Jupyter Notebooks.  DML is all about hands-on advice from our 10+ years of experience in AI.We are also on: Newsletter  GitHubSign up to discover human stories that deepen your understanding of the world.FreeDistraction-free reading. No ads.Organize your knowledge with lists and highlights.Tell your story. Find your audience.Sign up for freeMembershipAccess the best member-only stories.Support independent authors.Listen to audio narrations.Read offline.Join the Partner Program and earn for your writing.Try for $5/monthAbout9191FollowWritten by Paul Iusztin2.4K FollowersEditor for Decoding ML Senior ML Engineer | Helping machine learning engineers design and productionize ML systems. | Decoding ML Newsletter: [URL] from Paul Iusztin and Decoding MLPaul IusztininDecoding MLAn End-to-End Framework for Production-Ready LLM Systems by Building Your LLM TwinFrom data gathering to productionizing LLMs using LLMOps good practices.16 min readMar 16, 20241.5K10Razvant AlexandruinDecoding MLHow to build a Real-Time News Search Engine using Serverless Upstash Kafka and Vector DBA hands-on guide to implementing a live news aggregating streaming pipeline with Apache Kafka, Bytewax, and Upstash Vector Database.19 min readApr 13, 20246873Paul IusztininDecoding MLThe LLMs kit: Build a production-ready real-time financial advisor system using streamingLesson 1: LLM architecture system design using the 3-pipeline pattern12 min readJan 5, 2024383Paul IusztininDecoding MLA Real-time Retrieval System for RAG on Social Media DataUse a streaming engine to populate a vector DB in real-time. Improve RAG accuracy using rerank & UMAP.12 min readMar 30, 2024305See all from Paul IusztinSee all from Decoding MLRecommended from MediumQwakHow to Build an End-to-End ML Pipeline in 2024Learn to build an end-to-end ML pipeline and streamline your ML workflows in 2024, from data ingestion to model deployment and performance24 min readApr 7, 20242071Paul IusztininDecoding MLThe LLMs kit: Build a production-ready real-time financial advisor system using streamingLesson 1: LLM architecture system design using the 3-pipeline pattern12 min readJan 5, 2024383ListsMedium's Huge List of Publications Accepting Submissions285 stories2516 savesYesmine RouisinTheFork Engineering BlogA Guide to MLOps with Airflow and MLflowIntroduction11 min readNov 6, 2023319ManralaiMastering MLOps for FreeYour Go-To GitHub Repositories Collection4 min readJan 28, 20246463Vipra SinghBuilding LLM Applications: Evaluation (Part 8)Learn Large Language Models ( LLM ) through the lens of a Retrieval Augmented Generation ( RAG ) Application.47 min readApr 8, 20241891Ramazan OlmezEnd-to-End Machine Learning Project: Churn PredictionThe main objective of this article is to develop an end-to-end machine learning project. For a model to be truly useful, it needs to be18 min readFeb 22, 202487See more recommendationsHelpStatusAboutCareersBlogPrivacyTermsText to speechTeams\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nTo make Medium work, we log user data. By using Medium, you agree to our Privacy Policy, including cookie policy."
  }
]

Conclusion

In Lesson 6, you learned how to:

• understanding fine-tuning importance
• use an LLM to generate instructions for crawled data
• load the cleaned and embedded data to Qdrant
• dataset quality is crucial
• data versioning with CometML: Implementing data versioning using CometML ensures reproducibility of experiments, aids in model diagnostics, and supports regulatory compliance. This is crucial for maintaining a robust and traceable MLOps workflow.

In Lesson 7, you will learn how to use the generated dataset to finetune a Mistral-7b for content generation building and deploying the training pipeline to Qwak an AI serverless platform.

🔗 Check out the code on GitHub [1] and support us with a ⭐️

Enjoyed This Article?

Join the Decoding ML Newsletter for battle-tested content on designing, coding, and deploying production-grade ML & MLOps systems. Every week. For FREE ↓

Decoding ML Newsletter | Paul Iusztin | Substack

References

Literature

[1] Your LLM Twin Course — GitHub Repository (2024), Decoding ML GitHub Organization
[2] MLFlow Alternatives, Neptune.ai

Images

If not otherwise stated, all images are created by the author.