Creating End-to-End AI Data Product(image processing) App with MLOps by using Docker, ELK, Airflow, RabbitMQ
Note: This is the main article about building the system above. Details and instructions on how to implement parts of the system are written in other Medium articles which are linked below this article.
In this tutorial I will try to explain how to create an End-to End data pipeline which we will build on the Google Cloud Platform. Be wary as I am not an expert on the subject, but a mere student and I am sure there are better ways to build such a system. My reason for building this project was to learn/practice. Also, English is not my first language.
The system’s function is to detect faulty marble products on a production line in real-time.
The system we will be building consists of:
- Getting image data form a source (our camera is fictional)
- Processing the data
- Storing data on a NoSQL database (ElasticSearch in our case)
- Classifying with machine learning model
- Publishing to a message broker (RabbitMQ)
- Consuming from a message broker
- Building a very basic interface to see the results.
The tech/tools we will be using:
- Google Cloud Platform (Compute Engine)
- Docker
- Linux (Debian)
- Python
- Elastic Stack (ElasticSearch, Filebeat, Kibana)
- Apache Airflow
- RabbitMQ
Tech Details
Google Cloud Platform: GCP is the cloud service of Google. Thanks to this service, we can build our system on Google’s servers. This practice has many advantages compared to a local system. Some of these advantages are as follows:
- We save from hardware costs.
- If things go wrong, we can just create a new machine on the cloud.
- Our machine on the cloud stays up all the time.
Docker: Docker lets us containerize programs and run them wherever we want. Docker containers contain apps and their pre-requirements. This allows us to install our tools without installing their requirements first.
Docker is especially useful in Cloud platforms since we don’t need to relay on what is currently installed on the host thanks to it.
Elastic Stack (ELK): By Elastic Stack, we mean ElasticSearch, Logstash and Kibana. I didn’t see any reason to use Logstash since we only have one data source in this project. Additionally to the ELK stack, we will install Filebeat to detect when data arrives at somewhere.
ElasticSearch is used as a NoSQL database. It searches text-based data with ease. Our images will be converted to a text-based format called Base64 before saving to the database, so it fits our case. Also ElasticSearch being compatible with Kibana and Filebeat is the most important reason we are using it.
Kibana lets us monitor what’s inside out ElasticSearch Database. We can see our data, indexes, documents and metadata thanks to Kibana.
Filebeat is a part of the ELK stack. It detects when new data arrives where it is looking for and sends it to our ElasticSearch database. We will use it to listen to the location where images from the camera arrives every few seconds.
Apache Airflow: Airflow schedules, orchestrates and monitors tasks. It allows us to make tasks dependent on other tasks. Also, we can schedule tasks to run every minute using Airflow.
We will create a DAG (directed acyclic graph) that contains our tasks. This DAG will run every minute to maintain a real-time data process. Inside our DAG we will have 2 tasks: An image to Base64 encoder and a prediction script that contains our trained machine learning model. The prediction script will depend on the encoder script running before it, so that it doesn’t try to predict without a Base64 formatted data existing.
RabbitMQ: This is a message broker. Thanks to RabbitMQ we can deliver data and messages between tools and applications.
In its most basic form, it has a producer, a queue and a consumer. Producer is any app that publishes messages to the RabbitMQ queue. Messages are kept in the queue until they are consumed by a consumer app. Messages from the queues are consumed as First-In-First-Out style, meaning first message that’s published to the queue is also consumed first, which makes it a “queue”.
Steps to follow for building the system:
I will explain every detail. So as not to turn this article to a messy salad, I will divide this tutorial into multiple articles.
Follow below articles in order to build the system from zero.
You can see the end product in this video:
