Design, Development and Deployment of a simple Data Pipeline

Data Engineering technical challenge (part 1)

Dockerizing a simple data pipeline

This article is based on solving a technical challenge for the data engineering position, in the development of this article we will show all the steps to follow in order to design a quick solution to the challenge and then deployment it, we will follow the next steps and I will try to go straight to the point on each of them:

1. Read and analyze requirements.
2. Ask ourselves questions about some requirements
3. Rule out quickly optional requirements
4. Design data architecture and choose technologies
5. Development
6. Deployment plan
7. Show results

Let’s start

1. Read and analyze requirements.

Let’s read some of them and highlight what matters:

You will create a mechanism to ingest data from one server to a postgres database and build a REST API to get the records from the database.

• ingest data from one server to a postgres database
• build a REST API to get the records from the database

The project consists to design the database…