Create AWS baby datalakes to handle ongoing daily data batch

In the era of micro-service architecture, AI (or BI)-powered applications are structured as a collection of services which are loosely coupled, easily maintained, independently organized, and owned by smaller teams. This is possible by containerizing different services within an application and using API calls to communicate among them. Finally, these AI-fueled-apps are also deployed in a cloud-native approach and consequently build upon service-oriented architecture (SOA) and event-driven architecture (EDA).

Problems

When you are a Business Analyst you spend most of your time to collect data coming from many sources, gather it in the same place to laboriously analyze chunks of files enabling you to draw some partial and inaccurate insights in order to finally leverage business opportunities. In this daily time-consuming routine, it should be noted Business Analysts are used to struggling to open a file on Excel — their favorite tool — due to a too large size of the given file or some poor memory capacity of their 32-bit PCs.

Here we offer to set up a robust serverless data pipeline to 1️⃣automatically ingest data…