This quote is said by Kurt Bollacker, a famous data scientist. Data Integration procedures are a way to love data in order to make it survive. These procedures involve many things, like cleansing, ETL/ELT mapping, and transformation.

In this blog, we are specifically going to discuss ETL and ELT, comparing both of them and discussing their specific use cases.

ETL

ETL stands for Extract-Transform-Load. ETL is a data integration process that extracts data from multiple data sources, transforms them by modelling, and combines them into a single, consistent data store that is loaded into a data warehouse.

Benefits of ETL

ETL has several business benefits, beyond extracting, transforming, and loading data, delivering it from source to destination.

• Context: ETL helps businesses gain deep historical context with data.
• Consolidation: It provides a consolidated view of data, for easier analysis and reporting.
• Productivity: It improves productivity with repeatable processes that don’t require heavy hand-coding.
• Accuracy: It improves data accuracy and audit capabilities that most businesses require for compliance with regulations and standards.

ELT

ELT stands for Extract-Load-Transform. It is a data integration process that extracts data from multiple sources and just loads it into a data lake. The transformation is done by hardware resources of the data lake itself, which is an advantage over ETL.

Benefits of ELT

ELT has many benefits, like the following:

• Agility: The data lake in the ELT process stores the unstructured data and makes it readily available for use. Data modelling to transform unstructured data can be set up as per requirement. One doesn’t have to think about the structure before loading data into the data lake.
• Simplicity: SQL transformations are generally used for data transformation in the data lake. This generalisation allows the entire team to work upon the transformation logic.
• Self-Service Analytics: Having all raw data in the data lake, one can use BI tools to fetch the required aggregated summary statistics of the raw data.
• Bug Fixing: If there is an error in the transformation algorithm, only that algorithm needs to be fixed. In contrast to this in ETL, we have to re-deploy the whole pipeline after bug-fixing.

ETL vs ELT: Detailed Comparison

In my previous blog, I gave an overview of the differences between ETL and ELT. Let’s do a detailed comparison over here:

Process:

ETL: The staging server transforms the data and then transfers the same to data warehouse DB.

ELT: Here data remains in the DB of the data warehouse.

Transformation:

ETL: ETL server/staging area handles the whole transformation process.

ELT: The target system performs the transformation process.

Complexity:

ETL: The ETL process loads only the important data, as identified at design time.

ELT: This process involves development from the output-backward and loading only relevant data.

Calculations:

ETL: Overwrites existing column and pushes the same to the targeted platform.

ELT: Easily add the calculated column to the existing table.

Time-Maintenance:

ETL: It needs high maintenance as you need to select data to load and transform.

ELT: Low maintenance as data is always available.

ETL vs ELT: When to use which

The following decision tree may help to choose which paradigm to pick while setting up data pipelines.

Simple conclusion, we can say:

• If our target system is a cloud-based data warehouse like AWS Redshift or Google BigQuery, then performing transformations after loading is extremely simple.
• If our target system is much like a traditional data lake (flat storage), then transformations should be performed before loading as traditional data lakes offer very few compute resources, for performing transformations. Moreover, if we have very few compute-intensive transformations, performing them before loading would be much better.

The nature of transformations, we require to run also plays a key role while choosing between ETL and ELT.

If your transformations are simple and/or your endpoint does not support complex transformations, then you should definitely choose an ETL process and perform the transformations prior to loading. However, if your transformations are complex and subject to frequent changes and your endpoint supports the processing power, then an ELT process gives you the flexibility to run any transformations after loading the data.

About Me

Myself Tejas Agrawal, an undergraduate student at the Indian Institute of Information Technology, Allahabad. Currently, I’m working as Data Engineering and Analytics Intern at Headout.com, where I find and develop data-driven solutions to business problems. I love to explore new industry standards and open source technology stacks.

Connect with me on LinkedIn.

Big Data Integration Procedures was originally published in Nybles on Medium, where people are continuing the conversation by highlighting and responding to this story.

“Use the right tool for the job!”

This saying, which sounds quite simple at first, is much complex to implement in a practical aspect.

Early-stage startups, find it difficult to choose the variety of tools that are available in the ecosystem because it is quite unpredictable how their data is going to evolve.

Need of a Modern Data Stack

In the last 10 years, the software industry has seen a rise in:

• Computational Power: Public Cloud Providers like AWS, Google Cloud offers enormous computing power at a standard market cost.
• Data Sources: The rise of the IoT ecosystem, smart devices, have caused an exponential increase in the amount of data generated per day. In 2020, every person on the planet has generated ~1.7MB of data per second.
• Data Literacy among business stakeholders: In the primitive software industry, analysts used to dig up excel spreadsheets manually to get some valuable insights about data. Today, many BI tools have proved useful in leveraging the power of data and giving valuable insights, thus creating literacy among business stakeholders.
• Open Source Adoption in Data Projects: Industry has seen a tremendous rise in open source communities in the last 10 years. Many cool data tools (~Apache Airflow, DBT, Metabase) have thrived and evolved in open source communities.

The shift from Legacy ETL to Modern ELT

In this modern era, where most of the corporates are leveraging data-driven solutions, we are seeing a consistent shift to ELT architecture, from primitive legacy ETL architecture.

Modern ELT processing has become popular over legacy ETL because of the following reasons:

1. Cheap, affordable, and efficient cloud storage and analytics services.
2. Legacy ETL pipelines are not that flexible to easily adapt according to the exponential data growth.
3. Modern ELT is fast as compared to legacy ETL as no strict transformation stage is involved before loading data into the warehouse.
4. ELT tools are much proficient at simply plugging source data into the target system with minimal manual work from the user given that user-defined transformations are not required.
5. Analysts can perform transformations on the data as needed in the warehouse with tools like DBT without taking care of insights and data types beforehand.

Adoption Strategy for startups

As mentioned previously in this blog, it is difficult for startups to anticipate the evolution of the data, they are going to deal with.

So, early-stage startups should consider the following things while choosing tools for their data stack:

1. High adoption and awareness among other startups and customers.
2. Which fits into the ELT model of the data stack.
3. Database paradigms (e.g. structured, geospatial, entity-relations, search engine), that would fit into the requirement of storing and querying data produced by their domain and market.
4. Equivalent open source alternatives of paid SaaS tools.

Extraction and Loading

Collection of data from all event sources, like web, app, backend services, and sending them to the data warehouse.

Paid SaaS Tools: Stitch, Fivetran, HevoData

Free and Open Source Alternatives: Singer, Meltano, Airbyte

Data Warehouse

A structured, nonvolatile, single source of truth of all data of the organization, where we can store and query all data.

Paid: AWS Redshift, Google BigQuery, Snowflake

Free and Open Source alternatives: Apache Druid

Transformation and Modelling

Creating models out of raw data with documentation for better consumption.

Paid: Dataform, DBT

Free and Open Source Alternatives: Talend Open Studio, Apache NiFi

Orchestration

Softwares for executing and orchestrating jobs that handle data flow.

Paid: Prefect.io

Free and Open Source Alternatives: Apache Airflow, Dagster

Visualization and Analytics

For better visibility and interpretation of data from different data sources.

Paid: Tableau, Microsoft PowerBI, Grafana

Free and Open Source Alternatives: Metabase, D3js, DyGraphs

About Me

Myself Tejas Agrawal, an undergraduate student at the Indian Institute of Information Technology, Allahabad. Currently, I’m working as Data Engineering and Analytics Intern at Saveo.in, where I find and develop data-driven solutions to business problems. I love to explore new industry standards and open source technology stacks.

Connect with me on LinkedIn.

Modern Data Stack for Startups was originally published in Nybles on Medium, where people are continuing the conversation by highlighting and responding to this story.

Introduction👼

Howdy Fellas😇!! I’m back with my another blog after a long time. Before starting, lemme give the first-time readers a brief intro of mine.

Myself Tejas Agrawal👦 (alias: cyber-venom003), a pre-final year undergraduate student at the Indian Institute of Information Technology, Allahabad🏫, an open-source enthusiast, and technology explorer of all time.

As the title suggests, I wanna share my experience of one year of my college life with you people. The thing special about this year is, that I was completely indulged in exploring the world of information technology by learning new frameworks, making some cool projects, and contributing to open-source software also.

How it all started

My development journey started in December 2019, my first vacation after my introduction to college. Some of you might know that my college is famous for its competitive programming culture after the legendary Humblefool. In my first semester, I was introduced to competitive programming by my college seniors. I found it interesting, I did it for almost 1.5 months, but then, I got carried away in the fun and enjoyment of college life xD.

Due to this break into my competitive programming journey, I never started it again due to lost interest in it. And just like most of the other college students, I got stuck in the following dilemma:

So, as I said, I love to explore new things, I chose the path of development. In December 2019, I started with Native Frontend Web Development(HTML, CSS, JavaScript). I found it much more interesting than competitive programming because, I can see my code taking the effect in real life(web browser, precisely). Besides this, I also started learning one of the famous version control tools Git, and source code management platform GitHub, and I found both of them as one of the most interesting things in this virtual world. Git and GitHub were just like magic for me because they enabled me with the following:

1. Sharing my projects to other people so that they can see and run them on their systems.
2. Collaborative working feature with other developers which gradually introduced into the World of Open Source.

Sticking on the path of development🚧

So, after getting started on the path of development, I was learning things at a quite good pace and then, I got an opportunity to participate in OpenCode IIITA: A month-long program starting in January for students to start their journey in the world of open source.

As I became quite proficient with Frontend web development by learning them in vacations, I learned further to contribute to open source projects by claiming issues and making pull requests.

I also got fascinated by knowing about GitHub Contribution Count, which gives that lovely green dots on your GitHub profile🤩. Those green dots motivated me to remain on the path of development.

Exploring new things🤠

So, as I was motivated to stay on this path, I explored new things, like development frameworks, open-source student programs, and pretty other stuff of the same kind.

I also came to know about GSoC (Google Summer of Code): an international annual program in which Google awards stipends to students who successfully complete a free and open-source software development project during the summer.

After getting to know the perks of GSoC from my seniors who were selected for this, I was very much motivated to learn in order to get selected for GSoC’21.

After learning Frontend web development, I learned Backend web development, which fascinated me much more than frontend. Because the backend is one that is purely based on business logic. I learned Backend by using Express.js as a web framework and MongoDB as a database.

After learning this, I moved to mobile application development. I started to learn Flutter: A cross-platform mobile development framework by Google. I made my first Android App “Cook Blog”: A blogging application for foodies and cooks to share their recipes, using Flutter.

Contributing to Open Source👨‍💻

After learning plenty amount of skills, I contributed to many open source projects, mostly in form of Pull Requests, but also in form of issues and code reviews. I majorly contributed during HacktoberFest by Digital Ocean and Script Winter of Code by Script Foundation.

I learned a lot by contributing to open source. I got to know how these frameworks and all work in real life(production environment, precisely). I learned how to deliver a better UX and how UI/UX affects the user engagement of the platform. I learned about bug hunting in a codebase of an open-source project. I learned about code writing conventions, best code development practices.

Lastly, I learned how to read large documentations of some frameworks and how to search for the solutions on StackOverflow xD.

Encountering Failures🥺

Just like a riveted part of everyone’s journey, failures were also destined to happen with me also.

Failure #1:

Not getting selected for MLH Fellowship Spring’21

The MLH Fellowship is an internship alternative for software engineers, with a focus on Open Source projects. Instead of working on a project for just one company, students contribute to Open Source projects that are used by companies around the world.

I applied for the MLH Fellowship Spring’21 batch. My application got shortlisted in the first round. I also cleared by behavioral round also. In a technical round, I was answering the questions of my interviewer very confidently. I left no stone unturned from my side. But there was something else destined for me:

Failure #2:

Not getting selected for GSoC’21

As I introduced you to GSoC in the above sections, you can get the idea that I was going to apply for it. I applied for GSoC’21 at Palisadoes Foundation. Again, from my side, I left no stone unturned. I contributed to the organization through some pull requests and issues, I regularly reviewed the codebase which was constantly changing due to pull requests of fellow developers aiming for that organization. I made a very elaborated proposal for integrating the chat feature in their existing Flutter application. But again, I faced failure:

Overwhelmed with sadness, a natural human emotion that triggers after some subsequent failures, I asked organization mentor if there were any unnoticed shortcomings from my side. My mentor from that organization gave me an elaborated review, which just made me calm down.

Priceless learnings from this 1 year of journey😇

So, as I said I learned many things in context of development as well as life during this 1 year of the journey, I would like to share them with you readers.

1. Always try to integrate your project with some source control manager like Git and keep them on some Git hosting platform like GitHub or GitLab. Always make a good README.md file and add a License to your project. This would help you in showcasing your project source code during interviews.
2. Make a habit of writing automated tests for your application. Try to use those tests before every commit. This method will prove very productive during the entire project development because you won’t have to hassle with manual testing. For Node.js projects, you can use mocha.js or jest.js. For python based projects, you can use PyTest.
3. For the frontend part of applications, always try to make a state-of-art UI/UX. UI/UX, in a layman terms, can be said as “Look and Feel” of the entire application. An application with a bad UI/UX is equivalent to an application offering bad performance. If you are not a UI/UX designer, you can try to take inspirations from Dribbble and Behance.
4. For backend part of applications, always “containerize” or “dockerize” them, so that other developers can clone it and run on their platform independently without any missing dependencies. Containerization can be done also for complex frontend applications which are built on some framework.
5. Always try to learn some basics of DevOps so that you can deploy your projects in a production environment (real-world environment, i.e. real live servers) so that people can actually use it and you can showcase it somewhere.
6. Never afraid of any type of errors. Always remember that you won’t be the first one to encounter it, so try to search it on google and find the relevant StackOverflow answer or Github Comment thread for it.
7. Always refer to documentations if you are stuck somewhere. Documentations also help in the removal of deprecation warnings and errors.
8. Try to come out of failure depression as soon as possible. It is a normal human nature that we feel low after a failure, when we work hard for an opportunity. Remember the quote said by Napoleon Hill:

“Every failure brings with it the seed of an equivalent success.”

9. Don’t afraid of getting feedback from your interviewer or mentor after getting rejected for an opportunity. Getting feedback will make to aware of your lacking and shortcoming. Feedbacks enable to you improve upon yourself and come back stronger than before.

What I missed in this 1 year😓

The major missing I feel in this 1 year of the journey was the knowledge of basic Data Structures and Algorithms. As I said, I left out from Competitive Programming due to loss of interest, I never tried to revive that interest after getting into development. Due to the result, I missed out on the knowledge of basic optimization techniques that can be proved useful in development also.

Application of the concepts, that are learned in competitive programming can be proven useful when we try to improve the performance of a project. So, it took me a very long time for learning the performance improvement of a project.

My say in Competitive Programming vs Development🧐

Competitive Programming is equally important just like Development. Development enables you to learn how things work in the real world in form of projects and platforms, similarly, Competitive Programming helps you to develop skills for finding efficient solutions to some complex problems.

If you are uninterested in competitive programming just because you think that CP questions would not be there in real-world projects, then lemme clear this misunderstanding, that CP questions help a lot in real-world project development also. The skills that you learn while doing CP questions(like, solving an O(n²) time problem in O(n) time) would help you a lot in improving the performance of your project.

My say would be, that both are equally important. So you should choose any one according to your interest but should also try to build up interest in other also. CP should be much considered as a habit rather than a task. Make a good habit of giving CP contests(like CodeForces weekly contests and CodeChef monthly contests) try to solve contest problems in given time constraints. Check editorials of questions after the contest to learn from your mistakes. All this will help you to build up your problem solving skills which would reflect in your project performance.

Continuing the Journey🎉

The journey is going on!!! Currently, I’m working on Project COMET, an ecosystem comprising of different platforms for connecting, collaborating and competiting with developers around the world. With the learnings I’ve acquired in this journey till now, I’m also learning Competitive Programming and Data Structures and Algorithms to strengthen my logic thinking ability and to learn optimization techniques. Also, I’m always eager to learn new things, hence seeking for learning new skills also.🎉🎉

1 year of Projects’ Development: A Developer's perspective👨‍💻 was originally published in IIITians Network on Medium, where people are continuing the conversation by highlighting and responding to this story.

“ Experience: Universal Mother of Sciences ”

Introduction

Hello Amigos!! Myself Tejas Agrawal (alias: cyber-venom003), an open-source enthusiast from the Indian Institute of Information Technology, Allahabad. I recently participated with 3 of my teammates in Hackout’20, a 36-hour national level online hackathon conducted by Headout. This was the first hackathon experience of my life which was really the best experience I’ve got till now in my professional life. So let’s get started.

Pre-Hackathon Planning

I got to know about hackathon through an e-mail from my college technical society, GeekHaven. I thought about my existing skills and decided that this would be the right time to start this. I went to the website of Hackout and joined the Discord server of the hackathon. Before teaming up with anyone, I ensured from organizers, on Discord, if the hackathon is an open track or it has some problem statements. I contacted one of my friends, Divyansh Rai, a pro-Kaggler and AI-ML enthusiast from my college itself for teaming up in the hackathon. We decided to make a team of 4 members, so he further contacted Aditya Raj, another AI-ML enthusiast, and Avneesh Kumar, a Flutter developer, and open source enthusiasts, both from our college.

At last, we got an efficient team of 4 members named “Cyber Symbiotes” and we started to think out of the box: about an idea that is something, useful for the masses. And then, there was a light bulb moment among us: Aditya Raj came up with an idea of a healthcare application to increase the efficiency of the medical architecture of India. We decided to build a cross-platform application that will take symptoms from users and will predict the probable diseases, and suggest to them the recommended tests. Moreover, users can upload the test results and the app would give them the risk and severity of the disease. Finally, we all decided to work upon this and started planning the workflow of the application.

Aditya Raj and Divyansh started their research on the Deep Learning Algorithms for disease detection using MRI Scans and Chest X-Rays, while I and Avneesh started our research on the implementation of AI in flutter applications.

6th November 2020: Day 0 of Hackathon

The hackathon started at 3:00 PM and by that time, Aditya and Divyansh managed to make the deep learning models of best accuracy to detect Brain Tumor, Pneumonia, and Covid-19. We all made a rough plan to commit and push timings of the project on Github so that we can work efficiently without much hassle and problems. The hackathon started with a talk given by Rachit Watts sir, VP Engineering in Headout. They explained about the work culture and life at Headout and his talk motivated me and my team to work efficiently with enthusiasm so that we can win and may get an internship opportunity at Headout.

After the speaker talks, we attended the Speaker AMA sessions to ask about their professional experiences which further motivated all of us to work upon. By the end of the day, I and Avneesh managed to make a dummy database of diseases mapped with their symptoms and recommended tests and hosted it on a cloud (Google Firebase). I implemented the feature of getting the probable diseases from the symptoms input given by the user till the end of day 0 itself. At night, before sleeping, we planned how to proceed with the project further in the next day on Discord Voice Channel.

7th November 2020: Day 1 of Hackathon

On day 1, we all started in the morning at 9 AM as we planned everything one night before. Avneesh started to work upon the feature of recommending tests on the basis of disease. On 7th Nov, there was my end semester exam of college which was a barrier to our hackathon progress, but I managed to dodge it(thanks to the online semester, xD).

By the afternoon, Avneesh managed to implement the disease to tests feature and 65% work of the application was completed. After that, mentors of the hackathon started the mid reviews of projects and our mid review was taken by Aakash Goel sir, engineering manager at Headout. He was impressed by our work momentum and wished us the best for further. Now, the remaining work was to integrate the deep learning models with our flutter application and none of us knew anything about MLOps. So all of us started to research the best frameworks which could provide us ML Deployment. Divyansh tried it using Streamlit library of python, Aditya tried it using Django and I started to research over Flask, a micro web framework written in Python. I googled things, searched over StackOverflow to know how this ML Deployment stuff works. After 4 hours of much hassle, finally, at the end of the day at 12 AM, I managed to make a local Flask server and tested it using a sample web app.

8th November 2020: Final Day of Hackathon

My day 2 started in continuity with day 1. As I was exploring the MLOps stuff, I didn’t sleep and tried to deploy my local Flask server on Heroku, but after the much hassle of 3 hours, all my efforts of mine went in vain. Hence, I started to integrate the local Flask Server with my flutter app.

Date: 8th November, Time: 5 AM IST, the dawn of the morning. Integration of local flask server with flutter app was successful, all testing checks were successful but, I ended up with eye fatigue and the app “Swasthyam” was completed. I notified all team members and they saw a celebration moment in the morning itself.

After that, I slept at 6 AM for 4 hours to relax, as I was having another exam at 11 AM. In the morning, mentors scheduled the final reviews at 11 AM, but due to my exams, I and my team requested mentors to take the final review of our project after my exams.

Finally, after my exam, our final review was taken by Ekansh Bansal sir, an intern at Headout. Firstly, they asked about our idea, which was explained by Aditya very well. He defended every cross-question of Ekansh sir in a very nice way. After that, Divyansh explained the ML and DL implementation in the project and I explained the MLOps and flutter application. Finally, after 30 mins, Ekansh sir wishes us all the best and we turned our eyes towards the #announcements channel in the discord server and waited for results.

All of my team members had butterflies in their stomachs and all of us were holding the breath to see the top 9 teams shortlisting. All of our fingers were crossed until…..

Cyber Symbiotes appeared nowhere in the top 9 shortlists. All of my team members were struck numb that why we weren’t shortlisted. After the shortlisting, we headed over to the YouTube live stream to see the presentations of the projects of the top 9 teams.

After getting disqualified, we just motivated each other not to get depressed and thought over why we weren’t shortlisted. We were just happy that we got a wonderful experience in our professional life and decided that we’ll always team up for any hackathon. Finally, there was an end to the exciting journey of 3 days which gave me a wonderful working experience under timed constraints and a fair knowledge of MLOps.

Lessons I learned during Hackout 2020

Hackout 2020 results taught me many lessons from the mistakes we committed as a team during the hackathon time period. I’m listing them here so that readers might learn from my experience to get an even better experience than mine:

1. Don’t ever think that you are just making a working prototype of your idea. Just keep working on the project and try to make it better and better until the last hour of your time range.
2. Try your best to make your working project live and open to all, i.e. deploy it on some cloud service. This would give a nice impression to anyone to whom you’re presenting the app.
3. Commit and push to the version control system at each and every milestone of the project. Believe me, it is very necessary to track your progress and it is a very nice preventive measure if you fuck up something in between development.
4. UI and UX of your project are as equally important as other aspects of your project because they are the very first things anyone sees before the core implementations of your project and as they say “First Impression is the last impression”. Hence your team should always contain a great designer.
5. Testing is very necessary at each and every milestone of your project development. While testing, you should think like a layman that what a user can do and what fuck ups can occur while the real-world usage of an app by a layman user. Before completion, you should try to clear all of the loopholes in your development phase only.

Links to Project

• cyber-venom003/Swasthyam-Hackout
• Swasthyam

Drop a star on the Github repository if you like it, any improvement suggestions are welcomed as Github issues.

Journey from a Rookie to Hacker: My first Hackathon experience at Hackout’20 was originally published in GDSC, IIIT Allahabad on Medium, where people are continuing the conversation by highlighting and responding to this story.