Why you should write your own ETL
In this story i’m going to tell you why and when you need to write your own ETL flow to feed your data warehouse.
What’s an ETL?
ETL is an acronym it stands for:
Extract the information from sources
Transform those information in any data type
Load into your data warehouse
You can change the order of those operations into:
Extract— Load — Transform
This depends on your needs and data base type.
How can i write an ETL or ELT flow?
Well a feeding flow could be written with two main tools:
1 — Drag and drop BI tools:
Those tools simplify a lot the enormous amount of code you need to write.
With these tools you need only to drag and drop the feature you need and configure it. Often these tools are based on Java and each feature is written with the aid of XML.
Pros: Every operation is easy and powerful
Cons: You need to wait the producer to release the feature you need for your ETL
When to use it?
It’s better to use those tools when you are not a developer and need to develop a fast paced BI project.
2 — Writing your own code:
By writing your own code it’s an another world, better or worse depending on your needs and project pace.
Personally i prefer this way, because you can add as many features you need at any pace you want.
But on the other side you need to be highly skilled on writing code and knowing at least one programming language between Python, Javascript or Java; you can write an ETL with C but it starts to be very difficult to maintain and to debug.
Pros: You write every single feature line by line
Cons: Huge number of code lines and huge project
Well this ends the thought about how and when to write your own code, let me know what you think about this topic.
