Stream Your Data Using Nothing But Python’s Requests Library
Understand when to stream data, how to configure a pipeline and learn from my mistakes building streaming pipelines.
I need your help. Take a minute to answer a 3-question survey to tell me how I can help you outside this blog. All responses receive a free gift.
The Forgotten Streaming Library Hidden In Python
When you think of building streaming data pipelines you likely think of using a library like Apache Kafka that is tailor-made for streaming.
Cloud platforms like Google Cloud Platforms (GCP) and Amazon Web Services (AWS) support the streaming of data into their respective SQL environments. GCP, in particular, even provides services like the Billing API or Google Analytics connections that stream source data on a recurring basis.
On top of the infrastructure giants countless third-party “code-less” platforms have taken on the heavy lifting of setting up, maintaining and scaling streaming infrastructure.
Even though third parties sell platforms that will alleviate the “challenging” builds like streaming pipelines, as a data engineer, it is still important to know how to configure a streaming pipeline.
