Published in Uncanny Recursions·Oct 4, 2020Introduction to KubernetesExecutor and KubernetesPodOperatorIn the previous tutorial, we didn’t delve into the concept of Executors in Airflow. In Airflow, Executors define a mechanism by which task instances get run. Repo link: https://github.com/mrafayaleem/etl-series When you spun up an Airflow cluster using helm in the previous tutorial, you might have noticed a scheduler and a…Airflow5 min readAirflow5 min read
Published in Uncanny Recursions·Jun 10, 2020Setting up Airflow on a local Kubernetes cluster using HelmIn this post, I will cover steps to setup production-like Airflow scheduler, worker and a webserver on a local Kubernetes cluster. Later on, I will use the same K8s cluster to schedule ETL tasks using Airflow. Quickstart a Kubernetes single-node cluster If you are on Mac, it’s just a matter of a ticking a checkbox…Airflow3 min readAirflow3 min read
Published in Uncanny Recursions·Jun 10, 2020Building an end-to-end data pipeline from scratchThis series is meant to cover a broad range of topics that involve setting up a production grade ETL pipeline. I have broken it down into chapters and more of them will be added as we move along this series. I am a Mac user so everything that is written is in context of software and tools installed on OS X. You should be able to find and install equivalent versions of those for your own setup. Feel free to leave any feedback on my Twitter handle or through my website.Data Pipeline1 min readData Pipeline1 min read
Published in Uncanny Recursions·Aug 5, 2019An introduction to Random Forest using Tesla stock pricesIn this post, we will aim at having a deeper understanding of Random Forest by taking OHLC (open-high-low-close) data for Tesla stock for 5 year period. We will keep unrelated discussion at a minimum and try to focus on everything that is related to our understanding of Random Forest for…Machine Learning9 min readMachine Learning9 min read
Published in HackerNoon.com·Mar 13, 2017A/B Testing Our Way to a Better Payment GatewayImagine you have to rewrite an existing web service to move to a new payment gateway (PSP or payment service provider) due to various business use cases. Your first thought might be to replace old with the new one in its entirety and roll that out. That is a naive…Payments9 min readPayments9 min read
Published in OLX Group Engineering·Mar 13, 2017How we changed our payment service provider over an A/B test via HAProxyImagine you have to rewrite an existing web service to move to a new payment gateway (PSP or payment service provider) due to various business use cases. Your first thought might be to replace old with the new one in its entirety and roll that out. That is a naive…Payments9 min readPayments9 min read
Published in OLX Group Engineering·Mar 6, 2017Live MySQL Schema Changes on Amazon RDS with Percona — A WalkthroughAt dubizzle OLX, we recently reached a roadblock on a project where we had to change schema on around 130 tables in our production database with master/slave replication. The simplest possible solution that could have been was to recreate same tables with new schemas, copy the data and update our…My SQL9 min readMy SQL9 min read
Published in Uncanny Recursions·Mar 1, 2017How We Changed Our Payment Service Provider Over an A/B Test via HAProxyImagine you have to rewrite an existing web service to move to a new payment gateway (PSP or payment service provider) due to various business use cases. Your first thought might be to replace old with the new one in its entirety and roll that out. That is a naive…Ecommerce9 min readEcommerce9 min read
Published in Uncanny Recursions·Mar 31, 2016Apache Kafka Producer Benchmarks — Java vs. Jython vs. Python31 Mar 2016 v1.0 Note: This post is open to suggestions that can help achieve fairer results with these benchmarks. It is a versioned post and I would be incrementing it if anything related to these benchmarks changes. Changes can be tracked here as well. In my previous post, I…Big Data4 min readBig Data4 min read
Published in Uncanny Recursions·Mar 19, 2016Interfacing Jython with Kafka 0.8.x19 Mar 2016 Note: A complementing repository for this post can be found at: https://github.com/mrafayaleem/kafka-jython With the release of Kafka 0.9.0, the consumer API was redesigned to remove the dependency between consumer and Zookeeper. Prior to the 0.9.0 release, Kafka consumer was dependent on Zookeeper for storing its offsets and…Apache Kafka7 min readApache Kafka7 min read
