Originally posted to my personal blog available here. At QueryClick, we use Azure Functions as the runtime for a number of our data collection pipelines. Serverless technology has been a recent adoption here and it’s worked out well for us so far. …

Originally posted to my personal blog here. Databricks Notebooks are commonplace at QueryClick, they’re the combination of notebooks and PySpark for EDA and simple Spark jobs. Notebooks are controversial to say the least; they encourage bad coding standards and have a non-linear flow to their code. Notebooks Learn how to manage and use notebooks in Databricks.docs.databricks.com However, they do have…

Originally posted to my personal blog here. As part of the Microsoft Partner Hack in November 2020, I decided to use this opportunity to try out a new method of ingesting Fluentd logs. What is Fluentd? Fluentd is a log collector which takes a declarative config file containing input (or “source”) and output…