Lim Yow ChenginTowards Data ScienceFirsthand Experience with Polynote — A Better Notebook for Data EngineersNetflix recently announced Polynote, which looks promising for Data Engineers working on Spark and Scala. There are many great features…Oct 27, 20191Oct 27, 20191
Lim Yow ChenginThe StartupThree Things I Learnt From Using Scala Option Type in ETL Code BaseFor the past year, I have the opportunity to migrate a Java ETL code base to Scala. A typical flow for an ETL program takes data in its…Oct 20, 2019Oct 20, 2019
Lim Yow ChengMonitoring Spark Structured Streaming/Kafka Offsets with Prometheus and GrafanaVisualizing Kafka offset value can be useful to monitor the performance of spark structured streaming appApr 15, 2019Apr 15, 2019