Published inMiro EngineeringWriting data product pipelines with AirflowA framework to make your data organization transition from simple DAGs to trustworthy data products.Jan 26, 20235Jan 26, 20235
Published inMiro Engineering5 Strategies for data workflows scheduling at MiroSome strategies we follow to make our scheduling capability escalableMar 14, 2022Mar 14, 2022
Published inMiro EngineeringAgile Data Engineering at MiroThis is how we do it: at Miro we leverage Agile concepts to drive data engineering projects. We get improved predictability, and a smoother…Nov 22, 2021Nov 22, 2021
Published inTDS ArchiveIntroducing a new pySpark’s library: owl-data-sanitizerA library to democratize data quality within companies with pySpark data pipelines.May 5, 2020May 5, 2020
Published ininganalytics.com/inganalyticsHow to review ETL pySpark pipelinesA guide about how to perform pySpark code reviews to comply with python standards, guarantee data quality and keep your code extensible.Mar 23, 2020Mar 23, 2020
Published inTDS ArchiveWrite Clean and SOLID Scala Spark Jobsnowadays extensive pipelines are written as simple SQL queries, neglecting important development concepts as writing clean and testeable…Dec 30, 20196Dec 30, 20196
Published inTDS ArchiveUnderstanding the Spark insertInto functionProblems found while using the spark insertInto with HiveOct 22, 20192Oct 22, 20192
Published inThe StartupIngesting Raw Data with Kafka Connect and Spark DatasetsIn this blog post I will explain how we use kafka-connect and spark orchestrated by platforms like kubernetes and airflow to create a Raw…Oct 15, 2019Oct 15, 2019
Dockerizing a Hand Written Digits Predictor ServiceDockerizing a Flask service that uses keras to classify Hand Written Digits.Mar 21, 2019Mar 21, 2019
Published inTDS ArchiveDataset deduplication using spark’s MLlibSpark Machine Learning: 2 approaches to deduplicate Dataframes using sparkMLib and Scala.Mar 17, 20191Mar 17, 20191
