Dominik SchauerOptimizing storage for your data lake in 6 waysData lakes are a cost-effective solution for storing large amounts of data in the cloud, for example on AWS S3 or Azure Data Lake Storage…Mar 2Mar 2
Dominik SchauerAWS Glue job development in VS Code — unit testing with Docker and pytest on an EC2 development…This article describes how to setup a remote development environment to develop and unit test AWS Glue Pyspark jobs locally. You will use…Apr 2, 2023Apr 2, 2023
Dominik SchauerDefeating Impostor Syndrome: Overcoming Self-Doubt in Your Professional LifeImpostor syndrome, also known as impostor phenomenon, is a feeling of inadequacy and self-doubt that affects many professionals in…Mar 4, 2023Mar 4, 2023
Dominik SchauerProfessional AWS Glue PySpark Development — Mocking AWS services for unit testsIn this post we take a look at mocking AWS resources for unit tests in a local development environment.Dec 25, 2022Dec 25, 2022
Dominik SchauerProfessional AWS Glue PySpark Development — Interactive SessionsThis is the second part of my series of developing AWS Glue jobs. Last time we took a look at local development in VS Code utilizing a…Dec 20, 2022Dec 20, 2022
Dominik SchauerProfessional AWS Glue PySpark Development — Local Development and Unit TestsPart 1: Local development environment and unit testingDec 18, 2022Dec 18, 2022
Dominik SchauerHow to save money when developing AWS Glue Spark JobsOne way to reduce the cost of developing AWS Glue Spark jobs is to use the Glue Python Shell environment instead of a dedicated Glue…Dec 12, 2022Dec 12, 2022