Dominik SchauerOptimizing storage for your data lake in 6 waysData lakes are a cost-effective solution for storing large amounts of data in the cloud, for example on AWS S3 or Azure Data Lake Storage…8 min read·Mar 2, 2024----
Dominik SchauerAWS Glue job development in VS Code — unit testing with Docker and pytest on an EC2 development…This article describes how to setup a remote development environment to develop and unit test AWS Glue Pyspark jobs locally. You will use…11 min read·Apr 2, 2023----
Dominik SchauerDefeating Impostor Syndrome: Overcoming Self-Doubt in Your Professional LifeImpostor syndrome, also known as impostor phenomenon, is a feeling of inadequacy and self-doubt that affects many professionals in…2 min read·Mar 4, 2023----
Dominik SchauerProfessional AWS Glue PySpark Development — Mocking AWS services for unit testsIn this post we take a look at mocking AWS resources for unit tests in a local development environment.7 min read·Dec 25, 2022----
Dominik SchauerProfessional AWS Glue PySpark Development — Interactive SessionsThis is the second part of my series of developing AWS Glue jobs. Last time we took a look at local development in VS Code utilizing a…9 min read·Dec 20, 2022----
Dominik SchauerProfessional AWS Glue PySpark Development — Local Development and Unit TestsPart 1: Local development environment and unit testing10 min read·Dec 18, 2022----
Dominik SchauerHow to save money when developing AWS Glue Spark JobsOne way to reduce the cost of developing AWS Glue Spark jobs is to use the Glue Python Shell environment instead of a dedicated Glue…4 min read·Dec 12, 2022----