Our data orchestration platform is using several lambda functions to manage, audit, and monitor data processes.
Our Python lambda functions share some logic such as logging, SQS utilities, slack communication code, and more.
To reuse Python modules we used to copy a bunch of .py files containing our shared modules into multiple lambdas. This, obviously is not a good practice as we had to build, pack, and deploy all lambdas upon changes in shared modules.
Instead of duplicating code, we decided to share code between lambdas using Lambda Layers.
Docker is a great choice for development runtime hosting. It makes it easier to keep your development components such as Spark, Python, Scala and could offer data science libraries out-of-the-box.
Fortunately, Jupyter Project offer various docker images in their Github account. In this short guide I will walk you through the process of running your local Jupyter/JupyterLab.
Note: the steps describe the process on Mac machines. Windows/Linux users will have slightly different process.
2. Download docker from docker store and install it on…
Fortunately, Jupyter Project offers various docker images in their Github repo. In this short guide I will walk you through the process of running your local Jupyter/JupyterLab.
Note: The following steps will describe the process on Mac machines. Windows/Linux users will have slightly different process.
Create local Jupyter or JupyterLab dev environment using Docker