https://www.pexels.com/photo/rows-of-wooden-panels-of-lumber-construction-4178808/

Our data orchestration platform is using several lambda functions to manage, audit, and monitor data processes.
Our Python lambda functions share some logic such as logging, SQS utilities, slack communication code, and more.

To reuse Python modules we used to copy a bunch of .py files containing our shared modules into multiple lambdas. This, obviously is not a good practice as we had to build, pack, and deploy all lambdas upon changes in shared modules.

Lambda Layers

Instead of duplicating code, we decided to share code between lambdas using Lambda Layers.


Docker is a great choice for development runtime hosting. It makes it easier to keep your development components such as Spark, Python, Scala and could offer data science libraries out-of-the-box.

Fortunately, Jupyter Project offer various docker images in their Github account. In this short guide I will walk you through the process of running your local Jupyter/JupyterLab.

Note: the steps describe the process on Mac machines. Windows/Linux users will have slightly different process.

Steps:

  1. First we will need to create a docker hub account (free). Just load https://hub.docker.com/ and register.
Docker Hub registration

2. Download docker from docker store and install it on…


Overview

Docker is a great choice for development runtime hosting. It makes it easier to keep your development components such as Spark, Python, Scala and could offer data science libraries out-of-the-box.

Fortunately, Jupyter Project offers various docker images in their Github repo. In this short guide I will walk you through the process of running your local Jupyter/JupyterLab.

Note: The following steps will describe the process on Mac machines. Windows/Linux users will have slightly different process.

Goal

Create local Jupyter or JupyterLab dev environment using Docker

Steps:

  1. First we need to create a docker hub account (free). Just load https://hub.docker.com/ and register.

Tomer Levi

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store