PyFinance
Published in

PyFinance

Deploy Jupyter Notebook on AWS EC2 instance

To make Python environment setup in cloud instance easier, I have added a shell script that installs required packages and a conda virtual environment that contains frequently used python libraries in this repository.

But I recommend that you go through all the steps at least once before executing aforementioned shell script, just so you have an idea of what we are doing.

Advantages of using cloud instance

Photo by Sergey Zolkin on Unsplash

By using jupyter notebook on cloud, anyone can take benefits of professional grade hardware from their personal computer system. Rather than setting up a workstation or server and keeping it up to date with latest hardware as time passes, you can just pay for a virtual machine that suits your needs and get your work done easily.

Other than hardware to test our models, we also get storage for scripts and all the files containing data.

Some options for cloud computing are : Google Cloud Platform, Digital Ocean, and Amazon Web Service(AWS).

You are charged on the basis of hours. See AWS pricing here.

Tutorial

The Virtual machine I am using here has following specifications but the choices are completely arbitrary :

Amazon Machine Image (AMI) : Ubuntu Server 18.04 LTS

Instance Type : t2.micro

Physical Memory : 1GB

Storage : 8 GB (Amazon EBS)

CPU : Intel Xeon family

Step 1 : Make an AWS account from here

Step 2 : Sign in

Step 3 : Go to EC2 and Launch Instance

Step 4 : Select Ubuntu

Step 5 : Select t2.micro as instance type

Step 6 : Go to “Configure Security Groups” tab

Give group name and group description that you like .

Step 7 : Add a rule and enter the following values

Type: Custom TCP

Protocol : TCP [Set by Default]

Port Range : 8888

Source : Anywhere

Step 8 : Review & Launch

Step 9 : AWS will prompt you to create a new key pair or select a new one

NOTE : If you select an existing key pair, you need to have that “.pem” file on your local machine.

For this tutorial we will make a new key pair.

Download the key pair you’ve just made.

Step 10 : Click on Launch Instances.

Scroll down and click on View Instances.

Your instance will have status : running.

Step 11 : Copy the downloaded key pair to any directory you want and change access permissions for it.

I am making a new directory “notebooks” and copying key pair into it. I will also change its permissions to 400(read permission to root user only).

Use these commands in your bash shell or you can do it using GUI.

# Making a new directory "notebooks" and switching to it
mkdir notebooks && cd notebooks
# Copying key pair into "notebooks" directory
cp ~/Downloads/for-tutorial.pem ./
# Changing permissions for key pair
chmod 400 for-tutorial.pem

Step 12 : Right click on your running instance and click on Connect.

The example will have a command like

ssh -i "for-tutorial.pem" ubuntu@ec2-XX-XXX-XX-XX.XX-XXXX-X.compute.amazonaws.com

Run that command on shell in your local machine.

Answer yes to any prompt by shell and you’ll be connected to terminal of your cloud instance.

You could notice that your terminal now identifies machine as

@ubuntu@ip-xxx-xx-xx-xx:~$ 

Step 13 : Now you have to do following at cloud instance :

1. Install bzip2, gcc, git, htop, screen, vim, and wget.

2. Download and install anaconda (or miniconda).

3. Add conda to PATH variable.

4. Install necessary python libraries from conda (in this tutorial : jupyter notebook and ipython)

All of these four steps are covered in following command

or you can download the bash script that does it all from here.

#Step 1
apt-get update
apt-get upgrade -y
apt-get install -y bzip2 gcc git htop screen vim wget
apt-get upgrade -y bash
apt-get clean
#Step 2
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O Miniconda.sh
bash Miniconda.sh -b
rm -rf Miniconda.sh
#Step 3
echo "export PATH=~/miniconda3/bin:$PATH" >> ~/.bashrc
source ~/.bashrc
#Step 4
conda install -c anaconda notebook --yes
conda install ipython --yes

Step 14 : Generate an OpenSSL certificate for Jupyter Notebook

Run this command on the cloud instance terminal.

openssl req -x509 -nodes -days 365 -newkey rsa:1024 -out mykey.pem -keyout mykey.pem

The name of file after -out and -keyout flag should be same.

Shell will prompt you to enter some details about yourself, you can fill them or just leave them blank using “ . ” as input.

A file named “mykey.pem” will be generated.

Step 15 : Generate a password hash for jupyter notebook

Run the following command in cloud instance terminal

ipython

This will execute an interactive session for ipython in the terminal, now import passwd() method from notebook.auth and generate hash for any string, here I used “mypassword” but you can use something more secure.

In [1]: from notebook.auth import passwdIn [2]: passwd("mypassword")                                                                                                           
Out[2]: 'sha1:553c099eea73:2e8fda20a9878dd5abc84965d4a81e09eb48995f'
In [3]: exit

The string you generate hash for will be the password of jupyter notebook.

Do not clear the terminal window, you will need this hash in following steps.

Step 16 : Generate a Jupyter Notebook Configuration File (jupyter_notebook_config.py)

Run this in the terminal of cloud instance

jupyter notebook --generate-config

Step 17 : Open config file (~/.jupyter/jupyter_notebook_config.py)

and using text editor (like vim or nano), add the following lines

c.get_config()
c.NotebookApp.certfile=u'home/ubuntu/mykey.pem'
c.NotebookApp.ip='0.0.0.0'
c.NotebookApp.port=8888
c.NotebookApp.password=u'sha1:553c099eea73:2e8fda20a9878dd5abc84965d4a81e09eb48995f'

Here, c.NotebookApp.certfile takes the location of certificate generated by OpenSSL (mykey.pem)

and c.NotebookApp.password takes the hash generated from Step 15 as input.

Step 18 : Run jupyter notebook on cloud instance

jupyter notebook --no-browser --port=8888

Step 19 : Open a new terminal tab or window on your local machine (keeping the existing jupyter notebook terminal open)

connect to cloud instance terminal but with a new flag -L

ssh -i "for-tutorial.pem" -L 8890:localhost:8888 ubuntu@ec2-XX-XXX-XX-XX.XX-XXXX-X.compute.amazonaws.com

This command will forward the output from 8888 port of cloud instance to port 8890 of your local machine.

Step 20 : Open localhost:8890 in your browser on you local machine and enter the password (string from step 15).

NOTE : If your browser warns you about an insecure connection, you can just choose to “continue anyway”.

If password is said to be incorrect, then copy the token from terminal that is running jupyter notebook and paste it in password field.

If something goes wrong, relaunch jupyter notebook on a different port and use a different port on your local machine as well.

--

--

--

Sharing articles about the applications of Python programming language in finance.

Recommended from Medium

More with less; Client Side Rollups for Video Quality Metrics

Journey with Dlithe BootCamp Java_Fullstack_Development | week3

Build a Calculator with PHP OOP and Procedural Error Handling (Simple Method)

Track Paper Trading Results with Trade Performance Window

Your First Python Tutorial

Computer Network: Application Layer

How to reuse complex xib-designed views in storyboards using modern swift generics, property…

Keep it Secret…! Keep it Safe…! with Docker Secrets in WSO2 Micro Integrator

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Avnish

Avnish

Associate SQE @ Red Hat | bovem.netlify.app | linkedin.com/in/avnish-pal/

More from Medium

Signed URLs and Signed Cookies in AWS CloudFront using Python

A Python Script to Fetch and Process AWS Security Hub Findings Using the AWS CLI

AWS Lambda Function using Python packaging with serverless framework

Integrating AWS APIs in Python — Using AWS S3 with the boto3 SDK