ML Cloud Computing Part 2: Connecting PyCharm to Paperspace
Setting up a PyCharm project with a remote interpreter
As much as I love terminal I am a big believer in GUI tools. I would much rather minimize the amount of exposure to anything command line in my daily development workflow. And honestly, this is one of my biggest peeves about the state of ML workflows today with Terminal, SSH, and local hosts. I see many people using text editors and running their code using command line execution. Mobile developers are spoiled by Android Studio and XCode, ML devs should have the same luxury.
Lucky for us there is a lovely python IDE called PyCharm. It is basically XCode, but for all things python, which happens to be the most common language interface for all things ML :) An even bigger bonus is that PyCharm has the built-in ability to remotely connect to a server and run your code remotely. Console, Logs, debugger, and plots all work.
Step 0: Setup a Paperspace machine:
- If you have not already gone through Part 1 of setting up Paperspace, you can find it here.
Step 1: Download and install PyCharm Set:
- Head over to — https://www.jetbrains.com/pycharm/.
- Download the Professional version. This is necessary if you want to connect it to your Paperspace remote machine.
- There is a 30-day free trial and if you are a student they will give you the hookup. They also have startup discounts and if you are a student it is free.
I must stress this. It is not worth trying to save yourself a few hundred dollars and using a free product. If you are serious about development you should be serious about your tools.
Step 2: Install anaconda on your Paperspace machine:
- Head to the Paperspace console and make sure your machine is running.
- Open a terminal and using the ssh jujutsu from Part 1 type:
- To find the latest version of Anaconda follow this link.
- Copy the download link for the python 3.7 64 bit version of anaconda. For me, it is: https://repo.anaconda.com/archive/Anaconda3-2018.12-Linux-x86_64.sh.
- Type the following into the terminal to download the installer to your Paperspace box.
- Next, run the installer using the following command.
- Follow the prompts in the terminal and finish the installation.
Step 3: Create and set up your virtual environment
A quick note on why this is important. As you get into training ML models and using a variety of different frameworks you will find that they all have different requirements and dependencies. Creating virtual environments gives you an easy way to have many different versions of python and other frameworks just by switching environments. It is also handy for when you are playing around and you break something. Instead of breaking the global machine you just break the environment and it is easy to delete it and try again.
- Use the following command to make an environment named ml_env that has python 3.6 installed. Anything greater than python 3.6 and you are going to have trouble later with Turi Create as well as other ML frameworks.
conda create -n ml_env python=3.6 anaconda
- To activate/deactivate your environment use the following.
source activate ml_env
- Here is a quick conda cheat sheet.
Use to update conda:
conda update conda
Create a new environment with a specific python version:
conda create -n yourenvname python=3.6 anaconda
Activate an existing conda environment:
source activate yourenvname
Deactivate an existing conda environment:
To list all existing environments on the system:
conda info --envs
Install a specific package using conda:
conda install -n yourenvname [package]
Uninstall and delete everything in the environment:
conda remove -n yourenvname -all
If you accidentally installed or upgraded python to a version that breaks things you can roll back. Say you installed 3.7 and need to go back to 3.6:
conda install python=3.6 -- downgrade to python 3.6
Step 4: Hooking up PyCharm to your Paperspace remote:
- Open PyCharm and select “Create New Project”.
- Pick a folder where the local files will be saved. I am using ml_project.
- Now to set up the remote environment click the “…” to get started.
- On the left select SSH Interpreter.
- For Host: use the public IP for your Paperspace box.
- Username: you will want to use paperspace.
- Select Next.
Here we want to point to the python interpreter in the ml_env virtual environment on the Paperspace box.
- Navigate to /home/paperspace/anaconda3/envs/ml_env/bin/python.
- Select OK,then Finish.
Finally, we need to select where the PyCharm project files will live on the remote server side.
- Select the folder icon in the lower right.
- Navigate to where you want to have PyCharm sync files.
- You can right click to make a new folder. I am using /home/paperspace/PycharmProjects/ml_project.
- Click OK, then Create.
PyCharm will take a few minutes to upload its helpers to the remote.
When a remote Python interpreter is added, at first the PyCharm helpers are copied to the remote host. PyCharm helpers are needed to run remotely the packaging tasks, debugger, tests, and other PyCharm features.
Step 5: Running your first program on your Paperspace machine:
- Right click on the project folder ml_project.
- Hover over New.
- Select Python File.
- Name the file test and click OK.
One great feature in PyCharm is that it will automatically sync the files in your local project folder to the one on your remote.
- In the upper right-hand corner click the tab “Remote Host.”
- Navigate to home/paperspace/PycharmProjects/ml_project.
You will see your newly created test.py file is not on the server. Now you may be thinking why would we want to use CyberDuck when we get similar functionality with PyCharm. While the PyCharm system works great for code, it starts to breakdown when transferring large amounts of data. So when adding large amounts of training data, CyberDuck or command line is the way to go.
- In the test.py file type print (“hello world”).
- Right click on test.py on the left-hand side, and select Run ‘test’.
- In the console, you will see the output “hello world”.
Congratulations you now have an IDE that is backed by a very powerful GPU in the cloud! Next, we will properly take advantage of this setup and train some networks!
As ever, QuarkWorks is available to help with any software application project — web, mobile, and more! If you are interested in our services you can check out our website. We would love to answer any questions you have! Just reach out to us on our Twitter, Facebook, or LinkedIn.