Coding on the Remote Server via pyCharm

Photo by Johan Mouchet on Unsplash

As data enthusiasts, we often need to run codes in remote servers because we need computation power and sufficient memory. As I came across, the most popular tool people use to code in remote servers is the Jupyter lab and notebook. Although Jupyter is quite satisfying, pyCharm professional comes with its advantages too.
Here are some aspects of pyCharm Professional that make me write this article.

• Needless to say, I love JetBrains Product and I’m quite familiar to them for years.
• Rather than being just a notebook, PyCharm provides a complete IDE with the tools that make coding easier and more comfortable.
• If you are developing an actual project rather than doing just some analysis, you’ll have to test your code right away and for that a debugging tool is crucial. Although Jupyter accepts debugging mechanisms, such as PDB, debugging in PyCharm feels more solid.
• All of the utilities that I just mentioned also presented in pyCharm Community, but what makes pyCharm Professional special is that it has support for Jupyter notebook.
• Remote connection via Jupyter leaves no physical clue in your local machine, that means, in case of server miss-function you may lose your data and code if you hadn’t back them up. Nevertheless, pyCharm Professional works synchronizing the remote directory with a directory in your local machine. So you do not have to fear of server ill-functioning.

Photo by Casey Horner on Unsplash

If I have your attention, let’s move forward.

Recipe to set up PyCharm Connection

• Open PyCharm File > Settings > Build, Execution, Deployment
• On the right tab, choose Deployment
• Click the + appeared in the most left.
• We’ll now define a new connection, an SFTP connection should do the trick, so choose SFTP
• Enter the Host info which is the IP address to the remote machine like xx.xx.xx.xx
• Fill the port as 22
• Enter the username, it’s the ssh username@xx.xx.xx.xx (remember the command when you try to connect via terminal)
• Choose authentication as the password if you use the password to access the server. Enter your password in the password section.
• If you use PEM to access, choose the appropriate option and map your PEM path.
• Now test the connection, if success, you’re good to go.
• Now we will specify the root path, here are some tips to root mapping:

> Be careful at this stage, because all the mappings that we build will be built upon root. And this is crucial because pyCharm is all about synchronizing the right folders.

> If your interpreter resides in a folder outside of the project(which is the right way), you may want to map the folder, which is the parent to your project and interpreter. Down below there is a sample root with the actual project structure.

• So map the root path that suits you.
• Fill the web server URL as http://xx.xx.xx

Setting up the mappings

Do not close the recent window, we are continuing with it.

• Open the next tab mappings
• Fill the deployment path. Deployment path is the project’s root folder in remote and its a relative path to your project root above.
• Local path is absolute, it is the directory in your local machine and this is where you will synchronize the remote path with.
• We will add one more mapping, it is the interpreter. We will map our local interpreters’ path to the remote. My personal advice is to use virtualenvs every time. For further information on virtuelanvs you can read my blogpost. So only thing you have to do is to map local path (C:\\Users\imauser\someplace\envs|env-of-the-project) to the remote interpreter(\envs\env-of-the-project-on-remote). Recall that the local path is absolute and remote path is relative.

Mapping interpreters will make package management far easier,
so you wouldn’t have to connect the server via ssh and install packages in there, you can simply use pyCharm’s package management interface in order to do so.

Your life after pyCharm

• So, now you can comfortably use pyCharm on remote as if you’re coding in your local machine.
• Every time you make changes in a file and save (Ctrl+s) pyCharm automatically uploads your local changes to the remote.
• If you think automatic upload is ill-functioning, you can use Deployment utilities in Tools> Deployment section.
• If you like to see what is inside of your project on remote, you may use Tools> Deployment > Browse Remote Host, just comfy as a file explorer.
• Also, you can download/upload files by just dragging them between project explorer on the left and browse remote host toolbox on the right.

It is easy as it is.

How to run Jupyter notebooks via pyCharm on remote

This headline for me was a dream come true. Notebooks are great if you are covering a one-time analysis, and you’re in a rush to deal with data. They’re also great for prototyping your project. For all aforementioned stuff, Jupyter accepts your messy code and does not criticize it for all logical mistakes, typos, etc. But when it comes to implementing a project from scratch based on your prototype, things can go ugly and it turns out that the way pyCharm rebukes you was required after all. Neither Jupyter lab nor notebook provides you the guidance pyCharm is able to present and also editing .py files with them is really awkward. And copy+pasting from a web browser to pyCharm gives discomfort along with the displeasing nature of Jupyter that it does not let you jump between dependencies and it prevents you from constructing the project efficiently. Although I feel like I’m ambitiously trying to convince you of using pyCharm, I believe those who are suffered from the same issues will sympathize.

So, let’s get our hands dirty.

Recipe to run Jupyter notebooks via pyCharm

The first few steps will be exactly the same as when you try to use Jupyter for remote connections.

• Connect to the remote server, you may use ssh if you’re given username, password and IP address.
• $ ssh username@xx.xx.xx.xx
• Activate your virtual environment. For python-virtualenv you may use:
• $ source ~/envs/env-myenv/bin/activate
• For further information about environments refer to my blogpost.
• Cd into your projcts’ working directory.
• $ cd ~/repos/my_project
• Enable jupyter connection
• $ jupyter lab --no-browser --port=8890

You may use any valid port. I just like 8890.

• After connection established copy the connection-token and URL. It will be similar to the example right below:
  http://127.0.0.1:8890/?token=lalalalalalallalalalalallalalalalalallalalalalal
• Open an extra terminal window and from your local machine just type
• $ ssh -N -f -L localhost:8888:localhost:8890 username@xx.xx.xx.xx
• Open pyCharm and create an .ipynb file or just open an existing one.
• From the top tab of the ipynb file open Configure Jupyter Server.
• We will now create a configured server because remote servers are configured servers where the local ones are the managed ones.

In order to use the local interpreter in pyCharm-Jupyter tool,
you may use managed servers and chose your local interpreter.
But note that using local interpreters doesn’t provide the speed or
whatever the reason you are using the remote machine
in the first place.

• Fill the configured server with the token path you just copied. But this time instead of 127.0.0.1 write down the IP of the remote machine. Just as:
  http://xx.xx.xx.xx:8890/?token=lalalalalalallalalalalallalalalalalallalalalalal
• Click apply and quit.
• Now click to the blob icon to start the server.

Most of the short-cuts in pyCharm-Jupyter are the same as in the Jupyter notebook.
For instance, in order to run a cell you may use (shift+enter).

For further information about short-cuts just play with the toolbox or go to the pyCharm official doc.