So you want to use Jupyter notebook on a cluster?
I wanted to use Jupyter, but some of the analyses I want to do will take up a lot of file space. So, I want to run Jupyter remotely on a cluster, but I can’t install things on the cluster. Here’s how I got things to work.
(1) You only need to do these ‘workarounds’ if you can’t install stuff on your cluster. Even then, if Singularity is installed, use that instead!!
(2) Anaconda needs to be installed on the cluster.
(3) This is set to just run Jupyter on the head node of the cluster, will need some adjusting to run on a compute node.
Now that we have that out of the way…
(1) Load anaconda, if needed. For me, I needed to do ‘module load anaconda’. If I didn’t load anaconda, python was accessible, but not pip.
(2) Use pip to install jupyter, using the ‘user’ flag: pip install jupyter --user. This is necessary since we can’t install it using sudo — we just don’t have the requisite permissions. (Otherwise all this would be easy!)
(3) Unfortunately, the usual way of starting Jupyter (jupyter notebook) does not work. This is related to installing in the user path.
(4) Jupyter can be started using: python -m notebook. You should now be able to access the Jupyter web interface at http://localhost:8888 (as usual).
(5) There are some limitations to starting Jupyter this way, relative to being able to type jupyter notebook, namely, if you want to pass additional input arguments in when Jupyter starts.
If this is something you want to fix, add ~/.local/bin to your PATH (using setenv).
(6) For bonus points, you can make an ssh tunnel so you can access Jupyter from your local machine, without using a remote display protocol (like VNC or NoMachine).
For Mac/Linux, you can just do this with some extra options on the usual ssh command: ssh email@example.com -L8888:localhost:8888.
For Windows, you can set an ssh tunnel in PuTTY, similar to as described here: https://crl.ucsd.edu/handbook/vnc/ (but use ports 8888 instead of 5900/5901). There are some more elaborate ways to do this remote Jupyter set-up, but they seemed more complicated (see http://laserplasma.ninja/