17 days running Google Colab

For my degree dissertation at the university, I need to run several experiments with different models of neural networks and download compressed files with the results and models learned, for this I am using an environment known as Google Collaboratory or Google Colab. In total, I need to download more than 23000 files.

In the last 17 days, I have successfully executed 19923 experiments and I have encountered certain problems that have delayed me, in this post I will talk about those problems so that they do not go unnoticed in the future.

1. Definitions

Google Colaboratory: is a free Jupyter Notebook environment that requires no configuration and runs entirely in the cloud. On the other hand, Jupyter Notebook is an interactive environment that allows to develop Python code dynamically and runs as a client-server application.

Notebook: files created in Colab or Jupyter.

Cell: the notebooks are divided into small blocks called cells, in them you can execute Python code or render text using Markdown.

2. Virtual Machines (Sessions)

Colab uses virtual machines to execute the notebook code on the server, these machines have a maximum life span of 12 hours. Therefore, anything that takes more than 12 hours to run will be incomplete.

Each Google account can have a maximum of 5 open sessions simultaneously, in this context, a session is a virtual machine. This limit can be reduced over time, for example: if you chain several runs of 5 sessions and 12 hours each, you will end up with a limit of only one open session. This limit lasts approximately 12 hours.

For this reason, if you need to have 5 active sessions at all times, it’s best to have a second Google account to fall back on when the limit appears in the first one.

3. Internet connection

For a Colab notebook to run, you must be connected to the internet. If you have bad WiFi coverage, it is possible that the execution will stop because the server interprets that you have disconnected, so it is better to connect the computer to the router using a wire and turn off the computer’s WiFi card.

4. Display Charts

Displaying graphs in Colab is useful, but they will only be shown when the execution of the corresponding cell is finished. If you use a cell that takes 12 hours to execute, like me, you will not see the graphs until that time passes, or worse, it is possible that so many graphs are accumulated, that the execution of the virtual machine stops because you have consumed all the available RAM memory. Also, when you have more than 20 figures opened, there is a warning quite annoying at the output of the execution.

To avoid all this, the best thing is to save the graphics as an image and then close them, this way:

5. Preventing Data Loss: Connect to Google Drive

Once the virtual machine is disconnected, which can happen because of all the above mentioned, the local files that were in it are lost, so it is a good idea to connect the notebook with Google Drive and save all the generated files in a folder. This also makes downloading files easier, since there is no option in Colab to download several files at once, they have to be downloaded one at a time.

To connect with Drive, just execute the following code:

This will generate a link that will open a dialog box, where we will select the Drive account we want to connect to and it will generate a password that must be entered in the notebook.

6. Uploading Files from Google Drive

Uploading several files to several sessions can be a bit annoying as it is a repetitive and time-consuming task, depending on the number of files and their size. Virtual machines use Linux, so to avoid this delay, you can connect the sessions with Google Drive, copy from there the necessary files and import the functions later.

As for the data files, they can be accessed directly from Drive without copying them, assuming that they will not be modified during the notebook execution.

For example, if you have 3 source files, you can save time this way:

The symbol “%” means that we are going to execute a Linux command instead of Python code.