Recovering from a Jupyter Disaster

Published in

Flatiron Engineering

5 min readMar 29, 2019

So you’ve done some really great work in a Jupyter notebook. The model finally works, results are excellent, and you’re almost certainly going to get promoted because of this.
It is late, you’ve been up for 80 hours without sleep and you accidentally delete the notebook.
Surely it’s backed up somewhere?
You check the backups only to realize they haven’t been running for months and the alerting email goes to somebody that has left the company years ago.

Don’t panic. Things are bad but you might be able to recover.

Jupyter snapshots

Jupyter stores snapshots by default for all notebooks

They can generally be found in:

{notebook_directory}/.ipynb_checkpoints/

These notebooks are copies at a point in time of a notebook, when it was last saved. By default only one snapshot exists per notebook.

Copy the snapshot over to somewhere else, open it up and see if it has what you want.

You open the snapshot only to find its empty. Somehow in a caffeine frenzy you must have deleted all the code and saved over the notebook before deleting it??

Things look grim but there are still avenues open to you.

IPython kernel history

IPython by default stores all the commands you’ve issued to it in a sqlite database. This is used to drive things like CTRL-R searches in the terminal but can also be used to recover code that has been executed

These can be found in:

~/.ipython/profile_default

Inside this directory is a sqlite database:

`history.sqlite`

This contains every command you’ve ever sent to the IPython kernel and not actively removed from history. Odds are you’ve never removed anything from then history so your code could well be in there.

There are some caveats though.

No file names
Since this is the history of the kernel session there is no way to associate a particular piece of code with a file as there is no mentions of filenames in this history table. There is a session_id though which will have to suffice.
No cell output
Since this is a listing of history of command sent to the kernel, you do not have results of a command. But if you can recover the code, you should be able to rerun your notebook and get output.
No non-code cells
All of your beautifully written markdown prose is gone. You’re just going to have to redo that sadly.

Recovery process

Now is a very good time to make a backup of the `history.sqlite` database. You can generally do this just by copying the file somewhere else.

There are two tables of interest here `session` and `history`. There may be others, but they are likely empty, and uninteresting.

TABLE history
    (session integer, line integer, source text, source_raw text,
    PRIMARY KEY (session, line));TABLE sessions (
    session integer primary key autoincrement,
    start timestamp,
    end timestamp,
    num_cmds integer,
    remark text);

Using the sessions `start` timestamp we can filter the list of sessions to ones related to when we started our session. This corresponds to the time where you restarted the kernel for a notebook. The `end` timestamp might be set, but it is best to not rely on it. The remark field looks like it shows promise, but it is empty, nothing to see there.

To find the right session we can generally run some queries like this to narrow on on the particular session we want.

Start a sqlite session by doing the following in a bash shell.

sqlite3 history.sqlite

If the sqlite3 command line is not install you can get it from your system package manager or by simply running something like

# using conda
conda install sqlite3# using apt
sudo apt-get update && sudo apt-get install sqlite3# using yum
sudo yum install sqlite3# using brew
brew install sqlite3

Run a query similar to the following to see which sessions you last ran:

select 
    sessions.session, sessions.start, sessions.end, history.source
from sessions 
join history using (session)
where
    start > '2019–02–27' and start < '2019–03–01';

This will show you every cell you’ve run in that time interval. Scroll through the results and find the session you are interested in.

This means that we can run a query like:

select '# @@ Cell '|| line || char(10) || source || char(10) from history where session = 1634;

to get our output as a `.py` file that we can convert back to a notebook with the excellent jupytext

sqlite3 history_bak.sqlite \
    "select '# @@ Cell '|| line || char(10) || source || char(10)    from history where session = 1634;" \
> myoutput.pyjupytext — to ipynb myoutput.py

Yay, you have your notebook back!

Um, actually you didn’t delete the notebook. Somehow your entire home directory was deleted. What??

Recovering from total madness

You’re in trouble now. You’ve probably lost months of work. This is really bad.

There is still a small glimmer of hope.

If you have not closed any of the python kernels that you are using; one of them might still have a handle to the sqlite database file — even though that no longer exists.

The solution outlined here only works on linux, but can be done in other operating systems by using similar concepts. Relevant google search terms are “recover file open file descriptor ${OS_NAME}”.

Use a tool like ps or htop to find the pid of your process in question.

$ ps -u mvanniekerk -f | grep ipykernelmvannie+ 7408 5835 0 Feb19 ? 00:00:45 /home/mvanniekerk/miniconda3/envs/py36/bin/python -m ipykernel_launcher -f /home/mvanniekerk/.local/share/jupyter/runtime/kernel-3c5a9702-fbdd-4f2c-a2f2–0fe79872edf6.json
mvannie+ 7426 5835 0 Feb19 ? 00:00:45 /home/mvanniekerk/miniconda3/envs/py36/bin/python -m ipykernel_launcher -f /home/mvanniekerk/.local/share/jupyter/runtime/kernel-9f2329f7–8059–4645–972d-4e7589f88c27.json

The first integer here is the PID, in this case `7408`. Let’s go explore its filesystem.

$ cd /proc/7408/fd$ ls -lah | awk ‘{print $9, $10, $11}’ | grep sqlite35 -> /home/mvanniekerk/.ipython/profile_default/history.sqlite
38 -> /home/mvanniekerk/.ipython/profile_default/history.sqlite

These file descriptor are still open and thus this file still exists.

Just copy this file to somewhere and you can perform the recovery procedures outlined in the previous directory.

In this case:

cp /proc/7408/fd/35 /somesafepath/ipython_history.sqlite

Conclusion

None of this hard work would be needed if you made backups or used version control, but hopefully this can help as a disaster recovery guide in the future.