Recovering from a Jupyter Disaster

Marius van Niekerk
Mar 29, 2019 · 5 min read
Image for post
Image for post

So you’ve done some really great work in a Jupyter notebook. The model finally works, results are excellent, and you’re almost certainly going to get promoted because of this.

It is late, you’ve been up for 80 hours without sleep and you accidentally delete the notebook.

Surely it’s backed up somewhere?

You check the backups only to realize they haven’t been running for months and the alerting email goes to somebody that has left the company years ago.

Don’t panic. Things are bad but you might be able to recover.

Jupyter snapshots

Jupyter stores snapshots by default for all notebooks

They can generally be found in:

These notebooks are copies at a point in time of a notebook, when it was last saved. By default only one snapshot exists per notebook.

Copy the snapshot over to somewhere else, open it up and see if it has what you want.

You open the snapshot only to find its empty. Somehow in a caffeine frenzy you must have deleted all the code and saved over the notebook before deleting it??

Things look grim but there are still avenues open to you.

IPython kernel history

IPython by default stores all the commands you’ve issued to it in a sqlite database. This is used to drive things like CTRL-R searches in the terminal but can also be used to recover code that has been executed

These can be found in:

Inside this directory is a sqlite database:

`history.sqlite`

This contains every command you’ve ever sent to the IPython kernel and not actively removed from history. Odds are you’ve never removed anything from then history so your code could well be in there.

There are some caveats though.

  • No file names
    Since this is the history of the kernel session there is no way to associate a particular piece of code with a file as there is no mentions of filenames in this history table. There is a session_id though which will have to suffice.
  • No cell output
    Since this is a listing of history of command sent to the kernel, you do not have results of a command. But if you can recover the code, you should be able to rerun your notebook and get output.
  • No non-code cells
    All of your beautifully written markdown prose is gone. You’re just going to have to redo that sadly.

Recovery process

Now is a very good time to make a backup of the `history.sqlite` database. You can generally do this just by copying the file somewhere else.

There are two tables of interest here `session` and `history`. There may be others, but they are likely empty, and uninteresting.

Using the sessions `start` timestamp we can filter the list of sessions to ones related to when we started our session. This corresponds to the time where you restarted the kernel for a notebook. The `end` timestamp might be set, but it is best to not rely on it. The remark field looks like it shows promise, but it is empty, nothing to see there.

To find the right session we can generally run some queries like this to narrow on on the particular session we want.

Start a sqlite session by doing the following in a bash shell.

If the sqlite3 command line is not install you can get it from your system package manager or by simply running something like

Run a query similar to the following to see which sessions you last ran:

This will show you every cell you’ve run in that time interval. Scroll through the results and find the session you are interested in.

This means that we can run a query like:

to get our output as a `.py` file that we can convert back to a notebook with the excellent jupytext

Yay, you have your notebook back!

Um, actually you didn’t delete the notebook. Somehow your entire home directory was deleted. What??

Recovering from total madness

You’re in trouble now. You’ve probably lost months of work. This is really bad.

There is still a small glimmer of hope.

If you have not closed any of the python kernels that you are using; one of them might still have a handle to the sqlite database file — even though that no longer exists.

The solution outlined here only works on linux, but can be done in other operating systems by using similar concepts. Relevant google search terms are “recover file open file descriptor ${OS_NAME}”.

Use a tool like ps or htop to find the pid of your process in question.

The first integer here is the PID, in this case `7408`. Let’s go explore its filesystem.

These file descriptor are still open and thus this file still exists.

Just copy this file to somewhere and you can perform the recovery procedures outlined in the previous directory.

In this case:

Conclusion

None of this hard work would be needed if you made backups or used version control, but hopefully this can help as a disaster recovery guide in the future.

Flatiron Engineering

Thoughts from the Engineering team at Flatiron Health.

Marius van Niekerk

Written by

Flatiron Engineering

Thoughts from the Engineering team at Flatiron Health. We're building technology that enables cancer researchers and care providers to learn from the experience of every patient.

Marius van Niekerk

Written by

Flatiron Engineering

Thoughts from the Engineering team at Flatiron Health. We're building technology that enables cancer researchers and care providers to learn from the experience of every patient.

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store