You need to be careful on what you print on Jupyter Notebooks
Today, I was working on a dataset, using Jupyter Notebook application.
I was trying to parse the dataset from one format to another. And the dataset size was relatively large, about 2 Gigabytes of text.
While parsing, I fell into the carelessness of printing the parsed data values. Which resulted in printing a significant part of the 2 Gigabyte text.
It resulted in having a notebook with 200 Megabytes of size. I guess since the information in it corresponded to millions of text lines, browser was not able to open the Jupyter Notebook anymore. That means I could not even copy paste my code to recover my work.
I didn’t want that. I wanted my work back, so I tried a few things.
Things I have Tried
1- Cell -> All Output -> Clear
I tried to clear all cell outputs, and then save the file.
It did not work, even though I managed to press “Clear” and then hit Ctrl+S before the webpage died. I tried this multiple times, but there was no hope.
2- Deleting Lines Manually
I knew that Atom handled large text fıles better than the default text editor, so I tried editing the .ipynb just like we would edit a .txt file, to delete the unnecessary parts.
Atom was crashing after I scrolled more than a few thousand lines in the .ipynb file. I did not know how many lines it contained, but I changed the difference in file size.
0,5 Megabytes was reduced from the 200 Megabyte file, with a deletion of few thousand lines. So Atom was not a good solution, since it crashed when it loaded more than 0,5 Megabytes of lines.
3- Deleting Lines with Python
I think this might have been done much better, however I was confident that this approach would solve my case, so I tried the particular solution:
- First, I created a backup of the corrupted file, just in case I do anything wrong.
- I have read the .ipynb file from python, then wrote back only a few lines back.
- To be more clear, I did not open the .ipynb file in Jupyter. I opened it as a readable and writable file in Python, to modify it. Just like you would open a .txt file in Python.
- First and last lines, I wrote back to the file. The millions of other lines that were in between, I did not write back.
This resulted in me being able to open the .ipynb in Atom again, since file size was reduced to 2 Megabytes.
There, I found the relevant cells contatining the code, copied the code, and my work was recovered.
This is a basic issue, however it seemed interesting to me. Turns out you need to be careful on what you print on Jupyter Notebooks.
Note: Modifying an .ipynb like you would modify a .txt file corrupts the .ipynb if you don’t do it carefully. So take a backup of your notebook before you modify it like .txt, if you will.