A Rant On Why I Despise Jupyter Notebooks
I remember one day messaging one of my seniors asking them, “Dude, how do you write such good, structured code”. But unfortunately, since he was so used to good coding practices it was natural for him, and he didn’t have an answer. Feeling dissatisfied, I spent the next 2 months thinking , looking at repositories on GitHub, watching random videos on YouTube trying to understand what was causing me to write such spaghetti, unusable code, until it finally hit me.
One of the worst mistakes I ever made was to use Jupyter notebooks to create projects. I still regret it , but hopefully I know now, how one should use them.
I seriously hate Jupyter notebooks. It’s a mess. I wouldn’t be using it if it were not for Colab.
If you are beginner to Machine Learning, then I’m sure you might have seen those huge lists of skills required which contains editors like VS Code and then Jupyter notebooks or Jupyter Lab. Next thing you might jump to writing notebooks , feel great, list them in your Resume as a skill BUT forget an essential skill.
Whenever you read any job description it’s often mentioned that the candidate should have the ability to read and write clear code of large codebases. Do you think writing in Jupyter notebooks is going to give you that experience?
Not, if you are a beginner. As a beginner you might not follow the principles of OOP. You might write only simple functions. You might not even realize the need to write functions! And what this leads to is embarrassing .
This leads to projects which are contained in Jupyter notebooks that you cannot deploy in production, cannot reuse it by importing it as a module? or a library?
Even if you can, why take the pain to use a NotebookLoader and other hacks to import a .py file into a notebook. Reproducibility of Jupyter code is sh*t which is due to the problem of “cell states”. Good luck trying to wonder which state, corresponds to which run, and the weird outputs which you will receive that will make you want to give up on coding altogether or if you realize rerun the entire notebook.
One such incident I recall where I tried a Leetcode question in a Jupyter notebook, thinking back of it I feel like jumping off the top of the Burj Khalifa in utter dismay and cringe.
According to a study 73% of all Jupyter notebooks are not reproducible with straightforward approaches [1]
There is no fucking way I’m using a tool called OSIRIS now to check for reproducibility, as If I have enough time. I’m using notebooks only if I’m forced too or trashing the code away.
By writing in plain files, you will be able to learn loads of Python concepts that you might never will in Jupyter notebooks due to laziness of copy-pasting code and since you aren’t handling many files, and complexity at once. It’s painful trying to reproduce your own Jupyter code from months back. I would rather take the pain now to write better modularized code, or even a simple design pattern, so that everything works fine with a simple command than worry about cell states breaking my entire freaking execution workflow.
Yes, you can deploy a Jupyter notebooks using Kedro[2] but why make things difficult? I don’t think most products would ever deploy a notebook to production, especially with the latency issues that come with it.
The goal of projects should be not only to display what you have made but also to be reproducible, usable, modifiable, fork able by other users. One great advantage is the ability to directly deploy than having to restructure from notebooks for deployment, if it is too specific.
A better solution would be to make a small module, and then showcase it or use it via a Jupyter notebook. By doing so not only do you get to use your beloved Notebooks, with its ecosystem but also create something that is usable.
The only good thing about Jupyter is the ability to experiment code or train a machine learning model, save its weights and, trashing the entire notebook as and Yet_Another_Untitled_.ipynb to adapt it into modular code.
Well, I chose to spend hours on a REPL or even take the pains to use VIM than open a notebook to execute things in cells.
Feel free to follow up with your idea on Twitter @agrover112, for taking on the Herculean task to change my mind.
Join my email list for more helpful insights on tech and non-tech !
References
[1]Assessing and Restoring Reproducibility of Jupyter Notebooks (lilicoding.github.io)
