How to update Overleaf from a Juypter Notebook

Jupyter on the left and Overleaf on the right. If you follow the steps, you can pretty much update Overleaf from Jupyter

Why Overleaf and Jupyter together

Jupyter is a powerful workhorse where you can integrate data, text and graphics together. If you ever work on a document where you will need to pull in data, analyse any form of data using say something like R, or Python, or Julia, (Jupyter evolved from the combination of these three Programming languages), give Jupyter a try. You will not likely to go back to any other workflows and processes. Besides, You can easily use Jupyter to develop a workflow that scholars like Kieran Healy write about. You can download and install Jupyter on your computer and work from there. Or you can use something like Sagemathcloud to work, or use Wakari.io or tmpnb.org. If you use Julia use juliabox.org. There are many many ways you can try Jupyter notebooks and they are worth learning.

You can write markdown in Jupyter (that is its main format), and weave together what is known as ipynb workbooks (interactive python workbooks). But Jupyter also bundles in it a nice terminal, and you can import plain text documents and write them using Jupyter.

Overleaf is a powerful web app that can weave documents in LaTeX and can be used to push contents to external sites and academic blogs and to journals for publication. You do not need to know much LaTeX to work with Overleaf (it helps though).

While using Jupyter makes the work of continuous integration with data and text very systematic and brings in the references stored in the computer and on the web in one platform, and offers the ease of writing in markdown and presenting the initial piece of work seamlessly with friends and colleagues who either or not have ipython/jupyter, having Overleaf makes it easy to control the final appearance of the document to be pushed everywhere really easy.

So, having Overleaf and Jupyter play with each other will be particularly beneficial in those situations where we have to do data analysis, and then would like to put everything out to seamlessly to other scholarly documents and servers. Besides, as Jupyter stores everything in markdown and it can be converted to a markdown document, it can be hosted in blogs such as Ghostblog, or directly uploaded as ipynb notebooks through github to display the work done. Not only this. It is possible to actually set up what is known as a gist to display the analysis and web page together. This can also be helpful to take in data and other tools if you are connected to the net all the time.

What do I want to do?

The idea then is to use both Jupyter and Overleaf to update each other. At this stage, it is an instance of Jupyter hosted in my desktop computer where I compose my statistical data analysis and graphics, and bring in bibtex files from all over the web, and import from my reference manager. Then write and then with the push of a simple set of terminal commands, I can dynamically update an Overleaf document. Want to give it a try? Let’s go.

First Steps, use Overleaf to create a document

1. You will need a free Overleaf account and you will need an access to Jupyter notebooks.
2. In the Overleaf, go and open up a new document. I usually use a blank document to start with which I will gradually fill in.
3. Type in the name of the paper that you want to write in Overleaf.
4. Next, click on the “share” button on the Overleaf page and note where they say that you can connect to git
5. If you have done that, you are all set, :-)
6. Next, start a terminal and type jupyter notebook. I used the terminal in my Mac to start a jupyter notebook, you will need to have jupyter installed. If you do not have jupyter, go install it from here

Installing Jupyter Notebook - Jupyter Documentation 4.1.1 alpha documentation

The documentation is quite comprehensive and easy to follow.

Move to Jupyter

First start a jupyter notebook by changing into a folder and typing:

jupyter notebook

This will start a Jupyter notebook. Stay in the terminal.

In the same folder, type

git clone <URL> (where URL is going to be the URL you copied from Overleaf)

1. Jupyter notebook will start in your browser. Write your text in it.
2. When you are ready to push to Overleaf, bring up a terminal. You can do it within the Jupyter notebook. Click on the home button and add a new terminal, if one is not already running. Alternatively, and I find it a little more intuitive, you can open a new shell or a terminal window anyway rather than working from the emulated terminal
3. After you have typed the git clone code in the terminal, you are pretty much set

Next steps: Let’s build the bridge

• For this you will need pandoc in your machine or installed in the server from where you are working. You can find more information on Pandoc here

Pandoc - About pandoc

• At this stage, you have a document that reads xxx.ipynb. That is an interactive python notebook. To move over to Overleaf, you will need to create a latex document from it. Here are the steps:
• First, convert the ipynb file to a markdown file. Issue

jupyter nbconvert --to markdown <jupyter.ipynb>

Replace <jupyterfile.ipynb> with whatever filename that ipynb file you have.

• Then, process the markdown file with Pandoc, thus

pandoc -f markdown -t latex <filename>.md -o filename.tex

• Now, copy the latex file and whatever other image files etc you have to the folder that Overleaf git created for you. Here is the code:

cp filename.tex <overleaf_git_folder> 
change the <overleaf_git_folder> with the actual folder name.

• Then use your git to push it to Overleaf thus:

git pull
git add .
git commit -m “type a message here”
git push

At this stage, you may want to move over to Overleaf and tweak your files. Add a bibliography file, and add latex file you created in python notebook in Overleaf and you will be all set. Open the main.tex file and do:

\include{overleaf}
\bibliography{bibliographyfile}
\bibliographystyle{put your favourite style here}

Add these in the body of the document.

These few steps will set up your Overleaf and Jupyter to play with each other as you keep updating your paper in Jupyter, bring in more analyses, and keep updating your Overleaf file till you are happy to submit or stop working with it.