Writing an Interactive Book πŸ“– over the Threat Hunter Playbook 🏹 with the help of the Jupyter Book Project πŸ’₯

Roberto Rodriguez
Dec 18, 2019 Β· 10 min read

Well, I decided to start writing a book πŸ˜†, but an interactive book 😎! Yes, a book where I could not only share detection concepts, but also allow the readers to interactively run every analytic provided as part of my research in a public computing environment all from a web-browser. It is still a work-in-progress, but I believe it is time to start putting together some of the projects that I have built so far and share them in a more traditional reading experience, a book, and for free of course πŸŽ„!

In a previous post, I shared how I was able to integrate detections from the Threat Hunter Playbook initiative and pre-recorded datasets from Mordor with the amazing BinderHub project. That was something that I had not seen before, but I felt that I still needed to package all those ideas and resources in a more practical and easy-to-read format.

How can I share the rich interactive Jupyter notebook experience in a book format?

In this post, I will show you how I took all the Jupyter notebooks available in the Threat Hunter Playbook project and some markdown files to create a static website leveraging the amazing Jupyter Book project originally developed by Chris Holdgraf and Sam Lau with support of the UC Berkeley Data Science Education Program and the Berkeley Institute for Data Science. In addition, I will show you how I create Jupyter notebooks programmatically from YAML files using a python library named nbformat to enhance the Jupyter Book content update and documentation of the project.

Pre-Requirements

I highly recommend to read the following post to have a better understanding of most of the projects I will be mentioning in this post:

What is a Jupyter Book?

Jupyter Books lets you build an online book using a collection of Jupyter Notebooks and Markdown files. Its output is similar to the excellent Bookdown tool, and adds extra functionality for people running a Jupyter stack.

The result is a static website with notebooks and markdown files underlaying HTML pages πŸ˜‰ Also, an online book with capabilities to interactively run notebook cells either with a local JupyterHub deployment or online via a public computing environment such as BinderHub.

What is a Static Website?

The concept of a static website comes from taking static files such as markdown files and turning them into HTML files that can be deployed online. For example, this is something you can do with GitHub pages. I find it very flexible and easy to do when deploying a website without the need to know how websites are developed. The Jupyter book project uses the structure of a Jekyll website.

A Jupyter Book Recipe

With Jupyter Book, all you need in order to build your own book are the following files:

The notebooks and markdown files for the static website need to be all in one folder as shown in here. That’s all you need to start creating a Jupyter Book 🍻

Installing Jupyter Book CLI

Before continuing with this process, you will need to install the jupyter-book python library. This library allows you to create, build, upgrade, and otherwise control your Jupyter Book. You can install it via pip:

pip install jupyter-book

Or from the latest version from GitHub (still via pip):

pip install git+https://github.com/jupyter/jupyter-book

The book building process

According to the official Jupyter Book docs, building a Jupyter Book consists of three steps:

Jupyter Book expects a particular collection of files and folders that work with the Static Site Generator Jekyll. In my case, I used the following Threat Hunter Playbook files , toc.yaml, and config.yaml files with the jupyter book CLI to create the book template. You just need to run the following command:

jupyter-book create mybook --content-folder /path/to/content --config myconfig.yaml --toc mytoc.yaml

You will get a similar book structure as shown in here:

mybookname/
β”œβ”€β”€ assets
β”‚ └── custom
β”‚ β”œβ”€β”€ custom.css
β”‚ └── custom.js
β”œβ”€β”€ _config.yml
β”œβ”€β”€ content
β”‚ β”œβ”€β”€ features
β”‚ β”‚ β”œβ”€β”€ features.md
β”‚ β”‚ └── notebooks.ipynb
β”‚ └── LICENSE.md
β”œβ”€β”€ _data
β”‚ └── toc.yml
└── requirements.txt

Jupyter-book can also help you to convert each page of your book into HTML. This converts your .ipynb, .md, etc files into HTML that can be understood by a website. See the building each page's HTML section for more information in their official docs. Use the jupyter book CLI to run the following:

jupyter-book build mybook/ --overwrite

you will see a new folder being created named_build which will host all your markdown and notebook files in HTML format similar to the ones in here.

Once you have HTML files for each page, you will need to put them all together as a standalone HTML file that can be hosted online. See the build your book’s html section for more information. I built mine leveraging GitHub Pages so I did not have to do it myself.

Once your book’s HTML is built, you can host it online as a public website. Once again, in my case GitHub Pages takes care of it for me. Basically, after creating and building the book under the folder docs in the ThreatHunter Playbook repo, I went to the repository settings and selected the source of my GitHub Page as shown below. I also added the following custom domain threathunterplaybook.com to it.

Once the GitHub pages feature is enabled successfully in my repository, I can go to the following site and see the whole project in a book format:

https://threathunterplaybook.com

That’s it πŸ™?

Yeah, that’s pretty much it from a Jupyter Book perspective! However, I added some extra automation on the top of this process to make the contribution process and update of the book much easier for future builds.

Automating the Creation of Notebooks πŸ—

As you already know, the Threat Hunter Playbook project documents detection strategies in the form of interactive notebooks to provide an easy and flexible way to visualize the expected output and be able to run the analytics against pre-recorded mordor datasets through BinderHub cloud computing environments. This has been a great integration for the project; however, not everyone has the skills, resources or time to create Jupyter hunting notebooks from scratch. Therefore, I needed to find a way to automate the creation of Jupyter notebooks from a template with an easy to use format. What if I document every threat hunter playbook as a YAML file and convert them into notebooks via code? How would I do that?

Enter NBFormat

nbformat contains the reference implementation of the Jupyter Notebook format, and Python APIs for working with notebooks.

The Threat Hunter Playbook notebooks follow always a similar format/structure. Therefore, I started testing if it was possible to use nbformat APIs to create notebooks automatically with some Python code from a YAML file.

The following document summarizes the main sections of every playbook provided by the project.

So far, this is how I document every single playbook to share with the community. However, how can you also create something similar without standing up your own Jupyter Notebook server?

Constructing Notebooks Programmatically 😎

You can use nbformat Python APIs to create markdown and code cells. The following is a basic example to show you how easy it is to create a notebook with a few lines of code:

  • Import nbformat library
  • Create a new notebook object
  • Initialize notebooks cells as an empty list
import nbformat as nbfnb = nbf.v4.new_notebook()
nb['cells'] = []
nb['cells'].append(nbf.v4.new_markdown_cell("# Remote Service Creation"))
nb['cells'].append(nbf.v4.new_code_cell(
"""from openhunt.mordorutils import *
spark = get_spark()"""
))
nbf.write(nb, "test.ipynb")

The result will be the following notebook 😱

All I need to do now is pass the Threat Hunter Playbook format as a YAML file through the nbformat APIs and create a Jupyter Notebook.

YAML File > Jupyter Notebook πŸ’₯😱

I translated the following Jupyter Notebook used as the main example in this post and translated it to the following YAML file:

Next, I created the following Python script, as a proof of concept, to convert the YAML file to a Jupyter Notebook with markdown and code cells derived from the YAML file.

The result was the following Jupyter Notebook with the same format as the one shown before:

YAML Playbooks > Jupyter Book Notebooks 🍻

Finally, I took this concept and added it as part of the Python script I use to update my Jupyter Book. One difference from the proof-of-concept script is that instead of converting one file only, I iterate over all the YAML files I created from the notebooks available in the Threat Hunter Playbook. All the notebooks created by the script get stored in folders inside of the Jupyter Book docs>content>notebooks folder and categorized following the MITRE ATT&CK structure (Platform>Tactic)

How is this an Interactive Book Again πŸ˜†?

Great question! According to the Jupyter Book docs:

because Jupyter Books are built with Jupyter Notebooks, you can connect your online book with a Jupyter kernel running in the cloud. This lets readers quickly interact with your content in a traditional coding interface using either JupyterHub or BinderHub.

If you read this previous post, you already know that I use BinderHub public infrastructure in order to share my research with other hunters in the 🌎. Therefore, I enabled the Binder button on the top of every page that is backed by a Jupyter notebook as shown below:

All you have to do is click the Interact button, and you will be taken to BinderHub’s public infrastructure, where the notebook will be hosted by a server created by my own BinderHub repository (Dockerfile).

There you will be able to interactively run analytics provided in the notebooks as shown below:

According to the Jupyter Book docs

If you’d like to provide interactivity for your content without making your readers leave the Jupyter Book site, you can use a project called Thebelab.

This provides you a button that, when clicked, will convert each code cell into an interactive cell that can be edited. It also adds a β€œrun” button to each cell, and connects to a Binder kernel running in the cloud.

I enabled it in my Jupyter Book as shown below:

If you click on that button, you will see the Run and Restart buttons on every single code cell in pages backed by a Jupyter notebook.

Remember that this is still an experimental feature! In the meantime, I still ❀️ the BinderHub redirection with the Interact button to use other features available in the Jupyter Notebook interface.

Future Work

  • Add more chapters to the Jupyter Book!
  • Add more analytics in other platforms!
  • I added a basic Jupyter Notebooks tutorial for this first release. However, I want to add more tutorials to show the power of notebooks in our industry
  • I am preparing some material for a few workshops and training classes for 2020 using all the projects mentioned in this post so stay tuned! πŸ˜ƒ 🍻 πŸ’œ

Thats it! I hope you enjoyed this post! Im so happy I got the Jupyter Book to work and added other enhancement to the projects I have been working on and updating for the last few months. My goal is to write more about every open source project I have worked on in the past couple of years and put it all together in this Jupyter Book πŸ™

Book Link: http://threathunterplaybook.com/

Finally, with this new project, I am also very happy and honored to share that I just got accepted to the GitHub Sponsors program! 😱 If you would like to fund my contributions and become a sponsor to my open source work, you can read about it here: https://github.com/sponsors/Cyb3rWard0g Thank you so much in advance πŸ™ 😊🍻

References

https://jupyterbook.org/intro.html

https://nbformat.readthedocs.io/en/latest/index.html

https://threathunterplaybook.com/

.

Threat Hunters Forge

Threat Hunting, Data Science & Open Source Projects

Roberto Rodriguez

Written by

Threat Hunters Forge

Threat Hunting, Data Science & Open Source Projects

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium β€” and support writers while you’re at it. Just $5/month. Upgrade