Visual Studio Code — the swiss army knife for threat hunting with Azure Sentinel

Jupyter is a great platform for threat hunting where you can work with data in-context and natively connect to Azure Sentinel using Kqlmagic, but adding Visual Studio Code to the mix will give you even more superpowers!

When working with Visual Studio Code and Jupyter you get intellisense, debugging, a variable and data explorer, and live sharing; making the life of security analysts a bit easier. In this blog I’ll show you how.

Threat Hunting with python, Jupyter and Kusto

Jupyter Notebook, formerly called IPython, is an open-source application that allows you to create and share documents that contain live code, equations, visualizations and narrative text through markdown. It is already broadly used in cybersecurity for threat hunting, and has support for lots of programming languages such as R, Python, etc. The multi-user version of Jupyter is called JupyterHub.

Microsoft created a magic extension for Jupyter called Kqlmagic that allows you to work with Kusto-based workspaces such as Log Analytics, Azure Security Center, Azure Sentinel and more from a Jupyter notebook using KQL (Kusto Query language).

PRO TIP: I’ve written a blog earlier this year that helps you get started and provides a step-by-step tutorial to connect to Azure Security Center and Azure Sentinel. Read it here.

How do I get Jupyter up and running?

Microsoft’s offers a free hosted service called Azure Notebooks to develop and run Jupyter notebooks in the cloud with no installation. Although it is a free service, each project is limited to 4 Gb of memory and 1 Gb data. However, if the Azure Active Directory account you sign in with is associated with an Azure subscription, you can connect a Data Science Virtual Machine (DSVM) instances. DSVM’s are available from the Azure Marketplace and provide you with better processing power and removes any of those limits.

If you want to run Jupyter locally, you can install it using pip. However, I strongly recommend installing Python and Jupyter using the Anaconda distribution, which includes Python, the Jupyter Notebook, and other commonly used packages for scientific computing and data science.

For even more convenience, Jupyter is available as a Docker image. The Jupyter team maintains a set of Docker image definitions in the https://github.com/jupyter/docker-stacks GitHub repository. Not sure which Docker image to choose? Here’s documentation describing them, and their images and relationships. My suggestion: use scipy-notebook.

PRO TIP: Use this command to load a Docker-based Jupyter container on port 8888: docker run -p 8888:8888 — name jupyter jupyter/scipy-notebook

Why use Visual Studio Code?

Jupyter is a great platform for threat hunting where you can work with data in-context and natively connect to security backends in Microsoft Azure using Kqlmagic, but adding Visual Studio Code to the mix will give you even more superpowers!

Microsoft released a Python extension for Visual Studio Code in the extension marketplace that now supports Jupyter and works both on Windows and MacOS. If you are working on Windows and want an isolated environment for working with Python, the Windows Subsystem for Linux (WSL) is a great option. You can enable WSL and install a Linux distribution on your Windows machine, completely isolated from your normal development environment and use the VS Code Remote — WSL extension.

When working with Visual Studio Code and Jupyter you get:

A familiar interface

You might already be using Visual Studio Code for other things, and can just add this to the list of great things to do with VSCode :-)

IntelliSense

The Python Interactive window has full IntelliSense — code completions, member lists, quick info for methods, and parameter hints. You can be just as productive typing in the Python Interactive window as you are in the code editor.

Live Sharing

The Python Interactive window also supports Visual Studio Live Share for real-time collaboration. Live Share lets you co-edit and co-debug while sharing code, terminal, comments and more.

Variable Explorer and Data Viewer

Within the Python Interactive window, it’s possible to view, inspect, and filter the variables within your current Jupyter session. By expanding the Variables section after running code and cells, you’ll see a list of the current variables, which will automatically update as variables are used in code.

Debugger

The Visual Studio Code debugger lets you step through your code, set breakpoints, examine state, and analyze problems. Using the debugger is a helpful way to find and correct issues in notebook code. Open the command palette (Ctrl+Shift+P) and run the Python: Debug Current File in Python Interactive Window command.

Show me the money!

To work with Jupyter notebooks, you must activate an Anaconda environment in VS Code, or another Python environment in which you’ve installed the Jupyter package. To select an environment, use the Python: Select Interpreter command from the Command Palette (Ctrl+Shift+P).

To connect to the Jupyter server running in the Docker container:

· Run the Python: Specify Jupyter server URI command from the Command Palette (Ctrl+Shift+P).
· When prompted, provide the server’s URI (hostname) with the authentication token included with a ?token= URL parameter.

PRO TIP: You define Jupyter-like code cells within Python code using a #%% comment.

Run Cell applies to only the one code cell. Run Below, which appears on the first cell, runs all the code in the file. Run Above applies to all the code cells up to, but not including, the cell with the adornment. You would use Run Above, for example, to initialize the state of the runtime environment before running that specific cell. Selecting a command starts Jupyter (if necessary, which might take a minute), then runs the appropriate cell in the Python Interactive window:

When you’ve activated an environment with Jupyter installed, you can import a Jupyter notebook file (.ipynb) in VS Code as Python code. Once you’ve imported the file, you can run the code as you would with any other Python file and also use the VS Code debugger. When you open a notebook file, the Python extension prompts you to import the notebook as a Python code file:

To export content from VS Code to a Jupyter notebook (with the .ipynb extension), open the command palette (Ctrl+Shift+P) and select Python: Export Current Python File as Jupyter Notebook. It creates a Jupyter notebook from the contents of the current file, using the #%% and #%% [markdown] delimiters to specify their respective cell types.

PRO TIP: You define Jupyter markdown text cells within Python code using the #%% [markdown] comment.

Conclusion

While Jupyter, together with Azure Sentinel, already provides a powerful combination for threat hunting, adding Visual Studio to the mix supercharges it even more. Intellisense, debugging, live sharing and the variable viewer are very useful if you’re a security analyst working with notebooks.

Happy hunting!

— Maarten Goet, MVP & RD