Google Colab + .NET Core
I recently moved a data analysis side project of mine onto Google Colab (https://colab.research.google.com). Colab in conjuntion with BigQuery and Google Drive have been fantastic. I obtain raw data, parse it and then ingest into BigQuery. I’ve been using Colab + Google Drive to run these ETL pipelines into BigQuery, because hey it’s free. Thanks Google, Colab is awesome!
My process has always been to run the dataset through the parser locally, ship the bits up to Google and then kick off the BigQuery ingestion. The parser process inflates the data substantially, so I decided to see if I could rather ship the raw bits and then include the parser in the ETL pipeline.
The Google Colab host runs on Ubuntu 17.10 and my parser was written in F#. With .NET Core being cross platform, if I could port my app across I could theoretically get it running inside Colab.
After a bit of trial and error I managed to get it working. When I showed it to a colleague he encouraged me to write this, my first article. So here goes…
.NET Core Console App running inside Colab
We are going to be working directly inside a Google Colab notebook for the duration of this article. To get started head over to https://colab.research.google.com and fire up a new Python3 notebook.
Installing dotnet core
The Colab environment is targeting Python users, as such the dotnet runtime + sdk are not installed out of the box. Since Google gives us a lot of freedom on these Colab machine, we can install it ourselves.
Colab is a Linux host environment, which meant I could simply follow the appropriate guidance, provided by Microsoft, for the target platform.
You might be asking “what’s the deal with all the exclamation marks?” In any notebook cell if you start a line with an exclamation mark (!) the command gets piped to the underlying shell.
We only need the SDK because we’re building the application in situ. If you were to bring your own precompiled .NET Core app you would only need to pull down the runtime.
Installing the dotnet core 2.1 SDK
Setting up the console app
For the purposes of this demo we’ll be using the HelloWorld console app that the dotnet cli generates when bootstrapping a new console application.
At this point in time we have a .NET Core 2.x app sitting at /content/HelloColab/bin/Debug/netcoreapp2.1/HelloColab.dll.
Running the console app (where’s my exe?)
Something that confused me when compiling my first cross platform .NET app was the output type. In order to make all platforms happy, a .NET Core app gets compiled down to a dll (you can get a target specific output if you target a specific platform, but that’s a story for another day…).
You can also pass python variables through as command line arguments. I used this to compose my .NET core parser into my ETL pipeline.
I’ll admit this example is a little contrived because the HelloWorld app doesn’t do anything with the inbound args array, but hopefully this illustrates the point.
Being able to run .NET Core inside Colab has been super useful to me. It has allowed me to re-use existing code and get more done. Hopefully you’ll find this equally useful.
Bonus: Web Api running inside Colab
Armed with my new found .NETty goodness inside Colab, I was curious to see how far I could take this. The challenge was to stand-up a web api inside Colab and then call it from another cell.
I am personally still trying to find a use-case for this, so if you think of anything let me know.
Creating the web api
Again we’ll be leveraging one of the dotnet cli project templates as our demo application.
Ok so now that we have a web api, let’s try to run it just like the console app.
There’s just one small problem…it’s blocking. Great! So we have an api, but we can’t use it. We need to get this up and running on a separate thread. Time for some magic, or should I say magics.
%%script me an api
IPython, and by extension Colab, includes a concept known as cell magics. Basically it is just syntactic sugar that calls a function. You can read more about magics here.
The magics (magic? magics? whatever…) we are interested in is %%script (docs). By calling it with the optional bg argument we can have it run on another thread. Additionally we use the optional proc argument so we can get a handle on the process (more on that later.)
Let’s take a quick peek at the output (stdout) from the process, to see if we are getting the same console output as in the blocking case.
Excellent! So far so good.
Next we need to verify that the api is actually up and running. We will use the very popular Requests Python library to try and hit the /api/values endpoint.
Let’s create a helper function to do this as we’ll use it a few times.
Now we call the function and see if we can get a response.
Awesome, we just got a response from a .NET web api that we hosted inside Colab itself.
Shutting down the web api
Last but not least we need a way to clean-up after ourselves. This is where the process handle comes in very handy. The web api starter project is waiting for a CTRL-C input (SIGKILL) to start its shutdown process. The process handle we got earlier is actually an instance of a Popen object (doc), which has a handy function (send_signal) for passing signals down to the process.
There we have it. We took a .NET Core web api through its full life-cycle and used it all from within Colab.
Conclusion
In this article I’ve demonstrated that it is possible to combine the power of .NET and Google Colab. I am hoping to play with using the api inside Colab a bit more in the future.
You can access the full notebook here. Thanks for reading.
