My 2015–2016 changelog, a story of 2000 commits
My GitHub graph just ticked over 2,000 last week. I wasn’t aiming high, actually I don’t really know how that happened. In previous years it’s been in the low hundreds. It seems like a lot, to others it might not seem like that much, but the point I’m proud of is it covers over 50 projects and 7 languages.
2015 was a breakthrough year for me, starting to contribute more to OpenSource, realising it’s not all that scary, learning new languages and embracing the OSS community in a way I hadn’t done before.
So, imagine if you will, this is my personal CHANGES.rst file for release 2015–2016.
OSS Work
Apache Libcloud (Python)
This started off as a need to have a Dimension Data Cloud API driver for Libcloud, I didn’t know Python, I was really sick and stranded in a hotel in Seattle, so I learnt Python and put something together.
It was a copy of an original “Opsource” driver which is the company Dimension Data acquired to run the cloud, so I then made a number of changes to it whilst learning Python.
As I learnt more about Python I rewrote the whole thing and now it covers every function of the API (also thanks to another 8 engineers at Dimension Data).
I was asked to jump onboard at Libcloud and then ended up merging, maintaining and documenting a lot of the project along with Tomaz Muraus (StackStorm), Eric Johnson (Google), Jerry Chen (Union Metrics), Paul Querna (ScaleFT) and others.
I added over Christmas a “container-as-a-Service” abstraction subsystem to libcloud and also rewrote the HTTP library using requests (but never had the guts to merge it since I broke >4000 of the 6321 tests in the PR).
Tomaz Muraus has done a huge amount of work on this project, it’s received well over 2million downloads and it’s being used in some major open-source and commercial projects. I owe a lot from Tomaz pulling my first few PR’s to shreds as I was still learning Python (and continuing to).
You can hear me on FLOSS weekly episode 373 and on Podcast.__init__ Episode 55 talking about libcloud, Python and why vendor API lockin is a terrible idea.
StackStorm (Python)
I stumbled across StackStorm (or “ST2") in it’s infancy purely because I noticed Tomaz was working there. After checking it out I realised this thing was going to be big.
What StackStorm does is allow you to programmatically react to events using complex workflows, it’s not just your typically run-book automation tool but a pluggable system of workflows, sensors, and also some really slick chatbot integration with Slack.
Having attempted to figure out how to do similar things using Microsoft System Center Orchestrator, Operations Manager and Service Center in the past, this was like jumping out of cold treacle and onto a hover-board.
Because StackStorm was prerelease when I started using it, I decided to get behind the project and whilst I was developing what I needed I would contribute it upstream.
- A pack for the Dimension Data cloud, Qualys, Tesla Motors, Octopus Deploy, Yammer
- Improvements to the ServiceNow pack, the Windows pack
- Integrated Cisco Spark in as a Chat provider
Some key lessons from working with StackStorm were that community engagement is make-or-break these days. The whole team were approachable, smart and quite fun to collaborate with. We replaced a Opalis-based workflow that took 5 days to complete into a modular series of workflows in StackStorm that takes a few hours to finish. We can jump into certain steps, remediate failures at any point in the chain and the whole thing is stored on Git.
StackStorm was acquired by Brocade, I’m continuing to work with it and we’re using it daily in our DevOps team. Check it out https://stackstorm.com/
I also added a C# client and a PowerShell module for calling their APIs https://github.com/tonybaloney/St2Client
Pyjion (C++, Python)
Brett Cannon (Core CPython developer) and Dino Viehland put together a proof-of-concept extension point for CPython that allows you to branch the execution of the byte-code into a seperate JIT engine. They did this using the CoreCLR JITer, which was beta at the time. I heard about it on 2 podcasts in the same week and jumped on it to see how it worked. Sadly, Microsoft were making major changes to the CoreCLR project at the time and it no longer built, so I and my colleague Adam Friedman worked on updating it for RC2. There’s still a task to update this for 1.0.
Pyjion was later proposed as a Python Enhancement or PEP and is being discussed as a permanent feature of CPython.
Checkout my talk on what it how this works http://www.slideshare.net/AnthonyShaw5/pyjion-a-jit-extension-system-for-cpython
SaltStack (Python)
I got introduced to SaltStack via a colleague who wrote a wrapper for our cloud API so that users can provision servers on the Dimension Data Cloud. I offered to help get that merged and review the code for him. It was pretty seamless, the team at Salt are really easy to work with.
SaltStack is a DevOps tool, based on agents (minions) that take command from the master via ZeroMQ protocol. Its a hugely flexible replacement to the SSH-style of operations (login, command, logout).
SaltStack has the dangerous combination of an awesome community, simple extensibility and lots of opportunity for modules that I was hooked.
Since I’ve picked it up I’ve contributed a number of new modules as well as fixing and adding tests to existing libraries.
- Pushed Salt-Cloud support for dimension data cloud
- NSO support (see the PyNSO module) as a proxy minion
- Added a module for the WMF5/PowerShell 5 “PSGet” module installation sytem
- Added a servicenow module
- Added a state and an execution module for libcloud’s DNS abstraction
- Added a way of quickly extending salt using a “salt-extend” command
csco-spark, spark-hubot (Javascript, CoffeeScript)
As part of extending StackStorm to work with Cisco Spark (Cisco’s chat and video/voice system), I needed to build a Hubot module. Hubot is GitHub’s open-source chatbot framework.
Hubot modules are typically written in CoffeeScript, so I had to learn that first, then set about working with the Cisco Spark APIs to build a module. I hit a hurdle early on because the NPM package I was using was missing some functionality, so I raised a couple of PRs to get it to where I needed it to be.
Then, I worked on the Hubot module, which can send and receive chat messages to the Spark APIs. It’s had a few improvements and I have received a couple or PRs and issues from the community. https://github.com/tonybaloney/hubot-spark
Checkout a demo of the working integration
SphinxContrib-ConfluenceBuilder (Python)
Atlassian Confluence is a very-mature CMS/Wiki. We use it a lot for work and for public facing content. It does have it’s own proprietary markup language, which makes auto-generating content from say, an API specification very hard.
Sphinx makes that really easy, but it works with RestructuredText formatting, so I developed a Sphinx Extension to generate the confluence format and then talk to a Confluence API endpoint (optional) and publish the pages. We’re going to use this to auto-publish API specifications and documentation out of our Git repositories in a similar way to ReadTheDocs.io.
https://github.com/tonybaloney/sphinxcontrib-confluencebuilder
Work projects
Dynamic PowerShell API (C#.NET)
This is an interesting project, it’s written in C# and it turns a directory of PowerShell scripts (with known parameters) into REST API endpoints. Why? Well, we wanted to automate some Exchange Server functionality, Remote PowerShell is not my favourite solution and this seemed like a better approach. This was we can get control over exactly which scripts get run on the box and test them prior to production, instead of-runtime-hacking-over-RPC (Remote PowerShell)
https://github.com/DimensionDataCBUSydney/PowerShell.REST.API
PSUnit (PowerShell)
I took a project for unit testing PowerShell, updated it for v4 of PowerShell and made a number of other improvements and raised a pull-request. It hasn’t been touched (dead project?)
https://github.com/DimensionDataCBUSydney/PSUnit
Citrix Provisioning Plugin (C#.NET)
I attempted to integrate the Dimension Data Cloud into the Citrix provisioning worker for an upcoming and since released version of Citrix. I think I was and still am the first person outside of the Citrix engineering team to do this, so I was shooting from the hip. I hit a few hurdles really from lack of documentation and this was put on hold.
Fog module (Ruby)
Fog is a Ruby library for calling cloud provisioning APIs, similar to Apache Libcloud but with some significant differences.
I started to build out a Fog module for the Dimension Data Cloud, realised that Fog doesn’t normalise models and therefore wouldn’t be much use over a direct Ruby client, which I have yet to work on.
Plumbery (Python)
Plumbery is an Infrastructure-as-Code tool specifically for the Dimension Data Cloud, it was developed by Bernard Pacques and built on the Apache Libcloud module. Bernard has done some wonderful work on this project, we’re using it to automate all sorts of complex infrastructure and applications on the Dimension Data Cloud. I helped extend it to support Windows, run on Docker and various refactorings.
PyNSO (Python)
NSO, or Network Services Orchestrator is a product by Cisco that stores all your network configuration in a single database and you can work on that datastore and it will be pushed out to your endpoints.
There was a REST API for NSO, but I had to beg, borrow and steal to get my hands on it. I finally managed to then write a Python client around that API which I could use for a number of other integrations (like the SaltStack one).
Enterprise Open-Source (work in progress)
I’m working on documenting a framework for running a modern OpenSource project, or applying OpenSource principles to a software project.
ScienceLogic EM7 API Client (Python)
I put together a Python client (and PyPi module) for ScienceLogic’s EM7 monitoring system. We use it on our cloud service, but there weren’t any working API client libraries out there so I had to put one together. This has been deployed in production since.
RightScale libcloud UCA agent (Python)
I put together an agent for RightScale’s universal cloud-appliance using Libcloud, this UCA subsystem is still in beta so you need to be working closely with RightScale to get access. With this module you can plug into any cloud that Python supports using a drop-in configuration file and then see the assets in RightScale and track costs etc.
https://github.com/tonybaloney/rightscale-agent-libcloud
Minor projects
wntf (Python)
As part of a blog post I wrote an anti-recommendation algorithm for Twitter, called “who not to follow”, this was really fun. The blog was posted on kdnuggets.com and has received nearly 1000 views. https://github.com/tonybaloney/wntf
readysalted (Python)
I’m working on design patterns for Internet-of-things mesh control planes using SaltStack. Still experimenting with different approaches when I get the odd afternoon spare.
https://github.com/tonybaloney/readysalted
What did I learn from all this?
Coding is 50% of the work, documenting, packaging, testing, fixing and sharing projects takes up a lot of time. But just throwing projects up on GitHub isn’t much use to anyone. I’ve started to document what is required for a complete project on my enterprise open-source project (see above) before people would ever consider using it.
What next?
I want to make sure that if I do start projects that I see them through, so I’ll continue to polish and maintain the libraries I shared.
Realistically, I want to halve the contributions and focus on the people and process. I’m hoping to get more and more people at my company involved in OpenSource, work with the wider community and teach what I can to people who can provide real value.
I’ve put forward a number of initiatives internally to try and improve awareness in OpenSource, drive some training and learning platforms around Python and my goal is to teach 1000 people in our company Python by this time next year.
