Driving GDS-vCloud-Tools with Jenkins

Why might you want Jenkins to drive vCloud Tools? For us, it came down to 3 simple reasons:

1. To minimize the time between configuration changes being pushed to git and actually being deployed.
2. To eliminate the risk of manual changes being made through the vCloud GUI, leaving undesired configuration on a live platform
3. Automate all the things!

When I started work on this story, I had a quick google to see if there was any material on existing implementations. I couldn’t find anything, so thought it was worth a blog.

Provisioning tools such as gds-vcloud-tools allow us to define our infrastructure through code. Which holds a number of benefits that although aren’t specific to being driven by Jenkins, are still worth quickly noting.

1. It allows changes to infrastructure to follow a formal review and approve process through git, giving confidence that the configuration in place on a live platform is correct.
2. It allows us to quickly glance at a few configuration files and fully understand the infrastructure that is in place, rather than clicking through a GUI
3. Significantly reduces time and effort to make changes to your infrastructure, by allowing you to define shared variables and re-use blocks of code.

The Goal

We currently have 1 Jenkins box (configured by puppet) controlling 2 development VDCs on vCloud.

We want Jenkins to update the configuration in vCloud as soon as a change is pushed to gitlab.

We also want Jenkins to ensure the configuration in vCloud matches that of our config files in gitlab every 2 hours. Just to be sure any ad-hoc changes that have been made in the mean time are removed.

Restrictions

One restriction we had was the version of ruby running on our ubuntu trusty VMs. It is 1.9 and the application being served from these VMs required it to stay at that. This is a restriction as vCloud tools requires version 2 or higher. To get round this restriction we used the rbenv-jenkins plugin which easily allows you to run Jenkins jobs in different rbenv’s. This plugin did require the installation of some dependencies which we included in our Jenkins puppet class.

The other restrictions are to do with the vCloud-tools themselves.

The most annoying of which was with the vCloud-net-launch tool, which fails as soon as it comes to a network in the configuration file which already exists. This means you need to add new networks to the top of the file so they are created before the command inevitably fails when it comes to an existing network. This means it exists with a status code of 1 and so the build in Jenkins will show as “Failure” despite actually being a success. And so we used a dirty hack to turn that red failure light green, but ill explain that later.

The next most annoying is a lack of functionality which means the tools will not delete networks or VMs if they are removed from the configuration files. It essentially acts in a “create-only” mode. Whereas with NAT and Firewall rules, if you remove them from the configuration file, they will be deleted, which is excellent.

Password Concerns

In order to run vCloud tools, you need to authenticate to your vOrg as a user with the appropriate permissions. Obviously, as we are automating the creation of this job we need the password to be stored securely.

Although the Jenkins password parameter doesn’t store the password in plaintext, even in the config.xml it generates, It does hash it using a private key that is stored alongside the rest of the configuration. And so we took the approach that if it could be encrypted using that key, it could be decrypted almost as easily. Also, we are already using secure Hiera, so it was easy to just use that.

As I mentioned before, we use Puppet with Hiera for configuration management. The things we use Puppet for here are:

1. To configure the fog file with the required vCloud specific parameters that vCloud-tools requires to connect
2. To install the rbenv plugin and its dependencies
3. To manage the Jenkins sudoers file to allow the specific commands we need in this job
4. To store and deploy the jobs config.xml

vCloud Tools

For those who are unfamiliar with vCloud tools, they are split into 6 tools, more info can be found here.

We use them to configure vApps & VMs, vNetworks, NAT rules and Firewall rules. And so for this we need 3 of the 6 tools:

• vcloud-launch
• vcloud-net-launch
• vcloud-edge-configure

In order to be able to only apply what has changed, we have split the configuration for 1 VDC into 4 different yaml files, as follows:

firewall-<VDC NAME>.yaml — For firewall rules

nat-<VDC NAME>.yaml — For NAT rules

vms-<VDC NAME>.yaml — For vApp/VM configuration

networks-<VDC NAME>.yaml — For vNetworks configuration

The Approach

We decided to achieve our goal with 2 jobs.

One which will trigger each time an update is pushed to our vCloud repo on gitlab. It will read which files have been changed and apply only these changes.

Another job will run only the Firewall and NAT rule vCloud tools every 2 hours. The vNetworks and vApp/VM tools are left out here, because as mentioned before they don’t have the ability to remove VMs or Networks and so there is no value in having them running on a schedule as new configuration will be applied via the on push.

The Jobs

Common to both the On Push and On Schedule Jobs

The first thing common to both jobs is simple, the connection to our vCloud repo on our gitlab server.

The next thing common to both is the rbenv config. Below you can see we set it to run under ruby version 2.

The next common component is part of the “Build Execute shell”. This part does the following things.

1. Checks if a Gemfile.lock exists, if it does not, it will run bundle install to the specified path.
2. Runs bundle install in deployment mode so that its defaults are set to deployment instead of development.
3. Turns off the trace of execution of the password sensitive commands so that the password is not noted in the console output.
4. Makes a call to Hiera to retrieve the password for the vCloud user.
5. Uses the credentials stored in the fog file under “<vOrg-Name>” along with the vcloud_password to login to vCloud, storing the token it gets in response to a variable

Specific to On Push job

The On Push Job then does a number of things of its own.

1. It reads in the latest push to the git repo and stores it in a “CHANGES” variable
2. Sets up a function to inspect the “CHANGES” variable which files have been edited and executes the vcloud-tools command required for those files. As mentioned before, we force the vCloud networks tool to exit with a 0 status instead of 1 so build does not show as failed.
3. As mentioned before, this Jenkins box manages 2 VDCs and so we have 2 if statements which query the CHANGES variable for the names of these VDCs, calling our function from above for whichever has been updated.

Specific to On Schedule job

The On Schedule Job is much simpler.

The first difference is the trigger, which is scheduled for every 2 hours.

It has the same function but without if statements as we simply want it to run an update on the firewall and NAT rules.

Desired Improvements

• Gracefully handle fail of vNetworks tool
• vApp/VM and vNetwork deletion