Open-sourcing The Uplift project — a Packer/Vagrant toolkit for SharePoint professionals

Published in

SubPointSolutions

12 min readMar 15, 2019

Building infrastructure for SharePoint is not an easy task. A bare minimum farm setup is hardly fun: it involves dozens of files to be downloaded, several dozens of configurations, several virtual machines, and hours of work.

Wouldn’t it be great just to run a few commands, and end-to-end setup happens automatically? Wouldn’t it be great to have this experience on MacOS as well? And what about my team members and other projects?

With this post, we start a mini-series on how we solved the complexity of SharePoint development, test or lab environments setup, and open sourced The Uplift Project — a Packer/Vagrant toolkit for SharePoint professionals.

Challenges with SharePoint infrastructure

Okay, what is so complicated about SharePoint infrastructure?

A very minimal development, test or lab environment consists of at least two virtual machines: the domain controller and SharePoint Server itself. High level setup looks more or less the same: provision domain controller, provision and join SQL Server, join and setup SharePoint Server itself, configure needed services and solutions, optional additions such as Visual Studio and/or other products.

The overall complexity is an incredible high. End-to-end installs might take from several hours to a day or more. Unless the process is scripted and fully automated, there is a fair chance of running into a constant battle of misconfigurations, updates, patching, and troubleshooting. Even then, manual configurations can hardly be scale across the development team.

Therefore, maintenance overheard, configuration drifts and random issues add up and at scale can impact the team, the project or even whole organisation.

Fast forward to Infrastructure as Code

How can this be improved?

To overcome increasingly complex infrastructure setup, provision and configuration pieces can be seen as a set of programmatic or declarative definitions.

Dealing with infrastructure provision via codified configurations called “Infrastructure as code” and considered as a modern approach of scalable, disposable and repeatable infrastructure management.

In other words, shifting from manual, wizard-driven configuration into programmatic or declarative configuration reduces the effort required to create and maintain complex infrastructure setups. The infrastructure builds become immutable and highly predictable.

Improving SharePoint infrastructure provision

With the context of the SharePoint infrastructure setup, every farm provision might be split into several phases:

Prepare binaries: ISO images, installation media, updates, patches, etc.
Prepare Sysprep (generalized) image with pre-installed software
Provision and configure virtual machines
Apply application configurations on newly provisioned virtual machines
Test everything in automated way!

Splitting complex workflow into smaller steps helps to come up with the right tool for the job. Smaller tasks are always a win. They are easier to solve, implement, refactor, automate and test. The end-to-end workflow still exists, but it is seen as am orchestration of a much smaller, replaceable steps.

Now that we see infrastructure setup as a sequence of smaller steps, we can come up with the tools. In our case, after a few years of trials and errors, we organically came up with the following toolchain:

Prepare binaries: custom PowerShell 6 module
Prepare Sysprep (generalized) image: HashiCorp’s Packer
Provision and virtual machines: HashiCorp’s Vagrant + VirtualBox
Apply application configurations: PowerShell + PowerShell DSC
Testing: a combination of Pester, Kitchen CI and Jenkins

Some tools might look unfamiliar, we will dig into them later in this post while explaining design choices.

Design choices: why’s and how’s

One of the challenges while designing a new solution is picking up the right tool(s) for the job. There are tons of tools which can help with infrastructure automation, but we deliberately choose these particular tools. Let’s quickly dig into why’s and how’s in more details.

1/4 — Prepare binaries: custom PowerShell 6 module

Average SharePoint infrastructure setup might require dozens files — ISO images, installation media, patches and updates, language packs. Ideally, we should have a tool which would:

download files
validate checksum, ensure that files were downloaded correctly
structure downloaded files (for example, with folders)
work on Windows, MacOS, and Linux platforms
integrate with CI/CD smoothly
be driven via configuration (such as JSON or YAML)
download ISO and installs for Windows, SharePoint, SQL, Visual Studio

While curl, wget and other download managers can solve some of these problem , we didn’t find a great answer to these needs (it can also mean that we just didn’t find it yet!).

Therefore, we come up with a PowerShell 6 module which downloads over 120 installation files for Windows, SharePoint, SQL, CRM Servers for you. It comes with a command line interface, JSON driven configurations, built-in checksum validation and a few more nice features. Being written in PowerShell 6, it works on Windows and MacOS platforms. Altogether, it provides an enjoyable experience and takes away the complexity of downloading and organising installation media for your SharePoint and Windows environments.

More details on this tool and how to use it can be found here:

While this module does not solve all the problems, at least we won’t have to deal with manual browser-based downloads anymore. An hour or so is back!

2/4 — Prepare Sysprep (generalized) image: HashiCorp’s Packer

When it comes to baking virtual machine images, there might be no better tool than HashiCorp’s Packer.

In a nutshell, Packer provides an easy way to create any type of the machine images, including Sysprep-ed Windows images for Hyper-V, VirtualBox, Azure, Amazon and many others.

No manual steps are involved: Packer embraces modern configuration management and provides an easy way to plug PowerShell, custom script, DSC, Puppet, Chef and other tools while preparing machine images.

More details on why to use Packer and its benefit can be found on the official Packer web site:

In our case, we use Packer to build Sysprep-ed Windows images in highly automated, unattended way. No manual steps are involved whatsoever.

Every image runs really heavy operations which you wouldn't want to run every SharePoint farm build. This includes Windows patching, preparing SQL image, pre-installing SharePoint binaries, patches and language packs, pre-installing Visual Studio 2017. In average, pre-baked image saves 3–5 hours because of pre-installed tooling and software.

Once the image is built, it can be shared on Vagrant Cloud and used by Vagrant to create new virtual machines. This what makes Packer/Vagrant so cool!

See Vagrant Cloud as a NuGet or PowerShell Gallery but for virtual machine images. How cool is this?

Currently, we published dozen images available on Vagrant Cloud. These images focus on several areas: Windows 2016, SQL Server 2016, SharePoint 2016 and Visual Studio 2017.

Windows 2016 images

These images focus on Windows 2016 itself, with pre-installed patched and several PowerShell modules. The aim is to get a pre-patched baseline, a starting point for other images or applications, and eliminate several hours of applying patches and initial Windows server configurations.

win-2016-datacenter-bare — a minimal image, as it comes with ISO
win-2016-datacenter-soe-latest — latest updates as of the release date
win-2016-datacenter-app — same as before, but with .NET 35 feature

SOE stands for “standard operating environment”

Windows + SQL 2016 images

All these images focus on SQL Server 2016. A good thing is that these images are based on the previous fully patched Windows 2016 images. Therefore, we spend time only on SQL Server provisioning, all patches and base PowerShell modules are already pre-installed. A few more hours are back!

Be aware that these images pre-install SQL Server and require SQL image completion after bootstrap.

win-2016-datacenter-app-sql16-sp2 — SQL 2016 with Service Pack 2
win-2016-datacenter-app-sql16-latest— SQL 2016 with Service Pack 2

SharePoint 2016 images

The following images are based on SQL Server 2016 and come with pre-installed SharePoint 2016 RTM, language packs, updates and Visual Studio 2017. Because we don’t have to worry about Windows patching and SQL, we can focus on SharePoint install. Save up to 4–5 hours of boring routines!

These images pre-install SQL Server, hence require SQL image completion after bootstrap. As they come with SharePoint binaries pre-installed, we can start SharePoint farm creation straight away. With Visual Studio 2017 pre-installed, total time saved comes up to a day worth of work.

win-2016-datacenter-sp2016rtm-sql16-vs17 — SharePoint 2016 RTM
win-2016-datacenter-sp2016latest-sql16-vs17 — same, with latest updates

More images variations; Packer template variables

As you can see, the main pillars for SharePoint farm build are Windows Server, SQL Server and SharePoint server itself. Because every product can have patches, updates and customisation, it is nearly impossible to produce and maintain a machine image for every case.

Packer solves this complexity via template parameters. A generic build template can be crafted, and then variables can be used to produce specialised machine images. We leverage this in uplift-packer project to produce mentioned early templates and more variations of Windows 2016, SQL Server, SharePoint and Visual Studio — with different patches, service packs and other tools.

There are much more to Packer, we highly recommend to explore it further.

3/4 — Provision virtual machines: HashiCorp’s Vagrant + VirtualBox

Vagrant is another tool by HashiCorp which comes handy while building development, test or lab virtual machine environment.

It provides an easy way to build and managing virtual machine environments in a single workflow abstracting low-level details of a particular virtualisation platform. Out of the box, Vagrant support Hyper-V, VMware, VirtualBox, Azure, AWS and many other virtualization platforms.

More details on why to use Vagrant and its benefit can be found on the official Vagrant web site:

Why we use Vagrant + VirtualBox

In our case, we decided to use Vagrant + VirtualBox due to

Consistent, clean and easy virtual machine setup
Ability to codify provision configurations
VirtualBox supports Windows/MacOS platforms (!)

That said, our limited research hints that VMWare and Hyper-V is the first choice for SharePoint professionals. Azure comes as the primary cloud offer, and only then comes VirtualBox.

Even then, we went with VirtualBox because it gives a portable environment on Windows/MacOS platform and provides a good level of automation around virtual machines, networks and other programmatic setups.

Additional support for Hyper-V and Azure might come later. Give us a shout if you need these.

Again, similar to Packer, it might be a good idea to explore more Vagrant features and play around with existing Windows Vagrant boxes done and shared by others:

Dealing with repetitive Vagrant configurations

While Vagrant provides an easy way to manage and provision virtual machines, we still need to apply additional automation to turn virtual machines into domain controllers, finish SQL Server image, configure SharePoint farm and so on.

These are combinations of the PowerShell scripts and PowerShell DSC. Because SharePoint farm provision can be seen as a set of smaller steps, we wrote automation to solve the most common tasks:

configuring virtual machine resources (CPU, RAM, network)
creation of the domain controller (dc promo)
joining to a domain controller
completing SQL Server image
creating SharePoint farm
creating a minimal set of SharePoint Service Applications
etc.

We deliberately use PowerShell/DSC to ensure reasonable learning curve of the scripted automation. This is why no Puppet, Chef or Ansible can be seen.

Now that we have these scripted, we saw two ways of bringing these scripts to Vagrantfile:

Including scripts as a git sub-module in the project with Vagrantfile
Crafting Vagrant plugin which would deliver automation scripts

The decision was made to go with Vagrant plugin which would wrap all PowerShell scripts into nice-looking helpers within Vagrantfile.

The plugin is called vagrant-uplift, it provides a simplified configuration of DC, SQL, SharePoint and VS and designed to be used with uplift packer boxes. It is open-source and more details can be found on the GitHub:

uplift-vagrant @ GitHub

At this very points, we’ve got three things ticked! Bear a few more minutes, we are almost there!

[x] 1/4 Prepare binaries: custom PowerShell 6 module
[x] 2/4 Prepare Sysprep (generalized) image: HashiCorp’s Packer
[x] 3/4 Provision virtual machines: HashiCorp’s Vagrant + VirtualBox

The next section covers testing, the approaches we took to automate Packer and Vagrant builds, ensure high-quality setups and virtual machine configurations.

4/4 — Testing: combination of Pester, Kitchen CI and Jenkins

As you can see, not only building infrastructure for SharePoint is hard but also building automation for this can be super-hard as well.

In nutshell, there are two key points to successful testing in this project:

Small, manageable automation pieces which can be tested in isolation
Automated workflows which are run locally and under CI/CD pipelines
Avoid dependency on a particular CI/CD server

Local development and testing

PowerShell functions and DSC can be tested in isolation using Pester and Kitchen CI. At this stage, ad hoc local testing is used with a combination of Vagrant or Docker driver for Kitchen CI or just ad hoc virtual machines and snapshots.

Local Packer and Vagrant builds

Once a positive outcome is achieved, full Packer or Vagrant build can be run. Two crucial bits here are Pester and Invoke-Build build automation.

We tend to have a Pester test suite for most of the PowerShell DSC configurations. That means that once domain join or SharePoint farm provision happens, Pester tests checks if domain join actually happen or needed SharePoint services are up and running. This ensure high quality of the produced images and configurations.

Invoke-Build solves task management and orchestration problem. It provides an easy way to run several tasks in a sequence; build packer image, run vagrant provision, clean up and destroy boxes, etc.

Avoiding dependency on a particular CI/CD server

CI/CD pipelines is a crucial pillar for automation at scale. While testing, hundreds of builds can be run in the background every week. As mentioned early, Invoke-Build provides an entry point, a set of tasks which then can be run locally by people or by any CI/C server on your choice — Azure Pipelines, Jenkins, Bamboo or others.

As an example of this approach, we use a pre-configured Jenkins server with to builds and test mentioned Packer images and Vagrant configurations. it comes as a Vagrant box, and can be run on Windows/MacOS or other platform. Additional automation helps with initial build agent setup, provisions pipelines and view.

It is also open-sourced as a part of Uplift Project and can be found on the GitHub:

uplift-cicd-jenkins2 @ GitHub

Pre-configured Jenkins server for the Uplift Project

Uplift Project summary

This really long post (sorry!) wraps up an overview and structure of the Uplift Project, an attempt to bring consistent Packer/Vagrant workflows and base boxes for SharePoint professionals.

The main goal is to reduce complexity and effort while building development, test and lab SharePoint environments by leveraging existing tooling or providing automation in several areas:

Reduce effort to download, store and manage installation files
Provide Packer templates for Windows, SQL, SharePoint and Visual Studio
Provide ready-to-use Vagrant base boxes
Provide additional automation to streamline Vagrant configurations

These areas are mapped into smaller projects and open-sourced on the GitHub:

uplift-powershell — reusable PowerShell modules; simplified file download
uplift-packer — Packer templates for SharePoint professionals
uplift-vagrant — Vagrant plugin to simplify Windows infrastructure provisioning
uplift-cicd-jenkins2 — ready-to-use Jenkins server and pipelines to build uplift Packer images and Vagrant boxes

Also, we made uplift-contrib repository which contains examples on Vagrant configurations:

uplift-contrib — Vagrant config examples for the Uplift Project

Meanwhile, there are several artefacts which can be used straight away:

Links, shoutouts and inspirations

While the below list cannot be a full guide to learn Packer, Vagrant or other tools, it might give a few ideas on where to get started.

A special thanks to Matt Wrock and Gusztáv Varga for sharing so much! Your public blog posts and GitHub repositories heavily influenced design and decision around the Uplift Project including decision to make it public.

If you like what we did, feel free to share feedback in the comments. We are keen to hear back and see what can be improved.