Config Management in a VFX Studio- SaltStack

Is your VFX studio still employing raw SSH to manage your infrastructure? Maybe you still deploy code manually with a collection of hacked-together, cowboy-coded shell scripts using rsync and — dare I say — FTP? Luckily, the year is 2015 and there are many modern, well-supported solutions to choose from for managing your infrastructure.

Using a popular, well-tested and well-documented solution has many advantages — security being one of them. With vulnerabilities like Shellshock and Heartbleed in the wild, you don’t want your studio to end up like Sony did in 2014 with the Sony Pictures hack.

It seems lately a lot of visual effects studios are falling behind in their backend and pipeline infrastructure. Why is this the case?

• Small studios don’t have the engineering talent or resources to build up the necessary infrastructure, while large facilities have numerous home-grown tools that are in desperate need of updating.
• Studios are either afraid of implementing new technology or just don’t care enough about innovation. They often live by the mantra, “if it ain’t broke, don’t fix it.”
• Old technology is difficult to scale in a large environment. Having worked in both large and small facilities, I’ve written numerous scripts around SSH and Fabriq. Although they can do the trick, they tend to be slow, they need to be constantly maintained by someone and they do not scale well.

There are a lot of better tools out there and this post will explore which Configuration Management (CM) tool is the best fit for a visual effects studio workflow.

Why is having a configuration management and remote executing infrastructure important within a studio environment?

Adopting the notion of “Infrastructure as Code” is key in creating a scalable and efficient studio pipeline.

Configuration management tools are designed to automate the application of configuration states. Having to deal with hundreds of render nodes and numerous VMs — quite often in remote data centers — one can easily spend hours updating and deploying apps across the studio infrastructure.

When choosing a configuration management solution we looked at the following requirements:

1. It needs to manage server configurations, and serve as a remote execution and deployment system.
2. Simplicity and ease of use. Code should be as simple as possible, and the learning curve for developers should be small.
3. Python Based (very important in the visual effects industry)
4. Multi-OS Support (Linux, OSX, Windows)
5. Supports multiple cloud platforms and hybrid infrastructures
6. Supports global facility and distributed setup. If the connections between studios fail, the setup needs to work.
7. Works well with continuous integration tools such as Jenkins.

Types of CM tools

There are several CM tools available, but we looked at the following choices: CFEngine, Ansible, Chef, Puppet and SaltStack (Salt) to review.

CFEngine

CFEngine began development in 1993 making it one of the first configuration management solutions. It’s an industry veteran that’s been around for decades — and it is written in C — which makes it very fast and stable.

The software is open source, however, you need to be a competent C developer in order to make changes. Some other drawbacks include a steep learning curve and lacking extensive community support when compared to the other tools.

I personally have limited experience with CFEngine, and in general, it is difficult to find easily accessible information about it within the tech community. It’s also unclear how it performs at scale and within a cloud environment. Large companies are using it in the wild — LinkedIn, for example, runs its infrastructure on top of CFEngine and SaltStack.

Puppet

Puppet is Ruby based and has a large community behind it.

It’s the most complete in terms of modules and user interfaces, which are built with Puppet-specific language.

Having spent time setting up Puppet infrastructure at Method Studios, I can say that it’s not as streamlined as Salt — and the configurations can get very messy, very quickly.

Other drawbacks include:

• The Ruby manifest execution order can be unpredictable.
• Puppet manifests are executed in a declarative manner that can jump around the manifest. If not done properly, this can lead to a headache when trying to troubleshoot manifests.
• For a Python developer, learning Ruby and the Puppet DSL can be difficult.

Chef

Chef is also Ruby-based and similar to Puppet. It’s very popular these days — Disney Interactive is using it, and they love it. It integrates well within the Amazon AWS, Microsoft Azure and Rackspace cloud environments. Chef has a lot of community support behind it, and you will have no trouble finding information and examples on the web. The drawback for Python developers, again, is having to learn Ruby.

Ansible

Ansible performs all functions over SSH, which is one of its biggest strengths — yet also one of its weaknesses when it comes to scaling. It’s a minimal tool — designed to be very secure — and it doesn’t require agent software to be running on any remote machines.

Speed is not it’s strong point, but if you’re looking for something that doesn’t need to scale to thousands of machines, you don’t require strong Windows support, and you don’t want to be managing a client, Ansible may be the right tool for the job. It’s very popular, has a great community and can be a great tool for a VFX workflow.

SaltStack

Salt is a configuration management solution and remote execution system that stands out from the other tools mentioned.

There are several positives about Salt. These include:

• Open source software and one of the leading projects on GitHub
• Multi-platform
• Python-based
• Highly scalable

It offers a large number of pre-built modules to address specific software, operating system, and cloud architectures — also, custom modules can be written in Python.

Drawbacks include:

• A steep learning curve for a beginner
• Documentation is lacking clear examples
• Being a new tool on the market, there are a significant amount of bugs
• Having to manage a salt-minion client adds a level of complexity

Why Do We Like SaltStack?

SaltStack shares a lot of the same concepts as the other CM tools. Even though the tools are similar, Salt allows you to have more control with less of the headache.

The fact it’s Python-based was one of the key reasons we swayed towards it. In general, Salt is easier to pick up and troubleshoot should problems arise.

Also, Salt “state files” are plain text files written in YAML — a superset of JSON. This makes them clear, structured, and the syntax can be picked up rather quickly. It also provides top-down execution order configurations, which can simplify debugging. And, it has a convenient Python client API.

While Salt is our primary CM choice, it can have a steep learning curve — the training manual is dense and has an unorganized layout making it difficult for newcomers to pick up. But, once you pass the initial setup, Salt is manageable.

Perhaps one of the most compelling aspects of SaltStack is its vibrant community. Having used a number of different configuration management systems across a wide variety of environments, I’ve yet to find a solution that’s as rapid to deploy, simple to scale, or as well architected as Salt.

The Best Tool For The Job

We’ve listed various CM solutions for your infrastructure. We personally use Salt and find it to be highly scalable, extendable and well supported, making it the best choice for our company. For the VFX studio just getting started — or looking to upgrade to a modern, well-supported tool — I’d suggest starting out with Salt or Ansible.

At the end of the day, however, all of these configuration management tools can get the job done. The particular tool you choose may depend on your organization’s existing infrastructure and toolset. If, for instance, you’re running Rails and have Ruby developers on staff, looking into Chef or Puppet is a no-brainer.

As a general rule, no single tool can be effective on its own — your chosen CM solution should be integrated into a comprehensive DevOps toolchain that is built with precise tools in alignment. Only then can you know that your CM system is optimized for efficiency.

Now, it’s your turn — tell us what your CM system plan is for your VFX studio?