Pulumi — first impressions

Pulumi, which came out of stealth mode yesterday, is an interesting technology. It allows someone to describe their cloud application stack — on any cloud — using general-purpose programming languages. So it’s like CloudFormation, but not just for AWS. It’s like Terraform, but instead of using a Domain-Specific Language (in which you may not have primitives for iteration and abstraction) you use Javascript, Typescript, Python, Go, and potentially others.

Pulumi’s offering consists of a a set of command-line tools for Linux/Mac/Windows, which take your code + metadata + configuration and give it to their SaaS (at the core of which is a language-agnostic engine, open sourced as of yesterday and freely hosted for now, though clearly part of the monetization strategy). The engine runs Pulumi applications using language-specific runtimes and comes up with a schedule of operations (e.g., create bucket, run VM, update a Lambda function, etc) for modifying cloud resources. If the schedule looks good to you, the engine executes it in your cloud account via cloud-specific resource providers. As you change your code and redeploy it, the engine calculates the smallest set of cloud operations needed to adjust the state in the cloud to the new version. From an OO perspective, objects in the language runtime are mapped to cloud entities (via names).

There are three main points of extensibility (which will blow up their QA matrix, but what are you gonna do). One can add support for:

1. New languages. The team originally thought about creating a new general-purpose language, but then realized that it would be hard, unnecessary, and limiting to the appeal of the system.
2. New clouds. They started with AWS, Azure, GCP, and one of the early testers added Kubernetes support.
3. New abstractions. For example, their cloud-specific providers are expected to expose all features of any given cloud, but their multi-cloud provider leverages the lower-level providers to offer abstractions that should work on multiple clouds (though only on AWS for now). Such is the power of an approach based on general-purpose object-oriented languages.

Pulumi is well positioned to eliminate syntactic differences in how similar services on different clouds are used. At a first glance it’s strictly about cloud deployment, but it helps with developing, building, and testing, too.

• In the pure serverless world there isn’t much building, you push code into the cloud and it runs. Which Pulumi, with support for deployment versioning, can accomplish: change a Lambda function embedded in your Pulumi app, do ‘pulumi update’, and you’ve deployed a new version of your serverless app. (So this is competitive with the Serverless Framework et al.)
• For containerized applications, Pulumi will build and deploy an image from a Dockerfile, as shown in the cloud-js-thumbnailer example, potentially simplifying an error-prone build pipeline.
• An explicit set of update steps, which are consistently logged, which can be tried out on a test copy of the infrastructure or reversed in production, are supposed to help with testing and diagnostics. I believe that.
• I am guessing here, but extensibility of the system — both on the development side and the deployment side — tells me that additional machinery for building, packaging, and testing, particularly of non-serverless applications, even VM-based dinosaurs, can be added or integrated in later.
• And to really go on a limb, I think down the road Pulumi can be used not only to eliminate uninteresting differences in how similar cloud services are configured, but in how similar cloud services with I/O — persistence and messaging (the hard ones!) — are invoked. There are interfaces for a generic Table and Bucket in the multi-cloud provider, and their implementations for AWS. These can be used by Pulumi code, but not by the “business logic,” the stuff in your Lambdas or your containers. It would be a shame if all those abstractions couldn’t be reused! And while I get the concerns about reducing rich functionality of clouds to the lowest common denominator, having layers of abstractions lets us to have the cake and eat it, too. Developers can use cloud-specific APIs when needed and higher-level cloud-agnostic APIs when that’s adequate.

Philosophically, this technology obliterates the last traces of the dev/ops divide, for better and for worse. It brings cloud infrastructure into your code the way system libraries bring operating system details into your code. And on top of that, with language facilities for abstraction, some of the low-level details can be hidden, in the name of portability, in the same way that Java and its successors tried hiding details specific to operating systems. I suspect that, as with JRE and friends, high-level abstractions will see mixed success as the unique and powerful features of underlying cloud services will remain attractive to a large subset of developers. But it’s an interesting approach and I look forward to seeing where it takes us.

For individual apps, serverless platforms like App Engine, AppScale, and Heroku don’t need a sophisticated deployment language: they provision cloud resources automatically, with things like load balancers, API Gateways, and Auto Scaling groups magically present under the covers. As deployments get more sophisticated, though, spanning multiple apps and utilizing an array of cloud resources shared by the apps, a deployment tool like Pulumi will come in handy in the PaaS world, too.

For more information:

• A good introductory blog post from Pulumi’s CEO, Joe Duffy
• Another good first blog post from Pulumi’s CTO, Luke Hoban
• Serverless tooling comparison presentation by Pulumi’s Product and Community Manager, Donna Malayeri
• For technical details and a walk-through of the code repositories, see this 40-min presentation from Joe

Originally published at blog.appscale.com.