Programming the Cloud
Over the last 5 years while working at Microsoft and AWS I’ve frequently day-dreamed about the potential of a truly programmable cloud development platform. Of making the managed cloud services provided by platforms like AWS, Azure, GCP and others as easy to use as the APIs in my favorite programming languages and package managers. And ultimately, of deeply empowering application developers to take advantage of the best of the modern cloud platforms.
Just over a year ago, I joined Joe Duffy and Eric Rudder to build Pulumi — a startup creating a new kind of cloud development platform. Today, we’re taking the wraps off of what we’ve been working on, and I couldn’t be more excited!
You can read lots more about Pulumi in Joe’s blog post, or jump in and try it out at https://pulumi.io — but I thought I’d share a few of my thoughts as well on why I’m so excited about what we’ve built, and what Pulumi can enable for cloud developers.
A Programmable Cloud
Today’s cloud platforms offer an amazing and ever-expanding collection of building blocks for developers, managed services that offload operational burden from application development teams, and offer incredible elasticity and pay-as-you-go or even consumption-based pricing. Together these cloud platforms are enabling a new generation of applications to be imagined and delivered. Services like AWS Lambda, Google Kubernetes Engine, Azure Cosmos DB, etc. are building blocks that are becoming part of the platform for almost any new application being developed today.
And yet, taking advantage of these new managed services is still too hard. Application developers are kept at arms-length from the features of the cloud platforms, needing to point and click in consoles, use low level REST APIs or CLIs, or — in the best case — use rather esoteric YAML and JSON documents to define the cloud resources they want to provision.
Back when using the cloud meant provisioning a virtual network and a few VMs, this may have been okay. You created these resources once, and then touched them rarely. Application developers treated them as black boxes, deploying code into a VM with little regard for the details of the cloud platform.
Today’s cloud couldn’t be more different. Application developers provision dozens of granular cloud resources for each service they are building — a NoSQL store, some blob storage, a queue and some timers for background job processing, a variety of serverless or container-based compute components, and little bits of logic to describe how each of these pieces of managed infrastructure will be wired together.
In some specific niches, more developer-friendly tools have started to become available. For serverless applications, where large numbers of granular cloud resources are the norm, tools like Serverless Framework, SAM, Up and many more have become popular — each providing it’s own interpretation of a higher-level way for application developers to reason about the underlying cloud platform and to deploy applications. But in each of these niches, the tools have specialized just for that niche, making it difficult or impossible to use the best capabilities of the platforms when they are needed most. What’s been missing, is a way to treat each of these as just a library, and enable an ecosystem of these libraries on top of a unified programming model, such that developers can mix and match the best features of each library for their needs, and mix high-level abstractions where they make sense with low level details where they are necessary.
A Simple Example
A quick example of what I’m talking about here. As an application developer, I see a diagram like below and think to myself — “looks like that should be about 20 lines of code!”.
It should create a bucket, a containerized task, and hook up event handlers on the bucket that kick off the containerized task. And if I wanted to run something like this as a single process on my laptop, watching my filesystem and launching a process (or container) — it would indeed be about 20 lines of code!
But what happens when I try to bring this to the cloud? The good news is there are great managed services I get to build on top of, all of which are robust, effectively infinitely scalable, consumption-priced, and available in dozens of regions around the globe. For example, in AWS, I can combine S3, Lambda and Fargate (serverless containers) to build this application. The bad news is that each of these services requires configuring a half dozen discrete cloud resources, each with complex configuration. In fact, building the application above requires deploying about 25 different cloud resources in AWS. Specifying the configuration for these resources means copy-pasting a dozen pages of esoteric YAML or JSON, tweaking things until the pieces all wire up correctly. And then, after all of that — writing the few lines of actual application logic that are needed.
But it doesn’t have to be like that. We can have a more programmable cloud. We can enable developers to turn a diagram like this into a piece of software at the level of abstraction they expect. We can create new kinds of components on top of the amazing building block services in the cloud so that we don’t have to constantly copy-paste to build our cloud software. And we can ultimately empower developers to use the best of the cloud to build applications.
Pulumi
Pulumi is a programming model for the cloud, and a programming model imagined from the ground up for truly cloud native application development.
While traditional software can run “in” the cloud, Pulumi programs run “on” the cloud. Pulumi programs manage both their cloud infrastructure and the application logic that will run on and between those managed services.
As a programming model, Pulumi at it’s core enables developers to imagine new abstractions and components. To write functions and classes, to publish packages, and to build on an ecosystem of cloud components built by other developers.
Here’s an example:
Here we use JavaScript to define our cloud application (Pulumi supports Python, Go and TypeScript as well), and we use an open-source package cloud-aws from NPM which offers some handy high-level components for building modern cloud applications on AWS (Pulumi supports Azure, GCP and Kubernetes as well).
This example uses a custom component called cloud.API which exposes an HTTP API to the internet. We use .static to serve some static resources at the root, and .get to serve a REST API with some simple application logic. And finally, we expose this API and export an internet-facing URL.
Under the hood, cloud.API provisions many AWS resources — an API Gateway, an S3 Bucket and some S3 Objects, a Lambda, along with the supporting Roles and Policies. But Pulumi also deploys application code — the implementation of the Lambda to return the string “AWS”.
Ultimately, this doesn’t look much different than a simple Node HTTP program (perhaps using Express.js). But instead of running as a single process on a single VM somewhere (and leaving the responsibility to deploy, manage, scale, load balance, service, etc. to the developer), the code above can be deployed in seconds and will be running as a managed, scalable, and extremely low cost service on the internet.
Just as importantly, if I want to mix in some low level AWS details, I can do that. The cloud-aws package is itself just built on top of AWS primitives, and I can add my own as directly as well — a DynamoDB table or an Elasticache managed Redis instance. I can run a containerized Service. Or I can even spin up an EC2 network and VM and specifying the precise networking configuration for my application.
But that’s just one very simple example. You can use Pulumi for an amazing variety of cloud programming applications. Want to run containerized and scale-out Airflow on AWS? Or stand up an Azure Kubernetes Service cluster?Perhaps deploy Jenkins on Kubernetes? Or serve a static site using S3, Route53 and CloudFront? Or create a new AWS virtual network with just new awsinfra.Network(...)? All of these and more are easy to express with Pulumi.
Returning to the simple example from earlier —here’s what that looks like in Pulumi. Note that each piece of code here is 1:1 with one of the concepts in the plain-english description and the architecture diagram — a bucket, a containerized Task, and two event handlers that wire these up. This is enabled by combining low-level infrastructure, high-level components, and the ability to write pieces of runtime glue as callbacks within our program. Notably though, we can still include whatever application logic we want into this deployment, as part of the container provided to the build of the cloud.Task. Turning a high-level description of a cloud application into a program that can be deployed directly to the cloud can be as simple as this across a wide variety of applications.
With this application deployed, you can update the application code in the container, provision new managed services, or adjust the wiring between the existing components as simply as a pulumi update. And you can get aggregate logs, deep links to the underlying AWS resources and insight into the history of deployments using the pulumi CLI and https://app.pulumi.com console. More details and a walkthrough of this example are available in a great blog post from Donna Malayeri.
Summary
Pulumi is a new open source development platform for the cloud, offering a fun and empowering new way to build cloud software. With today’s releases, we’re just getting started — and we’d love you to check it out and join us on the journey.
