Introducing CloudEvents 0.1

Last week at CloudNativeCon & KubeCon, a specification was announced that aims to help people and applications handle event data, called CloudEvents.

CloudEvents’ scope is simple: provide a consistent set of metadata which you can include with event data to make events easier to work with for publishers, middleware, subscribers & applications in general. In short, it’s a standard event envelope.

Here’s what version 0.1 of the spec looks like in an example event represented in JSON:

An example CloudEvent v0.1 for when an object is created in AWS S3

The scope of CloudEvents is pretty humble, however many believe it has big potential. It’s being incubated within the CNCF and an increasing number of industry stakeholders have been actively contributing to the project since January. These include Google, Microsoft, IBM, VMware, SAP, Oracle, Huawei, Alibaba, Red Hat, Iguazio & more. Further, after version 0.1 of the specification was announced last week, Microsoft announced their immediate support for CloudEvents natively for all events in Azure, via Azure Event Grid. Other vendors have started incorporating it just as quickly.

What’s the big deal? Why is this important and why is everyone getting involved?

I can’t speak for everyone, but as a developer who helps contribute to the effort, I can chime in on why I’m excited about CloudEvents and why it can raise the tide.

The Serverless Movement

In 2015, I worked on an open-source project called the Serverless Framework, which helps the world build entire applications on new serverless services, like AWS Lambda. These serverless applications are easier to make, easier to maintain and incredibly efficient overall. If you want to move fast, increase innovation and reduce overhead, there is no better option right now than the serverless application.

Since 2015, I’ve witnessed many developers and orgs use serverless architectures to deliver incredible amounts of amazing work. At the same time, I’ve seen many struggle with event-driven design, which is inherent in serverless architectures since FaaS functions are event-driven.

For example, developers have a hard time discovering events they can work with, understanding event schemas, mocking events for development, testing events and event-driven workflows, observing event-driven workflows, tracing events as they are transported across environments, evolving event data safely, correlating events, transporting events via common protocols, ensuring event delivery, and the list goes on…

What’s most frustrating is event-driven design and serverless technologies enable developers to reach unprecedented levels of productivity, yet our tooling isn’t designed to help us manage this new level of output. We can each provision a thousand AWS Lambda functions right now, which will scale massively and cost nothing until they’re executed. This is incredible power, but I don’t believe we are ready to reason about this new level of output. That is, until we have better tools.

This is a problem which CloudEvents can solve. Consistent event metadata creates a stable foundation on top of which we can build better tools for developing, running and operating serverless & event-driven architectures.

Our Serverless & Event-Driven Future

Events have been around for ages, but our future will contain more events and event-driven applications than ever before.

The digital world is storming into our human reality. IoT devices and sensors are proliferating, monitoring us and our environments. At the same time, our applications are becoming more distributed and the quantity of our integrations is growing. As a result, we’re increasingly publishing events, leveraging event-driven design patterns and transporting events across environments.

Meanwhile, serverless computing (originally known as event-driven computing) can process events at scale on a pay-per-execution basis. It has never been easier to write code to react to anything that can possibly happen. This is a powerful incentive to create more event-driven applications.

Suddenly, the world is richer with events than ever, awaiting developers with vision to put them to productive use.

Unfortunately, as the quantity of events and event-driven integrations increase, there still is no consistency in how publishers describe events. Developers looking to easily work with event data have to first learn platform-specific terms and semantics. Some platforms even format events differently across their own services (*cough* AWS *cough*). Meanwhile, the ability to transport events is hindered as logic and infrastructure have no consistent information which they can use to make smart decisions to handle and route events. Future versions of CloudEvents will focus on routing and tracing events, making event-driven patterns and integrations easier.

Lock-in, Gravity & Vendor Choice

Lock-in is a key concern in serverless architectures. A common question in the serverless community is how to make serverless logic portable.

I’m not convinced we should prioritize moving serverless logic around. The economics of serverless computing change the conversation. Serverless logic can reside everywhere with little overhead and cost. Serverless functions can be provisioned simultaneously across regions and providers. Events can now be consumed by one function, or multiple functions at the same time, more easily than ever. I favor offering developers a uniform experience for provisioning and managing the lifecycle of serverless logic, over making it portable.

Meanwhile, our serverless lock-in concerns should be directed at the gravity of data.

Events bring anti-gravity to data. Our applications currently focus on collecting and protecting static data “at rest”, but this doesn’t allow apps to be as agile and extensible as we’d like. If our architectures prioritize it, they can first keep data “in flight” in the form of events, and leverage serverless functions to consume events easily and concurrently, to deliver valuable outcomes.

CloudEvents’ common metadata makes events easier to route, fan out, trace, replay and generally keep “in flight”. They are more portable, more liquid, easier to transport across environments. We also have specs for mapping CloudEvents metadata into existing protocols (HTTP, AMQP, MQTT, and HTTP WebHooks are in PRs) as appropriate for each protocol. Network bandwidth, cost and latency are still major challenges. But CloudEvents simple metadata can bring anti-gravity to data in a lot of use-cases today.

Lastly, is the increasing desire for vendor choice. Serverless compute has always been only half of the value. The other half of the value comes from the other services platforms provide which developers can use to deliver innovative outcomes with their serverless functions. We see this every day from the vantage point of the Serverless Framework. The vast majority of serverless applications built on AWS Lambda use other serverless services AWS provides, like S3, DynamoDB, Kinesis or Rekognition. Who cares about your serverless compute service. Developers and businesses need other serverless services that offer easy, high-level abstractions to help them innovate. I expect cloud providers to offer significantly more serverless services in the near future. Vendors (especially nascent ones) have an opportunity in serverless, if they decrease emphasis on trying to copy each other’s services, and instead innovate new serverless services that offer unique, best-in-class, value propositions. With CloudEvents plus serverless and event-driven architectures, developers can leisurely distribute data across vendors and clouds to take advantage of the unique value they provide. CloudEvents helps level the playing field.

At CloudNativeCon, I did a live demo of a massive multi-cloud scenario using CloudEvents. The demonstration involved integrating 11 different serverless compute providers, which to their credit, all contributed to the development of the demo. The scenario was completely over the top, but it showed what’s possible.

In the architecture, a CloudEvent is published from AWS S3 and another from Azure Blob Storage. Both CloudEvents are then distributed via new types of event routers to 11 serverless functions across 11 different vendors. Each function used a proprietary service their respective platform offers to perform an action based on the event to create an valuable outcome. The results were then logged to a Twitter feed. It was incredible.

Video of the demo at CloudNativeCon 2018

Goodbye API Gateway. Hello Event Gateway.

CloudEvents’ consistent metadata can be generically tracked, routed, and dispatched by infrastructure, which gives us the opportunity to build smarter pipes.

Microsoft’s Event Grid is one of the first to provide a new take on an event router with a serverless focus, and it can especially help build event-driven and serverless applications on the Azure platform.

The other pioneer in the space is an open-source project which my company has been building for over a year. It’s called the Event Gateway and it uses CloudEvents as the default event format.

CloudEvents envelopes are one-way letters. Its metadata enables the creation of complex workflows, which can include request/response. So we designed the Event Gateway to offer both event-driven (pub/sub, async) and request-driven (request/response) flows.

You can send custom CloudEvents into the Gateway and route them to multiple serverless functions in a pub/sub pattern. You can also expose HTTP endpoints and receive HTTP requests (which the Gateway converts to HTTP CloudEvents) and respond to them synchronously with serverless functions. Plus, the Gateway makes it easy to integrate with multiple serverless compute options as well as other sources and sinks, making it the ultimate event-driven integration infrastructure, for the serverless and multi-cloud era.

You can use the Event Gateway today via our recently released Hosted Event Gateway Beta. You can also deploy the open-source project. Kelsey Hightower gave a great demo at CloudNativeCon on running the Event Gateway on Kubernetes.

There is a clear opportunity for new event-driven middleware to help us build the next generation of applications and integrations. More importantly, both of these new event routers are clear examples of what CloudEvents’ simple metadata helps enable. As the CloudEvents specification advances, so will the tooling and ecosystem around it.

Fostering Serverless & Event-Driven Community

Of course, this is just the beginning of CloudEvents. It’s remarkable the effort has such significant industry support already.

More importantly, CloudEvents must be owned by the community in order for it to be successful, which is why it was important to hand over to the CNCF. The CNCF provides a neutral ground where industry titans and startups alike can collaborate on solutions for the greater good. The CNCF is also exceptional at fostering community. Dan Kohn, Kelsey Hightower, Chris Aniszczyk & more have done a tremendous job there. Now, as we move higher up the stack and shift the focus of software development tools from infrastructure to outcomes, I have only one question for them... When will the Serverless Foundation be established? 😏

Ultimately, this is not about vendors or the CNCF. What matters most is that we continue to make things simpler, more accessible, and democratize the power to create software, regardless of location, gender, race, age, background, resources, power, etc.

Serverless, CloudEvents and community can do this.

Selfie from CloudNativeCon & KubeCon featuring featuring myself and CloudEvents contributors from Google, VMware, IBM, Red Hat, Huawei, Oracle, Microsoft, SAP & Iguazio, all working together to make serverless/event-driven interoperability easy.

Shout-out to the great CloudEvents contributors. Dream team.

Check out the CloudEvents spec for more info and ways to contribute. We’d love to start seeing some end users from various verticals involved. Also this Twitter thread by Clemens Vasters rocks.