Platform Engineering: Creating your Internal Developer Platform (Part 2)

In the first part of this article I gave an introduction to platform engineering and explained what an internal developer platform is. In this article I will explain how you can create your own using a variety of open source and commercial tools.

Open Source tools

There is no single open source tool that will give you an entire internal developer platform (IDP), but you can if you wish aim to create your own by combining multiple tools like ArgoCD and Crossplane to manage your kubernetes workloads and infrastructure, with Backstage acting as your service catalog. This section assumes you would be using a CI tool (preferably gitlab or anything with a simple language), kubernetes, git repositories, observability tools, etc.

Backstage

For your Service Catalog you would use Spotify’s backstage:

• Backstage is actively being developed by Spotify and a strong community behind it, so it has virtually unlimited backing at the moment.
• You can browse any service and configure it to give you additional observability via plugins.
• You can template services with it. You can use it to create all the components of a new service, template code, configuration, new repository, pipeline, etc.
• It allows you to define documentation and golden paths with it.

ArgoCD

ArgoCD is a gitops tool used to dynamically deploy your services to Kubernetes, it does this by reconciling the state in your cluster with your repository in real time and more importantly, without needing to grant access to the cluster to an external CI tool. ArgoCD has the following features that make it suitable for your platform:

• It has a great UI; relatively easy to understand and it allows you to trigger deployments and roll them back
• It allows you to define guardrails on projects with RBAC, such as defining what certain users can do, what type of resources they can apply etc.
• Create groups and assign users different levels of permissions to do various tasks.
• You can also do all your deployments in the cluster without ever needing access to kubectl. Effectively abstracting the developer away completely from using kubernetes. Every time you change code and commit it, argocd is there to deploy your changes.

Crossplane

To enable your developers to provision infrastructure you would use Crossplane. Crossplane is a component you can install in your kubernetes cluster that enables you to manage any kind of infrastructure or anything with an API. With this tool you give a way for your developers to provision infrastructure with simple yaml files.

• Crossplane is a set of Kubernetes operators that extend kubernetes with the ability to provision and manage cloud infrastructure, services, and applications.
• Operation engineers can create composite resources that simplify and define how the developers can provision infrastructure. This provides abstraction and guardrailing.
• Crossplane can be used to generate infrastructure with pretty much everything that has an API as long you create a provider for it. There are already providers for all major cloud platforms like GCP or AWS that you can use out of the box.
• Crossplane watches your infrastructure and ensures it stays as you defined it.
• It makes creating infrastructure a lot simpler than using plain terraform.

Application Management

This IDP I have just described is missing one key element that would aid abstraction and ease of use for developers. With the above setup you can automate most things and template kubernetes or Helm files. The problem with this is that our developers will still need to work with kubernetes yaml files. Sure, our service catalogue can do this for us to a certain extent with good templating, but isn’t the aim of a platform to abstract away complexity and hence not having to deal with kubernetes as well?

Open Application Model (OAM) and Kubevela

Open application model was designed to abstract kubernetes applications, meaning that a developer would only need to provide a simple yaml configuration based on whatever the platform team defines and the resources would be created according to that definition. This means that instead of having to deal with yamls for services, ingress, deployments, etc you just create a simple yaml file that does it all for you behind the scenes.

Originally OAM was implemented with Crossplane but later it became a standalone product called Kubevela, created and actively maintained by Alibaba and Microsoft. Apparently the original plan was to integrate this with crossplane but they had to become an standalone product according to a statement made in the Crossplane Slack by one of the maintainers of the project:

Kubevela offers a platform framework to abstract kubernetes applications in an opinionated way. In that sense Kubevela would be more flexible than using more sleek, commercial tools than Shipa, but perhaps not as production ready yet as it is still relatively new.

It is still early days to say whether OAM will become the standard to abstract applications away or something else will take over, but right now the only contender I could find is PAWS, and there are no open source implementations of it, and even the implementation in Humanitec who created the standard is still green — time will tell. So far it seems only a handful of companies including creators Alibaba and Microsoft are giving their support and contribution to the OAM standard.

Either way, kubevela may be worth considering if you are creating your own platform with open source tools, but I would leave it as a nice to have part of your implementation at the end.

Ketch

Ketch is another tool you can use to abstract kubernetes applications away, it is an open source version of Shipa. Ketch works by mapping frameworks to namespaces and packing your applications and deploying them to this framework based on what you define. It doesn’t seem to be as feature rich or flexible as kubevela but if it neatly fits your workflows it may also be a better option.

The downside with implementing your IDP with open source tools is obviously the lack of support and also the additional work involved. All of the tools described above require extensive configuration and potentially months of work until you are at a place where you can run with them. They also may have a higher learning curve than paid tools

Commercial tools

In preparation for writing this blog I did a small foray into a few commercial platform tools. What follows is just my first impressions of the tools and the companies behind them, just bear in mind that I didn’t have a chance to test these extensively in a working environment with hundreds of people using them so results may vary. Attempts were made to reach all vendors to clarify questions I had, and ask for a demo, details will be provided below.

Tools are in alphabetical order

Humanitec

I don’t think any other company has done more than Humanitec to define, evangelize and expand the term “internal development platform” and platform engineering in general. Even though they obviously have a vested interest in seeing this market expand, they have done their best to keep their resources as agnostic as possible and provide an outlet to showcase competing products and opinions as well.

If you want to try their tool they generously give you a 45 day trial with plenty of documentation for you to toy around with it.

Humanitec markets itself as a platform orchestrator, and you can use it to choreograph all layers of a deployment, from application to infrastructure. Humanitec uses PAWS to define abstract workloads and infrastructure from developers. Some parts are defined by the platform teams, whereas others are defined by developers as shown in the graph below. Humanitec can orchestrate all layers of your platform for you and it is to be used in conjunction with a service catalog and other GitOps and IaC tools.

What I liked about it:

• Really good team behind it, they genuinely seemed eager to answer questions and show me the product, even their CEO Kaspar von Grünberg was kind enough to have a call and correspond with me. In total I had a chat with four people from the team.
• The tool allows you to orchestrate and abstract both kubernetes services and infrastructure with the use of dynamic configuration management and PAWS in a way that’s quite unique and comprehensive.
• Easy to get access to a demo.
• Lots of information and blogs explaining IDP and platform engineering as well as an extensive study of its benefits.

What I liked less about it:

• The UI is not the most beautiful or intuitive. Had to follow a tutorial to get my bearings. That being said, it seems they are working hard to make it better and have taken my feedback onboard. They also mentioned their ultimate goal is to be a component of an IDP, hence UI is not their most important feature, they rather yield that to something like backstage.
• Documentation and definitions are currently out of date so it can be confusing. For example there is a lot of talk about PAWS and dynamic configuration management in their blogs but there is a palpable lack of practical documentation on how to do this with their tool. I’ve mentioned this to them and they admit they are a bit behind but working hard to improve this in the near future.

Mia Platform

Mia platform markets itself as an end-to-end hub for development and operations. The product itself is a UI first abstraction of kubernetes and it is very sleek and easy to use, with many additional features that a service catalogue would offer, so in that sense, it is the closest you can get to a standalone internal development platform.

What I liked about it:

• It is older and feels more mature than the other tools mentioned here.
• The UI is clean and easy to navigate, which makes sense because it is an UI first product.
• If your developers don’t want to touch configuration files for kubernetes and do all of this via the UI, Mia platform is phenomenal as it works very well.
• It provides a public and an internal marketplace to either create new template services or find one that suits your needs created by another company.
• All the configuration changes in the UI are written to a repository where you can find all kubernetes yamls neatly organized.
• It is a very good option for enterprise customers due relative ease of implementation.
• They have a variety of options to install their product, self-managed, PaaS and hybrid.
• The team comprehensive and helpful to demoing the tool for me.
• It works like an internal PaaS with additional IDP features.
• In the near future they will offer backstage integrations as well as packaging applications in Helm.
• Some open source initiatives like kube-green, micro-lc and Rond

What I liked less about it:

• Not as easy to get your hands on to try the platform as other tools described here.
• Being an UI first product it has some downsides:
• Not all kubernetes resources and features are supported on the UI, to use other resources you need to push yaml files to the repository and these will get applied via gitops model.
• If you change any of the files created with the UI and push to the repo, these get overruled by the UI. The workaround is to edit a json config file that holds all the yaml configuration stored by the UI. While this is a solution, it isn’t very elegant if you want to follow a GitOps model and not use the UI that much.
• They don’t have a way to define applications and resources using something like PAWS or OAM, therefore your developers still need to understand the underlying mechanisms of kubernetes to a certain extent and there is no way for a platform team to abstract away complexity and define something like composite resources.
• Currently they don’t have a way to create resources outside of Kubernetes, it is very much a kubernetes abstraction so far. But perhaps in the future they will have crossplane integrations or other methods to do this.
• Mia Platform seems to be a bit of an island compared to other competing or complementary tools out there. You don’t see much of their opinions or contributions to the open source space or platform conferences. That’s not to say they don’t open source anything, it will just be nicer to hear from them more on platform conferences and around because I think they have a great product.

Opslevel

Opslevel is a service catalog similar to backstage

What I liked about it:

• UI is nice, intuitive and clean
• You have ways to define the quality of an application based on your custom criteria.
• The team behind it was kind enough to give us a demo and show us the product.
• It is quicker to get started than backstage.

What I liked less about it:

• Plugins are not community driven like backstage.
• Fewer features than backstage, for example templates are still experimental.

Port

The Israel based tool Port is a Service Catalog, much like Backstage

What I liked about it

• Easy to navigate UI.
• Relatively quick to get started as it is a SaaS offering and designed to be easier to use.
• Very easy to add blueprints or any kind of resource, you just need to add simple json configuration via the API or UI and you are done.
• It is very extensible and can be integrated with pretty much anything. You can define any action you want and then you trigger whatever tool can perform that action.
• Phenomenal support. I had a chat with the CEO Zohar and he was extremely helpful answering all my questions and demoing the tool for me. I have no doubt that he would be very proactive and hands-on assisting his customers to achieve their goals.
• You can define RBAC for your users.
• You can template microservices
• In the future you will also be able to use Backstage plugins with it.

What I liked less about it:

• No on-premise version of the tool yet. (But it’s coming)
• Even though you can pretty much integrate Port with anything, the integrations are almost entirely your responsibility as of the writing of this article, usually done by sending events to a queue and then you handle those events yourself. That being said, there are a few exporters that work directly with Kubernetes, github and terraform that help with data fetching.
• This will change once you can use backstage plugins with it, but right now you don’t have much of an observability or documentation pane in Port, you essentially get links to go somewhere else to get it.

Shipa

Shipa describes itself as an application management framework and it is essentially an abstraction for kubernetes applications. Developers don’t need to see or edit yamls, shipa does all the work of packaging the application and delivering it to the cluster.

What I liked about it:

• It’s a sleek product and it looks trivial to use for developers.
• Support was helpful to answer questions even though the time difference made it hard to find common hours to speak (I assume this would not be a limitation if you paid for the product).
• You can easily get a demo to play with, though you need to provide your own cluster.
• Easy to get started and run with it.
• Shipa is open sourced as Ketch

What I liked less about it:

• It works by mapping frameworks to namespaces and it may be a bit too opinionated about how your applications will be deployed. If your application requires various kubernetes resources that don’t fit Shipa’s deployment model, it may not be the right tool for you.

Upbound

Upbound is a managed crossplane offering and as such you use it to create control planes of any kind, which can be used to create any resources you want, as long as they have an API. There are many providers created for all major platforms, hence, it is a good choice to manage your infrastructure and your kubernetes workloads.The fact that crossplane is open source and providers are community driven makes this platform particularly appealing.

Upbound’s aim is to be the platform for building internal cloud platforms, so if your aim is to be able to provision resources in multiple cloud providers from one place and do it with simple yaml, upbound excels at it.

What I liked about it:

• Free demo, easy access.
• Good team behind it, endorsement by Kelsey Hightower makes it more appealing.
• Helpful to answer my questions.
• The UI of the platform is clean and easy to navigate.
• Crossplane is open source.

What I liked less about it:

• Many of the commands on the documentation didn’t exist in the tool to manage the cluster. For example, token authentication didn’t work for me. They acknowledged this issue, so this should be fixed soon (or perhaps already has as of the publishing of this post)
• Strange flow to authenticate cluster by providing a token, even after providing your credentials.
• After following all the instructions, I still wasn’t able to add providers, and kept getting an “unknown error”.
• I was told all of the above are being worked on, but I was surprised by the large number of issues I got from following their getting started guide and how out of date it was.
• Upbound doesn’t seem to be working on a way to abstract applications the same as other products like Shipa or Humanitec with PAWS. Instead you can use Kubevela or Ketch for that.
• Crossplane is great but it has some limitations, and you still indirectly rely on terraform for full resource compatibility.

Conclusion

I have no doubt that platform engineering is the pinnacle of the DevOps evolution and where the industry is inexorably heading. Companies are going to need to have self-service capabilities for developers if they want to be able to grow and innovate at scale.

The increasing complexity and the near comedic sprawl of tools to accomplish similar things makes this even more of a necessity to bring sensible choice and order to an organization. Widely used tools in DevOps like kubernetes are so complex that they are often a treasure trove of memes and not all cloud platforms are that intuitive either.

That all being said, all of the tools I’ve tested above are anywhere from half baked to raw in their development lifecycle, with incomplete documentation, functionality that doesn’t fully work and/or lack of features. The lack of widespread use also makes third party documentation and blogs sparse, meaning your team is likely to have to do a lot of trial and error, unless you pay for a commercial product with an eager and hardworking support and development team that’s going to be there to assist you.

This is by no means a criticism of either platform engineering or the tools to implement it because it is still very early days as you can see by this Gartner graph:

Platform engineering is relatively new but gaining momentum fast, it will be very exciting to see where we are in a few years from now. Companies that begin their journey early by at least learning about it will benefit from better engineering environments and faster software delivery in future.

If you feel compelled to kindly give praise, remember you can hold down the clap button to lazily express your enthusiasm 😉