Securing your microservices with Pivotal Cloud Foundry
Securing microservices is hard. It’s not that the tools for implementing security are hard to understand or use. What is hard is understanding the security implications of breaking up the monolith into potentially hundreds of services.
If you read “The separation of concerns between API Gateway and microservice”, you know that it’s not as simple as just defining a barrier around all of your services. When thinking about security, you should absolutely think of your microservices as separate systems that you individually will be securing and not a single platform which is the word you might use when trying to sell your microservice architecture to your customers. Each individual microservice should have all layers, according to the OSI model, secured. Do not fall into the trap of relying too much on your IaaS or PaaS platform. It won’t solve all your issues. Especially not on layers 4–7. Security for layers 4–7 in the OSI model is typically configured/implemented on an individual microservice basis and not system-wide. The reason for this rests in the nature of microservices. Your services will have different responsibilities and some of them will, therefore, be more sensitive than others. You will want to restrict access between them to mitigate the risk of sensitive data getting leaked. You could end up exposing all your data if one of your services got exposed otherwise.
The following high-level drawing address security concerns on layers 4–7 in the OSI-model in a typical microservice architecture.
We are using OAuth2 client credentials grant flow since the system won’t have any notion of a user. User authentication and authorization is delegated to the client application. The resource server will trust that the user has given proper consent to the external client application for using the data on the resource server. This is fine as long as the user isn’t the owner of the data that he/she wants to access. The typical use case could be catalogue services and registers for lookup and search.
The client credentials grant flow will cover security on layer 7. Additionally, we will be using mTLS and network security policies to secure layer 4–5 (TCP). The example is done for Pivotal Cloud Foundry, but it is applicable to Kubernetes/OpenShift as well. The authorization and Identity server needs to be set up on a private network. An API Gateway for the authorization server will expose the functionality necessary for the client application. Another API gateway will expose your APIs.
We are using open source technologies to avoid vendor lock-in. Apigee Edge is a terrific API gateway, but using it will make you depend on it. Open source versions like Zuul and the Spring Cloud Gateway might not do everything you need, but you can switch them out for something else when you need to or when the time is right.
Both Pivotal Cloud Foundry and Openshift/Kubernetes have their own authorization servers that you can use, but none of them has a full implementation of OAuth2.
When securing Ingress into your platform, make sure you use a proper authentication mechanism. The basic client credentials grant flow isn’t much more secure than using Basic Auth so you should consider your options. The Oauth2 specification support what is called a private key jwt assertion. The W2O Identity server supports using this type of assertion. Keycloak also can do client secret authentication using Signed JWT assertions which is basically the same thing. I would not recommend developing a version of the PK JWT assertion yourself unless you are exceptionally gifted, so if this is not a readily available option, then use mTLS instead and rely on the information related to the public part in the TLS certificate for doing the authorization. mTLS authorization is good to do in your API Gateway, but you can do it in the service instead if you want to. In many cases, doing the full mTLS+OAuth2 authorization for all services is to be prefered. It might seem overkill to do both mTLS authentication and authorization and OAuth2, but you should consider securing all the layers in the OSI-model. mTLS will work for layer 4–5, but you should absolutely not skip layer 7. The combination will make your Ingress almost impossible to breach. One issue you might get from using certificates is maintenance and rotation. Hashicorp Vault could help you to securely store and rotate your certificates. Hashicorp has made a service broker for Cloud Foundry that I would consider using. Please remember that even if you just use a client secret assertion, that you still need to implement a way for the client applications to change their client secrets.
For Pivotal Cloud Foundry, your ingress will be the go-router. The go-router will make sure you achieve non-repudiation by verifying the client’s mTLS certificate with the mTLS certificate’s CA. You will terminate the mTLS between the external client applications and your microservices in the go-router, but the go-router will forward the public part of the mTLS certificate through the X-Forward-Client-Cert header so that you can authorize the client in the API Gateway and the microservices. This, however, requires mTLS between the go-router and the API Gateway to be effective.
From version 2.3, PCF has added support for Mutual TLS App Identity Verification. This makes mTLS possible between the go-router and the application services. The prerequisite is that the service has a URL. The only service that you want to bind to a URL is your API gateway. You, therefore, need to setup mTLS connection between the go-router and your API gateway. It should be as simple as flipping a switch for PCF. Still, if you want to setup mTLS between your microservices, you have to register the public routes with the go-router. That would expose your internal apps. You don’t want that. In PCF you have what is called App Instance Container Identity Credentials. An application gets assigned a certificate and a certificate chain when the application is created. The App certificate gets rotated every 24 hours. It’s, therefore, possible to configure your applications to use mTLS and authorize request based on the identity of the source application. You could use the certificate and certificate chain to set up mTLS connections, and based on the app-guid, which you would find written as the Organizational Unit value under the Subject field of the certificate, compare the app-id with the app-id you have pre-registered for the receiving service. Doing this extra authorization means you lend a bit of your level 4–5 security to the levels above. You could opt for only relying on mTLS and App Instance Container Identity Credentials and rely on that to be secure enough for your microservices.
For Kubernetes you have Istio. Istio can be compared with Spring Cloud although it’s slightly different. You can think of Spring Cloud as a microservice chassis while Istio is a microservice grid. In that respect, Spring Cloud is a set of applications that together give you a microservice application architecture. Istio, on the other hand, is injected into the platform and operate alongside your services without you noticing they are there. That offloads many concerns that previously needed to get addressed in the application layer down to the platform. Istio has an incredible security model which I am personally a big fan of. Out of the box, it supports SPIFFE — PKI, mTLS and OAuth2 integration for all your services. When looking at securing your Kubernetes applications, you should definitely look into Istio. PCF is actually leveraging parts of Istio in their Cloud Foundry architecture. This is actually how they are able to deliver App Instance Container Identity Credentials and Mutual TLS App Identity Verification. To get a grip on how this is done for Cloud Foundry, read here. Looking ahead, Pivotal Cloud Foundry is working on switching out their entire routing layer with Istio, but the Istio CF project has been under heavy development for some time and it may take a while before it is stable enough for production.
There are a couple of downsides to mTLS and App Instance Container Identity Credentials though. For example, it’s quite difficult to do more fine-grained authorization per operation based on the origin of the request or the original requestor. This is where OAuth2 can be handy.
The following drawing shows how you can leverage OAuth2 to do fine-grained OSI level 7 authorization and security for services. Each of your microservices can have controllers, or even endpoints, with different levels of sensitivity. With the prerequisite that the source application has been granted a scope through the user’s consent, the client’s request can be granted access based on which scope that is registered for the microservice endpoint in question. This allows much more flexibility in terms of defining the sensitivity and functional scope for each microservice. Using OAuth2 in combination with mTLS gives you the best of both worlds. A strong security architecture coupled with the flexibility of OAuth2 authorization. You may be tempted to rely on network policies for securing your internal services, but you should be very careful about doing so as they are very easy to circumvent. Even if you have defined network policies between your services, circumventing these just require someone to log into the PCF application manager and assign a route to one of the services. It can also happen that, by accident, a developer leaves the no-route option to false in your application manifest during development. PCF will assign a public route and expose your microservice if the same manifest gets used to deploy your application to production. I would still use network policies between my services to secure them from external networking access, but I would use mTLS and OAuth2, as well, to mitigate the possibility for an accidental exposure or a security breach.
We haven’t talked about encryption. Encryption at rest is out of scope for this article. If you want to know whether you should encrypt at rest or not, then read my article, “To encrypt, or not to encrypt, that is the question”. I also haven’t talked about how your PCF foundation could be secured on the platform or networking plane. Platform security is out of scope for this post as well.
If all this made you unsure on whether you should choose Kubernetes or Pivotal Cloud Foundry, be aware of the following. Pivotal Cloud Foundry has, within a year, been able to implement mTLS and SPIFFE based App Instance Container Identity. Enabling these only require you to flip a switch. The learning curve is also considerably lower for both your ops and your developers. PCF may be lacking in flexibility, but it is considerably quicker to roll out. I talked to an Ops member that was working on a private Kubernetes cloud platform project and they had used 18 months on setting up their platform from scratch. They still weren’t finished and hadn’t even begun application development. From my experience, it will take you half that time to set up the Pivotal Cloud Foundry Platform in a private cloud and get your first release out into production. 2–3 months installing the basic installation and get you ready for development, another 3 months to integrate your Spring Cloud microservice chassis and then another 4 months to get you ready for production, and that without having any prior knowledge of Pivotal Cloud Foundry. If you require more flexibility, Pivotal also have their PKS service. If you are going to build up your own private cloud from the ground up, then Pivotal is not your worst option. What you pay for, you gain in less effort. You should also think of whether you will be able to onboard developers and ops that actually have worked with Kubernetes. It’s one thing to know it from a purely technical point of view, but it can’t make up for the lack of experience. In my experience, Pivotal Cloud Foundry will lower that risk considerably.
