Credhub as an Opinionated Service
Being secure is the mantra that every enterprise/product/vendor is focussed on. To be agile, to achieve greater speeds in the release train, enterprises are soaked into this “so fashioned cloud native” digital transformation. At the same time being cloud secure is an essential aspect to be a successful agile innovative organisation. Speed and being Secure is kind of oxymoron, but to be successful there has to be a lot of synergy that should exist between these two. If not, will go down the rabbit hole if these two don’t work together.
When it comes to Enterprise Security Pivotal’s Chief Security Officer Justin Smith talks about the importance of Cloud Native Security and the ‘3R’ principles and how PCF is helping customers heading towards this transformation journey. The first R ‘Rotate’ where he mentions credentials are often long-lived, meaning in the event of a credential leak, attackers could have prolonged access to your environment. To avoid these security threats, there has to be an easy and efficient mechanism to generate, create, rotate and delete the credentials. Clearly a stronger security posture is mandated in this service driven anything.
Here comes Credhub that generates, stores and manages the lifecycle of credentials in CF.
Here we look at how we can create a Credhub service broker using Spring boot based on Open Service Broker 2.x. We use the broker to secure the credentials in CF, and then consume it in a client written using Spring boot.
Sample Credhub Service Broker
Spring Credhub provides support for storing, retrieving, and deleting credentials from a credhub server running in a CF platform. This sample spring boot based broker generates and secures credentials in credhub server running in CF. More about credhub in CF docs.
Credhub supports two authentication modes: mTLS and oAuth2. You can use either of these in CF. Based on the properties that you specify in application.[yml|properties]/manifest.yml, one of the authentication methods will be leveraged.
Authentication
manifest.yml
env:
SPRING_CREDHUB_URL: https://credhub.service.cf.internal:8844If you specify only the SPRING_CREDHUB_URL, then mTLS will be used. Apps will be authenticated using mTLS as CF platform manages this automatically for platform related components.
env:
SPRING_CREDHUB_URL: https://credhub.service.cf.internal:8844
SPRING_CREDHUB_OAUTH2_CLIENT_ID: clientid
SPRING_CREDHUB_OAUTH2_CLIENT_SECRET: clientsecret
SPRING_CREDHUB_OAUTH2_ACCESS_TOKEN_URI: https://uaa.sys.pcf.com/oauth/tokenGenerate a UAA client and use the details in the properties file. If you specify all 4 aforementioned properties, then oAuth2 will be used. Credhub interacts with UAA for the given secrets to facilitate oAuth2. You can mention the properties in application.[yml|properties] file as well
Authorization
AuthZ is determined/set while binding a service to an application. This happens during cf bind-service call when access permissions for the credential object are set.
JsonCredentialRequest credhubRequest = JsonCredentialRequest.builder()
.overwrite(true)
.value(credentials) .permission(CredentialPermission.builder().app(request.getBindResource().getAppGuid()).operations(READ).build()) .name(ServiceInstanceCredentialName.builder() .serviceBrokerName(request.getServiceInstanceId()) .serviceOfferingName(request.getPlanId()) .serviceBindingId(request.getBindingId()) .credentialName(request.getBindingId()).build()) .build();Permission for the generated credentials is set while persisting the details in credhub server by looking up the guid of the application to be bound. Access controls are restricted through this permission object setting and in this case this blocks all but the bound application from reading the credentials.
This sample broker uses an in-memory db, however it can be customised based on the need. Actual source code and instructions for getting this up and running in CF is available in the repo credentials-service-broker.

Sample Credhub Client
This is the consumer, client side part of the program. This sample client interpolates credentials persisted in credhub server in a non-assisted mode using spring cloud libraries when you bind a broker service like credential-service-broker. Spring applications using Spring Cloud Connectors or Spring Boot ${vcap.service.} properties will have framework support for auto resolution. Also, it uses spring-boot-starter-cloud-connectors to interject the bound service info as a custom cloud service connector.
When you run this magic command ‘cf push’ by binding the broker service, you won’t see the credentials in raw format. You would only get to see the `credhub-ref` in `VCAP_SERVICES` binding like,
"VCAP_SERVICES": {
"credentialstore": [
{
"name": "credstore-svc",
"instance_name": "credstore-svc",
"binding_name": "credstore-svc",
"credentials": {
"credhub-ref": "/c/57744a22-7ff9-491b-b73e-6df70501fa38/68b4882d-8607-40b6-99ce-44cc285ddf54/04b68e3b-7433-4238-a61e-515ec8b07348/04b68e3b-7433-4238-a61e-515ec8b07348"
},
"syslog_drain_url": null,
"volume_mounts": [],
"label": "credentialstore",
"provider": null,
"plan": "standard",
"tags": [
"credhub",
"secrets",
"credentails",
"certs"
]
}
]
}
Even if you do cf ssh -> env or cf env or manual ssh -> env, everywhere it would get displayed as above. During ‘cf push’ spring libraries would do the interpolation and the decrypted raw values would get injected only into the process memory. As in this case we have defined our own reference custom service connector for credhub, it would get created and initialised with the decrypted values. This will be fully transparent to the end users. It uses mTLS for authentication between credhub server and the app that does the interpolation that is managed by the CF platform. Now the credentials are completely secured and nowhere it gets exposed in raw format except in the actual running process memory.

When you hit the actuator endpoint /actuator/env (spring boot 2.0 and above), you would get to see the unwinding of the credhub-ref env variable. Actual source code and instructions for getting this up and running in CF is available in the repo credentials-service-client.
Links:
https://github.com/srinivasa-vasu/credentials-service-broker
https://github.com/srinivasa-vasu/credentials-service-client
https://spring.io/projects/spring-credhub
https://docs.cloudfoundry.org/credhub/
https://www.youtube.com/watch?v=gQnM_PIMuE8
https://builttoadapt.io/the-three-r-s-of-enterprise-security-rotate-repave-and-repair-f64f6d6ba29d