Top 10 Things To Secure Your Microservices (Chintan Jain, May 2017)

Executive Summary

Microservices are here to stay. They offer tremendous benefits as compared to fat Monolithic applications and are perfect for today’s production systems. Microservices developers gain by having more independence in all aspects of SDLC including design, development and rapid deployment of these services in production and having less dependency on other teams. However, with great freedom also comes great responsibility. The microservices developers don’t have the advantage of having pre-defined security and governance controls for confidentiality, availability, integrity and accountability of their services like their predecessors. The developers are responsible for the designing secure microservices for all stakeholders of the microservices with little help available from traditional security departments.

In this blog, I wanted to cover top ten must dos that every microservices developer should know and practice to have a secure microservice in production.

Here are the top 10 things you need to do for a secure Microservices:

1. Employ TLS 1.2 for point to point encryption

2. Authentication your microservice consumers

3. Proper authorization of your microservice consumers

4. Rate Limit and throttle your microservice traffic

5. Enable Alerting and Monitoring to detect abnormal patterns, system outages and security events

6. Design a secure network architecture

7. Make your microservices resilient across different data centers or cloud regions

8. Store private keys, system or database credentials securely

9. Encrypt very sensitive data like passwords or pins at a field Level

10. Catch all for all other security best practices

Description of these things in detail

1. Employ TLS 1.2 for point to point encryption

This is a no brainer. Every client that connects to your microservices should use Transport Layer Security(TLS) security and use TLS 1.2 to connect to your endpoint. TLS is used for point to point encryption between the client and your micro service and offers the advantages of data confidentiality and origin authentication. Hence, all the data that is exchanged between your client and the microservice is protected against eavesdropping and tampering and the client can be rest assured that it is talking to your service only and not a malicious user’s service.

Furthermore, use of TLS 1.2 ensures that only the strongest symmetric and asymmetric algorithms and hashing algorithms with large key sizes are used which makes it very secure.

One important point to note is that you should not restrict yourself to using TLS 1.2 only between your consumer and your service endpoint. Even various layers of your architecture should use TLS 1.2. This may include Load Balancer and your web server, your web server and your application server or between your application server and database server and all other downstream applications that you call from your application. Remember, your security is only as powerful as your weakest link — so it is of utmost importance to employ the security principle of defense-in-depth and protect each layer from eavesdropping and tampering and against man-in-the-middle attacks. And this you can do by using TLS 1.2 at all relevant layers.

2. Authentication your microservice consumers

This is very important that you secure your microservices using various standard methodologies available for authentication today. This can be a certificate based mutual authentication, a client id and secret given to each client based or using various variations of industry standard protocols such as OAuth, Open ID Connect or SAML.

Which authentication scheme you deploy with depend on your use case however I would recommend that for high security use cases, you would be better off having multi factor authentication scheme that employs two factors such as what you have, what you know and what you are. As an e.g. if you have a very high security use case then you may require your client to provide a client id and secret and provide a client certificate (something you know and something you have) whereas for a medium or low security use cases you would be fine just requiring a client id and secret to authenticate the client. You may also employ context based schemes such as IP address, Device ID, One Time Password etc. to factor something you have as an authentication factor.

3. Proper Authorization of your microservice consumers

Once you authenticate the client, you must make sure that the client is only authorized to perform what they were specifically authorized for. This can be based on their client certificate name, or client id with their authorizations saved in a database, caching layer or shared file system. The authorization checks not only applies to communication between the client and your microservices endpoint but also between each layer of your microservices. As an e.g. suppose your microservices accept an image from the end user and saves it in S3 bucket hence the microservice needs to have the permission to upload into S3 but it should not have permission to delete from the S3 bucket or even read from the S3 bucket if it does not need that permission. In short, each layer in your microservices architecture need to work on a need to know and least privilege security principle.

4. Rate Limit and throttle your microservice traffic

I have seen many microservices developers boast that their services can scale infinitely and ready to accept all requests that come their way only to fail miserably in production. It is of utmost importance that application teams focus on performance testing to find out how much peak load their service can support and what is the best response times as per business requirements. If the service developer does not know what is their peak capacity and they start accepting all traffic, different layers in their architecture will start choking and will impact all their clients. Even if the application layer can handle the increase in traffic may be the database will choke because the connection pool was not big enough, there was not enough memory available or there is not enough CPU power. In short, you will need to be very methodical in planning your capacity and then apply rate limit rules to your clients so as not have availability issues later on.

If the client is sending more traffic to you than you planned for, then you need to start rate limiting them. This can take several forms:

(1) Limit the number of requests the client can send hourly.

(2) Limit the number of requests the client can send daily.

(3) Throttle the requests so the clients don’t consume all your bandwidth. Taking from a real-world example see how T-Mobile gives you 4GLTE speed until you reach your 6GB monthly allowance thereby slowing your data speed considerably down after that limit is reached.

In short, you need to do a better job doing capacity planning on your microservice so you clients don’t have availability issues and your microservice does not go down due to DDoS attacks or erroneous usage. Rate Limiting if done correctly will help prevent DDoS and subsequent availability issues.

5. Enable Alerting and Monitoring to detect abnormal patterns, system outages and security events

It is very important that you keep an audit trail of all client calls made to your microservices. However, great care should be taken that any sensitive information such as client secret, User Personally Identifiable Information, passwords, Account Numbers, Session Tokens etc. are not logged. Automated alerts and monitors should be set up on your logging system based on certain patterns. The logging systems should then trigger emails to the system owners. The alerts can be triggered on events such as the service giving many 500 error responses, many unauthorized requests to access the microservices, many failed attempts to authentication using a valid client id but incorrect password or secret, user of expired tokens, service throwing 404 errors etc. As availability is also an important pillar of security, your microservice not only need to be preventive but also be detective to discover and remediate availability issues. Logging plays an important part in detecting the availability issues and detecting any malicious or abnormal use of your microservice.

6. Design a secure network architecture

As I mentioned earlier, your security is as good as your weakest link. Hence it is important to think about security at every layer of your network. This can include the technology stack you choose for your microservices (and related security issues with that), the operating system you choose to develop on, the web server, application server, caching layer and database server you choose, the routers, switches and the firewall rules you put in to allow traffic to your microservice. In short, you would need to configure each of these components securely and encrypt the traffic that flows between these different layers using TLS 1.2 as described above. Furthermore, your microservice may also be calling some other external or internal services. It is very important that those services are secure as well and employ all the security best practices as mentioned in this paper. Remember that your clients may not know that you are using these other services and any breach in these services is same as the breach in your microservice from a client perspective. Hence, great care should be taken in securing each layer of your network architecture.

7. Make your microservices resilient across different data centers or cloud regions

You would need to ensure that your microservice is not just running in one datacenter or a cloud region. Your microservice needs to be resilient along different datacenters or cloud regions. This can take a form of an active-active or active-passive deployment. This is because there are some data centers or cloud provider issues that are outside your control and the region may go down anytime due to external factors outside your control. In the same vein, you would also need to ensure that all your dependencies (other services you rely on) are also resilient. As an e.g. let’s say that you are reliant on a microservice that is also deployed in AWS-East Region just like your service which is Active-Active across AWS-East and AWS-West Region. However, if this other service is not resilient across West Region then it does not matter even if your service moves to West due to AWS-East Outage because your clients will still have service availability issues due to your dependence on this other service. Hence, you need to ensure that not only is your microservice resilient but also all your dependencies are resilient across different regions or data centers.

8. Store private keys, system or database credentials securely

I have seen security applied at each layer however TLS private keys being stored in plaintext in the file system on the web or application server. If an attacker or a malicious insider can get to your server, the first thing they will do is try to find this private key and then they can intercept and decipher all communication between your server and your client. Hence, it is very important to secure the TLS private key. There are several ways to do this. The top most security systems would be better off encrypting the private keys using physical or cloud based HSMs and storing the encrypted private keys on the server and then decrypting them on the fly and storing them in server memory temporarily (you may enable a keep alive to keep the private key active in memory for 30 minutes and extending it further if there are more access requests with an upper limit of expiring it daily). Some other cheaper yet very secure options to encrypt the private key is using services such as AWS KMS or Hashicorp vault.

The same patterns of encrypting the private key can also be applied to storing other sensitive data such as database credentials, authentication credentials to talk to other microservices etc. The point is that none of the authentication credentials to any internal or external dependencies should ever be present in plain text in your servers and you should use symmetric cryptographic algorithms such as AES to encrypt the key with another key that is stored in systems such as HSM, KMS or Hashicorp Vault based on your use case.

9. Encrypt very sensitive data like passwords or pins at a field Level

Many times, microservices developers take the approach of encrypting the entire database rather than focusing on encrypting or hashing sensitive values such as user passwords, credit card numbers, session tokens (OAuth access or refresh tokens or session tokens), social security numbers or sensitive account numbers. Note that entire database encryption does not provide adequate security against several attacks such as a malicious database administrator, malware reading memory or SQL injection etc. It is recommended that following techniques be used to encrypt various data types in database at a field level:

1) Password — Hashing using BCrypt or SCrypt

2) Credit Card Numbers — PCI compliant Tokens

3) Various tokens such as OAuth Access or Refresh Tokens or Session tokens — AES Encryption

10. Catch all for all other security best practices

The last but not the least is all security best practices that also apply to monolithic services. The main ones are:

1) Protect your microservices from OWASP Top 10 vulnerabilities. Now a days, a lot of framework will provide nice inbuilt features to protect you against these but you still need to use them in order to protect against these.

2) Keep your OS and your open source software, 3rd party software versions up to date with security patches. Again, this is a legacy top best practice that still applies to microservices.

3) Make your developers and agile team trained in Application Security and have them practice Secure Software Development Lifecycle so security is thought through since the design phase and not put on as a bandage later. Bandage security never works and causes lots of problems for every stakeholder alike — so why not bake security in from get — go?

4) Engage in penetration testing first time you develop a microservice to find out security flaws in your microservices and then perform penetration testing yearly. If the pen testers know what they are doing, you would be surprised with what they find.

Conclusion

This paper gives you a very good overview of what you need to do to run a secure microservice in production. However, there is no one size fits all so what security you apply to your microservice will totally depend on your use case. My hope is that with this paper you would appreciate the complexity of the security problem for microservices however also appreciate how it is possible to create a very secure microservices by fragmenting a very complex security problem into tiny simpler bites. Ultimately having simple solutions based on solid security fundamentals will help you in creating a very secure microservices for all stakeholders for your microservice.