Recently, while preparing for our Alpha launch, I ran across a very strange issue that took the better part of a day to debug. I thought it would be a great idea to share my findings here so that others might benefit from my tribulations.
First, a little background… At Preset (my company), we are building a web app, along with a pile of infrastructure that’s all run on top of Kubernetes (EKS) which is in turn managed by Helm. In order to deploy things in response to end users’ requests, we are reaching out to Tiller (https://helm.sh/docs/glossary/#tiller) directly via its API using a pretty sweet lib (https://pypi.org/project/pyhelm/). Under the covers, Tiller communicates over GRPC. GRPC libs tend to be async in nature, however Python typically doesn’t. Fortunately (or unfortunately), there are libraries and even web servers, that can help with that! We are running Gunicorn+gevent in order to handle requests in an async way and yet still retain synchronous semantics. …
Imagine you’re a Spring developer working for company X on a micro-service based platform with services written in Java/Spring Boot and deployed using Kubernetes (Docker). Your various services communicate with one another using JSON over HTTP, making things simple. You assume that all inter-process communication happens over your “trusted” network (think private datacenter or AWS VPC, etc.) and therefore doesn’t need any sort of transport level security.
Now imagine that your company hires a new information security lead who gets to work on his first day auditing your beloved platform — including your method for dealing with transport security, or lack thereof. After doing a quick review, he adds a requirement for you to add TLS for all inter-service transport in your datacenter, citing the fact that data needs to be protected in motion, as well as at rest in order to adhere to standard security best practices. …
