Sentry and Kubernetes
Take-aways from deploying Sentry in a k8s cluster
If you haven’t heard about Kubernetes yet, it’s a platform that allows you to run and orchestrate containers. Currently Kubernetes is about six years old, and over the last two years, it has risen in popularity.
At xgeeks we are currently working on a client that is looking to create a platform based on Kubernetes. Using this platform they can deploy services on their own and have the ownership of the data generated, besides being able to have more or less computational power on those services.
With a large quantity of deployments we need a way to know and be notified of any errors occurred during runtime on our services. Being notified of this will allow us to iterate faster on bugs that happen in runtime.
Since we want to have our own software running on our platform we need to find a way to have a centralised error management on our cluster. Sentry is an Open-source error tracking application that helps developers to monitor errors in real time.
Our Solution
Our solution will be to deploy a Sentry instance in our Kubernetes cluster using Helm Charts. Although there are no official charts for Sentry, the community has created one that we will be using.
When deploying locally everything seems to go well, but when you try to deploy to the cluster, problems start to appear.
Problems that we found
Here is our list of problems that we found and how we solved them.
#1 — You should not use root images
For security reasons our platform does not allow to have containers with root privileges by default. Here is an explanation from the Bitnami Engineering blog:
The key take-away from the blog post is the following:
“Changing the configuration of your containers to make them run as
non-rootadds an extra layer of security.”
How we solved it
We created a new image based on the sentry one where we add the UID to the user:
#2 — Source maps don’t work
The user created previously doesn’t belong to the sentry user group and doesn’t have permissions to write source maps in the right folder.
How we solved it
Add the user to the sentry user group:
#3 — Multiple pods accessing the same storage
Sentry has 3 containers that need access to the same storage volume aka PVC:
- cron
- web
- worker
So the containers need ReadWriteAccess. Many of the k8s storage systems don’t support that kind of permission, so looking to the Sentry documentation we can see that we can change to an S3 or GCS(Google Cloud Storage) bucket. For this implementation we use GCS storage.
How we solved it
Create a bucket in GCS and update helm values to use the GCS bucket as storage.
In theory this should do it but when you try to deploy this you notice another problem.
#3.1 — Sentry only compiles with minimum needed packages
When we deploy with GCS credentials to a bucket we see an error of “failing to find packages”. We can see in the documentation that sentry in version 9.1.2 and prior only compiles its source code with the minimum needed packages.
Since Sentry was created in python we need to create a requirements file to add the missing packages and rebuild the docker image.
The new Dockerfile:
#4 — Generated passwords come with a cost
Sentry helm chart is configured to generate random passwords to administration, PostgreSQL and Redis if you don’t set them. The problem is that every time you have a new deployment a new password will be generated and the previous one will be deleted. We are using CI/CD to deploy sentry to our platform which causes us to lose the credentials and the ability to redeploy, or in our case, to run “helm upgrade”.
How we solved it
We have to clear the namespace and redeploy with the previously generated password.
There is another solution that is to deploy once and on the redeploy set the secret that contains the passwords. We choose not to follow this approach because we are deploying different containers at each time and for us they should be unique and the process should be omnipotent.
Final thoughts
We have succeeded in the deployment of Sentry in our platform and it is a tool that we are using to actively reduce the bugs in our software. When we started the deployment of Sentry, the version that was available was 9.1.3, nowadays you have version 10 available. That version has a lot of new features and we plan to update to that version.
Once we do this will share another list of problems 😄.
I hope I shared a few solutions for your problems in case you are trying to deploy Sentry instance. Feel free to drop any comments! 🙂
If you find this article interesting, please share it, because you know — Sharing is caring!
Also, if you enjoy working at a large scale in projects with global impact and if you enjoy a challenge, please reach out to us at xgeeks! We’re always looking for talented people to join our team 🙌
