Building a Kubernetes Cluster on a Budget

Scaling organizational tools to hundreds of users

Photo by Maximilian Weisbecker on Unsplash

I thought it would be a good idea if UX Society had some self-hosted tools that we can use to encourage inter-organization communication and to standardize documentation for the next generation of students who join the organization.

The goal is simple. How do we provide tools like email, real-time chat, and CI/CD in a cost-efficient manner? When we grow as an organization, how can the infrastructure scale while still keeping costs down?

The SaaS Problem

The easy way out would be to go for SaaS subscriptions. It’s really expensive though. $5/mo/user for GSuite, $12/year/user for Slack, $25/mo for a GitHub organization and an additional $9/mo for extra users… you can see how all of this can really pile up. It even becomes more of a problem when you scale these tools for use across the entire organization. We estimated that UXSoc would have to pay a whopping PHP 270,000.00 if these SaaS tools were used for every member of the organization.

Calculations that Meg and I did if we used these tools across the entire organization

It might be possible to implement these tools if you let people pay dues to join the organization. The UXSoc chapter heads, Laura, Mike Jaren (MJ), Chynna, and I, had a discussion about how we could streamline our communications into a single tool. I argued that it would be worth it for a UXSoc member to pay $12/year for them to stay connected with current UXSoc members, as well as access the depth of knowledge that previous leaders of UXSoc have left behind for the organization to continue flourishing.

But, this strayed away from the our mission of letting our organization be open to the student community. It wasn’t meant to be exclusive, so I can see why some executive board members could be averse to involving money so that someone can be a part of UXSoc. Some might also think that $12/year is too much of a price to pay. Compare this to how some AISEC chapters require its members to pay $25/year, while most social fraternities and sororities in the USA have its members pay $60/year in dues. $12/year to stay connected with other UXSoc members is a small price to pay.

The Self-Hosted Solution

Each master or node represents a VM or a physical machine

The opposite of subscribing to these tools would be to run a bunch of self-hosted applications. What better way to do this than running all your apps on a Kubernetes cluster? It’s not that simple, as I eventually found out.

The Setup

Two questions come to mind when setting up a cluster. How much is our budget, and how many machines do we need?

To figure out the budget, I based the ceiling amount of what I can spend on infrastructure based on what UXSoc Ateneo earns from consultancy. So, I set the bar at around $500/year, which is just over $40/month. A single consultancy project can usually cover these costs.

The number of machines we need could be somewhat estimated by the number of apps we are going to run. I decided to go for a 2-node cluster, with one master on the $5 tier, and one worker on the $10 tier that DigitalOcean provides. I followed this guide to get me up and running. With this 2-node cluster, we are currently consuming $15/month.

Deploying a mailserver

This is what can happen if your mailserver runs out of space.

A mailserver would be really important to have so that matters regarding org work can be sent and received in a centralized system. With every member or executive board member having a @uxsociety.org email, we can further standardize both our internal and external communications.

Using this guide, I set up Mailu to run on a DigitalOcean droplet. On the $10 tier, 50GB would be enough space for 50 users, if you assigned a quota of 1GB/user. It would cost $0.01/GB of storage on DigitalOcean if we needed to scale the number of users. Not bad, since extra space was quite cheap.

However, when actually using the mailserver, sending mail from my own relay would always end up in spam. I’m guessing that a lot of the DigitalOcean IPs were used at one point to deliver spam, so the IPs were placed on a global blacklist.

To mitigate this, I set up an external SMTP relay with SendGrid. After signing up with SendGrid, you can send up to 100 emails/day with their relay before you have to pay for its services.

After setting up the relay with SendGrid, my emails weren’t appearing in spam anymore. But, I had to use an email client like Thunderbird to manage my emails, which is a far cry from the Gmail interfaces we regularly use. Even though self-hosting email could work, I think that people would rather prefer the usability of their email — which is why the $5/mo/user for GSuite is justified.

Deploying a chat application

I first did some research into Slack alternatives and found three options:

Zulip, which grouped its messages through topics/subjects, much like email:

Mattermost, which calls itself the “open source, private cloud, Slack alternative:”

and RocketChat, a plugin-based enterprise chat system:

I decided to go for RocketChat. What made RocketChat really easy to deploy was that there was a helm chart available. This made it really easy to install RocketChat onto my cluster, which is actually live and accessible at: https://chat.uxsociety.org.

The concept of helm is similar to npm, but not exactly. While npm is a package manager that helps you add libraries to your code, on the other hand, helm manages what it calls “charts,” which are a set of configuration files that you can add to your cluster so that you can easily deploy and expose services.

helm install --name rocketchat --values values.yaml stable/rocketchat

However, deploying this onto a cluster that you manage could have potential issues — what if you run out of SSD space? Is there some kind of backup process that we’d need to follow? Would we need to add more machines or block storage to handle more users? I guess only time would tell how to manage these problems when they occur in the future.

Deploying a CI/CD application

Drone’s interface is quite clean and simple

At Bloom (my previous internship), I was interested in reducing the costs of managed systems like CI/CD. Using Travis, you can run private CI with 1 concurrency at $69/month. With Drone, you can scale to as many build agents you need, but for a lot cheaper.

I initially looked up Jenkins as a build server, but decided to go for Drone instead. These two features were what made Drone really appealing to me:

1. It runs everything in containers, and is coded in Go, so the application did not take much resources to run compared to other CI/CD apps
2. Drone has a helm chart which made it really easy to install

helm install --name drone --values values.yaml stable/drone

When installing helm charts, make sure to edit the configuration file to add your GitHub client ID and secret, which you can find when registering OAuth apps on your account.

Not surprisingly, I had to scale my nodes, because we would need more memory and CPU resources to run Drone. Currently, from the 2-node cluster, I added one, $20-tiered VM, which leads us to a total cost of $35/month. Looks like we’re still under budget!

You can use the ansible playbook from this guide to add more worker nodes to your cluster when you need them.

What I learned

I feel like in the long term, I would spend a lot more money managing these systems. Managing SSD space and uptime can be a real pain if you’re trying to be careful with how much you spend. I think that the ease of using managed systems is well worth the cost than self-hosting, but if you are looking for a setup that can deploy applications easily, but cheaply, then this is the setup for you. In the long run, I really hope that UX Society will adopt some standardized practices for communication, and although this may happen later, rather than sooner, the change requires more than just a change in tools, but a culture change.

I will also be publishing my k8s configurations on my GitHub soon so stay tuned!