My Experience Building A Web Infrastructure From Scratch
Having a proper infrastructure is one of the most important things a development team needs to develop peacefully.
And I mean not only a powerful and secure one but a comfortable one.
Finding this balance between performance, security, and comfortability is not that easy.
I want to tell you what I have learnt while I was creating an infrastructure from scratch for Elma.
The development workflow
Many people in charge of infrastructure think only in terms of the production environment.
They think mainly about how the infrastructure will perform in terms of final user usage.
Don’t misunderstand me, I really think this is a very important point of view. But I think there’s another one: development workflow.
Per each dollar you spend in infrastructure, chances are you are spending much more on developers. If you want your investment worth it, you need to think about how developers will behave with this infrastructure.
So yes, the first thing I thought in terms of infrastructure was the development workflow. How developers we’re going to develop, test, integrate, code review or deploy. How much time I could save for developers in terms of infrastructure.
And the first I decided was we need to use less tools. Having the same tool for most of the workflow is great in terms of productivity. And Gitlab was our choice.
Gitlab for task management, version control, and pipelining.
Before the new infrastructure we were using Jira for task management, Bitbucket for version control and Jenkins for pipelining. Although you can integrate this tools more or less, Gitlab provided all of this at a few clicks distance.
With Gitlab on the cloud, we saved on having to maintain Jenkins needed infrastructure, using also shared runners. While this have a cost, AWS machines used for Jenkins weren’t free at all.
This, for sure, have its drawbacks, like having trouble when Gitlab servers are experiencing problems. But, in the end, it saves us the work of maintaining Jenkins and the correspondent machines.
As Jira isn’t free, for the cost of it we had the three problems solved.
Docker for the job
In the past, I have made the mistake of considering Docker over complicated. While it’s not easy to use and understand technology, I have found it very useful last year.
We were going to create a microservices architecture, so it made a lot of sense to use a provisioning technology that allowed us to use very different programming languages or technologies.
One important thing about Docker: learn how inheritance works. Use it to define kind of a tree for the different environments you may have -like development, integration, production…
The main objective is to have the same infrastructure both in development and in production, and the only differences should be in terms of scalability.
This takes me to…
Don’t use any technology you cannot use or, at least, simulate in development
One of the most stupid time developers loose is trying to check things like in production.
Every time developers need to deploy something to a different environment to check something is time lost while the deployment process occurs and an obvious demonstration that your environments are not equal.
So you are not developing in the same infrastructure you will deploy for users. Bad bad bad.
For example, we wanted to use Google Pub/Sub as a managed queue system, and we found an emulator, so we can use it in the development environment without any trouble. What about BigQuery? We were using BigQuery and it’s an important piece for BI team, but BigQuery doesn’t have any emulator, so developing it is quite a pain in the ass.
Kubernetes as a Docker cluster
When you put Docker in a cluster you arrive at the “only scalability differences” I was talking below. We were using Kubernetes for development and Docker Swarm for production which was obviously a bad decision -probably we were in the middle of a change.
So one of the first things we made was to put Kubernetes in all the environments except for development. This way we have, at least, an environment to check all things like in production: staging. The problem is we have to deploy to test, which is a loss of time as I was talking before, so we are developing an integration environment where we can share folders and check the actual code we are developing.
Kubernetes is not an easy technology by any means. You need to test, fix, test, fix, and fix again to get things working. But, in the end, you have a reliable cluster that can be scaled at will.
Deploying code to Kubernetes is also a difficult thing, which takes me to Helm.
Kong as a proxy
To add some actions after and before every request, we have configured Kong and the Kong ingress controller.
Is ridiculously simple to add things like CORS, API key or event proxy cache.
All you need is to add some YAML files to your charts to create consumers, plugins, and to relate everything together.
If you want to avoid some application code that is common to all of your microservices, using a proxy like Kong is a great idea. Try it!
Cert manager for SSL certificates
You can pay -a lot- for wildcard certificates or you can just use LetsEncrypt with Cert Manager configured with Kong like us.
Technically speaking is another chart installed in your cluster and some more YAML’s files in your Helm deployment charts.
Is not the most simple configuration I have seen, but once you can make it work everything is the same.
Helm for deployments
Deploying on Kubernetes consists on apply YAML files defining pods, services, ingresses, jobs and more. Doing this by hand is not usable nor scalable and we wanted to do everything from Gitlab pipelines, so Helm makes a lot of sense.
You define a “chart”, which consists on the stated YAML files, and Helm is in charge of deployment history, versioning, etc.
You can also find charts for lots of different uses if you search on Google. For example, if you search “Kibana chart” you will find a pre configurated chart you can use to install Kibana in your cluster. Easy.
While Helm is arriving at its third version, which it’s said it’s gonna be much better, even now is a great option. It simplifies a lot the deployment process, so don’t hesitate to use it on your own projects.
Terraform to manage it all
Before Docker, people use to have Terraform creating infrastructure and Ansible provisioning it with the needed software. While Docker and Kubernetes make a lot of this by itself, Terraform stills make a lot of sense to manage the surrounding cloud platform -in our case Google Cloud, we will talk later.
So we use Terraform to create the clusters, the nodes, to install general needed charts, databases, permissions, etc.
We should be able to remake the whole infrastructure from scratch with a single terraform apply command. And this is something to chase, having into account disasters occur from time to time.
Another interesting point in favour of Terraform is that it can be used as a documentation of the whole infrastructure.
Google Cloud as a cloud provider
We were in AWS, and there is nothing wrong about it. But Google Cloud is a little bit cheaper -especially because the cluster itself is free and you only pay for nodes- and has better agreements for startups, so we are saving some money.
Also, we already were using some tools from Google, like GSuite and Bigquery, and made sense.
In general, Google Cloud is a great cloud provider, having lots of useful tools and very interesting prices. Is quite interesting what a combination of Firebase and Cloud functions can do for companies looking for a serverless infrastructure.
To note some drawbacks, permission system for Google Cloud -IAM, service accounts, etc- is a little bit confusing compared with AWS security groups and some tools like Pub/Sub are still a little bit rudimentary -there’s no control panel to see actual messages.
So, in the end, our developers do code on the local environment, push it to Gitlab, and pipeline starts. After passing tests, code is built and developer can deploy to staging to check it out if is needed. If not, merge to master, tests, build and deploy to production. Everything in one tool -Gitlab.
This deployment process generate needed daemonsets, services, ingresses and kong needed stuff and apply it to our Kubernetes cluster, which consists of two identic nodes.
And you have everything working. Just like that.
