Setup Kubernetes cluster and deploy drone on AWS — Part I

Before we talking about what is Kubernetes, let’s talk first on the Monolithic Architecture and Microservices. In the past, what engineers are familiar with usually follows the monolithic architecture, monolithic means composed all in one piece, build a single backend codebase, inject business logic and components within, when company scope becomes bigger and want to scale, we increase more backend service.

What’s wrong with that?

No, it’s not about right or wrong, using monolithic architecture still can establish your business, just that the scaling strategy will be a bit difficult to maintain, also it increased the complexity of engineers development.

Back to 2016, when honestbee only have Groceries vertical, our Rails backed backend codebase is quite clean and can be easily maintained, but when things coming to 2017, Laundry and Food vertical came in and introduced more different business logic on Brand / Store / Zone / Order / Fulfillment handling, we starting to add different logic to the original codebase, using state machine to deal with multiple statuses, more design pattern to handles the serializer, polymorphic, etc.

Things are getting worse

From the business perspective, we can edit the existing backend codebase to achieve the targets and we did, but, from an engineering management perspective, more things get affected.

1. The entire backend development progress and performance getting slower from time to time
2. New engineer on-boarding speed slow down, people are getting confused and takes a longer time to understand such large codebase with complex business logic
3. Hard for debugging, we need to trace more than 3-tiers to deep dive to the root cause.
4. Setup multiple instances to serve all countries, but new launched cities stability may get affected by another huge traffic cities.

Team morale also damaged because of this, because engineers cannot contribute using fast speed as they expected in the beginning, there are too many constraints.

How to solve it?

Then, we start talking about micro-services, but we don’t think micro-service is a silver bullet, we use it to benefit us based on these facts

1. Separate services based on different business logic, make the code clean and small, easy to maintain and understand, also faster development
2. Deployment is independent and not blocking each other
3. Teams can choose their preferred languages and technology, encourage them to discover new tech
4. Using different strategy to deploy and scale, regional, domains, etc.

Of course, some side effects are:

1. Deployment management of multiple services (load balancing, scaling, monitoring)
2. Partitioning the application and definition into microservices is an art (sometimes we over-engineering)
3. Extra effort and monitoring of services communication (e.g. Using message Pub/Sub like Kafka)
4. Integration test becomes more complicated

From our perspective, the Cons < Pros, because the Pros solves the process conflicts, team management difficulties, and the best part is “Teams can choose their preferred languages and technology, encourage them to discover new tech”, it encourages our engineers can feel freer to try & error, which is a good thing for culture.

Why Kubernetes?

Since we had many micro-services running independently, we definitely don’t want to manually deploy all those containers into each instance, Kubernetes helps us to do the automatic deployments, expansion, and management, the benefits are:

1. Deploy container to multiple instances at the same time
2. Manage container status and doing crash fallback handling, make sure service is alive
3. Transfer all containers from instance to instance
4. The flexibility of expansion by the k8s cluster
5. Open source & active community, backed by Google

kops — Kubernetes Operations

kops is a component provided from Kubernetes as an operation usage, it can help you to deploy on AWS or GCP.

First, you will need kubectl toolchain (On mac I’m using homebrew)

brew install kubernetes-cli

then can check the installation is correct

kubectl version

Once it’s set up, let’s install kops

brew update && brew install kops

After this, you already had kops installed, can type kops in the console and check if the help document shows.

AWS Configuration

If you don’t have an account yet, please go to AWS Console to signup. AWS has free tier so don’t worry if you just trying to explore.

And you also need awscli to be installed.

sudo easy_install pip && pip install awscli

and check the current version with aws --version

Now we are going to create a user for kops usage, since we will be using awscli to access the AWS resources, please choose Programmatic access, and Attach existing policies directly, with AdministratorAccess. After user creation completed, copy the value of Access key ID & Secret access key.

Back to the terminal, type aws configure and set the correct value to it.

You should be able to find the set values in $HOME/.aws

cat ~/.aws/credentials

S3 bucket

Please go to AWS S3 to create a bucket for Kubernetes, input the bucket name and assign permission to the account you just created, make sure the region is the same as what you selected on AWS panel. This bucket will store the relevant Kubernetes files.

Route53

We will need one domain registered for accessing the Drone service, for detail registration can refer to this.

Basically, after these, your configuration and preparation on AWS are all done.

Establish Kubernetes Cluster

Before doing further action for install, we need an SSH key to access for login into Kubernetes Cluster in the future.

ssh-keygen -f ~/.ssh/id_rsa will save the generated public / private key into the ~/.ssh/id_rsa, to check if the key has been generated, type cat ~/.ssh/id_rsa you should see this.

Create a cluster

kops create cluster \
--name=drone.samwang0723.com \
--state=s3://k8s-samwang0723 \
--zones=ap-southeast-1a \
--master-size=t2.micro \
--node-size=t2.micro \
--node-count=2

name — cluster name
state — cluster data storage
zone — AWS node deploy zone
master-size — EC2 instance type for master
node-size — same as master-size, setup the instance type for nodes
node-count — setup Node count

After executing the command, you should see the preview of AWS changes, to confirm the update, type below command and kops will perform the change to AWS.

kops update cluster drone.samwang0723.com --yes --state s3://k8s-samwang0723

Now, check your ~/.node/config to see the config of this Kubernetes cluster

Remember that username and password, also you will see api server configuration

server: https://api.drone.samwang0723.com

Use your account password to login this server URL, it will show all the API usage of Kubernetes.

Go back to AWS admin panel and check, you will see 3 instances being set up already (1 master 2 nodes)

Also, Route 53 recordset created as well.

Finally, in your console, check the nodes using kubectl get nodes

So far, you already have complete cluster established on AWS and you already know that kops really help developers to reduce the painful process while deployment.

Next, in Part II we will introduce how to deploy Drone to this cluster.

Side note, if you don’t want to waste money and wanted to delete everything we just setup using kops, you can use:

kops delete cluster drone.samwang0723.com — state s3://k8s-samwang0723 — yes