Rock-Solid K3s on OCI — part 1

Photo by Leon Macapagal: https://www.pexels.com/photo/aerial-view-of-city-buildings-beside-water-6622898/

Kubernetes (K8s) is a terrific orchestration engine. It canbe used in heavy-weight enterprise environments down to embedded/edge IoT use-cases (sometimes called “fog computing”). Full-blown K8s is a great choice for most users, as it offers a rich and robust feature-set in a stable and scalable format. For environments that have fewer resources available, a lightweight K8s implementation is ideal.

If you’re needing a managed K8s service, I’d highly recommend that you use OCI Container Engine (often abbreviated as OKE). It offers an enterprise-grade K8s environment that’s fully-managed, all powered by and integrated with OCI. Things like OCI load balancers, Block Storage Volumes and more can be managed for you in the OCI Container Engine. Node pools (supporting both x86_64 and aarch64 architectures), K8s version upgrades and more is easily accomplished with OCI Container Engine.

If you’re using an OCI Always Free account, you’ll need to find another option as OKE isn’t available in Free Tier accounts. With that being said, you do get a quite a lot of processing “umph” in a free tier account! It’d be a shame to let that go to waste. This could be ideal for a lightweight K8s deployment…

K3s is a lightweight Kubernetes (K8s) implementation that is designed for embedded and lightweight environments. It’s very small and imposes very minimal resource requirements. Much of this is accomplished by combining processes and eliminating unnecessary components. It uses a concept of the K3s Server, which runs the control plane, and K3s Agents, which are largely used to run containers (worker nodes). There are other lightweight K8s implementations available besides K3s, such as Minikube and MicroK8s.

There are several steps involved in getting this stood up. To make this more manageable, I’ve organized this into a multi-part series. Here’s a quick overview of what we’ll be looking at over this series:

• Introduction (part 1 — this article)
• OCI Infrastructure (part 1 — this article)
• K3s installation (part 2)
• OCI Cloud Controller Manager Installation (part 3)
• OCI Cloud Controller Storage Installation (part 3)
• Ingress Controller (part 3)
• Sample application deployment (Jupyter and Node-RED) (part 4)

At the end of the series, we’ll have deployed applications that might be used for machine learning and IoT experimentation. Of course you might use it for other applications as well! The choice is up to you. You’ll have a great lightweight K8s platform that’s able to run different applications.

Getting Started

First things first, you need to sign-up for an OCI account. Do this by going to https://cloud.oracle.com/ and clicking the “Sign Up” button. Make sure that you can login to the OCI Console and get ready to deploy K3s.

The remainder of this article is going to focus on getting the needed OCI infrastructure created and ready for us to install K3s on. Let’s get going!

OCI Infrastructure

I’d recommend using a single compartment of your choosing for all of the OCI resources. Since a picture is worth a thousand words, here is a structural view of the OCI resources needed:

Structural view of OCI resources

Start by creating a VCN, with the following attributes:

• Name: k3s
• IPv4 CIDR: 192.168.0.0/16

Here’s a logical diagram of the resources, showing the routing policies we’ll be configuring:

Logical diagram of OCI infrastructure

Create a NAT Gateway and an Internet Gateway in the VCN, along with two Route Tables:

Name: private

• Route Rule: 0.0.0.0/0 > NAT Gateway

Name: public

• Route Rule: 0.0.0.0/0 > Internet Gateway

Proceed by creating three Subnets in the newly-created VCN:

Subnet 1

• Name: k3s
• IPv4 CIDR Block: 192.168.0.0/24
• Subnet Access: Private Subnet
• Route Table: private
• DNS Label: k3s

Subnet 2

• Name: bastion
• IPv4 CIDR Block: 192.168.255.0/24
• Subnet Access: Public Subnet
• Route Table: public
• DNS Label: bastion

Subnet 3

• Name: lb
• IPv4 CIDR Block: 192.168.1.0/24
• Subnet Access: Public Subnet
• Route Table: public
• DNS Label: lb

Let’s create the network security policies that we’ll be using. To do this, we’ll setup several Network Security Groups (NSGs), as shown in the following diagram:

OCI Network Security Groups

Create the following NSGs and NSG Rules in the same VCN. Note that you’ll want to go through and create the empty NSGs first, then go back and add rules to them. This is because many of the rules are to/from NSGs (not CIDR blocks), which means the NSGs need to exist before an associated rule can reference them.

Name: servers

Rule 1:

• Direction: Egress
• IP Protocol: All Protocols
• Destination Type: Network Security Group (NSG)
• Destination NSG: servers
• Description: Permit intra-subnet traffic (outbound).

Rule 2:

• Direction: Ingress
• IP Protocol: All Protocols
• Source Type: Network Security Group (NSG)
• Source NSG: servers
• Description: Permit intra-subnet traffic (inbound).

Rule 3:

• Direction: Ingress
• IP Protocol: All Protocols
• Source Type: Network Security Group (NSG)
• Source NSG: agents
• Description: Permit traffic from agents (inbound).

Rule 4:

• Direction: Egress
• IP Protocol: All Protocols
• Destination Type: Network Security Group (NSG)
• Destination NSG: agents
• Description: Permit traffic to agents (outbound).

Rule 5:

• Direction: Ingress
• IP Protocol: TCP
• Source Type: CIDR
• Source: 192.168.255.0/24
• Destination Port Range: 22
• Description: Permit SSH (tcp/22) from bastion subnet

Name: agents

Rule 1:

• Direction: Egress
• IP Protocol: All Protocols
• Destination Type: Network Security Group (NSG)
• Destination NSG: agents
• Description: Permit intra-subnet traffic (outbound).

Rule 2:

• Direction: Ingress
• IP Protocol: All Protocols
• Source Type: Network Security Group (NSG)
• Source NSG: servers
• Description: Permit traffic from servers (inbound).

Rule 3:

• Direction: Egress
• IP Protocol: All Protocols
• Destination Type: Network Security Group (NSG)
• Destination NSG: servers
• Description: Permit traffic to servers (outbound).

Rule 4:

• Direction: Ingress
• IP Protocol: All Protocols
• Source Type: Network Security Group (NSG)
• Source NSG: agents
• Description: Permit intra-subnet traffic (inbound).

Rule 5:

• Direction: Ingress
• IP Protocol: All Protocols
• Source Type: Network Security Group (NSG)
• Source NSG: bastions

Rule 6:

• Direction: Ingress
• IP Protocol: TCP
• Source Type: CIDR
• Source: 192.168.1.0/24
• Description: Permit all TCP ports from LB subnet

Rule 7:

• Direction: Ingress
• IP Protocol: TCP
• Source Type: CIDR
• Source: 192.168.255.0/24
• Destination Port Range: 22
• Description: Permit SSH (tcp/22) from bastion subnet

Name: inet_out

Rule 1:

• Direction: Egress
• IP Protocol: TCP
• Destination Type: CIDR
• Destination: 0.0.0.0/0
• Destination Port Range: 443
• Description: Permit HTTPS (tcp/443) to the web

Rule 2:

• Direction: Egress
• IP Protocol: TCP
• Destination Type: CIDR
• Destination: 0.0.0.0/0
• Destination Port Range: 80
• Description: Permit HTTP (tcp/80) to the web

Rule 3:

• Direction: Egress
• IP Protocol: UDP
• Destination Type: CIDR
• Destination: 0.0.0.0/0
• Destination Port Range: 53
• Description: Permit DNS (udp/53) to the web

Name: bastions

Rule 1:

• Direction: Egress
• IP Protocol: All Protocols
• Destination Type: Network Security Group (NSG)
• Destination: servers
• Description: Permit outbound access to servers

Rule 2:

• Direction: Egress
• IP Protocol: All Protocols
• Destination Type: Network Security Group (NSG)
• Destination: agents
• Description: Permit outbound access to agents

Rule 3:

• Direction: Egress
• IP Protocol: All Protocols
• Destination Type: Network Security Group (NSG)
• Destination: lb
• Description: Permit outbound access to LB

Rule 4:

• Direction: Ingress
• IP Protocol: TCP
• Source Type: CIDR
• Source: <your public IP here>
• Destination Port Range: 22
• Description: Permit SSH (tcp/22) from permitted IP

Name: lb

Rule 1:

• Direction: Ingress
• IP Protocol: All Protocols
• Source Type: Network Security Group (NSG)
• Source: bastions
• Description: Permit all traffic from bastion

Rule 2:

• Direction: Egress
• IP Protocol: All Protocols
• Destination Type: Network Security Group (NSG)
• Destination: agents
• Description: Permit all traffic from LB to K3s agents

Rule 3:

• Direction: Ingress
• IP Protocol: All Protocols
• Source Type: CIDR
• Source: <your public IP here>
• Description: Permit all traffic from permitted access CIDR

Compute Instances

We are now at the point where we can create the OCI Compute Instances that we’ll be using to run K3s on! We’ve setup a lot of resources and now is the culmination, in preparation for installing K3s.

I’m going to assume that you are not using any of your computing resources in your OCI Always Free tenancy, allowing us to use them for this K3s solution.

Proceed by creating the following compute instances. For all of the instances, I am using Oracle Linux 8 and the same SSH keys. Note that OCI can generate SSH keys for you, or you can use your own SSH keys. Either way, keep track of them as you’ll be needing to use them later.

Name: bastion

• Placement: <pick an AD>
• Shape: VM.Standard.E2.1.Micro (assuming you’re using an OCI Always Free account, otherwise pick an appropriate shape)
• VCN: k3s
• Subnet: bastion
• Public IP Address: Assign a public IP address
• (click Show advanced options in the Networking box)
• Check the Use network security groups to control traffic check box, adding the following NSGs: inet_out, bastions

Name: server1

• Placement: <pick an AD>
• Shape: VM.Standard.E2.1.Micro (assuming you’re using an OCI Always Free account, otherwise pick an appropriate shape)
• VCN: k3s
• Subnet: k3s
• Public IP Address: Do not assign a public IP address
• (click Show advanced options in the Networking box)
• Check the Use network security groups to control traffic check box, adding the following NSGs: inet_out, servers

Name: agent1

• Placement: <pick an AD>
• Shape: VM.Standard.A1.Flex
• VCN: k3s
• Subnet: k3s
• Public IP Address: Do not assign a public IP address
• (click Show advanced options in the Networking box)
• Check the Use network security groups to control traffic check box, adding the following NSGs: inet_out, agents

Name: agent2

• Placement: <pick an AD>
• Shape: VM.Standard.A1.Flex
• VCN: k3s
• Subnet: k3s
• Public IP Address: Do not assign a public IP address
• (click Show advanced options in the Networking box)
• Check the Use network security groups to control traffic check box, adding the following NSGs: inet_out, agents

IAM Policies

We’ll be using Instance Principals for authentication with OCI. What this means is that one or more OCI instances will be given an identity, allowing it to make calls to the OCI API and be treated as the identity.

Create a Dynamic Group with a name of k3s_master, description of K3s master dyn group and a single Matching Rule:

All { instance.compartment.id = ‘<COMPARTMENT_OCID>’, instance.id = ‘<SERVER1_OCID>’ }

Next create an IAM policy in the compartment you’re using that references the k3s_master Dynamic Group (name: k3s, description: K3s instance principal statements). Add the following Statements to the k3s IAM policy:

allow dynamic-group k3s_master to use instance-family in compartment id <COMPARTMENT_OCID>,
allow dynamic-group k3s_master to manage virtual-network-family in compartment id <COMPARTMENT_OCID>,
allow dynamic-group k3s_master to manage load-balancers in compartment id <COMPARTMENT_OCID>,
allow dynamic-group k3s_master to manage volume-family in compartment id <COMPARTMENT_OCID>

Accessing the Instances

With all of the OCI Compute Instances created, you’ll want to SSH to the bastion, ideally enabling Agent Forwarding with the -A SSH command-line parameter (assuming you’re using MacOS or Linux). Similar configuration for enabling Agent Forwarding is available with Putty, SecureCRT or many other terminal emulators.

Here’s an example of how you’ll get to the different instances:

Your computer
      |
      |
  (SSH session with Agent Forwarding enabled)
      |
      V
bastion (OCI instance)
    |
    |
 (SSH session)
    |
    V
server1, agent1 or agent2

Here are command examples:

ssh -A -i <your SSH priv key> opc@<bastion_pub_ip>

From within this session:

ssh <priv_ip>

Where <priv_ip> is the private IP (or FQDN — see next paragraph) of either server1, agent1 or agent2. These commands (and flow) will be used heavily in the upcoming parts of this series, when we’ll be installing K3s and other components.

As a shortcut, it’s ideal to use the fully-qualified domain name (FQDN) of the OCI instances (instead of the private IP of the instance), which means you don’t have to know (or look up) the private IPs of the different OCI instances. For example:

ssh server1.k3s.k3s.oraclevcn.com
ssh agent1.k3s.k3s.oraclevcn.com
ssh agent2.k3s.k3s.oraclevcn.com

The above commands are the fast and friendly way of SSH’ng from the bastion to the different internal OCI Compute Instances.

Wrapping It Up

With a few clicks and keystrokes, we’ve been able to setup the needed OCI resources for running K3s in our OCI Always Free tenancy! In the next part of the series, we’ll setup K3s. See you in the next article.

If you’re curious about the goings-on of Oracle Developers in their natural habitat, come join us on our public Slack channel! We don’t mind being your fish bowl 🐠