OpenShift 4.3 OKD Bare Metal Install on VMWare Home Lab

Craig Robinson
Feb 28 · 17 min read


Let’s face it, the cloud is too expensive for OpenShift enthusiasts to set up a test OpenShift environment. Used hardware for a home lab that could run an OCP cluster among other things is relatively inexpensive these days, $250–$350, especially when compared to a cloud-hosted solution costing over $250 per month. The way many of us learn is through experience and with using home labs we can learn new technologies by installing, breaking, and fixing them ourselves.

Before deploying several OpenShift 3.11 clusters in a production environment, I perfected my playbooks and deployments in a home lab using the upstream version, OKD (previously known as origin). The amount of knowledge I gained by building, tearing down, and rebuilding the cluster several times has been valuable and has since been used that experience to solve production issues. When I found out in October of 2018 that OpenShift 4 was built following an immutable infrastructure principal and that when combined with CoreOS OpenShift offered OTA like updates, I was all in.

It has taken some time, but I have succeeded in building an OpenShift 4.x OKD cluster in my home lab. It can be difficult and overwhelming to design an architecture in your home lab specifically for ODK. Fortunately, I was able to come up with a design that worked and I would like to share the process with others.

Note: At the time of writing OKD is in preview and is still a work in progress so expect bugs and workarounds.

Screenshot from github.com/openshift/okd

This guide assumes you have a VMWare host, Linux terminal (Ubuntu Terminal, putty, or similar for Windows users), basic knowledge of Linux, the ability to Google, and a separate network that is not 192.168.1.0/24.

Note: If you are already using a 192.168.1.0/24 network, be sure to use a different IP scheme for your OKD environment.


VM Overview:

Note: “There are terraform scripts available to setup vSphere-enabled kubelet” -/r/roignac. I chose not to use the scripts as it added an extra layer of complexity, but they are available.


Create a new network in VMWare for OKD:

Add a new port group.
Name your Group and set your VLAN ID.

Create a pfSense VM:

Download pfSense.

Create a new Virtual Machine. Choose Guest OS as Other and Select FreeBSD 64-bit.

Use the default template settings for resources. I prefer to thin provision my disks for non-production use.

Select your home network for Network Adapter 1, and add a new network adapter using the OKD network.


Setup pfSense:

pfSense Console

Using the VMWare console on the pfSense VM, select option 8 to access the shell. Type “pfctl –d” to turn off the packet filter temporarily. Next, add a firewall rule by typing “ easyrule pass wan tcp any any 443”. This will allow you to access the pfSense web GUI from your workstation.

Source: https://docs.j7k6.org/pfsense-wan-webinterface/

Login to pfSense via your web-browser. The default username is “admin” and the password is “pfsense”.

After logging in, click next and use “okd4-pfsense” for hostname and “okd.local” for the domain. Also, enable DNS Server Override and Disable DNS Forwarder. Next.

Select your Timezone. Next.

Use Defaults for WAN Configuration. Uncheck “Block RFC1918 Private Networks” since your home network will be the “WAN” in this setup. Next.


Create bootstrap, master, and worker nodes:

The latest stable version at the time of writing is 31.20200210.3.0

Create the six ODK nodes (bootstrap, master, worker) on your ESXi host using the values in the spreadsheet:

You should end up with the following VMs:


Setup DHCP reservations:

Login into pfSense. Go to Services → DHCP Server and change your ending range IP to 192.168.1.99, then click Save.

On the DHCP Server, page click Add at the bottom.

Fill in the MAC Address, IP Address, and Hostname, then click save. Do this for each ODK VM. Click Apply Changes at the top of the page when complete.


Create and set up VM to host various services:

Download CentOS 7 minimal ISO (no GUI) and upload it to your ESXi datastore. Example: CentOS-7-x86_64-Minimal-1908.iso

Create your okd4-services virtual machine. The 20GB drive is for the OS. The 100GB drive is for NFS and will be persistent storage for the OpenShift registry. #ThinProvision

After creating the VM copy the MAC address on the VM Hardware configuration page and set up a DHCP Reservation for this VM using the IP address 192.168.1.210. Hit “Apply Changes” at the top of the DHCP page after you are done.

Power on the okd4-services VM.

Run through a typical CentOS 7 install, use the 20GB drive to install the OS on. On the networking page, it should pull the 192.168.1.210 IP address from DHCP. Set a root password and an admin user.

While CentOS is installing, go to the pfSense GUI and setup a port forward to the okd4-services for remote ssh/scp access. This will allow you to ssh from your workstation to easily copy and paste bash commands. Go to Firewall → NAT → Port Forward → Add

Destination port range “22” to “22”; redirect target IP “192.168.1.210”; redirect target port “22”

Click Save, then Apply Changes.

Reboot the okd4-services VM after installation and ssh to the WAN IP of your pfSense VM that you port forwarded to the okd4-services server.

ssh user@pfsense_ip

Install open-vm-tools, vim, wget, and git packages. Also yum update the OS, and reboot after completion.

sudo yum install –y open-vm-tools vim wget git && sudo yum update –y && sudo init 6

After updated and a restart. Use git to download the DNS, HAProxy, and install-conf.yaml example files via git:

git clone https://github.com/cragr/okd4_files.git
cd okd4_files

Install bind (DNS)

sudo yum -y install bind bind-utils

Copy the named config files and zones:

sudo cp named.conf /etc/named.conf
sudo cp named.conf.local /etc/named/
sudo mkdir /etc/named/zones
sudo cp db* /etc/named/zones

Enable and start named:

sudo systemctl enable named
sudo systemctl start named
sudo systemctl status named

Create firewall rules:

sudo firewall-cmd --permanent --add-port=53/udp
sudo firewall-cmd --reload

Set the DNS server IP to 192.168.1.210 in the pfSense DHCP server settings. Services → DHCP → Servers

Restart the network services on the okd4-services VM:

sudo systemctl restart network.service

Test DNS on the okd4-services

dig okd.local
dig –x 192.168.1.210

Install HAProxy:

sudo yum install haproxy

Copy haproxy config from the git okd4_files directory :

sudo cp haproxy.cfg /etc/haproxy/haproxy.cfg

Start, enable, and verify HA Proxy service:

sudo systemctl enable haproxy
sudo systemctl start haproxy
sudo systemctl status haproxy

Add OKD firewall ports:

sudo firewall-cmd --permanent --add-port=6443/tcp
sudo firewall-cmd --permanent --add-port=22623/tcp
sudo firewall-cmd --permanent --add-service=http
sudo firewall-cmd --permanent --add-service=https
sudo firewall-cmd --reload

Install Apache/HTTPD

sudo yum install –y httpd

Change httpd to listen port to 8080:

sudo vim /etc/httpd/conf/httpd.conf

Change “Listen 80” to “Listen 8080” around line 42:

Enable and Start httpd service/Allow port 8080 on the firewall:

sudo systemctl enable httpd
sudo systemctl start httpd
sudo firewall-cmd --permanent --add-port=8080/tcp
sudo firewall-cmd –reload

Test the webserver:

curl localhost:8080
A successful curl should look like this.

pfSense Tweaks

Add 4433 to Firewall rule on PFSense; Firewall → Rules → WAN → Add

Change PFSense default port to 4433; System → Advanced → TCP Port Change to 4433

Add 6443, 80, 443 port-forward rules to okd4-services for access from your home network.


Congratulations You Are Half Way There!


Download the openshift-installer and oc client:

To download the latest oc client and openshift-install binaries, you need to use an existing version of the oc client.

Download the oc client from the OKD releases page. Example:

wget https://github.com/openshift/okd/releases/download/4.3.0-0.okd-2019-11-15-182656/openshift-client-linux-4.3.0-0.okd-2019-11-15-182656.tar.gz

Extract the okd version of the oc client:

tar -zxvf openshift-client-linux-4.3.0-0.okd-2019-11-15-182656.tar.gz

Move the kubectl and oc to /usr/local/bin and show the version:

sudo mv kubectl oc /usr/local/bin/
oc version

The latest releases are listed on https://origin-release.svc.ci.openshift.org/.

The version I installed was 4.3, specifically 4.3.0–0.okd-2020–02–29–000821

oc adm release extract --tools registry.svc.ci.openshift.org/origin/release:4.3

This downloads two tar.gz files, one containing a new version of the oc client and the openshift-install binary.

Extract and move the new version of the oc client, openshift-installer and kubectl. Verify the version:

tar -zxvf openshift-client-linux-4.3.0-0.okd-2020-02-29-000821.tar.gztar -zxvf openshift-install-linux-4.3.0-0.okd-2020-02-29-000821.tar.gzsudo mv oc kubectl openshift-install /usr/local/bin/oc version

Setup the openshift-installer:

Optional: Go to https://try.openshift.com/ and get a pull secret from RedHat. If you do not have an account I recommend creating a no-cost developer account.

Create an install directory and copy the install-config.yaml file:

cd ~
mkdir install_dir
cp okd_files/install-config.yaml ./install_dir

Edit the install-config.yaml in the install_dir, insert your pull secret and ssh key, and backup the install-config.yaml as it will be deleted in the next step:

sudo vim ./install_dir/install-config.yamlcp ./install_dir/install-config.yaml ./install_dir/install-config.yaml.bak

Generate the Kubernetes manifests for the cluster, ignore the warning:

openshift-install create manifests --dir=install_dir/

Modify the cluster-scheduler-02-config.yaml manifest file to prevent Pods from being scheduled on the control plane machines:

vim install_dir/manifests/cluster-scheduler-02-config.yml

Locate the mastersSchedulable parameter and change the value to False:

Now you can create the ignition-configs:

openshift-install create ignition-configs — dir=install_dir/

Note: If you reuse the install_dir make sure it is empty. Hidden files are created after generating the configs and they need to be removed if you use the same folder on a 2nd attempt.


Host ignition and Fedora CoreOS files on the webserver:

sudo mkdir /var/www/html/okd4

Copy the install_dir contents to /var/www/html/okd4:

sudo cp -R install_dir/* /var/www/html/okd4/

Test the webserver:

curl localhost:8080/okd4/metadata.json

Download the Fedora CoreOS bare-metal bios image and sig files and shorten the file names:

cd /var/www/html/okd4/sudo wget https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/31.20200210.3.0/x86_64/fedora-coreos-31.20200210.3.0-metal.x86_64.raw.xzsudo wget https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/31.20200210.3.0/x86_64/fedora-coreos-31.20200210.3.0-metal.x86_64.raw.xz.sigsudo mv fedora-coreos-31.20200210.3.0-metal.x86_64.raw.xz fcos.raw.xzsudo mv fedora-coreos-31.20200210.3.0-metal.x86_64.raw.xz.sig fcos.raw.xz.sig

Begin the Install:

sudo rm -rf install_dir/*

If you are concerned you may have a bad install and want to start with a clean install, you can.

To start a clean install, edit the VM and delete the HD on the nodes that have gone through the install process and click save to remove the drive. Next, edit the VM again and add a new drive and click save. Once the drive has been removed and re-added, start back from this point of powering on the bootstrap and try again.


Staring the bootstrap node:

coreos.inst.install_dev=sdacoreos.inst.image_url=http://192.168.1.210:8080/okd4/fcos.raw.xzcoreos.inst.ignition_url=http://192.168.1.210:8080/okd4/bootstrap.ign

You should see that the fcos.raw.gz image and signature have been downloaded:

After the image is installed, the VM will restart to the login screen and begin to show the following (do not panic, this is normal):

The console window will show a lot of messages during the install. At this point, I recommend connecting to the bootstrap node via SSH and watching the bootkube service logs. You can ssh from your okd4-servers node by using the following command:

ssh -i ~/.ssh/rsa_id core@192.168.1.200

Upon login you will see this screen:

To watch the bootkube logs enter this command:

journalctl -b -f -u bootkube.service

The logs should look similar to this:

Keep an eye out for errors. This is updating the rpm-ostree and will take 3–10 minutes. The VM will reboot once completed and end your ssh session.

After the VM reboots reconnect to your bootstrap node via SSH again. Notice this time there is an error with the crio service. This appears to be an issue with the 4.3 branch but we will work around it:

Edit the /etc/crio/crio.conf file

sudo vi /etc/crio/crio.conf

Edit the line that sets the OCI hooks directories and change “/usr/share/containers/oci/hooks.d” to “/etc/containers/oci/hooks.d” and save.

To verify crio is now working do systemctl status crio and you should see the service is now active:

sudo systemctl status crio

Now check the logs on the bootkube service:

journalctl -b -f -u bootkube.service

You will see it failing to check the etcd nodes and the logs report the cluster is unhealthy which means you are ready to power on the control nodes.

Starting the control plane nodes:

coreos.inst.install_dev=sdacoreos.inst.image_url=http://192.168.1.210:8080/okd4/fcos.raw.xzcoreos.inst.ignition_url=http://192.168.1.210:8080/okd4/master.ign

Just like the bootstrap node you should that the fcos.raw.gz image and signature have been downloaded:

After the image is installed, the VM will restart to the login screen, begin to update the ostree, and then reboot once completed just like the bootstrap node:

After the 2nd reboot, you will need to ssh into the node from the okd4-services node and edit the crio.conf. Remove the line “/usr/share/containers/oci/hooks.d” and save your changes.

Now repeat the same process for the other two control nodes.

After the other two nodes have reached this point, you can monitor the bootstrap process from the okd4-services node:

openshift-install --dir=install_dir/ wait-for bootstrap-complete --log-level=info

Once the installer shows it is safe to remove the bootstrap node it is time to boot the compute nodes.

Note: At this point, you can shutdown your bootstrap node. Now is a good time to edit the /etc/haproxy/haproxy.cfg, comment out the bootstrap node, and restart the haproxy service.

sudo vi /etc/haproxy/haproxy.cfg
sudo systemctl restart haproxy

Starting the compute nodes:

coreos.inst.install_dev=sdacoreos.inst.image_url=http://192.168.1.210:8080/okd4/fcos.raw.xzcoreos.inst.ignition_url=http://192.168.1.210:8080/okd4/worker.ign

Just like the other nodes you should that the fcos.raw.gz image and signature have been downloaded:

After the image is installed, the VM will restart to the login screen, begin to update the ostree, and then reboot once completed just like the other nodes:

After the 2nd reboot, you will need to ssh into the node from the okd4-services node and edit the crio.conf. Remove the line “/usr/share/containers/oci/hooks.d” and save your changes.

Now repeat the same process for the other worker node.


Login to the cluster and approve CSRs:

export KUBECONFIG=./install_dir/auth/kubeconfig
oc whoami
oc get nodes
oc get csr

You should only see the master nodes but plenty of csr’s waiting for approval. Install the jq package to assist with approving multiple csr’s at once time.

wget -O jq https://github.com/stedolan/jq/releases/download/jq-1.6/jq-linux64chmod +x jq
sudo mv jq /usr/local/bin/
jq --version

Approve all the pending certs and check your nodes:

oc get csr -ojson | jq -r '.items[] | select(.status == {} ) | .metadata.name' | xargs oc adm certificate approve

Log in to the web console using your workstation. Before you do, you will need to add console-openshift-console.apps.lab.okd.local to your workstation’s host file:

<workstation ip addr> console-openshift-console.apps.lab.okd.local

Also, get your kubeadmin password from the install_dir/auth folder.

cat install_dir/auth/kubeadmin-password

Check out that beautiful dashboard. Take a break and look around the admin panel.

The only items remaining to be addressed are operators, persistent storage, and configuration of the registry.


Checking on Operators:

oc get clusteroperators

Persistent Storage:

Login to your okd4-services VM and being to set up an NFS server. The following commands install the necessary packages, enable services, and configure file and folder permissions.

sudo yum install –y nfs-utils
sudo systemctl enable nfs-server rpcbind
sudo systemctl start nfs-server rpcbind nfs-lock-nfs-idmap
sudo mkdir –p /var/nfsshare/registry
sudo chmod –R 777 /var/nfsshare
sudo chown –R nfsnobody:nfsnobody /var/nfsshare

Create an NFS Export

sudo vi /etc/exports

Add this line in the new /etc/exports file “/var/nfsshare 192.168.1.0/24(rw,sync,no_root_squash,no_all_squash,no_wdelay)”

Restart the nfs-server service and add firewall rules:

sudo systemctl restart nfs-server
sudo firewall-cmd --permanent --zone=public --add-service mountd
sudo firewall-cmd --permanent --zone=public --add-service rpc-bind
sudo firewall-cmd --permanent --zone=public --add-service nfs
sudo firewall-cmd --reload

Registry configuration:

oc create -f okd4_files/registry_pv.yaml
oc get pv

Edit the image-registry operator:

oc edit configs.imageregistry.operator.openshift.io

Change the managmentState: from Removed to Managed. Under storage: add the pvc: and claim: blank to automatically attach the PV and save your changes:

managementStage: Managedstorage:
pvc:
claim:

Check your persistent volume, it should have been claimed:

oc get pv

To test the registry and underlying storage, create a new php project in OKD and use the example repository. After it has been built you can check your NFS export on okd4-services for data:

The end.

Known Issues:

The cluster operator machine-config error shown below is a known issue in OKD 4.3 and has been resolved in OKD 4.4 which is the recommended version.

 by the author.

Craig Robinson

Written by

OpenShift Enthusiast

More From Medium

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight.
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox.
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month.