Guide: OKD 4.5 Single Node Cluster

Craig Robinson
Jul 15 · 12 min read

Updated: 7/29/2020

After listening to some feedback in the chat on a recent Twitch stream and the okd-wg mailing list I decided to create a guide for installing an OKD 4.5 SNC (single node cluster). This guide will use no worker nodes and only a single control-plane node with the goal of reducing the total amount of resources needed to get OKD up and running for a test drive.

VM Overview:

For this installation, I used an ESXi 6.5 host on my local network and allocated the total virtual machine RAM usage to 22GB. ESXi 6.5 has a minimum of 4GB of RAM but you should be able to run this on an ESXi host with 24GB of RAM. Here is a breakdown of the virtual machines and network layout:

Create the okd4-services VM:

Download the Fedora Server Standard ISO and upload it to your ESXi host datastore.

Create a new Virtual Machine. Choose Guest OS as Linux and Select CentOS 7 (64-bit).

Select the Datastore and customize the settings to 2 vCPU, 2GB RAM, 100GB HD. Attach the Fedora Serer ISO to the CD/DVD drive.

Review your settings and click Finish.

Install Fedora Server 32 on the okd4-services VM:

Run through the installation.

I prefer to use the “Standard Partition” storage configuration to use the entire drive at “/”. On the “Installation Destination” page, click on Custom under Storage Configuration, then Done.

On the Manual Partitioning page, select Standard Partition, then click “Click Here to Create them automatically.”

Select the “/” partition and click the “-” to delete it. Then click the “+” to re-add it. At this point, it should use the entire drive. Click Done, then Accept the changes.

For Software Selection, choose “Fedora Custom Operating System” and include the “Guest Agents” add-on.

Enable the NIC connected to the VM Network and set the hostname as okd4-services, then click Apply and Done.

Set the Root password, and create an admin user. Click “Begin Installation” to start the install.

After the installation has completed, login, and update the OS.

sudo dnf update -y
sudo dnf install -y git wget vim
sudo init 6

Create bootstrap and master nodes:

Download the Fedora CoreOS Bare Metal ISO and upload it to your ESXi datastore.

The latest stable version at the time of writing is 32.20200715.3.0

Create the two ODK nodes (okd4-bootstrap and okd4-control-plane-1) on your ESXi host using the values in the spreadsheet at the beginning of this post:

okd4-bootstrap:

okd4-control-plane-1

You should end up with the following VMs:

Configure okd4-services VM to host various services:

The okd4-services VM is used to provide DNS, NFS exports, web server, and load balancing.

Open a terminal on the okd4-services VM and clone the okd4_files repo (branch snc) that contains the DNS, HAProxy, and install-conf.yaml example files:

cd
git clone -b snc https://github.com/cragr/okd4_files.git
cd okd4_files

Install bind (DNS)

sudo dnf -y install bind bind-utils

Copy the named config files and zones:

sudo cp named.conf /etc/named.conf
sudo cp named.conf.local /etc/named/
sudo mkdir /etc/named/zones
sudo cp db* /etc/named/zones

Enable and start named:

sudo systemctl enable named
sudo systemctl start named
sudo systemctl status named

Create firewall rules:

sudo firewall-cmd --permanent --add-port=53/udp
sudo firewall-cmd --reload

Change the DNS on the okd4-service to 127.0.0.1:

sudo nmcli connection modify ens192 ipv4.dns "127.0.0.1"

Restart the network services on the okd4-services VM:

sudo systemctl restart NetworkManager

Test DNS on the okd4-services.

dig okd.local
dig –x 192.168.60.240

With DNS working correctly, you should see the following results:

Change your workstation's primary DNS to match the okd4-services IP, 192.168.60.240 so you can resolve the DNS entries.

Install HAProxy:

sudo dnf install haproxy -y

Copy haproxy config from the git okd4_files directory :

sudo cp haproxy.cfg /etc/haproxy/haproxy.cfg

Start, enable, and verify HA Proxy service:

sudo setsebool -P haproxy_connect_any 1
sudo systemctl enable haproxy
sudo systemctl start haproxy
sudo systemctl status haproxy

Add OKD firewall ports:

sudo firewall-cmd --permanent --add-port=6443/tcp
sudo firewall-cmd --permanent --add-port=22623/tcp
sudo firewall-cmd --permanent --add-service=http
sudo firewall-cmd --permanent --add-service=https
sudo firewall-cmd --reload

Install Apache/HTTPD

sudo dnf install -y httpd

Change httpd to listen port to 8080:

sudo sed -i 's/Listen 80/Listen 8080/' /etc/httpd/conf/httpd.conf

Enable and Start httpd service/Allow port 8080 on the firewall:

sudo setsebool -P httpd_read_user_content 1
sudo systemctl enable httpd
sudo systemctl start httpd
sudo firewall-cmd --permanent --add-port=8080/tcp
sudo firewall-cmd --reload

Test the webserver:

curl localhost:8080

A successful curl should look like this:

Congratulations, You Are Half Way There!

Download the openshift-installer and oc client:

SSH to the okd4-services VM

To download the latest oc client and openshift-install binaries, you need to use an existing version of the oc client.

Download the 4.5 version of the oc client and openshift-install from the OKD releases page. Example:

cd
wget https://github.com/openshift/okd/releases/download/4.5.0-0.okd-2020-07-29-070316/openshift-client-linux-4.5.0-0.okd-2020-07-29-070316.tar.gz
wget https://github.com/openshift/okd/releases/download/4.5.0-0.okd-2020-07-29-070316/openshift-install-linux-4.5.0-0.okd-2020-07-29-070316.tar.gz

Extract the okd version of the oc client and openshift-install:

tar -zxvf openshift-client-linux-4.5.0-0.okd-2020-07-29-070316.tar.gz
tar -zxvf openshift-install-linux-4.5.0-0.okd-2020-07-29-070316.tar.gz

Move the kubectl, oc, and openshift-install to /usr/local/bin and show the version:

sudo mv kubectl oc openshift-install /usr/local/bin/
oc version
openshift-install version

The latest and recent releases are available at https://origin-release.svc.ci.openshift.org

Setup the openshift-installer:

In the install-config.yaml, you can either use a pull-secret from RedHat or the default of “{“auths”:{“fake”:{“auth”: “bar”}}}” as the pull-secret.

Generate an SSH key if you do not already have one.

ssh-keygen

Create an install directory and copy the install-config.yaml file:

cd
mkdir install_dir
cp okd4_files/install-config.yaml ./install_dir

Edit the install-config.yaml in the install_dir, add your ssh key, and edit the controlPlane master replicas to 1. Backup the install-config.yaml as it will be removed when generating the manifests:

vim ./install_dir/install-config.yaml
cp ./install_dir/install-config.yaml ./install_dir/install-config.yaml.bak

Generate the Kubernetes manifests for the cluster, ignore the warning:

openshift-install create manifests --dir=install_dir/

Now you can create the ignition-configs:

openshift-install create ignition-configs --dir=install_dir/

Note: If you reuse the install_dir, make sure it is empty. Hidden files are created after generating the configs, and they should be removed before you use the same folder on a 2nd attempt.

Host ignition and Fedora CoreOS files on the webserver:

Create okd4 directory in /var/www/html:

sudo mkdir /var/www/html/okd4

Copy the install_dir contents to /var/www/html/okd4 and set permissions:

sudo cp -R install_dir/* /var/www/html/okd4/
sudo chown -R apache: /var/www/html/
sudo chmod -R 755 /var/www/html/

Test the webserver:

curl localhost:8080/okd4/metadata.json

Download the Fedora CoreOS bare-metal bios image and sig files and shorten the file names:

cd /var/www/html/okd4/
sudo wget https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/32.20200715.3.0/x86_64/fedora-coreos-32.20200715.3.0-metal.x86_64.raw.xz
sudo wget https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/32.20200715.3.0/x86_64/fedora-coreos-32.20200715.3.0-metal.x86_64.raw.xz.sig
sudo mv fedora-coreos-32.20200715.3.0-metal.x86_64.raw.xz fcos.raw.xz
sudo mv fedora-coreos-32.20200715.3.0-metal.x86_64.raw.xz.sig fcos.raw.xz.sig
sudo chown -R apache: /var/www/html/
sudo chmod -R 755 /var/www/html/

Starting the bootstrap node:

NOTE: I recommend you use a DHCP lease for both the bootstrap and control-plane nodes if possible instead of the line below beginning with “ip=” which will set static IP addresses.

Power on the odk4-bootstrap VM. Press the TAB key to edit the kernel boot options. Once completed, press Enter to boot.

ip=192.168.60.242::192.168.60.1:255.255.255.0:::none nameserver=192.168.60.240 coreos.inst.install_dev=/dev/sda coreos.inst.image_url=http://192.168.60.240:8080/okd4/fcos.raw.xz coreos.inst.ignition_url=http://192.168.60.240:8080/okd4/bootstrap.ign

You should see that the fcos.raw.xz image and signature are downloading:

Starting the control-plane-1 node:

Power on the odk4-control-plane-1 VM. Press the TAB key to edit the kernel boot options. Once completed, press Enter to boot.

ip=192.168.60.241::192.168.60.1:255.255.255.0:::none nameserver=192.168.60.240 coreos.inst.install_dev=/dev/sda coreos.inst.image_url=http://192.168.60.240:8080/okd4/fcos.raw.xz coreos.inst.ignition_url=http://192.168.60.240:8080/okd4/master.ign

You should see that the fcos.raw.xz image and signature are downloading:

Monitor the bootstrap installation:

You can monitor the bootstrap process from the okd4-services node:

openshift-install --dir=install_dir/ wait-for bootstrap-complete --log-level=info

Once the bootstrap process is complete, which can take upwards of 30 minutes, you can shutdown your bootstrap node. Now is a good time to edit the /etc/haproxy/haproxy.cfg, comment out the bootstrap node, and reload the haproxy service.

sudo sed -i '/ okd4-bootstrap /s/^/#/' /etc/haproxy/haproxy.cfg
sudo systemctl reload haproxy

Login to the cluster and check the status:

Now that the control-plane node is online, you should be able to login with the oc client. Use the following commands to log in and check the status of your cluster:

export KUBECONFIG=~/install_dir/auth/kubeconfig
oc whoami
oc get nodes

Check the status of the cluster operators.

oc get clusteroperators

Once the console operator is available login to the web console.Get your kubeadmin password from the install_dir/auth folder:

cat install_dir/auth/kubeadmin-password

Open your web browser to https://console-openshift-console.apps.lab.okd.local/ and login as kubeadmin with the password from above:

The cluster status may still say upgrading, and it continues to finish the installation.

Persistent Storage:

We need to create some persistent storage for our registry before we can complete this project. Let’s configure our okd4-services VM as an NFS server and use it for persistent storage.

Login to your okd4-services VM and begin to set up an NFS server. The following commands install the necessary packages, enable services, and configure file and folder permissions.

sudo dnf install -y nfs-utils
sudo systemctl enable nfs-server rpcbind
sudo systemctl start nfs-server rpcbind
sudo mkdir -p /var/nfsshare/registry
sudo chmod -R 777 /var/nfsshare
sudo chown -R nobody:nobody /var/nfsshare

Create an NFS Export

Add this line in the new /etc/exports file “/var/nfsshare 192.168.60.0/24(rw,sync,no_root_squash,no_all_squash,no_wdelay)”

echo '/var/nfsshare 192.168.60.0/24(rw,sync,no_root_squash,no_all_squash,no_wdelay)' | sudo tee /etc/exports

Restart the nfs-server service and add firewall rules:

sudo setsebool -P nfs_export_all_rw 1
sudo systemctl restart nfs-server
sudo firewall-cmd --permanent --zone=public --add-service mountd
sudo firewall-cmd --permanent --zone=public --add-service rpc-bind
sudo firewall-cmd --permanent --zone=public --add-service nfs
sudo firewall-cmd --reload

Registry configuration:

Create a persistent volume on the NFS share. Use the registry_py.yaml in okd4_files folder from the git repo:

oc create -f okd4_files/registry_pv.yaml
oc get pv

Edit the image-registry operator:

oc edit configs.imageregistry.operator.openshift.io

Change the managementState: from Removed to Managed. Under storage: add the pvc: and claim: blank to attach the PV and save your changes automatically:

managementState: Managedstorage:
pvc:
claim:

Check your persistent volume, and it should now be claimed:

oc get pv

Check the export size, and it should be zero. In the next section, we will push to the registry, and the file size should not be zero.

du -sh /var/nfsshare/registry

In the next section, we will create a WordPress project and push it to the registry. After the push, the NFS export should show 200+ MB.

Create WordPress Project:

Create a new OKD project.

oc new-project wordpress-test

Create a new app using the centos php73 s2i image from docker hub and use the WordPress GitHub repo for the source. Expose the service to create a route.

oc new-app centos/php-73-centos7~https://github.com/WordPress/WordPress.git
oc expose svc/wordpress

Create a new app using the centos7 MariaDB image with some environment variables:

oc new-app centos/mariadb-103-centos7 --name mariadb --env MYSQL_DATABASE=wordpress --env MYSQL_USER=wordpress --env MYSQL_PASSWORD=wordpress

Open the OpenShift console and browse to the WordPress-test project. Once the WordPress image is built and ready, it will be dark blue like the MariaDB instance show here:

Click on the WordPress object and click on the route to open it in your web browser:

You should see the WordPress setup config, click Let’s Go.

Fill in the database, username, password, and database host as pictured and run the installation:

Fill out the welcome information and click Install WordPress.

Log in, and you should have a working WordPress installation:

Check the size of your NFS export on the okd4-services VM. It should be around 300MB in size.

du -sh /var/nfsshare/registry/

You have just verified your persistent volume is working.

HTPasswd Setup:

The kubeadmin is a temporary user. The easiest way to set up a local user is with htpasswd.

cd
cd okd4_files
htpasswd -c -B -b users.htpasswd testuser testpassword

Create a secret in the openshift-config project using the users.htpasswd file you generated:

oc create secret generic htpass-secret --from-file=htpasswd=users.htpasswd -n openshift-config

Add the identity provider.

oc apply -f htpasswd_provider.yaml

Logout of the OpenShift Console. Then select htpasswd_provider and login with testuser and testpassword credentials.

If you visit the Administrator page you should see no projects:

Give yourself cluster-admin access, and the projects should immediately populate:

oc adm policy add-cluster-role-to-user cluster-admin testuser

Your user should now have cluster-admin level access:

Congrats! You have created an OKD Single Node Cluster!

Hopefully, you have created an OKD single node cluster and learned a few things along the way. At this point, you should have a decent basis to tinker with OKD and continue to learn.

Here are some resources available to help you along your journey:

To report issues, use the OKD Github Repo: https://github.com/openshift/okd

For support check out the #openshift-users channel on k8s Slack

The OKD Working Group meets bi-weekly to discuss the development and next steps. The meeting schedule and location are tracked in the openshift/community repo.

Google group for okd-wg: https://groups.google.com/forum/#!forum/okd-wg

The Startup

Medium's largest active publication, followed by +729K people. Follow to join our community.

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store