IBM Cloud Private Offline Installation

Published in

IBM Cloud

10 min readMay 2, 2018

By Francis Nazareth, Cloud Architect, IBM & Haytham Elkhoja, Cloud Platform Architect, IBM.

Enterprises around the world are revamping the way they build applications by adopting DevOps, Microservices, Cloud and Containers. But more often than not, enterprise customers working in highly regulated industries mandate a disconnected, private and air-gapped cloud — a cloud solution that is not connected to the internet.

IBM Cloud Private is a modular platform built on top of Kubernetes, Cloud Foundry and Terraform to help enterprises endorse a cloud-native approach to development and deployment of applications and rapidly bring app ideas to market. For more info on IBM Cloud Private (ICP) check out this post.

Goals

This article has 2 goals:

Demonstrate how to install and deploy IBM Cloud Private (ICP) in an offline enterprise environment (with no internet connectivity).
Populate the IBM Cloud Private App Catalog with offline Docker images to the private Docker registry and the private Helm chart repository.

The following sections go through installing IBM Cloud Private 2.1.0.2 on Red Hat Enterprise Linux 7.4 and publishing local Helm charts in the App Catalog.

There are 2 cases: Case 1: For IBM Cloud Private Cloud Native and Enterprise Editions and Case 2: For IBM Cloud Private Community Edition.

Assumptions and guidance

This article assumes that:

You have a common understanding of IBM Cloud Private. Please review the overview, architecture and components sections of the IBM Cloud Private documentation in IBM Knowledge Center.
You have an internet connected laptop or host somewhere that you can use to git clone and do some docker pull.
You installed Red Hat Enterprise Linux 7.4 and have a valid ISO image or DVD medium at your disposal.
Enough disk space on /opt and /var. For more information on hardware and storage requirements, click here.
You are a root. If not, all operations in this article should performed by a privileged user by prepending all commands with sudo.
Look out for the target of each command in the command line prompt. Some commands need to be applied on all nodes, others on the boot node and others on your internet connected laptop or host.

Installation

Download the Installer

Case 1: For IBM Cloud Private Cloud Native and Enterprise Editions:

You must have a valid IBM Passport Advantage Subscription. Download the IBM Cloud Private 2.1.0.2 Cloud Native or Enterprise editions ibm-cloud-private-x86_64-2.1.0.2.tar.gz file and the supplied IBM Cloud Private Docker binary package icp-docker-17.09_x86_64.bin provided by IBM.

Case 2: For IBM Cloud Private Community Edition:

To install the IBM Cloud Private Community Edition, the ICP installer is pulled via Dockerhub and saved on your laptop. Then, it is copied to the offline boot node. Run this command from your internet connected laptop:

user@macbookpro:~$ sudo docker pull ibmcom/icp-inception:2.1.0.2 && sudo docker save -o icp-inception.tar ibmcom/icp-inception:2.1.0.2

Regardless of the medium that you want to use, whether it’s a Red Hat Enterprise Linux 7.4 DVD or an ISO image, you must mount it on your system.

Mount the RHEL DVD drive or ISO Image

Run the following command, substituting the name of your CDROM/DVDROM device (mine is /dev/sr0).

Repeat the following task on all the nodes.

DVD

user@allnodes.icp:~$ sudo mount /dev/sr0 /media/cdrom

ISO Image

user@allnodes.icp:~$ sudo mount -t iso9660 -o loop path/to/image.iso /media/cdrom

Create a local Yum repository

Repeat the following task on all the nodes.

user@allnodes.icp:~$ sudo vi /etc/yum.repos.d/RHELDISC.repo[RHELDISC]
name=RHEL_7.4_x86_64_Disc
baseurl="file:///media/cdrom" -- or /media/iso
gpgcheck=0

Run yum repolist to validate the list of repositories, including the local repo you’ve just created.

Install other prerequisites

Policycore utilities is a library needed by Docker. Install policycoreutils before installing Docker. This should use the local Yum repository previously created. Repeat the following task on all the nodes.

user@allnodes.icp:~$ sudo yum install policycoreutils-python.x86_64

Prepare your cluster

Follow the steps here to prepare your cluster and nodes. Repeat the following tasks on all the nodes. This will involve:

Editing /etc/hosts.
Because IBM Cloud Private uses Ansible and SSH to install itself on all the nodes, from the boot node do an ssh user@<node> to all the nodes and to itself (localhost). This will add the nodes on the boot node’s ~/.ssh/known_hosts and clear the message The authenticity of host ‘xxx.xxx.xxx.xxx (xxx.xxx.xxx.xxx)’ can’t be established.
Installing python and python-pip.
Making sure NTP is deployed on all the nodes and time is sync’ed.
Tweaking some Kernel parameters in /etc/sysctl.conf.

Disable the Firewall

Repeat the following tasks on all the nodes.

user@allnodes.icp:~$ sudo systemctl stop firewalld
user@allnodes.icp:~$ sudo systemctl disable firewalld

Install Docker and load the IBM Cloud Private installation images to Docker

Case 1: For IBM Cloud Private Cloud Native and Enterprise Editions:

Install Docker by running the following commands on all the nodes (not just the boot node). Please note that any dependencies will be resolved against the local yum repository.

user@allnodes.icp:~$ sudo chmod +x icp-docker-17.09_x86_64.bin
user@allnodes.icp:~$ sudo ./icp-docker-17.09_x86_64.bin –install

On your boot node, load the installer into Docker:

user@bootnode.icp:~$ sudo tar xvf ibm-cloud-private-x86_64-2.1.0.2.tar.gz -O | docker load

Case 2: For IBM Cloud Private Community Edition:

Follow this link to download the Docker CE daemon and client binaries to your laptop. Copy the binaries with their prerequisites to all the nodes (not just the boot node), and continue the installation steps of Docker CE.

On your boot node, load the installer into Docker:

user@bootnode.icp:~$ sudo docker load -i ~/icp-inception.tar

Create folders for the ICP installation and extract the sample configuration file from the installer image:

user@bootnode:~$ sudo mkdir /opt/ibm-cloud-private-2.1.0.2; \ cd /opt/ibm-cloud-private-2.1.0.2

Case 1: For IBM Cloud Private Cloud Native and Enterprise Editions:

user@bootnode.icp:~$ sudo docker run -v $(pwd):/data -e LICENSE=accept \ ibmcom/icp-inception:2.1.0.2-ee \ cp -r cluster /data

Case 2: For IBM Cloud Private Community Edition:

user@bootnode.icp:~$ sudo docker run -v $(pwd):/data -e LICENSE=accept \ ibmcom/icp-inception:2.1.0.2 \ cp -r cluster /data

Further prepare the installation

Follow the steps here to further prepare the installation. This will involve:

Generating SSH keys to allow for password-less SSH connections for Ansible. Follow the steps here.
Copying the SSH private key to /opt/ibm-cloud-private-2.1.0.2/cluster/ssh_key. Note that your copied private key is ssh_key and proper permissions should be applied.
Copying the installer to /opt/ibm-cloud-private-2.1.0.2/cluster/images/.
Customize and edit the IBM Cloud Private Installer files ./cluster/config.yaml and ./cluster/hosts.

Installing IBM Cloud Private

You’re now ready to deploy IBM Cloud Private. From the boot node, run the following command:

user@bootnode.icp:~$ sudo docker run --net=host -t -e LICENSE=accept \ -v "$(pwd)":/installer/cluster ibmcom/icp-inception:2.1.0.2-ee install

Alternatively, I prefer to perform my installation using screen to allow the installer to continue in case my terminal is killed or disconnected and verbosing as much as I can, then outputting the logs to an install.log file for debugging.

user@bootnode.icp:~$ screen sudo docker run --net=host -t -e LICENSE=accept \ -v "$(pwd)":/installer/cluster ibmcom/icp-inception:2.1.0.2-ee install -vvv | tee install.log

After the installation is complete, you should hopefully see this message:

UI URL is https://<ip_address>:8443 , default username/password is xxx/xxx

Install the CLI tools

There are 4 CLI tools that you need to use to continue your offline installation: kubectl, helm, the IBM Bluemix CLI bx and the bx icp plugin.

Install the kubectl command

Case 1: For IBM Cloud Private Cloud Native and Enterprise Editions:

user@bootnode.icp:~$ sudo docker run -e LICENSE=accept --net=host -v /usr/local/bin:/data ibmcom/icp-inception:2.1.0.2-ee cp /usr/local/bin/kubectl /data

Case 2: For IBM Cloud Private Community Edition:

user@bootnode.icp:~$ sudo docker run -e LICENSE=accept --net=host -v /usr/local/bin:/data ibmcom/icp-inception:2.1.0.2 cp /usr/local/bin/kubectl /data

The preceding commands will copy the kubectl command into /usr/local/bin/kubectl. To test kubectl, open your installed IBM Cloud Private https://<ip_address>:8443, browse to the Configure client link on the top right side, and then copy and paste the kubectl config ... commands on the boot node command line prompt.

Install the helm command

The IBM Cloud Private installation ships with helm. In the example below, you’ll be downloading the Linux (64-bit) version of the command.

On your boot node, download it from the local /helm-api/cli URL.

user@bootnode.icp:~$ curl --insecure -O https://mycluster.icp:8443/helm-api/cli/linux-amd64/helm
user@bootnode.icp:~$ sudo mv helm /usr/local/bin/
user@bootnode.icp:~$ sudo chmod  +x /usr/local/bin/helm

You also must download the helm command on your laptop or internet connected host: (assuming that your laptop can reach the master node to download the file, if not, then copy it via USB or another method). This will help us for later to package Helm charts for offline use.

If you’re on a macOS/Linux variant or on Windows with Chocolatey, you can download helm using the following commands:

macOS with brew
user@macbookpro:~$ brew install kubernetes-helmWindows with Chocolatey
C:\Users/user> choco install kubernetes-helm

Or simply follow the steps to download the Helm binaries here.

Install the bx command

From your laptop, download the Bluemix CLI from here, then transfer the downloaded archive file to the offline machine.
From the boot node, unzip the tar.gz file and install.

user@bootnode.icp:~$ sudo tar -zxvf IBM_Cloud_CLI_0.6.6_amd64.tar.gz
user@bootnode.icp:~$ sudo ./install_bluemix_cli

Install the bx cloud private plugin

On your newly installed IBM Cloud Private instance, browse to Command Line Tools > Cloud Private CLI.
Download the plugin for your operating system/architecture. In our case, it’s Linux (64-bit). An icp-linux-amd64 binary file should start downloading. Transfer the downloaded binary file to the offline machine.

From the boot node, install the downloaded plugin using the bx command

user@bootnode.icp:~$ bx plugin install icp-linux-amd64

Now try logging in to your IBM Cloud Private instance using the bx command

user@bootnode.icp:~$ bx pr login --skip-ssl-validation -a https://<ip_address>:8443
username: xxx
password: xxx

Updating Helm Charts to use the local Docker registry

Because your environment is not connected to the internet, the App Catalog will be empty. The App Catalog uses Helm chart repositories to populate Docker based applications that can be configured and deployed in seconds. By default, the App Catalog has 2 repositories, the publicly accessible Helm chart repo located at https://github.com/IBM/charts and a local one, which is empty. Let’s populate it.

Here are the steps that you will perform from your internet connected laptop or host:

Pull the Docker images locally on your laptop, and create a local copy of necessary Docker images.
Tag these images (for example, a cluster named mycluster.icp and for the namespace default, an image named Jenkins needs to be tagged as mycluster.icp:8500/default/jenkins.
Push these Docker images to IBM Cloud Private’s private Docker registry.
Change the Helm charts to use Docker images from IBM Cloud Private’s private docker registry.
Update the Helm charts’ values.yaml files to the locations of the private Docker registry.

This process can be tedious for a big number of images. To simplify the process, we have written a few shell scripts to automate all of this.

Download ibm-helm-offline from github.com

From your laptop, do the following:

user@macbookpro:~$ git clone https://github.com/fnazaret/ibm-helm-offline.git
user@macbookpro:~$ cd ibm-helm-offline/
user@macbookpro:~/ibm-helm-offline$ chmod +x getRepos.sh

Edit DockerTags.txt

Open the DockerTags.txt file. Edit the images, change the versions of the images if needed, or add more images from different Docker Hub repositories.

user@macbookpro:~/ibm-helm-offline$ vi DockerTags.txtubuntu:14.04
ubuntu:16.04
websphere-liberty:latest
ibmcom/transformation-advisor-db:1.4.0
ibmcom/transformation-advisor-server:1.4.0
ibmcom/transformation-advisor-ui:1.4.0
ibmcom/postgresql:9.6.6
ibmcom/icp-nodejs-sample:latest
...

Execute the getRepos.sh file against the updated DockerTags.txt

user@macbookpro:~/ibm-helm-offline$ sudo ./getRepos.sh DockerTags.txt

This script will use docker pull to pull the Docker images from Docker Hub, docker save, and then archive package them to a tar.gz file, ICP-EE-2.1.0.2.tar.gz. Depending on what you have included or excluded from the DockerTags.txt file, the size of the archive package might be slightly less than 9GB, hence the file creation will take time.

After completion, move the file ICP-EE-2.1.0.2.tar.gz and copy the contents of ~/ibm-helm-offline/ from your laptop to the bootnode host.

Log in to the IBM Cloud Private private registry

On your boot node do:

user@bootnode.icp:~$ sudo docker login mycluster.icp:8500
username: xxx
password: xxx

Load the archive package into the IBM Cloud Private private registry

user@bootnode.icp:~$ tar xf ICP-EE-2.1.0.2.tar.gz -O | docker load

Execute the tagAndUpload.sh file against the DockerTags.txt

user@bootnode.icp:~$ chmod +x tagAndUpload.sh
user@bootnode.icp:~$ sudo ./tagAndUpload.sh DockerTags.txt

Depending on the number of images and size, this might take some time to complete.

The images can now be deployed on IBM Cloud Private from the private registry, but the App Catalog is still empty. For that, you’ll need to perform 2 final tasks.

Modify Helm chart entries from public to private registries

The Helm charts need to be updated to point to the IBM Cloud Private private registry. For each Helm chart, you need to edit the registry entries in the values.yaml file, change the public registry entry to the private registry, create an archive of the Helm chart, and load the Helm chart to IBM Cloud Private.

For example, in ibm-datapower-dev/values.yaml, the line:

repository: ibmcom/datapower

should be changed to:

repository: mycluster.icp:8500/default/ibmcom/datapower

This has already been done for you in the in the ibm-helm-offline/stable folder on your laptop.

On your laptop, package the Helm charts needed. You need to execute helm package against that folder.

user@macbookpro:~/ibm-helm-offline$ cd stable
user@macbookpro:~/ibm-helm-offline/stable$ helm package *

This produces a .tgz file for each Helm chart. Copy them to the bootnode host.

Load Helm charts to the private Helm charts repository

You can now load the Helm charts that you packaged earlier into the IBM Cloud Private local Helm chart repository using this command from the boot node:

user@bootnode.icp:~$ bx pr load-helm-chart --archive <<helm chart.tgz>>

Alternatively, you could run the script loadHelmCharts.sh, provided in ibm-helm-offline:

user@bootnode.icp:~$ chmod +x loadHelmCharts.sh
user@bootnode.icp:~$ sudo ./loadHelmCharts.sh

The catalog should appear immediately in IBM Cloud Private App Catalog as soon as the command is run … and you’re done. Happy Kube’ing.

You can find more info on:

IBM Knowledge Center page for IBM Cloud Private.
On the Public IBM Cloud Private Slack channel.
On Stackoverflow.

IBM Cloud Private Offline Installation

Goals

Assumptions and guidance

Installation

Written by Haytham Elkhoja