PowerVC — Redhat and Centos 8 Image creation with pvsadm

Updated to Rhel 8.4

I have been thinking about how to improve the image creation for PowerVC for some while now, and I like to use code to create OS Images or packages Images, (for example Hasicorp Packer or just plain Kickstart)

So after I found the tool pvsadm on the IBM Github page. Tool to help with managing of resources in IBM Power Systems Virtual Server. In short, this tool is to create OS Image and upload image to IBM Power System Virtual Server that is cloud-based. (And I guess more is coming)

So I thought this tool should also be possible to use on On-prem PowerVC to create images and have most of the job in code.

So I have written a small guide on how to use pvsadm in combination with PowerVC to create and customise qcow2 images to OVA.
I'm also thinking of Creating an Ansible role that does this in one go, maybe in the future.

Note: There is a small Document that describes how to convert RHEL qcow2 to ova image format on the GitHub page. Guide

Updated to Rhel 8.4 (17.08.21)
New image template,cloudinit config and pvsadm tool.

Workflow:

The typical image workflow comprises of the following steps:

1. Download the qcow2 image. (Can be done by the tool)
2. Customize the image with pvsadm image qcow2ova --prep-template-
3. Convert the downloaded qcow2 image to ova using pvsadm image qcow2ova command.
4. Upload the ova image to PowerVC Server.
   Note: There is an option to upload image to IBM Cloud Object Store Bucket using pvsadm image upload command, but i have not tested this yet.
5. Import the ova image to IBM PowerVC powervc-image import

Prereqs: ✅

• PowerVC with FC connection to Storage, or you need to use another method for importing Images to PowerVC
• VM/Workstation to run pvsadm.
• > 200GB of memory where you run pvsadm and also on PowerVC if importing.

Note: this done with version pvsadm version 0.1.2, so there could be changes in the future.

Installation of pvsadm 🏗

1. Install the following packages:
   sudo dnf install -y qemu-img cloud-utils-growpart
2. Go to the releases page and download the latest or stable version of the pvsadm helper tool

sudo curl -sL https://github.com/ppc64le-cloud/pvsadm/releases/download/v0.1.2/pvsadm-linux-ppc64le -o /usr/local/bin/pvsadm
sudo chmod +x /usr/local/bin/pvsadm
pvsadm version

3. Run the pvsadm --helpcommand to check the available subcommands and the options.

Customize the image preparation script for RHEL/CentOS distro

pvsadm image qcow2ova --help

Customize the image preparation script for RHEL/CentOS distro, e.g: add additional yum repository or packages, change name servers etc.

Before creating we can check the default prep image template with the following command:

pvsadm image qcow2ova --prep-template-default

If you want to change this:

Step 1: Dump the default image preparation template

pvsadm image qcow2ova --prep-template-default > image-prep.template

Step 2: Make the necessary changes to the above-generated template file(bash shell script) image-prep.template

There is a lot to digest here, some of the commands to notice. (Se example below)

• Yum update OS.
• cloudinit will be configuration,
• firewall
• multipath setup
• IBM Power rpms
• reset_rmc seems to be added into cloud_final_modules automatically.

Then I have modified some of the values in the default template:

SSH
Normally SSH with root will be disabled. ssh_pwauth: 0
So if you want to have SSH access with a password:
We can add a sed line:

# Cloudinit customize — allow ssh password authentication
sed -i -e ‘s/ssh_pwauth: *0/ssh_pwauth: 1/g’ /etc/cloud/cloud.cfg

IPv6 seems to be disabled.

bootcmd:
- ‘echo “IPV6_AUTOCONF=no” >> /etc/sysconfig/network-scripts/ifcfg-$(ls /syys/class/net -1| grep env.|sort -n -r|head -1)

However, I can still see that IPv6 is listed.
This command will temporarily disable IPv6 address, have not digged into this future for now.

sysctl -w net.ipv6.conf.all.disable_ipv6=1

Redhat Subscription
There is also a line with Redhat Subscription activation.

• I don’t want to add my user and password, so i tried to change the activation method to org= and use the same variables.
• subscription-manager register -org=1269xxx activationkey=redhat_os

Org={{ .RHNUser }}
activationkey={{ .RHNPassword }}

subscription-manager register --org={{ .RHNUser }} --activationkey={{ .RHNPassword }}
#subscription-manager register --force --auto-attach --username={{ .RHNUser }} --password={{ .RHNPassword }}

Nameserver
Changed the nameserver to an internal one. will only be used for update the image.

echo “nameserver 9.9.9.9” | tee /etc/resolv.conf

firewalld service
I don’t like that they remove the firewalld service,
So use masked or disable instead. Note: If you mask you will have a problem if some of the installation is using Firewalld and is trying to start firewalld service.
FYI: Rhel 8.4 complaint about the systemctl disabled firewalld so remove this in the updated script.

#rm -rf /etc/systemd/system/multi-user.target.wants/firewalld.service
systemctl mask firewalld

TimeZone
The time and date is ETC something, so I tried to add this line.

# Set timezone
sudo timedatectl set-timezone UTC

If the goal is to create a generic image, then It’s important that we don’t add too much here, and then we could use other tools like packer or cloudinit to configure the system.

My image template example:

Default cloud.init

Step 3 — Download image

• Goto Red Hat Customer Portal
• Find the latest KVM Guest Image and copy the link from the Download Now link

Notes:

• The web page got a session timeout, make sure your session is active while copying the link
• The download link has a & char, make sure to add a proper escape character to download file properly

Step 4: Run the qcow2ova with the modified image preparation template

Note: Convert the rhel 8.4 Qcow2 image to ova format by installing all the prerequisites required for the image to work in the IBM Power Systems

We are going to use the command pvsadm with the subcommands image qcow2ova
Note: Version: v0.1.2, GoVersion: go1.16.5

pvsadm image qcow2ova [flags]

Flags:
 — image-name string Name of the resultant OVA image
 — image-url string URL or absolute local file path to the <QCOW2>.gz image
 — image-dist string Image Distribution(supported: rhel, centos, coreos)
 — image-size uint Size (in GB) of the resultant OVA image (default 120)
 — rhn-user string RedHat Subscription username. Required when Image distribution is rhel
 — rhn-password string RedHat Subscription password. Required when Image distribution is rhel
 — os-password string Root user password, will auto-generate the 12 bits password(applicable only for redhat and cento distro)
 -t, — temp-dir string Scratch space to use for OVA generation (default “/tmp”)
 — prep-template string Image preparation script template, use — prep-template-default to print the default template(supported distros: rhel and centos)
 — prep-template-default Prints the default image preparation script template, use — prep-template to set the custom template script(supported distros: rhel and centos)
 -h, — help

The options we are going to use is:

--image-name rhel-84–170821
--image-url ./rhel-8.4-ppc64le-kvm.qcow2
--temp-dir /data/tmp 
--os-password mypassword 
--image-dist rhel 
--rhn-user xxxx 
--rhn-password redhat_os
--prep-template image-prep.template

Take the command and change if your environment:

$ pvsadm image qcow2ova --image-name rhel-84–170821 --image-url ./rhel-8.4-ppc64le-kvm.qcow2 --temp-dir /data/tmp --os-password mypassword --image-dist rhel --rhn-user xxxx — rhn-password redhat_os --prep-template image-prep.template
Output:
I0817 12:36:28.187569  619324 qcow2ova.go:75] Overriding with the user defined image preparation template.
I0817 12:36:28.187684  619324 validate.go:40] Checking: platform
I0817 12:36:28.187695  619324 validate.go:40] Checking: user
I0817 12:36:28.187707  619324 validate.go:40] Checking: image-name
I0817 12:36:28.187743  619324 validate.go:40] Checking: tools
I0817 12:36:28.187785  619324 tools.go:43] qemu-img found at /usr/bin/qemu-img
I0817 12:36:28.187816  619324 tools.go:43] growpart found at /usr/bin/growpart
I0817 12:36:28.187826  619324 validate.go:40] Checking: diskspace
I0817 12:36:28.188017  619324 diskspace.go:49] free: 260G, need: 61G
I0817 12:36:28.188541  619324 get-image.go:43] Copying /data/rhel-8.4-ppc64le-kvm.qcow2 into /data/tmp/qcow2ova550813861/rhel-8.4-ppc64le-kvm.qcow2
I0817 12:36:29.324576  619324 get-image.go:47] Copy Completed!
I0817 12:36:29.324612  619324 qcow2ova.go:176] downloaded/copied the file at: /data/tmp/qcow2ova550813861/rhel-8.4-ppc64le-kvm.qcow2
I0817 12:36:29.324863  619324 qcow2ova.go:204] Converting Qcow2(/data/tmp/qcow2ova550813861/rhel-8.4-ppc64le-kvm.qcow2) image to raw(/data/tmp/qcow2ova550813861/ova-img-dir/disk.raw) format
I0817 12:36:36.390430  619324 qcow2ova.go:209] Conversion completed
I0817 12:36:36.391073  619324 qcow2ova.go:211] Resizing the image /data/tmp/qcow2ova550813861/ova-img-dir/disk.raw to 11G
WARNING: Image format was not specified for '/data/tmp/qcow2ova550813861/ova-img-dir/disk.raw' and probing guessed raw.
         Automatically detecting the format is dangerous for raw images, write operations on block 0 will be restricted.
         Specify the 'raw' format explicitly to remove the restrictions.
Image resized.
I0817 12:36:36.542536  619324 qcow2ova.go:216] Resize completed
I0817 12:36:36.542567  619324 qcow2ova.go:218] Preparing the image
/dev/loop10
CHANGED: partition=2 start=10240 old: size=20961280 end=20971520 new: size=23058399 end=23068639
xfs
meta-data=/dev/loop10p2          isize=512    agcount=4, agsize=655040 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=1, rmapbt=0
         =                       reflink=1
data     =                       bsize=4096   blocks=2620160, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
log      =internal log           bsize=4096   blocks=2560, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
data blocks changed from 2620160 to 2882299
nameserver 9.9.9.9
The system has been registered with ID: 927d70a5-cc28-4830-8de
The registered system name is: rhel84-test.oslo.forum.com
Installed Product Current Status:
Product Name: Red Hat Enterprise Linux for Power, little endian
Status:       Subscribed
And it will contiune on the prep script.
Changing password for user root.
passwd: all authentication tokens updated successfully.
Unregistering from: subscription.rhsm.redhat.com:443/subscription
System has been unregistered.
All local data removed
I0817 12:45:13.704677  619324 qcow2ova.go:223] Preparation completed
I0817 12:45:13.704715  619324 qcow2ova.go:225] Creating an OVA bundle
I0817 12:45:22.543614  619324 qcow2ova.go:230] OVA bundle creation completed: /data/tmp/qcow2ova550813861/rhel84-170821.ova
I0817 12:45:22.543674  619324 qcow2ova.go:232] Compressing an OVA file
I0817 12:45:48.033033  619324 qcow2ova.go:238] OVA file Compression completed
Successfully converted Qcow2 image to OVA format, find at /root/rhel84-170821-v2.ova.gz
OS root password: mypassword

If we check the folder we will now have ova.gz file.

[root@rhel84-test data]# ls -lh | grep rhel
-rw-r — r — . 1 root root 2.5G Feb 5 14:34 rhel84-170821.ova
-rw-r — r — . 1 root root 1.2G Feb 5 14:19 rhel-8.4-ppc64le-kvm.qcow2

Centos:
Note: remember you can also use URL.

pvsadm image qcow2ova --image-name centos-82 --image-dist centos --image-url /root/CentOS-8-GenericCloud-8.2.2004–20200611.2.ppc64le.qcow2 --prep-template image-prep.template

Step 5: Import it to PowerVC (Need FC connection)

When we converted the Image, the default image size I 120GB (Resizing the image disk.raw to 120G)

This can be changed if you specify: image-size 50

We need minimum 120GB available in /var/opt/ibm/powervc/

The default temp_staging_parent_dir is /var/opt/ibm/powervc/imgstaging, (The folder imgstaging/ will be created.)

powervc-image config
List powervc-image configuration properties:
+ — — — — — — — — — — — — — — -+ — — — — — — — — — — — — — — — +
| Configuration Property | Value |
+ — — — — — — — — — — — — — — -+ — — — — — — — — — —— — — — — — — +
| default_ova_export_dir | /var/opt/ibm/powervc/ova |
| enforce_unique_image_names | True |
| enforce_unique_volume_names | False |
| polling_timeout | 86400 |
| powervc_image_log | /opt/ibm/powervc/log/powervc-image.log |
| temp_staging_parent_dir | /var/opt/ibm/powervc/imgstaging |
+ — — — — — — — — — — — — — — -+ — — — — — — — — — — —— — — — — +

If you want to change Imagestaging:

powervc-image config --set temp_staging_parent_dir -value /var/opt/ibm/powervc/imgstaging

If you don’t use PowerVC VM to create the image: SCP the image over to PowerVC

scp rhel-84-**.ova.gz root@powervc01:/root/powervc

Check that you have the available space.

[root@powervc01 ~]# df -h /var/opt/ibm/powervc/
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/rhel-root 181G 23G 158G 13% /

NB: Remember this will only work if the PowerVC has an FC connection to storage!

Test this with powervc-image test-conn

powervc-image test-conn <target_storage_template_id>
powervc-image test-conn 915e1785–3e4d-4a8d-8dcd-2275d3aeece1
[root@powervc01 ~]# powervc-image test-conn 915e1785–3e4d-4a8d-8dcd-2275d3aeece1
Output: 
Running ‘cleanup’ flow for any stale SCSI devices before connection activity…
+ — — — -+ — — — — — — — — — + — — — — — — — — — + — — — — + — — — — | host | wwpn | wwnn | speed | port_state | time_since_last_reset | targets |+ — — — -+ — — — — — — — — — + — — — — — — — — — + — — — — 
host1 | c050760839170054 | c050760839170054 | 8 Gbit | Online | [unknown] | 2 (*) |+ — — — -+ — — — — — — — — — + — — — — — — — — — 
+ — — — — -+ — — — — — — — — — + — — — — — -+ — — — — — +
| row_num | target_wwpn | target_id | num_luns |
+ — — — — -+ — — — — — — — — — + — — — — — -+ — — — — — +
| 1 | 500507680c1511c8 | 0 | 1 |
| 2 | 500507680c1511c6 | 1 | 1 |

+ — — — -+ — — — — — — — — — + — — — — — — — — — + — — — — + — — — — 
| host | wwpn | wwnn | speed | port_state | time_since_last_reset | targets |
+ — — — -+ — — — — — — — — — + — — — — — — — — — + — — — — + — — — — 
| host2 | c050760839170056 | c050760839170056 | 8 Gbit | Online | [unknown] | 2 (*) |
+ — — — -+ — — — — — — — — — + — — — — — — — — — + — — — — + — — — — 
+ — — — — -+ — — — — — — — — — + — — — — — -+ — — — — — +
| row_num | target_wwpn | target_id | num_luns |
+ — — — — -+ — — — — — — — — — + — — — — — -+ — — — — — +
| 1 | 500507680c1511c6 | 1 | 1 |
| 2 | 500507680c1511c8 | 0 | 1 |
+ — — — — -+ — — — — — — — — — + — — — — — -+ — — — — — +
The storage template with ID 915e1785–3e4d-4a8d-8dcd-2275d3aeece1 has the supported Fibre Channel provider ‘svc02’.
Creating a test volume ‘image-agility-priming-vol-915e17’ on storage provider ‘svc02’…
.Done creating.
Mapping test volume on the storage provider to the local server’s ports…
There are already attached devices from one or more of the mapped target WWPNs: {‘500507680c1511c8’, ‘500507680c1511c6’}
Device attachment information returned: {‘type’: ‘block’, ‘scsi_wwn’: ‘3600507680c81008e400000000000010a’, ‘path’: ‘/dev/disk/by-path/fc-0x500507680c1511c8-lun-1’}
Connection of test volume worked on initial try.
Disconnecting test volume ‘image-agility-priming-vol-915e17’ from local server since it was created during this run and there was no connection failure…
Unmapping the test volume from the storage provider…
Deleting test volume…
Test Fibre Channel connectivity complete.

TIP: How can you find the storage template IDs from the command line?

export OS_USERNAME=root
export OS_PASSWORD=password
source /opt/ibm/powervc/powervcrc
#Run the [cinder type-list] OpenStack command and get the ID:
cinder type-list
+ — — — — — — — — — — — — — — — — — — — + — — — — — — — — — — — — — 
| ID | Name | Description | Is_Public |
+ — — — — — — — — — — — — — — — — — — — + — — — — — — — — — — — — — 
| 915e1785–3e4d-4a8d-8dcd-2275d3aeece1 | Storage_template_svc02_FS840–1 | — | True |
| e76737d3–6c57–47bc-8b0c-4220fa2690cf | svc02 base template | — | True |
+ — — — — — — — — — — — — — — — — — — — + — — — — — — — — — — — — — 

Import image:

Rember to check what project/tentant you importing the image to, default i ibm-default. (check /opt/ibm/powervc/powervcrc)

powervc-image import
-p <path_to_ova>
-t <storage_template_id>
-d <desciption>

powervc-image import -p <path_to_ova> -t <storage_template_id> -d <desciption>
powervc-image import -p /root/powervc/rhel-84–170821.ova.gz -t 915e1785–3e4d-4a8d-8dcd-2275d3aeece1 -d “rhel84 created with pvsadm”

This will take some time, (26min for me) so you can start a new session and follow that the stage volume will fill up or do something else.

# df -h /var/opt/ibm/powervc/
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/rhel-root 181G 99G 82G 55% /

List images: powervc-image list

[root@powervc01 powervc]# powervc-image list
[Set the OS_PASSWORD environment variable to avoid this prompt] Enter password for ‘root’:
+ — — — — — — — — -+ — — — — — — — — — — — — — — — — — — — + — — — — 
| Name | ID | Status | Volumes | Size | Description | Architecture | OS Distro | Exportable |
+ — — — — — — — — -+ — — — — — — — — — — — — — — — — — — — + — — — — 
| rhel-84–170821 | 317b27bd-d17d-4199–8217–047a4e503165 | active | 1 | 120 | rhel84 created with pvsadm | ppc64le | rhel | True |
+ — — — — — — — — -+ — — — — — — — — — — — — — — — — — — — + — — — — 
1 managed images returned.

Ta-da 🏁 👏

You should now be able to start using your image