This is pretty straight forward. Be able to execute commands effectively. Tools that can help with this are man pages and bash completion. You can type a command such as systemctl then hit tab twice to see which options are available. Type man systemctl to get more detailed information, remember to look for examples in man pages.

Use input-output redirection (>, >>, |, 2>, etc.)

There are three I/O redirection terms to know

STDIN — Basically what comes from your keyboard. It’s file descriptor number is 0. It can be used in redirection with ‘<’ which is the same as ‘0<’. For example lets say we have a file called test.txt with the following content: echo “lol” we can redirect the contents of this file to bash to execute the command using the following:

bash < test.txt
#notice that the command in test.txt is executed

STDOUT — Basically what you see on your computer monitor after running a command. Its file descriptor number is 1. It can be used in redirection with ‘>’ which is the same as ‘1>’. For example we can redirect output that would typically be sent to STDOUT to a file with:

cat /etc/passwd > copy-of-passwd
#notice that nothing is output to STDOUT

STDERR — This outputs errors that occurs in commands to your computer monitor. Its file descriptor number is 2. It can be used in redirection with ‘2>’. For example lets create an error by trying to read from a file that does not exist and then redirecting the error output to /dev/null.

cat bogusfile.txt
#this generates an error that is printed to your screen
cat bogusfile.txt 2>/dev/null
#the error message is no longer displayed

#output both STDOUT and STDERR to /dev/null
find -type f -perm /4000 &>/dev/null

Use grep and regular expressions to analyze text

#Simple grepping. searches for the string 'lol' in the file test.txt
grep 'lol' test.txt
#Search for strings that start with asdf
grep -E '^asdf' test.txt
#Search for strings that end with lol"
grep -E 'lol$' test.txt
#Search for string using wildcard '.' to match any single character
grep -E 'lo.' test.txt
#Search with matching will match any of the following lol/eol/col
grep -E '[lec]ol'
#match zero or one of the preceding character with '?': will match lol/ol
grep -E 'l?ol'
# match one or more of preceding character with '+' will match llol/lllol etc
grep -E 'l+ol'
#Matches exactly two of previous character with '{2}' will match llol
grep -E 'l{2}ol'
#Matches minimum of 1 and max of 3 of previous char with '{1,3}' will match lol/llol/lllol
grep -E 'l{1,3}ol'
#example regular expression to match IP address '-o' only outputs matches. note that we need to escape '.' so it is not interpreted
ip a | grep lo | grep -E -o '[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}'
#other useful text tools
#using sed to substitute lol with lmao in a file
sed -i 's/lol/lmao/g' test.txt
#sed to print 2nd line of file
sed -n 2p test.txt
#awk to print home directory of current user. grep is used with cmd substitution to get user name. then awk to print relevant line
cat /etc/passwd | grep $(echo $USER) | awk -F : '{ print $6 }'

Access remote systems using SSH

SSH is a utility for accessing remote devices via a terminal it offers a ton of functionality such as: remote access, file transfers, command execution, remote/dynamic/local port forwarding and more. I have used many of these features in my learning-cybersecurity publication if you are interested.

#SSH-server config file where you can change options such as listening port, password auth and more.
cat /etc/ssh/sshd_config
#SSH basic connection syntax
ssh username@8.8.8.8.8
#SSH with non-standard port
ssh -p 777 username@8.8.8.8.8
#copy file to remote device using SCP (Secure Copy Protocol)
scp -P 777 /etc/passwd username@8.8.8.8:/remote/directory/forfile
#copy file from remote device to local machine with SCP (notice -P is capital in SCP for whatever stupid reason)
scp -P 777 username@8.8.8.8.8:/etc/passwd /home/user/Documents
#Transfer files with SFTP. This provides a more interactive experience
sftp username@8.8.8.8.8
#available SFTP commands: (list with ?)
ls #list remote files
pwd #remote current directory
lpwd #local current directory
lcd /home/user/.config #changes local directory to /home/user/.config/
put /etc/hosts #will upload /etc/hosts file to remote server
get /etc/hosts #will download the /etc/hosts file from remote server
exit #obvious

#configuring key-based authentication
ssh-keygen #generates public/private key pair (follow prompts)
ssh-copy-id username@8.8.8.8.8 #adds public key to remote server
#edit the /etc/ssh/sshd_config and uncomment/change following line
PasswordAuthentication no
#login with key
ssh -p 777 username@8.8.8.8 -i .ssh/id_rsa

# Working with multiple terminals in a non-graphical environment
Alt+F1 #Gnome display manager graphical login
Alt+F2 #Access to current graphical console
Alt+F3 #Access to current graphical session
Alt+F4-F6 #Access to nongraphical consoles
#you can also use chvt command instead
chvt 1

Log in and switch users in multiuser targets

emergency.target: Minimal number of units started. Only used to fix a system if something has gone terribly wrong

rescue.target: Starts units for a fully functional Linux system. Is minimal, does not start bloat

multi-user.target: Starts all units for full functionality, usually used on servers

graphical.target: Similar to multi-user.target except it also starts a graphical interface

#More information on targets
systemctl cat multi-user.target
#using isolate to switch current state of computer to target 
#note this needs to have the AllowIsolate=yes line
#check with: 
find /usr/lib/systemd/system -name *.target -exec grep -H Isolate {} \;
systemctl isolate rescue.target

Archive, compress, unpack, and uncompress files using tar, gzip, and bzip2

Used to create backups using tapes. It creates archives and has options to compress, list and extract files using various compression algorithms

#create archive
tar -cvf /tmp/home.tar /home/user
#list contents of archive
tar -tvf /tmp/home.tar
#add to existing archive
tar -rvf /tmp/home.tar /etc/passwd
#update archive
tar -uvf /tmp/home.tar /home
#extract archive
tar -xvf /tmp/home.tar -C /tmp/
#compress archive with gzip (-J for xz, -j for bzip2)
tar -cvfz /tmp/home.tar.gz /home/user
#uncompress archive
tar -xvfz /tmp/home.tar.gz /tmp/

Create and edit text files

#create file
touch test.txt
#edit
vi test.txt 
echo "something" >> test.txt
#search and replace
sed -i test.txt 's/something/nothing/g'

Create, delete, copy, and move files and directories

# creating directory
mkdir /home/user/test
#creating directory path with directories that don't exist
mkdir -p /home/user/test/test2/test3
#remove empty directory
rmdir /home/user/test/test2/test3
#remove non empty directories
rm -rf /home/user/test
#copy
cp /etc/passwd /home
#copy recursive
cp -R /home/user /tmp
#move
mv /tmp/test /home

Create hard and soft links

Files in general are simply links to an inode. Inode is a data structure that keeps track of all files/directories in a filesystem. This links each file/directory to a unique identifier so it can be found in on disk.

Hard links- A hard link creates another file with a link to the same inode, so if you delete the original file, the hard link file still points to the inode and the original content is still available. You can test this be creating a hard link of a file and then deleting it, then checking to see if the file content is still available. Note that hard links must reside within the same filesystem.

#creates a hard link of the test.txt to the file /tmp/test.txt
ln test.txt /tmp/test.txt

Soft links- A soft link is different from a hard link in that it simply points to another file name. This means if you were to delete the original file, the soft link would no longer have the file content available to it. One of the benefits of soft links is that it can span different file systems and partitions.

#creates a soft link of test.txt to the file /tmp/test.txt
ln -s test.txt /tmp/test.txt

List, set, and change standard ugo/rwx permissions

Source: https://pressidium.com/blog/deciphering-linux-file-system-permissions/

There are three basic permissions that can be applied to files/directories. They are: read, write, execute. An important thing to remember when using numeric representations of permissions is that: r=4, w=2, x=1. Therefore if a file permission has been set using chmod 777 then the user, group and others all have read write execute permissions.

There are also special permissions: SUID = 4000, SGID = 2000, sticky = 1000. SUID allows a binary to be executed with the permissions of the file owner. The most common example of this is the passwd binary which allows users to change their own passwords. Without SUID set on passwd, users would be unable to do so. SGID is similar only that it executes the binary with the permissions of the group owner. SGID is useful in shared group environments where users requiring similar access to a directory need to write files. The sticky bit is used for preventing file deletion. If the sticky bit is set, the only user who can delete a file is the owner of said file or directory that contains the file.

#setting permssions using numeric representation, User=rw, group=r, others=0
chmod 640
#setting using symbolic
chmod u+rw
chmod g-w
#setting SUID on file numerically and u=rwx, g=rw, o=x
chmod 4761
#setting sticky bit in relative mode, suid, then sgid
chmod +t /directory
chmod u+s /executable
chmod g+s /directory # or executable

Locate, read, and use system documentation including man, info, and files in /usr/share/doc

Man pages are our best friends when it comes to the RHCSA exam, if you forget how to use a command such as grep, then use man grep to get detailed explanations and examples. Additionally we can use apropos or man -k to search for key-words in man pages.

#man page for grep
man grep
#searching man pages for volume group
man -k "volume group"
apropos "volume group"
#update man pages (must be root)
sudo mandb
#additional information about commands can be found in /usr/share/doc
ls /usr/share/doc

Create simple shell scripts

Shell scripts should always start with a line defining which shell will execute the script. For operating systems with bash that line will be #!/bin/bash. In minimal environments you may need to use #!/bin/sh. Script files must have the execute permission to be executed. This can easily be done using chmod u+x myscript.sh.

Conditionally execute code (use of: if, test, [], etc.)

#spaces are important in an if statement
#checks to see if the number of arguments is greater than 0, if so echo arg num
# if no arguments are specified it tells you so
if [[ $# > 0 ]];
then
  echo "you entered $# arguments"
else
  echo "you entered no arguments"
fi

# '$#' = number of arguments
# '$@' = refers to each argument, can get arg values unlike $#
# '#?' = exit status of last command executed, if no err #? =0
# '$0' = name of script

#using elif in an if statement
#Print all of the arguments if two are provided, if there are more than two arguments then say so, else no arguments
if [[ $# = 2 ]];
then
  echo "the arguments you provided are: $@"
elif [[ $# > 2 ]];
then
  echo "too many arguments"
else
  echo "not enough arguments provided"
fi

# working with errors
# asks user to enter file and checks to see if it exists based on exit status
echo "enter a filename:"
read filename
ls $filename &>/dev/null
if [[ $? = 0 ]];
then
  echo "$filename exists"
else
  echo "$filename does not exist"
fi

Use Looping constructs (for, etc.) to process file, command line input

# for loop to print all numbers from 1-100 in a one-liner
for i in {1..100}; do echo $i; done

# accomplishes the same thing differently
for (( i=1; i<100; i++ )); do
  echo $i
done

# while loop that stops when another user logs in
# type sudo su in another terminal 
while [[ $(who | wc -l) = 1 ]]
do
  echo "only one user is logged in"
  sleep 5
done
echo "another user has logged in"

Process script inputs ($1, $2, etc.)

# case statement that checks to see if first arg matches, then executes cmd
case "$1" in
  proc)
    ps aux
    ;;
  list)
    ls -l ~/
    ;;
  listall)
    ls -al ~/
    ;;
  *) #catch all
    echo "usage: enter proc/list/listall"
    ;;
esac

Processing output of shell commands within a script

#!/bin/bash
#search path for binary using cmd substitution
#use script with ssh as argument to see all binaries in path containing 'ssh'
ls $(echo $PATH | sed s/:/\\n/g) | grep $1)

Operate running systems

Boot, reboot, and shut down a system normally

#various ways to manipulate system
shutdown now
reboot
systemctl suspend
systemctl hibernate
systemctl reboot

Boot systems into different targets manually

https://linuxconfig.org/boot-systems-into-different-targets-manually-rhcsa-objective-preparation

# systemctl set-default multi-user.target
#More information on targets
systemctl cat multi-user.target
#using isolate to switch current state of computer to target 
#note this needs to have the AllowIsolate=yes line
#check with: 
find /usr/lib/systemd/system -name *.target -exec grep -H Isolate {} \;
systemctl isolate rescue.target

Interrupt the boot process in order to gain access to a system

To interrupt the boot process in RHEL simply press e when the GRUB boot menu is displayed. For troubleshooting purposes it is recommended to remove the ‘rhgb’ and ‘quiet’ options. This makes the boot process more verbose and you will be able to see any boot-time errors

Identify CPU/memory intensive processes and kill processes

#view processes in real time/ interactive
top
#print all processes
ps aux
#check exact command used to start process
ps -ef
#get highest memory usage processes %cpu can also be used
ps aux -sort -%mem | head -n 10
#get pid of highest usage memory process
ps aux -sort -%mem | head -n 2 | tr -s ' ' | cut -d ' ' -f 2 | grep -E -o '[[:digit:]]*'
#get pid
pidof firefox
#kill firefox
kill $(pidof firefox)
#make sure its dead
kill -9 $(pidof firefox)

Adjust process scheduling

There are two tools for managing process priorities; nice and renice. If you want to start a process with an adjusted priority use nice. If you want to change the priority of a running process then use renice. Values of niceness range from -20 to 19 (default set to 0) negative numbers increase the priority, higher numbers decrease the priority.

#change niceness of firefox to 10
renice -n 10  -p $(pidof firefox)
#start process with niceness of 5
nice -n 5 firefox

Manage tuning profiles

the tuned daemon in RHEL is used to automatically tune a system for the better performance.

#install
dnf install tuned -y
#enable and start
systemctl enable --now tuned
#list current profile
tuned-adm active
#get recommended profile
tuned-adm recommend
#set profile
tuned-adm profile virtual-guest

Locate and interpret system log files and journals

On RHEL logs are stored in the /var/log directory. Some important logs are

/var/log/messages: most messages are written here
/var/log/dmesg: kernel related events
/var/log/secure: authentication events
/var/log/boot.log: boot events
/var/log/audit/audit.log: SELinux events
/var/log/httpd: Apache web server log

The systemd-journald service is responsible for storing log messages in the journal. The journal file is temporarily stored in /run/log/journal, it can be easily accessed using the journalctl command.

#creating an authentication log
logger -p auth.err "ooops"
#display live and most recent entries, ooops should appear here
journalctl -f
#view all logs with pager
journalctl
#no pager
journalctl --no-pager
#filter by uid
journalctl _UID=0
#get last 10 lines
journalctl -n 10
#get errors only
journalctl -p err
#events since yesterday
journalctl --since yesterday
#get specific service information
journalctl -u sshd
journalctl _SYSTEMD_UNIT=sshd.service -o verbose #extra verbose

Preserve system journals

The journald configuration file is located at /etc/systemd/journald.conf

#preserve system journals by changing storage=
vim /etc/systemd/journald.conf
#modes
#storage=auto: journal saved to disk if /var/log/journal exists
#storage=volatile journal stored in mem at /run/log/journal dir
#storage=persistent dir /var/log/journal is automatically created
storage=persistent

Start, stop, and check the status of network services

#start a service (not persistent)
systemctl start sshd
#stops servie (not persistent)
systemctl stop sshd
#get status
systemctl status sshd
#checking listening ports
ss -lt
nmap -p- localhost
#list UDP and TCP ports
ss -tul

Securely transfer files between systems

#copy file to remote device using SCP (Secure Copy Protocol)
scp -P 777 /etc/passwd username@8.8.8.8:/remote/directory/forfile
#copy file from remote device to local machine with SCP (notice -P is capital in SCP for whatever stupid reason)
scp -P 777 username@8.8.8.8.8:/etc/passwd /home/user/Documents
#Transfer files with SFTP. This provides a more interactive experience
sftp username@8.8.8.8.8
#available SFTP commands: (list with ?)
ls #list remote files
pwd #remote current directory
lpwd #local current directory
lcd /home/user/.config #changes local directory to /home/user/.config/
put /etc/hosts #will upload /etc/hosts file to remote server
get /etc/hosts #will download the /etc/hosts file from remote server
exit #obvious

Configure local storage

Overall steps for creating LVM

Partition disks using fdisk/gdisk (fdisk for MBR, gdisk for GPT)
Create physical volumes using pvcreate
Create/add drives to volume group with vgcreate/vgextend
Create logical volume with lvcreate/lvextend

List, create, delete partitions on MBR and GPT disks

fdisk/gdisk can be used to partition drives. Note that while partitioning the drive you need to ensure that it has the correct hex code. gdisk code = 8e00 fdisk code = 8e (you do not need to remember the code as you can look it up while in the fdisk/gdisk interactive terminal). Note you need to press ‘t’ in fdisk to change the label type (gdisk forces you to label it when creating partition).

Create and remove physical volumes

Once you have labeled a partition as an LVM partition type, you can then mark it as a physical volumes using pvcreate.

#Create pv on sdb1
pvcreate /dev/sdb1
#list pv
pvs

Assign physical volumes to volume groups

After physical volumes are created they need to be added to a volume group before they can be used. If the volume group is already created you will need to use vgextend to add the physical volume to the volume group. The syntax:

#Create vg and add drive
vgcreate myvolumegroup /dev/sdb1
#Specify physical extent size
vgcreate -s $size myvolumegroup /dev/sdb1
#Add sdb2 pv to existing vg
vgextend myvolumegroup /dev/sdb2
#Display vg
vgs && vgdisplay

About extents:

Within a volume group, the disk space available for allocation is divided into units of a fixed-size called extents. An extent is the smallest unit of space that can be allocated. Within a physical volume, extents are referred to as physical extents.

Create and delete logical volumes

Once volume groups are created you can create one or more logical volumes. A name and size are required to create logical volumes.

#Creating lv named mylv with half the size available by myvolumegroup
lvcreate -n mylv -l 50%FREE myvolumegroup
#Similar but specifying the extents
lvcreate -n mylv -L 10G myvolumegroup
#note you need to add a filesystem
mkfs.xfs /dev/myvolumegroup/mylv
#resizing lv (need to extend volume group first)
#use -r to automatically resize the filesystem
#the use of '+' is important, if not used the space is not added
lvextend -r -l +100%FREE /dev/myvolumegroup/mylv
#using logical extents
lvextend -r -L +10G /dev/myvolumegroup/mylv

Note that the resize options will only work if supported by the filesystem. So for example some file systems such as xfs cannot decrease its size but can extend it’s size. So if you attempt to reduce the size of a logical volume with an xfs filesystem it will fail. If you do need a filesystem that supports shrinkage you need to use ext4.

Configure systems to mount file systems at boot by universally unique ID (UUID) or label

There are two ways to mount filesystems. Because the RHCSA exam requires configurations to be persistent you need to edit the /etc/fstab file. The format of the fstab file is as follows (by column)

Device: which can be a label (if configured) or more commonly the device UUID which can be retrieved using blkid (no output will appear without sudo).
Mount point: which is the directory on the local system where you can access the filesystem. ex /mnt/extraspace
File system type: simply the filesystem type such as xfs/ext4/swap
Options: comma separated list of values such as exec/noexec
Backup operation: Used for backup operations, just leave set to 0
Filesystem check order: Uses fcsk to check the filesystem, just leave set to 0

Example file:

# LVM @ /dev/sdb1
UUID=1f8e2964-7181-56a4-a931-9c04c7bd5f2a  /mnt/extraspace  xfs  relatime,noexec  0  0

Note that after adding you can either restart the computer to have the drive mounted or you can type ‘sudo mount -a’ to mount all devices specified in /etc/fstab. Although a restart is recommended for exam as it ensures your configuration is working correctly upon reboot.

Add new partitions and logical volumes, and swap to a system non-destructively

To add more swap space to a Linux system you first need to create a swap partition. This can be done using fdisk/gdisk, you will need to set the swap label to the partition for fdisk the label hexcode is 82, for gdisk it’s 8200. After partitioning you need to to format it as swap space using ‘mkswap’. The next step is to use the swapon/swapoff to activate/deactivate the swap space.

#partition swap space
fdisk /dev/sdb
#check swap space
free -m
#format partition as swap space
mkswap /dev/sdb1
# enable
swapon /dev/sdb1
#verify
free -m

Note that with this method the added swap space is not persistent and therefore not available upon restart. To make the swap space persistent we once again need to edit the /etc/fstab file.

#example fstab entry
#note the 'none' as swap does not require a mount point
/dev/sdb1 none swap defaults 0 0

Create and configure file systems

Create, mount, unmount, and use vfat, ext4, and xfs file systems

#creating vfat, ext4, xfs filesystems
mkfs.vfat /dev/sdb1
mkfs.ext4 /dev/sdb1
mkfs.xfs /dev/sdb1
#mount not persistent
mount /dev/sdb1 /mnt/newspace
#mount persisitent with /etc/fstab
/dev/sdb1 /mnt/newspace xfs defaults 0 0
#unmount
umount /mnt/newspace
#use partprobe if having issues listing partitions/drives
partprobe

Mount and unmount network file systems using NFS

#install nfs utilities
dnf install -y nfs-utils
#discovering remote NFS server mounts
showmount -e 10.1.1.125
#mount shares - not persistent
mount 10.1.1.125:/remote/dir /mnt/nfs_share
#mount with /etc/fstab - persistent
#the sync option ensures that changes are immediatley available to prevent data loss
10.1.1.125:/remote/dir /mnt/nfs_share nfs sync 0 0

Configure autofs

Autofs is an alternative to using the /etc/fstab file to persistently mount NFS shares. This is slightly different from the fstab file in that it mounts on demand. This means that the NFS share is only mounted when an access attempt is made. This ensures the system is not dedicating resources to a mounted NFS share that is not being used (as is the problem with automounting using fstab). It also allows for mounting without root level user permissions. Mounting with autofs takes two steps:

Edit the master config file /etc/auto.master. This file contains the local mount point then on the same line the child config file is identified. example file: (I could not get this to work in the way it is specified in the cert guide, but below worked for me)

# /- tells autofs that mount points are defined in /etc/auto.nfsdata
/- /etc/auto.nfsdata

2. In the child config file you put the subdirectory in the root directory. Example file: (/nfsmnt will be created when autofs is restarted/started)

#you do not need to create dir, they will be autocreated
/nfsmnt -rw 10.1.1.15:/remote/dir
#using wildcards
#is useful if you for example store user home dir in NFS
# *=local mount point which matches everything
# & represents matching item on remote server
* -rw 10.1.1.15:/remote/users/&

#note restart autofs after making changes
systemctl restart autofs

Note if autofs is not installed you can install with: dnf install -y autofs. ensure you also enable the service: systemctl enable — now autofs. Note that when you list files in the NFS root directory you may not see the nfsmnt directory, don’t panic simply cd into /nfsmnt and the share will become available to you.

Extend existing logical volumes

#resizing lv (need to extend volume group first)
#use -r to automatically resize the filesystem
#the use of '+' is important, if not used the space is not added
lvextend -r -l +100%FREE /dev/myvolumegroup/mylv
#using logical extents
lvextend -r -L +10G /dev/myvolumegroup/mylv

Create and configure set-GID directories for collaboration

#change group ownership of groupdir
chown :sysadmin /groupdir
chgrp sysadmin /groupdir
#add set-GID permission
chmod g+s /groupdir

#example scenario where sysadmin john wants other sysadmins to be able to -
#create files in /sysadmin while not allowing other sysadmins to delete -
#each others files in the /sysadmin dir

# add john as owner of /sysadmin and sysadmins as group owner
chown john:sysadmins /sysadmin
# add sticky bit to prevent file deletion unless you are owner
chmod 1770 /sysadmin

#only allows owner of file (and john -because he owns dir) to delete files

Diagnose and correct file permission problems

#change owner and group of file
chown john:sysadmin /groupdir
#check permissions of directory
ls -ld /groupdir
#check permissions of file
ls -l /groupdir/file.txt
#check permissions and SELinux context
ls -lZ /groupdir/file.txt
#change permissions remove others from being able to read file.txt
chmod o-r /groupdir/file.txt
#do the opposite
chmod o+r /groupdir/file.txt
#change all permissions at once for user, group and others
chmod a+rx /groupdir/file.txt
#change group permissions to read write execute
chmod g=rwx /groudir/file.txt

Deploy, configure, and maintain systems

Schedule tasks using at and cron

If you struggle with remembering the format you can use the following command to get the example format: ‘cat /etc/crontab’

Cron format:

|------------------------------- Minute (0-59)
|     |------------------------- Hour (0-23)
|     |     |------------------- Day of the month (1-31)
|     |     |     |------------- Month (1-12; or JAN to DEC)
|     |     |     |     |------- Day of the week (0-6; or SUN to SAT; or 7 for Sunday)
|     |     |     |     |
|     |     |     |     |
*     *     *     *     *

Options can also use ‘@reboot’ to run upon a restart. Cronjobs can be added using the crontab -e command

Example cronjobs:

#Runs backup.sh at 11pm monday through saturday
0 23 * * 1-6 /root/backup.sh
#It can also be written like this
0 23 * * MON-SAT /root/backup.sh
#execute backup script everyday at midnight
0 0 * * * /root/backup.sh

Start and stop services and configure services to start automatically at boot

Systemd is the default system and service manager for RHEL versions its syntax is pretty easy to understand.

#start a service (not persistent)
systemctl start sshd
#start a service automatically (persistent)
systemctl enable sshd
#starts and enables service in one command (persistent)
systemctl enable --now sshd
#stops persistent starting of service
systemctl disable sshd
#stops servie (not persistent)
systemctl stop sshd
#get status
systemctl status sshd
#get detailed info about service
systemctl status -l sshd
#list all services
systemctl -t service
#list failed services
systemctl --failed -t service

#note that you do not need to type out the full service name, systemctl-
#appends .service to the command. You can also use autocompletion TAB,TAB

Configure systems to boot into a specific target automatically

To configure a system to boot into a specific target automatically you need to set the default target.

#list all targets
systemctl --type=target --all
#list active targets
systemctl --type=target
#find targets with isolate 
find /usr/lib/systemd/system -name *.target -exec grep -H Isolate {} \;
#listing default target
systemctl get-default
#setting default target
systemctl set-default graphical.target

Configure time service clients

The main command used for displaying and making changes to time is timedatectl.

#Show current time configuration
timedatectl status
#Change current time
timedatectl set-time n
#List timezones
timedatectl list-timezone
#set timezone
timedatectl set-timezone EST
#configuring NTP
timedatectl set-ntp 1 #0 for off

When timedatectl uses NTP it works with the chronyd process, so if you see an error message when trying to configure NTP then it is likely that chronyd is not installed or not running.

#Trouble shooting NTP failure
#Ensure chronyd is installed
dnf install -y chrony
#verify config file
cat /etc/chrony.conf
#check status of chronyd service
systemctl status chronyd

Install and update software packages from Red Hat Network, a remote repository, or from the local file system

The default application used to install software packages on RHEL is dnf. These software packages are in the Red Hat Package Manager (RPM) format. Note that you may need to use the subscription-manager package to authenticate to certain repositories before being allowed access to download. Repositories are specified in the /etc/yum.repos.d. If you are like me and struggle to remember the format in which repositories are configured, simply read a file from this directory to get the correct structure.

[label] #used as identifier in repo file
name= #mandatory option specifying name of repo
mirrorlist= #optional refers to URL about mirror servers
baseurl= #mandatory option with URL pointing to packages
gpgcheck= #specifies if integrity checking is performed on packages 1=on 0=off
gpgkey= #location of gpg key for integrity checking

#Example of creating a local repository from the RHEL ISO:
#first mount iso to filesystem
lsblk
mkdir /mnt/rhel.cd
mount /dev/sr0 /mnt/rhel.cd

#configure repository
[rhel_cd_repo]
name=rhel.cd
baseurl=file:///mnt/rhel.cd/BaseOS/
enabled=1
gpgcheck=0

#easier way to create this repository
dnf config-manager --add-repo=file:///mnt/rhel.cd/BaseOS/
#configuring remote repository
dnf config-manager --add-repo=http://SERVER.com/BaseOS/
#Note: if you are using the config-manager you may need to disable the
#gpgcheck as it may be enabled

Modify the system bootloader

The bootloader is located in the boot sector of your RHEL hard drive. It’s purpose is to load the kernel and initramfs. The configuration file for grub is located at /etc/default/grub. This file contains the options that are used by the GRUB boot loader. The most important line for the RHCSA is the GRUB_CMDLINE_LINUX line. This line contains the arguments for the way the kernel is booted. As previously stated it is useful to remove ‘rhgb quiet’ to make the boot process more verbose. If you modify this file it is necessary to regenerate the configuration file. The modification is made with grub2-mkconfig, along with the path of the of the main configuration file ending with .cfg. This file will be in different locations depending on whether the system is UEFI or BIOS as shown below.

#edit the /etc/default/grub file and set GRUB_TIMEOUT to 10 seconds
vi /etc/default/grub
GRUB_TIMEOUT=10
#writing modified configuration file on a system with BIOS
grub2-mkconfig -o /boot/grub2/grub.cfg
#writing modified configuration file on a system with UEFI
grub2-mkconfig -o /boot/efi/EFI/redhat/grub.cfg

# an easy way to check if your system is UEFI or BIOS is to check if the
# following directory exists: /sys/firmware/efi
#https://askubuntu.com/questions/162564/how-can-i-tell-if-my-system-was-booted-as-efi-uefi-or-bios

Manage basic networking

Configure IPv4 and IPv6 addresses

There are plenty of ways to edit the network configuration of a RHEL system, although there are two easy ways to make changes. The first way to edit the settings is simply through the GUI with nm-connection-editor. The second method is using the nmtui tool which has an interactive menu allowing you to easily change IP settings.

#check ip settings
ip a
#check link state of network interfaces
ip link show
#show routes
ip route show
#checking listening ports
ss -lt
nmap -p- localhost
#list UDP and TCP ports
ss -tul

Network configuration files are located: /etc/NetworkManager/system-connections/

Configure hostname resolution

Hostname resolution can be configured locally with the /etc/hosts file. The OS will first try to see if a name can be resolved locally before reaching out via DNS.

#set hostname resolution for google dns
8.8.8.8 google.dns
#check to ensure it worked
ping google.dns

#set hostname
hostnamectl set-hostname lol.mydomain.com
#verify
hostnamectl

Configure network services to start automatically at boot

#start a service (not persistent)
systemctl start sshd
#start a service automatically (persistent)
systemctl enable sshd
#starts and enables service in one command (persistent)
systemctl enable --now sshd
#stops persistent starting of service
systemctl disable sshd
#stops servie (not persistent)
systemctl stop sshd
#get status
systemctl status sshd
#get detailed info about service
systemctl status -l sshd
#list all services
systemctl -t service
#list failed services
systemctl --failed -t service

#note that you do not need to type out the full service name, systemctl
#appends .service to the command. You can also use autocompletion TAB,TAB

Restrict network access using firewall-cmd/firewall

Firewalld is the default firewall used in RHEL9 replacing iptables. Firewalld can be configured on the command line using the firewall-cmd utility. There is also a GUI version that can be accessed with firewall-config, you will obviously need to be in a graphical environment for this to work.

#get zones
firewall-cmd --get-zones
#get default zone
firewall-cmd --get-default-zone
#set default zone
firewall-cmd --set-zone=public
#get current firewall configuration
firewall-cmd --list-all
#add a service to firewall configuration (not persistent)
firewall-cmd --add-service=ssh
#add a service to firewall config (persistent)
firewall-cmd --add-service=ssh --permanent
#add a port to firewall config (persistent)
firewall-cmd --add-port=8000/tcp --permanent
#remove port (persistent)
firewall-cmd --remove-service ssh --permanent
#reload firewall to ensure changes are made
firewall-cmd --reload
# make runtime config persistent
firewall-cmd --runtime-to-permanent

# note if you do not add the --permanent options, the config will not be persistent

Manage users and groups

Create, delete, and modify local user accounts

create: useradd | modify: usermod | delete: userdel

#create user
#-m = create home
#-u = set UID
#-G = add user to group(s)
#-N = do not create user group, typically the primary group is same as uname
#-g = set primary group
useradd -m -u 1001 -G friends,family -N -g lolgroup john
#home directories are created based on /etc/skel
#useradd default file /etc/default/useradd

# ensure user cannot login
usermod -s /sbin/nologin john

#delete user and home dir
userdel -r john

#/etc/login.defs contains login-related variables including password expiration

Change passwords and adjust password aging for local user accounts

chage and passwd are the main tools used to adjust password aging.

#list password expiry data for user
chage -l john
#make user account expire
chage -E 2026-01-01 john

#adjusting with passwd
# -n 5 = must wait 5 days before changing
# -w 7 = warns user 7 days before expiring
# -x 120 = password expires in 120 days
passwd -n 5 -w 7 -x 120 john

Create, delete, and modify local groups and group memberships

Groups config file: /etc/group

#creating groups with group id
groupadd lolgroup -g 1111
#check members of group
lid -g lolgroup
#check your groups
groups ; id
#add john to lolgroup
usermod -aG lolgroup john
#change user primary group
usermod -g lolgroup john

Configure superuser access

sudoer file /etc/sudoer (visudo to edit)

#add user to wheel group (admin access group)
usermod -aG wheel john
#verify
visudo
#you should see: %wheel ALL=(ALL) ALL

# allowing user to execute specific commands with sudo
visudo
#add line
john ALL=/usr/bin/passwd, !/usr/bin/passwd root
#allows john to execute passwd as admin but does not allow changing root pass

Manage security

Configure firewall settings using firewall-cmd/firewalld

#get zones
firewall-cmd --get-zones
#get default zone
firewall-cmd --get-default-zone
#set default zone
firewall-cmd --set-zone=public
#get current firewall configuration
firewall-cmd --list-all
#add a service to firewall configuration (not persistent)
firewall-cmd --add-service=ssh
#add a service to firewall config (persistent)
firewall-cmd --add-service=ssh --permanent
#add a port to firewall config (persistent)
firewall-cmd --add-port=8000/tcp --permanent
#remove port (persistent)
firewall-cmd --remove-service ssh --permanent
#reload firewall to ensure changes are made
firewall-cmd --reload
# make runtime config persistent
firewall-cmd --runtime-to-permanent

#note if you do not add the --permanent options, the config will not be persistent

Manage default file permissions

Umask: umask sets the default permissions when files are created. If you create various files on a Linux filesystem you will notice that the permissions set are consistent. To change this default you must edit umask. umask like regular permissions can be expressed numerically, but that confused the hell out of me so I just use the symbolic form. To make the changes persistent there are two files we can modify: /etc/profile or our other option is to create /etc/profile.d/umask.sh with the desired umask. Lastly you can set the umask in the individual profile of said user /home/user/.profile (will not apply to all users).

#getting default umask (numeric) example output: 0022
umask
#getting default umask (symbolic) example output: u=rwx,g=rx,o=rx
umask -S
#set umask symbolic mode
umask u=rwx, g=rx, o=rx

Configure key-based authentication for SSH

#configuring key-based authentication
ssh-keygen #generates public/private key pair (follow prompts)
ssh-copy-id username@8.8.8.8.8 #adds public key to remote server
#edit the /etc/ssh/sshd_config and uncomment/change following line
PasswordAuthentication no
#login with key
ssh -p 777 username@8.8.8.8 -i .ssh/id_rsa

Set enforcing and permissive modes for SELinux

SELinux modes:

Enforcing: fully enforcing all SELinux rules in policy.

Permissive: SELinux activity is logged to /var/log/audit/audit.log but no activity is blocked

Disabled: Disabled -No SELinux policy is loaded.

Configuration file for SELinux: /etc/sysconfig/selinux. This configuration file is useful as it tells you exactly how to reconfigure SELinux enforcement modes as well as updating grub.

#view SELinux enforcment status
sestatus -v
#put SELinux in permissive mode (not persistent)
setenforce 0
#put SELinux in enforcing mode (not persistent)
setenforce 1
#making SELinux modes persistent
vi /etc/default/grub
#add following to line that contains LINUX
selinux=1
#update grub different for BIOS and UEFI systems
grub2-mkconfig -o /boot/grub2/grub.cfg
grub2-mkconfig -o /boot/efi/EFI/redhat/grub.cfg

#note in RHEL9 it is not possible to set the DEFAULT mode to disabled

List and identify SELinux file and process context

Context labels provide information about an object by defining a user, role and type. Some command line utilities use the argument ‘Z’ to list or work with these context labels. For example listing the context label for the webroot directory of Apache can be done with:

#List label
ls -lZ /var/www/html

#Note that if you are changing the default web-root directory for Apache-
#it can be helpful to copy this label and re-apply to new directory.

#Setting context types (do not use chcon, not persistent)
semanage fcontext -a -t httpd_sys_content_t "/new_web_root(/.*)?"
#next you will need to write the change to policy
restorecon -R -v /new_web_root

Restore default file contexts

There are two ways you can restore the default file contexts. The first is by creating the file /.autorelabel in the root directory. If this file exists then the entire file system will be relabeled upon a restart. This may not be ideal if you are only looking to restore the file context of one file/directory. Similarly you can use restorecon to relabel the entire file system or on a single file on disk.

#relable entire filesystem
touch /.autorelable
restorecon -Rv /
#relable directory recursivley
restorecon -Rv /new_web_root
#relabe single file
restorecon -v /home/user/test.txt

#the man page semanage-fcontext provides example config's and is useful

Manage SELinux port labels

SELinux port labels are particularly important when services are configured to use alternate ports. If for example you were to configure SSH to listen on port 777 rather than 22, then access to the port by default would be denied. You can see this if you type systemctl status sshd, an error message will appear noting that access to the port is denied. If you would like to test this, then edit /etc/ssh/sshd_config file and ensure you restart, enable and start the sshd service

#setting SELinux to non default port label
semanage port -a -t ssh_port_t -p tcp 777
#confirm ssh can now use non standard port
semanage port -l | grep ssh

Use boolean settings to modify system SELinux settings

SELinux policy uses boolean rules to allow or deny specific features in applications. A commonly used example is the anonymous write feature in FTP. This feature allows unauthenticated users to write to FTP directories. If you were to enable this feature in the FTP configuration file on a system with SELinux it would not work. To enable this feature you would need to modify the ftpd_anon_write boolean which is by default set to off.

#list booleans
getsebool -a | grep ftp
#list booleans (more detailed)
semanage boolean -l
#changing booleans (Not persistent)
setsebool ftpd_anon_write on
#changing booleans (persistent)
setsebool -P ftpd_anon_write on

Diagnose and address routine SELinux policy violations

The primary source to diagnose any problems with SELinux is to read the log file at /var/log/audit/audit.log. SELinux log entries contain the string AVC so you can easily filter out these messages using:

grep -i avc /var/log/audit/audit.log
#easier way with sealert
#install
dnf install -y setroubleshoot-server
#analyzing log
journalctl | grep sealert
#this will also give you another command to run if you want more detail
#it also provides recomendations which is nice

Manage containers

Find and retrieve container images from a remote registry

container images are specified in /etc/containers/registries.conf. Users running rootless containers can create ~/.config/containers/registries.conf. If there is a conflict the user file overrides the system file.

#install podman
dnf install container-tools
#list registries
podman info | grep -A 10 registries
#run from specific registry
podman run -d docker.io/library/nginx
#search registries for container
podman search nginx

Inspect container images

There are two tools used to inspect images; skopeo inspect and podman inspect. The advantage of skopeo is that you can inspect the image from the registry without having to pull the image. Podman inspect on the other hand can only inspect images that are on the local machine.

#list images
podman images
#inspect local image
podman inspect nginx
#inspect from registry
skopeo inspect docker://docker.io/library/nginx

Perform container management using commands such as podman and skopeo

#login to registry
podman login registry.access.redhat.com
#search specific registry
podman search registry.access.redhat.com/nginx

Build a container from a Containerfile

podman -t imagename:version ~/containerfiledir
#verify
podman images

Perform basic container management such as running, starting, stopping, and listing running containers

#list containers
podman ps -a
#start container
podman start nginx
#stop container
podman stop nginx
#restart
podman restart nginx
#kill
podman kill nginx
#remove image
podman rmi mariadb

Run a service inside a container

#start nginx container in detatched mode
podman run -d --name nginx -p 8000:80 nginx:latest
#open bash shell on container
podman exec -ti nginx /bin/bash
#start nginx but remove files after run
podman run --rm -d --name nginx -p 8000:80 nginx:latest

Configure a container to start automatically as a systemd service

#enable linger for user
loginctl enable-linger user
#generate systemd files
mkdir -p ~/.config/systemd/user ; cd ~/.config/systemd/user
podman generate systemd nginx --files
#configure container to start as service
systemctl --user enable container-nginx.service

Attach persistent storage to a container

mkdir /webroot
chown user /webroot
podman pull nginx
podman run -d --name nginx -p 8000:80 -v /webroot:/usr/share/nginx/html:Z nginx:latest
firewall-cmd --add-port=8000/tcp --permanent
fireall-cmd --reload

Extra

Reset root password

#Step 1 reboot computer
# press 'e' when GRUB boot menu is displayed
#Step 2 place the following in the 'linux' line
init=/bin/bash
#Step 3 press CTRL+x to boot with bash
#Step 4 mount filesystem with rw access type:
mount -o remount,rw /
#Step 5 change password
passwd
#Step 6 relable neccessary becasue of /etc/shadow change:
touch /.autorelable
#Step 7 start systemd to replace /bin/bash as PID 1
exec /usr/lib/systemd/systemd
#Step 8 reboot
reboot
#note without performing step7 you will be unable to reboot as it is a function
#of systemd

Lab setup

The goal of my lab setup was to create a VM that I could access while I had down time at work. I show how I configured this in windows 11 using Hyper-V here. I also used the trial version of RHEL 9 which is available for 60 days. I figured that should be enough time to study for/pass the exam. Also, for your lab setup I recommend adding extra storage, in Hyper-V I was able to use some extra storage space on my hard drive to create a vhd and attach it to the RHEL VM. This will let you configure partitioning and creation of LVM’s and mount them persistently.

Cockpit

The cockpit service came pre-installed on my RHEL, it can be used to get a GUI overview of the configuration of a RHEL server (a web terminal exists too). It could come in handy on the exam, although definitely not necessary as the command line is much quicker at getting things done. Documentation can be found here.

Troubleshooting issues

There are several troubleshooting options available when booting RHEL

rd.break: stops boot procedure in initramfs stage. Root directory is not yet mounted in this mode. Requires root password
init=/bin/sh: starts /bin/sh shell after loading kernel and initrd. Root directory is mounted, but is read only. Press CTRL+X to enter this mode.
systemd.unit=emergency.target: Enters a bare minimum mode with minimal systemd units loaded.
systemd.unit=rescue.target: Slightly more systemd units then emergency.target. Requires root password.

Reinstalling grub: If grub is damaged and is still in a bootable state you can use ‘grub2-install’ with the desired device name as an argument.

Fixing initramfs: If the root file system is not getting mounted on root directory and no systemd units are started then there is likely a problem with initramfs. A potential solution is to re-create initramfs by using ‘dracut — force’.

File system issues: If there is a misconfiguration regarding file-system mounts you typically see the message “Give root password for maintenance” generated by fsck. You should check blkid/lsblk with /etc/fstab to verify configurations.’ journal-xb’ to view messages related to this issue. If the file system itself has issues you may need to mount it with: ‘remount,rw /’

Use journalctl to read kernel messages with ‘journalctl -k’

Troubleshooting SELinux issues:

grep -i avc /var/log/audit/audit.log
dnf install -y troubleshoot-server
journalctl | grep sealert

RHCSA9 Exam Notes

Table of contents

Understand and use essential tools

Access a shell prompt and issue commands with correct syntax