source: Wikipedia

Dealing with Disks in Fedora VM

How to handle the “Low Disk Space on ‘Filesystem root’” error

Intro

When I started the Software Engineering program at the Flatiron School I hadn’t had much experience with MacOS. Having used PCs and dabbling with *nixes since the 90s, I never felt compelled to try MacOS out. However, for the in-person program it was highly recommended (read: requirements) to use a MacBook. Luckily, at that time (now, discontinued) they had a leasing program and I borrowed a Macbook Air, and soon fell in 💖.

Everything worked together comparatively seamlessly, and I should have expected it. My transition from a Motorola phone with Android to the iPhone (at my wife’s behest) was life-altering. Well, the months have flown by and my lease is now over and today I had to send back my Macbook to the Flatiron School via Fedex 😥.

Now, Mac-less, and afraid to buy a new Macbook with the recently announced change in processor technology at Apple, I’m planning to use a Linux Virtual Machine (referred to hereafter as VM) until the new models launch.

My Backup Plan — Linux Virtual Machines

As some of you know I set up Fedora (a popular Linux distribution) on a virtual machine using VMware’s Workstation to do my Flatiron coursework on my PC (bigger monitor, better processor and more memory). At the time, I wrote up an article here on Medium about my experience setting it up. It wasn’t as pretty as MacOS, but it was functional and much more straightforward of an installation than trying to install and run Ruby/PostgreSQL on Windows.

Some months have gone by, and I’ve slowly added more applications and updates to the Fedora VM. Now, some of you may know the saying:

Data expands to fill the space available for storage.

Which is an extension to C. Northcote Parkinson’s satirical Parkinson’s Law[1], which is an adage that says:

IT is a commonplace observation that work expands so as to fill the time available for its completion.

I’ve found that I have nearly filled the 40 GB of disk space I originally allotted and thought I’d never use. As I neared max capacity, a warning appeared thusly:

source: https://forums.linuxmint.com/viewtopic.php?t=246520

Well, I remember seeing an option to expand the hard disk size in the VM settings. A quick update to the maximum size (➡100 GB) and I reboot the machine to update the settings and we should be good, right?

Not quite. We’re about halfway there. What I’ve done is expand the size of the virtual hard drive from 40 GB to 100 GB, but VMware’s disk utility does not alter the partitions and file system of the operating system.

Working with Partitions and File System on Fedora

The first thing I have to say is BACKUP, BACKUP, BACKUP! If you are using VMware Workstation 15 Pro, there’s an option under the “Manage” menu to create a backup. If you’re using the (free) Workstation 15 Player version, its not listed as an option but is relatively straightforward to do.

I copied my VM folder to a new folder. Although copying the 39.1 GB of data (on an HDD) to that new folder took some time (~20 minutes), its worth it. Better safe than sorry. I’ve had the experience of corrupting a file system or partition and although it’s sometimes possible to recover data, in most cases, its simply lost.

If you want to learn more about partitions, Chris Hoffman’s article “Beginner Geek: Hard Disk Partitions Explained” at How-to Geek is a good article to start with.

Before I start, let me share my current system configuration:

• Fedora Thirty One — 5.7.11–100.fc31.x86_64
• VMware Workstation 15 player, version 15.5.2 build-15785246
• Windows 10, 64-bit (Build 19041) 10.0.19041

After backing up and starting the copied VM to make sure it was working properly, I closed the copy. Then, I start up the VM I’ll be working on and logged in as the root user. You can do this from the login screen or by using the su - (substitute user). Be mindful that if you use su (without the dash ‘-’) you will keep (most of) your current environment settings. Whereas, su - will reset your environment to emulate logging in as the user (setting home directory, environment variables and path)

Once you’re logged in as root, you can use df (disk free) to show the current file systems and their available disk space. We can see my / directory named /dev/mapper/fedora_localhost--live-root is at 97% capacity.

On Linux/*nix systems, there’s oftentimes different ways to receive output from commands and the df command is no different. I found it awkward working with 1k-blocks of data to show disk space, so I used the man command to reference the df manual pages.

(aside 1: if you’ve ever read the term RTFM, these are the manual pages they are referring to. And it was a kind of snarky way of telling people to learn for themselves. I was never personally a fan of the practice back in the 90s/2000s, having been one of those newbies who always needed help, and I’m glad the tech culture is changing to one of helpfulness and information-sharing)

(aside 2: a nice feature of zsh is the ability to reference the man command while you are typing it. For example, you might have typed the command ps (which shows active processes — much like Activity Monitor on MacOS and Task Manager on Windows) but forgot the exact options (switches) you want to use. You can type [ESC], then ‘h’ and it will pull up the manual page. After you look up your options you can hit ‘q’ and you cursor will be back at spot you hit [ESC].

$ ps [ESC]h

This may not be useful in this case, but if you have already invested time in typing a long command such as:

$ for i in {1..30}; do echo $i; done | cut -c 2 | sort | uniq

And want to know the options for the uniq command (the last command before [ESC] is the one referenced), it can save you time and keystrokes.

(aside 3: You’ll find that your default shell as the root user might be bash , and you might not want to change that. There may be scripts or processes that are relying on the root user’s shell to be bash . In this case, you can set the shell to zsh just for interactive shells (as opposed to a non-interactive one) by adding this code to your /root/.bashrc:

if [ ! -z "$PS1" ]     # -z returns TRUE if length of $PS1 is zero
then                   # $PS1 is the primary prompt string that is
    exec /bin/zsh $*   # loaded when a user logs in.
fi

To see what your current shell is, you can use ps -p $$ or echo $0 . Be careful when using echo $SHELL because it may display the default shell for the current user, but not the active shell. For example, when we use the code above. The environment variable $SHELL will show /bin/bash , while echo $0 will show /bin/zsh.)

All asides aside, let’s look at the manual and we can see two options (aka ‘switches’) that might make the output more readable. The -h or --human-readable switch and the -H or --si option.

I don’t know which side fence you are on regarding 1000 vs 1024 (or Kilo/Mega/Giga vs Kibi/Mebi/Gibi) but I prefer 1000 or the terms kilobyte (kb), megabyte (mb), gigabyte (gb) for their familiarity. This would be the -H or --si (for International System of Units,SI abbreviated from French — Système international).

We re-run the df command with our new switch df --si or df -H and the terminal outputs:

Taking this idea of “different ways to receive command output” a step further, I did a search to see if there were any packages for df in Fedora’s software package manager dnf using dnf search df and received back many screen-fulls of data that looked like this:

It looks like dnf gave us back every listing that had the letters ‘d’ and ‘f’ in it. That’s not quite what I wanted, what I really wanted was any packages with the phrase “(space)df(space)”. So, let’s try and use the grep command to globally search for a regular expression and print matching lines (g/re/p). We can do this by using a pipe | to send the output of the dnf search df command to input of the grep command.

$ dnf search df | grep " df "

And we see one match — described as “Fully colorized df clone written in python”. Let’s go ahead and install it! (yeah, yeah I know, I’m running out of disk space and that’s how I got in this mess, but its only 18 KB! 🤓)

Let’s take a look and see how pydf looks!

Sweet!

Wait, what were we doing again?

Oh yeah, okay, need to use our expanded disk capacity.

Now that we’ve (colorfully and beautifully organizadly) confirmed we are low on disk space, we can move on. Our next step is to use the fdisk command.

The fdisk command let’s us manipulate the disk partition table. Its a command line interface (CLI) and reminds me of the first Ruby project we did at Flatiron School. That brings back fuzzy, happy memories of my first stumbling steps into programming, but I’m not deceived because I know that fdisk is dangerous.

As soon as you start the command it warns us “Changes will remain in memory only, until you decide to write them. Be careful before using the write command.” because you can literally delete your whole OS.

Create a New Partition

First, we’ll initialize that new (virtual) disk space and add it as a partition to our Fedora file system using fdisk .

Let’s get started. We run:

$ fdisk /dev/sda

Then:

1. Enter “n” to “add a new partition”.
2. Enter “p” to create a “primary partition”.
3. I’ll take the default Partition number of 3.
4. I’ll also choose defaults for the First sector and last sector.

And, a new partition 3 of type ‘Linux’ and of size 60 GiB was created. The last thing I want to do is set the type to Linux LVM (code 8e).

5. Enter ‘t’ and select our partition number, 3, and enter 8e.

6. Lastly, we ‘w’rite the changes and reboot the machine.

Adding a Disk to LVM

After logging back in, and then switching to the root user, I’m ready to use the lvm (Logical Volume Manager) to create a new PV (physical volume) from our previously created partition. With this PV, I will extend the size of the VG (volume group). And this, in turn, will be used to extend the LV (logical volume).

“LVM Hierarchy” by xmodulo is licensed under CC BY 2.0

For more information on LVM and the way storage is handled, check the Red Hat Deployment Guide, Chapter 11. LVM (Logical Volume Manager) page.

I can see each of the different parts of the file system using the commands:

• pvs — display info about physical volumes
• vgs — display info about volume groups
• lvs — display info about logical volumes
• lsblk — list block devices

First, we run lsblk (list block devices) to show all our block devices and confirm that our newly created partition exists. We can see it under the sda listings as sda3 , with our 60G of space. We want to connect it to sda2 under live-root .

What is a “block device”?

Briefly, “Block devices are characterized by random access to data organized in fixed-size blocks. Examples of such devices are hard drives, CD-ROM drives, RAM disks, etc.” [2] This is opposed to character devices which work byte by byte (keyboard, mouse, etc).

Create PV (Physical Volume):

Using pvcreate on our partition /dev/sda3 and then view it with pvs . We can see our new partition has been converted to a PV and we see 60.00g free, along with /dev/sda2 which is our original root drive.

Extend VG (Volume Group)

Using vgextend we attach our newly created PV to the root drive’s VG. After our VG has been extended, we can see that the 60 GB of space has been added to the original 40 GB and we see the VSize is 98.99g (100 GB) with a Vfree size of 60.00 GiB.

Grow LV (Logical Volume)

Finally, we want to grow the LV and allow it access to disk space we added. The command lvdisplay gives us a summary of our LV. What I’m interested here is the second Logical Volume, /dev/fedora_localhost-live/root .

Next, I’ll use the vgdisplay command to show the Volume Group overview, and I can see here that my Free PE/Size is 60.0 GB. The total VG size is 98.99 (100 GB) and everything looks set for me to extend that second Logical Volume (LV), /dev/fedora_localhost-live/root so I run the lvextend command using the switch -l +100%FREE which utilizes all available free space in the VG.

It looks like it has successfully resized the LV! I confirm by using lvdisplay which shows an updated LV Size of <95.05 GiB.

Last Step: Resize the File system

The final step is to use the resize2fs command to enlarge the filesystem to fill the LV (Logical Volume). Then I use lvs to check the size of the /root LV and then df to check that our free space has increased, and it has!

Conclusion

Looks like I’m all set to continue using and upgrading Fedora for the foreseeable future without worrying about running out of disk space. I’m pretty amazed and thankful to the FOSS coders who built such a resilient, novice proof system. Other than the (sometimes) overwhelming documentation and manuals, the process was pretty painless and if I hadn’t been writing at the same time I was learning, I’m sure it would have taken a half-hour to finish. I’m really glad that worked out so smooth!

Footnotes and References:

1. “Parkinson’s Law”. C. Northcote Parkinson. The Economist. 11/19/1955. https://www.informit.com/articles/article.aspx?p=2429466. Accessed 8/17/20.
2. “Block Device Drivers”. The Linux Kernel; 4.19.0. https://linux-kernel-labs.github.io/refs/heads/master/labs/block_device_drivers.html#. Accessed 8/17/20.
3. “How to create a volume group in Linux with LVM”. Tyler Carrigan. Red Hat. 04/06/20. https://www.redhat.com/sysadmin/create-volume-group. Accessed 8/17/20.
4. “How to Extend/Reduce LVM’s (Logical Volume Management) in Linux — Part II”. Babin Lonston. TecMint. 06/27/17. https://www.tecmint.com/extend-and-reduce-lvms-in-linux/. Accessed 8/17/20.
5. “zsh set as default shell for root”. Stack Exchange — Unix & Linux. 11/23/16. https://unix.stackexchange.com/questions/318418/zsh-set-as-default-shell-for-root. Accessed 8/17/20.
6. “How do I find out what shell I am using on Linux/Unix?”. NixCraft. 12/14/19. https://bit.ly/2FtDwFX. Accessed 8/17/20.