Extending an EBS Root Volume

Today I was tasked with the simple exercise of extending an EBS root volume for a client and figured I would document the process in a blog entry while waiting for the drive to optimize.

Step 1

Per the AWS doc on Modifying the Size, IOPS, or Type of an EBS Volume on Linux, I verified whether this particular c4.4xlarge instance is a member of the current generation instances, and it is. So long as the volume was created after Amazon pushed out the updates required to be considered ‘current generation’, this means we should not need to shut the instance down during the process of modifying the volume.

Because the one partition we are modifying however is the root partition, we will need to shut the instance down and detach the volume later on to complete the process of extending the volume size.

Next, I used the console to find the respective volume and issue the `Modify Volume` command:

Once I updated the size from 150GB to 250GB, the IOPS automatically updated from 450 to 750 as well:

Step 2

After clicking `Modify`, you can monitor the progress via the console or the command line as follows:

AWS CLI

I used a Linux `watch` command to keep an eye on the progress without needing to constantly refresh the console. Note the “Progress” value; also notice it shows both the target size and IOPS as well as the original:

Every 2.0s: aws ec2 describe-volumes-modifications --region us-east-1 --volume-id vol-0032717c9c20XXXXX Fri Jan 5 20:16:03 2018
{
    "VolumesModifications": [
        {
        "TargetSize": 250,
        "TargetVolumeType": "gp2",
        "ModificationState": "optimizing",
        "VolumeId": "vol-0032717c9c2XXXXXX",
        "TargetIops": 750,
        "StartTime": "2018-01-05T19:47:35.249Z",
        "Progress": 35,
        "OriginalVolumeType": "gp2",
        "OriginalIops": 450,
        "OriginalSize": 150
      }
    ]
}

Per the AWS documentation on Monitoring the Progress of Volume Modifications, you should still be able to use your volume during this current generation modification process. Your performance will be between the source and target configuration, no less than the source volume performance if you are increasing the size. If you are decreasing the size of your volume, the transitional performance will be no less than the target volume performance.

AWS Console

Using the AWS Console, you can navigate to the EC2 Service and click on Volumes. Locate the volume you are modifying and expand the ‘State’ column. It should show “in-use - modifying” initially and then “in-use - optimizing” with the percentage complete.

Once complete, you should see “in-use - completed (100%)” and you can view the volume modification details as shown:

Step 3

In this example, as shown below, we have only one partition to extend and we must make the file system aware of the new volume size before it can be used. Using `lsblk`, we can see our root partition is only aware of 150G of our now 250G sized volume. We must resize this root partition to use the remaining space on the volume.

[root@ip-10-X-X-X ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvda 202:0 0 250G 0 disk
└─xvda1 202:1 0 150G 0 part /

This AWS doc on Expanding a Linux Partition does a nice job of laying out the steps necessary to expand a root partition, so we will be following the steps as outlined.

Prepare Root Partition for Expansion

First, we’ll use the following command to test the read integrity of the entire device without writing to it:

$ sudo badblocks -b 4096 -c 4096 -s /dev/xvda
Checking for bad blocks (read-only test): 46.68% done, 1:02 elapsed. (0/0/0 errors)
# When completed, it shows ‘done’ with no errors:
$ sudo badblocks -b 4096 -c 4096 -s /dev/xvda
Checking for bad blocks (read-only test): done
# We could try a non-destructive read-write check, but since our partition is mounted at root, badblocks won’t let us:
$ sudo badblocks -b 4096 -c 4096 -s /dev/xvda -n
/dev/xvda is apparently in use by the system; it's not safe to run badblocks!

Next, we shutdown our server so we can detach the root volume and we will also take a snapshot as a safety precaution. This way we can start over in case we make a mistake or find something is indeed corrupt on the new volume.

We will make note of the device name the volume is currently attached to, which in our case is /dev/xvda. Now we can detach the volume from our primary instance.

Once the volume is detached, we must attach it as a secondary volume to another instance in the same Availability Zone. Our volume is not encrypted, so I have launched a temporary t2.micro instance for our purposes. Once our t2.micro instance is up and running, we can SSH in and attach the secondary volume. Run `lsblk` to see our volume:

# Before attaching secondary volume
[root@ip-10-N-N-N ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvda 202:0 0 8G 0 disk
└─xvda1 202:1 0 8G 0 part /
# After attaching secondary volume
[root@ip-10-N-N-N ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvda 202:0 0 8G 0 disk
└─xvda1 202:1 0 8G 0 part /
xvdb 202:16 0 250G 0 disk
└─xvdb1 202:17 0 150G 0 part

Per Amazon, regarding `gdisk` vs. the `parted` utility for editing MBR and GPT partition tables:

Some versions of parted (newer than version 2.1) have limited support for GPT partition tables and they may cause boot issues if their version of parted is used to modify boot volumes. You can check your version of parted with the parted — version command.

[root@ip-10-N-N-N ~]# fdisk -l
WARNING: fdisk GPT support is currently new, and therefore in an experimental phase. Use at your own discretion.
Disk /dev/xvda: 8589 MB, 8589934592 bytes, 16777216 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: gpt
# Start End Size Type Name
1 4096 16777182 8G Linux filesyste Linux
128 2048 4095 1M BIOS boot parti BIOS Boot Partition
Disk /dev/xvdb: 268.4 GB, 268435456000 bytes, 524288000 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x00000000
Device Boot Start End Blocks Id System
/dev/xvdb1 1 314572799 157286399+ ee GPT
[root@ip-10-N-N-N ~]# parted --version
parted (GNU parted) 2.1
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later .
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Written by http://parted.alioth.debian.org/cgi-bin/trac.cgi/browser/AUTHORS.

Since our volume is using a GPT partition table, I’ve chosen to give the `gdisk` utility a test run since I’ve never used it before, rather than `parted`, even though `parted --version` shows we are at version 2.1. You can read more about the gdisk utility here if you’re interested.

Jumping to the section on expanding a Linux partition using gdisk in the AWS documentation, we continue to follow along. We’ve already identified our partition to expand as /dev/xvdb on our t2.micro instance with 250G available but only 150G allocated.

We can also see our partition is not mounted so there is no need to unmount, and we have already taken a snapshot earlier in case something goes awry and we need a do-over. Be sure to take note of all of your GUID, partition numbers, start and end sectors, the code indicating partition type and the names as shown below using the `gdisk -l` command.

# Notice there is no MOUNTPOINT for our xvdb1 partition
[root@ip-10-N-N-N ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvda 202:0 0 8G 0 disk
└─xvda1 202:1 0 8G 0 part /
xvdb 202:16 0 250G 0 disk
└─xvdb1 202:17 0 150G 0 part
# We can use the gdisk command to verify it is a GPT partition
[root@ip-10-N-N-N ~]# gdisk -l /dev/xvdb
GPT fdisk (gdisk) version 0.8.10
Partition table scan:
MBR: protective
BSD: not present
APM: not present
GPT: present
Found valid GPT with protective MBR; using GPT.
Disk /dev/xvdb: 524288000 sectors, 250.0 GiB
Logical sector size: 512 bytes
Disk identifier (GUID): C322C84F-CCFE-4D5A-914A-8CBE33A33826
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 314572766
Partitions will be aligned on 2048-sector boundaries
Total free space is 2014 sectors (1007.0 KiB)
Number Start (sector) End (sector) Size Code Name
 1 4096 314572766 150.0 GiB 8300 Linux
 128 2048 4095 1024.0 KiB EF02 BIOS Boot Partition

Using gdisk to Expand a Linux Partition

Now we are ready to run the gdisk utility to resize our partition.

[root@ip-10-N-N-N ~]# gdisk /dev/xvdb
GPT fdisk (gdisk) version 0.8.10
Partition table scan:
MBR: protective
BSD: not present
APM: not present
GPT: present
Found valid GPT with protective MBR; using GPT.
# Here are a list of valid gdisk commands, very similar to fdisk but
# comparing the two would require another blog post on its own
Command (? for help): ?
b back up GPT data to a file
c change a partition's name
d delete a partition
i show detailed information on a partition
l list known partition types
n add a new partition
o create a new empty GUID partition table (GPT)
p print the partition table
q quit without saving changes
r recovery and transformation options (experts only)
s sort partitions
t change a partition's type code
v verify disk
w write table to disk and exit
x extra functionality (experts only)
? print this menu

We can use the `p` option to view the existing partition table; remember to make note of your GUID and ALL of your partition numbers, start and end sectors, code and names here!

Command (? for help): p
Disk /dev/xvdb: 524288000 sectors, 250.0 GiB
Logical sector size: 512 bytes
Disk identifier (GUID): C322C84F-CCFE-4D5A-914A-8CBE33A33826
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 314572766
Partitions will be aligned on 2048-sector boundaries
Total free space is 2014 sectors (1007.0 KiB)
Number Start (sector) End (sector) Size Code Name
 1 4096 314572766 150.0 GiB 8300 Linux
128 2048 4095 1024.0 KiB EF02 BIOS Boot Partition

Next we’ll use the `o` option to create a new empty partition table.

Command (? for help): o
This option deletes all partitions and creates a new protective MBR.
Proceed? (Y/N): Y

Now we can use the `n` option to create new partitions. This is where you will need to enter the information you collected earlier; the partition number, start and end sectors, code type and name for each of your partitions. You can use the `c` option to customize the name if desired, otherwise leave them as the default names given such as in this example.

Command (? for help): n
Partition number (1-128, default 1): 1
First sector (34-524287966, default = 2048) or {+-}size{KMGTP}: 4096
Last sector (4096-524287966, default = 524287966) or {+-}size{KMGTP}:
Current type is 'Linux filesystem'
Hex code or GUID (L to show codes, Enter = 8300): 8300
Changed type of partition to 'Linux filesystem'
Command (? for help): n
Partition number (2-128, default 2): 128
First sector (34-4095, default = 2048) or {+-}size{KMGTP}: 2048
Last sector (2048-4095, default = 4095) or {+-}size{KMGTP}: 4095
Current type is 'Linux filesystem'
Hex code or GUID (L to show codes, Enter = 8300): EF02
Changed type of partition to 'BIOS boot partition'

Use `x` to enter into the advanced ‘experts functionality’ menu and then `g` to update the disk’s unique GUID that you made note of earlier.

Command (? for help): x
Expert command (? for help): ?
a set attributes
c change partition GUID
d display the sector alignment value
e relocate backup data structures to the end of the disk
g change disk GUID
h recompute CHS values in protective/hybrid MBR
i show detailed information on a partition
l set the sector alignment value
m return to main menu
n create a new protective MBR
o print protective MBR data
p print the partition table
q quit without saving changes
r recovery and transformation options (experts only)
s resize partition table
t transpose two partition table entries
u replicate partition table on new device
v verify disk
w write table to disk and exit
z zap (destroy) GPT data structures and exit
? print this menu
Expert command (? for help): g
Enter the disk's unique GUID ('R' to randomize): C322C84F-CCFE-4D5A-914A-8CBE33A33826
The new disk GUID is C322C84F-CCFE-4D5A-914A-8CBE33A33826

Now we can enter `w` to write our changes and quit.

Expert command (? for help): w
Final checks complete. About to write GPT data. THIS WILL OVERWRITE EXISTING
PARTITIONS!!
Do you want to proceed? (Y/N): Y
OK; writing new GUID partition table (GPT) to /dev/xvdb.
The operation has completed successfully.

Running `lsblk` now shows our updated partition using the full 250G of space.

[root@ip-10-N-N-N ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvda 202:0 0 8G 0 disk
└─xvda1 202:1 0 8G 0 part /
xvdb 202:16 0 250G 0 disk
└─xvdb1 202:17 0 250G 0 part

Lastly, since we are running on ext4, we will use `e2fsck` to check the file system integrity.

[root@ip-10-N-N-N ~]# file -sL /dev/xvdb1
/dev/xvdb1: Linux rev 1.0 ext4 filesystem data, UUID=5730e0f7-7693-4888-8ec0-9863d114470c (extents) (large files) (huge files)
[root@ip-10-N-N-N ~]# e2fsck -f /dev/xvdb1
e2fsck 1.42.12 (29-Aug-2014)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/: 90431/9830400 files (0.2% non-contiguous), 19035275/39321083 blocks

Attach Volume to Primary Instance

Now that our volume is extended we are ready to detach from our secondary t2.micro and attach to the original primary instance.

Once reattached, we can see using `lsblk` that our partition is making full use of our updated volume size. `df -h` shows our partition data is still intact right where we left off with 71G in use. We are all done!

[root@ip-10-X-X-X ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvda 202:0 0 250G 0 disk
└─xvda1 202:1 0 250G 0 part /
[root@ip-10-X-X-X ~]# df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 15G 60K 15G 1% /dev
tmpfs 15G 0 15G 0% /dev/shm
/dev/xvda1 246G 71G 176G 29% /