SHRINK an Amazon AWS EBS root volume (2020 Update)

I had an AWS EC2 Instance which started at 20gb, gradually my disk usage increased so I expanded to 30gb, before I knew it I was already using 50gb.

The cost already doubled and it was necessary I audited my files to reduce the disk usage. It was really interesting figuring out what actually ate up my space (I will cover “How to Fee-up Some Space on your Linux Machine/Server” in another topic) and I recovered 34gb.

Now I need to shrink my disk from 50gb back to 29gb. All I could find in Google search results were outdated and some would even make you loose all your data.

These steps highlight how I did it successfully with 10 minutes downtime:

1. Take a snapshot of the EBS volume you want to shrink. (If you screw up at any step, this is your safeguard for recovery.)
2. Create a new EBS volume with the new shrunken size that you want. Please be sure that your new size will be enough to hold your existing data and have some free space. If the original volume is encrypted, make sure the new one is too.
3. Since this is a root volume, you can’t use the instance that you boot off of it. You will need to create a new AWS instance (a temporary one, with its own root volume). The easiest way to do this would simply be to select your existing instance and click on “Launch more like this” (in the Actions menu) which basically duplicates the configuration.
4. Detach the root volume you want to shrink from the instance you boot off of it, then attach it to the new instance as /dev/sdf. Also, attach the new empty volume to the new instance as /dev/sdg.
5. Start the new instance and once it’s online, SSH to it. If you had added your own key when creating the instance, you can just use the key to login. Otherwise you may need to enter a password.
6. Execute: sudo ls -l /dev/nv* … You should see a list of devices. Since this is a root volume, you should see nvme1n1 and nvme1n1p1 but for the new volume, you would see nvme2n1 only. (Note that /dev/s* becomes /dev/nv* in Ubuntu kernel. This is normal — just look at the suffixes like me1n1 and me2n1 to figure out the volume.)
7. Execute sudo dd bs=16M if=/dev/nvme1n1 of=/dev/nvme2n1 count=100 … This will copy the first 1680 MB of data from disk sdf to disk sdg. This is a real low-level clone of the first 1680 MB, which is much more than enough to clone the boot loader and the existing partition information.
   
   Side note: The goal here is only to clone the boot loader but since we don’t know the exact locations, we just copy the first 1680 MB whatever that maybe. The boot loader itself may not be more than a couple of MBs if it even goes beyond 1 MB. Also, the partition information just comes with this clone, but we will actually discard it in the next step.
8. Execute sudo fdisk /dev/nvme2n1 … This will launch the fdisk utility. Reminder: It’s NOT /dev/nvme1n1.
9. Once you’re in the fdisk command line, press p to see the partition information. You will see that there is 1 partition. Take note of the start sector number. You will need it.
10. Press d and delete the existing partition. Then press n to add a new partition and use the same start sector number that you noted as the “First Sector”. The rest of the options can be left at default. It looks like this:
    .
    Command (m for help): n[ENTER]
    Partition type:
    p primary (0 primary, 0 extended, 4 free)
    e extended
    Select (default p): [ENTER]
    Using default response p
    Partition number (1–4, default 1): [ENTER]
    Using default value 1
    First sector (2048–12582911, default 2048): 16065[ENTER]
    Last sector, +sectors or +size{K,M,G} (16065–12582911, default 12582911): [ENTER]
    Using default value 12582911
    .
    Side note: If you have more than one partition in the original volume, you will need to create as many partitions in the new volume as well, giving each partition enough space to hold the same amount of data. It would be a good idea to launch fdisk on /dev/nvme1n1 and view the partition information of the original volume.
11. Press a and select the new partition to add a boot flag on it.
12. Press w to save the changes and exit fdisk.
13. Now execute sudo ls -l /dev/nv* … You should see a list of devices. If you did everything correctly, you should be able to see nvme2n1p1 now.
14. Execute sudo e2fsck -f /dev/nvme1n1p1 … This checks the file system of the old volume and ensures that there are no errors.
15. f there are no errors, execute sudo resize2fs -M -p /dev/nvme1n1p1 … This is a very interesting command which pushes all the data to the beginning of the partition and resizes it down to fit just the data with no free space at all. The goal is to make sure there is no free space in between data blocks.
    .
    This command can take a while to finish. Keep calm and grab a coffee. If you’re worried about session disconnect, execute this using nohup (command) &
16. In the previous command’s output, the last line tells you the number of blocks. Each block is sized 4K but when we clone the partition, we are going to do it in 16 MB blocks. So, in order to compute the number of 16 MB, blocks, multiply the number in the last line by 4 / (16 * 1024). Round this number UP (not down) to the nearest integer.
    .
    Example: 1252939 (number in last line) * 4 / (16 * 1024) = 305.893310546875 … But round this UP to 306 or even 310 (it doesn’t matter as long as you don’t go below).
17. Execute sudo dd bs=16M if=/dev/nvme1n1p1 of=/dev/nvme2n1p1 count=(PUT THE NUMBER FROM PREVIOUS STEP HERE) … So for example, you would execute sudo dd bs=16M if=/dev/nvme1n1p1 of=/dev/nvme2n1p1 count=306 … This will clone the entire partition 1 from the old volume to the new volume.
    .
    This command can take a while to finish, and there won’t be any progress display. Keep calm and grab a coffee. You can test if session is still alive by pressing Enter key. If you’re worried about session disconnect, execute this using nohup (command) &
18. Execute sudo resize2fs -p /dev/nvme2n1p1 … This will expand the partition to fill up the remaining space of the volume.
19. Execute sudo e2fsck -f /dev/nvme2n1p1 … This will check the cloned data for any error. If there are no errors, you should be able to see that the number of files will match the same number from the output of step 14.
20. Now you may log out the temporary instance and stop it (shut it down). Careful, do NOT terminate it. Once it’s stopped, detach both the old and new volumes from it.
21. Attach the new volume to the original instance as /dev/sda1 (this is the root volume). Start the instance, and if everything went well, the instance should boot up like nothing ever happened.
22. You can delete the old volume and terminate the temporary instance. Keep the snapshot for a couple of weeks, then delete it once you’re sure everything is working fine.