What to do when your Bitcoin Full Node running on Digital Ocean fills up your storage space
A few months ago I started running a Bitcoin Full Node on a Digital Ocean droplet using their Volumes to store the ~160GB of blockchain data. I selected a 200GB volume (which, after partitioning and formatting, gave about 190GB of storage space). I wrote about it here.
Fast-forward to a couple weeks ago, and the blockchain data could had completely filled up the volume and the Bitcoin daemon had stopped running. I took some time today to increase the size of the volume to handle more data and the process wasn’t as easy as I hoped. If you followed the instructions from my original article, and find yourself in the same situation, follow the steps below to increase your volume size and get your full node back up and running.
Resize the Volume in the Digital Ocean Control Panel
Login to your Digital Ocean account, and navigate to the full node droplet Volumes section, expand the “More” dropdown and click the “Resize volume” link:
Pick a new size for your Volume (I increased mine by 20GB, which should buy be a couple more months time):
After a couple of seconds your volume should be resized and the entire process is now finished!
Just Kidding.
Server Prerequisites
Before the full node server can make use of those new gigs of space, we will need to resize the partition we created when we first setup the volume. In my case, I’m resizing the partition from a 200GB one to a 220GB one.
First, open the terminal and ssh back into your droplet. If you haven’t done this in awhile you may be faced with a message like this:
# ssh bitcoin@<droplet ip address>bitcoin@<droplet ip address>: Permission denied (publickey).
This happens when the ssh client doesn’t know about the private ssh key used to login to this server. In my original article, the private key we created was located at ~/.ssh/digital_ocean_droplet
so to add it to the ssh client just issue this command:
# ssh-add ~/.ssh/digital_ocean_droplet
After logging into the server, we need to get it ready for resizing the partition. Remember that we mounted the volume as a partition at /mnt/volume-lon1-03-part1
. To view the amount of free space this volume has (or in our case, doesn’t have), issue the df
command like so:
df -h -x tmpfs -x devtmpfsFilesystem Size Used Avail Use% Mounted on/dev/vda1 58G 1.5G 57G 3% //dev/vda15 105M 3.4M 102M 4% /boot/efi/dev/sda1 197G 187G 1.8m 100% /mnt/volume-lon1-03-part1
Before we start the resizing process, we need to unmount the partition from the filesystem, but we can’t do that until we ensure that no processes are currently accessing the partition. You can run this command to list all processes accessing the partition:
sudo lsof +f -- /mnt/volume-lon1-03-part1
If there are any processes listed, make sure you stop them first. If you see a long list of bitcoind entries, that means the bitcoin daemon is still running and can be stopped with sudo service bitcoin stop
.
NOTE: Whenever you run a command prefixed with
sudo
, you are running that command with root privileges.We setup thebitcoin
user with sudo capabilities so if asked for a password when running asudo
command, you will need to enter thebitcoin
user’s password you chose when doing the Initial Server Setup from my previous article.
After stopping all access to the partition filesystem, it’s time to unmount it with the umount
command:
sudo umount /mnt/volume-lon1-03-part1
You can confirm the partition has been unmounted by again running the df
command above and checking that the partition is no longer listed.
Rewriting the Partition Table
Now that we’ve unmounted the partition, it’s time we rewrite the partition table to take advantage of the extra space on the volume. We will first backup the existing partition table just in case anything goes wrong, and then resize it. For this, we will use the gdisk
tool. Before start the gdisk
process, make sure to switch to a directory that is writable:
cd ~
sudo gdisk /dev/disk/by-id/scsi-0DO_Volume_volume-lon1-03
Note how the file name passed to gdisk
above does not include the -part1
suffix. After running this, you will be shown the gdisk
command prompt like so:
GPT fdisk (gdisk) version 1.0.1Partition table scan:
MBR: protective
BSD: not present
APM: not present
GPT: presentFound valid GPT with protective MBR; using GPT.Command (? for help):
To perform the backup, type b
into the command prompt and hit enter, like so:
Command (? for help): bEnter backup filename to save:
Type in a filename for the backup (I chose partition_table.bak
):
Enter backup filename to save: partition_table.bak
The operation has completed successfully.Command (? for help):
Next we need to do a little bit of work to relocate the backup data structures to the end of the disk, so enter “Expert” mode by issuing the x
command, followed by e
, and then m
to get back to “Normal” mode:
Command (? for help): xExpert command (? for help): e
Relocating backup data structures to the end of the diskExpert command (? for help): mCommand (? for help):
Finally, we’re going to remove and recreate the partition. Remove the partition by issuing the d
command:
Command (? for help): d
Using 1Command (? for help):
Next, recreate the partition using the new size of the volume by issuing the n
command:
Command (? for help): n
Directly after issuing the n
command, the gdisk
tool will ask you to a enter a series of info about the new partition you want to create. You shouldn’t need to enter anything different than the default values offered by the prompt, so just keep hitting Enter/Return until you reach another command prompt. It’ll look something like this:
Command (? for help): nPartition number (1-128, default 1):
First sector (34-461373406, default = 2048) or {+-}size{KMGTP}:
Last sector (2048-461373406, default = 461373406) or {+-}size{KMGTP}:
Current type is 'Linux filesystem'
Hex code or GUID (L to show codes, Enter = 8300):
Changed type of partition to 'Linux filesystem'Command (? for help):
Now we just need to exit the gdisk
tool and confirm we want to make the changes by typing Y
when prompted, like so:
Command (? for help): wFinal checks complete. About to write GPT data. THIS WILL OVERWRITE EXISTING PARTITIONS!!Do you want to proceed? (Y/N): YOK; writing new GUID partition table (GPT) to /dev/disk/by-id/scsi-0DO_Volume_volume-lon1-03.The operation has completed successfully.
Expanding the Filesystem
Now that the partition is ready, we need to expand the filesystem to make use of the extra space. First, we need to run the e2fsck
command to make sure the unmounted partition is free of faults:
sudo e2fsck -f /dev/disk/by-id/scsi-0DO_Volume_volume-lon1-03-part1
After that has completed, the filesystem can be resized using the resize2fs
command:
sudo resize2fs /dev/disk/by-id/scsi-0DO_Volume_volume-lon1-03-part1
Now, after remounting the volume and running df
again, you should see the extra space as available to use:
sudo mount -a
df -h -x tmpfs -x devtmpfsFilesystem Size Used Avail Use% Mounted on
/dev/sda1 217G 187G 19G 91% /mnt/volume-lon1-03-part1
Now that you have more room to play with, you should be able to successfully restart the bitcoind daemon using sudo service bitcoin start
and confirm new blocks are being verified with the bitcoin-cli getblockchaininfo
command
Wrap up
If you have any problems with the above procedure and aren’t able to get your full node back up and running, please send me a note on Twitter. If you enjoyed this article please give me a clap or two and let me know. Happy noding!