Moonbeam Node Snapshots — The CertHum Approach — Part III
Deploy the Scripts!
In Part II of this series we looked at some of the reasons that we think object storage on hyperscale cloud is the best destination to send your Moonbeam parachain and relay chain database snapshots. We also looked at the CertHum snapshot technology stack, as well as the dependencies needed to run the CertHum scripts. In this part we will provide the scripts we use to make a parachain and relay chain database snapshot for a node running on the Moonbeam network. These scripts can easily be adapted to for use on both Moonbase Alpha and Moonriver.
The reason we use shell scripts for these snapshots is because we want something which is simple, quick, and which will get the job done. We’ll be the first to admit that there are more elegant ways to accomplish our objectives that would perhaps be more extensible, but that’s not what we were looking for in creating these scripts. Simple was the operative word. The shell scripts we are using have been loosely developed based on the Ubuntu documentation found here.
We are also aware that improvements can be made to variable usage and other areas in these scripts and welcome any forks of the repo which we will publish in Part IV of the series.
Caution: These scripts will stop your Moonbeam service for an extended period of time. We only use this on a collator node which is dedicated for snapshot generation purposes. Be sure to disable the cron job if you use this on a node that you bring into production.
First, let’s create the empty file in nano editor which will hold the script to backup the Polkadot relay chain database on our Moonbeam node:
sudo nano /usr/local/bin/moonbeam-polkadot-backup.shThen, paste in the below script which will:
- Stop your Moonbeam service
- Compress the Polkadot relay chain database to a separate file
- Upload the file to Azure Blob storage
- Restart the Moonbeam service
- Delete the local file
We’ve provided extra details in the script comments so that you can see what each part does, and make it easier to edit the script if your setup requires something different.
Also, always make sure you have enough space on your server to accommodate the local snapshot file, and also plan for disk usage growth.
####################################
#
# Backup Moonbeam-Polkadot Database
#
#####################################Chain - We set the variable for the chain we will be using. This
#will help with using the script on different chains
chain="moonbeam-polkadot"
chaindb="polkadot"#User where we will create the snapshot file
#We are using a generic user for the directory
#An improvement would be to create a specific backup service account
user="ubuntu"#Azure Blob target URI. This will be specific to your you blob #deployment and will be how your server communicates from Azure. You #generate this URI via the Azure portal. Think of it as a secure key #into your object storage with a direct locatorazure_uri='YOUR-URI-FROM-AZURE - KEEP IT SECRET -- KEEP IT SAFE'#Source RocksDB Directory -- we use the default location as #described in the CertHum blog.
backup_files="/var/lib/moonbeam-data/polkadot/chains/$chaindb/db"#Target is our backup user home Directory.
dest="/home/$user"#Target filename for our backup, we are passing in the variable #defined above for our chain. Perhaps 'archive' could be improved.
archive_file="$chain-backup.tar.zst"#A dict file is specific to Zstandard and helps with decompression
dict_file="$chain-sst.dict"#A text file which provides details of our snapshot
archive_text="$chain-info.txt"#Stop Moonbeam service - put in your service name and this could be #improved by useing a variable.
systemctl stop 'YOUR-SERVICE-NAME'.service# Print start status message.
echo "Moonbeam service stopped"#Create new dictionary - this creates the dict file and you can find #more details on the Zstandard documentation
zstd --train `find "$backup_files" -name '*.sst' | shuf -n 200`#Zstandard names the file dictionary and we change that to our name
mv dictionary $dest/$dict_file#create new text file for our snapshot info
touch $dest/$archive_text
chmod 777 $dest/$archive_text# Print start status message.
echo "Backing up $backup_files to $dest/$archive_file"
date
echo# Backup the files using tar. NOTE -- we have to put in the static #dict file name. A todo is determine how to pass in the variable to #the zstd command. There are a lot of useful features including how #many processors to use and we recommend reading the Zstandard #manual to see how to use them.
tar -I 'zstd -v -15 -T0 -D /home/ubuntu/moonbeam-polkadot-sst.dict' -cvf $dest/$archive_file --strip-components=5 $backup_files# Print end status message.
echo
echo "Backup finished"
date#Append date to file
printf "Generated on: " >> $dest/$archive_text
date >> $dest/$archive_text#Append Moonbeam version to file
printf "Extracted from Moonbeam version: " >> $dest/$archive_text
/var/lib/moonbeam-data/moonbeam --version >> $dest/$archive_text#Append size file to info
ls -l -h $dest/$archive_file | awk '{print "Size =",$5}' >> $dest/$archive_text#Start Moonbeam service - we don't need the service stopped to push #the data up to Azure
systemctl start 'YOUR-SERVICE-NAME'.service# Print start status message.
echo "Moonbeam service started"# Print start status message.
echo "Uploading to Azure BLOB"#Send to BLOB
azcopy copy "$dest/$archive_file" "$azure_uri"
azcopy copy "$dest/$dict_file" "$azure_uri"
azcopy copy "$dest/$archive_text" "$azure_uri"#If you want to move these files to another server, this is where
#you can add the commands to SCP the files to the destination# Print start status message.
echo "Uploading to Azure Blob"#Cleanup - we delete the files we created. When you first test your #script it may help to comment these out in case you have problems #uploading, so you don't delete the files and need to compress them #again. Once you know the upload works, uncomment them.
rm $dest/$archive_file
rm $dest/$dict_file
rm $dest/$archive_text# Print start status message.
echo "Deleted backup files"# Print start status message.
echo "Upload Complete"
Once we have finishe creating the script, we want to make it executable by issuing the following command.
sudo chmod +x /usr/local/bin/moonbeam-polkadot-backup.shNow let’s run the script. You should see a lot of output on the terminal if Zstandard is working — it will journal all of the files it is compressing.
This can take a while, and if you run the command from your terminal session, depending on your setup, you may timeout so here’s a way to disown the process so that it continues even if your session ends:
ctrl-z
disown -h //you may need to put in the process id here
bgThe status messages should continue to write to the terminal and if all goes well you should see your snapshot uploading to your Azure Blob container.
Life has taught us that things don’t always go as planned the first time, and so if you run into issues, you can write the script output to a text file for analysis as follows:
sudo /usr/local/bin/moonbeam-polkadot-backup.sh >> my_snapshot_output.txtWhen the script successfully completes you should see the new snapshot in the Azure portal. Note: this may take up to an hour or more depending on your bandwidth and server hardware.
That takes care of the Polkadot relay chain, and now we need a snapshot of the Moonbeam parachain. This is just a matter of creating a similar script to the one we just created for the relay chain.
NOTE: Why do we create one script for the relay chain and one for the parachain? For a couple of reasons. First, it’s a bit easier to troubleshoot errors by breaking it up into two parts, and if we were using a more powerful language we would just create these as separate modules. Second, there may be times we want just want to create a one-off backup of either the relay chain or parachain database and this makes it easier to do that.
Again, we create the empty file in nano editor which will hold the script to backup the Moonbeam parachain database on our Moonbeam node:
sudo nano /usr/local/bin/moonbeam-backup.shThen, we paste in the following script which will follow the same steps as when we made a snapshot of the relay chain database:
####################################
#
# Backup Moonbeam Database
#
#####################################Chain - We set the variable for the chain we will be using. This
#will help with using the script on different chains
#Chain
chain="moonbeam"
chaindb="moonbeam"#User where we will create the snapshot file
#We are using a generic user for the directory
#An improvement would be to create a specific backup service account
user="ubuntu"#Azure Blob target URI. This will be specific to your you blob #deployment and will be how your server communicates from Azure. You #generate this URI via the Azure portal. Think of it as a secure key #into your object storage with a direct locatorazure_uri='YOUR-URI-FROM-AZURE - KEEP IT SECRET -- KEEP IT SAFE'#Source RocksDB Directory -- we use the default location as #described in the CertHum blog.
backup_files="/var/lib/moonbeam-data/chains/$chaindb/db"#Target is our backup user home Directory.
dest="/home/$user"#Target filename for our backup, we are passing in the variable #defined above for our chain. Perhaps 'archive' could be improved.
archive_file="$chain-backup.tar.zst"#A dict file is specific to Zstandard and helps with decompression
dict_file="$chain-sst.dict"#A text file which provides details of our snapshot
archive_text="$chain-info.txt"#Stop Moonbeam service - put in your service name and this could be #improved by useing a variable.
systemctl stop 'YOUR-SERVICE-NAME'.service# Print start status message.
echo "Moonbeam service stopped"#Create new dictionary - this creates the dict file and you can find #more details on the Zstandard documentation
zstd --train `find "$backup_files" -name '*.sst' | shuf -n 200`#Zstandard names the file dictionary and we change that to our name
mv dictionary $dest/$dict_file#create new text file for our snapshot info
touch $dest/$archive_text
chmod 777 $dest/$archive_text# Print start status message.
echo "Backing up $backup_files to $dest/$archive_file"
date
echo# Backup the files using tar. NOTE -- we have to put in the static #dict file name. A todo is determine how to pass in the variable to #the zstd command. There are a lot of useful features including how #many processors to use and we recommend reading the Zstandard #manual to see how to use them.
tar -I 'zstd -v -15 -T0 -D /home/ubuntu/moonbeam-sst.dict' -cvf $dest/$archive_file --strip-components=5 $backup_files# Print end status message.
echo
echo "Backup finished"
date#Append date to file
printf "Generated on: " >> $dest/$archive_text
date >> $dest/$archive_text#Append Moonbeam version to file
printf "Extracted from Moonbeam version: " >> $dest/$archive_text
/var/lib/moonbeam-data/moonbeam --version >> $dest/$archive_text#Append size file to info
ls -l -h $dest/$archive_file | awk '{print "Size =",$5}' >> $dest/$archive_text#Start Moonbeam service - we don't need the service stopped to push #the data up to Azure
systemctl start 'YOUR-SERVICE-NAME'.service# Print start status message.
echo "Moonbeam service started"# Print start status message.
echo "Uploading to Azure BLOB"#Send to BLOB
azcopy copy "$dest/$archive_file" "$azure_uri"
azcopy copy "$dest/$dict_file" "$azure_uri"
azcopy copy "$dest/$archive_text" "$azure_uri"#If you want to move these files to another server, this is where
#you can add the commands to SCP the files to the destination# Print start status message.
echo "Uploading to Azure Blob"#Cleanup - we delete the files we created. When you first test your #script it may help to comment these out in case you have problems #uploading, so you don't delete the files and need to compress them #again. Once you know the upload works, uncomment them.
rm $dest/$archive_file
rm $dest/$dict_file
rm $dest/$archive_text# Print start status message.
echo "Deleted backup files"# Print start status message.
echo "Upload Complete"
Again, we want to make the script executable by issuing the following command.
sudo chmod +x /usr/local/bin/moonbeam-backup.shRun the script and if all things go well, and you’ve copied over any updates to the first script which might apply to this one, you should have a snapshot of the Moonbeam parachain in your Azure Blob object storage.
Once you’ve completed the backups you should test, at least once, to a recovery server. We will take you through the steps to do this in Part IV of the series (including how to prevent any possibility of equivocation).
Now that we are confident our scripts work the way we want them to work, let’s set up a cron job to run multiple times a week, or whatever frequency you want (remember — this will stop your service and get it out of sync for a number of hours). Be aware that we want to offset the time from one script to the next — we want to make sure the first one completes before the second one starts.
Always make sure you have the correct path to AzCopy and to your script. We’ve also added some logging to the cron file so that we can see what happened if anything breaks.
#Don't forget to add path to AzCopy
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin#Run moonbeam db script at select days
0 5 2,5,8,11,15,18,22,25,28 * * moonbeam-backup.sh > /var/log/moonbeam-backup.log 2>&1#Run moonbeam db script at select days
0 12 2,5,8,11,15,18,22,25,28 * * moonbeam-polkadot-backup.sh > /var/log/moonbeam-polkadot-backup.log 2>&1
Now you have a weekly backup of our parachain and relay chain database writing to Azure blob at low cost, and you know you’ll always have them available to restore in a pinch.
If you want to monitor your cron job to make sure it’s successful, or so that you are notified of problems, there are a number of inexpensive SaaS options, as well as free software which will help you to do this. We won’t make any recommendations here but encourage you to explore what best meets your needs.
This completes Part III of the series. You should be able to alter these scripts to suit your specific needs, whether it’s to use on a different chain or a different cloud provider or dedicated server. In Part IV we will also publish the above scripts along with our scripts for Moonriver and Moonbase Alpha to our GitHub account.
Next week we’ll publish the final part of the series where we will take you step by step through a recovery to a new server, and show you how you can be up and running in less than an hour with appropriately powered hardware and network connectivity.
What changes would you make to these scripts? We would love to hear about improvements in the comments, specifically around how to improve the security of them, and feedback is always welcome via comment or direct message. Thanks for reading.
