How to use Google Cloud Platform to train an AWS DeepRacer AI model

A GCP Compute Instance simulating AWS DeepRacer 3D model training on the Shanghai Sudu track.

Have you burned through your Amazon Web Services(AWS) Free tier credits trying to train your new AWS DeepRacer model? Maybe you just have $300 dedicated towards Google Cloud Platform(GCP) services? (You could.) In any case, it’s a great opportunity to experience the Cloud, and get your feet wet with reinforcement learning!

This guide is designed for those who are new to GCP, in order to teach you how to train an AWS DeepRacer reinforcement learning model outside of the Amazon Web Services(AWS) Console. You’ll be able to watch your training from anywhere, without tying up local resources or racking up a huge bill!

This walkthough guide is designed to be followed in order, and some steps a designed as setup in anticipation in future steps. This is noted where applicable

(Special Thanks to jarrett jordaan for some important corrections and tips!)

Creating a Virtual Machine(VM) in GCP

The first step to use GCP for training an Amazon DeepRacer model “locally” is creating a virtual machine(VM) instance in the GCP console.

• Sign into your GCP Console and click Navigation Menu>Compute Engine>VM Instances (Clicking on “Compute Engine” has the same effect)

• Click the “+Create Instance” button in the top bar of the page

VM Attributes

• Below are the attributes of the VM instance we’ll be creating. Begin by giving the VM an appropriate name. Choose a region and zone in which your quotas will allow the machine types we will create in the next step.

Creating an custom VM instance with 14 vCPU cores and 68.75 GB of memory

• Next we will configure the Machine configuration of our VM. These settings were automatically recommended for me by GCP based on my low usage patterns on a larger machine(More vCPUs and memory).

Note about speed: Any machines offered are capable of running the DeepRacer Github repo we will be cloning, but your “Real Time Factor” may be less than 1. This will increase training simulation time. Similarly, adding a GPU to your device is not necessary for the software to function, but my understanding is that it is utilized to shorten policy-updating iterations. All considerations that increase speed increase cost, so use the estimations provided and plan accordingly!

Boot Disk Options

• Next, we will examine our boot disk. Two changes need to be made for the intended usage.

• First, change the OS from “Debian GNU/Linux 9(stretch)” to “Debian GNU/Linux 10 (buster)”. Updating the OS version will allow the log analysis tools to run after training and evaluation.

• Second, choose a larger Boot disk Size.
  Note: persistent disk prices are relatively low, so I recommend at least 200GB.

Network Tag

• Now, click Management, security, disks, networking, sole tenancy to expand those respective tabs. Click on the Networking tab.

• In the field “Network tags” enter vnc-server. It should become a tag with an “x” to its right. We will setup the firewall rule affecting this tag in a later step. This is another preparatory step that allows screen access to the VM from a remote machine, such as your local computer or mobile device.

Preemptibility

• Click back to the Management tag, scroll to the Availability policy, and examine the “Preemptibility” option.

Note: Choosing “On” means that your instance cannot persist for more than 24 hours(your data will persist between instances, but training and access will immediately stop!), and demand spikes will cause your computers resources to be “pre-empted” and shut down. Preemptible instances cost substantially less(estimate for yourself!), so plan accordingly.

Create the VM

All set? Click Create!

Double-check all of your choices, and then click Create once you are satisfied. You will receive a notification when your VM is ready!

…or if it fails.

Don’t worry! It’s very normal to experience issues creating VMs with a new account, especially in the Free Tier. The good news is that the failure message should include a reason.

Just click Request Increase to start the process!

Just click on the notification to find out the reason. My experience is that most early rejections are related to quotas, so request the appropriate increases today for the VM you want to build tomorrow ;)

Once your VM specifications are in order, we’ll move on to creating the rule to allow a VNC(Virtual-Network Computing) connection.

Create a VNC Firewall Rule

Before we look inside of our newly-created VM instance, we will take one more step of setup in the GCP Console: creating the firewall rule to allow remote access.

• Click Navigation Menu>VPC Network>Firewall Rules

• At the top of the page, click +Create Firewall Rule

• Give your rule an appropriate name and description

• The “Targets” dropdown menu should default to “Specified target tags”. Type vnc-server into “Target tags” (It should become a tag with an “x”)

• “Source filter” should default to “IP ranges.” Type 0.0.0.0/0 into “Source IP ranges.”

• Under “Protocols and ports” select the “Specified protocols and ports” radio button and type tcp:5901. Double-check your settings and click Create. The firewall rule is ready!

Install Prerequisites

Enter the Command Line Interface(CLI)

Now we’ll finally connect to our VM instance through SSH(Secure Shell) to set up the ecosystem we’ll be using to train our model. Most of the installations are in no particular order, as their dependencies and additional required packages will be installed as we go.

This guide generally follows the steps laid out by the ARCC.

Most of the remaining commands will be issued from the command line. Don’t worry if you aren’t familiar with the command line. You should be able to enter exactly these commands in this order and end up with a running simulation!

Start SSH Session

• Return to Navigation Menu>Compute Engine>VM Instances (Clicking on “Compute Engine” has the same effect)

• In the row containing the newly-created VM, click the word SSH under the column “Connect.” This will launch an SSH session in a new window.

Install Git

The first package we’ll be installing is git version control system.

• To do that, enter this command, then hit enter:

sudo apt-get install git

Important Note: Any time I use the command apt-get, you may choose to add the “-y” flag to the command, in order to “assume Yes” (i.e. sudo apt-get -y install git )

You should have a similar output

Install Pip3

• Install the Python 3 package manager, pip(pip is a recursive acronym– “Pip Installs Packages”). Accept dependencies and additional packages, then upgrade to latest version

sudo apt-get install python3-pip
pip3 install --upgrade pip

• Before moving on, go ahead and update your package list by running

sudo apt-get update

Install AWS CLI

• Using your new pip3 command, download the AWS CLI:

sudo pip3 install awscli

If you don’t use sudo in the previous command, then AWS CLI will not be install in your path, and the following steps would be required. Unless you specifically prefer not to install as a root user, you can skip to AWS Credentials:

• Next, we’ll need to add the installation to our $PATH, in order to run the aws command in any directory. We’ll do this by editing the .bashrc file in the text editor nano. If you’re already familiar with the command line, and you have a preferred editor, feel free to use that instead! If you’re less confident in the CLI, then follow along!

nano ~/.bashrc

End of the `.bashrc` file as viewed in nano

• Scroll to the bottom of the file using the arrow keys(Or other nano shortcuts), and add the following after the rest of the file, replacing *YOUR_USERNAME*with your actual username(e.g. paul_m_reese):

export PATH=$PATH:/home/*YOUR_USERNAME*/.local/bin

• If you’re using nano, use Ctrl-x to close the file, type yto save the file before close, and then Enter to confirm file name. You’ll need to restart your SSH session for the changes to take effect.

sudo reboot

• Reboot the system, wait about 30 seconds, and then click “Reconnect”

• Re-open the SSH session from the GCP Console, and verify that your new path directory has been successfully added:

aws --version

In order to actually use the features of the AWS CLI, you’ll need your AWS Credentials. Leave this SSH session open while you retrieve your AWS Access Key ID and Secret Access Key from the AWS Console. If you already have both the ID and the Secret Key, you can skip the next section!

AWS Credentials

If you used Amazon’s Setup guide, then you should have created an IAM User. You’ll want to access that account in the AWS Console now.

• In the upper-right corner, click your Username>My Security Credentials

• If you don’t already have an ID/Secret Access Key pair, click Create New Access Key under the “Access keys” section of the page.

Note: This secret key will only be provided at this time.

• Return to your SSH session and configure the AWS CLI with your ID and secret key. After running the following command, enter each of them when prompted, as well as the Region you chose when creating your AWS DeepRacer model(“us-east-1”). Leave the last option blank to default to “json”:

aws configure

AWS CLI is now ready for use in our environment!

Install Docker

Next, we will install Docker according to provided instructions. I’m not going in to the level of detail provided by the developer instructions, but I will provide the commands you’ll need to run to get Docker up and running. You’ll need to confirm a few of these installations for them to finish, so I recommend running them one at a time.

sudo apt update
sudo apt install apt-transport-https ca-certificates curl gnupg2 software-properties-common
curl -fsSL https://download.docker.com/linux/debian/gpg | sudo apt-key add -
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/debian $(lsb_release -cs) stable"
sudo apt update
sudo apt install docker-ce
sudo systemctl status docker

• Once you’ve made it through these commands, make sure to hit Ctrl-c to exitsystemctl

Add Docker User

Docker is now installed, but you can’t run it with root user privileges unless you run the next series of commands.

Note: You may find that the group already exists when you run sudo groupadd docker, and that’s not going to affect the rest of things!

sudo groupadd docker
sudo usermod -aG docker $USER
newgrp docker
docker run hello-world

Docker running the “hello-world” container!

If all of these commands were successful, Docker should load the hello-world container. Congratulations: non-root users can now run the docker command! Let’s go ahead and setup the local Sagemaker network we’ll be using later using that new docker command:

docker network create sagemaker-local

Install Docker Compose

Next we will install Docker Compose. Finally a simple installation!

sudo pip3 install docker-compose

Nvidia Docker Runtime

Note About GPUs: As I am not employing a GPU, I will not cover the additional installation of the Nvidia Docker Runtime. If you are using a GPU, you’ll want to ensure that is running, as well, before moving on.

Install VNC Viewer

The simulation uses a VNCViewer window to show a 3D view of your model, so we will install it next.

• Download the VNCViewer installer, then upload the .deb file to your VM instance through the SSH window we’ve been using. The file will be uploaded to your home directory(which is where you should be located if you haven’t changed directories), so run the following command:
  Note: your package name may differ. Please compare your file name!

sudo dpkg -i VNC-Viewer-6.19.715-Linux-x64.deb

You’ll be informed that some dependencies aren’t currently installed. Fix those dependencies!

sudo apt-get install libx11–6 libxext6
sudo apt-get — fix-broken install
sudo apt-get install gnome-terminal
sudo pip3 install urllib3 — upgrade

Fix the broken installation!

Install Jupyter

Log analysis depends on Jupyter notebooks to operate, so we will install it next:

sudo pip3 install jupyter

Jupyter notebooks run in a web browser, so I’d recommend downloading one besides the default included with your desktop. Firefox(formerly Iceweasel) or Chrome are the most popular choices, and I’ll be using Chrome:

sudo apt-get install wget
wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
sudo apt install ./google-chrome-stable_current_amd64.deb

The notebooks also lean on quite a few Python libraries. You can get them all by running:

sudo pip3 install boto3 tensorflow sklearn pandas matplotlib shapely
sudo apt-get install virtualenv

GUI Prerequisites

If you’d like to use the GUI to input your DeepRacer parameters and hyperparameters, then you’ll need QT:

sudo apt-get install qt5-default
sudo apt-get build-dep qt5-default

Install the Virtual Desktop (Xfce)

Because we are accessing a virtual desktop, we’ll need to a desktop to access! My intention is to use this vm solely for DeepRacer modeling and analysis, so I value a slim operating system like Xfce. Both Xfce and Gnome installation instructions are covered in another article, but these instructions will be specific to Xfce.

• Install a VNC Server so that we can access the VM from a remote device. I’ll show you TightVNC Server, but Tiger VNC is another popular option. If you’d like to follow along, or simply have no preference of VNC server:

sudo apt-get install tightvncserver

• Next, we’ll install Xfce

sudo apt-get install xfce4 xfce4-goodies
sudo apt-get upgrade xfce4 xfce4-goodies

You get 8 characters, make it strong! …but memorable.

• Now, configure the VNC server. Enter a strong 8-character password when prompted, then verify the password. You will also be given the option to create a password for view-only usage:

 vncserver

Note: You need to start the VNC server every time you stop(or get pre-empted) and re-launch your instance. Now that you’ve set your password, you’ll just need to run the vncserver command to start the server!

• Once your credentials are configured, we need to edit the .vnc/xstartup file to automatically launch our desktop environment. Open the file in nano(Or any text editor), and then add the command startxfce4 & as the last line:
  Note: Gnome would require gnome-session & in xstartup instead

nano .vnc/xstartup

Take note of the bottom line!

• Once you’ve made your addition, hit Ctrl-x to exit nano, y to confirm saving before exit, and Enter to confirm the original filename

Download DeepRacer-For-Dummies Github Repo

At this point, we are going to go ahead and download the repository containing all of the DeepRacer-related files that we will be using for simulation, modeling, data analysis, etc.:

• Create github directory and move into it

mkdir github && cd github

• The next series of commands clone the ARCC repo onto our VM, then runs the init.sh script in the appropriate directory

git clone https://github.com/ARCC-RACE/deepracer-for-dummies.git && cd deepracer-for-dummies && ./init.sh

We will take a look at the most important files once the environment is fully set up and running!

Access your VM desktop

Finally, You can access you VM instance. On most computers, you can simply use a web browser enter the IP address of your newly-created instance in the following format:

vnc://IPADDRESS:5901

As seen from my local machine!

• Enter the password your created after creating your VNC Server, and you should see your desktop!

Note: Remember that you’ll need to run vncserver each time that the VM instance gets pre-empted or restarts!

Signs of life!

I like to choose the default configuration. Once you’ve gotten comfortable with the desktop GUI, get ready to dive right back into the terminal– these machines charge by the second!

Set root Password

First, go ahead and set a root user password. The GUI will need this password to run some commands, most notably the memory manager:

sudo passwd root

If you’d like to set all of your values in the same place, and minimize your time in the command line, then setup the DeepRacer for Dummies desktop GUI:

cd /github/deepracer-for-dummies/build-gui-Desktop-Release
./gui

GUI/CLI Commands for DeepRacer

This is your control hub!

This GUI allows you to run all of the terminal commands with just the click of button. The ARCC offers this GUI setup advice, most of which will be covered here. The reward function, action space, track, and hyperparameters are represented by text fields in the GUI. Alternatively, you may alter the files directly in the command line!

• Your hyperparameters:
  Note: This is also where you would edit the code to run a pre-trained model!

~/github/deepracer-for-dummies/rl_deepracer_coach_robomaker.py

• The track:
  Note: You also need to change `GPU_AVAILABLE` to `False`, unless you’ve enabled a GPU

~/github/deepracer-for-dummies/docker/.env

• Track choices are contained in the repo, in case you forget. Just take the file name and leave off the .world suffix:

~/github/deepracer-for-dummies/deepracer/simulation/aws-robomaker-sample-application-deepracer/simulation_ws/src/deepracer_simulation/worlds

I’ll also list them, for convenience:

AWS_track
Bowtie_track
China_track
New_York_Track
Oval_track
Straight_track
Tokyo_Training_track
Tokyo_Training_track_original
Virtual_May19_Train_track
easy_track
hard_loopy_track
hard_speed_track
hard_track
medium_track
reinvent_base
reinvent_base_jeremiah
reinvent_carpet
reinvent_concrete
reinvent_wood

• Both the Action Space and Reward Function are kept in the same folder as the files “model_metadata.json” and “reward.py,” respectively:

~/github/deepracer-for-dummies/docker/volumes/minio/bucket/custom_files

Once you’ve edited your models, parameters and hyperparameters(either in the CLI or the GUI), it’s time to start training!

If you’re using the GUI, click the Save button before starting training. The next scripts are all represented by a button with the same name(i.e. Start button starts the `start.sh` script). If you’re using CLI, open the terminal from your new desktop, and switch into the training script directory:

cd ~/github/deepracer-for-dummies/scripts/training

Now that our training environment is set, you can run the training script. I’m going to describe the rest of the GUI before I move on, but you can go ahead and Start training while you take a look at this relatively simple GUI:
Note: Multiple windows will open, don’t be alarmed!

./start.sh

• When you are finished training your model, this is also where the script to shut down(Stop) the processes created by the start script:

./stop.sh

• If you are satisfied with a run, and would like to use it as a starting point, you can use the Pretrained button in the GUI, or from this directory use the script:
  Note: if you are using the GUI, this button is a toggle between using the pre-trained model or not!

./set-last-run-to-pretrained.sh

• To Upload a specific snapshot to S3 in AWS:
  Note: If no checkpoint is provided, it attempts to retrieve the latest snapshot

./upload-snapshot.sh`

• If you would like to completely Delete the last run, including log files:

sudo ./delete-last-run.sh

The GUI has four additional buttons, as well: Init, Save, Restart, and Refresh.

• Init — This initializes the DeepRacer repo downloaded from Github according to a script taking these steps:
  1. Clone https://github.com/crr0004/deepracer.git
  2. mkdir -p ~/.sagemaker && cp config.yaml ~/.sagemaker
3. Sets the image name in rl_deepracer_coach_robomaker.py to crr0004/sagemaker-rl-tensorflow:nvidia
4. Sets the instance_type in rl_deepracer_coach_robomaker.py to local_gpu
5. Copies the reward.py and model-metadata files into your Minio bucket

If you’re encountering an error that you just can’t place, it may be worth it to re-initialize. To run init.sh from the command line, you can

cd ~/github/deepracer-for-dummies/
./init.sh

• Save — From ARCC:

Saves the settings in the GUI to the files that are used for training. Make sure you do this before starting or else the training session will not actually be using the hyperparameters/reward function/actions space/ track that you see in the console

• Restart — Again, from ARCC:

This is cool! What this button does is allow you to tweak some values in the GUI, save them, and in the click of a button: stop training, save the model to the pre-trained directory so it can be the base for the new model that it is about to train, set the rl_coach (sagemaker) to use a pre-trained model, and then restart training with the model that was originally training but with a new values that you set for it. This allows you to quickly iterate through hyperparameters, reward functions, and even action spaces (although be careful to not change the number of possible actions)

• Refresh — This ensures that the GUI is in sync with your file system, and will overwrite any unsaved changes you have!

Those are the basic controls for your DeepRacer!

Start Training your Model

Because of our diligent prep work, we can run the simulation programs without issue.

• Hit the “Start” button in the GUI, or execute the ./start.sh command from the command line, and things will spring to life! If you’re using the GUI, you’ll be asked whether or not you’d like to use the memory manager, which will cap the amount of data saved at 3GB. If so, you’ll need to enter your newly-created root password.

Two new terminal windows will open, one with realtime logging, another cryptically stating vncviewer, and your original terminal window, which now contains a link with workbooks to analyze your logs.

• If you’d ever like to view these workbooks on your own, run

cd ~/github/deepracer-for-dummies/scripts/log-analysis/
./start.sh

Either way, you’ll want to take a look at DeepRacer Log Analysis.ipynb, Evaluation_analysis.ipynb, and Training_analysis.ipynb. These files will be useful once your model has completed some training, and created some data for you to analyze.

Another popup should have loaded from VNCViewer, as well. Accept this warning to get to the moment you’ve been waiting for: your model trying the simulation!
Note: If a 3D View of the track you chose isn’t loading, then there is likely a syntax error somewhere in your settings, so double-check them(Maybe validate your reward function again?), and then try again. If it still isn’t working, then run the init script one more time. As a last resort, you can delete and re-initialize the deepracer-for-dummies repo, but then you’ll need to re-initialize your parameters, hyperparameters, and track selection.

If your car was making progress for a few runs, when suddenly it starts mindlessly driving straight into oblivion, or even stops moving entirely: don’t panic! If you look at the terminal window containing your logs, you’ll likely see that the model is in a policy-updating iteration!

This is very CPU-intense time, and may slow your instance slightly, but the code will iterate the number of “epochs” you set, and then begin new episodes!

That’s it!

My intention in writing this was to make it simple for someone with moderate technical ability to get started training DeepRacer models on Google Cloud Platform. I couldn’t cover everything, but I think these are the tools you need to focus on improving your DeepRacer!

If you see any problems, please let me know! Hopefully, my trial and error has created an easy enough process for you that the result seems inevitable– trust me, it wasn’t. :)