Have you burned through your Amazon Web Services(AWS) Free tier credits trying to train your new AWS DeepRacer model? Maybe you just have $300 dedicated towards Google Cloud Platform(GCP) services? (You could.) In any case, it’s a great opportunity to experience the Cloud, and get your feet wet with reinforcement learning!
This guide is designed for those who are new to GCP, in order to teach you how to train an AWS DeepRacer reinforcement learning model outside of the Amazon Web Services(AWS) Console. You’ll be able to watch your training from anywhere, without tying up local resources or racking up a huge bill!
This walkthough guide is designed to be followed in order, and some steps a designed as setup in anticipation in future steps. This is noted where applicable
(Special Thanks to jarrett jordaan for some important corrections and tips!)
Creating a Virtual Machine(VM) in GCP
The first step to use GCP for training an Amazon DeepRacer model “locally” is creating a virtual machine(VM) instance in the GCP console.
- Sign into your GCP Console and click Navigation Menu>Compute Engine>VM Instances (Clicking on “Compute Engine” has the same effect)
- Click the “+Create Instance” button in the top bar of the page
- Below are the attributes of the VM instance we’ll be creating. Begin by giving the VM an appropriate name. Choose a region and zone in which your quotas will allow the machine types we will create in the next step.
- Next we will configure the Machine configuration of our VM. These settings were automatically recommended for me by GCP based on my low usage patterns on a larger machine(More vCPUs and memory).
Note about speed: Any machines offered are capable of running the DeepRacer Github repo we will be cloning, but your “Real Time Factor” may be less than 1. This will increase training simulation time. Similarly, adding a GPU to your device is not necessary for the software to function, but my understanding is that it is utilized to shorten policy-updating iterations. All considerations that increase speed increase cost, so use the estimations provided and plan accordingly!
Boot Disk Options
- Next, we will examine our boot disk. Two changes need to be made for the intended usage.
- First, change the OS from “Debian GNU/Linux 9(stretch)” to “Debian GNU/Linux 10 (buster)”. Updating the OS version will allow the log analysis tools to run after training and evaluation.
- Second, choose a larger Boot disk Size.
Note: persistent disk prices are relatively low, so I recommend at least 200GB.
- Now, click Management, security, disks, networking, sole tenancy to expand those respective tabs. Click on the Networking tab.
- In the field “Network tags” enter
vnc-server. It should become a tag with an “x” to its right. We will setup the firewall rule affecting this tag in a later step. This is another preparatory step that allows screen access to the VM from a remote machine, such as your local computer or mobile device.
- Click back to the Management tag, scroll to the Availability policy, and examine the “Preemptibility” option.
Note: Choosing “On” means that your instance cannot persist for more than 24 hours(your data will persist between instances, but training and access will immediately stop!), and demand spikes will cause your computers resources to be “pre-empted” and shut down. Preemptible instances cost substantially less(estimate for yourself!), so plan accordingly.
Create the VM
Double-check all of your choices, and then click Create once you are satisfied. You will receive a notification when your VM is ready!
Don’t worry! It’s very normal to experience issues creating VMs with a new account, especially in the Free Tier. The good news is that the failure message should include a reason.
Just click on the notification to find out the reason. My experience is that most early rejections are related to quotas, so request the appropriate increases today for the VM you want to build tomorrow ;)
Once your VM specifications are in order, we’ll move on to creating the rule to allow a VNC(Virtual-Network Computing) connection.
Create a VNC Firewall Rule
Before we look inside of our newly-created VM instance, we will take one more step of setup in the GCP Console: creating the firewall rule to allow remote access.
- Click Navigation Menu>VPC Network>Firewall Rules
- At the top of the page, click +Create Firewall Rule
- Give your rule an appropriate name and description
- The “Targets” dropdown menu should default to “Specified target tags”. Type
vnc-serverinto “Target tags” (It should become a tag with an “x”)
- “Source filter” should default to “IP ranges.” Type
0.0.0.0/0into “Source IP ranges.”
- Under “Protocols and ports” select the “Specified protocols and ports” radio button and type
tcp:5901.Double-check your settings and click Create. The firewall rule is ready!
Enter the Command Line Interface(CLI)
Now we’ll finally connect to our VM instance through SSH(Secure Shell) to set up the ecosystem we’ll be using to train our model. Most of the installations are in no particular order, as their dependencies and additional required packages will be installed as we go.
Most of the remaining commands will be issued from the command line. Don’t worry if you aren’t familiar with the command line. You should be able to enter exactly these commands in this order and end up with a running simulation!
Start SSH Session
- Return to Navigation Menu>Compute Engine>VM Instances (Clicking on “Compute Engine” has the same effect)
- In the row containing the newly-created VM, click the word SSH under the column “Connect.” This will launch an SSH session in a new window.
The first package we’ll be installing is git version control system.
- To do that, enter this command, then hit enter:
sudo apt-get install git
Important Note: Any time I use the command apt-get, you may choose to add the “-y” flag to the command, in order to “assume Yes” (i.e.
sudo apt-get -y install git )
- Install the Python 3 package manager, pip(pip is a recursive acronym– “Pip Installs Packages”). Accept dependencies and additional packages, then upgrade to latest version
sudo apt-get install python3-pip
pip3 install --upgrade pip
- Before moving on, go ahead and update your package list by running
sudo apt-get update
Install AWS CLI
- Using your new
pip3command, download the AWS CLI:
sudo pip3 install awscli
If you don’t use
sudo in the previous command, then AWS CLI will not be install in your path, and the following steps would be required. Unless you specifically prefer not to install as a root user, you can skip to AWS Credentials:
- Next, we’ll need to add the installation to our
$PATH, in order to run the
awscommand in any directory. We’ll do this by editing the
.bashrcfile in the text editor nano. If you’re already familiar with the command line, and you have a preferred editor, feel free to use that instead! If you’re less confident in the CLI, then follow along!
- Scroll to the bottom of the file using the arrow keys(Or other nano shortcuts), and add the following after the rest of the file, replacing
*YOUR_USERNAME*with your actual username(e.g.
- If you’re using nano, use
Ctrl-xto close the file, type
yto save the file before close, and then
Enterto confirm file name. You’ll need to restart your SSH session for the changes to take effect.
- Reboot the system, wait about 30 seconds, and then click “Reconnect”
- Re-open the SSH session from the GCP Console, and verify that your new path directory has been successfully added:
In order to actually use the features of the AWS CLI, you’ll need your AWS Credentials. Leave this SSH session open while you retrieve your AWS Access Key ID and Secret Access Key from the AWS Console. If you already have both the ID and the Secret Key, you can skip the next section!
If you used Amazon’s Setup guide, then you should have created an IAM User. You’ll want to access that account in the AWS Console now.
- In the upper-right corner, click your Username>My Security Credentials
- If you don’t already have an ID/Secret Access Key pair, click Create New Access Key under the “Access keys” section of the page.
Note: This secret key will only be provided at this time.
- Return to your SSH session and configure the AWS CLI with your ID and secret key. After running the following command, enter each of them when prompted, as well as the Region you chose when creating your AWS DeepRacer model(“us-east-1”). Leave the last option blank to default to “json”:
AWS CLI is now ready for use in our environment!
Next, we will install Docker according to provided instructions. I’m not going in to the level of detail provided by the developer instructions, but I will provide the commands you’ll need to run to get Docker up and running. You’ll need to confirm a few of these installations for them to finish, so I recommend running them one at a time.
sudo apt update
sudo apt install apt-transport-https ca-certificates curl gnupg2 software-properties-common
curl -fsSL https://download.docker.com/linux/debian/gpg | sudo apt-key add -
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/debian $(lsb_release -cs) stable"
sudo apt update
sudo apt install docker-ce
sudo systemctl status docker
- Once you’ve made it through these commands, make sure to hit
Add Docker User
Docker is now installed, but you can’t run it with root user privileges unless you run the next series of commands.
Note: You may find that the group already exists when you run
sudo groupadd docker, and that’s not going to affect the rest of things!
sudo groupadd docker
sudo usermod -aG docker $USER
docker run hello-world
If all of these commands were successful, Docker should load the
hello-world container. Congratulations: non-root users can now run the
docker command! Let’s go ahead and setup the local Sagemaker network we’ll be using later using that new
docker network create sagemaker-local
Install Docker Compose
Next we will install Docker Compose. Finally a simple installation!
sudo pip3 install docker-compose
Nvidia Docker Runtime
Note About GPUs: As I am not employing a GPU, I will not cover the additional installation of the Nvidia Docker Runtime. If you are using a GPU, you’ll want to ensure that is running, as well, before moving on.
Install VNC Viewer
The simulation uses a VNCViewer window to show a 3D view of your model, so we will install it next.
- Download the VNCViewer installer, then upload the .deb file to your VM instance through the SSH window we’ve been using. The file will be uploaded to your home directory(which is where you should be located if you haven’t changed directories), so run the following command:
Note: your package name may differ. Please compare your file name!
sudo dpkg -i VNC-Viewer-6.19.715-Linux-x64.deb
You’ll be informed that some dependencies aren’t currently installed. Fix those dependencies!
sudo apt-get install libx11–6 libxext6
sudo apt-get — fix-broken install
sudo apt-get install gnome-terminal
sudo pip3 install urllib3 — upgrade
Log analysis depends on Jupyter notebooks to operate, so we will install it next:
sudo pip3 install jupyter
Jupyter notebooks run in a web browser, so I’d recommend downloading one besides the default included with your desktop. Firefox(formerly Iceweasel) or Chrome are the most popular choices, and I’ll be using Chrome:
sudo apt-get install wget
sudo apt install ./google-chrome-stable_current_amd64.deb
The notebooks also lean on quite a few Python libraries. You can get them all by running:
sudo pip3 install boto3 tensorflow sklearn pandas matplotlib shapely
sudo apt-get install virtualenv
If you’d like to use the GUI to input your DeepRacer parameters and hyperparameters, then you’ll need QT:
sudo apt-get install qt5-default
sudo apt-get build-dep qt5-default
Install the Virtual Desktop (Xfce)
Because we are accessing a virtual desktop, we’ll need to a desktop to access! My intention is to use this vm solely for DeepRacer modeling and analysis, so I value a slim operating system like Xfce. Both Xfce and Gnome installation instructions are covered in another article, but these instructions will be specific to Xfce.
- Install a VNC Server so that we can access the VM from a remote device. I’ll show you TightVNC Server, but Tiger VNC is another popular option. If you’d like to follow along, or simply have no preference of VNC server:
sudo apt-get install tightvncserver
- Next, we’ll install Xfce
sudo apt-get install xfce4 xfce4-goodies
sudo apt-get upgrade xfce4 xfce4-goodies
- Now, configure the VNC server. Enter a strong 8-character password when prompted, then verify the password. You will also be given the option to create a password for view-only usage:
Note: You need to start the VNC server every time you stop(or get pre-empted) and re-launch your instance. Now that you’ve set your password, you’ll just need to run the
vncserver command to start the server!
- Once your credentials are configured, we need to edit the
.vnc/xstartupfile to automatically launch our desktop environment. Open the file in nano(Or any text editor), and then add the command
startxfce4 &as the last line:
Note: Gnome would require
- Once you’ve made your addition, hit
Ctrl-xto exit nano,
yto confirm saving before exit, and
Enterto confirm the original filename
Download DeepRacer-For-Dummies Github Repo
At this point, we are going to go ahead and download the repository containing all of the DeepRacer-related files that we will be using for simulation, modeling, data analysis, etc.:
- Create github directory and move into it
mkdir github && cd github
- The next series of commands clone the ARCC repo onto our VM, then runs the
init.shscript in the appropriate directory
git clone https://github.com/ARCC-RACE/deepracer-for-dummies.git && cd deepracer-for-dummies && ./init.sh
We will take a look at the most important files once the environment is fully set up and running!
Access your VM desktop
Finally, You can access you VM instance. On most computers, you can simply use a web browser enter the IP address of your newly-created instance in the following format:
- Enter the password your created after creating your VNC Server, and you should see your desktop!
Note: Remember that you’ll need to run
vncserver each time that the VM instance gets pre-empted or restarts!
I like to choose the default configuration. Once you’ve gotten comfortable with the desktop GUI, get ready to dive right back into the terminal– these machines charge by the second!
Set root Password
First, go ahead and set a root user password. The GUI will need this password to run some commands, most notably the memory manager:
sudo passwd root
If you’d like to set all of your values in the same place, and minimize your time in the command line, then setup the DeepRacer for Dummies desktop GUI:
GUI/CLI Commands for DeepRacer
This GUI allows you to run all of the terminal commands with just the click of button. The ARCC offers this GUI setup advice, most of which will be covered here. The reward function, action space, track, and hyperparameters are represented by text fields in the GUI. Alternatively, you may alter the files directly in the command line!
- Your hyperparameters:
Note: This is also where you would edit the code to run a pre-trained model!
- The track:
Note: You also need to change `GPU_AVAILABLE` to `False`, unless you’ve enabled a GPU
- Track choices are contained in the repo, in case you forget. Just take the file name and leave off the
I’ll also list them, for convenience:
- Both the Action Space and Reward Function are kept in the same folder as the files “model_metadata.json” and “reward.py,” respectively:
Once you’ve edited your models, parameters and hyperparameters(either in the CLI or the GUI), it’s time to start training!
If you’re using the GUI, click the Save button before starting training. The next scripts are all represented by a button with the same name(i.e. Start button starts the `start.sh` script). If you’re using CLI, open the terminal from your new desktop, and switch into the training script directory:
Now that our training environment is set, you can run the training script. I’m going to describe the rest of the GUI before I move on, but you can go ahead and Start training while you take a look at this relatively simple GUI:
Note: Multiple windows will open, don’t be alarmed!
- When you are finished training your model, this is also where the script to shut down(Stop) the processes created by the start script:
- If you are satisfied with a run, and would like to use it as a starting point, you can use the Pretrained button in the GUI, or from this directory use the script:
Note: if you are using the GUI, this button is a toggle between using the pre-trained model or not!
- To Upload a specific snapshot to S3 in AWS:
Note: If no checkpoint is provided, it attempts to retrieve the latest snapshot
- If you would like to completely Delete the last run, including log files:
The GUI has four additional buttons, as well: Init, Save, Restart, and Refresh.
- Init — This initializes the DeepRacer repo downloaded from Github according to a script taking these steps:
1. Clone https://github.com/crr0004/deepracer.git
mkdir -p ~/.sagemaker && cp config.yaml ~/.sagemaker3. Sets the image name in
crr0004/sagemaker-rl-tensorflow:nvidia4. Sets the instance_type in
local_gpu5. Copies the
model-metadatafiles into your Minio bucket
If you’re encountering an error that you just can’t place, it may be worth it to re-initialize. To run
init.sh from the command line, you can
- Save — From ARCC:
Saves the settings in the GUI to the files that are used for training. Make sure you do this before starting or else the training session will not actually be using the hyperparameters/reward function/actions space/ track that you see in the console
- Restart — Again, from ARCC:
This is cool! What this button does is allow you to tweak some values in the GUI, save them, and in the click of a button: stop training, save the model to the pre-trained directory so it can be the base for the new model that it is about to train, set the rl_coach (sagemaker) to use a pre-trained model, and then restart training with the model that was originally training but with a new values that you set for it. This allows you to quickly iterate through hyperparameters, reward functions, and even action spaces (although be careful to not change the number of possible actions)
- Refresh — This ensures that the GUI is in sync with your file system, and will overwrite any unsaved changes you have!
Those are the basic controls for your DeepRacer!
Start Training your Model
Because of our diligent prep work, we can run the simulation programs without issue.
- Hit the “Start” button in the GUI, or execute the
./start.shcommand from the command line, and things will spring to life! If you’re using the GUI, you’ll be asked whether or not you’d like to use the memory manager, which will cap the amount of data saved at 3GB. If so, you’ll need to enter your newly-created root password.
Two new terminal windows will open, one with realtime logging, another cryptically stating
vncviewer, and your original terminal window, which now contains a link with workbooks to analyze your logs.
- If you’d ever like to view these workbooks on your own, run
Either way, you’ll want to take a look at
DeepRacer Log Analysis.ipynb,
Training_analysis.ipynb. These files will be useful once your model has completed some training, and created some data for you to analyze.
Another popup should have loaded from VNCViewer, as well. Accept this warning to get to the moment you’ve been waiting for: your model trying the simulation!
Note: If a 3D View of the track you chose isn’t loading, then there is likely a syntax error somewhere in your settings, so double-check them(Maybe validate your reward function again?), and then try again. If it still isn’t working, then run the
init script one more time. As a last resort, you can delete and re-initialize the
deepracer-for-dummies repo, but then you’ll need to re-initialize your parameters, hyperparameters, and track selection.
If your car was making progress for a few runs, when suddenly it starts mindlessly driving straight into oblivion, or even stops moving entirely: don’t panic! If you look at the terminal window containing your logs, you’ll likely see that the model is in a policy-updating iteration!
This is very CPU-intense time, and may slow your instance slightly, but the code will iterate the number of “epochs” you set, and then begin new episodes!
My intention in writing this was to make it simple for someone with moderate technical ability to get started training DeepRacer models on Google Cloud Platform. I couldn’t cover everything, but I think these are the tools you need to focus on improving your DeepRacer!
If you see any problems, please let me know! Hopefully, my trial and error has created an easy enough process for you that the result seems inevitable– trust me, it wasn’t. :)