How our company’s AWS DeepRacer Team finished *SPOILER* in the World Finals at AWS re:Invent
After fighting through a tough round of 64 internal teams to win our company league, we stood on the shoulders of giants to duke it out with the best in world.
If you’re interested in reading how we got this far and how our company wildcard worked, check out my first blog post here: https://medium.com/libertyit/how-we-won-our-company-deepracer-league-and-qualified-for-re-invent-30ec9be7c8fa.
If you are unfamiliar with DeepRacer and would like to know more, this link is a great starting point: https://aws.amazon.com/deepracer/
Las Vegas 2019 — it’s time for the jewel in the crown of AWS conferences: re:Invent. With around 65,000 attendees and over 3000 sessions, this year marked the biggest re:Invent conference to date. While we filled our boots with sessions, workshops, parties and swag (so much swag) we were there with one goal in mind: take home the coveted DeepRacer world championship cup. In our way stood the winners of every AWS Summit DeepRacer competition in 2018 as well as the formidable top 18 finishers in the virtual league. It’s also worth mentioning the fact that we’d be competing on a whole new track that nobody has had the chance to race on physically up until this tournament.
As a bit of background we are team Outrun. We won the Liberty Mutual company league (a gruelling 64 team tournament that took place over 4 months) and we were awarded the once in a lifetime prize of 5 golden tickets to re:Invent and a wildcard place in the grand finals. Unlike the company league, we could only have one racer for our wildcard place, this was luckily an easy choice — our star driver and trainer of over a thousand models (probably) Darren. Fortunately for us, re:Invent was also host to the final qualifying race of the year where the top three would get a place in the Championship.
The last qualifier
The first issue we had was that we didn’t know what track this qualifier would be raced on, would it be the new 2019 championship track, for which we have a few models trained, but untested on a physical track, or would it be the 2018 track used in all the summits up until now, for which we have a model that can set a blistering 6.9 seconds? We also didn’t know how many tracks would be available, and how many tries we would get at setting our time. As the hour drew near in the Aria hotel, we did what we do best at re:Invent: we queued. We joined a massive queue for around 90 mins to get into the room hosting the finals, still not knowing what the set up would be. When the doors opened we waited our turn then rushed in to see one track set up in the corner, one glorious track, one 2018 track!
So we joined the queue to race which was quickly closed behind us. With only one track and 2 and a half hours they were only letting ~20 people race. Disappointed is not the word for some of the people who were unable to join the queue, but at least we were there. We also spotted the runners up from our company league (team Slow Down for What) who were also racing for a place in the championship. While we were waiting in the queue, Blaine, the face of DeepRacer TV, began interviewing people in the queue who had featured in previous episodes. The excellent commentators did a great job of building hype in the room and as a big fan of DeepRacer TV, I for one was delighted just to be a small part of it. As we didn’t know which track to train for we had a selection of models, but perhaps less coverage than we would have liked for this particular track (most of our models were clones from the same family of model), but we weren’t too phased. Inevitably the talk around the queue was around what kind of models people were running with people bragging about sub 9 second models and sub 8 second models, but no other person in the queue had seen what we had: a sub 7 second model. In addition to this, the people who were racing before us weren’t putting amazing times on the board — none of them were under 10 seconds. We were feeling confident! Until…
We only beat our only known competitors (team Slow Down For What) very narrowly in our company league — they had set a sub 8 second time and only had time to improve since then so we were very keen to see what time they’d set. But instead of knocking our socks off, the car struggled to get around the track. It kept going off at the hairpin, it kept being drawn towards to AWS logo printed on the track, when it did finish it was going slow. It was a shadow of the model that pushed us right to the edge just a month ago. A quick look at the leaderboard with no good times suddenly had us thinking that maybe there was just something up with the AWS set up — were the cars not calibrated correctly? Was the lighting off on the track? Or maybe something we’d not encountered yet? What if our models performed the same way? Well that’s exactly what happened. One out of four of us managed to nearly finish a lap with what would have been a winning time, but the car came off the track right at the last. The rest of us barely finished a lap. It was disappointing considering that on another day this model can drive smoothly around a track lap after lap consistently doing sub 9 seconds without any input from the driver. But it was not meant to be. We left the track while the winners were photographed with their trophies and turned our focus to the big Championship starting the next day.
Championship Day 1: Open play and the group stages
The first day of the championship had the 64 finalists split into four groups with the last group being made up entirely of online qualifiers. Darren (representing team OutRun and Liberty Mutual) was in group C. Each racer had four attempts, each lasting four minutes to set their fastest time and the fastest four times from each group would qualify to the final 16 for the head to head stages the next day.
The racing wasn’t scheduled to start until 1pm, but fortunately the venue was open from 9am for “open play”. At 9am the doors opened with some of the finalists running to make sure they were first to get practice time in, however there was no need as there were five tracks set up. The venue, the MGM Garden Arena was amazing. The 16,800 capacity arena has been host to professional wrestling matches, championship boxing matches and just a couple of short weeks post re:Invent, the Ariana Grande world tour, but today it belongs to the DeepRacers. As we walked down the stairs we got our first glimpse of the new championship track in the flesh and it was indeed massive. I don’t envy the pit crew who have to chase the car around the track all day! There was a cage set up in the middle of the room where the engineers could work on the cars themselves and at the back of the arena the commentators desk was set up, flanked by two huge screens streaming all of the live action. It was awesome.
It was time to get down to business. Having not raced on this track physically, we got to work testing as many models as possible between us. We each took a USB stick and joined a queue to try and find out which models were going to perform the best in the real world. This is where having a team really paid off. It took around 30 minutes of queuing before getting a 3 minute slot on the track. This way each of us could queue, try out 2–3 models, then join the queue again. its important to note that there were leaderboards set up for local play. This meant that we could really get a feel for what sort of times were being set by everyone else racing with a particular interest being given to the times being set by the other finalists. At first we set an ~12 second time, but didn’t know if that would be good or not for this new track, probably not since it put us in fourth at an early stage. Next time round we set some ~11 second times which barely put us in the top 15 times on the board. With open play running out fast, Darren took to the track for one last shot at setting a good time and absolutely smashed our previous fastest time rocketing us up to 2nd place on the leaderboard with a very impressive 9.886 seconds. No prizes for practice at this stage, but 2nd place is an impressive accolade going into the first round of the finals, we were elated and feeling confident. Bring on the group stages!
Things were running way behind schedule, around three hours behind schedule to be exact. This was an issue for the fatigue laden Outrun who were still very much suffering the effects of jet lag and 24 hours of travel just a few days ago. Regardless we had our game faces on as group C was called to the floor and Darren took to the track for his first four minute shot. Groups A and B were all but finished by this stage and our previous time of 9.886 set in the practice play would be enough to have us qualify comfortably. Inevitably this was not the case with group C. In the first of our four attempts Darren set a time far from the top four, the car didn’t perform and everyone else seemed to be doing much better, by the second attempt we had a set a time of ~10.1 seconds which put us in third. This was good, but the way the sub 9 second times were going up on the board in both group C and D it was looking like we’d need to reach a bit deeper to get qualified.
We looked with envy at groups A and B knowing 10.1 would have landed us a place in the next stage, but all we could do was keep our heads in the game and keep trying to improve in attempt three. This attempt was a disaster, no matter how hard he pushed it Darren couldn’t improve on the previous time. The car just wouldn’t stay on the track at any speed. Worse still, our 10.1 second time was now in fourth place, right on the bubble for elimination and we only had one last shot to improve. Darren took to the track, the car instantly looked better and it was just a case of getting up some speed and setting one last good time for the day. Our strategy from the practice day was to let the car find its line, do a couple of laps then crank up the throttle for the last couple of corners. The car went around once, twice and on the third time it came off right at the last corner to groans from the crowd. We were consistently setting low 10 second times but it wasn’t enough. Again the car started doing its consistent laps and then just as it swung round the now infamous horse shoe, Darren began frantically tapping the throttle. The DeepRacer glided round the last corner at top speed and crossed the finish line setting a blazing time of 9.331 topping our group and landing him a spot in the commentators chair to run them through what just happened. In typical Outrun fashion we smashed it when we needed it most and were through to the Head to Heads.
Championship Day 2: The quick and the dead
Day two was host to the nefarious and unforgiving head to head stage of the competition. In our opinion this was a well designed stage of the tournament, with provisions to insure that no racer had an advantage from using a “better” car, or could set a time on a faster track. To digress slightly, AWS were trialing DeepRacer analytics for the tournament and for the first time we had data showing which of the racers were most reliable, who had completed the most laps, who had set the best average lap times, which track was fastest and even who was statistically more likely to win in different head to head match ups. For a statistics geek like myself it was all very impressive and it did show that one of the four tracks available was performing faster than the others. To negate such variations, each head to head matchup would take place such that each racer was on the same track, and would get one attempt with each of the two cars available.
We spent the morning racing more models in open play to try out a few last minute clones trained by Darren the night before. We were confidently setting times between 9–10 seconds and a couple of the new models made the final cut onto the USB and into the golden notebook filled with model names, times, throttle percentages etc. Fail to prepare, prepare to fail! Throughout the day we also had supporters drop in from Liberty IT and Liberty Mutual to cheer us on, their support was invaluable.
Our first race went spectacularly well. We stuck to our plan of getting laps on the board before blitzing the last of the track and set a record time of 8.9 seconds, the fastest that had been set on this new track to date. It was beginning to feel like we might just be able to win this thing. We won the match up and progressed to the next stage. Our record stood for a full 50 minutes before an 8.8 second time was set and as the day went on that time got smaller and smaller until it was a tiny 8.02 seconds set by the current world record holder of the previous year’s track — Sola, the Usain Bolt of DeepRacer.
We didn’t know if we could dig deep enough and knock another second off our time, but all we needed to do for now was get to the final three people, so we raced some more. Two more races stood between us and a place in the grand final at the Thursday morning keynote. The next one got off to a rocky start, but in our second run with a better car we set a personal best time of 8.8 seconds and got through to our final race against our toughest competitor yet who had set a best time of 8.4 seconds. At the start it was clear that our first car was the lesser of the two available for this track. Our adversary set a great time of 9.1 seconds, not unbeatable, but it’d be tough considering our best on this run so far was over 10 seconds. We’d have to wait until we got our turn on the better car to have a chance of beating it. As soon as our model was loaded into the next car things immediately looked better as the car flew around it’s first couple of laps according to plan. Then disaster struck, the car came off the track and one of the wheels became dislodged. Our pit crew decided that because it wasn’t out fault the clock would get stopped while the car was replaced. While this was good news it meant that we were back using the car that was struggling with our model and we inevitably lost this head to head match up.
We weren’t out yet. Similar to having a 3rd place playoffs in popular sporting events, the AWS championship had a losers bracket: two more head to heads and one last chance to qualify for the big final. The bad news, we were exhausted. Darren had been in the room since 8am and it was now approaching 6pm. Just standing all day on top of the emotional stress and the adrenaline of racing all day had taken it’s toll on everybody. The first race was against somebody we had already beaten in a previous matchup with our 8.8 second time. We took to the stand and raced for our DeepRacer lives! But it was not meant to be… the first car didn’t behave for us and the second car, while looking better, began to have issues after a minor crash into the wall. Our fastest time was 11.8 seconds and we dropped out of the last chance bracket and out of the tournament in very respectable 4th place.
Championship Day 3: The final
The championship took place before the closing keynote of the conference on Thursday morning, the finalists were Fumiaki, Roger and Sola. Each racer had a meagre 90 seconds to set their best time, making this the most cut throat stage yet. After what must have felt like forever, the double record holder Sola emerged victorious with a time of 10.236 seconds. She currently holds the “official” world record for the 2018 track and now for the 2019 track. This is important, because while we experienced inconsistencies with the cars/tracks (the fastest time in the final was more than 10 seconds, despite 8 second times being set the previous day) etc. the best racer won and has won consistently throughout the whole process. AWS have done great job putting this tournament together even if it isn’t perfect in places and we can’t wait to see how this tournament evolves over the next few years.
Final thoughts
We had the time of our lives at AWS re:Invent, we’d like to thank everybody involved at AWS, Liberty IT, Liberty Mutual and the DeepRacer community on Slack for all the help and support over the last few months. DeepRacer isn’t perfect, but then every sport has its issues and certainly every new sport will have its teething issues. Despite this, the best racers won their places in the final and we were delighted to have been a part of it.
