Developer Blog 2: Role Selection Algorithm
An 80’s Cartoon Approach
I’m a pretty big fan of 80’s era cartoons (maybe my gamer tag makes more sense now), and even though I know this catch phrase is pretty cheesy, it has been a source of motivation for me throughout my life. Sure, I have since learned that this concept was not coined by a fictional cartoon hero, but that doesn’t take away from the message. It is with this mantra that I approach working with Mobalytics: Give a player knowledge in the right way, and they are halfway to a ‘W’.
When we approached our plans for our Pre-game Analytics, we knew that what we wanted to create was ambitious. We were trying to create a tool that, frankly, didn’t exist anywhere else. Sure, there are plenty of tools people use prior to their champion appearing on the Rift, but none of them were smart. None of them could tell you what you should do to win the game. There was too much left to interpretation.
People are smart; it really is astounding at the marvels we are capable of, but we have limitations. We can only focus on so much at one time. We fall into old habits. We forget. It was our intention to create a tool to help mitigate some of those weaknesses.
Comprised of career researchers, computer/data scientists, and some of the most knowledgeable League of Legends (LoL) players on the planet, the R&D team quickly noticed two things lacking with the suite of tools currently available. First, the team found that players were consistently making incorrect conclusions, primarily attributed to the low depth of information they had available. Second, there hadn’t been earnest attempts, in our opinion, to look at how each player strategically fit into their respective team before a match had loaded.
The team spent a great deal of time tapping into the LoL experts’ sheer knowledge of the game to brainstorm and create a pretty neat advice system. We would give players advice on how to win the lane matchup they were in, and how to succeed in the teamfight portions of the game. It was my job to design the software systems that would support this. Hoping to get the “easy” one out of the way first, I started thinking about the “Win the Lane” advice (WTL). We wanted advice that was specific to the champion — little nuggets of wisdom — information that was present, but maybe not distilled down in bite-sized chunks for players. League is a game that revolves around windows of opportunity, but sometimes knowing what those windows are is difficult to understand. It was our goal to try and highlight some of these very specific opportunities. We knew most players expected to be using our system would already be skilled Summoners, so, we made it a priority to try to provide advice most hadn’t thought of already.
Problem Numbero Uno
We needed a system with the capability to give advice for champions based upon the role they were being played in, even the specific matchup! So, after designing the data structure, I dove into the Riot API. For those that don’t know, the Riot API is the toolset that Riot gives developers to access the interesting data about their game. It powers their match history, amongst other things. This is where we ran into problem numero uno. Riot’s API provided data about what lane a player played in after a match completed, but it didn’t tell us anything about the match they were currently playing in. Makes sense, right? After all, hindsight is 20/20 (remember this…).
This became our first challenge. We needed to create a system that could reliably and consistently predict which lane a player would be playing in, a difficult problem for an experienced player (let alone a computer). Just think about how often you guess wrong, and hey, that’s okay because it’s part of the game design. Champions are intended to be flex picks sometimes precisely to make the draft phase more difficult.
Now, there are several types of developers, and, like any nerdy group, we have vehement arguments about which programming language is better, or which database type, or hell, how many spaces should belong in a tab in code. I consider myself to be a data-driven developer, being a huge cheerleader for big data’s ability to reveal patterns that not even the best expert might catch (along with its numerous applications in statistics and machine learning). It is with this bias that I approached the problem.
Some members of my team said to simply look at champion and summoner spells, which tend to be consistent per lane. Ghost is almost never in the bot lane. Smite is almost always in the jungle. Though these things are true, simply using summoner spells just didn’t sit right with me. I remembered Runeglaive Ezreal or the slew of top-laners that took Smite in Season 5. I called out Ivern supports with Smite. I rejected that Teleport was only used in solo lanes, recalling so many games I had played with or against a Shaco or Yi in the jungle teleporting to a ward in the bot lane. I knew that some supports, just from their summoner spells, could look like mid-laners, and vice versa.
So, I did what most data scientists would do, got a whole lot of data and tried to look at probabilities. I sketched out an algorithm design and passed it off to the dev team to code and test. A couple of days later, I reviewed the test results, fully aware there would need to be tweaks, but maybe a bit optimistic on how much small changes would influence things.
(It should be emphasized here that the development team, the software engineers actually charge in implementing our hair-brained ideas Are The Real Heroes Here(TM). They are always quick to implement our ideas and were invaluable to this process, interjecting their opinions to help improve the algorithm throughout)
We had something that kinda worked. About 70% of the time it was guessing an entire team correctly, which seems pretty good. I knew there were improvements to the code that hadn’t been implemented yet. But, I wanted this thing to be perfect and, because I knew that numbers tell a story, but people frequently misunderstand the moral of that story, I dove down deeper. When a team wasn’t correctly assessed, how bad was it screwing up? How many players were being put in different roles? How many times was one team correctly guessed, but the other team incorrectly guessed? After all, it doesn’t matter if we get your lane correct, we need to know both what lane you are in, and who you are fighting against.
So, we iterated. This is a word that is used a lot in the tech world. Iterate is like a four-letter word for the business people in a tech company, because it means, “we made a small, incremental change that is going to need probably many more small, incremental changes until we get it ‘right’.” (To be fair, ‘pivoted’ is the term the sales and business staff use that most often gets a developer’s blood boiling.) And we weren’t even fixing our algorithm, we were tweaking the script we were using to test our algorithm.
After doing some tests, the news wasn’t exactly good. Some 4 in 10 games would yield inaccurate results for at least 2 players. That wouldn’t do. However, it was also during this time that we ran into something quite interesting. Remember what I said before about hindsight being 20/20? Yeah, forget that.
Hindsight isn’t always 20/20
We tested our algorithm by feeding the input data we had available to us (champions and summoner spells) into our code from games that had already finished. These were games in which Riot had already figured out who was in what role.
But, there was a problem…
During our tests, we would rate the “amount correct” a team composition would be, compared to what the Riot API said. We started consistently getting results with 80% accuracy in roughly 10% of analyzed games, which shouldn’t be possible. Given LoL’s team structure, it would be reasonable to expect 100% or 60% accuracy as there can never only be one role wrong (4/5 = 80%).
We evaluated the games more carefully, and found a Worrying Trend™: sometimes Riot API was getting the roles wrong. It usually stemmed from the API accidentally saying there were two junglers when there clearly weren’t. This was a big deal because our solution needed to look at past games to make assumptions about future games. So, we made some tweaks (aka iterated), and got rid of all games that were messed up from the Riot API end, and moved on.
(Note: The above paragraph greatly minimizes the amount of time and effort it took to find this anomaly and suss out a solution to it. Knowing something is wrong, and finding out what is causing the problem are two very different things to solve. Ask a mechanic.)
The Off Meta Conundrum
This brought us to our next problem. What happens when someone plays a champion in a role that champion isn’t normally played? People do that. Sometimes they don’t know the meta (we found this true in a lot of the lowest brackets) or, maybe, they just don’t care. People try something new because they think it will be OP. Sometimes it is (see: Miss Fortune Support, circumstantially). Sometimes it isn’t (see: Miss Fortune Support, universally).
Good or bad, we still needed to handle this. So, we relied on the fact that people, in general, will be most likely to play a champion in a particular role, and started building a bit of a personal history to add to our accuracy. Now, of course, there are a lot of technical details worth glossing over here for brevity (ha! brevity, this is already quite long): how to balance meta and personal history? How do we get information about people that we don’t have a track record in our database without slowing the whole system to a crawl? How are we going to handle data limitation?
Needless to say, it took a few more days, but in the end, the results were well worth the additional effort. At this point, we had jumped complete match accuracy (meaning both teams were completely correct) from around 58–60% all the way up to 81.6%. This number doesn’t tell the real story, so let me break it down a bit further: even when there was a failed match, most players in that match were getting their correct roles (since all it took was one player incorrectly identified for us to flag a match as incorrect). Our system was guessing individual roles correctly 95.2% of the time. It also meant that at least 91.2% of the time a player would be matched with the correct lane opponent. (We say at least, because, sometimes the system could swap roles around, but “accidentally” put people against the correct champion, just maybe in the wrong lane). We had an internal target of 90% accuracy or higher, and we had hit it! We were off to the races…until we weren’t.
Recognizing mistakes
I’d rather make mistakes than make nothing at all. ~Ekko
Testing our system with live data, we began getting great results, but also noticed an interesting pattern. When games incorrectly assigned roles, it was most often due to the jungle role being mismatched — an issue that was easily identified by the human eye, but somehow not to a computer.
Now, the heading of this section is about recognizing our mistakes. It should be known that, given unlimited time and bandwidth, we could solve this problem using big data. However, we didn’t have either. Players have roughly 30–60 seconds before a game starts to explore our tool, collect the desired information, and digest it. Acquiring any additional data would need to be fast and precise.
So, I backtracked a bit. After talking with the other members of the R&D team and thoroughly investigating several case studies, we determined that these problems would be resolved with additional conditions informed by expert domain knowledge. So we iterated… again.
That brings us to our current state of affairs. Our role prediction across an entire match is close to 96% accuracy right now. Our algorithm picks roles better than I do, and I have been playing the game (poorly) since Season 1.
Always improving.
We might be close to 100% accuracy, but we’re not even close to being done. We are constantly looking at ways to iterate and tweak the system to make it even smarter. We want to stay up to date with the changing meta. As players begin to use our tools and we get more data, we promise we will make whatever tweaks we need to give the absolute best possible experience we can to our users. That is our commitment to not only the community but also to ourselves.
I wonder what’s around next corner. ~Braum
(Thank you for reading. Sign up for Mobalytics Beta today!)
