Did Pokemon GO Try Too Hard to be Agile?
When Pokemon GO hit the App Store in July of 2016, it quickly became a global phenomenon. What was also quickly apparent, was that the game had flaws that undermined the user experience, and that users were leaving the game in large numbers.
While the game issues have received the lions share of blame for the loss of users, Niantics apparent lack of preparation in dealing with these issues has also (rightly) been criticised.
However, blaming Niantic for not correctly anticipating the response to Pokemon GO is somewhat shortsighted. Operating under the assumption that Hanke and employees are smart, talented and driven to produce the best possible product, what specific part of their decision making process was responsible for Niantic being unable to deal with the success of the game?
Is the project management framework utilised (Agile) responsible?
Niantic and Pokemon GO
To attempt to understand how Niantic found itself in this position, some context is required.
The first iteration of Niantic began with John Hanke developing an overlay for Google Maps in 2010. This led to the creation of the company within Google, and the development of 2 apps, Field Trip and Ingress, based on Augmented Reality technology. When Google announced the creation of Alphabet in August of 2015, Niantic Labs, already a somewhat standalone company within Google, was spun off.
The rationale for this decision was partly based on the perception that Niantic ‘wasn’t as agile’ as other startups it was competing against. Running the company outside the Alphabet umbrella allowed Niantic to truly adopt Agile philosophies. In a techcrunch.com interview, Hanke said of the spinoff from Google,
Being independent allows us to be more nimble across the board
In September of 2015, Niantic announced that it was developing Pokemon GO, and a month later announced it had raised $20m of funding from Google, The Pokemon Company, and Nintendo.
Pokemon GO launched on July 6th, 2016 to a huge response from fans. It had the biggest opening week of any app according to Apple. Estimates suggested that 1 in 10 Americans played the game, and that it was generating $6m USD in daily revenue from US players alone. Despite only being available in 3 countries at launch, players all around the world spoofed their GPS locations or created fake app store accounts to play. Hillary Clinton referenced the game on the campaign trail, and multiple Norwegian members of parliament were caught playing the game while in chambers.
All metrics were smashed, with Daily Active Users reaching 45 million by the 3rd week of July (2 weeks after launch), and Time Spent In App beating even Facebook’s numbers.
Niantic and Agile
While Hanke and Niantic have not publicly commented on Project Management methodologies utilised during the development of Pokemon GO, we can infer from the speed of development and the incremental rollout of Pokemon GO that an agile framework was used.
The initial release of Pokemon GO has been considered a Minimum Viable Product, or
Niantic utilised this popular Agile technique of maximising capital efficiency, which also quickly creates a healthy backlog to feed Scrum teams. The MVP method is the quickest way to develop a ‘Build-Measure-Learn’ feedback loop upon which Scrum thrives.
Pokemon GO and Users
With the release of Pokemon GO, the Scrum Backlog filled up quickly. Despite the apps popularity, it was plagued by issues. Server overloads prevented millions from playing each day. Buggy GPS tracking means players were engaging with the game but the game was giving nothing back. Constant app crashes meant restarting progress constantly. Crippling battery usage resulted in a boom for portable power banks, but constant frustration from players.
Teams worked quickly to resolve these issues, but more kept appearing. As more servers were bought online, the infamous ‘Three Step Bug’ appeared. At San Diego ComicCon, Hanke plead with players to stick by the app, promising many more features, and that only 10% of the envisaged final product had been released to players. He also addressed the ‘Three Step Bug’, promising that Niantics engineers were working hard to fix it. Despite this, within a week the ‘Pokemon distance indicator’ that was affected by the bug was simply removed from the game.
These promises and what work Scrum teams were making on the backlog did little to appease annoyed users. By mid-August, 6 weeks after launch, Bloomberg estimated that Daily Average Users had dropped to 30 million, and further to 20 million by October. Daily Time Spent in App was down to 26 minutes, half that of Facebooks which they had previously topped. Daily revenue was down 75% from the peak to USD 1.5 million per day.
While these numbers are still very strong, and the estimated USD 500 million of revenue generated by the app represents a huge return on investment for Niantic and its investors, one has to wonder about the amount of money left on the table.
Agile and Results
The Minimum Viable Product approach to building a Scrum backlog works well when a company can control the level of reach their product has. Software firms creating products for clients are able to limit distribution, and hence the impact of flaws in the MVP. Companies creating products that already have a huge viral following should be careful when following in Niantics footsteps, as even the most loyal fans only have so much patience.
The Scrum philosophies may have been to blame for some of Niantics shortcomings in developing Pokemon Go. Prioritising ‘Individuals and Interactions over Processes and Tools’ may have lead to oversights in rigorous testing of the weakest links of system architecture. ‘Responding to Change over Following a Plan’ may have lead to overconfidence in the MVP and Niantics ability to fix problems as they arose, not before they arose.
Finally, Niantic appears to have suffered from Confirmation Bias when planning release scenarios for the app. Not only did they massively underestimate user demand, but the sometimes strange locations of poke-stops, robberies, and ‘unexpected discoveries’ show that not enough effort was put into truly understanding what effect the game would have on people’s lives.
While Niantics Agile development of Pokemon GO cannot be considered a failure (it was a huge financial success and continues to generate substantial revenue), it provides valuable insights to the shortcomings of Scrum and Agile.
Note: This article was written in October 2016, some figures / links may be out-of-date, but the overall argument isn’t affected.