Is Pokémon Go success going to come to a Weezing halt?

It's probably fair to say that the excitement surrounding the release of Pokémon Go is unprecedented in both nature and scale. Rarely has a game ever gained such rapid and widespread popularity across just about every key demographic. This being the case, it's natural to ask "Can it possibly continue?". In this blog post, we'll utilise the powerful underlying AI systems of the latest iteration of our mobile games analytic platform, Player XP, to understand what players are saying about the game, and what this can tell us about the long term sustainability of its amazing popularity.

Popularity vs Satisfaction

The first thing to note is that Pokémon Go has already racked up a massive 106,079 appstore reviews in the first week since its launch. To put that into perspective, Mobile Strike, which currently sits fourth in the iTunes list of top grossing apps has a total of 7,145 reviews in our system. Of those 106,079 reviews, our system has automatically identified 54,221 reviews as containing valuable constructive feedback. The numbers are, as we expected, very impressive. However, our new Player XP scoring system has given Pokémon Go a meagre 34/100 score! This score, which is automatically calculated using the average sentiment of user reviews, can be seen as a major red flag that tells us that the game's users are actually very dissatisfied with the current state of the game. We will now take a look at some of the platforms new features in order to get a better understanding of the game's problems, and what they mean for its long term prospects.

Issues and Sentiment

Of the 54,221 constructive reviews for Pokémon Go, our systems categorised 76% of reviews as talking about 'issues' (bugs, connection problems etc). Clearly, this is not a good sign. But with our new sentiment analysis feature, we can go further and look at how the people writing these 'issues' reviews generally feel about the game. Specifically, our new sentiment analysis feature takes each review and scores it on a scale of 0 to 100 (the higher the score, the happier the review). It's important to note that this is different from the star rating, which is generally a very poor metric for understanding how players feel about your game.

For example, by applying the sentiment and star rating filters in the review explorer, we can immediately find 18 5-star reviews with very negative sentiment scores, 14 of which are tagged as 'issues'. Generally, these are reviews by players that like the game, but are frustrated by the technical problems that appear to have plagued its first week on the market ('It's a good game, but it crashes for me a lot...', 'This game is awesome but you guys need to fix the freezing...').

As well as working out the sentiment scores for each individual review, our system works out averaged overall scores for each different aspect of the game (graphics, audio, issues, gameplay etc), and displays them all in a single radar chart, so you can see exactly where the game's strengths and weaknesses lie. In this case, the 'issues' category has an average sentiment score of 19/100!!! Given that 76% of all constructive Pokémon Go reviews talk about the game's issues, this goes a long way to explaining the overall dissatisfaction of the user-base, as measured by the Player XP score. Looking at the other categories, the only aspect of the game to attain a satisfaction score of over 50% was the graphics, but only 1% of constructive reviews talked about the game's graphics, so the contribution of this category to the overall PXP score is negligible.

Key Phrase Extraction

Okay, so we've seen that technical issues are a huge problem for Pokémon Go, and are contributing to a generally negative reaction amongst players (hence the low score for this aspect of the game). But we'd like to go further. In particular, we'd like to know exactly what these technical issues are. But given that there are over 33,000 reviews talking about technical issues, this obviously isn't something we can achieve manually. Luckily, we've recently been working on a very cool new feature that reads through all these reviews for us, and identifies the game's main technical issues all by itself! To give you a taste of what it can do, below is a list of key phrases that the system identified amongst the issues reviews. As you can see, they tell us a lot about the main technical issues that players are currently experiencing.

• 'Servers Offline'
• 'No GPS Signal'
• 'Low Population Destiny'
• '2 Step Verification'
• '1 HP'
• 'Three Step Glitch'
• 'Claw Marks'
• 'Power Saving Mode'
• 'Red Exclamation Mark'
• 'Sign Out Button'

These are the first 10 key phrases (of over 30) that the system automatically extracts from the game's issues reviews. In the finished feature, the user will be able to see the frequency of each phrase's occurrence plotted over time and updates, and will be able to click on the phrase to see all the reviews containing it.

Most of these phrases are quite self-explanatory, but let's focus on the phrase '1 hp' for now. Looking at any of the reviews associated with the phrase, we can immediately see that there is a major bug in the battling system, whereby the health of gym leaders often freezes at 1hp. Similarly, a quick glance at the reviews associated with the 'low population density bug' tells us that there is a big problem with Pokémon being unavailable for players in rural location.

So, as well as telling us that technical issues are wreaking havoc with the players' user experience, Player XP has actually identified and quantified the specific issues that players are complaining about.

Tags and Comparisons

Another very cool new feature that we've now integrated into Player XP is the 'tagging system'. This is a feature that automatically goes through each individual game and, by analysing the content of its reviews (utilising cutting edge machine learning techniques), assigns it a variety of 'tags' that tell us about its genre, graphics, price, game mechanics etc.

For example, Pokémon Go was automatically assigned the tags 'Free to Play', 'Multiplayer', 'Card Game' and 'Simulator'. Using these tags, we can find similar games through PXP's advanced search functionality. The first game that shows up through this search is Clash Royale (which is tagged as 'Free to Play', 'Multiplayer', 'Card Game' and 'Strategy). Like Pokémon Go, Clash Royale is a hugely popular game with over 50,000 collected reviews.

Another major similarity that Clash Royale shares with Pokémon Go is that its user satisfaction score is badly marred by complaints about technical issues (25% of all constructive reviews concern technical issues, and the average sentiment score of this category is 20/100). In light of these similarities, we can use Clash Royale, which has already been out for six months, as an indicator of Pokémon Go's medium term prospects. To this end, we can use PXP to track the number of reviews per month for Clash Royale since its release.

These figures show that, like Pokémon Go, Clash Royale enjoyed great early popularity, peaking one month after release (in March). However, the game has subsequently failed to maintain its initial success, and has steadily lost users throughout the year. Now, we've already seen that Player XP has identified technical issues as the main source of dissatisfaction amongst players of Clash Royale, and this looks like a major factor in the game's inability to maintain its original popularity. This puts into perspective the size of the task facing the Pokémon Go development team as they try to keep their users happy. For, while 25% of constructive reviewers complained about technical issues in Clash Royale, that number increased to a massive 76% for Pokémon Go. On Nostalgia (An Aside)

Enough about the issues, what explains the immense popularity that Pokémon Go has achieved thus far? Well, we can reuse the key phrase extraction feature but this time, instead of focusing on the reviews classed under 'issues', we can restrict ourselves to all those reviews with a positive sentiment (of which there are currently just under 6,000). This will allow us to isolate the key phrases that correlate with positive experiences amongst the players, thereby telling us where the game has succeeded, giving us insight into the game's current popularity. Below are the first nine phrases (of 31) that our system automatically identified.

• 'Child Hood'
• 'Apple Watch Integration'
• 'Red Blue'
• 'Brings back Memories'
• 'My Inner Child'
• 'Gameboy Color'
• 'Fresh Air'
• 'Bragging Rights'
• 'Never Knew Existed'

Without even looking at the associated reviews, we can immediately see that the phrases 'child hood', 'brings back memories', 'my inner child' and 'gameboy color' express the players' nostalgic relationship to the Pokémon Franchise, a relationship which Pokémon Go seems to be exploiting very successfully. The phrases 'Fresh Air' and 'Never Knew Existed' are testament to the game's success in getting players to use the app to interact with their local environments in entirely new ways, while 'Bragging Rights' and 'Red Blue' speak of Niantic's success in creating an engaging competitive culture amongst players. The question of whether these strengths will be sufficient to maintain the game's popularity in the face of immense technical issues is a difficult one, made all the more intriguing by the fact that Pokémon Go represents an entirely new kind of game. However, we have already seen that technical issues can be enough to derail the popularity of major games like Clash Royale, and unless something is done soon to improve the overall technical performance of the game, question marks will remain around the long term sustainability of Pokémon Go's success.