Pokémon Go Fest attendees to get refunds as technical issues break the event

https://techcrunch.com/2017/07/22/pokemon-go-fest-attendees-to-get-refunds-as-technical-issues-break-the-event

by janober • 9 years ago

109 99 9 years ago

techcrunch.com

vortico• 9 years ago

This game has been so much of a trainwreck, I don't see why Game Freaks licensed Niantic to use their IP.

sddfd• 9 years ago

Niantic seems to have expertise in Mobile Gaming from ingress. Which other company has?

If your comment is based on this event, it seems unfair: Who else has successfully done an event with similar bandwidth requirements before? Also, my understanding is that the cellular network is the problem, but you might correct me if I'm wrong.

Besides, Blizzard had spectacular problems in the past with WOW launch events, simply because no one had ever done something of that /scale/ before.

gambiting• 9 years ago

It sounds like they planned an event for 20k people without telling mobile networks they are doing it. For festivals mobile networks usually deploy extra capacity to handle larger than usual number of people. It sounds like this could be handled by large number of local WiFi spots as well.

dungle62• 9 years ago

20k is actually not that many people as far as these things go. Over 50k in that confined stadium area and you really start hitting some hard limits with current cell tech. 20k should have been no problem with careful planning.

40acres• 9 years ago

Has it? I read recently that they still had around 70m MAU.

FullMtlAlcoholc2• 9 years ago

The integration of ltheir location api with AR was reasonably well done at the time. I agree though, Niantic has no business making games. I try to imagine a world in which they license their AR SDK

mrisoli• 9 years ago

You think the AR was well done? Serious question, not trying to flame or anything, but I disagree. For me, the AR is pretty much just overlaying a Pokemon graphic on top of camera footage, there's very little interaction to the camera input.

In fact, after playing for a bit, I turned off the AR because of loading times.

hkmurakami• 9 years ago

Because ingress was an enormous success and a cultural phenomenon in Japan.

yoodenvranx3• 9 years ago

Not only in Japan but also in Germany and the US

monksy• 9 years ago

The weather in Chicago since 5pm till early this morning has been filled with storms. Storms were also threatening this afternoon, but surprisingly they didn't come. I had to cancel a beach meetup that I was looking forward do over this.

kermittd• 9 years ago

The event looked interesting and the game is a great concept. Sorry to hear about this fail as well.

kayoone• 9 years ago

Even at big music festivals, mobile networks are usually totally overloaded just by the sheer amount of people logged in at the same time, without even doing anything. Even recieving an SMS can take minutes. I don't assume they would do an event like this without letting providers know in advance to upgrade capacity, but it seems like this.

iDemonix• 9 years ago

I've always wondered why cellular is affected by this, anyone have a link?

TrickyRick3• 9 years ago

There's a limited amount of "space" in the air, ie not everyone can be transmitting at the same time. This is fixed using a few different techniques depending on the cell network but for GSM something called TDMA (Time division multiple access) is used. This basically gives your phone a certain amount of time to broadcast on before it should leave the airwaves for someone else to transmit on. Too many people in one location means not enough time for everyone to transmit.

mschuster913• 9 years ago

I believe it's the backhaul capacity that's the problem. Even assuming "only" 1 MBit/s constant traffic per customer, an uplink of 1 GBit/s can only handle a thousand clients tops.

Cheap-ish providers are worse - they usually don't run fiber to every tower because laying fiber is expensive, but instead only to one tower which then distributes uplink via directional microwave to nearby towers. Once that link gets saturated or has issues, the whole area is screwed.

Providers like Deutsche Telekom (or other former monopolists) usually have it easier as they have huuuge amounts of fiber and especially conduits in the ground, which means it's easier for them to wire up even small towers with real dedicated fiber uplink.

foobaw• 9 years ago

Disclaimer: I worked at an OEM phone company.

Cell towers get saturated pretty easily by crowds. Network providers probably did not account for the crowd, or even if they did, it wasn't enough.

I would assume a company like Niantic asked network providers to install additional cell towers beforehand, so my best guess is they tried their best but still failed.

The technical details are quite complex, and you would need to read up on GSM,UMTS,LTE,TDMA,CDMA, but the bottom-line is, the overall technology isn't good enough yet.

fencepost5• 9 years ago

The technologies used and underlying principles have changed enormously from 2G/GSM to 3G/WCDMA to 4G/LTE/OFDMA but a lot of it comes down to questions of how many individual connections each tower can handle and the minimum amount of airtime communicating with each device requires. At one point (still?) there are also questions of how strongly each device and tower must be broadcasting to keep the received signal at each end within acceptable levels.

Edit: Expanding on this a little while I still have time to edit, and all of this is from a non-telecom person so it's likely I have errors in here. Still, this should provide a starting point for the history and what to look for to get more current/accurate info.

Most 2G was GSM which used TDMA (Time Division) - basically, each device connecting to the tower was assigned a "slice" of the available time and had to broadcast within that time slice. When the tower ran out of slices, some devices simply couldn't connect to it. Sprint and Verizon were the exceptions, they used CDMA which allowed many devices to broadcast at the same time and was able to decode data for/from each device based on the Code Division used to encode it. CDMA towers "breathed" which meant that their coverage area could vary based on how many devices were connected because the devices and towers had to keep broadcasts within particular power levels at the endpoints. CDMA allowed more devices, TDMA could allow larger coverage from each tower by reducing the number of time slices (which also reduced the number of devices that could connect). Notably, CDMA allowed devices to talk to multiple towers at once with reconciliation being handled by a smarter backend, while TDMA required more explicit handoffs between towers. Way back when this was why on GSM your call would simply drop, but on CDMA you'd simply develop a severe case of robot voice.

3G almost all moved to WCDMA, which I believe had many of the same advantages in density as CDMA.

4G moved to somethind different (OFDMA?), which I haven't read enough about to really understand but which is technically quite different from WCDMA.

Most LTE uses yet another protocol. On T-Mobile, this was noticeable when T-Mo "refarmed" its network, converting a bunch of frequency bands from 4G to the newer higher-capacity but incompatible LTE. I know a few people who had 4G phones on T-Mobile and were offered free replacement phones because T-Mobile had basically yanked the actual 4G network out from under them.

Basically, there's a lot out there, it has changed a LOT over the last 10-15 years, and many of the changes were actually complete replacements rather than incremental upgrades.

lnanek2• 9 years ago

What's to wonder about? Each tower can only handle a certain number of people, and the cell phone companies don't put the number of towers needed for an event like this in permanently.

mschuster913• 9 years ago

Even at big music festivals, mobile networks are usually totally overloaded just by the sheer amount of people logged in at the same time, without even doing anything

In Germany, ordinary political demonstrations (from my experience, everything from 2k to 80k is enough) are sufficient for O2 service to totally break down, even in the center of major cities.

The Oktoberfest in Munich has similar problems despite operators putting up dozens of small BTS, but then again, it's up to 350.000 people concentrated on maybe 200.000 m² and it's really hard to get them all serviced good enough to live-stream their Brathendl and beer Mass to Instagram...

SOLAR_FIELDS• 9 years ago

They have this figured out in Austin for huge events such as ACL. Temporary towers erected in the correct locations can easily handle the extra load with proper planning. I went two years ago and had absolutely no issue with LTE or service

fencepost5• 9 years ago

Just as glad I didn't get a ticket ($20? sure. Scalper prices? Heck no) and waste the time. This whole thing is going to turn into a real fiasco for some time due to the number of people who bought resold passes (so the original seller sold it and now gets their original $20 back), people who traveled (I know many got hotels, not sure how many flew in), and questions of how they're going to get that in-game currency to people (reportedly some are still in line to get in, while others are waiting in a not-quite-as-long line just to get out).

Each person attending has a uniquely-numbered wristband pass, and apparently on entry they're getting an envelope with a patch and a unique-to-them QR code which needs to be scanned after spinning a special "Pokestop" and effectively checks you into the event. Since the game is crashing basically at startup for most folks, that makes it a bit of a challenge just to get in. Still, I suspect the in-game credit is going to be based on people who checked in with that code since the ordering process didn't involve providing a game login.

Also, I've seen some folks questioning why they didn't have WiFi set up, but I'm pretty sure that the way WiFi works would make that almost impossible anyway (2.4 would be a bad joke no matter what, 5GHz might be possible over most/all available channels, APs that can handle hundreds or thousands of simultaneous connections are rare and probably require major advance planning, AP broadcast power would probably have to be quite low to allow large numbers of APs in a physically small area, etc.).

maxerickson4• 9 years ago

People in a stadium aren't going to be quite as focused on their phones, but lots of people in a tight space isn't an unusual occurrence for networks.

fencepost5• 9 years ago

~15k people in a small section of Grant Park all trying to play a somewhat data-intensive online game that requires constant communication with servers is a VERY different activity stream than people in a stadium. Might be closer to think of those people in a stadium all trying to stream video of the event.

ericabiz• 9 years ago

Stadiums have issues with this too. I was at Marlins Park for the All-Star series. During the Home Run Derby I had press access, which had special Wifi credentials on a completely separate network.

It was down for the entire game. A lot of hotspots popped up in the press area.

The during the actual All-Star game I tried the regular stadium Wifi. It was down for most of the game.

T-Mobile was up for most of the game, so I was still able to publish photos and read people's reactions. A good thing as T-Mo was the main sponsor for the games.

aeleos• 9 years ago

There are definitely companies have the kind of wifi infrastructure you are talking about who offer rentals to the people who organize these kinds of events. Here is an example of one[0].

Wifi is often deployed at large events (PAX East for example) and it is more then feasible if the company is willing to pay the money for another company to do it.

However, their server infrastructure is another question entirely. In today's age, making scale-able network services with very little downtime is not something that requires a dedicated networking team and hundreds of millions of dollars. I can't imagine they have very many good excuses to why they continue to allow this to happen other than that they don't value investment into their infrastructure enough (despite being a company completely dependent on the internet)

The problems that they are experiencing are ones that have caused problems and had solutions created for them for years. The just seem to be unwilling to properly invest in the solutions to fix them, which are cheaper than ever (considering the complexity of the issues in the grand scheme of things).

[0] https://tradeshowinternet.com/solutions/event-organizers

fencepost5• 9 years ago

They may also be having load-balancer issues assuming they have front-end servers that are checking whether accounts are supposed to be directed to a specific set of servers. Assuming that they normally load-balance based on geographic location, the check for specialized server connections may be something that they never actually load-tested before after confirming that it worked for their staff in testing.

I asked earlier and have asked again whether anyone at the location but not officially attending is having better luck connecting, that would help to narrow it down between carrier issues, Niantic server issues and Niantic server-for-this-event issues

Edit: They expanded the area for special/regional spawns to 2 miles around Grant Park but still only for people who "checked in" at the park, and people who've left are reporting much better luck when they're away from the park. That may indicate that most of the issue was insufficient networking.

ploxiln• 9 years ago

A company I used to work for specialized in exactly this case - high density WiFi. They have an 8 radio / 24 antenna AP: https://www.xirrus.com/product/xr-4000-series/

You can use 6 20 MHz channels in the 5GHz band and 1 or 2 channels in the 2.4 GHz band, on each of these multi-APs, and you can get much smaller "cell sizes" (low power signal, smaller signal area) than cellular providers. You can bump clients from busier channels on to less busy ones to force better balance.

WiFi drivers for clients and APs are a big pile of bugs - the resilience (retries at lower rates, chip-timeout-resets, etc) covers a lot of them up. You can get more consistent throughput and latency by digging in and fixing these bugs (and working around bugs in most clients). Most good WiFi APs are debugged until they work pretty well for the "typical" home or office use-cases. But the Xirrus APs (and some competitors to some degree) are debugged for the "high density" use-case - multiple hundreds of users per multi-AP, and maybe 8 to 30 of these multi-APs in a large conference hall or stadium.

You have to know what you're doing. It is expensive. Niantic has a hugely popular service. They're making money. But they never seem to know how to handle the massive scale that was predictable after the first month or so. They sold tickets, they knew how many were coming. There are people who know how to handle that. Niantic just didn't feel like doing the research to find out how or who to handle it. Their systems fall over again and again and again.

intopieces• 9 years ago

Anyone who has ever tried to send a photo at Soldier Field during any large event could have predicted the crippling of the network that happened here. Maybe I missed it in the article, but did the event organizers not set up an ad-hoc network for this?

Hydraulix989• 9 years ago

This is a situation where a mesh network topology (like Firechat) would be necessary.

azernik2• 9 years ago

Mesh is unfortunate for these situations; the limiting factor at these scales isn't infrastructure size, but EM spectrum. The accepted solution is mandating players go onto WiFi, where you can get them to much lower emitted power by deploying enough APs, close enough to any player location, that no clients need to shout to get their traffic heard.

dboreham3• 9 years ago

Sigh. Except in battlefield and deep off grid situations mesh is always a bad idea.

Thriptic• 9 years ago

Agreed, this should have been totally predictable based on Bears games, lolla, etc.

yoodenvranx3• 9 years ago

Niantic organized a lot of big Ingress events for the last 3 years so they _must_ have known about these problems but somehow they still managed to fuck it up.

yoshamano• 9 years ago

What they should have done is made arrangements to have COWs on site from the major providers.

https://en.wikipedia.org/wiki/Mobile_cell_sites#Cell_on_whee...

Matt3o12_• 9 years ago

How do networks manage to have sufficient backbone for those mobile towers?

What good are those towers if thousand people can connect but cannot use any internet because their backbone cable is so slow. I once was at a festival in Germany where they had such measures but lacked backbones (I had full signal strength in a remote array but the internet was unusable slow).

Not every town has fiber optic to support such a surge of people.

dboreham3• 9 years ago

Or Disneyland.

free_everybody• 9 years ago

When Niantic’s John Hanke took the stage, he was greeted by an audience a few thousand deep, many of them chanting “FIX YOUR GAME” or “WE CAN’T PLAY!”

Can you imagine how stressful this would be? It's one thing to get disappointing feedback data on bugs, UX, etc. Standing in front of thousands of angry people screaming at you to fix your software this instant? Yikes!!

capkutay• 9 years ago

I remember Google Cloud was doing a ton of press about how Niantic was using their platform for Pokemon Go. I wonder if that will backfire for them now.

dmoy• 9 years ago

These are completely unrelated, no? One is about scaling the servers and stuff behind the scenes, this thing is about cellular data networks failing and dying.

cptcobalt• 9 years ago

Humans just aren't made for that sort of feedback. I couldn't imagine 10 or 100 people yelling genuine critical feedback like this at me -- let alone thousands. I imagine he didn't have a great day.

ShabbosGoy• 9 years ago

Could they have used a mesh network to help load balance connections to WiFi APs?

NightlyDev2• 9 years ago

I guess that's no suprise? Downtime with every single event som far, so why should this one be anything different?

WiFi is a thing and should he able to handle it just fine. Heck, they could even block anything but PGO and get away with low bandwidth too.

When voulenteers can manage to get WiFi working for thousands of nerds at indoor LANs... Let's just say that Niantic should be able to avoid this with a little bit of planning.

This is the company running one of the most unstable and buggy games I have ever played, so I would expect no less. At least the game is very fun for a lot of people... When it works.

make3• 9 years ago

Don't large lans use.. LAN ? (Ethernet). I could be wrong, but as many desktops still don't have wifi and you know where people are and it's so much more stable large scale.. it would make sense

NightlyDev2• 9 years ago

Ofc ethernet is used, but WiFi is often used for phones and additional devices like game consoles as it's mostly 1 port/person. The gathering and Dreamhack are two good examples of events with WiFi scaling to many thousands of connected devices without much problems. More or less every AP provider has solutions for this.

yoodenvranx3• 9 years ago

As a long term Ingress player i am not surprised by this _at all_. Niantics management has proven itself again and again and again as being completely incompetent at everything they do. (They even to manage to ban their own official event photographer after he had GPS problems while traveling to Paris).

I used to be angry at them for ruining Ingress (one of the best games ever made) but nowadays I am just amazed at the level of sheer incompetence.

They managed to learn _nothing_ from 4 years of Ingress and I think now it's time that the whole upper management is fired so that the company can start fresh.

lini• 9 years ago

The scale of this event is bigger than any Ingress anomaly until now. Also, all players seem to be concentrated into a smaller area than Ingress events, which further complicates things. I think the main issue they could not control today is that Verizon ran out of bandwidth and people simply couldn't connect. Another difference with Ingress events - the Go Fest is a closed venue with security and ticket validation at the gate. In Ingress, you usually validate your ticket beforehand and don't have to worry about queues in the hours before the actual event stats.

grogenaut2• 9 years ago

Did they buy extra coverage for the event? I've been to football games and mardi gras and they don't have issues with coverage and that's a lot more than 20k people even if they're not all playing it should be similar unless PMG is really really bad with bandwidth.

CydeWeys• 9 years ago

I think that's overstating it a bit. I went to an Ingress anomaly in Brooklyn last summer and everything worked fine. It wasn't the smoothest run event ever, and they could have used more staff, but it was an enjoyable time.

There's something different about what they were trying to do with Pokemon Go that made it suck. They don't have experience with these yet. And if someone made long distance travel plans, just getting your entry ticket refunded doesn't begin to make up for your loss.

syntheticnature• 9 years ago

As an Ingresser myself, I'd say there were two big sources of issues vs. your typical Ingress anomaly, and one that would happen in Ingress if it was a bit closer in popularity.

First off, they had everything in one place. For those who haven't played Ingress to that point, anomalies in Ingress usually take place with a primary and two secondary locations in, roughly, Asia, the Western Hemisphere, and Europe. This means that people aren't usually traveling from far away (which probably would increase overall participation) but also that each site's numbers are lower. I figure overall anomaly attendance on a given anomaly weekend might well rival that of this event -- but they're spread out in geography and time.

The second aspect is that Ingress has much more encouragement in-game of local organization. I understand quite well why Pokemon Go lacks in-game chat, but it saps local organization. Locals play an important, perhaps critical, role in Ingress event organization, and provide Niantic with free boots on the ground to figure things out. I know there is some local organization going on in Pokemon Go, but it's forced to be based on random encounters, and folks are still unsure whether to organize on team lines or just geographically. (With the current setup, I'd say geographically -- if there's anything to be learned from Ingress, it's to tamp down factionalism.)

The last thing, the one that I think would hurt Ingress equally? It seems there wasn't a limit on ticket sales. In Ingress some special extra-value packs that get you extra tchotchkes both virtual and physical exist, but the base 'free' ticket is unlimited. Similarly, here. They probably wanted to avoid any chance of scalping or the like, but imagine if they limited it to, say, a quarter the attendance?

FreakyT2• 9 years ago

I realize this statement probably won't go over well here, but Niantic's behavior strikes me as the kind of thing that happens when a company is entirely run by software engineers who refuse to accept relevant advice from anyone who isn't a software engineer.

There are people who understand event planning, there are people who understand how to build F2P mobile games, and I get the impression that exactly zero of these people were consulted, or, if they were, their advice was immediately ignored.

quasse• 9 years ago

If that were the case you would hope they would at least be good at software engineering, but all evidence points to that being completely false.

awalton• 9 years ago

Anyone who's played Pokemon Go or Ingress can refute this; their UI design is abysmal at the very best, the game suffers a million bugs and patches introduce at least as many as they fix, etc. A company ran by Software Engineers would have hired competent UI developers and QE engineers and would have demoed the hell out of the event before putting it into production...

No, I'd argue Niantic may originally have been engineering driven, but once Ruthless cut them from Alphabet, they went straight for the greed play, developed with an IP that would sell billions with next to no effort, and are now tightening the screws on the player-base with Pay-to-Win gym tickets and events requiring as many Lures and Incenses as you can buy to continue extracting income.

davidmurdoch3• 9 years ago

I'm making a massive location-based game myself right now (currently have 140 million POIs in the USA alone) and have been studying how Niantic is making decisions for about 9 months now. It's pretty crazy how poorly they treat every misstep they make, though I think they did a good job quickly giving refunds, rewards, etc after this one, as well as a decent sounding apology.

Of course, they should have been better prepared for the event, but they did seem to do better at the cleanup this time. So... baby steps?

dEnigma• 9 years ago

Niantec just posted an update[0]. These are the steps they will be taking:

All registered attendees will soon receive an email with instructions on how to receive a full refund for the cost of their ticket. These instructions will be sent to the email addresses associated with your Pokémon GO account.

All registered attendees will receive $100 in PokéCoins in their Pokémon GO account.

Special Pokémon, Eggs, and check-in PokéStops appearing during Pokémon GO Fest have had their range increased to a two mile radius surrounding Grant Park through Monday morning, July 24. These Pokémon and Eggs will only be visible to Pokémon GO Fest attendees who validated the QR code they received when they entered Pokémon GO Fest. Attendees who were unable to validate their QR code during the event can do so through the special PokéStops through Monday morning.

All registered attendees will have the Legendary Pokémon, Lugia, added to their account.

[0]http://pokemongolive.com/en/post/pokemongofestupdate

kendallpark• 9 years ago

All registered attendees will have the Legendary Pokémon, Lugia, added to their account.

Lugia? That’s some damage control right there.

mizzao• 9 years ago

Beg for forgiveness and give out free stuff, that generally makes people much happier :)

cloudkj• 9 years ago

Are there any Niantic employees active here on HN? As an engineer, I would be very interested in hearing some first hand accounts of some of the technical problems Niantic's engineers have had to deal with, not necessarily just for the Go Fest, but for the game in general. To the extent that they're allowed to talk about it, it would be very interesting to get an AMA style take on some of the challenges.

davidmurdoch3• 9 years ago

As the lead engineer on an up coming game of a similar scale, YES! This would be so awesome!

em3rgent0rdr• 9 years ago

The silver lining is that all these people would would only interact with each other virtually then had a perfect opportunity to talk to each other face-to-face, fwiw.

jbmorgado• 9 years ago

What surprised me in this article is that people still gather to play Pokemon Go.

I personally never got into the game, while my cousin was totally into it, but where I live (which is a medium city for Europe), the fad has totally passed in less than 6 months after launch.

Pokémon Go Fest attendees to get refunds as technical issues break the event

Related Stories