"The next frontier for AR": Why Niantic is sharing its tech with Lightship

It's safe to say that augmented reality has struggled to find its moment.

Nintendo 3DS came bundled with a series of marker cards that could be used for AR-powered minigames when it launched in 2011, and came preloaded with AR title Face Raiders, which saw you firing tennis balls at flying robots with faces taken from photographs captured using the 3DS' camera.

Years before that, I recall magazine covers, pages and leaflets that served as markers for AR minigames or dioramas to promote a larger, non-AR game. You can go back even further -- after all, what is EyeToy: Play but a collection of augmented reality mini-games for the living room? These are concepts that have been incrementally expanded over the years, from Sony's Invizimals to Microsoft's many promises for HoloLens.

In 2016, the launch of Pokémon Go prompted a wave of AR photos featuring Pikachu and pals imposed on real-world landscapes, and at the time it felt like the beginning of a new wave for augmented reality. Of course, the novelty of this visual trick wore off and developers instead focused on recreating Niantic's location-based gameplay rather than its AR functionality.

At the time, Niantic's mobile phenomenon did feel like a step forward for the technology, but five years on it's hard to argue that augmented reality has yet to find its feet as a staple technology with which to make games, or indeed any other type of experience. The repeated attempts to make AR happen suggest the technology certainly has promise, so what's holding it back?

"AR projects have been done and published for over 20 years, but [it] only started five or six years ago in earnest," Niantic's chief research scientist Gabriel Brostow tells GamesIndustry.biz. "The reason we say it's now really here is a little bit like when Skype happened -- there had been video calling before then, but suddenly my elderly parents could do it without a lot of instruction. I think that's where we are now."

We're speaking to Brostow at the recent launch event for Niantic's Lightship platform, a suite of developer tools to help other studios tap into the company's learnings around augmented reality, as well as more recent advancements in its tech.

One of these is semantic segmentation: the ability for a computer or even a smartphone, to analyse a camera feed and identify aspects such as sky, ground, grass, trees, solid objects and more. Brostow says this technique has been known to the computer vision community for several years, and was even possible on computers and laptops three years ago, but has since been condensed down to a smartphone. If nothing else, Pokémon Go proved this: augmented reality is likely to only ever take off once the functionality is available in smart devices and engaged with across multiple locations rather than just the home.

"That's absolutely fair to say," says Brostow. "AR should really be a broad church. [EyeToy], that is AR, it's tied to your console, and it's fine. But being able to take it with you, that's a new generation."

Sematic segmentation has wider ramifications than simply identifying your surroundings; it also enables occlusions, so on-screen objects may be obscured by real-world ones or even interact with them. A demo for Niantic Lightship shows a mobile user throwing virtual balls around a real-world environment, which then bounce off objects such as lampposts and benches.

"We have ways of enjoying the technology, we have great game designers, but we can't do it all. And we know this sector is just going to keep growing, so why try to strangle it?"

Gabriel Brostow, Niantic

It's the sort of thing the human brain takes for granted at first glance but when you realise the implications, that the virtual world can interact with the physical one, it opens up more possibilities for AR games. Without it, augmented reality is just a digital layer over a video feed. Pikachu isn't really in your living room; he's on your phone and it only looks like he's sitting on your sofa is you position the phone correctly. But with semantic segmentation and occlusion, the opportunities for interaction in AR increase dramatically. It also helps take AR away from being a novelty and makes it integral to the experience.

"If you had to make an AR experience before, you'd focus on things that fly so they don't have to interact with any surfaces," says Brostow. "Humans would accept that. But the contact with the real world... it's totally analogous to digital special effects in film. You can have special effects that are totally over-the-top -- Jar-Jar Binks, for example. But there are other special effects that are so subtle that you don't notice them, like the android boy in AI: Artificial Intelligence never blinking.

"It's the next frontier for AR, so let's get it outside and in front of people."

Phil Stuart, founder and CEO of UK studio Preloaded -- one of the developers that has already experimented with Lightship -- says augmented reality is all about world context. He adds: "The ability to be able to understand the world allows us to create more contextual experiences. To know that I'm looking at the sky, the grass... when the camera knows what it's looking at and can create digital experiences that reflect that, it's incredibly powerful. It has a level of intelligence that makes it feel believable and really removes the suspension of disbelief. That's the opportunity we have. It's early days, but it's the beginning of a really exciting journey."

Watch on YouTube

Lightship launched last month and is currently free to all developers -- although studios with multiplayer games that host over 50,000 users will face a yet-to-be-disclosed charge in six months' time to compensate for the impact on Niantic's servers. A revenue model will be fully announced later, but for now the company is keen to get more developers experimenting with AR.

Some of the examples shown off around launch include partnerships with Historic Royal Palaces (where users can plant and water flowers in the moat around the Tower of London), the PGA of America (where players can start a round of AR golf in any open space) and the Coachella Valley Music and Arts Festival (whose app allows you to see a giant butterfly land on its rainbow-coloured Spectra tower). More partnerships are planned with the Science Museum Group, Japanese manga publisher Shueisha and wellness app Tripp.

"Niantic will stay with games and always make games. We might make other things as well, but games are where we're coming from"

Gabriel Brostow, Niantic

The fact that Niantic is making its technology available for free (for now, at least) to such a wide range of companies is an odd one. Since Pokémon Go launched, scores of other developers have tried to build similar games around IP such as Jurassic Park, Ghostbusters, Minecraft and The Witcher, yet none have achieved the levels of success enjoyed by Niantic's flagship game. Why give away the secret sauce behind this and future projects?

"I asked a very similar question when I joined Niantic," Brostow admits. "They announced on my first day [back in 2018] that they were going to open up this platform, and I was like, '...What?!' And I've been convinced, gradually.

"Niantic is a start-up -- a big-ish start-up, over 700 people now -- and we have so many projects that we can't take on all the cool, creative things that are out there. And it's not a zero sum game. The projects you see here are coming from teams who have been chosen not because they don't compete, but because they're interested and savvy enough to want to try bleeding edge technology. They're going to do great things with it.

"We have ways of enjoying the technology, we have great game designers, but we can't do it all. And we know this sector is just going to keep growing, so why try to strangle it?"

The growth of the AR sector is a key motivator here. As we've established, augmented reality has struggled to achieve the adoption -- both among developers and users -- that might have been expected in 2016. Brostow even suggests there may be "yet another generation of 'oh, now this is real AR'" still to come, pointing to the potential for AR headsets -- something he tells us Niantic is collaborating on.

Certainly Stuart, as a developer, expects big things for AR on the horizon: "When I first started making games, it was about making the most compelling experiences on screens. With AR, we have the opportunity to look through our screens back into our world and bring gameplay and playful experiences back into our world. AR is a blurry line between the digital and physical world -- is it screen activity if you're looking at the real world and playing in the real world? It's a different type of experience.

"For me, it's a significant moment because it's the attraction of a screen but the reality of our world and I think that's something that's incredibly powerful. Pokémon Go does it to a certain extent, but we're still early, we're still trapped by the screen form factor. Really, this is the stepping stone towards hands-free AR glasses. All of this is a precursor, it's the preface to the main event that's coming in two or three years time."

He acknowledges that AR will continue to be a "subset of the massive mobile game [market]," but maintains it is "something which is powerful, interesting and will continue to grow." He is especially excited about the multiplayer technology available via Lightship, pointing to the effect Pokémon Go has had in bringing people together -- even strangers -- to engage in games together and adds that real-time multiplayer is a particularly big opportunity.

He also suggests, as the array of partnerships Niantic has already secured indicates, that the promise of AR stretches far beyond games: "Pokémon Go and Pikmin will be the foundational pieces for that platform, but the opportunity they're trying to pursue is about looking at these new types of users, new types of behaviours that go across different verticals. The way I look at it, how does Niantic take the way Pokemon Go and games like that changed people's behaviour and bring that to new types of sectors like retail, concerts, arts and culture?"

That's certainly in line with the ambitions Niantic displayed at the launch event, although Brostow is keen to emphasise that the company won't be straying too far from its roots.

"Niantic will stay with games and always make games," he concludes. "We might make other things as well, but games are where we're coming from. They are the perfect challenge -- if you want to know a programming language, if you want to know a platform, write a game in it. You'll have to do the logic, the graphics, the networking -- you have to know it because you can't get away with half-assing any part of the technology in a game. It's the ultimate proof for your technology."