The ‘Killer Apps’ That Could Make Apple’s New Headset a Winner
Augmented and virtual reality have mostly been dead ends so far. How Apple’s vision could succeed where others have failed.
By Christopher Mims
Wall Street Journal
May 26, 2023 9:00 pm ET
DOMINIC BUGATTO
------------------------------------
Explaining the inner workings of a nuclear-powered octocopter that will someday explore Jupiter’s moon Titan is a tall order for even the most skilled teacher.
Luckily, when I recently paid a visit to the lab where it’s being designed and tested, an engineer tasked with building the Dragonfly probe for NASA pulled out a pedagogical aid sufficient for the task. It was a HoloLens 2, the augmented-reality headset made by Microsoft.
Each of us wore a HoloLens headset that projected a realistic, shared, three-dimensional image of the Dragonfly in front of our eyes—a sort of shared hallucination. We were able to walk around the space probe, my guide peeling back its layers and pointing to individual components.
That experience was remarkable precisely because it couldn’t have been more compelling and edifying in any medium other than augmented reality—the name for tech that imposes digital images onto our view of the real world. In the not-too-distant future, that kind of experience could be utterly mundane. Just as personal computers and then smartphones were once exotic and now both teach and distract us daily, augmented reality is coming for how we interact with the digital and physical worlds.
Apple’s forthcoming headset, which the company is expected to reveal at its developers’ conference starting June 5, is likely to accelerate the adoption of augmented reality—and the broader phenomenon of “mixed reality,” which encompasses full-immersion virtual-reality headsets that completely occlude a person’s vision.
Apple AAPL 1.41%increase; green up pointing triangle has a chance to succeed where others have failed. Microsoft, a pioneer of augmented reality and maker of the impressive, but limited, headset I wore that day, is a good example of why it hasn’t caught on yet. Despite an early lead in the field—the first HoloLens shipped in 2016—the company largely failed to make a compelling case for its technology, and has seen an exodus of talent from the team responsible for it. A similar fate befell Google Glass, which Google stopped selling earlier this year.
The key to Apple’s success—or failure—will be its ability to mobilize not just its fanatical, and growing, fan base, but also an army of developers, to make apps and services for its headset the way they have for its other devices.
Motivating those developers will require that Apple demonstrate what has so far been lacking for mixed reality: a blueprint for “killer apps” that will be so compelling that early adopters plunk down for this and future Apple headsets. (No small ask given that the device is expected to cost $3,000 or so.)
My experiences with many of the state-of-the-art AR and VR systems that have come out so far suggest that these killer apps will do things that the flat screens of our current devices struggle with. Most important and transformative: They will let us interact, intuitively, with content in three dimensions.
Attendees try out Microsoft’s HoloLens 2 augmented reality headsets at a trade show in February. PHOTO: ANGEL GARCIA/BLOOMBERG NEWS
-------------------
Into the third dimension
Wallace Lages, an assistant professor of computer science at Northeastern University in Boston, has seen mixed-reality headsets go from nearly unwearable behemoths to something more manageable in the past 10 years. When he started working with augmented and virtual reality, in 2014, his first headset weighed almost 3 pounds, and had a heavy cable attached to it. “It was huge!” he says, laughing. (For comparison, Microsoft’s HoloLens 2 weighs about 1.2 pounds, and is self-contained.)
In that same time, these devices have become vastly more capable—if still far from perfect. Their displays have become brighter, more colorful, and bigger, which is essential to making use of the enormous field of view the human eye possesses.
The best way to use all these extra pixels and processor cycles, Lages says his own research has revealed, is to make our interactions with computers more intuitive—and simpler.
A key way to do that is to take advantage of something that our current computing devices are generally very poor at: our spatial cognition. We see and think in three dimensions, we can build vast maps in our heads, and when given the opportunity, we will use our hands and bodies to manipulate objects and move through spaces in ways that aid us in solving problems and recalling information.
Here’s a mundane but potentially important example of how humans use all those capabilities in real life, but are unable to with current computer interfaces: Imagine if we could organize our documents not just in lists and folders and two-dimensional digital desktops, but in the physical space around us.
Those of us old enough to remember organizing a project by spreading physical documents out on a desk and the walls of an office know this process can work even better—not just for organization but also collaboration.
Apple is expected to announce a mixed-reality headset in 2023. When the company released the iPod and Apple Watch those markets ballooned. Will the AR/VR headset market do the same? WSJ’s Joanna Stern looks back and explains by inflating giant balloons. Photo illustration: Annie Zhao for The Wall Street Journal
Organizing a file system spatially is just one idea, of course—there’s every reason to believe that, just as humans have learned to use charts, graphs, spreadsheets and countless other systems for displaying and manipulating information in two dimensions, we might someday feel just as at home accomplishing the same tasks in three.
One example that is already compelling is a type of app that allows a person to create 3-D content for mixed reality, in mixed reality. Currently, creating 3-D content for games and other entertainment requires learning complicated software that translates 3-D models for a two-dimensional screen. But apps that let you sculpt in three dimensions—painting, stretching, and chiseling away at materials of any kind you can imagine—allow creation of 3-D content in a way that’s far more intuitive than using traditional modeling tools.
Game on
The first major augmented-reality hit was a 2016 game called Pokémon Go, in which players wander the real world with their phones out, collecting pet monsters. It showed that there can be an appetite for gaming using this technology, but the fact that in all the years since, there has yet to be another AR game that matched its success suggests future AR games might be quite different from walking through your neighborhood trying to “ catch ‘em all.”
The augmented-reality game ‘Pokémon Go’ by Nintendo requires players to wander the real world with their phones out, collecting pet monsters. PHOTO: SAM MIRCOVICH/REUTERS
------------------------
Something that doesn’t require gamers to actually go outside might do better. In the first six weeks since its launch in March, the Sony PlayStation VR2 headset sold around 600,000 units, the company revealed in a recent presentation to investors. That’s a tiny fraction of the number of iPhones sold in the same period, but it indicates that games are a place where there is genuine interest in mixed-reality devices. There are countless examples of AR games already available in Apple’s app store that can be played on iPhones, evidence of Apple’s yearslong commitment to educating both its users and developers about how this technology works. While some, like the Lego AR Studio, are well-designed, almost all of them still feel like tech demos, not fully-formed products.
It’s clear that for AR games to take off, a fully immersive, headset-based experience that frees the player’s hands to interact with the world—like Sony’s VR system—will be required.
A tall orderBeyond 3-D interfaces for apps we already use on our existing devices, mixed-reality systems have the potential to help us better interact with information that is already inherently three dimensional.
For example, when we navigate with a map, we have to translate from the real world to the map, and back. This is tolerable for people who have grown accustomed to it, but there is a more intuitive way: What if we could simply overlay arrows telling us which way to go on the real world? Since 2019, Google Maps Live View has made this possible on our phones, and it’s indisputably handy when trying to walk around an unfamiliar city.
Making augmented-reality navigation work on a headset, versus a phone, requires that the headset be something comfortable, safe and stylish enough that people aren’t afraid to wear it on their face, in public. Accomplishing that while also giving a device a powerful enough processor and a good enough display is a tall order, says Ari Grobman, chief executive of Lumus. His company has spent 22 years developing components he says will someday enable just such a pair of smart glasses.
A prototype of augmented-reality glasses from Lumus. PHOTO: LUMUS
-----------------------------
Apple’s forthcoming headset seems to be the opposite of Grobman’s vision of smart glasses—apparently more like a traditional VR headset, and resembling ski goggles. Even if users aren’t at first comfortable walking around outside in Apple’s headset, there is plenty of inherently 3-D information worth consuming on such a device, indoors.
Andrew Garnar-Wortzel, a principal at architecture firm Gensler, has used systems like The Wild, which allows architects and their clients to experience an in-progress building in virtual reality. Doing so can enhance the process of designing and tweaking such a building, by making it more real than simply looking at blueprints or renderings on a flat screen. “While in the middle of designing with clients, we could meet in the space we were designing,” says Garnar-Wortzel.
Apple’s challenge
For now, Apple appears to be opting for a more maximalist approach than many of its competitors in augmented reality, releasing a headset that resembles today’s VR headsets—such as those made by Meta and Sony. This allows for more computing power in the headset, and more capabilities, but such a design sacrifices stylishness, comfort and wearability, and is a far cry from lightweight “smart glasses.”
In some ways, this form factor could be a mistake.
Tamir Berliner is a former executive at augmented-reality startup Magic Leap, and now the chief executive of Sightful, a company that just unveiled a laptop whose only display is a pair of lightweight augmented-reality glasses.
Sightful laptop where the only display is an AR headset.SIGHTFUL
-----------------------------------
One of the first things he discovered when the company began testing its product with focus groups was that if he gave testers VR-style displays, they automatically excluded about half the people who might otherwise use the company’s device. People with long hair generally don’t like putting bulky straps on their heads, says Berliner. And anyone wearing makeup is disinclined to put on something that resembles ski goggles, which will press into their face.
All the smarts and capabilities Apple can cram into its headset won’t matter if people don’t feel comfortable wearing it. Apple certainly has to start somewhere, in terms of getting developers accustomed to what will likely be just the first device of many, and getting consumers excited about its potential.
What remains to be seen is whether the capabilities of this device, and the potential of 3-D content and 3-D interfaces, will overcome the issues that have plagued so many other tech giants that have tried and failed to make face-worn computers go mainstream.
Write to Christopher Mims at christopher.mims@wsj.com
Copyright ©2023 Dow Jones & Company, Inc. All Rights Reserved.
Appeared in the May 27, 2023, print edition as 'The ‘Killer Apps’ That Will Make Apple’s Headset a Winner'.
The ‘Killer Apps’ That Could Make Apple’s New Headset a Winner - WSJ |
