Apple and Facebook mom Meta are expected to release mixed reality headsets in the coming year that can finally deliver on the industry’s promise to turn head-worn devices into the next big shift in personal computing.
But there is one major potential problem: sticker shock.
The best-selling virtual reality headset, the Meta Quest 2, costs $400 and accounted for 78% of the emerging VR market by 2021, according to IDC. Consumers who want the next-generation technology will have to spend multiples of that.
Meta’s upcoming high-end headset, codenamed Cambria, is expected to cost at least $800, the company said earlier this year. Apple’s unannounced device could reportedly cost thousands of dollars. That is a significant burden for products in a category that has yet to become mainstream. Just 11.2 million VR units were shipped last year, IDC said. Apple sells so many iPhones every few weeks.
To expand the market, Meta and Apple will have to convince consumers that more advanced systems will be worth the investment. Both companies are reportedly betting on a new technology called passthrough mixed reality, which requires better displays and more processing power.
If mixed reality passthrough works as advertised, a VR headset would also function as a set of augmented reality glasses, enhancing the possibilities for real-world applications and use.
With existing VR devices, the experience is limited to what’s on the headset’s display. In passthrough AR, powerful cameras on the outside of a VR headset capture video of the outside world and send it to two or more screens, one in front of the user’s eyes.
This allows developers to play with mixed reality, software overlapping or images on the video of the real world from just outside.
Mixed reality believers say we will eventually be able to condense the technology into lightweight glasses with clear lenses. But that’s for the future.
The pass-through approach is emerging as the preferred option in the near term, as optical transparent displays are far from ready for primetime. The problem for today is that mixed reality passthrough requires a lot of expensive parts and a powerful headset, limiting the size of the market.
In addition to the advanced cameras, pass-through devices require depth sensors that can take detailed videos and measurements of the user’s environment. They should also follow the user’s eyes so as not to waste energy on graphics that go unseen. And they need powerful processing capabilities and software to reduce latency so that what the user sees in the headset isn’t slowed down or blurry.
Most importantly, the high-resolution screen has to be much closer than a smartphone’s screen because it’s so close to the user’s eyes. Smartphone screens average about 550 pixels per inch, but mixed reality devices require screens at about 3,500 PPI, according to CounterPoint Research.
While Meta and Apple have not released their headsets, a few devices currently on the market support mixed reality passthrough. The experiences are usually limited – black and white or low quality video – due to a lack of processing power.
A few weeks ago, I was able to test a headset from Varjo, a Finnish company co-founded by Urho Konttori, a former director of Microsoft and Nokia. Last year, Varjo released the XR-3, which offers full-color, low-latency passthrough mixed reality. It is expensive, heavy and aimed at businesses. It costs $6,495 to buy it or about $1,500 to rent it for a year.
While playing with the XR-3, I felt less isolated than with other VR headsets.
Varjo’s XR-3 headset
I could access a virtual world with the push of a single button and bring up games that took over my entire field of view. I could use virtual computer monitors that display Windows applications in the virtual world.
I was also able to interact with the world around me through Varjo’s passthrough view. In the demo, Varjo placed a life-size car model in space. I was able to walk around it and inspect the interior and discuss what I saw with someone who wasn’t wearing a VR headset.
Most impressively, when passthrough was enabled, I was able to interact with the actual environment around me, have a conversation with the person next to me or find a seat and sit in it. That is not possible with existing VR technology, which forces you to remove yourself from the physical world.
Konttori told me that was one of his main goals. The company wants to almost mimic the display quality of the human eye, which it calls the “holy grail” of mixed reality.
‘A single coherent scene’
The XR-3 has two screens measuring 2880 by 2720 pixels, and the company uses eye tracking to focus its processing power to deliver better image quality where your eyes look.
The key is “being able to merge the physical reality around you with the virtual reality objects and make it a single coherent scene, where you can no longer distinguish what is real and what is virtual,” Konttori said. “Part of this evolution is that you can see that at some point the fidelity of this experience is equal to what you would perceive by looking at it with your own eyes.”
To use the XR-3, however, you must be wired to a high-performance gaming PC. Meta and Apple are focused on developing devices that don’t need to be connected to a separate computer. Konttori knows it will be difficult for his startup to compete with some of the largest tech companies in the world, but he says Meta and Apple still face challenges.
That’s because developing an easy-to-use product with the right weight and power consumption is very tricky, especially when it comes to keeping costs down and sending millions.
“Companies are focused on consumer experiences, which means they are still really driven by size, weight, ergonomics and cost,” Konttori said.
A participant wears an HTC Corp. Vive virtual reality (VR) headset at the Apple Worldwide Developers Conference (WWDC) in San Jose, California, USA, on Monday, June 5, 2017.
David Paul Morris | Bloomberg | Getty Images
Apple is notoriously secretive about its product roadmap, especially when it comes to new categories. The company has invested heavily in virtual reality research and development in its Technology Development Group and has acquired several startups specializing in mixed reality technology.
According to reports from Bloomberg and The Information, Apple is developing a mixed reality headset that resembles ski goggles with a powerful homegrown chip, similar to what powers its MacBook laptops, and screens with a higher resolution than what’s currently on the market. .
The headset reportedly supports passthrough video and offers games and other applications. At one point, Apple aimed for a resolution comparable to that of a 4K TV per eye for its first headset, because slightly less can cause users to see individual pixels, The Information reported.
Apple has not confirmed its plans to release a mixed reality headset and the company has not responded to a request for comment on this story. In an interview with Chinese media earlier this year, Apple CEO Tim Cook suggested something is in the works.
Meta has said that Project Cambria, with color passthrough support, will be released later this year. Based on public renderings of the device, it also resembles ski goggles. It includes pancake optics, a type of lens that doesn’t require as fine calibration as other VR lenses.
Meta said in May that the price for Cambria would be “significantly higher” than $800.
While passthrough technology has yet to hit the market in a real way and will be quite pricey if it does, metaverse developers are rallying behind it. The primary alternative, optically based mixed reality, uses transparent screens built into lenses to integrate computer graphics with the real world. Microsoft’s Hololens and Magic Leap use optical waveguides, a kind of transparent display.
Transparent displays are also expensive and come with their own challenges. They are not good when used in bright daylight, and current offerings can suffer from poor image quality and blurry text.
Varjo is betting on passthrough technology, and Konttori says it’s the better approach, largely because it’s all-digital, giving developers more control.
“It becomes calculable,” Konttori said. “It becomes a tool for artificial intelligence to participate in your world, improve your vision or your intellect, and you can distort the world in the smallest or the largest possible ways.”
He expects passthrough to be “the winning approach for a very, very long time.”
WATCH: The future of entertainment is mixed reality gaming experiences