HOW AR AND VR TECHNOLOGIES MIGHT CHANGE THE WORLD AS WE SEE IT NOW?

Who else can tell us more about AR and VR than someone with over 20 years practice in the field? Ville Miettinen has a solid experience in computer graphics and vision industries. For the last 25 years, he was founding, building and exiting graphics technology companies in the Mummy Valley a.k.a. Finland. Some of them, such as  Hybrid Graphics and drawElements, received global success and were acquired by NVIDIA and Google respectively. Currently, Ville works as a Principal Algorithm Developer and Co-Founder at Varjo. During this exciting speech at BUILD stage at EMERGE 2019, Ville talked about differences in AR and VR and supported them with demo examples from Varjo.

The other type is wearable AR, where you have heavy headsets which are typically untethered, as you don't have a cable going. In wearable AR there is a graphics process running somewhere on the headset and it's able to display graphics on top of the real world. However, in wearable AR there are some fundamental challenges mainly because displays are very limited to a narrow field-of-view. Whatever objects you're seeing, there're just in the middle of your vision. And typically the waveguide technology darkens the real world quite a bit.

What's Mixed Reality? The XR term is not very well-defined, but the idea is that the next generation of Augmented Reality, where we pick the good parts of VR and the good parts of AR, will represent XR. The end goal of mixed reality is to create an experience where you cannot distinguish any difference between the real and virtual worlds. Computer vision developers are able to take virtual objects, virtual experiences, and combine them seamlessly with the real world. Of course, this is an old truck complex problem, because in order to do that properly, what you need to do is to 3D capture everything that surrounds you. Also, you have to figure out the entire light transportation problem, where light is coming, what's hitting and what's bouncing. All these can help you to reconstruct what's happening between virtual and real worlds. Eventually, what we want to have is a virtual object, which will be reflecting the real world and vice versa. Virtual objects are casting shadows on a real-world surface, and these virtual light sources will be lighting the real-world environments on so forth.

It's worth highlighting that the whole AR/VR/XR industry is completely driven by the consumer market, as the main applications are games.

There are three problems that computer graphics coders are facing while working with AR technology.

1. When working with AR, we need to deal with display resolutions. The existing headsets have fairly low display resolutions. If you are rendering something virtual there, it's not going to look very real.
2. The existing AR technologies have also such problem like the narrow field-of-view, because of that, all the objects are located only in the center of the view.
3. Objects displayed in AR are hazy. Due to the specificity of AR technology of adding and image, objects don't look realistic.

However, there are solutions to tackle the above listed issues. Jason Paul, who runs VR strategy at NVIDIA, says that it would take us about 20 years to reach human-eye resolutions. Then the question is how to get there. When we are talking about reaching human-eye resolutions, the following features should be taken into consideration. The human eye doesn't have a fixed resolution, it varies. But if it were a single resolution it would be roughly a 100 megapixel. It seems that we can just build a 100-megapixel display and problem solved. But it's not.

First of all, the highest resolution displays in existence, in laboratory conditions, are about 16 megapixels. Also, building high-res displays is very expensive. A couple of big display manufacturers in the AR/VR land, such as Microsoft and Magic Leap, both spent more than a billion dollars on the display technology. All the other big boys and small players like Varjo, they just take displays created for mobile phones, which are big enough to be used in a creative way. However, in the mobile phone world, the maximum display resolution has already been reached. Now you can watch Full HD videos on mobile phones, thus, there are no real drivers to bump the resolution.

Also, virtual reality needs much higher frame rates, and, unfortunately, mobile phone displays are not very suitable for that. Let's imagine for a moment that someone proposed us to build a mega display with 100 megapixels which we can put in our headsets. But again, there is an issue. We still need to render 100 megapixels per frame, and that's not going to happen, even with the latest generation display. Most of those pixels, that would be rendered, they would go to a peripheral vision where you cannot really see anything. It means that the bigger the field-of-view is, the more pixels will be spread beyond your eyesight.

It's possible to do variable rate rendering, meaning that we are not rendering at the same rate in all directions, but we only focus on the areas where a human eye can actually see better. For example, if you put your two fingers up in front of you, that's your foveal vision and an area where you can actually see accurately. When you are moving your hand to different sides, it's your peripheral vision where you can still see, but things are not sharp anymore and you just can detect motions happening there, but cannot actually see any details. There are rendering algorithms that are called foveated rendering, which try to imitate this.

Let's say someone builds us a mega display with nice foveated rendering technology, and, finally, we got our human-eye resolution. But it doesn't work properly that way. The problem is with pixels that consume power. The power consumption is a massive issue in anything that you're wearing. All the headsets are using several watts of power and these plates are just in front of your eyes. Even the current generation displays are unbearably hot.

Speaking about the VR industry, it's not working the classical way. Whenever new technology brought to the market, it first comes to the high-end pro user land where the initial device will be tested with prosumers, and, after that, the product gets simplified, became easier to use, much cheaper for ordinary consumers. In the VR market, it goes the other way round. Back in decades, they started with consumers and fairly crappy products, and now the VR industry is trying to quickly fix itself by pushing to consumers products that are not technologically up there. Ville claimed that at Varjo they started with in a traditional way with pro products for the ultra-high-end users and hopefully quite soon they will introduce headsets to the consumer market.

When we're talking about AR/VR/XR, there is a place for disruption. For Ville, disruption is about moving a decimal point. Any decimal point, if you move it to the left or to the right, for example, you cut the price by 10X, you make something 10X faster and 10X more beautiful, means that you have proper disruption. What computer vision developers and graphics designers should do is to disrupt resolution. At Varjo they built a foveal screen putting two displays per eye. One is a small micro-LED display and another one like background display. These two displays overlay on top of each other using an optical combiner. That sounds like a really simple idea, but the devil is in the details. When you have two different devices from different manufacturers running at different resolutions, combining those images seamlessly is a pretty nasty engineering challenge, as you need to blur them properly. But once you've managed to do that, the end result will show you a display that roughly matches the resolution of the human eye.

In order to push forward their technology, they decided to build their own custom eye-tracking solution at Varjo. They placed high speed infrared cameras inside the headset to track eyes moving. Such eye-tracking technology gives an accuracy of 0.2−0.3 degrees.

Varjo has been building demos and use cases with eye-tracking for implementing further features in AR/VR. It turned out that eye-tracking is not only useful in UI and UX, but it's very useful in lots of different rendering algorithms, for example, computing the levels of detail of textures or geometry, procedural rendering, and streaming of 3D content. There's a prediction that eye tracking will become a standard component in AR/VR technology.

Ville summarized his talk with the presentation of Varjo's products they have been working on over the last years. The first headset they produced was VR-2, which was launched in February 2019 and now it's available for online purchase.

Later on in May 2019 they launched their second product, which is called XR-1. It's the same headset that solves the resolution problem, but it also has added cameras in the front. In this headset, there are cameras that can take images, which can be manipulated at will. In order to have an idea how XR-1 works, we suggest to watch the video below.

EMERGE team hopes you found this article insightful.
Stay tuned with us for further updates!