Sony Files Patent On Letting Observers Dynamically Look Into VR Game Environments

Back in April 2018, Sony Interactive Entertainment filed a patent at the US Patent Office on a process of letting observers of people playing VR dynamically watch what their friends are doing on a second screen, such as a smartphone. The patent is currently still undergoing the approval process, and was published on November 1, 2018.

According to the patent, the video feed data of what the player is doing is sent through a client system, and then displayed on a second screen. The process is doable both locally and through the internet via a cloud gaming service.

So far, it seems like a regular streaming service to a different device, right? Here’s the interesting bit: the process patented also allows the second player or observer to dynamically look at the surroundings of what the first player sees in VR, through the device like a smartphone.

This means that, as the illustrations above show, you can actually gain a look at the environment around the player at different angles and how close you are to the player, sort of like peering through a window, except that you can move the window frame however you want.

But how is this done? It seems that the process will ask for the observing device to use both the forward-facing and selfie camera at once, in order to track both the observer’s head movements and distance from the screen, as well as the observer’s position relative to the VR player. In the theoretical instance of a VR device that also has a forward-facing camera, it may also be used to detect where the observer is in relation to the player.

In another instance, the VR player is seen looking at a man riding a skateboard. Meanwhile, through the screen that is communicating with a computer, the second player is watching the first player’s view. Like the previous examples, you can change the angle and direction of where you look, but instead, a wider view is generated through synchronization via the computer, meaning that you can see generated parts of the scenery even if the first player isn’t looking in that direction. The second player is essentially able to glance around the VR world, although they stay tethered to the first player.

As for how this process can be used in games, at its base, what this allows for is a better way of implementing replays of VR content via the Dualshock 4’s Share button. If the spectator is using a VR headset of their own, they can look through the VR replay from vantage points that the first player wouldn’t have seen. Because the observer is watching through their “window”, you won’t have to follow the original player’s head movements, and potentially get nauseous and dizzy.

Apart from just replays though, the patent provides us with an interesting use of this process above in a first-person shooter. By looking through the “window” into the game world, and being able to process different angles and directions, the observer can help the player look out for obstacles and enemies from different angles, acting as a navigator.

Finally, the process provides a look at how video game streaming for VR content on sites such as Twitch could work. The patent details further examples of how the player’s gaze can be contextualized to the viewers watching via computer monitor with visual indicators, or even change the contrast to focus on what the player is looking at.