How It Works...
The problem with typical stereoptic 3D imaging methods is that they are heavily reliant on binocular vision... and, for that matter, even position and alignment of one's eyes. If you tilt your head off of dead center ever so slightly, the effect suffers. Furthermore, the 3D images you see in these entertainment media are tied to a camera that is not directly tied to the viewer, so even though the 3D view has a sweet spot from which an observer should view it, the 3D image will be the same regardless of the position from which you view it. This can allow for the appropriate unique views for the left and right eye, but doesn't allow for interactive occlusion, the effect of changing your viewpoint a bit and seeing one object getting partially or completely hidden by a closer object. This occlusion is a natural part of how people determine the relative depth of objects.
Given a single viewer, however, there are ways to reproduce this interactive occlusion in an interactive entertainment product, such as a game. By capturing positional information about where the viewer is looking from, the in-game camera can be tied to the viewer's actual viewing position. Then, when you look left, right up or down, the image shifts so that you're peering around a close object to see an object that's further away. The result is that things can appear to be deep inside of a "box" whose glass top is the screen of the NGP.
The first time I saw this effect, it was done using a Wii-mote as a camera positioned on top of a monitor, pointing toward the viewer and a pair of glasses with two L.E.D.s, one mounted on either side, to provide positional information. This creates a very convincing effect, but required that the user wear glasses or a headband of something similar with mounted lights to provide tracking information. By replacing these position tracking elements with facial tracking, the system only requires the built-in camera; no external hardware is required. I would expect to see this used in several Sony first party titles, and, most likely, made into part of a developer toolkit library, to make it an easy-to-drop-in feature available to developers creating games and applications for the NGP.
Citations:
- Sony Patent: Method and Apparatus for Adjusting a View of a Scene Being Displayed According to Tracked Head Motion on Google Patents
- Head Tracking for Desktop VR Displays using the WiiRemote on YouTube (Effect Shown With Wii-mote) - by Johnny Chung Lee, Carnegie Mellon University
