{"id":1025,"date":"2010-08-24T23:09:43","date_gmt":"2010-08-24T22:09:43","guid":{"rendered":"http:\/\/cb.nowan.net\/blog\/?p=1025"},"modified":"2010-08-24T23:09:43","modified_gmt":"2010-08-24T22:09:43","slug":"normalizing-vr-systems-and-applications","status":"publish","type":"post","link":"http:\/\/cb.nowan.net\/blog\/2010\/08\/24\/normalizing-vr-systems-and-applications\/","title":{"rendered":"Normalizing VR systems and applications"},"content":{"rendered":"

I’m still processing this article from Mel Slater<\/a>, where he also defines Place and Plausibility illusions<\/a>.\u00c2\u00a0It has a lot of fundamental ideas in it, so let’s now talk about “sensorimotor contingencies” (SC) and Valid Actions and how they could help normalize VR systems and applications.<\/p>\n

Bear with me, it’s long but interesting, and it’s a good warmup for the upcoming year \ud83d\ude42<\/p>\n

Again complicated names for a simple idea : Sensorimotor contingencies and Valid Actions represent the set of possible actions and perceptions of your physical<\/strong> VR system.<\/p>\n

Valid Actions<\/strong><\/p>\n

Immersive systems can be characterised by the sensorimotor contingencies (SC) that they support.<\/strong> SCs refer to the actions that we know to carry out in order to perceive, for example, moving your head and eyes to change gaze direction, or bending down and shifting head and gaze direction in order to see underneath something.<\/p>\n

The SCs supported by a system define a set of\u00c2\u00a0valid actions that are meaningful in terms of perception<\/strong> within the virtual environment\u00c2\u00a0depicted. For example, turn your head or bend forward and the rendered visual images\u00c2\u00a0ideally change the same as they would if you were in an equivalent physical environment.\u00c2\u00a0If head tracking was not enabled, then turning your head would have no effect, and\u00c2\u00a0therefore such an action could not be useful for perception. We define the set of Valid\u00c2\u00a0Sensorimotor Actions <\/strong>with respect to a given IVR system to be those actions that\u00c2\u00a0consistently result in changes to images<\/strong> [Cb: in the sense of ‘perceptive images’] (in all sensory modalities) so that perception may\u00c2\u00a0be changed meaningfully<\/strong>.<\/p><\/blockquote>\n

We define the set of Valid Effectual Actions<\/strong> as those actions\u00c2\u00a0that the participant can take in order to effect changes in the environment<\/strong>. We call the\u00c2\u00a0union of these two sets the set of Valid Actions \u00e2\u20ac\u201c the actions that a participant can take\u00c2\u00a0that can result in changes in perception, or changes to the environment<\/em><\/strong>.<\/p>\n

For example, consider an environment displayed visually through a head-tracked HMD.<\/div>\n
A participant in such an environment can usually quickly learn the effect of head<\/div>\n
movements on visual perception \u00e2\u20ac\u201c the SCs. Such head movements will be Valid<\/div>\n
Sensorimotor Actions. However, suppose the participant reaches out to touch a virtual<\/div>\n
object, but feels nothing because there is no haptics in this system. Here, the reaching out<\/div>\n
to touch something is not a valid sensorimotor action for this IVR.<\/div>\n

For example, consider an environment displayed visually through a head-tracked HMD.\u00c2\u00a0A participant in such an environment can usually quickly learn the effect of head\u00c2\u00a0movements on visual perception \u00e2\u20ac\u201c the SCs. Such head movements will be Valid\u00c2\u00a0Sensorimotor Actions. However, suppose the participant reaches out to touch a virtual\u00c2\u00a0object, but feels nothing because there is no haptics in this system. Here, the reaching out\u00c2\u00a0to touch something is not a valid sensorimotor action for this IVR.<\/p>\n

Now imagine an\u00c2\u00a0environment displayed visually on a large back-projected screen \u00e2\u20ac\u201c again with headtracking.\u00c2\u00a0However, now when the participant looks far enough to one side visual\u00c2\u00a0elements from the surrounding real world would intrude into the field of view. Actions\u00c2\u00a0that result in perception from outside of the virtual environment are also not valid\u00c2\u00a0sensorimotor actions<\/strong>.<\/p><\/blockquote>\n

This is also why HMDs may result in less breaks in presence since, if the outside world is blocked, well you won’t see the outside world ! Which happens a bit too often in caves unless you’re in a 6-sided cave.<\/p>\n

Please note that we’re only talking about the physical actions of the user<\/strong>, and not what happens in the virtual environment.<\/p>\n

Defining your system<\/strong><\/p>\n

You might think that all those definitions are useless. \u00c2\u00a0It looks like simply putting words on things you already intuitively know.<\/p>\n

But actually they’re the root of the definition of a VR system !<\/p>\n

When you’re talking about the displays and devices of your system, you’re describing the technical solution to the question : “What is possible with my VR system ?”.<\/p>\n

The real answer is the set of Valid Actions as described above. Guess what type of VR system I’m describing :<\/p>\n

– “I can move my head and one hand, and they wil l tracked in a 3x2x2m box with a resolution of 0.1cm”<\/p>\n

– “If I look forward, to the right, to the left and down, I will see the virtual environment”<\/p>\n

– “My field of view is always at least 60\u00c2\u00b0”<\/p>\n

So, what system fulfills those Valid Actions ?<\/p>\n

A typical cave with 4 faces is a good candidate… but a (good) HMD based setup will also work !\u00c2\u00a0You can see now that an application should provide exactly the same experience on different systems as long as those systems provide the same Valid Actions!<\/p>\n

Different VR systems are equivalent if they allow the same set of Valid Actions.\u00c2\u00a0Right now, Idon’t think any Cave is equivalent to a HMD based system, and vice versa, but one day it could be!<\/p>\n

Those definitions can also be used to classify VR systems :<\/p>\n

In this view therefore, we describe immersion<\/strong> not by displays plus tracking, but as a\u00c2\u00a0property of the valid actions that are possible within the system<\/strong>. Generally, system A is at\u00c2\u00a0a higher level of immersion<\/strong> than system B if the valid actions of B form a proper subset\u00c2\u00a0of those of A.<\/p><\/blockquote>\n

Now, every aspect of your VR system is linked to the Valid Actions you want to get :<\/p>\n

In this framework displays and interactive capabilities are inseparable<\/strong>. Consider for\u00c2\u00a0example the issue of display resolution. At first sight this may appear to have nothing to\u00c2\u00a0do with \u00c2\u00a0interaction or SCs, but in fact if the participant wants to examine an object very\u00c2\u00a0closely, then the extent to which this is possible will be limited by the resolution of the\u00c2\u00a0display. Relatively low visual display resolution will mean that the normal action of\u00c2\u00a0bringing an object closer in vision by moving the body, head and eyes closer to it, will\u00c2\u00a0fail earlier than it would in physical reality, and at different times in different systems.<\/p><\/blockquote>\n

That’s why, rather than seeing the technical specifications of a VR system (it has 4 faces, 2x2x2m, 1 hd projector per face and optical tracking), we should instead have the set of Valid Actions that are possible in it : the head and hand are tracked in a 2x2x2m area with 0.1cm precision and 120hz update, the horizontal fov is 200\u00c2\u00b0, vertical is 90\u00c2\u00b0 up, 100\u00c2\u00b0 down, when standing at the origin, the pixel angular resolution is …<\/p>\n

What particular hardware is used to achieve that doesn’t really matter.\u00c2\u00a0We still have to define the set of existing Valid Actions, but I think with current systems the set will be pretty small !<\/p>\n

This framework also has the advantage of putting the user at the center of the VR system<\/strong>. Everything is defined with respect to the user’s perception and actions. This is of course how systems are designed, but those requirements are never printed in press releases or websites.<\/p>\n

Portability<\/strong><\/p>\n

Now suppose you want to write a VR application that can run on many VR systems. How do you define the set of valid systems for your application and how can the application know about those requirements ?<\/p>\n

Some VR software offer virtual devices that allow your application to be abstracted from the physical device that will be used. Simply instantiate a compatible device at runtime and your application will work. But if you only define a 3d tracker, your application won’t know the available range of tracking. If the application knew that when the head is 20 cm near the boundary of tracking (or 20cm to a wall), it could warn the user, or trick him into moving in another direction<\/a>.<\/p>\n

More advanced software will provide a user abstraction. This way, your application only relies on this virtual user’s head and hand position, plus a virtual device for interaction. How each part of this virtual user is actually moved depends on the physical system. This is a step forward compared to only virtual devices since it’s formally putting the user back in the loop, and provides more abstraction (and more standardized) than virtual devices only.<\/p>\n

Defining the compatible VR systems for you application<\/strong><\/p>\n

All this is only one part of the solution.<\/p>\n

Suppose you’re writing a VR tennis game. Your game will only work if you have a minimum volume of tracked area, and if your hand tracking device is able to acurately record hand movement when doing a very fast movement to hit the ball.<\/p>\n

Or you want to create a VR climbing game. You will need to be able to track the user’s hand to at least 2m high and probably want to have a way for the user to look down to feel the height.<\/p>\n

You’re simply defining the set of Valid Actions that your application requires to function correctly.<\/p>\n

And any VR system that offers all those Valid Actions will be compatible with your application.<\/p>\n

Now how you define those Valid Actions for applications and systems? How do you match them ? How will this impact the creation of an application? Will it be simpler or more complicated to create a VR application?<\/p>\n

Anyway it should improve application portability and overall user experience by making clear the minimum requirements needed by the application.<\/p>\n

This is a nice research topic and is left as an exercise to the reader \ud83d\ude09<\/p>\n","protected":false},"excerpt":{"rendered":"

I’m still processing this article from Mel Slater, where he also defines Place and Plausibility illusions.\u00c2\u00a0It has a lot of fundamental ideas in it, so let’s now talk about “sensorimotor contingencies” (SC) and Valid Actions and how they… <\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[9],"tags":[435,116],"_links":{"self":[{"href":"http:\/\/cb.nowan.net\/blog\/wp-json\/wp\/v2\/posts\/1025"}],"collection":[{"href":"http:\/\/cb.nowan.net\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/cb.nowan.net\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/cb.nowan.net\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/cb.nowan.net\/blog\/wp-json\/wp\/v2\/comments?post=1025"}],"version-history":[{"count":4,"href":"http:\/\/cb.nowan.net\/blog\/wp-json\/wp\/v2\/posts\/1025\/revisions"}],"predecessor-version":[{"id":1107,"href":"http:\/\/cb.nowan.net\/blog\/wp-json\/wp\/v2\/posts\/1025\/revisions\/1107"}],"wp:attachment":[{"href":"http:\/\/cb.nowan.net\/blog\/wp-json\/wp\/v2\/media?parent=1025"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/cb.nowan.net\/blog\/wp-json\/wp\/v2\/categories?post=1025"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/cb.nowan.net\/blog\/wp-json\/wp\/v2\/tags?post=1025"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}