Passive haptics based mixed reality system

Two headset views of a map puzzle of German states, each piece labeled in Chinese. In panel a, with passive haptics, the pieces lie on a wooden board and a robotic virtual hand reaches for one. In panel b, with free-hand gestures, the pieces float in an empty virtual room and a cartoon virtual hand grabs one.
Figure 4 from Yang and Weng, VRCAI 2016: the learner's view of the map puzzle with passive haptics (a) and with free-hand gestures (b).

Most virtual reality systems convey touch poorly: controllers vibrate, and force-feedback devices tie the user to a stylus inside a small workspace. Passive haptics takes a cheaper route, using ordinary physical objects as stand-ins that the user touches while the headset shows a virtual counterpart, so the virtual hands and objects must line up closely with the real ones.

The ISMAR paper builds such a system, in which users handle real objects with bare hands at close range, proposes a calibration method for head and hand tracking, and measures the accuracy of each. The VRCAI paper applies it to geography teaching, with laser-cut acrylic puzzle pieces tracked by markers, and compares learning with passive haptics, with free-hand gestures in VR and with a real puzzle. A later room-scale version tracked the objects and the user’s head with PhaseSpace optical tracking and the arms and hands with Perception Neuron sensors.

Main results

  • In a study with 18 participants learning the German states through a map puzzle, quiz accuracy was highest with passive haptics (mean 95%), against 84% both for a real puzzle and for free-hand gestures in VR.
  • Completing the puzzle took longest with passive haptics (mean 769.4 s, against 465.3 s for the real puzzle and 350.3 s for free-hand gestures), mainly because of the camera's narrow field of view and hands hiding the markers.
  • Asked which virtual setup they would choose for learning if tracking were reliable, 15 of the 18 participants chose passive haptics.