Tricking the brain: New dimensions of haptics in virtual reality

How can virtual reality (VR) be experienced haptically, i.e., through the sense of touch? This is one of the fundamental questions that modern VR research is investigating. Computer scientist André Zenner, who is based in Saarbrücken, Germany, has come a significant step closer to answering this question in his doctoral thesis – by inventing new devices and developing software-based techniques inspired by human perception. He has now been awarded the prestigious “Best Dissertation Award” at the world’s leading VR conference.

André Zenner with the tubular controller “Shifty”, in which a movable weight is installed. (c) DFKI /Oliver Dietze

The award-winning work is about how physical props (technical term: “proxies”) can be used to make objects in virtual environments tangible. “Of course, you can’t have a proxy for every virtual object, then the approach wouldn’t be scalable. In my dissertation, I, therefore, thought about what devices could look like that could be used to simulate the physical properties of several different virtual objects as effectively as possible,” explains André Zenner, who completed his doctorate at the Saarbrücken Graduate School of Computer Science at Saarland University and is now conducting research at Saarland University and the German Research Center for Artificial Intelligence.

This resulted in the prototypes for two special VR controllers, “Shifty” and “Drag:on.” VR controllers are devices that can be held in the user’s hand to control or manipulate objects in virtual reality using tracking technology.

“Shifty” is a tubular controller in which a movable weight is installed. The weight can be moved along the lengthwise axis by a motor, changing the center of gravity and inertia of the rod. “In combination with corresponding visualizations in virtual reality, Shifty can be used to create the illusion that a virtual object is getting longer or heavier,” explains André Zenner. In experiments, he was able to prove that objects are perceived as lighter or smaller when the weight is close to the user’s hand and that, coupled with the corresponding visual input; they are perceived as longer and heavier the further the weight in the rod moves away from the user. “This is mainly due to changes in the inertia of the controller, as the overall weight does not change,” explains André Zenner. The research and development department of gaming giant Sony is already experimenting with this concept and cites André Zenner’s work in the development of new VR controllers.

The second controller, “Drag:on,” consists of two flamenco fans that can be unfolded using servomotors, thus increasing the air resistance of the controller. This means that the further the fans are unfolded, the more force the user has to exert to move the controller through the air. “Coupled with the right visual stimuli, Drag:on can be used to create the impression that the user is holding a small shovel or a large paddle, for example, or that they are pushing a heavy trolley or are twisting a knob that is difficult to turn,” explains André Zenner. Both controllers are basic research and so-called “proof of concepts.” This means that the prototypes can be used to show in user experiments that different controller states can improve the perception of different VR objects. Still, specific products using this technology are not yet available on the market.

With the controllers, the Saarbrücken-based computer scientist first addressed the so-called ‘similarity problem.’ The aim here is to ensure that virtual and real objects feel as similar as possible. In the second part of his work, he dealt with the so-called ‘colocation problem,’ i.e., the question of how the proxy can be spatially located in real life where the user sees it in virtual reality. This is particularly challenging as the controllers act as proxies for different virtual objects. Consequently, the user must be given the illusion that they are reaching for various objects, although in reality, they will always grasp the same proxy.

To achieve this, the researcher made use of the already established method of “hand redirection.” As the name suggests, this involves redirecting the movement of the hand in virtual reality so that the user thinks they are reaching to the left, for example, even though they are actually stretching their hand forward. “We conducted experiments to investigate the point at which users realize that their hand has been redirected. Our results showed that this point was reached quickly, so we thought about how we could better conceal the hand redirection,” says André Zenner. The solution: he tricked the brain by only redirecting the hand when the brain was blind to visual changes – namely during blinking. Together with a student under his supervision, he developed the appropriate software and used the eye trackers built into many VR headsets. In control studies, the team was then able to show that their new controllers, in combination with hand redirection algorithms, led to more convincing VR perceptions than previously possible.

This research achievement has now been recognized internationally. The work entitled “Advancing Proxy-Based Haptic Feedback in Virtual Reality” was awarded the “IEEE VGTC Virtual Reality Best Dissertation Award” in mid-March at the “IEEE Conference on Virtual Reality and 3D User Interfaces”, which took place this year in Orlando, Florida. According to the IEEE, the award is presented each year to the author of the most outstanding dissertation defended in the previous two calendar years in the research fields of virtual and augmented reality. A total of 16 dissertations were submitted to the 2024 edition of the conference and reviewed by an international committee of experts. The thesis was supervised by Prof. Dr. Antonio Krüger, CEO of the German Research Center for Artificial Intelligence and Head of the Ubiquitous Media Lab at Saarland University.

Background Saarland Informatics Campus:
900 scientists (including 400 PhD students) and approx. 2500 students from more than 80 nations make the Saarland Informatics Campus (SIC) one of the leading locations for computer science in Germany and Europe. Four world-renowned research institutes, namely the German Research Center for Artificial Intelligence (DFKI), the Max Planck Institute for Informatics, the Max Planck Institute for Software Systems, and the Center for Bioinformatics along with Saarland University and its three departments and 24 degree programs, together cover the entire spectrum of computer science.

Background German Research Center for Artificial Intelligence:
The German Research Center for Artificial Intelligence GmbH (DFKI) was founded in 1988 as a non-profit public-private partnership (PPP). It has locations in Kaiserslautern, Saarbrücken, Bremen, and Lower Saxony, laboratories in Berlin, Darmstadt, and Lübeck, and a branch office in Trier. DFKI combines scientific excellence and business-oriented value creation with social appreciation. DFKI has been researching AI for humans for over 35 years and is oriented towards social relevance and scientific excellence in the key future-oriented research and application areas of artificial intelligence. DFKI is one of the most important “Centers of Excellence” in the international scientific community. Around 1,560 employees from over 76 nations are currently researching innovative software solutions.

Editor:
Philipp Zapf-Schramm
Saarland Informatics Campus
Telefon: +49 681 302-70741
E-Mail: pzapf@cs.uni-saarland.de

Wissenschaftliche Ansprechpartner:

Dr.-Ing. André Zenner
Universität des Saarlandes & Deutsches Forschungszentrum für Künstliche Intelligenz
Tel.: +49 681 85775 5117
E-Mail: Andre.Zenner@dfki.de

Originalpublikation:

Zenner, A. (2022). Advancing Proxy-Based Haptic Feedback in Virtual Reality. Universität des Saarlandes. doi:10.22028/D291-37879

Weitere Informationen:

https://ieeevr.org/vgtc/VR_best_dissertation/index.html

https://www.uni-saarland.de/aktuell/haptik-in-der-virtuellen-realitaet-32286.html

Video „Shifty”

Video „Drag:on”

Video „Blink-Suppressed Hand Redirection“