VoluProf facilitates individual and interactive online lectures
The rotunda is equipped with 32 cameras that film the lecturer from all sides. A lifelike 3D avatar is generated from the video data.
© Fraunhofer HHI
Using AR glasses for online lectures.
Online offers by universities are mostly limited to playing back videos of lecturers’ presentations. There aren’t any opportunities for direct interaction. Scientists at the Fraunhofer Institute for Telecommunications, Heinrich-Hertz-Institut, HHI, want to change that with their VoluProf project. This project involves 32 cameras recording the lecturer’s presentation from all angles. The video data is used to generate a photo-realistic animated avatar that appears lifelike via AR glasses, and can even answer questions.
Almost all universities of applied sciences and colleges also offer their lectures and courses online. Students can access learning content from anywhere at any time, and the number of participants isn’t limited to the size of the lecture hall. Until now, however, this service has been limited to passive viewing of video recordings.
Now, the Fraunhofer Institute for Telecommunications, Heinrich-Hertz-Institut, HHI in Berlin has developed a solution that enables online lectures or courses to be designed in an interactive and customized manner for participants. The lecturer appears in photo-realistic quality as a moving and talking avatar who addresses his or her audience personally, and even responds to questions. “The universities’ online lectures will be taken to a new level in terms of visual, acoustic, and didactic quality,” says Dr. Cornelius Hellge, Head of Multimedia Communications Group and leader of the project.
Photo and audio-realistic avatar
To generate the avatar, the first step is for the presenter to stand in a rotunda. This is equipped with stereo microphones and a total of 32 video cameras. Now, the lecturer can give their talk the way they want, moving and gesticulating as freely as they would like. An animated 3D avatar in photo-realistic quality is created from the video footage of the lecture. Characteristic movements are also included. Since the volumetric aspect of each person’s body is also included, the videos are referred to as “volumetric videos,” hence the name of the project: VoluProf (volumetric professor).
The lecturer then provides the lecture notes in text form. On one hand, this serves as a basis for the audio script, in which the text is reproduced with the lecturer’s voice. On the other hand, the lecture notes serve as a basis for the 3D avatar’s animation, both for the facial expressions that match the text and for the appropriate body movements. The same applies to the voice, which is synchronized with the lip movements.
In order to participate in the virtual courses, all students need is a smartphone and AR (augmented reality) glasses. With the help of the glasses, the animated 3D avatar is superimposed directly into the real-life environment. For the students, it looks as if the lecturer is standing in front of them in the room. In addition, the participants’ position in the room and their line of vision can be detected by the glasses. This information is transmitted via smartphone to the provider’s server, where the avatar’s image is continuously rendered so that it faces the students at all times, appearing to speak directly to them. The avatar’s movements and the reaction to the participants’ input take place almost in real time; the latency is a maximum of 40 ms. “You get the impression that the lecturer is giving a one-to-one lecture to the student in question,” Hellge explains.
Interaction through questions and dialog
In contrast to traditional, passive online lectures, interaction with the lecturer is possible at any time with VoluProf. For example, a student could ask: “Can you repeat that, please?” or “I didn’t understand that.” These kinds of questions are stored as commands in the avatar’s neural network and then trigger either repetitions or a more detailed explanation by the avatar.
However, the students can also ask more specific questions. Since all conceivable questions can be saved in the lecture text in advance, the virtual professor can respond to the respective question. To achieve this, the system’s speech recognition function converts the spoken question into text. An AI-based chatbot then links the question text with the matching answer text, which is spoken by the virtual professor — including gestures, facial expressions, and synchronized lip movements.
Despite the solution’s technical sophistication, it places only low demands on the end user’s equipment. Since computationally intense tasks such as animation and rendering, audio synthesis, or speech recognition take place on the provider’s server, participants only need a standard smartphone that supports at least 4G and a pair of lightweight AR glasses. “We deliberately designed the concept to keep the barrier to entry for students as low as possible,” explains Hellge.
Expertise in graphics, audio, and video codecs
For VoluProf, Fraunhofer HHI has leveraged its long-standing expertise in the areas of computer vision, video work, and machine learning. One team designed the photorealistic representation of people as avatars using volumetric video data. Another team took care of the efficient transmission of video data. Researchers developed a transmission method specifically for this purpose that guarantees low latency, while adapting to different network conditions and enabling smooth motion at reduced resolution even when the connection is poor. Hellge emphasizes the whole project’s innovative approach, which incorporates these technologies: “Photorealistic avatars are already well-known from the movies. But what’s new and unique about VoluProf is that the photo-realistic avatars interact with people and answer questions in real time via online connection.”
Hellge, a Fraunhofer researcher, initiated the project, has taken qualified technology partners on board, and is driving its further development as project manager. Initial trials at the University of Rostock have already taken place. “The feedback has been very positive and there’s a lot of interest in the final implementation,” Hellge is happy to report.
Weitere Informationen:
https://www.fraunhofer.de/en/press/research-news/2023/august-2023/voluprof-facil…
Media Contact
All latest news from the category: Information Technology
Here you can find a summary of innovations in the fields of information and data processing and up-to-date developments on IT equipment and hardware.
This area covers topics such as IT services, IT architectures, IT management and telecommunications.
Newest articles
SpinMagIC: ‘EPR on a chip’ ensures quality of olive oil and beer
Spin-off company offers a tiny solution for a big problem. The first sign of spoilage in many food products is the formation of free radicals, which reduces the shelf-life and…
Impressive results against autoimmune disease
Severe systemic lupus no longer detectable after cancer medication treatment. The team at Charité – Universitätsmedizin Berlin is astounded by the huge improvement seen in a female patient with severe…
UCF researchers develop rapid test to detect dopamine
The sensor could serve as a low-cost and efficient tool for early detection of neurological disorders and conditions, including Parkinson’s disease, Alzheimer’s disease and depression. Dopamine, a neurotransmitter in our…