Virtual Reality Training Environment for Surgical Operations

An innovative combination of tissue simulation, an authoring tool for creating custom learning content, and special haptic arms will make this possible.

The long and complex training of surgeons is intended to prepare them as best as possible for the successful performance of complex procedures such as bone operations. With the help of virtual reality (VR), it is possible to realistically depict surgical situations and test training with virtual patients free of risk. However, current methods still have their limits in practice, as there is no haptic feedback, i.e. noticeable feel via the finger or hand. The aim of the VIRTOSHA project, which is carried out by the consortium consisting of the University Hospital Bonn (UKB), the TH Köln, MindPort GmbH as consortium leader and Haption GmbH, is to develop a VR training environment for realistic perception during surgical operations.

(from left): Anna Jansen, David Lähner, Markus Wellmann, Kristoffer Waldow, Prof. Dr. Arnulph Fuhrmann, Dr. Turhan Civelek, Prof. Dr. Björn Krüger, Dr. Jérôme Perret, PD Dr. Kristian Welle, Julian Marques. Photo: University Hospital Bonn (UKB)/PD Dr. Welle

“During their training, future surgeons need to experience what operations feel like and train their fine motor skills. Therefore, we are developing a new VR training tool that enables lifelike interaction with the tools. We are designing a drilling and screwing simulation with haptic feedback robotic arms and will implement the characteristics of different types of tissue. In this way, the resistance when drilling and screwing into the bones can be replicated and interventions can be trained realistically,” says PD Dr. Kristian Welle, senior physician at the University Hospital Bonn (UKB), who is responsible for the project at the UKB together with Prof. Björn Krüger.

Simulating Different Patients

By analyzing real operations, the steps and options of a procedure, the behavior of tissue, implants and surgical tools, as well as the haptic feedback, i.e. the noticeable feedback via the finger or hand, are recorded. Various processes and avatars, i.e. virtual representations of people, can be combined to create new training scenarios and train the collaboration of interdisciplinary surgical participants.

“We use software to design different scenarios where tissue and bone properties can be adjusted to simulate different patients. A realistic representation of the tissues and bones as well as the precise integration of hand movements are crucial for the real feeling in the VR environment,” says Prof. Björn Krüger, head of the “Personalized Digital Health and Telemedicine” research group at the UKB’s Epileptology Department.

The team at TH Köln is developing the AI-based simulation system with which users can realistically simulate interventions in human hard and soft tissue such as bones, periosteum and muscle tissue during training. The researchers also calculate when the surgical tools hit the simulated tissue in order to optimize the force feedback of the haptic arms. “We are bringing our many years of expertise and know-how in the application of haptic technologies and the real-time simulation of deformable materials to the project,” reports Prof. Dr. Arnulph Fuhrmann from the Institute of Media and Imaging Technology at the TH Köln.

Authoring System for Wide Applications

“In the past, attempts have been made to train surgical procedures using virtual reality,” says David Lähner, founder and CEO of MindPort. “We have developed a new approach to this goal: Using an integrated authoring tool, surgeons will be able to create new content and update existing content themselves. This allows many more different cases to be covered. Our long-term vision is that a ‘digital twin’ of the patient can be used to practice in virtuality before the real operation begins.”

“This approach is not only exciting for the training of our new colleagues, but will also greatly simplify the introduction of new implants,” adds PD Dr. Kristian Welle.

About the Project

The research project VIRTOSHA (author’s tool for virtual reality training simulations of osteosynthesis with haptic feedback and tissue simulation) involves MindPort GmbH as consortium leader, the University Hospital Bonn (UKB), the TH Köln and Haption GmbH. The project is funded with 1.5 million euros as part of the Gesünder.IN.NRW innovation competition. The project runs for three years.

Press contact:
Viola Röser
Press Officer at the University Hospital Bonn (UKB)
Communications and Media Office at the University Hospital Bonn
Phone +49 228 287- 10469
E-mail: viola.roeser@ukbonn.de

The Consortium Partners

At the University Hospital Bonn (UKB), around 500,000 patient treatments take place every year, there are around 9,500 employees and the balance sheet total is 1.8 billion euros. In addition to the 3,500 medical and dental students, 550 people are trained in numerous health professions every year. The UKB is in first place in the Focus clinic list among the university hospitals (UK) in North Rhine-Westphalia and has the second highest case mix index (case severity) in Germany. The F.A.Z. Institute has recognized the UKB as Germany’s most sought-after employer and training champion among public hospitals in Germany in 2022 and 2023.

The TH Köln – University of Applied Sciences offers students and scientists from Germany and abroad an inspiring environment to learn, work and research in the social, cultural, engineering and natural sciences. Currently there are 23,500 students enrolled in about 100 undergraduate and graduate programs. The TH Köln creates social innovation – this is our main idea as we approach the challenges of today’s society. Our interdisciplinary thinking and acting, as well as our regional, national, and international activities, have made us both a valued partner and a trailblazer in many areas. The TH Köln was founded as the Fachhochschule Köln in 1971 and is regarded as one of the most innovative universities of applied sciences.

MindPort GmbH is a tech startup founded in 2021. With its software product VR Builder, it provides enterprises with the capabilities to independently create high-quality VR applications. Thanks to an open source software core and no cloud dependency, it is particularly flexible and secure. MindPort’s innovative technology has already received international recognition and is used successfully worldwide and in various industries.

Haption Group develops, produces and sells hardware and software solutions based on force feedback. Founded in 2001, the company partners with the most advanced leaders to provide professional force feedback solutions for medical, robotics, nuclear and industrial applications.

https://www.ukbnewsroom.de/wp-content/uploads/2024/08/240814_PM_Projekt-VIRTOSHA_ENG.pdf