Event camera-based 3D capture for mixed reality
Depth estimation using an event-based camera and a beamer.
© Fraunhofer HHI
VIT team presents paper at CVPR 2023.
A paper by researchers from the Vision and Imaging Technologies (VIT) department of Fraunhofer Heinrich Hertz Institute (HHI) has been accepted at the CVPR 2023 Workshop for Event-based Vision. The world-renowned workshop covers novel event-based cameras and algorithms that are significantly more energy-efficient, light-sensitive, and dynamic than conventional cameras. The paper, “X-maps: Direct Depth Lookup for Event-based Structured Light Systems” examines the use of event-based cameras to minimize latency in mixed reality environments and improve the user experience.
Unlike conventional cameras, event-based cameras do not capture complete images that then have to be processed and stored in a time- and energy-consuming manner. Instead, they generate continuous pixel streams with a temporal resolution of microseconds. This fundamentally different approach to signal generation has several advantages: event-based cameras have a high dynamic range and very low latencies. Because of these characteristics, they unlock opportunities that are unthinkable with conventional cameras (which have been the main sensor technology for the last 60 years).
Due to their low latency, these new sensors are particularly interesting for use in mixed reality technologies (XR), where even minimal delays in image processing can have a strong negative impact on the user experience.
In their paper, Fraunhofer HHI researchers investigate the combination of an event-based camera with a beamer to achieve real-time depth estimation for Spatial Augmented Reality (SAR) applications. SAR is also referred to as projection-based augmented reality and describes the superposition of virtual content on the physical environment by projection. This can create interactive, and immersive experiences in various application areas such as industry, culture, or healthcare. Interactive feedback and high frame rates are essential in this context.
The research team measures the 3D structure of the projected surface with the help of the event camera, adjusting the projection automatically. This process happens so quickly that the human eye can only perceive a coherent image and not a line-by-line appearance of the image. This is why adjusting the projection even in dynamic scenes with mobile elements is possible. Such a technique could also be used outside of XR when using a projector on a curved projection surface, for example, to adjust the projection to the surface in real-time. In addition, the paper presents new techniques for rapidly processing detected objects and events and for calibrating the nonlinear behavior.
Wissenschaftliche Ansprechpartner:
Martina Müller
Press Officer
Phone +49 30 31002 242
Email martina.mueller@hhi.fraunhofer.de
Dr.-Ing. Anna Hilsmann
Technical Contact
Phone +49 30 31002-569
Email anna.hilsmann@hhi.fraunhofer.de
Originalpublikation:
https://tub-rip.github.io/eventvision2023/papers/2023CVPRW_X-Maps_Direct_Depth_L…
Weitere Informationen:
Media Contact
All latest news from the category: Information Technology
Here you can find a summary of innovations in the fields of information and data processing and up-to-date developments on IT equipment and hardware.
This area covers topics such as IT services, IT architectures, IT management and telecommunications.
Newest articles
NASA: Mystery of life’s handedness deepens
The mystery of why life uses molecules with specific orientations has deepened with a NASA-funded discovery that RNA — a key molecule thought to have potentially held the instructions for…
What are the effects of historic lithium mining on water quality?
Study reveals low levels of common contaminants but high levels of other elements in waters associated with an abandoned lithium mine. Lithium ore and mining waste from a historic lithium…
Quantum-inspired design boosts efficiency of heat-to-electricity conversion
Rice engineers take unconventional route to improving thermophotovoltaic systems. Researchers at Rice University have found a new way to improve a key element of thermophotovoltaic (TPV) systems, which convert heat…