Powerful object recognition for diverse applications using AI and robotics

New algorithms for object recognition and environment perception have been developed at Fraunhofer IPA, enabling sophisticated automation solutions for the service and production sectors. They are a result of the “Knowledge4Retail” research project, in which twelve project partners have developed solutions for future-oriented retail.

Making stationary retail more attractive to customers again with the help of digital solutions – this was the goal of the partners working together on the Knowledge4Retail (K4R) project from January 2020 to December 2022. The group presented the results at the final meeting this spring. The core of the project is a K4R open-source platform available on internet host GitHub, a version control system, which makes the results accessible to retailers. Retail companies can use the platform to connect analog and digital worlds as well as analyze processes more easily and generate purchasing behavior forecasts, for example. In the process, the project partners developed solutions for applications in logistics, store set-up, service robotics and a fully automated kiosk.

Fast, flexible 3D image processing

The IPA researchers worked on several development focuses as part of the project, during the course of which they expanded their expertise as well as the resulting range of 3D image processing-based products and services. One core aspect was the creation of what is known as semantic digital twinning of retail products. This involves using sensors to capture any product in a scanning station, for example. The product is then modeled three-dimensionally based on a wide range of information such as shape, color, and texture. The modeling process, in which Fraunhofer IPA collaborated with the Kaptura company, also involved machine learning methods, a subfield of AI (Artificial Intelligence).

With the help of the technologies developed, it is possible to model wide product ranges quickly and in 3D with high resolution texture. During the modeling process, semantic information such as logos, nutritional value or quantitative information on products is also recognized and digitally collected. The models generated are the basis, for example, for digital twins of a real store, a web store (see the link to the demo website of a virtual store in the info box below) or automated product analyses, such as those carried out in laboratories.

In addition, a scalable object recognition solution optimized for retail has been created, enabling automation solutions for production and logistics through precise optical product identification. Application examples include robot-based bin picking or bin packing, i.e. the packing of goods of all types and shapes. The latter task, also known as “order picking” or “pick & pack”, in which goods are removed from crates or shelves and neatly packed into a target carton, is becoming increasingly important, especially in these times of booming online retail. The object recognition solution developed to support these processes can be taught fully automatically on digital 3D product models in less than an hour for each new product.

Smart shelving and kiosk systems

Another result from Fraunhofer IPA is a mobile pointer unit that “helps” to move goods. The solution is aimed at skilled workers in logistics and retail and makes it easier for them to stock shelves. Employees wear a small barcode ring scanner on their finger which registers a product or its box to be placed on the shelf. A moving spotlight then illuminates the location where the product needs to be placed based on a digital store twin. The application is particularly useful for small products as well as those that are very similar to one another and saves time (see video link below).

The Smart Kiosk System is a sales solution that was developed and tested. It uses the micromarket concept in order to branch out from supermarkets to include additional sales channels in the form of canteens and break rooms. For this purpose, weight sensors and cameras are installed in a refrigerator. With the help of object recognition, they make it possible to automatically recognize hands and products as well as remove or deposit products. This is combined with personal identification via a user interface and connection to a billing system (e.g., in this project, to SAP through partner Nagarro). Consequently, product withdrawals can be automatically booked in the virtual shopping cart and later billed digitally. In addition, service personnel receive a message when products need to be replenished, as the kiosk system constantly updates the digital twin on the connected K4R cloud platform. The project work took place in consultation with practice partner Livello.

Integration in the open-source platform

Last but not least, the IPA experts also advised on how the solutions developed as part of the project could best be specified and integrated in the K4R open-source platform. The openly expandable platform provides functions and interfaces for connecting to various logistics systems, sensors, robots or the extensive software modules from the Robot Operating System (ROS) environment, as well as providing the necessary infrastructure for store and product networked digital twins.

Fraunhofer IPA has been leading the European part of the ROS-Industrial consortium for years and has consequently acquired extensive experience in the development of powerful, modular open-source systems. In fact, in predecessor projects such as Service Robot Network (SeRoNet), one of the project goals was to make the results sustainably accessible beyond the end of the project. For SeRoNet, this was achieved through the Xito.one virtual marketplace, a start-up development, and the ROStooling development tools, which were also posted on GitHub and used as a knowledge base for the K4R platform.

Project profile
Sponsor: German Federal Ministry for Economic Affairs and Climate Action BMWK: funding code 01MK20001G
Duration: 01 January 2020 to 31 December 2022
Project partners: Nagarro Allgeier ES GmbH, Deutsches Forschungszentrum für Künstliche Intelligenz [German Research Center for Artificial Intelligence] GmbH, dm-drogerie markt GmbH + Co. KG, dmTECH GmbH, EHI Retail Institute GmbH, fortiss GmbH, Fraunhofer-Gesellschaft, neusta GmbH, neusta software development West GmbH, team neusta GmbH, Technical University of Munich, Ubimax GmbH, University of Bremen

Wissenschaftliche Ansprechpartner:

Jochen Lindermayr | Phone +49 711 970-1565 | jochen.lindermayr@ipa.fraunhofer.de
Richard Bormann | Phone +49 711 970-1062 | richard.bormann@ipa.fraunhofer.de

Weitere Informationen:

http://github.com/knowledge4retail -> GitHub-Link
http://www.object-twin.ai/ -> Online shop website
http://knowledge4retail.org/ -> project website

http://www.ipa.fraunhofer.de/