Combining robotics and ChatGPT

TUM professor uses ChatGPT for choreographies with flying robots.

• Prof. Angela Schoellig has proved that large language models can be used safely in robotics.
• ChatGPT develops choreographies for up to nine flying robots performing to selected music.
• An additional algorithm guarantees that the drones will fly safely.

Prof. Angela Schoellig from the Technical University of Munich (TUM) uses ChatGPT to develop choreographies for swarms of drones to perform along to music. An additional safety filter prevents mid-air collisions. The researchers’ results demonstrate for the first time that large language models (LLMs) such as ChatGPT can be used in robotics.

The web interface is easy to use: Doctoral student Martin Schuck selects a music track and enters a text requesting a suggested choreography. Additional instructions can be given to the drone swarm via another prompt in the ChatGPT tool before an algorithm checks whether the suggested flight paths are feasible. The screen in the Learning Systems and Robotics Lab now shows a simulated airfield with six drones flying in circles to the music. If the scientist from the Chair of Safety, Performance and Reliability for Learning Systems at TUM likes this choreography, he logs it in. Soon afterwards, six palm-sized drones take off from the floor of the robotics lab.

Choreography with three flying robots is completed within five minutes

In Prof. Angela Schoellig’s laboratory, the research team has installed six ceiling-mounted cameras in a room measuring around 40 square meters and three meters high. Crosses are marked on the floor with insulating tape. These indicate the starting positions of the drones. Once the computer has verified a possible choreography, they can take off. The cameras detect the position of the quadrocopters, which are equipped with four propellers and motors, 200 times a second. The system compares them with the desired position. The “airshows” in the Learning Systems and Robotics Lab, which the research team realizes with up to nine drones, are 100 percent safe today. Without the special safety filter, only one in four demonstrations is accident-free.

SwarmGPT: additional algorithm ensures safety of flying robots

Prof. Angela Schoellig combined ChatGPT with the safety filter for the “Dance of the Flying Robots”. “The ChatGPT AI tool was primarily created to generate texts, but it can also suggest choreographies,” says the professor. “However, it initially knows nothing about the properties of drones and physical limits for the flight paths. So it is clear that ChatGPT makes mistakes.”

The additional safety algorithm closes this gap by mapping out flight paths for the proposed choreography so precisely that mid-air collisions are completely avoided. Drones can even approach each other diagonally. Prof. Schoellig calls the overall concept of ChatGPT and security filter designed to use several flying robots “SwarmGPT”. The tool generates the processes in the air and at the same time serves as an interface between the robot and the human, who does not require any expert knowledge.

A quantum leap through ChatGPT

When Angela Schoellig began her drone research almost 15 years ago, choreographies were hand-crafted. It took more than three years to develop the first six choreographies for six drones and get them up and running. “ChatGPT has brought about a quantum leap,” says Prof Schoellig. Over the past three months, the researchers have experimented with over 30 choreographies for up to nine drones. Today, it takes the researchers only around five minutes to develop a safe choreography for 30-second music clips with three drones. The more drones are added, the more time ChatGPT needs for its calculations and the longer it takes to propose a choreography. But Schoellig is certain: “The concept is scalable.”

Robotics: ChatGPT becomes an interface for non-experts

Can other robots be used with a similar interface via ChatGPT? For robots that use voice control to pick up objects, lay cables or open doors, the success rate of those actions is currently only 63, 56, and 80 percent. So far, applications in other robotics scenarios have been somewhat unreliable. For Prof. Angela Schoellig, this is an incentive: “I assume that our approach will keep getting better in other scenarios, too.” It may soon be possible to reprogram suction-based and industrial robots simply by voice command without the need for expert knowledge or programming skills.

More information
– Prof. Angela Schoellig is Head of the Chair of Safety, Performance and Reliability for Learning Systems at TUM and Director of the Munich Institute of Robotics and Machine Intelligence (MIRMI). At TUM’s Robotics and AI Institute, more than 70 professors are focusing on developing robotic and intelligent solutions in the fields of health, the environment, mobility, and work.

– Swarm-GPT: Combining Large Language Models with Safe Motion Planning for Robot Choreography Design ; NeurIPS 2023 workshop paper – https://tinyurl.com/23qwnyb8

Additional editorial information

Photo material: https://mediatum.ub.tum.de/image/1740784
Video: https://youtu.be/aKnhSnJHNpw

Wissenschaftliche Ansprechpartner:

Prof. Angela Schoellig
Chair of Safety, Performance and Reliability for Learning Systems
Technical University of Munich (TUM)
Angela.schoellig@tum.de

Originalpublikation:

Vivek K. Adajania, Siqi Zhou, Arun Kumar Singh, Angela P. Schoellig; AMSwarm: An Alternating Minimization Approach for Safe Motion Planning of Quadrotor Swarms in Cluttered Environments; IEEE; https://ieeexplore.ieee.org/document/10161063

https://www.tum.de/en/news-and-events/all-news/press-releases/details/tum-professor-uses-chatgpt-for-choreographies-with-flying-robots