MIT team guides airplane remotely using spoken English

Aeronautics researchers at MIT have developed a manned-to-unmanned aircraft guidance system that allows a pilot in one plane to guide another unmanned airplane by speaking commands in English.

In a flight test, the pilotless vehicle, called a UAV (unmanned aerial vehicle), responded to sudden changes in plan and avoided unexpected threats en route to its destination, in real time. “The system allows the pilot to interface with the UAV at a high level–not just ’turn right, turn left’ but ’fly to this region and perform this task,’” said Mario Valenti, a flight controls engineer for Boeing who is on leave to pursue a Ph.D. in electrical engineering and computer science at MIT. “The pilot essentially treats the UAV as a wingman,” said Valenti, comparing the UAV to a companion pilot in a fighter-plane squadron.

Tom Schouwenaars, a Ph.D. candidate in aeronautics and astronautics, and Valenti are principal researchers on the guidance system, which is part of the capstone demonstration of the Software Enabled Control (SEC) program. Professors Eric Feron and Jonathan How of the Department of Aeronautics and Astronautics (aero/astro) are among the principal investigators on the SEC program. The SEC program is a five-year, inter-university effort sponsored by the Defense Advanced Research Projects Agency (DARPA) through the Air Force Research Laboratory. As industry partner, Boeing provided the avionics test platform for the MIT guidance system and the planes used to demonstrate it.

The new guidance system is designed for volatile combat situations. For instance, a pilot might be commanded to gather images of an enemy site located in unknown territory. Rather than putting himself in danger, the pilot could assign a nearby UAV to the task. The UAV moves toward the enemy site, avoiding known threats (no-fly zones) and the unexpected (radar emanating from a missile site), all the while communicating its actions to the pilot in the other aircraft, which follows behind at a higher altitude and a safe distance. The technology also could have applications in the coordination of multiple air or space vehicles, such as in air traffic control or the reconfiguration of distributed satellite systems.

The guidance system performed flawlessly in flight tests involving a Boeing F-15 fighter jet and a Lockheed T-33 trainer fighter jet at Edwards Air Force Base in June. A pilot in the manned F-15 issued mission-level commands in everyday English–“fly to Task Area B”–to the T-33, and the T-33 executed them, maintaining a trajectory safe from threats, and at one point adjusting to a last-minute change in the predetermined mission plan. The T-33 was a substitute for the actual UAV in the test. It was manned by a pilot and crewperson who were on board to manage the aircraft in case of failure, but the vehicle was controlled entirely by MIT’s software, which ran on laptops placed inside each plane. “Through the recent experiments, the SEC program has demonstrated advanced behaviors that may now be integrated into the next generation of unmanned vehicles,” said John Bay, DARPA’s SEC program manager.

A paper published by the American Institute of Aeronautics and Astronautics (AIAA) in August discussed the results of the flight test in more detail. Aero/astro graduate student Yoshiaki Kuwata and James L. Paunicka, associate technical fellow at Boeing Phantom Works, authored the paper along with Feron, How, Schouwenaars and Valenti. Schouwenaars’ work on autonomous trajectory-planning algorithms earned him the AIAA’s Unmanned Aerial Vehicles Graduate Award, which he will receive at a conference in Reno, Nev., in January.

Teaching English to an airplane

Three elements combine to make MIT’s manned-to-unmanned air vehicle guidance system more flexible and more “intelligent” than previous systems. First, the team worked with Teragram Corp., a software company specializing in language technology, to create a natural-language interface through which the two vehicles communicate and coordinate their actions. The interface translates the pilot’s human language into the UAV’s machine language, and vice versa. “It allows us to task machines at a higher level, improving safety and efficiency,” said Feron.

Second, Valenti designed a task scheduler that keeps track of the oft-changing mission data from the manned vehicle and interprets it into tasks the UAV can perform. The task scheduler is integrated with the third element, Schouwenaars’ safe-trajectory-planning algorithm. The algorithm is based on mixed-integer linear programming (MILP), an optimization framework originally developed for operations research. Feron and Schouwenaars started applying MILP to the problem of aircraft routing in 2000. Unlike earlier technologies, MILP allows trajectory planning that can guarantee against collisions. Moreover, the trajectories can be computed while the vehicle is flying, requiring no pre-planning.

Using the off-the-shelf optimization software CPLEX, Schouwenaars fine-tuned the MILP-based guidance technique to enable the UAV to choose the fastest safe path to its destination–and then change course in a split second when faced with a new command or a sudden obstacle. The June flight tests marked the first time that a manned air vehicle used a MILP-based guidance system to control a UAV.

Already, said Feron, “the aerospace industry is using our system in its most advanced UAV programs.” He and his team are currently working toward implementing their guidance technology in systems with multiple air vehicles. Their work is being done in MIT’s Laboratory for Information and Decision Systems. A version of this article appeared in the November 3, 2004 issue of MIT Tech Talk (Volume 49, Number 8).