Let software catch the game for you

Software that can identify the significant events in live TV sports broadcasts will soon be able to compile programmes of highlights without any help from people.

The technology will save broadcasters millions in editing costs- and should eventually lead to new generations of video recorders that will let people customise their own sports highlights packages. But developing software that understands sport is no easy task.

Picking out the key events from a game- whether it be pool, rugby, baseball, soccer or basketball- is labour-intensive. As the footage streams into a TV station or outside-broadcast truck, a sports editor has to watch the action and keep notes on what happens and when. Only after that are the clips retrieved and put together to form a highlights package.

But as sports follow fixed rules, and take place in predictable locations, computers ought to be able to pick out the key pieces of play and string them together. “It is a situation that is ripe for automation,” believes Andrew Kilner at Sony Broadcast in Basingstoke, UK, which makes TV broadcasting equipment.

Anil Kokaram and colleagues at Trinity College in Dublin, Ireland are among the teams trying to turn the idea into reality. They have decided to analyse table-based ball games like snooker and pool. These are sports that a computer should find relatively easy to handle as the action is slow, indoor lighting is fairly consistent and cameras mostly shoot from fixed positions.

The Trinity team’s PC-based software uses the edges of the table and the positions of the pockets to work out where the balls are on the table. The software has the rules of the game programmed in, so it can track the moving balls and work out what has happened. For example, if a ball approaches a pocket and then disappears from view, the program assumes it has been potted. By working out how to detect foul shots – when a player hits the wrong ball – the team hopes to find a way to create a compelling highlights package that includes a varied selection of the action.

Sports like American football and soccer will be much more of a challenge, because they involve a far greater number of moving objects (both teams of players plus the ball) on the field which cannot be tracked easily without massive computer power. Hampering this process, too, is the fact that the colour of the playing field is often patchy and can vary with the weather and lighting. So when the camera moves across the field, the software could mistake the different-coloured patches for extra players.

Carlo Colombo and colleagues at the University of Florence, Italy, are trying out another idea. They found that they can compile highlights from soccer footage without tracking the ball or the moving players. Instead, one of their tricks is to look at the position of the players in set pieces like corners, free kicks and penalties. Their software detects the position of the pitch markings in a shot to work out which area is in the frame (see Graphic). Then, by checking the positions the players adopt in relation to the markings, the software can decide whether a player is about to take a penalty, free kick or corner, and whether or not a goal is scored as a result. The Florence team has not yet worked out how to enable the computer to determine when a goal is scored in open play.

Ahmat Ekin, a computer scientist from the University of Rochester in New York, may be close to solving that problem. He has designed software that looks for a specific sequence of camera shots to work out whether a goal has been scored. For example, player close-ups often indicate a gap in play when something important has happened, and slow-motion footage is another useful cue. If Ekin’s software sees a sequence of player close-ups combined with shots of the crowd and pictures in slow motion that lasts between 30 and 120 seconds, it decides that a goal has been scored, and records the clip in the highlights. But it could also be possible that a controversial incident is being analysed, and Ekin aims to get round this by combining sound analysis with the pictures to give a more accurate result. For example, the software could hunt for the commentator’s extravagant shouts of “gooooaall!”.

The electronics giant Sharp is now trialling a simple highlights package called Himpact with sports broadcasters. For soccer it simply searches for all replay footage, but in American football or baseball it captures all the “plays”- the action between the frequent pauses. In tests it has cut an hour of American football down to around 14 minutes, and an hour’s baseball to 10 minutes. Sharp is now seeking commercial partners to develop the technology for home video recorders. It hopes people will be able to choose a full highlights package or a customised one, in which they might choose to see only goals- or fouls, if the mood takes them.

The Japanese firm is also planning a heavyweight future for the technology: its next target market is sumo wrestling.