A future vision for media

New technologies to support the rapidly evolving world of media and communications are big business. Creating successful image recognition software is a key area of research, since multimedia relies heavily on the power of imagery and customers are demanding more information more rapidly.

Teaching computers to ‘see’ and recognize images accurately is difficult. The key is to train computers to recognize and group together images that have the same or similar numerical pixel data. The sheer amount and complexity of image data available means that advanced image recognition is still beyond current technologies.

An image recognition system called Snap2Tell, developed by Yiqun Li and co-workers at the A*STAR Institute for Infocomm Research is now being integrated into a new mobile phone app for Malaysian newspaper The Star as a new function called iSnap. This could have many potential applications for media use in future.

How Snap2Tell works

The initial objective of the Snap2Tell technology is to simplify the process of getting information on the move,” explains Li. “By capturing an image using the phone camera, relevant information on the topic at hand is displayed to the user immediately. Taking a picture is much easier than typing a sentence on the small key pad on the phone.”

The Snap2Tell software works by recognizing image information present in a pre-prepared database of Snap2Tell readable images. Snap2Tell matches the image to the database, and then accesses further information which has been pre-associated with that image.

Enhancing newspapers with iSnap

Snap2Tell is now being applied to mobile augmented reality applications, in the form of iSnap. iSnap is free to use for mobile phone and iPad users as soon they have downloaded the Star Mobile app. The user simply points the phone camera at an iSnap marked article or advert, and a set of icons will appear on top of the articles on the phone screen. The icons represent different additional information relevant to the article. When an icon is touched, iSnap will automatically take the reader to further information through a variety of multimedia, such as websites, social network forums, or video clips.

In The Star newspaper, certain articles and advertisements are now printed with an iSnap logo next to them. iSnap is different to QR codes, currently used widely in advertising print media such as billboard posters. Essentially barcode squares that phone users can scan, QR codes direct the user’s phone to a company’s product website. With iSnap, the code is not needed — the software recognizes images related to the product.

In addition, iSnap brings up a whole array of different multimedia information via icons on the touch-screen — not just one website. This means that the user has a choice of which media form they would like to view, rather than scrolling through websites to find a video, audio file, or written commentary, to give but a few examples. The user also has the choice to save their favorite information to their phone, should they wish.

Advantages and disadvantages

Media and advertising companies are delighted with the possibilities that the Snap2Tell technology brings. Paper advertisements can potentially come to life through mixed media — for example viewing the latest TV adverts or reading the specifications and reviews for the latest model of a car. News articles can be accompanied by film footage, interviews and associated story links and commentaries, and even billboard posters could carry the iSnap logo in future and lead the ‘snapper’ to film trailers or the latest news on a certain product.

Furthermore, a more targeted audience could also be reached in future, as Li explains: “The information can also be stored in the user's phone if the user is interested in the product being advertised. We have developed a mobile app called Snap2Remind for bringing up more information on printed advertisement pictures. The app also provides location-based reminders to alert users when they pass by the shops where the advertised products are being sold.”

For advertising and media companies, Snap2Tell opens up a wide range of possibilities in terms of consumers accessing further information without page or print limitations. Companies can also monitor the popularity of different articles and products by looking at the access rates for each individual item.

There are, however, some limitations of the technology as it currently stands. iSnap requires a phone with a touch-screen and access to the internet. In addition, media companies’ servers need to be big enough to host and produce multimedia on a large scale. They also need to develop a database of Snap2Tell readable images, by uploading each image as it is needed in printed form. This could eliminate some smaller local companies from participating.

Future plans for Snap2Tell technology

Li is quick to point out that, as in the West, newspapers may not last in Asia. However, she remains optimistic about the future of iSnap and Snap2Tell: “Even if there are no printed newspapers in the future, there may still be other paper media such as posters, signboards, paintings, drawings, CD covers, and book covers, for example.”

Snap2Tell image recognition is not limited to printed materials; it can be applied to physical objects or scenes as well — see Snap2Search for more details.

A*STAR researchers are now working to produce more complex versions of the Snap2Tell technology and expanding it into applications for 3D physical objects or scenes. Pointing the phone camera to a real building could bring up a similar touch-screen experience, providing the mobile users with information about the building and its location. This could mean that sight-seeing for tourists could take on a new twist, without the need for carrying guidebooks.

“We are planning to work on more complicated cases of augmented reality on mobile phones,” says Li. “The iSnap may not be just an overlay of icons on top of printed media. It may include 3D object rendering on top of physical objects or scenes in future.”

About the Institute for Infocomm Research

The A*STAR Institute for Infocomm Research (I²R) was established in 2002 with the mission of becoming a globally preferred source of innovations in interactive secured information, content and services. I²R conducts research and development in information technology, wireless and optical communications, interactive and digital media, signal processing and computing.
Associated links
http://www.research.a-star.edu.sg/feature-and-innovation/6516