Grid expectations for networked computing: from global Earth monitoring to black hole detection

Whether dealing with high-power particle accelerators, astronomical observatories or Earth-watching spacecraft, modern science involves vast volumes of information, and researchers require powerful Grid computing techniques to manage this data deluge.

And at a time when the research teams working on the biggest scientific problems of our time – from climate modelling to molecular biology or high energy physics – are larger and more diffuse than ever before, Grid computing offers a promising way for such groups to perform ’e-collaboration’: forming virtual organisations able to remotely work together, sharing data, tools and resources. A two-day workshop at ESA’s centre in Italy this week covered scientific and also industrial Grid computing applications, highlighting current systems as well as the likely course of future Grid development and applications. Around 40 local scientific researchers and representatives from industry gathered for the ’Grid and e-Collaboration for the Space Community’ workshop. The event took place on 2 and 3 February at the European Space Research Institute (ESRIN) in Frascati overlooking Rome. A day of discussion was followed by a second day of tutorials in Grid use.

It was an appropriate locale because ESRIN is dedicated to Earth Observation data archiving and distribution, an increasingly demanding task. For example, in just under three years of Envisat satellite operations, two Petabytes of derived data have now been archived, enough information to fill 20 million filing cabinets. The need to handle these much data and make them available to users has led ESRIN to take up Grid technology. “We have begun an Earth Science Grid-On-Demand service based on our local Grid,” explained organiser Luigi Fusco of ESRIN. “From this single site, users gain access to large volumes of Earth Observation data and can easily and quickly perform a variety of data reprocessing, including fine-tuning new algorithms to get optimal results.” Applications so far include satellite image mosaicking to monitor changing global chlorophyll concentrations on a monthly basis and tracking iceberg movement in polar regions.

ESA’s Internal University is interested in Grid computing not just as a subject but also a means of enabling staff education across Agency directorates and establishments. As an illustration of what is possible, interested groups remotely attended the Workshop from ESA’s European Space and Technology Centre (ESTEC) in the Netherlands, and European Space Operations Centre (ESOC) in Germany, as well as the University of Madrid, using Grid-supported collaborative work software called ISABEL.

ESRIN is also participating in the just-announced Megalab (Metropolitan e-Government Application Laboratory) initiative from the Lazio region, setting up a new high-speed data link from the centre of Rome to Frascati, linking up numerous research institutions in the area into a Metropolitan Area Network (MAN). This network is seen as a way of boosting regional competitiveness and offering innovative e-government services, enabling Grid computing for science and also business.

What are Grids?

The basic principle behind Grid computing is simple enough: anything one computer can do, a pool of computers can do faster and better. These machines do not have to be in the same building, country or even continent – they simply have to be linked together.

A geographically dispersed – or ’distributed’ – network of computers has the potential to provide users with access to advanced computing services, processing power or memory, enabling the solution of massively complex tasks beyond the capabilities of a single machine or local network. “The term ’Grid’ comes from the way electricity grids operate,” explained Domenico Laforenza of Italy’s National Research Centre Institute of Information Science and Technology (CNR-ISTI) in Pisa: “Nobody needs to know where the electricity comes from, or how it is transmitted, it is just there to be used. With Grids, the same is true of processing power.”

A variety of Grids are already in existence, utilised mainly by research institutions. The Workshop heard the Italian Agency for New Technologies, Energy and Environment (ENEA) has a Grid linking 12 of its sites across the country, and a nationwide ’Grid-it’ project being coordinated by the National Research Council currently links 20 sites with 1400 nodes and 2800 processors. On a larger scale, the Enabling Grids for E-sciencE (EGEE) project, supported by the European Commission (EC), is building a trans-continental Grid for science. Receiving €32 million in EC funding, the project involves 70 partners in 26 countries.

Putting Grids to work

Piero Benvenuti of the Italian National Institute for Astrophysics (INAF) recounted how use of Grids within astronomy has already led to the finding of some 30 new black holes. The discoveries came out of large-scale automated comparison of views of the same section of sky obtained from the Hubble and Chandra Space Telescopes. Common variations in emitted ultraviolet and x-ray energy was the sought-after clue. And a planned Virtual Observatory will use a Grid to enable astronomers to scrutinise a ’virtual sky’ that merges inputs from different instruments at different wavelengths at different times.

Benvenuti added that Grid use is set to further increase in future with the launch of ESA’s Planck spacecraft to measure the cosmic background radiation and the Gaia mission to create a three-dimensional map of our galaxy, as well as the ground-based Large Synoptic Survey Telescope (LSST).

Going beyond science

Just as the web initially developed for specialised scientific use but was enthusiastically taken up by wider society, emerging Grid and e-collaboration technology is likely to have a huge impact beyond science. Federico Rossi of the company Datamat said he predicted Grid use spreading beyond science to industry in the next few years, as initial ’hype and hope’ was replaced by early adopters demonstrating the solutions and convenience that Grids offer within an operational context.

The aerospace and automotive sectors were both obvious candidates for use of Grid technology, but he said that additional applications could be feasible within the fields of finance, health care, localised meteorology for tourism and sport and even the media and entertainment – Grid technology has the potential to make networks of local computer animators competitive with Hollywood.

With the distributed firm of Airbus, already a European success story, a wide variety of businesses could potentially use e-collaboration to coordinate with component suppliers and subcontractors. Rossi added that one survey of the oil and gas sector showed 84% of companies think e-collaboration is important.

ESA e-collaboration in action

Massimo Bandecchi of ESTEC discussed an ESA activity that shows how future industrial e-collaboration could work. The Agency has constructed a state-of-the-art Concurrent Design Facility (CDF) at ESTEC for designing future space missions. It is equipped with a network of computers, multimedia devices and software tools allowing multi-disciplinary teams to work together simultaneously.

The CDF has so far been used for designing around 30 mission proposals, and while it was initially conceived solely as an internal tool more recently external participation has taken place, including joint work with NASA and Stanford University. It has proved sufficiently popular that spin-off requests for the technology have been made by ESA member states and industry, and several new CDFs are taking shape within national space agency centres. The intention now is to apply Grid technology to enable distributed concurrent design from these facilities.

Meanwhile at ESRIN, a pilot e-collaboration study called THEmatic Vertical Organisations and Implementation of Collaborative Environments – or ’The Voice’ – intends to build up a user infrastructure allowing groups of users to collaborate in Earth Observation-related research or generate prototypes of scientific or value-added products. As currently envisaged, ’The Voice’ would also allow users the chance to integrate Earth Observation-derived products with products from other sources.

The Grid future

In his speech, Laforenza recounted the evolution of the Grid and e-collaboration concept so far, and how future progress would rely on interoperability standards – ensuring different Grids work together: “The Global Grid Forum is the organisation that supports the development of agreements and specifications, its membership spanning vertical layers from those doing Grid research, designing and building Grid software, deploying Grids and actually using Grids.”