From Testbed To Reality: Grid Computing Steps up a Gear

UK plans for Grid computing changed gear this week. The pioneering European DataGrid (EDG) project came to a successful conclusion at the end of March, and on 1 April a new project, known as Enabling Grids for E-Science in Europe (EGEE), begins. The UK is a major player in both projects, providing key staff and developing crucial areas of the technology. While EDG tested the concept of large-scale Grid computing, EGEE aims to create a permanent, reliable Grid infrastructure across Europe.

Grid computing pulls together the processing power and data storage of thousands of computers, spread over hundreds of locations. Professor Steve Lloyd, Chair of the UK Particle Physics Grid, explains that, “Individual scientists using the Grid won’t need to know where the data is held or which machines are running their programmes. So whereas a PC on the web provides information or access to services, such as banking or shopping, a PC on the Grid offers its computing power and storage.”

The European DataGrid (EDG) project started three years ago, with the UK Particle Physics and Astronomy Research Council (PPARC) providing £2.1m funding, as one of six main partners. EDG took a major step towards making the concept of a world-wide computing Grid a reality, building a test computing infrastructure capable of providing shared data and computing resources across the European scientific community. At peak performance, there were more than 1,000 computers on the EDG test bed, sharing more than 15 Terabytes (15 million million bytes) of data at 25 sites across Europe, Russia and Taiwan. Grid resources were provided to over 500 scientists.

After a massive software development effort involving seven major software releases over three years, the final version of EDG software is already in use in three major scientific fields: Particle Physics, Biomedical applications and Earth Observations. The software is exploited by ten bio-medical applications and five earth observation institutes.

In Particle Physics, Grid computing will help scientists deal with a data deluge from CERN’s new particle accelerator, the Large Hadron Collider (LHC), due to go online in 2007. LHC will produce millions of billions of bytes of real and simulated data. GridPP, the UK’s Particle Physics Grid, has been working with EDG over the last three years. GridPP resources contributed a large part to the EDG testbed, with processors at 16 UK sites and around 100,000 computing jobs submitted through UK computers.

GridPP also helped to develop much of the important ’middleware’ for EDG. This allows the software being used by the scientists to talk to the Grid’s hardware, distributing computing tasks efficiently around the network and dealing with issues such as security, ensuring that only authorised users can access the Grid. GridPP members will also be heavily involved in the next stage of European Grid computing, EGEE.

The EGEE project will build on the success of EDG and take Grid technology even further by establishing a Grid infrastructure available across Europe, 24 hours-a-day. Fabrizio Gagliardi, former DataGrid Project Leader and Project Director of EGEE, said: “Whereas EDG provided European scientists with the first convincing large-scale demonstrations of a functioning Data Grid, EGEE will make the technology available on a regular and reliable basis to all of European science, as well as industrial Research and Development. Like the World Wide Web, which was initially conceived at CERN for rather specialised scientific purposes, the impact of this emerging Grid technology on European society is difficult to predict in detail at this stage, but it is likely to be huge.”

EGEE will capitalise on the experience and achievements of EDG and many other EU, national and international Grid projects. It will primarily concentrate on three core areas:

– to build a consistent, robust and secure grid network.

– to continuously improve and maintain the middleware in order to deliver a reliable service to users.

– to attract new users from industry as well as science and ensure they receive the high standard of training and support they need.

EGEE consists of 70 partner institutions covering a wide-range of both scientific and industrial applications. Two pilot areas have been selected – the Large Hadron Collider Computing Grid; and Biomedical Grids, where several communities are facing equally daunting challenges to cope with the flood of bioinformatics and healthcare data.

Four UK organisations are partners in EGEE – PPARC, the Council for the Central Laboratory of the Research Councils (CCLRC), the National e-Science Centre in Edinburgh and University College London (UCL). In addition, there are five UK contributing organisations, which are part of the UK and Ireland Federation set up to extend deployment of this European-wide Grid: University of Glasgow, Imperial College London, University of Leeds (on behalf of the White Rose Consortium), University of Manchester, and University of Oxford.

The Grid will be built on the EU Research Network GEANT, as well as national infrastructure such as the UK’s SuperJANET academic network. UCL, through its e-Science Network Centre of Excellence, has primary responsibility for developing and deploying new EGEE network services, such as monitoring the networks and allocating space on them. Through this, the UK will play a major role in developing the critical relationship between EGEE and GEANT.

CCLRC (at the Rutherford Appleton Laboratory) will lead a partnership of a number of UK institutes in delivering production quality Grid services as part of EGEE and will provide core infrastructure services to the EGEE Grid. A programme focussed on producing high quality Grid information and monitoring services will also be developed.

This builds on the substantial experience built up over the last 3 years through participation in the EDG project and in running pilot Grid services.

The EGEE training programme, to be led by the UK National e-Science Centre (NeSC), will involve the active participation of 22 of the 70 EGEE partner organisations. During the next two years it will run training events and workshops all over Europe, as well as delivering customised training events within Grid computing conferences. The end-product of this work will be a series of tried and tested high-quality training modules, available for general use via the Web.

EGEE is a two-year project conceived as part of a four-year programme, where the results of the first two years will provide the basis for assessing subsequent objectives and funding needs.