Baudouin Raoult, ECMWF
As most meteorological organisations, the European Centre for Medium-Range Weather Forecasts (ECMWF) is facing the challenges created by ever growing data volumes. The Centre needs to make the results of its forecasting systems available to its Member States, Cooperating States in a timely fashion. This is currently done by pushing the data to users via a private network or the Internet, according to a prescribed schedule; this system is limited by the disk and network bandwidths. As a consequence, users cannot make use of the vast number of products generated, in particular the output of the Centre’s ensemble forecasting systems. ECMWF is also running one of the largest meteorological archives in the world, currently holding over 250 PB of data, and growing at a rate in excess of 1 PB per week. Thousands of users are issuing millions of requests per day to retrieve data from popular datasets such as ERA5, the Centre’s fifth generation reanalysis. Once again, usage of such data is limited by slow downloads over the Internet.
ECMWF believes that cloud-based services offer a way forward the address these challenges and embracing cloud technologies is now part of its strategy.
ECMWF plans to offer to its Member States and Cooperating States the ability to create virtual machines in a cloud infrastructure that is located in its own data centre, close to its high-performance computing facility and its data handling system. Although the virtual machines would be hosted on the Centre’s premises, its Member States and Cooperating States will have full control over them and could consider offering services to their own end-users, for example by running a web site offering products derived from ECMWF model outputs.
For this project, three challenges have been identified: technological, legal and security. These are currently being studied, and ECMWF is testing a series of uses cases with some of its users.