Germany’s National Meteorological Service Extends Far Beyond Its Borders

Meteorology today involves far more than regional weather forecasting; climate change has taken precedence as an international priority involving extremely complex political and socioeconomic challenges. These challenges call for advancements in meteorological research and service organizations worldwide. These challenges also call for data warehousing solutions capable of managing complex workloads and delivering real-time analytics for data volumes ranging into the hundreds of Terabytes.

Germany’s National Meteorological Service, the Deutscher Wetterdienst (DWD), residing in Offenbach, Germany, is one such organization. DWD monitors weather and meteorological conditions over Germany. In addition to weather services for the general public, DWD collaborates with national and worldwide environmental research institutes and organizations to assist in solving global challenges such as alternative energy sources, air quality, and disaster and water management.

DWD also provides a range of climatology services including analytical and evaluation services, expert reports, specialized forecasts, and weather information systems. In order to further improve its services, especially for warning purposes, DWD is introducing an innovative extremely short-range weather forecasting system.

Challenge:
DWD is launching its Ensemble Prediction System (EPS), an advanced solution for weather forecasting. DWD is one of the first meteorological organizations worldwide to undertake the challenge of operationally processing and analyzing ensemble data for extremely short-range weather forecasting. For this challenge a new database system was essential. The ensemble method allows estimating the reliability of a given forecast by averaging forecast outputs with slightly different starting conditions/measured values, taking into account the general uncertainties in modelling an extremely complex system like the atmosphere. These ensemble forecasts are planned to be recalculated every three hours. The new method involves the generation of more than 1GB per second of data. A large amount of this data is being processed by the database system and then made available for the meteorologists for further analysis. This new environment is also for one of DWD’s most valuable data assets – its database of meteorological data, the National Meteorological Library.

National Meteorological Library is a gem in meteorological history. It consists of both numerical data dating back to 1966 when the DWD undertook its very first numeric weather calculations and of non-numerical data reaching back as far as to the second half of the 15th century. A wealth of recent data is used as the basis for weather forecasts and for the verification of new weather models. In order to take on the challenge of extremely short-range weather forecasts a very substantial rise in data base performance is critical.

Growing data loads and heavy data traffic The existing data warehouse platform for meta data already strained under heavy user loads and frequent adhoc queries. Users – mainly meteorologists who process weather-related information - work with their data via a specialized application called CSOBANK developed inhouse at DWD. CSOBANK retrieves the required meta data via SQL from the meta data database which, as a next step, returns the requested real data from the file system, which holds up to 300 TB of data on disk as well as additional 1.5 PB on tape.

Weather forecasting is a very time critical business. There is nothing more outdated than yesterday’s weather forecast for today. For the new extremely short-range weather ensemble prediction system DWD expects twenty simultaneous weather forecasts will be calculated every three hours. Up to now only one short-range prediction has been calculated every 3 hours; worldwide predictions had only been calculated every 6 hours. Only 10 minute-old data sets are used as foundation configurations for highlycomplex calculations up to 4 GB in size. The input stream of external and calculated data that enters the filesystem and also involves a refreshment of meta data is estimated up to 800MB/s peak. This represents an ever growing challenge to the DWD team: Bottlenecks through simultaneous extensive queries by a growing number of users were already causing delays in response times. In order to come up to the new challenge of extremely short-range weather forecasts a very substantial rise in performance was necessary.

Requirements:

• Analytical and storage capabilities for increasing number of users and increasing analytic requirements
• Ability to consolidate multiple large and complex databases
• Increased performance and throughput in mixed workload and complex query environment
• Future system expansion: cost effective and with minimum time effort
• Rapid deployment

Solution:

• SGI® Adaptive Data Warehouse for meteorological metadata running on Oracle® 10g, built for exceptional scalability and throughput
• Rapid deployment: the installation was completed within three months with the support of SGI experts for data warehousing projects, simplified customizations and simplified solution architecture and deployment
• Flexible configurations that enable DWD to easily grow the system; Altix can readily adapt to match individual requirements and changing needs
• Minimum platform expansion costs through sharedmemory SGI® NUMAflex® architecture: configurations can add processors, memory and I/O independently
• Longevity and protection of technology investments ensured through industry standard components
• Total of 2.8 TB of memory allow ultrafast in-memory processing on five SGI Altix 4700 data base servers and two SGI® Altix® 450 test servers: a total of 472 Intel® Itanium® cores

New Capabilities:

• Timely decision-making for meteorologists even with large increases in complexity
• Significantly shorter time to insight with up to 15x faster query results
• Increasingly varied workloads can be handled
• Faster query results with the large increases in complexity
• Increased ability to quickly make informed decisions

Image courtesy of Deutscher Wetterdienst.