Research Software Engineer - Post-processing of Machine Learning Model Data

at  ECMWF

ABOUT ECMWF

The European Centre for Medium-Range Weather Forecasts (ECMWF) is a world-leader in weather and environmental forecasting. As an international organisation we serve our members and the wider community with global weather predictions and data that is critical for understanding and solving the climate crisis. We function as a 24/7 research and operational centre with a focus on medium and long-range predictions, holding one of the largest meteorological data archives in the world. The success of our activities builds on the talent of our scientists and experts, strong partnerships with 35 Member and Co-operating States and the international community, some of the most powerful supercomputers in the world, and the use of innovative technologies and machine learning across our operations. ECMWF is a multi-site organisation, with a main office in Reading, UK, a data centre/supercomputer in Bologna, Italy, and a large presence in Bonn, Germany. ECMWF has also developed a strong partnership with the European Union and has been entrusted with the implementation and operation of the Destination Earth Initiative and the Climate Change and Atmosphere Monitoring Services of the Copernicus Programme. Other areas of work include High Performance Computing and the development of digital tools that enable ECMWF to extend provision of data and products covering weather, climate, air quality, fire and flood prediction and monitoring.
See www.ecmwf.int for more info about what we do.

EDUCATION AND EXPERIENCE

• Advanced university degree (EQ7 level or above) or equivalent professional experience in computer science or engineering, computational science, physics or natural sciences, mathematics, or a related discipline.
• Demonstrated experience developing object-oriented software in Python.
• Experience designing complex data processing workflows.
• Experience developing scientific software to process large datasets.
• Experience in C++ development and interoperability between Python/C++ is a strong advantage.
• Experience developing or operating software in high-availability operational environments would be advantageous.
  Don’t be discouraged if you do not meet all the requirements. ECMWF invests in people with training and multiple opportunities for development.

SKILLS AND KNOWLEDGE

• Competence in Object-Oriented Analysis and Design, preferably in Python.
• Familiarity with software lifecycle maintenance is highly desirable.
• Demonstrated ability programming in UNIX/Linux systems.
• Ability to write software in a distributed computing or scientific computing environment.
• Knowledge of Machine Learning workflows and processing on GPUs would be advantageous.
• Candidates must be able to work effectively in English.
• Good knowledge of one of the Centre’s other working languages (French or German) would be an advantage.

YOUR ROLE

We are looking to hire a highly motivated Research Software Engineer (A2) to work on the development of data processing services for machine learning within Destination Earth (DestinE). This role is an integral part of a dynamic team, consisting of scientists and software engineers contributing to building ECMWF’s next generation of weather forecasting systems.
At ECMWF, you will find a passionate and diverse community, collectively aiming to bring novel technology and science to the cutting-edge of numerical weather prediction. ECMWF is one of the three organisations entrusted to deliver Destination Earth, a flagship initiative of the European Commission to develop a highly accurate digital model of the Earth on a global scale. In the framework of ECMWF’s contribution to the Destination Earth initiative of the European Commission (DestinE), you will work on the Digital Twin Engine (DTE), a set of interoperable software components and services which support end-to-end execution of the Digital Twins.
With the recent breakthrough in in AI-driven weather forecasting, it becomes clear that AI will play a key role in the next generation of forecasting systems. To this end, ECMWF is building a dedicated multi-disciplinary group to tackle these challenges. ECMWF has been the first operational weather centre to publish results of their own global machine learned weather model – the Artificial Intelligence Forecasting System (AIFS). Within DestinE, ECMWF will now develop and deploy workflows of machine-learned Earth-system components of a European foundation model. To leverage the enhanced interactivity of AI systems, the DestinE Digital Twin Engine will be further developed for efficient data output from machine learning systems.
In this role, you will explore and implement innovative solutions to adapt our data post-processing services creating workflows for machine learning systems developed in DestinE. You will be expected to bring creative solutions for handling large AI-generated output and ensuring the scalability and performance of the Digital Twin forecasting pipeline. This would include adapting the automated workflows to allow on-the-fly processing of AI-generated data on graphics processing units (GPUs).
The role sits in the Data Processing Services team, within the Development Section in the Forecasts and Services Department. The primary focus of the team is to ensure the scalability, performance, and robustness of the operational weather forecasting pipeline. The team is dedicated to navigating challenges posed by future model upgrades. This is done by exploring new technology, such as accelerators and novel storage hardware, and new algorithmic methodologies, including machine learning. This effort supports ECMWF’s strategy of producing cutting‐edge science and world-leading weather predictions and monitoring of the Earth system.

YOUR RESPONSIBILITIES

• Contribute to the global design of the DestinE machine learning post-processing workflow.
• Contribute to the development of the graph-based post-processing framework for DestinE workflows.
• Define the development requirements for dedicated post-processing components to be able to handle machine learning data efficiently.
• Design and implement the interface linking the different components of the DestinE post-processing workflow together.
• Link between different teams regarding the post-processing of ML model output.
• Contribute to the ECMWF and DestinE open-source software stack.