Title: Towards the combination of physical and data-driven forecasts for Earth system prediction
Due to the recent success of machine learning (ML) in many prediction problems, there is a high degree of interest in applying ML to Earth system prediction. However, because of the high dimensionality of the system, it is critical to use hybrid methods which combine data-driven models, physical models, and observations. I will present two such hybrid methods: Ensemble Oscillation Correction (EnOC) and the multi-model ensemble Kalman filter (MM-EnKF). Oscillatory modes of the climate system are one of its most predictable features, especially at intraseasonal timescales. It has previously been shown that these oscillations can be predicted well with statistical methods, often with better skill than dynamical models. However, they only represent a portion of the signal, and a method for beneficially combining them with dynamical forecasts of the full system has not previously been developed. Ensemble Oscillation Correction (EnOC) is a method which corrects oscillatory modes in ensemble forecasts from dynamical models. I will show results of EnOC applied to forecasts of South Asian monsoon rainfall, outperforming the ECMWF ensemble on subseasonal-to-seasonal timescales.
A more general method for combining multiple models and observations is multi-model data assimilation (MM-DA). MM-DA generalizes the variational, Bayesian, and minimum variance formulation of the Kalman filter. Here, I will show how multiple model ensembles can be combined for both DA and forecasting in a flow-dependent manner using a multi-model ensemble Kalman filter (MM-EnKF). This methodology is applied to multiscale chaotic models and results in significant error reductions compared to the best model and to an unweighted multi-model ensemble. Lastly, I will discuss the prospects of using the MM-EnKF for hybrid forecasting.
Title: Towards structure preserving discretizations of stochastic rotating shallow water equations on the sphere
We introduce a stochastic representation of the rotating shallow water equations and a corresponding structure preserving discretization. The stochastic flow model follows from using a stochastic transport principle and a decomposition of the fluid flow into a large-scale component and a noise term that models the unresolved flow components. Similarly to the deterministic case, this stochastic model (denoted as modeling under location uncertainty (LU)) conserves the global energy of any realization.
Consequently, it permits us to generate an ensemble of physically relevant random simulations with a good trade-off between the representation of the model error and the ensemble’s spread.
Applying a structure-preserving discretization of the deterministic part of the equations and standard finite difference/volume approximations of the stochastic terms, the resulting stochastic scheme preserves (spatially) the total energy. To address the enstrophy accumulation at the grid scale, we augment the scheme with a scale selective (energy preserving) dissipation of enstrophy, usually required to stabilize such stochastic numerical models. We compare this setup with one that applies standard biharmonic dissipation for stabilization and we study its performance for test cases of geophysical relevance.
Title: Scale-aware statistical space-time characterization of sub-grid air-sea exchange variability
We present a statistical scale-aware space-time model for the sub-grid variability of air-sea exchanges driven by surface wind speed. Quantifying the influence of the sub-grid scales on the resolved scales in physics-based models is needed to better represent the entire system. In this work, we evaluate and model the difference between the true turbulent fluxes and those calculated using area-averaged wind speeds. This discrepancy is modelled in space and time, conditioned on the low-resolution fields, with the view of developing a stochastic wind-flux parameterization. A locally stationary space-time Gaussian process is used to model this discrepancy process. Additionally, the Gaussian process is proposed in a scale-aware fashion meaning that the space-time correlation ranges depend on the considered resolution. The scale-aware capability is based on empirical observations from a systematic coarse-graining of a high-resolution model output dataset. It enables to derive a stochastic parameterization of sub-grid variability at any resolution and to characterize statistically the space-time structure of the discrepancy process across scales.
Title: On the crossing-point forecast
The crossing-point forecast (CPF) is a new concept in the field of probabilistic forecasting. A CPF is defined by the intersection between a forecast cumulative distribution and the corresponding climatology distribution. Focusing on this intersection point, a probabilistic forecast is summarized into a single number conveying information about a "probabilistic worst-case scenario" with respect to climatology. Is the predicted chance of suffering a loss, due to the occurrence of an (exceedance) event, higher than that event’s climatological frequency? The crossing-point forecast indicates the limit case for which the answer is positive.
The outcome corresponding to a CPF, called “crossing-point observation”, is directly related to the return period of the event that materializes. A simple error function that applies to the forecast and observed crossing-points is formulated. The resulting score is closely related to the diagonal score: it is proper and equitable which makes its application appealing for the comparison of competing forecasts.
The proposed scoring function is consistent for the crossing-point forecast in a similar way as the root mean squared error is consistent for the distributional mean forecast or the mean absolute error is consistent for the 50%-quantile forecast.
In weather forecasting, the information provided by CPF could be highly relevant for vulnerable users and more generally for users with interest for high-impact events. We propose here a comparison with the Extreme Forecast Index (EFI) using the ensemble forecast of the Integrated Forecasting System run at ECMWF and the corresponding model climatology. The EFI is designed to provide forecasters with general initial guidance on potential extreme weather events. Both EFI and CPF are derived using the same ingredients which makes their comparison particularly relevant. Based on case studies and verification metrics, we illustrate the complementarity of the two types of forecasts.
Titles: (1) Exponential stability of the optimal filter for signal processes with hyperbolic dynamics, (2) What is the correct cost functional for 4DVar? (3) Statistical forecast evaluation under serial dependence
Title: Inferring the instability of a dynamical system from the skill of data assimilation exercises
Title: Computer, how likely is it that I need my coat tomorrow? How neural networks can be used for both probabilistic weather forecasting and post-processing of NWP models
The success of machine learning techniques over the years, and in particular neural networks, has opened up a new avenue of research for weather forecasting. However neural networks suffer as decision-making tools because they lack the ability to express uncertainty. Here we show how this problem can be alleviated by transforming continuous data to categorical data. Specifically, we use neural networks to easily generate probabilistic data-driven forecasts of geopotential at the 500hPa level and the temperature at the 850hPa level, using the WeatherBench dataset (a processed version of the ERA5 reanalysis dataset regridded onto a coarse resolution). Furthermore, by using a combination of variable importance analysis and ensemble modelling, we show that our data-driven neural network approach can achieve better results than both some more complex neural networks and some simple NWP models. However, our approach is not more accurate than the existing operational ECMWF IFS model. Therefore, in the second part of this talk, we present ongoing work illustrating how neural networks can be used for post-processing to improve predictions from NWP models. In particular, we show how the relatively new technique of Bayesian Neural Networks may help to improve ensemble generation and uncertainty quantification of NWP models.
Title: Particle filters with nudging. Applications to Data Assimilation
This talk covers some recent work on developing particle filters based data assimilation methodology for high dimensional fluid dynamics models. The algorithm presented here is a particle filter with a so-called ”nudging” mechanism. The nudging procedure is used in the prediction step. In the absence of nudging, the particles have trajectories that are independent solutions of the model equations. The nudging presented here consists in adding a drift to the trajectories of the particles with the aim of maximising the likelihood of their positions given the observation data. This introduces a bias in the system that is corrected during the resampling step. The nudging procedure is theoretically justified through a standard convergence argument.
The corresponding Data Assimilation algorithm presented gives an asymptotically (as the number of particles increases) consistent approximation of the posterior distribution of the state given the data. The methodology is tested on a two-layer quasi-geostrophic model for a beta-plane channel flow with O(10^6) degrees of freedom out of which only a minute fraction are noisily observed. I will present the effect of the nudging procedure on the performance of the data assimilation procedure for a reduced model in terms of the accuracy and uncertainty of the results. The results presented here are incorporated in [1] and [2]. The talk is based on the papers:
[1] C Cotter, D Crisan, D Holm, W Pan, I Shevchenko, Data assimilation for a quasi[1]geostrophic model with circulation-preserving stochastic transport noise, Journal
[2] D Crisan, I Shevchenko, Particle filters with nudging, work in progress.
Title: Validating and improving the uncertainty assumptions for the assimilation of ocean--colour--derived chlorophyll into a marine biogeochemistry model
The correct specification of all sources of uncertainty is critical to the success of data assimilation (DA) in improving the realism and accuracy of forecasts and reanalyses. This work focuses on improving the uncertainty assumptions made during the assimilation of Ocean Colour derived chlorophyll into an operational marine coupled physical-biogeochemical DA system, which produces daily biogeochemistry forecasts on the North-West European Shelf Seas.
Analysis of the observation-model misfits shows significant biases in the chlorophyll, which strongly vary with season. The behaviour of these misfits agrees well with previous studies and can be attributed to systematic errors within the coupled model. Diagnostic metrics, frequently used within Numerical Weather Prediction, are applied to separate out the random component of the observation and model errors, allowing for the derivation of new error covariance matrices. These new error covariance matrices are then modified to account for the biases in the model that cannot be treated explicitly within the operational DA system. This has the effect of inflating both the error variances and the correlation length-scales. Experiments show that the new error covariances can result in significant improvements in the accuracy of the analysis and forecast.In particular, the new error covariance matrices reduce the bias in the Spring phytoplankton bloom present when using the previous error covariances. Validation against independent glider observations in the North Sea also shows reductions in bias in chlorophyll and oxygen that extend below the surface to the depth of the mixed layer. Accounting for the biases in the model in the error correlations can lead to much larger improvements than not accounting for them, however, there are also regions where large degradations are seen that may indicate model instabilities. This may be improved by estimating the bias separately for the different regions on the shelf.
Title: The Effective Use of Anchor Observations in Variational Bias Correction in the Presence of Model Bias
In numerical weather prediction, satellite radiance observations have a significant impact on forecast skill, but the data must be bias corrected before assimilation. Many operational centres use Variational Bias Correction (VarBC) to correct the biases in the satellite data. In VarBC it is assumed that model biases are negligible, however, this is often not the case. Unbiased (anchor) observations therefore play an important role in providing an independent estimate of the unbiased model state.
This work presents a systematic study of the properties of the network of anchor observations needed to reduce the contamination of model bias.
This allows VarBC to be robust to the continual growth of the satellite network.
We extend the theory of VarBC to include both bias-corrected and anchor observations, to find that the precision and the location of the anchor observations are important in reducing the contamination of model bias in the estimate of observation bias. Anchor observations work best at reducing the contamination of model bias in the observation bias correction when they observe the same states as the bias-corrected observations. When this is not the case, strong background error correlations allow information about the model bias to be passed from the anchor observations to the bias-corrected observations. These results show that, in operational systems, satellite data in regions with sparse unbiased observations could be more susceptible to model biases.
We demonstrate these results in a series of idealised numerical experiments that use the Lorenz 96 model as a simplified model of the atmosphere.
Title: Probabilistic forecasting to model uncertainty in Climate data for Weather Index Insurance
Weather Index Insurance predominantly depends on seasons and weather, hence a reliable and accurate weather forecasting model is significant to the insurance industry. However, the changes in climate do not only make weather more severe but also harder to forecast. For measuring the uncertainty in temperature forecast, a probabilistic forecasting model based on variational mode decomposition technique (VMD), quantile random forest, and a kernel density function (VMD-QRF-K) is constructed. VMD decomposes the temperature signal into intrinsic mode functions (IMFs) and the non-linear structure of random forest is applied to build the non-linear quantile regression forecast model. The kernel density is used in the density estimation. A case study using average temperature data from Ghana and Kenya are presented to illustrate the efficiency and robustness of the proposed technique.
Title: High dimensional particle filtering using lagged approximations
In the context of data assimilation and high dimensional models it is well known that naïve implementations of particle filters can have a cost that scales exponentially with the dimension of the hidden state. Many approaches have proposed to address this such tempering, blocking, using coordinates as time, localisation and others. One way to intuitively explain why these methods are effective is that they mitigate or break the correlations introduced by resampling. We will present a new approach that achieves this based on a deterministic lagged approximation of the smoothing distribution that is necessarily biased. The method is inspired by lag-approximation methods for the smoothing problem introduced by Kitagawa and Sato. For certain classes of state space models, particularly those that forget the initial condition exponentially fast in time, the bias of our approximation is shown to be uniformly controlled in the dimension and exponentially small in time. We develop a sequential Monte Carlo (SMC) method to recursively estimate expectations with respect to our biased filtering distributions. Moreover, we prove for a class of models that can contain dependencies amongst coordinates that as the dimension d grows the cost to achieve a stable mean square error in estimation, for classes of expectations, is of O(N d^2) per-unit time, where N is the number of simulated samples in the SMC algorithm. We will present results in challenging high-dimensional examples including the conservative shallow-water model.
This is joint work with Hamza Ruzayqat, Aimad Er-Raiy, Alexandros Beskos, Dan Crisan, and Ajay Jasra.
Title: The impact of hybrid oceanic data assimilation in a coupled model: A case study of a tropical cyclone
Tropical cyclones tend to result in distinctive spatial and temporal characteristics in the upper ocean, which suggests that traditional, parametrisation-based background-error covariances in oceanic data assimilation (DA) may not be suitable. Using the case study of Cyclone Titli, which affected the Bay of Bengal in October 2018, we explore hybrid methods that combine the traditional covariance modelling strategy used in variational methods with flow-dependent estimates of the ocean's error covariance structures based on a short-range ensemble forecast. This hybrid approach is investigated in the UK Met Office's state-of-the-art system. Single-observation experiments in the ocean reveal that the hybrid approach is capable of producing analysis increments that are time-varying, more anisotropic and vertically less uniform. When the hybrid oceanic covariances are incorporated into a weakly coupled DA system, the sea-surface temperature (SST) in the path of the cyclone is changed, not only through the different specifications of background-error covariances used in the SST assimilation, but also through the propagation of subsurface temperature differences to the surface as a result of vertical mixing associated with the cyclone's strong winds. The coupling with the atmosphere then leads to a discrepancy in the cyclone's central pressure, which brings forth further SST differences due to the different representations of the cyclone's emerging cold wake.
Tsz Yan Leung, Amos S. Lawless, Nancy K. Nichols, Daniel J. Lea, Matthew J. Martin
Title: Randomised Bayesian inference and stochastic representations of model error
Many models of natural phenomena are subject to uncertainty. Two important sources of uncertainty are parameters of interest that are not known precisely, and model error. 'Model error' may be due to errors in the mathematical model itself, such as unresolved scales or omitted nonlinearities, or numerical approximations that are made to improve the efficiency of a computational method. The uncertainty due to unknown parameters can be addressed by statistical inference. The uncertainty due to model error can be tackled by using stochastic representations, or equivalently, by using ensemble-based computation.
In this talk, we will present an approach for combining Bayesian inference for unknown parameters with stochastic representations of model error. The key idea of this approach is to randomise the Bayesian posterior probability measure, i.e. to use an ensemble of posterior measures that corresponds to a stochastic representation of model error. We will state error bounds for random posterior measures and illustrate the significance of these bounds, using time integration for differential equations as a guiding example.
Title : Mathematical Properties of Continuous Ranked Probability Score Forecasting
The theoretical advances on the properties of scoring rules over the past decades have broaden the use of scoring rules in probabilistic forecasting. In meteorological forecasting, statistical postprocessing techniques are essential to improve the forecasts made by deterministic physical models. Numerous state-of-the-art statistical postprocessing techniques are based on distributional regression evaluated with the Continuous Ranked Probability Score (CRPS). However, theoretical properties of such minimization of the CRPS have mostly considered the unconditional framework (i.e. without covariables) and infinite sample sizes. We circumvent these limitations and study the rate of convergence for a given class of distributions. Moreover, we show that the k-nearest neighbor method and the kernel method for the distributional regression reach the optimal rate of convergence in dimension d greater or equal to 2 and in any dimension, respectively.
Title: Score generative models for Bayesian inference
I will consider inference problems for which a prior distribution can only be characterise via available samples without access to their underlying probability distribution. Score generative models have recently become popular in the context of image generation. Related ideas can be utilised in the context of Bayesian inference and give rise to a novel class of approximate Langevin-type sampling methods. This is joint work with Georg Gottwald and Jakiw Pidstrigach.
Title: Bringing physical reasoning into statistical practice in forecast verification on S2S timescales
There is a longstanding disconnect between physical reasoning and statistical practice in mainstream climate science. This disconnect extends to forecasting on subseasonal to seasonal timescales, which generally employs a frequentist definition of probability, counting members of a forecast ensemble. Such an approach struggles in the face of systematic model error, the inevitably sparse sampling in a finite ensemble, and limited verification data. In this talk I reflect on this historical disconnect and illustrate a few ways in which physical reasoning can be brought into the statistical framing of the forecast verification problem, through Bayesian reasoning and causal networks.
Title: Evaluating probabilistic forecasts of extremes using continuous ranked probability score distributions
Verifying probabilistic forecasts for extreme events is a highly active research area because popular media and public opinions are naturally focused on extreme events, and biased conclusions are readily made. In this context, classical verification methods tailored for extreme events, such as thresholded and weighted scoring rules, have undesirable properties that cannot be mitigated, and the well-known continuous ranked probability score (CRPS) is no exception.
Here, we define a formal framework for assessing the behavior of forecast evaluation procedures with respect to extreme events, which we use to demonstrate that assessment based on the expectation of a proper score is not suitable for extremes. Alternatively, we propose studying the properties of the CRPS as a random variable by using extreme value theory to address extreme event verification. An index is introduced to compare calibrated forecasts, which summarizes the ability of probabilistic forecasts for predicting extremes. The strengths and limitations of this method are discussed using both theoretical arguments and simulations.
Title: Seasonal prediction using ECMWF’s global forecasting system
In this presentation I will give a brief introduction into seasonal forecasting and sources of long-range skill. Using ECMWF’s global forecasting system, I will provide a short overview of the current state-of-the-art and highlight some challenges related to variations in predictability and forecast performance in the tropics and extratropics.
Title: Long-term prediction using partial perfect observations and linear response theory
Chaotic systems are typically mixing, meaning that any "fuzzy" information gained from observing the system's state washes away over time, typically at a characteristic exponential rate. It is to be expected that improving the accuracy of our observations should improve future predictions, but in the natural case of partial observations it is not known what the ceiling of this improvement should be. Considering some toy models, we find that for all but "obvious" counterexamples, the predictive utility of perfect partial observations decays over time, at a slower characteristic exponential rate. This is a new result in the mathematical literature, and furnishes a mechanism for the emergence of linear response theory in larger systems.
Title: Stochastic modeling of a deterministic multi-scale Lorenz '96 dynamical system
A classical approach to the parametrization of subgrid-scale processes is to assume a large time scale separation between slow climate and fast weather processes. This assumption leads to a reduction of the full system to a set of slow variables that are impacted by a noise that parametrizes the fast processes. This type of reduction method has been called "Hasselman's program" or the "MTV approach" (after Majda, Timofeyev and Vanden-Eijnden) in the climate community and "homogenization" in the mathematical community.
Homogenization is an extension of the central limit theorem (CLT) to dynamical systems: we obtain convergence to a stochastic process with Gaussian white noise.
The Edgeworth expansion provides a description of deviations from the CLT behaviour. It can be used in the setting of time-scale-separated dynamical systems to develop extended parametrizations that more accurately describe the statistics of slow-fast systems than the limiting homogenized equations.
This is joint work with Georg Gottwald.