Data assimilation

Ghil, Michael. “Data Assimilation: Interesting Past, Bright Future.” CliMathParis 2019, Workshop 2: Big data, data assimilation, and uncertainty quantification, Institut Henri Poincaré, Paris, France, 2019. Workshop website Abstract

Metref, Sammy, Alexis Hannart, Juan Ruiz, M. Bocquet, Alberto Carrassi, and Michael Ghil. “Estimating model evidence using ensemble-based data assimilation with localization - The model selection problem.” Quarterly Journal of the Royal Meteorological Society (2019).
Hannart, A., A. Carrassi, M. Bocquet, Michael Ghil, P. Naveau, M. Pulido, J. Ruiz, and P. Tandeo. “DADA: data assimilation for the detection and attribution of weather and climate-related events.” Climatic Change 136, no. 2 (2016): 155–174. Publisher's Version Abstract

We describe a new approach that allows for systematic causal attribution of weather and climate-related events, in near-real time. The method is designed so as to facilitate its implementation at meteorological centers by relying on data and methods that are routinely available when numerically forecasting the weather. We thus show that causal attribution can be obtained as a by-product of data assimilation procedures run on a daily basis to update numerical weather prediction (NWP) models with new atmospheric observations; hence, the proposed methodology can take advantage of the powerful computational and observational capacity of weather forecasting centers. We explain the theoretical rationale of this approach and sketch the most prominent features of a ``data assimilation–based detection and attribution'' (DADA) procedure. The proposal is illustrated in the context of the classical three-variable Lorenz model with additional forcing. The paper concludes by raising several theoretical and practical questions that need to be addressed to make the proposal operational within NWP centers.

Merkin, V. G., Dmitri Kondrashov, Michael Ghil, and B. J. Anderson. “Data assimilation of low-altitude magnetic perturbations into a global magnetosphere model.” Space Weather 14, no. 2 (2016): 165–184. Publisher's Version
Podladchikova, T. V., Y. Y. Shprits, Dmitri Kondrashov, and A. C. Kellerman. “Noise statistics identification for Kalman filtering of the electron radiation belt observations I: Model errors.” Journal of Geophysical Research: Space Physics 119, no. 7 (2014): 5700–5724. Publisher's Version
Podladchikova, T. V., Y. Y. Shprits, A. C. Kellerman, and Dmitri Kondrashov. “Noise statistics identification for Kalman filtering of the electron radiation belt observations: 2. Filtration and smoothing.” Journal of Geophysical Research: Space Physics 119, no. 7 (2014): 5725–5743. Publisher's Version
Kellerman, A. C., Y. Y. Shprits, Dmitri Kondrashov, D. Subbotin, R. A. Makarevich, E. Donovan, and T. Nagai. “Three-dimensional data assimilation and reanalysis of radiation belt electrons: Observations of a four-zone structure using five spacecraft and the VERB code.” Journal of Geophysical Research: Space Physics 119, no. 11 (2014): 8764–8783. Publisher's Version
Ghil, Michael. “Lecture 1: Data Assimilation: How We Got Here and Where To Next?Workshop on Mathematics of Climate Change, Related Hazards and Risks, CIMAT, Guanajuato, Mexico, 2013. Abstract

Lecture 1: Data Assimilation: How We Got Here and Where To Next?
Daae, M., Y. Y. Shprits, B. Ni, J. Koller, Dmitri Kondrashov, and Y. Chen. “Reanalysis of radiation belt electron phase space density using various boundary conditions and loss models.” Advances in Space Research 48, no. 8 (2011): 1327 - 1334. Publisher's Version
Kondrashov, Dmitri, Michael Ghil, and Y. Shprits. “Lognormal Kalman filter for assimilating phase space density data in the radiation belts.” Space Weather 9, no. 11 (2011).
Carrassi, Alberto, Michael Ghil, Anna Trevisan, and Francesco Uboldi. “Data assimilation as a nonlinear dynamical systems problem: Stability and convergence of the prediction-assimilation system.” Chaos 18, no. 2 (2008): 023112. Abstract

We study prediction-assimilation systems, which have become routine in meteorology and oceanography and are rapidly spreading to other areas of the geosciences and of continuum physics. The long-term, nonlinear stability of such a system leads to the uniqueness of its sequentially estimated solutions and is required for the convergence of these solutions to the system's true, chaotic evolution. The key ideas of our approach are illustrated for a linearized Lorenz system. Stability of two nonlinear prediction-assimilation systems from dynamic meteorology is studied next via the complete spectrum of their Lyapunov exponents; these two systems are governed by a large set of ordinary and of partial differential equations, respectively. The degree of data-induced stabilization is crucial for the performance of such a system. This degree, in turn, depends on two key ingredients: (i) the observational network, either fixed or data-adaptive, and (ii) the assimilation method.

Kondrashov, Dmitri, Chaojiao Sun, and Michael Ghil. “Data Assimilation for a Coupled Ocean–Atmosphere Model. Part II: Parameter Estimation.” Monthly Weather Review 136 (2008): 5062–5076. Abstract

The parameter estimation problem for the coupled ocean–atmosphere system in the tropical Pacific Ocean is investigated using an advanced sequential estimator [i.e., the extended Kalman filter (EKF)]. The intermediate coupled model (ICM) used in this paper consists of a prognostic upper-ocean model and a diagnostic atmospheric model. Model errors arise from the uncertainty in atmospheric wind stress. First, the state and parameters are estimated in an identical-twin framework, based on incomplete and inaccurate observations of the model state. Two parameters are estimated by including them into an augmented state vector. Model-generated oceanic datasets are assimilated to produce a time-continuous, dynamically consistent description of the model’s El Niño–Southern Oscillation (ENSO). State estimation without correcting erroneous parameter values still permits recovering the true state to a certain extent, depending on the quality and accuracy of the observations and the size of the discrepancy in the parameters. Estimating both state and parameter values simultaneously, though, produces much better results. Next, real sea surface temperatures observations from the tropical Pacific are assimilated for a 30-yr period (1975–2004). Estimating both the state and parameters by the EKF method helps to track the observations better, even when the ICM is not capable of simulating all the details of the observed state. Furthermore, unobserved ocean variables, such as zonal currents, are improved when model parameters are estimated. A key advantage of using this augmented-state approach is that the incremental cost of applying the EKF to joint state and parameter estimation is small relative to the cost of state estimation alone. A similar approach generalizes various reduced-state approximations of the EKF and could improve simulations and forecasts using large, realistic models.

Ghil, Michael. “Data Assimilation for the Atmosphere, Ocean, Climate and Space Plasmas: Some Recent Results.” Dept. of Meteorology, University of Reading and the NERC Data Assimilation Research Centre (DARC), 2007. Abstract

Chin, T. M., M. J. Turmon, J. B. Jewell, and Michael Ghil. “An Ensemble-Based Smoother with Retrospectively Updated Weights for Highly Nonlinear Systems.” Monthly Weather Review 135 (2007): 186–202. Abstract

Monte Carlo computational methods have been introduced into data assimilation for nonlinear systems in order to alleviate the computational burden of updating and propagating the full probability distribution. By propagating an ensemble of representative states, algorithms like the ensemble Kalman filter (EnKF) and the resampled particle filter (RPF) rely on the existing modeling infrastructure to approximate the distribution based on the evolution of this ensemble. This work presents an ensemble-based smoother that is applicable to the Monte Carlo filtering schemes like EnKF and RPF. At the minor cost of retrospectively updating a set of weights for ensemble members, this smoother has demonstrated superior capabilities in state tracking for two highly nonlinear problems: the double-well potential and trivariate Lorenz systems. The algorithm does not require retrospective adaptation of the ensemble members themselves, and it is thus suited to a streaming operational mode. The accuracy of the proposed backward-update scheme in estimating non-Gaussian distributions is evaluated by comparison to the more accurate estimates provided by a Markov chain Monte Carlo algorithm.

Ihler, Alexander T., Sergey Kirshner, Michael Ghil, Andrew W. Robertson, and Padhraic Smyth. “Graphical models for statistical inference and data assimilation.” Physica D: Nonlinear Phenomena 230, no. 1 (2007): 72–87.
Kondrashov, Dmitri, Y. Shprits, Michael Ghil, and R. Thorne. “A Kalman filter technique to estimate relativistic electron lifetimes in the outer radiation belt.” Journal of Geophysical Research: Space Physics 112, no. A10 (2007).
Shprits, Yuri, Dmitri Kondrashov, Yue Chen, Richard Thorne, Michael Ghil, Reiner Friedel, and Geoff Reeves. “Reanalysis of relativistic radiation belt electron fluxes using CRRES satellite data, a radial diffusion model, and a Kalman filter.” Journal of Geophysical Research: Space Physics 112, no. A12 (2007).
Kao, Jim, Dawn Flicker, Kayo Ide, and Michael Ghil. “Estimating model parameters for an impact-produced shock-wave simulation: Optimal use of partial data with the extended Kalman filter.” Journal of Computational Physics 214, no. 2 (2006): 725–737.
Kao, Jim, Dawn Flicker, Rudy Henninger, Sarah Frey, Michael Ghil, and Kayo Ide. “Data assimilation with an extended Kalman filter for impact-produced shock-wave dynamics.” Journal of Computational Physics 196, no. 2 (2004): 705–723.
PDF (text only) PDF (figures)
Kao, J., D. Flicker, R. Henninger, Michael Ghil, and K. Ide. “Using extended Kalman filter for data assimilation and uncertainty quantification in shock-wave dynamics.” In Uncertainty Modeling and Analysis, 2003. ISUMA 2003. Fourth International Symposium on, 398–407. IEEE, 2003.