The FLAMINGO project: From High-performance computing to simulating the whole Universe

Summary

• Pillars of modern physics

  • General Relativity: gravity

  • Quantum Field Theory

  • Standard Model of Particle Physics

  • Lambda-Cold Dark Matter (λCDM) model of universe

• Cosmic microwave background: temperature variations in early universe

  • Can describe power spectrum using a 6-parameter model based on cosmological parameters

  • Cosmology tension: model’s predictions differ from astronomical measurements

  • Are we missing laws of physics or is it a measurement error?

• Approach: simulation of the universe to predict how it should look like after evolving from different starting conditions

  • EAGLE: Evolution and Assembly of GaLaxies and their Environments: https://icc.dur.ac.uk/Eagle

    • Gravitational collapse of more dense regions evolved into 

      • filaments and nodes of denser regions and 

      • then into individual galaxies

  • Depending on the different initial conditions the distribution of this filament graph changes

    • Simulations that incorporate different proposals for dark matter make very different predictions

      • Few, massive particles vs Many, light particles

  • Details that we may want to model:

    • Initial conditions: known from CMB

    • Physical dynamics: gravity, hydrodynamics, magnetic fields, radiative transfer, cosmic rays

    • Constituents: dark energy, dark matter, normal matter, other details of matter (stars, planets, etc.)

    • Depending on level of detail this can go from easy to very hard on available supercomputers

• Challenges of developing this simulation

  • Gravity is hard:

    • Very long-range: so need to exchange information among many different simulated objects 

    • Always attractive: errors accumulate during the simulation (additive errors not compensated by subtraction)

    • All scales matter almost equally (hard to approximate)

    • Cheap calculation: inefficient use of available compute resources

  • Gas and stars affect the evolution of the universe

    • Galaxy is emitting hot gas

    • Powered by galaxy center region 1e3 times smaller

    • Powered by a central black hole that is 1e6 times smaller than that

  • Full resolution computation on an exascale computer would take 1e24 seconds = 1e6 times age of universe

• Need software solutions to make this tractable: 

  • Multiscale

    • Multi-grid

    • Particle splitting

    • (semi-)Lagrangian

    • Focus most compute on one single region

  • Load varies in both time and scale, to need to dynamically adapt numerical scheme based on current state of the simulation

    • Dynamic re-meshing

    • Task-based parallelism

    • But this means that the computer spends more time on management logic at the expense of less computation

• Example: Astro-SPH

  • Task-based parallelism

  • Threads work on regions of space that are available

• Modeling astrophysical details

  • Large scale universal structures can be fully simulated

  • Small scale structures need to be approximated via sub-grid models

    • cooling/heating of gas

    • Star formation

    • Enriching of gas from stars

    • Black holes

    • ….

  • These models are approximate, so hard to capture their error or how it pollutes the dynamics of the larger simulation

    • Simple empirical parameter-fit models

    • Machine learning approach: 

      • Run simulation of target phenomenon and train an ML model on its dynamics

      • ML model is approximate and very cheap to run

      • Use ML model to simulate small-scale phenomena

    • Observational approach: take observations of relevant phenomenon

• FLAMINGO project: https://flamingo.strw.leidenuniv.nl/

  • Models many phenomena simultaneously: gravity, gases, stars, black holes

  • Calibration of sub-grid models

    • Run many simulations with different parameters for the sub-grid models

    • Use Gaussian processes to compute the probability distribution on the most likely values of these parameters

    • Validated against observational data

    • Fine-grained simulations not used for validation because the basic physics of stars, black holes, etc. are still being developed

  • Computation Cost:

    • 42 days on 30k CPUs

    • 31m CPI hours

    • 145 MWh of power use 

    • £20k cost

    • 3.9T of CO2

  • Validation against observational data must account for the properties of telescopes, atmosphere, interference from the moon (affects the distribution of sky patches that could be observed), etc.

  • Outcome: predictions still differ from the observations, so more work to be done on understanding the physics that drive the universe

• Debugging

  • For solvable/standard parts of the PDEs can compare to reference solutions or other simulations

  • To figure out which sub-grid models are most responsible for the error, need to look for which parts of the prediction are wrong

  • There are questions about whether the major laws (e.g. gravity) need revision. Simulation allows modeling of alternatives.