Predictions for events with significant inherent uncertainty should be probabilistic in nature to convey information on the uncertainty associated with the outcome. This holds, in particular, for settings where the prediction is subsequently used by many different users to derive further predictions for both expected outcomes and associated risks. Examples of such predictions include weather and climate forecasts such as predictions of extreme precipitation and flooding. We thus take a probabilistic view and assume that forecasts are given as predictive distributions. Evaluation of extreme forecasts then falls in three distinct categories, depending on the question being asked: 

1. A probabilistic forecast is issued for the extremes only and we want to know how good it is. 

2. A probabilistic forecast is issued for every type of outcome, and we want to know how good it is at predicting extreme outcomes. 

3. A probabilistic forecast is issued for every type of outcome, and we want to know how well certain tail properties or functionals of the predictive distribution match those of the true data distribution.

When predicting extreme events and assessing risk, the evaluation of the forecasts is additionally complicated by a lack of substantial observation set due to the rarity of the outcome of interest. We discuss how to perform the evaluation for all three categories above under these constraints within the frameworks of proper scoring rules and consistent scoring functions, and review the available literature on these topics. 

It is well recognized that classical cross validation (CV) methods such as leave one out CV (LOOCV) are often overly optimistic when applied to geostatistical models, underestimating their predictive error, yet there is still no consensus as to which CV method should be used and when. Moreover, many recent proposals for CV methods focus on reducing or accounting for spatial correlation between the left out observations and the in-sample data used to make predictions rather than on estimating the frequent target of interest: the spatial average of prediction error within the spatial domain. In this talk, I propose a method for CV in a geostatistical interpolation context I call spatial importance weighted CV, where importance weights are used within a CV framework to produce an estimator for the spatial average of prediction error (mean squared error or another scoring metric or rule) within the spatial domain that reduces bias compared to LOOCV. Importance weights are estimated via a Voronoi estimator requiring no tuning parameters, leading to a variation on spatial block CV estimators.

Template Model Builder (TMB) is a general tool for efficiently setting up highly parameterized models, also including random effects. TMB provides derivatives of the implemented function, enabling fast inference. Additionally, Markov structures are utilized to quickly marginalize over latent effects using the Laplace approximation.

At the course we will introduce TMB and focus on

1. Latent effect models

2. Spatial modeling

3. Non-standard models 

4. Model validation in TMB

During this course we will use the RTMB package, and the only programming language we will use is R.