Advanced Statistical Methods II

The most important unifying message of this course is the following: 

         Many modern statistical methods are built by changing one of three things: the data structure, the loss function, 

         or the optimization strategy. 

This line can be understood as follows. 

In elementary statistical modelling, we often begin with a relatively simple situation: the data are fully observed, the response variable is directly available, least squares or maximum likelihood has a simple form, and the resulting estimator can often be written down explicitly. In more advanced statistical problems, one or more of these convenient features breaks down. Sometimes the data structure changes: some values are missing, some event times are censored, or some labels are unobserved. Sometimes the loss function changes: instead of squared error, we may use absolute error, a quantile loss, or a penalized loss. Sometimes the optimization strategy changes: instead of solving an estimator in closed form, we construct iterative algorithms such as EM, MM, Newton’s method, Gauss-Newton, or gradient descent. 

Thus, the course should not be viewed as a collection of unrelated advanced topics. Rather, each topic is a response to a specific statistical difficulty. Missing or latent data leads naturally to EM-type algorithms. Censoring leads to survival likelihoods, Kaplan-Meier estimation, and proportional hazards regression. Multicollinearity or high dimensionality motivates ridge regression, PCR, PLS, LASSO, and LARS. Outliers and non-mean effects motivate LAD, robust regression, and quantile regression. Nonlinear scientific models require nonlinear least squares and iterative numerical optimization.