Model selection 2

R-squared increases every time you add a term, even if that term is not useful for predicting the outcome. However, adjusted R-squared takes into account the number of parameters in the model and does not necessarily increase just because a new predictor was added.

One is the sum of squared residuals - whether the model fits well. The other is the number of parameters - whether we are likely over-fitting.

The goal of cross-validation is to check whether you are over-fitting.

No!! Don't do this! This will re-estimate the coefficients. But, the whole point is that you estimate the coefficients by fitting the model to the train data, and that you use those exact coefficient estimates to make predictions for the test data and see how close your predictions are to the true outcome values.

If you are over-fitting, the model will fit the train data much better than it will fit the test data. So, the RMSE will be much bigger for the test data than it is for the train data.

Among other things, though the contest lasted three years, the winner was decided by 20 minutes. Netflix didn't end up using the winning algorithm. They tried to set up another contest later, but it was cancelled when they faced law suits about privacy. Now that Netflix centers around streaming video instead of mailing DVDs, the need to predict what movies people would enjoy has decreased.

Last time I checked, a total of $150,000 was available for a data competition about relating code to comments in python notebooks, with a first prize of $50,000.

Yes. It is a weird quantity that does not have a nice interpretation, the way that R^2 does.

See next video.

Perhaps save some of the data for cross-validation. Visualize the data in lots of ways. Consider what you know about the context. Consider transforming one or both variables. Take a look at residual v. fitted value plots. Assess whether a model's assumptions are true. You might run an ANOVA or look at a coefficient p-value to compare two models that you are interested in. Perhaps calculate measures of model fit like R-squared, adjusted R-squared, BIC, or AIC.

This is called "step selection." This is generally not a good idea, because you will miss the context in a way that might make the data less interpretable. For example, if your data set includes the two highly correlated variables "gender" and "hair length" as possible predictors for income, a step selection might choose to include "hair length" instead of "gender" if that reduces the residuals a tiny, tiny bit.