What is fit?
Model fit (or "goodness of fit) measures how good a job an estimated statistical model fits the data used to estimate the parameters of the model.
To take a familiar example, we might collect a series of data on a response (Y) and a paired predictor (X) and fit a linear regression model: Y^ =b0 +b1X. The parameters of this model are b0 and b1 and ones we have estimates of the parameters we can plug X back into the prediction equation and predict Y for each value of X. For example, if our estimates are b0=3 and b1=1.5 , then the model predicts Y^=7.5 when X=3. If our observation of Y is 8.1, then the difference Y-Y^ = 7.5-8.1 = - 0.6 is a measure of model fit known as a residual. The basis of common estimation procedures such as least squares is to find b0 and b1 to minimize these residuals over all the data, but there will always be some residual variation (otherwise the model would perfectly predict every data point).
We will apply these ideas more specifically to CMR data, where the observations are the capture histories, and where prediction is based on our assumptions about the underlying system (e.g., closed or open), parameters of interest (e.g., Phi and p), the design (CJS, Robust, etc.), and group and other effects such as sex and age structure.
Why might a model not fit?
There are a number of reasons why a model might not fit the data used to estimate the parameters. Here are some that are relevant to CMR modeling:
Model mis-specification/Data contain structure not in model (age, groups, heterogeneity, etc)
This will occur if the model simply doesn't capture the essentially elements in the data ('lacks structure') or if the model contains this elements but expresses them incorrectly ("mis-specification"). Examples:
Survival and capture vary between sexes but we fit a model that ignores sex effects
The population is open but we fit a closed abundance estimation model
The animals in the sample have heterogeneous capture probabilities and we fit a model that assumes that capture probabilities do not vary among individuals
The obvious solution in each of these cases is to try to anticipate how the data are going to vary, and collect the data in a way that allows identification of the relevant factors, and start with a model that takes these sources of variability into account. Specifically in the above cases:
Mark sufficient number of each sex, keep track of recaptures by sex, and start with a model that includes sex (along with time or other relevant structure)
Either reduce the sampling to a period where closure can be assumed, or use and open or Robust Design model
Fit a closed model that allows for individual heterogeneity (e.g., finite mixture or Huggins covariate model)
Latent (unobservable) structure
In some cases, structure exists in the data but we can't see it. For example, suppose that there are really 3 age classes with different survival: birth year, first year ('subadult'), and second year and later ('adult') but adults and subadults cannot be distinguished at first capture (this is common in birds). In this case we might fit a model allowing for 2 age classes but there are really 3, and consequently our model may not fit very well.
"Extra binomial" variation
Finally, even if we have perfectly captured the "structure" of the data with age, sex, time and other effects including covariates, the model will still typically lack perfect fit. To some degree this is expected (these are after all random samples) and is consistent with binomial sampling variation. Very simply, if we were to take 10 random binomial samples with n=10 and p=0.5, we might get results for the number of successes of x= 5 6 3 4 5 8 5 6 6 7, which is entirely consistent with binomial sampling variation. If we were to get instead x= 10 0 10 0 10 0 10 0 10 0, we would still on average have have p=0.5, but the variance of x would exceed (by quite a bit) what would be expected under the binomial model. Non-independence, random time or other effects, and other factors can all produce what is known as extra-binomial variation (EBV) count (e.g., binomial or Poisson) models.
EBV can be estimated by some methods discussed below. If EBV is "moderate" , we will make adjustments to likelihood values, AIC, CIs etc. using variance inflation factors.
There is no bright line between EBV and structural lack of model fit, other than to say that excessively large values of EBV probably are cause to revisit model structure or to fit random effects models. We will revisit this topic with specific examples, below.