I can only think of two scenarios thus far . . .
Scenario 1: This is a more qualitative example. Let’s say we are interested in gender differences in number of cigarettes smoked per day in a developing country. Because of the social and cultural stigma associate with women smoking, women may refuse to participate in the survey and this led to a lot of missing cases for women.
In this situation, we are assuming that women who responded are about the same as women who did not on the cigarette consumption variable we are trying to estimate. In other words, we are assuming that the frequencies of smoking for women who did not respond would be somewhat similar to women who responded. That’s a critical but untested assumption when we are trying to use weights to adjust for gender differences in the number of daily cigarette consumption.
Weighting will have no effect if there are no gender differences in smoking frequency.
Weighting will be able to improve the estimates if there are gender differences in smoking frequency and if the assumption that women who did not respond are similar to women who responded with respect to daily cigarette consumption. If women who did not responded are very different from women who responded, weight adjustments may make the estimates for gender differences in daily cigarette consumption worse than if the sample data is left unweighted.
Scenario 2: When the n to be selected is too small.
Table 1. Reasonable sample size (n)
As such, the variance of the mean in each stratum and the weighted sum of the variance across strata in Table 2 are inflated. The SE for the total sample are also inflated, though not as much as the SE for the variance of the mean in each stratum.
Since weight adjustments can have the impact of changing one respondent’s answer to a survey question relative to other respondents, it can have a very large impact on the estimates when the sample size is small. Thus,this can lead to unreliable estimates because the sample size may not be sufficiently robust, the variations will be too big and the data may be skewed.
Table 2. Small sample size (n)