Is the United States' improvement from 2003 to 2007 better than it appears due to the affect of Simpson's Paradox? Specifically, if we considered the change in population composition, and we looked at the performance of subsets of our United State's population (SES, ethnicity, etc.), would we find that the effectiveness and output of education has improved more than the report indicates, but that the degree of improvement is not evident in the report due to a higher proportion of lower performing groups within the whole at 2007?
The commissioners' report on this study (http://nces.ed.gov/whatsnew/commissioner/remarks2008/12_9_2008.asp) provides a little insight into performance by ethnicity and SES but it is not comprehensive. Gleaning from remarks in the commissioner's report, generally speaking the United States' improvement from the previous study to 2007's appears to be driven by non-ethnic and non-poverty populations. The commissioner’s report singles out high-poverty, black, and Hispanic populations in several comments indicating lackluster performance.
Therefore, this is not a perfect example of Simpson's Paradox, because an overall improvement in subgroups does not appear to be masked by the larger proportion of the lower-performing group to the total population. However, a differential in the performance of subgroups, masked in the overall results, is indicated by the commissioner's report.
Implications are two-fold. First, while very interesting and useful, we must not blindly generalize the overall results of this study (or others) without a reasonable understanding of its components (performance by subgroups within the whole). Second, and perhaps more important, what do these results say about teaching in the United States? Asked more specifically, why are we achieving more improvement in the more advantaged categories while lagging in the others? Educators will need to grapple with this to maintain and improve performance metrics as proportions of these relatively disadvantaged populations continue to increase in our schools.
Readings - Issues in Math & Science Curriculua > National Survey of Science and Mathematics Education: Status of High School Physics Teaching (2000) >