Suppose that you are evaluating two delivery companies. Your investigation shows that Company B has a better on-time rate for small packages and also for large packages. Despite Company B's performance, however, Company A has a better overall on-time rate. This situation exemplifies Simpson's Paradox, in which the judged relationship between two variables (e.g., company and performance) differs depending on whether that relationship is viewed within subcategories of a third variable (e.g., package size) or in the aggregate. A normative analysis is presented arguing that the reasonableness of using the third variable depends upon the sample as well as the separation between and variability within categories. To test subjects' abilities to behave appropriately in Simpson's Paradox situations, we examined responses to variations in these factors in five studies. Results showed that subjects had little or no sensitivity to differing stimulus set sizes. Also, subjects were sensitive to relationship strength and the variability within and between groups, and in non-normative ways. Subjects' judgment behavior is related to a broader perspective concerning selection among multiple available levels of analysis based on a consideration of argument strength.