Ames 2019

Appraisal of: Ames H, Glenton C, Lewin S. Purposive sampling in a qualitative evidence synthesis: a worked example from a synthesis on parental perceptions of vaccination communication. BMC Medical Research Methodology 2019;19:26

Reviewer(s): 

Andrew Booth

Full Reference:
Ames H, Glenton C, Lewin S. Purposive sampling in a qualitative evidence synthesis: a worked example from a synthesis on parental perceptions of vaccination communication. BMC Medical Research Methodology 2019;19:26

Short description: 

This methodological paper describes the development and application of a three-step purposive sampling framework for a Cochrane qualitative evidence synthesis examining parental perceptions of vaccination communication. The authors developed this approach to address the challenge of managing an excessively large number of eligible studies (79) that threatened the quality of their analysis.

The sampling framework prioritized studies across three sequential steps: (1) all studies from low and middle-income countries (n=9); (2) studies scoring four or more on a newly developed 5-point data richness scale (n=24); and (3) studies where objectives most closely matched the synthesis objectives (n=5), resulting in 38 sampled studies. The authors extracted information about country, setting, vaccine type, data richness, and study objectives from each eligible study.

Following completion of the synthesis, the authors conducted a retrospective analysis mapping which sampling step each study was selected in against the number of synthesis findings each study contributed to. This revealed that studies from low and middle-income countries (Step 1) contributed to the fewest findings on average (6 findings, range 2-13), while studies selected for data richness (Step 2) and closeness to synthesis objectives (Step 3) contributed to substantially more findings (both averaging 13 findings).

The paper includes critical reflection on the strengths and limitations of their approach, discussing potential missed studies (particularly regarding migration-related themes), the challenge of assessing data richness, and the implications for the GRADE-CERQual approach. The authors provide recommendations for future qualitative evidence synthesis, including suggestions for incorporating sampling strategies with CERQual assessment components (relevance, adequacy, methodological limitations, and coherence).

Limitations stated by the author(s): 

1. Missed relevant studies: The sampling framework may have overlooked primary studies that did not meet the sampling criteria but would have contributed meaningfully to the synthesis. Specifically, two studies on migration and access to health services did not meet sampling criteria but might have strengthened at least one finding.

2. No direct sampling for population variation: The sampling framework did not directly sample for variation in study populations, which meant some important perspectives may have been underrepresented.

3. Potential inclusion of thin data from LMIC settings: By prioritizing geographic location in Step 1, the authors may have sampled studies from low and middle-income countries that scored only 1 or 2 for data richness while excluding studies from high-income settings with richer data.

4. Lack of cross-checking methods: The authors acknowledge the need for methods to systematically cross-check for under-represented themes that may have emerged from non-sampled studies.

5. Potential limited contribution of LMIC studies: Studies sampled in Step 1 (LMIC settings) contributed to the least number of findings on average, raising questions about the balance between geographic representation and data richness.

Limitations stated by the reviewer(s): 

1. Lack of validation for data richness scale: The 5-point data richness scale was developed specifically for this review without prior validation, inter-rater reliability testing, or objective assessment of its consistency across different reviewers. While the authors acknowledge this and note it has been used in subsequent reviews, the subjective nature of these assessments remains a concern. [Measurement Bias]

2. Single reviewer application: Although the sampling framework was discussed among the review team, there is no indication that the application of sampling criteria (particularly the data richness scoring) was independently assessed by multiple reviewers. This introduces potential inconsistency and unreliability in study selection. [Selection Bias]

3. No prospective comparison: The paper presents a retrospective account of what was already done rather than a prospective comparison of different sampling approaches. A more robust methodological contribution would have involved comparing findings from the sampled versus complete set of studies, or testing different sampling strategies. [Limited Methodological Rigor]

4. Potential for systematic bias in sampling priorities: The hierarchical three-step approach meant that studies not from LMIC settings had to meet higher bars (data richness or objective matching) for inclusion. This predetermined hierarchy may have systematically excluded relevant high-quality studies from high-income settings that didn't score highly on steps 2 or 3. [Selection Bias]

5. Insufficient assessment of sampling impact: While the authors mapped contribution to findings by sampling step, they did not systematically assess whether important themes, perspectives, or findings were completely missed by the sampling approach. The acknowledgment of missed migration studies suggests this may be a broader issue. [Incomplete Analysis]

6. Complexity and reproducibility concerns: The three-step framework with its combination of different sampling strategies is complex and may be difficult for other researchers to replicate consistently. The paper would benefit from clearer decision rules and more explicit guidance on how to handle borderline cases. [Reproducibility Concerns]

7. Tension between stated aims: There is an inherent tension between wanting manageable data (which drives toward fewer studies) and wanting adequate geographic representation and rich data (which might require more studies). The paper does not fully resolve how to optimize this trade-off. [Conceptual Limitation]

8. Limited discussion of alternative approaches: While the paper references other purposive sampling methods from Suri's framework, there is limited critical discussion of why alternative approaches (such as sequential sampling to saturation) were not adopted or tested. [Incomplete Methodological Justification]

Study Type: 

Methodological paper / Worked example

Related Chapters: 

• Qualitative research

• Search strategy development

Tags: 

• Purposive sampling

• Qualitative evidence synthesis

• Systematic review methodology

• Data richness

• GRADE-CERQual

• Study selection

• Sampling framework

• Geographic representation

• Methodological development

• Worked example

• Vaccination communication