Embedded Files
Appraisal of: Cooper C, Booth A, Britten N, Garside R. A comparison of results of empirical studies of supplementary search techniques and recommendations in review methodology handbooks: a methodological review. Systematic Reviews. 2017; 6:234.
Reviewer(s):
Andrew Booth
Full Reference:
Cooper C, Booth A, Britten N, Garside R. A comparison of results of empirical studies of supplementary search techniques and recommendations in review methodology handbooks: a methodological review. Systematic Reviews. 2017; 6:234. doi:10.1186/s13643-017-0625-1
Short description:
This methodological review compared empirical evidence on supplementary search methods with guidance provided in systematic review methodology handbooks. The authors audited four influential UK methodological handbooks (Cochrane Handbook, CRD guidance, Campbell guide, and NICE manual) and systematically searched five bibliographic databases (MEDLINE, EMBASE, LISTA, ASSIA, Web of Science) to identify empirical studies evaluating supplementary search techniques. Forward citation chasing and bibliography checking were also employed.
Studies were included if they reported practical application of supplementary methods (descriptive), examined utility of methods (analytical), or explored factors impacting utility. Thirty-five empirical studies published between 1989 and 2016 were included alongside the four handbooks. The review examined five supplementary search methods: contacting study authors or experts, citation chasing, handsearching, trial register searching, and web searching.
For each method, the review synthesized evidence across five domains: what the method is used for, what the evidence says, claimed advantages, claimed disadvantages, and resource requirements. Results were presented in detailed narrative synthesis with summary tables. The authors concluded that there is reasonable consistency between handbook recommendations and current practice, though the empirical studies often sought to evaluate specific aspects to improve effectiveness or efficiency. However, significant gaps remain in understanding optimal application, with limited data on resource requirements and insufficient evidence to enable rational choices about which supplementary strategies to use in specific contexts. The review highlights the need for more consistent outcome measurement and reporting to enable generalization and comparison across studies.
Limitations stated by the author(s):
The authors acknowledge several limitations of their study. First, the date range and age of both the handbooks and included studies could be considered a limitation, as guidance and practice may have evolved. Second, the evidence base consists of comparative and non-comparative case studies that were taken at face value without formal quality appraisal, as no suitable appraisal tool exists for this type of methodological research. The authors note that supplementary search methods are typically evaluated primarily in terms of effectiveness, which may be a limited test of their actual contribution to study identification processes. They also recognize that different thresholds of effectiveness and efficiency may apply depending on review type—for example, systematic reviews of qualitative studies versus reviews of randomized controlled trials may have different requirements.
The authors acknowledge that while they aimed to comprehensively identify relevant studies, the broad nature of the field means some completed studies may have been inadvertently missed or overlooked. They note that while more systematic approaches such as double-screening might have improved rigor, they are confident that such approaches would be unlikely to alter the overall conclusions. Finally, the authors note that the included studies do not necessarily correlate directly to the concepts of advantages and disadvantages, and that in most cases, proposed advantages and disadvantages have not been empirically tested in practice.
Limitations stated by the reviewer(s):
Methodological and Design Limitations:
The most significant limitation of this review is the lack of a formal quality assessment framework for the included studies. While the authors acknowledge that no suitable tool exists, this absence means readers cannot gauge the reliability or risk of bias in the evidence base. Many included studies were small case studies or single-center evaluations, yet these are weighted equally with larger, more robust investigations in the narrative synthesis. The development of even a basic quality assessment framework considering factors such as sample size, comparison groups, replicability of methods, and transparency of reporting would have strengthened confidence in the findings.
The single-reviewer screening approach introduces potential selection bias and limits reproducibility. Given that 35 studies were included from an initial search of five databases, dual independent screening would have been feasible and would have enhanced methodological rigor without substantially increasing resources. The lack of inter-rater reliability testing means we cannot assess consistency in applying the inclusion criteria, which is particularly problematic given the relatively subjective inclusion criterion regarding whether studies "identified/explored factors that impact on the utility" of methods.
Synthesis and Analysis Limitations:
The narrative synthesis approach, while appropriate for heterogeneous evidence, lacks methodological transparency. The authors do not describe how they resolved conflicting findings across studies, weighted evidence from different sources, or made decisions about which studies to emphasize in their synthesis. For example, when studies reported contradictory findings about the effectiveness of citation chasing tools (Web of Science vs. Scopus vs. Google Scholar), no framework is provided for readers to understand how these conflicts were resolved or presented.
The framework for extracting data on "claimed advantages" and "claimed disadvantages" is problematic because, as the authors themselves note, these claims are often not empirically tested within the studies. This means the review may perpetuate unsubstantiated assertions rather than evidence-based conclusions. A more rigorous approach would have distinguished between empirically demonstrated advantages/disadvantages and those merely claimed by authors. Additionally, the five-domain framework (what it's used for, what evidence says, advantages, disadvantages, resources) was not clearly defined a priori, and it's unclear how comprehensively or systematically it was applied across all included studies.
Scope and Generalizability Concerns:
The decision to focus exclusively on four UK-based methodological handbooks substantially limits the international applicability of findings. Major guidance documents from other organizations (such as the Institute of Medicine in the US, the Joanna Briggs Institute in Australia, or GRADE working group) were excluded. This geographic and organizational limitation means the "recommended practice" baseline may not represent global consensus or best practice.
The review's scope is heavily skewed toward clinical and health services research, with limited consideration of how supplementary search methods function in other disciplines such as education, social sciences, or environmental reviews. The Campbell Collaboration guidance was included, but few Campbell-relevant studies appear in the synthesis. This limits the generalizability of findings to non-health systematic reviews, where databases, publication patterns, and reporting standards may differ substantially.
Resource Requirements Analysis:
One of the most significant weaknesses is the limited and inconsistent reporting of resource requirements across supplementary search methods. While the review identifies "resource requirements" as one of five key domains, the data extracted is sparse, inconsistent, and difficult to compare. For instance, handsearching times ranged from 6 minutes to 1 hour per journal, but there's no analysis of what factors explain this variation or what represents realistic expectations. Similarly, for contacting authors, no cost-benefit analysis is presented despite this being highlighted as resource-intensive. The review needed a more systematic approach to extracting, reporting, and comparing resource data, ideally with standardized metrics such as time per study identified or cost per included study. This would have substantially increased the practical utility of the findings for reviewers with limited resources.
Currency and Temporal Issues:
The search was conducted in July 2016, with included studies dating back to 1989. No consideration is given to how technological advances, changes in publishing practices, and evolution of database indexing may have altered the relevance of older studies. For example, studies from the 1990s evaluating handsearching may have limited relevance to current practice given improvements in database indexing. Similarly, early studies of web searching or trial registers may not reflect current platform capabilities. The review would have benefited from sensitivity analysis examining whether conclusions differ when restricted to more recent studies (e.g., last 10 years) versus older evidence.
Comparison Framework Limitations:
The review's primary aim was to compare handbook guidance with empirical practice, yet the comparison framework is underdeveloped. The authors note whether studies "cited" specific handbooks but don't systematically assess whether studies adhered to handbook recommendations, deviated from them, or tested their effectiveness. A more structured comparison might have involved rating each handbook recommendation against the strength of empirical evidence, identifying gaps where handbooks provide guidance without supporting evidence, or highlighting areas where empirical practice has moved beyond handbook guidance.
Missing Considerations:
Several important aspects of supplementary searching receive insufficient attention. First, the review doesn't adequately address the question of which combinations of supplementary methods are most effective or efficient—most systematic reviews use multiple supplementary methods, yet evidence on optimal combinations is not synthesized. Second, the review doesn't examine how the choice of supplementary methods should vary based on review characteristics such as topic maturity, publication patterns in the field, or review objectives (comprehensive effectiveness review vs. rapid review vs. qualitative evidence synthesis). Third, there's minimal discussion of when supplementary searching may not be necessary or worthwhile—not all reviews require all methods, yet guidance on tailoring approaches is limited.
The interaction between database search quality and supplementary search yield is inadequately explored. Studies that perform comprehensive, well-designed database searches may find less additional value from supplementary methods compared to studies with narrow database searches. This relationship deserves more explicit consideration when interpreting the value of supplementary methods.
Reporting and Transparency:
While the authors provide their search strategy, several methodological details are unclear. The process for forward citation chasing is not fully described—which studies were chased, how many generations, and what tools were used. The decision-making process for resolving disagreements during full-text review (where two authors were involved) is not reported, including the extent and nature of any disagreements. The data extraction process lacks transparency—there's no indication of whether extraction was conducted independently by multiple reviewers or whether any validation procedures were employed.
Implications and Recommendations:
The review's conclusions and recommendations are somewhat generic. The statement that "further research is required" is valid but not sufficiently specific about research priorities, optimal study designs, or key evidence gaps. More actionable recommendations would have enhanced the review's impact—for example, specifying minimum reporting standards for studies of supplementary search methods, proposing frameworks for decision-making about which methods to use, or providing algorithms based on review characteristics.
Overall Assessment:
Despite these limitations, the review makes a valuable contribution to systematic review methodology by bringing together diverse evidence on supplementary search methods and comparing it with handbook guidance. The comprehensive scope, systematic approach to study identification, and detailed synthesis across multiple methods provide a useful resource. However, the lack of quality assessment, limited analysis of resource requirements, absence of a structured comparison framework, and generic recommendations reduce the review's ability to provide definitive guidance for systematic reviewers. The review succeeds in describing current practice and identifying gaps but falls short of providing evidence-based, actionable recommendations for when and how to use specific supplementary search methods.
Study Type:
Methodological review (systematic review of methodology studies)
Related Chapters:
Tags:
• Supplementary searching
• Contacting authors
• Citation chasing
• Handsearching
• Trial registers
• Web searching
• Grey literature
• Systematic reviews
• Search strategies
• Methodological review
• Information retrieval
• Search methods evaluation
• B. Designing strategies - general
Page updated
Report abuse