Craven 2011

Appraisal of: Craven J, Levay P. Recording Database Searches for Systematic Reviews - What is the Value of Adding a Narrative to Peer-Review Checklists? A Case Study of NICE Interventional Procedures Guidance. Evidence Based Library and Information Practice. 2011;6(4):72-87.

Reviewer(s): 

Andrew Booth

Full Reference:
Craven J, Levay P. Recording Database Searches for Systematic Reviews - What is the Value of Adding a Narrative to Peer-Review Checklists? A Case Study of NICE Interventional Procedures Guidance. Evidence Based Library and Information Practice. 2011;6(4):72-87. doi:10.18438/B8MS3S

Short description: 

This case study examines the value of adding narrative explanations to peer-review checklists when recording systematic database searches. The authors analyze the search processes used at the National Institute for Health and Clinical Excellence (NICE) for Interventional Procedures Guidance in the United Kingdom. Through a detailed case study of a search conducted for evidence on non-rigid stabilization techniques for low back pain treatment, the study demonstrates how a search strategy can meet all technical criteria on a peer-review checklist yet still miss relevant papers. Three papers identified by clinical experts during the committee review stage were not retrieved by the original database search, despite the search having been approved through the standard checklist-based peer review process.

The case study reconstructs the original MEDLINE search strategy to understand why these papers were missed. The analysis revealed that including free-text terms for "flexible," "dynamic," and "non-rigid" significantly increased the precision of the search by reducing irrelevant results by over 90% (from 7,309 to 695 hits). However, this precision came at the cost of missing two relevant papers that did not use these specific terminological conventions in their titles or abstracts. The authors demonstrate that examination of the search records together with correspondence between the analyst and searcher revealed the peer reviewer had approved the search because its sensitivity level was deliberately calibrated to be appropriate for producing Interventional Procedures Guidance within NICE's time constraints.

The study compares NICE's processes with current methods identified in the literature, including the PRESS (Peer Review of Electronic Search Strategies) checklist, Cochrane Handbook guidance, and various reporting standards like PRISMA. The authors argue that existing checklists focus primarily on technical elements (spelling, syntax, Boolean operators, database selection) but provide no mechanism for capturing the rationale behind strategic decisions. The paper presents NICE's template for recording searches and discusses how it has been amended to include prompts for narrative explanations.

The study concludes that relying solely on checklists to ensure search quality has significant limitations when contextual information is absent. Peer reviewers may concentrate on mechanical correctness without understanding why a strategy was structured in a particular way or what the searcher intended to achieve. The authors recommend that searchers should record correspondence on key decisions and report a summary narrative alongside the search strategy to describe major decisions and provide insight into the search approach rationale.

Limitations stated by the author(s): 

The authors acknowledge that the case study is specific to the NICE Interventional Procedures Guidance context, where searches need to be completed within approximately 14 days and guidance must be issued in a timely manner to the NHS. They note that the same search might have been judged differently if conducted by an organization with more time available for comprehensive systematic reviews. The authors recognize that the three missed papers were ultimately identified through contact with clinical experts on the advisory committee, demonstrating the value of supplementary search methods beyond database searching. They also acknowledge that the search was designed with a particular level of precision appropriate to NICE's purpose, which involved trade-offs between sensitivity and specificity. The paper does not explicitly state whether the narrative approach has been formally evaluated for effectiveness or what the resource implications might be for implementing such documentation practices more broadly.

Limitations stated by the reviewer(s): 

1. Generalizability concerns [External Validity]: This is a single case study from one organization (NICE) examining one specific search on one clinical topic. The findings may not transfer to other organizational contexts, different types of systematic reviews (e.g., Cochrane reviews, rapid reviews, scoping reviews), or searches in different subject domains. The NICE Interventional Procedures Guidance process has unique characteristics (9-month timeline, specific sensitivity requirements) that may limit applicability of findings to other review contexts.

2. Lack of comparative analysis [Study Design]: The study does not compare NICE's approach with documented practices at other systematic review organizations. A comparative case study design examining multiple organizations' peer review processes would have strengthened the evidence base for the recommendations. The paper would have benefited from examining whether organizations that do include narrative documentation experience fewer search quality issues or better identification of relevant literature.

3. No empirical testing of proposed solution [Intervention Effectiveness]: While the authors recommend adding narrative summaries to search documentation, the study does not test whether this intervention actually improves peer review outcomes, search quality, or the identification of relevant studies. There is no before-and-after comparison or evaluation of whether peer reviewers who receive searches with narratives make different or better decisions than those reviewing checklist-only documentation.

4. Potential for organizational bias [Conflict of Interest]: Both authors are employed by NICE and are evaluating their own organization's processes. While they critically examine limitations of the current approach, there may be unconscious bias toward defending or rationalizing existing institutional practices. The selection of this particular case study (where the search "worked" appropriately for NICE's purposes despite missing papers) may reflect a defensive institutional response.

5. Post-hoc rationalization [Hindsight Bias]: The analysis is retrospective, conducted after the guidance was completed and the missed papers were identified. The interpretation that the search precision was "appropriate for the purpose" emerges from analyzing correspondence after the fact. The study cannot assess whether this justification would have been articulated prospectively or whether the peer reviewer would have raised concerns about the restrictive free-text terms had the narrative been provided during the initial review.

6. Incomplete consideration of alternative explanations [Confounding]: The three papers were identified through expert consultation, not through searching. The study focuses on how modifying the search strategy could have found these papers, but doesn't adequately consider whether the issue is that database searching has inherent limitations regardless of how well it is conducted or documented. The case might equally support conclusions about the importance of supplementary search methods rather than improved search documentation.

7. Limited discussion of feasibility and costs [Implementation]: The paper does not address the time and resource implications of requiring narrative documentation. For organizations conducting numerous searches, the additional burden of documenting and reviewing decision-making narratives could be substantial. There is no discussion of how detailed narratives need to be, how standardized they should be, or how much time peer reviewers would need to properly review both technical elements and narrative context.

8. Unclear standards for "appropriate" sensitivity [Lack of Operational Definition]: The study concludes that the search sensitivity was "appropriate for the purpose of producing Interventional Procedures Guidance" but does not provide clear criteria for determining what constitutes appropriate sensitivity for different review types. The decision appears somewhat subjective and context-dependent, which raises questions about how narratives would help peer reviewers make consistent judgments across different searches.

9. Missing methodological details [Incomplete Reporting]: The paper does not clearly describe how the peer review process worked at NICE, who the peer reviewers were (e.g., information specialists, methodologists, clinical experts), what training they received, or how disagreements were resolved. These contextual details would help readers understand the peer review system being evaluated. The correspondence between analyst and searcher is referenced but not fully presented, limiting transparency.

10. Limited engagement with counterarguments [Confirmation Bias]: The paper does not adequately consider potential disadvantages of narrative documentation, such as: narratives might become formulaic and add little value; peer reviewers might lack time to properly review detailed narratives; narratives might introduce inconsistency if different searchers document decisions in different ways; or technical checklists might still be more reliable for identifying genuine errors despite missing contextual nuance.

Study Type: 

Case study

Related Chapters: 

• Qualitative research

• Documenting and reporting the search process

Tags: 

• Case study

• Checklists

• Documentation

• NICE guidance

• Peer review

• Quality assurance

• Search recording

• Search reporting

• Search strategy development

• Systematic reviews

• Transparency

• PRESS checklist

• PRISMA

• Interventional procedures

• B. Designing strategies - general

• E. Quality assurance and reporting