Booth 2006

Appraisal of: Booth AC. "Brimful of STARLITE": toward standards for reporting literature searches. J Med Libr Assoc. 2006;94:421-9, e205.

This combines an appraisal prepared by Dagmara Chojecki and Lisa Tjosvold, with an appraisal prepared by Andrew Booth (2025)

Reviewer(s):
(1)

Dagmara Chojecki
Lisa Tjosvold

(2)

Andrew Booth

Full Reference:
Booth AC. "Brimful of STARLITE": toward standards for reporting literature searches. J Med Libr Assoc. 2006;94:421-9, e205.

Short description: 

Chojecki and Tjosvold: The aim of this paper is to stimulate improvements in conducting and reporting both qualitative and quantitative systematic reviews. The authors surveyed 44 reports of qualitative systematic reviews, characterized techniques used to identify articles for inclusion and proposed standards for reporting of literature searches called STARLITE which includes sampling strategy, type of study, approaches, range of years, limits, inclusion and exclusion, terms used and electronic sources. Findings from this work can inform groups of information specialists including the Cochrane Collaboration Information Retrieval Methods Group.

Booth: This systematic survey of the literature examined the reporting quality of search methods used in qualitative systematic reviews in healthcare published between 1988 and December 2004. The author searched MEDLINE, CINAHL, Web of Knowledge (including Science and Social Sciences Citation Index), the Cochrane Methodology Register, and conducted Internet searches using the Copernic Agent Professional meta-search agent. Of 64 studies identified, 43 met inclusion criteria and reported at least one element of their search methods.

The study extracted data on sampling strategies, databases searched, keywords used, and supplementary search approaches. Results revealed considerable variation and generally poor reporting standards, with only 64% reporting search terms used, 20% specifying study types, and 20% detailing inclusion and exclusion criteria. The most commonly searched database was CINAHL (31 times), followed by MEDLINE (30 times), with a median of five databases searched per review.

Based on these empirical findings and the author's extensive experience with systematic reviews, Booth proposes the STARLITE mnemonic as a framework for reporting literature searches. STARLITE encompasses eight essential elements: Sampling strategy, Type of study, Approaches, Range of years, Limits, Inclusion and exclusions, Terms used, and Electronic sources. The author concludes that there is a pressing need for international consensus on standards for reporting literature searches and that librarians have a key role in defining such standards for systematic reviews.

Limitations stated by the author(s): 

Chojecki and Tjosvold: Only one reviewer was used to make judgments on inclusion and exclusion which opens the possibility of bias.

Booth: The author explicitly acknowledges that using a single reviewer for judgments on inclusion and exclusion of studies opens the possibility of bias. This limitation would be particularly serious if the author had attempted to make value judgments on whether literature searching methods were "adequate" or not. However, the author mitigates this concern by maintaining a descriptive rather than evaluative focus, characterizing the population of qualitative systematic review studies without making quality judgments that would require additional reviewers, quality checklists, and assessment of inter-rater reliability.

The author also notes that the population of studies could be further extended through more exhaustive search procedures such as hand-searching of key qualitative journals like Qualitative Research and Qualitative Health Research. Additionally, the author acknowledges that the field of qualitative systematic review is relatively immature with no consensus on what constitutes such a review, making judgments about inclusion and exclusion necessarily subjective and rendering the survey less easily reproducible despite its measure of systematicity.

Limitations stated by the reviewer(s): 

Chojecki and Tjosvold: No additional limitations detected by the reviewers.

Booth: 

Strengths of the study:

The study makes an important contribution to the methodology of systematic reviews by highlighting the poor state of reporting for literature searches, which are fundamental to the validity and reproducibility of systematic reviews. The development of the STARLITE mnemonic provides a practical and memorable framework that addresses a genuine gap in the literature, as no formal standards for reporting literature searches existed at the time of publication. The author draws on experience in conducting HTAs, guidelines, and systematic reviews.

The study employs appropriate methodology for its aims as a systematic survey rather than a full systematic review, acknowledging resource constraints. The search strategy was comprehensive, covering multiple relevant databases and supplementary approaches including citation searching and Internet searches. The data extraction matrix and transparent reporting of findings allow readers to assess the evidence underpinning the STARLITE framework. The author is reflexive in acknowledging limitations and the distinction between poor reporting versus poor conduct.

Weaknesses and concerns:

1. Single reviewer bias: Despite the author's acknowledgment, the use of a single reviewer for all screening, inclusion/exclusion decisions, and data extraction represents a significant methodological limitation that could introduce systematic bias. Standard systematic review methodology requires at least two independent reviewers to minimize subjective judgments and errors. This is particularly problematic given the acknowledged subjectivity in defining what constitutes a "qualitative systematic review."

2. Lack of validation: The STARLITE framework, while intuitively appealing, is based primarily on the author's observation of what was missing in the literature rather than on empirical testing of what elements are necessary and sufficient for adequate reporting. No validation study is presented to demonstrate that using STARLITE actually improves the quality, reproducibility, or usability of reported searches. The framework would benefit from expert consensus methods such as Delphi panels or formal validation studies.

3. Limited scope: The study restricts inclusion to qualitative systematic reviews in healthcare, yet proposes STARLITE as a general framework for all systematic reviews. The generalizability to quantitative reviews or reviews in other fields is assumed rather than demonstrated. Different types of reviews may require different reporting elements or emphases.

4. Incomplete reporting of methods: Ironically, for a paper advocating improved reporting standards, some methodological details are underspecified. For example, the exact search strategies used are not provided, making the review difficult to replicate. The verification strategy (Figure 1) is mentioned but not fully explained. Inclusion and exclusion criteria could be more precisely defined, particularly regarding what constituted "significant part" of a qualitative systematic review.

5. No quality assessment: While the author states that quality assessment was not the aim, the descriptive approach means we cannot distinguish between reviews that conducted adequate searches but reported them poorly versus reviews that conducted inadequate searches. This limits the ability to identify which reporting gaps are most critical to address. Some evaluation of the likely impact of poor reporting on review quality would strengthen the argument for standardization.

6. Potential citation bias: The author does not disclose whether any of the included studies were authored or co-authored by Booth himself, which could introduce citation bias. Given that Booth authors papers in this field, this is likely and should have been transparently reported.

7. Limited stakeholder engagement: The development of STARLITE was a solo endeavour. Reporting standards benefit from multi-stakeholder input including librarians, information specialists, systematic reviewers, journal editors, and end-users. The lack of formal consensus-building limits the likely uptake and acceptance of the framework.

8. Temporal limitations: Searching only to December 2004 means the review may have missed emerging practices in qualitative synthesis methodology. Additionally, by 2006, other reporting guidelines (QUOROM, MOOSE) existed for quantitative reviews, and the paper could have engaged more deeply with how STARLITE complements or differs from these existing frameworks.

Study Type:
Chojecki and Tjosvold: Review

Booth: Systematic survey of the literature / Methodological review

Related Chapters: 

• Documenting and reporting the search process

• Qualitative Research

Tags: 

• Documenting

• Guidance document

• Literature searching

• Search methods

• Qualitative systematic reviews

• Reporting standards

• STARLITE

• Meta-synthesis

• Database searching

• Search strategies

• Methodological review

• Information retrieval

• Quality of reporting