Wright 2015

Appraisal of: Wright K, Golder S, Lewis-Light K. What value is the CINAHL database when searching for systematic reviews of qualitative studies? Systematic Reviews (2015) 4:104. 

Reviewer(s): 

Andrew Booth

Full Reference:
Wright K, Golder S, Lewis-Light K. What value is the CINAHL database when searching for systematic reviews of qualitative studies? Systematic Reviews (2015) 4:104

Short description: 

This retrospective study investigated the value of the CINAHL database for identifying qualitative research studies when conducting systematic reviews. The authors analysed 43 systematic reviews of qualitative studies published between 2007 and 2012, identified from the Database of Abstracts of Reviews of Effects (DARE). They systematically hand-searched 501 included references from these reviews across seven databases: CINAHL, MEDLINE/PubMed, PsycINFO, Embase, Web of Science, Scopus, and Google Scholar.

The authors aimed to determine two key metrics: the percentage of included studies available in CINAHL and the percentage of studies unique to CINAHL. The included reviews covered diverse health topics including pregnancy and childbirth, experiences of chronic conditions (diabetes, heart failure, respiratory infections), and healthcare interventions. Reviews searched between 3 and 20 databases (median 4), with MEDLINE/PubMed being universally used and CINAHL second most common (81%). Additionally, 86% of reviews employed supplementary search methods such as reference checking, citation searching, and hand-searching journals.

Results showed that all 43 reviews had at least some included studies available in CINAHL. For nine reviews (21%), all included studies were available in CINAHL, suggesting it could have been theoretically possible to identify all studies using this single database. For 30 reviews (70% of the sample), CINAHL contained 80% or more of the included studies, demonstrating high coverage. Regarding unique contributions, 18 reviews (42%) included at least one study uniquely identified from CINAHL. However, only 11 of these could be fully verified due to inability to access all databases used in the original reviews. For these 11 reviews, the median percentage of unique studies from CINAHL was 9.09%, with a range of 5.0% to 33.0%.

The authors concluded that CINAHL appears to be a valuable source for systematic reviews of qualitative research, with good coverage across diverse health topics. However, they were less confident about conclusions regarding unique study contributions due to access limitations to subscription databases used in original reviews.

Limitations stated by the author(s): 

The authors provide transparent acknowledgment of several methodological constraints. As a retrospective analysis of published systematic reviews, the study relied entirely on what was reported in the published papers regarding database selection, search strategies, and supplementary methods. Without access to complete search strategies or original search results, the authors could only assess current database availability of references rather than actual retrieval performance, which depends on search strategy quality, database interface functionality, and indexing accuracy.

The exclusive use of DARE to identify eligible systematic reviews, combined with a search strategy using only MeSH terms without free-text terms, may have resulted in selection bias and incomplete retrieval of all relevant qualitative systematic reviews in the PubMed database. The authors acknowledge they did not investigate reviews that had not used CINAHL, which could have provided useful comparative data. The restriction to one disease area or topic (though the sample actually covered diverse topics) potentially limits generalizability to other research areas.

A significant practical limitation was the inability to access numerous databases originally searched by the review authors, including subscription-only databases (CAB Abstracts, SocIndex, Ageline, ProQuest Dissertations), discontinued resources (Intute, SIGLE, National Research Register), and unclear sources. This meant that for 7 of the 18 reviews showing potential unique CINAHL contributions, the findings could not be verified because more than 50% of originally searched databases were inaccessible. Consequently, conclusions about unique study contributions could only be confidently based on 11 reviews rather than the full 18, substantially limiting confidence in this important finding.

Limitations stated by the reviewer(s): 

Methodological design limitations: While the retrospective approach was pragmatic given resource constraints, it introduces fundamental limitations that undermine the validity of the study's conclusions. The authors acknowledge they measured "current availability" rather than actual retrieval performance, but this distinction is critical and perhaps understated. The presence of a record in a database at the time of checking (2014) does not indicate whether it was indexed there during the original review searches (2007-2012), whether it was findable with the search strategies actually used, or whether indexing quality would have enabled retrieval. Database coverage, indexing practices, and journal inclusion all change over time, making retrospective availability a poor proxy for prospective performance. The study essentially measures "what's there now" rather than "what could have been found then," which is what systematic reviewers actually need to know. A prospective study design, though more resource-intensive, would have provided much more actionable evidence by testing actual search strategies and measuring real-world retrieval performance.

Sample selection and representativeness concerns: The reliance on DARE as the sole source of systematic reviews introduces multiple layers of potential bias. DARE's own search strategies and inclusion criteria may have preferentially identified certain types of qualitative reviews while missing others. The use of only MeSH terms ("Qualitative Research" and "Diabetes Mellitus") without free-text alternatives likely resulted in incomplete identification of eligible reviews, as qualitative methodology is often described using varied terminology not captured by controlled vocabulary. The authors acknowledge this but may underestimate its impact on the representativeness of their sample. The 43 included reviews, while covering diverse topics, were primarily published in 2011-2012 (86%), which limits generalizability across time periods when database coverage and practices may have differed. Furthermore, by only analyzing reviews that had actually searched CINAHL, the authors created a circular sampling issue that likely inflates CINAHL's apparent value. Reviews that successfully identified all necessary studies without using CINAHL were systematically excluded from analysis, yet these would provide the most powerful evidence about whether CINAHL is truly essential. This exclusion criterion essentially ensures that the sample is biased toward topics and questions where CINAHL appears useful.

Database access limitations critically undermine key findings: The inability to access many databases originally searched by review authors represents a fatal flaw for the study's primary research question about unique study contributions. For 39% of the reviews showing potential unique CINAHL studies, the findings could not be verified. This is not merely a limitation to acknowledge but rather fundamentally compromises the validity of conclusions about CINAHL's unique contribution. The authors report a median of 9.09% unique studies based on 11 reviews, but this figure has wide confidence intervals given the small sample size and may not be generalizable. More importantly, subscription databases that the authors couldn't access (particularly specialized databases in specific fields) may have contained many of the same "unique" studies attributed to CINAHL. The authors' decision to exclude reviews where inaccessible databases constituted more than 50% of those searched is sensible but further reduces the already small sample. This limitation essentially means the study can only provide weak evidence about one of its two primary research questions.

Insufficient analysis of contextual and quality factors: The study treats all 43 systematic reviews as equivalent units of analysis without adequately considering variation in review quality, comprehensiveness of searching, or appropriateness of database selection for the specific review questions. Some reviews may have had poorly designed search strategies that failed to retrieve relevant studies from any database, while others may have employed highly sensitive, well-designed strategies. These differences in search quality could substantially affect apparent database performance but are not examined. The authors note that 86% of reviews used supplementary search methods but provide limited analysis of how these methods interacted with database searching or whether certain databases were more likely to require supplementation. The diverse topics covered by the included reviews (pregnancy, chronic disease management, healthcare interventions) may have differential relationships with CINAHL's nursing and allied health focus, but this potential variation is not systematically explored. Some reviews covering predominantly nursing topics might show higher CINAHL coverage than those addressing broader health topics, but this hypothesis is not tested.

Measurement and interpretation issues: The study's two main outcome measures (percentage of included studies available in CINAHL and percentage unique to CINAHL) have important interpretational limitations that are not fully addressed. High availability percentages might reflect either good database coverage or poor searching of other databases by the original reviewers. The authors cannot distinguish between these scenarios. Similarly, low percentages of unique studies might indicate either that CINAHL rarely provides distinctive content or that review authors' search strategies in CINAHL were less effective than in other databases. The study measures presence/absence of records but not their findability, quality of indexing, or relevance ranking—all factors that affect practical utility for systematic reviewers. The authors' decision to treat MEDLINE and PubMed as a single database is pragmatic but potentially problematic, as PubMed includes content not in MEDLINE (as they acknowledge) and this additional content might have affected attribution of unique studies.

Limited practical guidance for systematic reviewers: While the study provides useful descriptive data, it offers limited actionable guidance for systematic reviewers making prospective decisions about database selection. The finding that CINAHL contained 80%+ of studies for 70% of reviews might suggest it's valuable, but the study doesn't address the critical question of cost-effectiveness: is searching CINAHL worth the additional time and subscription costs given that other database combinations might achieve similar coverage? The authors note that for two reviews, CINAHL coverage was much lower (38.5% and 14.3%) but provide insufficient analysis of what distinguished these reviews and how reviewers might prospectively identify when CINAHL would be less valuable. The study doesn't examine whether specific combinations of other databases could substitute for CINAHL, nor does it provide evidence about the relative importance of unique studies identified—were they central to review conclusions or peripheral? The discussion of qualitative evidence synthesis potentially requiring less exhaustive searching is interesting but underdeveloped.

Reporting and transparency concerns: While the authors provide reasonable detail about their methods, some aspects of their approach are unclear or potentially inconsistent. The inclusion/exclusion criteria are mentioned as being developed but are relegated to supplementary materials, making it difficult to fully assess their appropriateness. The process for resolving disagreements among the four information specialists conducting assessments is not described. The authors state they recorded whether records were "unique to CINAHL" but don't clearly specify their operational definition—does this mean unique among all seven databases examined, or unique among only those databases accessible and originally searched by each review? The handling of duplicates and the method for tracking individual references across multiple reviews is not fully explained. Some of the data presentation could be clearer; for example, Figure 3 shows distribution of CINAHL availability but doesn't indicate which reviews fall into each category, limiting ability to identify patterns.

Generalizability and external validity: The study's findings are explicitly limited to systematic reviews of qualitative research regarding diverse health topics, and the authors appropriately note that results may not apply to other research types or disease areas. However, even within qualitative research, the sample may not be representative. The included reviews came exclusively from DARE, which has specific inclusion criteria emphasizing methodological quality and systematic approaches. Lower-quality reviews or those using less rigorous methods might show different patterns of database contribution. The predominance of nursing and patient experience topics in the sample may inflate CINAHL's apparent value compared to qualitative research in other health domains such as health policy, implementation science, or basic behavioural research. The temporal concentration of included reviews in 2011-2012 means findings may not reflect current database performance given ongoing changes in journal coverage, indexing practices, and database functionality. The evolution of open access publishing, preprint servers, and other changes in scholarly communication since 2012 may affect the relative value of subscription databases like CINAHL.

Study Type: 

Retrospective bibliometric analysis / Methodological research

Related Chapters: 

• Qualitative research

• Search strategy development 

• Value of using different search approaches

Tags: 

• Bibliographic databases

• CINAHL

• DARE

• Database comparison

• Database coverage

• Database recall

• Evidence synthesis

• Information retrieval

• Literature searching

• MEDLINE/PubMed

• Qualitative research

• Search methods

• Search strategy

• Systematic reviews

• Unique references