Effective Clinical Practice
Effective Clinical Practice, July/August 2001
Kaveh G. Shojania, Lisa A. Bero
For author affiliations, current addresses, and contributions, see end of text.
Context. Systematic reviews of the literature are an important resource for clinicians. Unfortunately, the few published strategies for identifying these articles involve MEDLINE interfaces not widely available outside of academic medicine. In addition, the performance of these strategies is unknown.
Objective. To develop and evaluate a search strategy for identifying systematic reviews by using a publicly available MEDLINE interface (PubMed).
Design. Diagnostic test assessment.
Definition of Sensitivity. The proportion of recognized systematic reviews (indexed in the Cochrane Library's Database of Abstracts of Reviews of Effectiveness [DARE] or in ACP Journal Club) that are identified by the search strategy.
Definition of Positive Predictive Value. The proportion of articles identified in one of three sample searches (screening for colorectal cancer, thrombolytic therapy for venous thromboembolism, and treatment of dementia) that meet a minimum definition of systematic review.
Results. Our PubMed search strategy identified 93 of 100 DARE-indexed systematic reviews, a sensitivity of 93% (95% CI, 86% to 97%). For the sample of systematic reviews drawn from ACP Journal Club (n = 103), the PubMed strategy achieved a sensitivity of 97% (CI, 91% to 99%). When the three sample search strings were used, approximately 50% of retrieved articles met our minimum definition of systematic review. In contrast, the similar precision of a PubMed search restricted to review-type articles (as indexed by MEDLINE) was less than 10%.
Conclusions. This search strategy identified most systematic reviews without overwhelming users with numerous false-positive results. A "single-click" filter based on this strategy is now available as part of the Clinical Queries feature of PubMed.
Given the limited time available for searching and appraising the medical literature, systematic reviews are important sources of evidence for clinicians. In contrast to traditional narrative reviews, systematic reviews of the literature address a well-defined clinical question, use an explicit strategy to locate relevant evidence, evaluate the retrieved studies using prospectively defined methodologic criteria, and formally synthesize the results. (1-7)
Despite the importance of systematic reviews, MEDLINE does not index these reviews as a specific publication type. (8, 9) "Review" represents a publication type, but most of these articles are traditional narrative reviews. (4, 6) "Meta-analysis" also constitutes a publication type; however, not all systematic reviews are meta-analyses, and not all meta-analyses are indexed by this term. (8, 9) Thus, finding systematic reviews continues to be a major barrier to their use by clinicians. (10)
The few published search strategies for locating systematic reviews involve versions of MEDLINE not widely available outside of academic medicine. (8, 9, 11) More important, the performance of these strategies has not been evaluated. In this article, we present a search strategy for identifying systematic reviews that uses PubMed, a MEDLINE interface freely available via the Internet from the U.S. National Library of Medicine (NLM). (12) We assess the sensitivity of this strategy using as a "gold standard" systematic reviews recognized by two authoritative sources, and we evaluate the positive predictive value (or "precision") obtained by using three sample clinical searches.
In this article, we use the term systematic review to refer to any literature review, meta-analysis, guideline, or other article that explicitly indicates the use of a strategy for locating evidence. This purposefully broad definition is not intended to replace published recommendations for the conduct of systematic reviews. (1-7) Formal systematic reviews include other details of the search strategy and meet additional methodologic criteria for evidence appraisal and synthesis. (2, 6, 11) Of note, however, is that fewer than one third of review articles in core medical journals meet even our broad definition. (6)
MEDLINE is an on-line bibliographic database created by the NLM that contains more than 11 million references and abstracts published since 1965. (12) Multiple interfaces are available for searching MEDLINE; each differs in appearance and internal logic but seeks the same target content. Although most of these interfaces are proprietary, the NLM has provided free public access to MEDLINE through PubMed since 1997. (12)
The PubMed Search Strategy
As with other strategies for locating systematic reviews, (8, 11) the search presented in Table 1 combines relevant publication types ("meta-analysis," "review," "guideline") with title and text words typically found in systematic reviews. The "BUTNOT" phrase in the final search string eliminates extraneous publication types, such as letters, editorials, and case reports. The latter commonly includes a "review of the literature," but the thoroughness of such literature reviews varies widely. Moreover, case reports are generally not relevant to answering questions relating to clinical practice, and their sheer number can overwhelm clinicians searching for such answers.
We defined the sensitivity of the search strategy as the proportion of systematic reviews indexed in two repositories--the Cochrane Library's Database of Abstracts of Reviews of Effectiveness (DARE) or by the American College of Physicians-American Society of Internal Medicine's ACP Journal Club--that were successfully retrieved by our search strategy. The DARE contains quality-assessed systematic reviews identified by regular searches of important bibliographic databases (such as MEDLINE) and hand-searching of major journals. (13) Included reviews must meet at least four methodologic criteria, and all included articles must display "evidence of a substantial effort to search for all relevant research." (13) The Cochrane Library contains DARE and lists records (1899 at the time of this evaluation) in an apparently arbitrary order. We chose as a target sample the first 100 records that corresponded to articles indexed in MEDLINE. The total sample required to do this was 112 (i.e., 12 articles are not in MEDLINE).
Review articles included in ACP Journal Club must contain "an identifiable description of the methods indicating the sources and methods for searching for articles and state the clinical topic and the inclusion and exclusion criteria for selecting articles for detailed review." (14) In addition, at least one study reviewed in the article must meet the criteria required of original articles addressing treatment, diagnosis, prognosis, causation, quality improvement, or economics of health care programs. (14) These articles thus represent systematic reviews that clinicians would wish to retrieve with high sensitivity. We hand-searched the number of consecutive issues that was required to identify 100 "reviews" commented on in ACP Journal Club. Achieving this number required scanning from the most recent issue at the time of this search (September/October 2000) back to the first issue of 1999. This process identified 104 reviews, one of which was excluded because it represented a Canadian government report not indexed in MEDLINE.
Positive Predictive Value
We evaluated the positive predictive value of our search strategy by applying the strategy to each of three clinical topics: screening for colorectal cancer, thrombolytic therapy for venous thromboembolism, and treatment of dementia. The search strings for each clinical topic were created by using the Medical Subject Headings (MeSH) browser function, located in the left column of the main search screen in Figure 1. We then reviewed the titles and abstracts of each retrieved article to determine whether it met our criteria for a systematic review.
In the parlance of library science, we measured the "precision" of the search, which, continuing with the more familiar terminology of diagnostic tests, corresponds to the positive predictive value--the proportion of retrieved articles that represent "true positives." We regarded an article as a true positive only if the title or abstract explicitly identified the article as a systematic review or meta-analysis or if the article abstract indicated a strategy for locating the literature reviewed. Thus, an article that contained the phrase "literature review" in the title but merely stated that "relevant literature was reviewed" in the abstract would not count as a true positive. MEDLINE records without an abstract could be counted as true positives only if the title contained the words "meta-analysis," "metaanalysis," or "systematic review."
As shown in Table 2, our electronic search strategy retrieved 93 of the 100 target articles in the DARE (sensitivity, 93% [95% CI, 86% to 97%]). This sensitivity reflects our exclusion of 12 records not indexed in MEDLINE. Including such reviews in the target population reduces the sensitivity to 83% (95% CI, 74% to 89%). Four (15-18) of the seven DARE articles missed by our electronic search strategy involve aspects of behavioral medicine likely to fall outside the interests of most clinicians, reflecting the broad range of health care fields targeted by DARE.
The reviews drawn from ACP Journal Club, on the other hand, are explicitly chosen for their relevance to clinical practice. Among this sample, the electronic search retrieved 100 of 103 references (Table 2), yielding a sensitivity of 97% (CI, 91% to 99%). Careful review of the abstracts of the three missed references indicated that failure to capture one of them (19) reflected the imperfect sensitivity of the MEDLINE search engine for title words (in this case, "review"), (21, 20) not the logic of the search strategy.
Positive Predictive Value
As shown in Table 3, approximately 50% of the retrieved articles met the criteria for true positive in each of the three sample searches. In other words, half of the articles retrieved by combining a clinical search string with the "universe of systematic reviews" created by the general search (search 6 in Table 1) met our broad definition of systematic review. For comparison, we measured the positive predictive value of a search combining one of our three sample search strings (therapy for dementia) with the universe of articles referenced by MEDLINE as publication type "review" in 1999 (not shown in Table 1). Among the 170 articles retrieved, only 13 met the above definition for true positive, corresponding to a positive predictive value of only 8% (CI, 4% to 13%).
The modest positive predictive value for our search strategy, approximately 50%, is probably an underestimate, since we required true-positive results to be identifiable as systematic reviews solely on the basis of the MEDLINE record. On retrieval of the full-text articles, some of the retrieved citations not meeting this requirement (i.e., false positives) proved to be systematic reviews. For example, while none of the false-positive articles on colorectal cancer screening met our definition of a systematic review, three of the five false-positive references for thrombolytic therapy for venous thromboembolism (22-26) explicitly stated that the MEDLINE database was searched, (22, 23, 26) and one of the articles even provided the search terms used. (22)
We have presented a strategy that allows users to identify systematic reviews with a sensitivity of more than 90% for articles indexed in MEDLINE. For articles likely to have the greatest relevance to clinicians (those indexed in ACP Journal Club), the sensitivity exceeded 95%. This high sensitivity was achieved without many false-positive results. Approximately 50% of articles retrieved by sample searches met our broad definition of systematic review, compared with fewer than 10% obtained by simply restricting one of the sample searches to the publication type "review."
Some readers might reasonably be concerned about the broad definition used to evaluate performance of the search strategy. We chose this approach to impose a reasonable and easily applicable minimum requirement for a systematic review, rather than formally defining attributes required of a high-quality systematic review. Published recommendations for the methodologic features of systematic reviews (1, 2, 6) do not stipulate the minimum criteria that a review article must fulfill to count as a systematic review. Above all, we sought to distinguish between systematic reviews, which tend to share certain distinctive analytic features, and narrative reviews, an entirely distinct category. Of the various methodologic criteria typically applied to systematic reviews, the explicit mention of a strategy for locating evidence seemed the most basic and least controversial, hence our focus on this feature in constructing and evaluating our electronic search strategy.
The usefulness of previously published electronic search strategies for identifying systematic reviews (8, 9, 11) has been limited, since all have used MEDLINE interfaces that do not permit free public access (e.g., Ovid). Moreover, few studies have evaluated the performance of these strategies. One exception is a study describing a formal search strategy (using the Ovid interface) based on frequency analysis of terms appearing in a "quasi-gold standard" sample obtained by hand-searching high-impact journals. (27) The proposed strategy achieved 98% sensitivity while retaining a precision of 20%, but this performance was evaluated against the same gold standard sample from which the search terms were identified.
We have evaluated an electronic search strategy that identifies systematic reviews with high sensitivity without overwhelming users with numerous false-positive results. Readers are invited to store this search in a text file for use with subsequent searches (see the Appendix). While most systematic reviews have addressed aspects of therapeutics, articles reviewing diagnosis, prognosis, cause, and even outcomes and effectiveness research now appear with increasing frequency. (7) Thus, clinicians have at their fingertips a resource for identifying articles likely to contain evidence-based answers to important clinical questions.
|Take Home Points
1. Oxman AD, Cook DJ, Guyatt GH. Users' guides to the medical literature. VI. How to use an overview. Evidence-Based Medicine Working Group. JAMA. 1994;272:1367-71.
2. Oxman AD, Guyatt GH. Guidelines for reading literature reviews. CMAJ. 1988;138:697-703.
3. Oxman AD. Checklists for review articles. BMJ. 1994;309:648-51.
4. Cook DJ, Mulrow CD, Haynes RB. Systematic reviews: synthesis of best evidence for clinical decisions. Ann Intern Med. 1997;126:376-80.
5. Bero LA, Grilli R, Grimshaw JM, Harvey E, Oxman AD, Thomson MA. Closing the gap between research and practice: an overview of systematic reviews of interventions to promote the implementation of research findings. The Cochrane Effective Practice and Organization of Care Review Group. BMJ. 1998;317:465-8.
6. McAlister FA, Clark HD, van Walraven C, et al. The medical review article revisited: has the science improved? Ann Intern Med. 1999;131:947-51.
7. Petticrew M. Systematic reviews from astronomy to zoology: myths and misconceptions. BMJ. 2001;322:98-101.
8. Hunt DL, McKibbon KA. Locating and appraising systematic reviews. Ann Intern Med. 1997;126:532-8.
9. Glanville J, Lefebvre C. Identifying systematic reviews: key resources [Editorial]. ACP J Club. 2000;132:A11-2.
10. Bero LA, Jadad AR. How consumers and policymakers can use systematic reviews for decision making. Ann Intern Med. 1997;127:37-42.
11. Jadad AR, Cook DJ, Jones A, et al. Methodology and reports of systematic reviews and meta-analyses: a comparison of Cochrane reviews with articles published in paper-based journals. JAMA. 1998;280:278-80.
12. Lindberg DA. Internet access to the National Library of Medicine. Eff Clin Pract. 2000;3:256-60.
13. What are the criteria for the inclusion of reviews in DARE? NHS Centre for Reviews and Dissemination. Available at http://agatha.york.ac.uk/faq4.htm. Accessed 2 May 2001.
14. Purpose and Procedure. ACP Journal Club. Available at www.acponline.org/journals/acpjc/purproc.htm. Accessed 20 June 2001.
15. Bauman LJ, Drotar D, Leventhal JM, Perrin EC, Pless IB. A review of psychosocial interventions for children with chronic health conditions. Pediatrics. 1997;100:244-51.
16. Katon W, Gonzales J. A review of randomized trials of psychiatric consultation-liaison studies in primary care. Psychosomatics. 1994;35:268-78.
17. Moner SE. Acupuncture and addiction treatment. J Addict Dis. 1996;15:79-100.
18. Richmond RL. A comparison of measures used to assess effectiveness of the transdermal nicotine patch at 1 year. Addict Behav. 1997;22:753-7.
19. Magee LA, Ornstein MP, von Dadelszen P. Fortnightly review: management of hypertension in pregnancy. BMJ. 1999;318:1332-6.
20. Wilczynski NL, Walker CJ, McKibbon KA, Haynes RB. Reasons for the loss of sensitivity and specificity of methodologic MeSH terms and textwords in MEDLINE. Proc Annu Symp Comput Appl Med Care. 1995:436-40.
21. Allison JJ, Kiefe CI, Weissman NW, Carter J, Centor RM. The art and science of searching MEDLINE to answer clinical questions. Finding the right number of articles. Int J Technol Assess Health Care. 1999;15:281-96.
22. Arcasoy SM, Kreit JW. Thrombolytic therapy of pulmonary embolism: a comprehensive review of current evidence. Chest. 1999;115:1695-707.
23. O'Meara JJ 3rd, McNutt RA, Evans AT, Moore SW, Downs SM. A decision analysis of streptokinase plus heparin as compared with heparin alone for deep-vein thrombosis. N Engl J Med. 1994;330:1864-9.
24. Jones DR. Audit of attitudes to and use of postoperative thromboembolic prophylaxis in a regional health authority. Ann R Coll Surg Engl. 1991;73:219-21; discussion 222.
25. Gallus AS. Anticoagulants in the prevention of venous thromboembolism. Baillieres Clin Haematol. 1990;3:651-84.
26. Rogers LQ, Lutcher CL. Streptokinase therapy for deep vein thrombosis: a comprehensive review of the English literature. Am J Med. 1990;88:389-95.
27. Boynton J, Glanville J, McDaid D, Lefebvre C. Identifying systematic reviews in MEDLINE: developing an objective approach to search strategy design. Journal of Information Science. 1998;24:137-54.
The authors thank Dr. Brian Haynes for his comments and suggestions on an earlier draft of this manuscript.
Kaveh G. Shojania, MD, Department of Medicine, Box 0120 UCSF, San Francisco, CA 94143-0120; telephone: 415-502-1323; fax: 415-514-2094; e-mail: Shojania@medicine.ucsf.edu.
The search for systematic reviews shown in Table 1 is presented as a six-line search so that the logic of the search would be apparent. For everyday use, however, readers are invited to use a single-line version of this search, which can be copied into a text file and pasted into PubMed for subsequent searches:
((meta-analysis [pt] OR meta-analysis [tw] OR metanalysis [tw]) OR ((review [pt] OR guideline [pt] OR consensus [ti] OR guideline* [ti] OR literature [ti] OR overview [ti] OR review [ti]) AND ((Cochrane [tw] OR Medline [tw] OR CINAHL [tw] OR (National [tw] AND Library [tw])) OR (handsearch* [tw] OR search* [tw] OR searching [tw]) AND (hand [tw] OR manual [tw] OR electronic [tw] OR bibliographi* [tw] OR database* OR (Cochrane [tw] OR Medline [tw] OR CINAHL [tw] OR (National [tw] AND Library [tw]))))) OR ((synthesis [ti] OR overview [ti] OR review [ti] OR survey [ti]) AND (systematic [ti] OR critical [ti] OR methodologic [ti] OR quantitative [ti] OR qualitative [ti] OR literature [ti] OR evidence [ti] OR evidence-based [ti]))) BUTNOT (case* [ti] OR report [ti] OR editorial [pt] OR comment [pt] OR letter [pt])
To avoid the inconvenience of transcribing this tortuous string into a text file, copy the above search string. Please note: Users of older version of Netscape should be aware that the search string may be truncated.