Effective Clinical Practice
Douglas K. Owens, Dena M. Bravata
For author affiliations, current addresses, and contributions, see end of text.
If your heart is in your dream
No request is too extreme
When you wish upon a star
As dreamers do... (1)
In a landmark 1959 paper in Science, Ledley and Lusted (2) described principles of medical reasoning that could serve as the conceptual basis for a computer-based decision-support system (DSS). They described how logic, probability theory, and utility theory could serve as formal representations of the diagnostic process. Forty years later, has their vision been realized? Do DSSs help clinicians or their patients? If not, why have they failed?
In answering these questions, the most compelling evidence for the usefulness of DSSs would be studies demonstrating that they help patients lead longer or better lives. Without such studies, it would be helpful to show that DSSs assist clinicians in performing interventions, such as effective preventive services, that lead to better outcomes for patients. Although such studies would support the value of DSSs, they would leave open the question of whether changes in intermediate outcomes or in processes of care translate into improvements in length or quality of life.
In this issue of ecp, Kellett reports on use of a DSS to aid decisions regarding use of fibrinolytic therapy (3) in patients with suspected acute myocardial infarction. Also in this issue, Eytan and Goldberg describe possible designs for a trial to test the implementation of computer-based guidelines. (4) These interesting studies highlight the formidable challenges that investigators face when trying to determine whether DSSs are helpful. (5) In this editorial, we briefly review the literature on the effectiveness of DSSs and ask whether the current approach to development of such systems is likely to be fruitful.
Several systematic reviews have examined DSS use. (6-14) As shown in Table 1, studies evaluating DSSs have typically measured their effect on the process of care rather than on patient outcomes. We discuss findings in items of preventive care; drug dosing; diagnosis; and other aspects of medical care, including management. Because it was the most comprehensive and clinically relevant, we focus on the 1998 review by the McMaster group. (6)
Do DSSs Improve Process of Care?
Among the DSSs designed to improve process of care, those aimed at preventive interventions and drug administration have been the most successful. In the McMaster review, 14 of 19 studies of DSSs for preventive services found a positive effect on process of care. (6) A meta-analysis by Shea and colleagues (14) found improvement in rates of vaccination, improvement in rates of screening for breast and colorectal cancer, and a reduction in risk factors for cardiovascular disease. (An increase in rates of screening for cervical cancer did not reach statistical significance.) These studies provide convincing evidence that DSSs can improve the delivery of preventive services.
Of the 15 studies in the McMaster review that assessed the use of DSSs for drug dosing, 6 of 8 that evaluated administration of intravenous medications (e.g., theophylline, lidocaine, and heparin) found beneficial effects. Seven evaluated DSSs for warfarin dosing, but the results were inconsistent and did not permit broad conclusions. Three other systematic reviews also found convincing evidence of the effectiveness of DSSs for drug dosing.
Studies examining diagnostic DSSs have yielded disappointing results. For example, of the five studies on diagnostic DSSs that were included in the McMaster review, only one had significant positive results. More recently, Aase (15) found that the use of a diagnostic DSS for patients with chest pain was associated with a decrease in the number of patients referred unnecessarily to a monitored bed. Another study of two widely available general diagnostic DSSs for internal medicine, Quick Medical Reference (QMR) and Iliad, found that the programs improved clinicians' differential diagnoses. (16) However, because this trial did not assess the use of these programs in clinical care, the question of whether they would actually improve the process of care was not evaluated. Another potential problem with diagnostic DSSs is that they may considerably increase the length of consultations. (17, 18) To our knowledge, no general diagnostic DSSs are currently in widespread clinical use.
Nineteen of 26 studies in the McMaster review that addressed other aspects of care (e.g., management of diabetes and adherence to recommendations for general medical care) found improvements in the process of care. (6) In general, the review demonstrates that certain DSSs can improve adherence to established guidelines for care, although a substantial number of systems have failed.
Do DSSs Improve Patient Outcomes?
Only 14 of 68 studies in the McMaster review evaluated the effect of DSSs on patient outcomes; of these, 6 found that outcomes improved (Table 1). (6) The single study that examined a DSS for preventive care and evaluated patient outcomes found no change in blood pressure control. (19) One study of drug dosing found reduced complications and improved therapeutic benefit with heparin dosing. (20) The only study of a diagnostic DSS that evaluated patient outcomes found that using a system to identify patients at high risk for postoperative respiratory complications decreased rates of such complications. (21) Of 7 studies evaluating DSSs that dealt with aspects of medical care other than prevention, diagnosis, or drug dosing, 4 demonstrated improvements in patient outcomes. Of the 8 patient-outcome studies that found no improvements, 5 lacked sufficient statistical power to detect modest effects. (6)
A more recent study, which was not included in the McMaster review, examined use of an integrated DSS to manage administration of antibiotic agents at the LDS Hospital in Salt Lake City, Utah. This study demonstrated a reduction in adverse drug reactions, prescriptions for drugs to which patients were allergic, prescriptions for higher-than-recommended dosages, and mismatches in antibiotic susceptibility. In addition, the overall costs of antibiotic therapy decreased. (22) This study is important because it demonstrates that integrating a DSS with electronic patient data can improve outcomes.
In summary, for most DSSs, effect on patient outcomes has never been assessed. The few studies that have evaluated outcomes have had mixed results. Of the studies that were not positive, many were underpowered. Nonetheless, almost one half of the studies found that DSSs improved patient outcomes.
What Factors Are Related to DSS Success?
We can learn several general lessons from these studies about the characteristics that make a DSS more likely to succeed. First, the most successful DSSs fit well into the provider workflow. (23) Those that help clinicians perform tasks efficiently are more likely to be used than those that require additional effort. (10) For example, the effective antibiotic-management system at LDS Hospital required 3.5 seconds to obtain relevant information, whereas a human infectious-disease specialist required 14 minutes. (22) In addition, DSSs are of great benefit if they have easy-to-use interfaces and provide information at the point of care. (24) Finally, our view is that active systems that provide recommendations without provider input are more likely to be successful than passive systems from which the provider must actively seek advice. Reminder systems are usually active; diagnostic systems are often passive. Among the former, success is more likely if the DSS requires that providers acknowledge the reminder (25) or if the system provides personalized feedback. (26)
What Are the Barriers to DSS Implementation?
Although it is important as know what enhances the probability of success, we must also understand the obstacles that hinder it. Perhaps the most substantial barrier to DSS implementation is the lack of the necessary electronic infrastructure--especially electronic medical records, which allow a DSS to directly acquire patient data. Another challenge to widespread implementation is lack of a controlled medical vocabulary, which would facilitate data acquisition and processing. (23) A controlled vocabulary allows a DSS to determine that the terms heart failure, congestive heart failure, decreased left ventricular function, and low ejection fraction can refer to the same condition. Finally, fundamental questions about how to model and reason efficiently with medical knowledge remain unsolved. Some systems use probabilistic approaches to reasoning and require computational power that would exceed current capabilities, even if the relevant information could be modeled faithfully. Other systems use rule-based approaches that are difficult to apply to complex problems (e.g., general diagnosis) and require continual updating to reflect advances in medical knowledge.
Are We on the Right Path?
Several themes emerge from the growing literature on DSSs. Most studies have evaluated process of care, and a large proportion of those have demonstrated improvements with DSS use. The DSSs have worked well for relatively simple problems, such as providing reminders for preventive care or assisting with drug prescribing. The usefulness of these DSSs arises from their ability to use clinical information to answer straightforward questions, such as whether a certain drug is contraindicated. Another area in which initial results show promise is the use of computer-based guidelines. (6, 27) We believe that implementation of DSSs in these areas should be a high priority.
For the more complex problems of diagnosis and management, successful systems usually focus on a specific clinical problem, such as diagnosis of abdominal pain or choice of antibiotic agents. The challenges facing developers of broader diagnostic and management systems remain formidable. The diagnostic DSSs, QMR, and Iliad performed poorly in many of the same clinical scenarios that caused difficulty for physicians. (20) This finding illustrates the challenges of applying medical reasoning to complex cases.
Four decades after the essay by Ludley and Lusted was published, no general diagnostic systems are in widespread use. Have we been wishing on a star? We believe not. Rather than decrying the lack of comprehensive DSSs, we should recognize the value of computer systems that perform relatively simple tasks. Widespread implementation of DSSs for preventive interventions and drug prescribing, for example, could provide enormous benefit. The development of more comprehensive systems will depend on the availability of clinical information systems and on advances in ways to represent and reason with medical knowledge.
1. Harline L, Washington N. "When you wish upon a star." Available on Masquerade: Pinocchio [Original recording remastered] [Audio CD]: Oz; 1999.
2. Ledley RS, Lusted LB. Reasoning foundations of medical diagnosis. Science. 1959;130:9-21.
3. Kellett J. Decision support and the appropriate use of fibrinolysis in myocardial infarction. Eff Clin Pract. 2001;4:1-9.
4. Eytan TA, Goldberg HI. How effective is the computer-based clinical practice guideline? Eff Clin Pract. 2001;4:24-33.
5. Friedman, CP, Wyatt, J. Evaluation Methods in Medical Informatics. New York: Springer; 1997.
6. Hunt DL, Haynes RB, Hanna SE, Smith K. Effects of computer-based clinical decision support systems on physician performance and patient outcomes: a systematic review. JAMA. 1998;280:1339-46.
7. Johnston ME, Langton KB, Haynes RB, Mathieu A. Effects of computer-based clinical decision support systems on clinician performance and patient outcome. A critical appraisal of research. Ann Intern Med. 1994;120:135-42.
8. Montgomery AA, Fahey T. A systematic review of the use of computers in the management of hypertension. J Epidemiol Community Health. 1998;52:520-5.
9. Sullivan F, Mitchell E. Has general practitioner computing made a difference to patient care? A systematic review of published reports. BMJ. 1995;311:848-52.
10. Shiffman RN, Liaw Y, Brandt CA, Corb GJ. Computer-based guideline implementation systems: a systematic review of functionality and effectiveness. J Am Med Inform Assoc. 1999;6:104-14.
11. Balas EA, Austin SM, Mitchell JA, Ewigman BG, Bopp KD, Brown GD. The clinical value of computerized information services. A review of 98 randomized clinical trials. Arch Fam Med. 1996;5:271-8.
12. Walton R, Dovey S, Harvey E, Freemantle N. Computer support for determining drug dose: systematic review and meta-analysis. BMJ. 1999;318:984-90.
13. Austin SM, Balas EA, Mitchell JA, Ewigman BG. Effect of physician reminders on preventive care: meta-analysis of randomized clinical trials. Proc Annu Symp Comput Appl Med Care. 1994:121-4.
14. Shea S, DuMouchel W, Bahamonde L. A meta-analysis of 16 randomized controlled trials to evaluate computer-based clinical reminder systems for preventive care in the ambulatory setting. J Am Med Inform Assoc. 1996;3:399-409.
15. Aase O. Clinical experience with a decision support computer program using Bayes' theorem to diagnose chest pain patients. Cardiology. 1999;92:128-34.
16. Friedman CP, Elstein AS, Wolf FM, et al. Enhancement of clinicians' diagnostic reasoning by computer-based consultation: a multisite study of 2 systems. JAMA. 1999;282:1851-6.
17. Evans AR, Wilkes ER, Absolon PJ, Denner TJ, Kaye G. An interactive computerised protocol for the management of hypertension. Methods Inf Med. 1985;24:21-6.
18. Weingarten MA, Bazel D, Shannon HS. Computerized protocol for preventive medicine: a controlled self-audit in family practice. Fam Pract. 1989;6:120-4.
19. Barnett GO, Winickoff RN, Morgan MM, Zielstorff RD. A computer-based monitoring system for follow-up of elevated blood pressure. Med Care. 1983;21:400-9.
20. Mungall DR, Anbe D, Forrester PL, et al. A prospective randomized comparison of the accuracy of computer-assisted versus GUSTO nomogram-directed heparin therapy. Clin Pharmacol Ther. 1994;55:591-6.
21. Chase CR, Vacek PM, Shinozaki T, Giard AM, Ashikaga T. Medical information management: improving the transfer of research results to presurgical evaluation. Med Care. 1983;21: 410-24.
22. Evans RS, Pestotnik SL, Classen DC, et al. A computer-assisted management program for antibiotics and other antiinfective agents. N Engl J Med. 1998;338:232-8.
23. Musen MA, Shahar Y, Shortliffe EH. Clinical decision-support systems. In: Shortliffe EH, Perreault LE, Wiederhold G, Fagan LM, eds. Medical Informatics: Computer Applications in Health Care and Biomedicine. Springer-Verlag; 2000: 600-39.
24. Tierney WM, Hui SL, McDonald CJ. Delayed feedback of physician performance versus immediate reminders to perform preventive care. Effects on physician compliance. Med Care. 1986;24:659-66.
25. Litzelman DK, Dittus RS, Miller ME, Tierney WM. Requiring physicians to respond to computerized reminders improves their compliance with preventive care protocols. J Gen Intern Med. 1993;8:311-7.
26. Lobach DF. Electronically distributed, computer-generated, individualized feedback enhances the use of a computerized practice guideline. Proc AMIA Annu Fall Symp. 1996:493-7.
27. Owens DK. Use of medical informatics to implement and develop clinical practice guidelines. West J Med. 1998;168: 166-75.
Douglas K. Owens, MD, MS, 179 Encina Commons, Stanford University, Stanford, CA 94305-6019; telephone: 650-723-0933; fax: 650-723-1919; e-mail: firstname.lastname@example.org.