ACP American College of Physicians - Internal Medicine - Doctors for Adults

Effective Clinical Practice


How Effective Is the Computer-Based Clinical Practice Guideline?

Ted A. Eytan, Harold I. Goldberg

For author affiliations, current addresses, and contributions, see end of text.

See related editorial and Primer.

Context. The primary prevention of coronary artery disease in patients with diabetes could have a large impact on health care costs and outcomes. Guidelines for improving diabetic health indices are common, but significant challenges exist in implementing them.

General Question. How does integrating an evidence-based guideline into an electronic medical record affect patient care?

Specific Research Challenge. How can we implement the new guideline-enhanced medical record in a controlled manner and measure its impact on physician satisfaction, diabetes process measures, and the risk for cardiovascular disease?

Proposed Approach. All patients in the University of Washington system have an electronic Web-based medical record. Patients with diabetes will be randomly assigned to a guideline-enhanced or standard electronic medical record. The electronic medical record allows measurement of most clinical process measures and outcomes. Physician satisfaction will be measured by survey.

Potential Difficulties. Contamination may occur when guideline recommendations are applied to control patients as physicians gain experience with the guideline-enhanced record.

Take Home Points

Clinical practice guidelines have been defined as "systematically developed statements to assist practitioner and patient decisions about appropriate health care for specific clinical circumstances." (1) Guidelines are considered an important tool for improving the quality of health care because they can reduce variations in practice and change physician behavior to promote use of interventions supported by the best evidence available. (2) The dissemination of guidelines by themselves rarely improves the process of patient care. (3) The purpose of this proposed research is to determine whether integration of clinical practice guidelines in an electronic medical record system is an effective method of implementation.

Simple clinical reminders have been evaluated and shown to be useful in improving adherence to guideline process measures. (4) In diabetes, however, such improvement in process measures has not been tightly linked to improved outcomes. For example, as a result of the use of a simple point-of-care reminder system at the University of Washington in Seattle, Washington, 60% of diabetic patients adhere to the University of Washington's process recommendation that the hemoglobin Alc (HbAlc) level be measured every 6 months. However, the mean HbAlc level for these patients is 8.06%; only 30% of patients have HbAlc values less than the 7% recommended by the University of Washington's internal guidelines for diabetes care.

This discordance between process and outcomes partially explains why diabetes is cited as an area of medicine in which outcome-centered guideline implementation using computers is promising. (5-7) Another reason is the ease with which intermediate outcomes in diabetes can be measured by using laboratory data. A computerized guideline that builds on process reminders with evidence-based recommendations to take specific clinical actions (such as initiating lipid-lowering therapy or reducing blood pressure) may be able to leverage gains made in process reminders and outcomes.

We hypothesize that a complex guideline integrated at several appropriate points in an electronic medical record could increase accessibility and usability because complex assessments can be performed quickly. At the same time, point-of-care implementation will achieve consistent exposure to the principles of the guideline and the evidence behind it, allowing a lasting and effective educational intervention for providers and patients.


Different models proposed specifically for guideline implementation have been developed to overcome barriers to adherence, and each promotes interventions that address more than one key barrier. (3, 8, 9) An "information management services model" that describes the key components of successful guideline implementation has been proposed for the implementation of clinical practice guidelines. (10) Table 1 outlines these components and how they might be incorporated when a guideline is integrated into a Web-based medical record (already in use at the University of Washington). The combination of a successful and flexible electronic medical record (based on open standards) and a rigorous evidence-based guideline may be ideal for introducing evidence-based practice into a wide variety of environments.

Once the intervention has been designed, the next challenge is to test it and measure its effect on patient care. It is technically possible to set flags in the electronic medical record to alter the display of information according to which patient's record is being accessed or the identification number of the provider accessing the record. A randomized, controlled study is therefore feasible. The current production medical record system can also be queried interactively, on the Web, to measure such outcomes as new diagnoses, pharmacy prescribing patterns, or laboratory values.

Several methodologic challenges remain despite the ability to implement the intervention and measure its effects. Various levels of randomization bring advantages and disadvantages in terms of statistical power and threats to validity. In addition, measuring intermediate and long-term outcomes may be difficult in populations that receive care from more than one source. For example, if patients fill prescriptions at pharmacies outside of the University system, those prescriptions may not be captured in the medical record.


The Web-Based Medical Record

Consistent with the University of Washington's regional mission, construction of a large relational clinical database began in 1989. The goal of this database was to make both clinical and reference information available in real time to the primary and secondary care providers in the University of Washington's geographically diverse referral base.

Originally, the clinical database used interfaces with registration, billing, pharmacy, laboratory, radiology, pathology, and transcription computing systems to generate text patient records that were viewable on the local area network. In late 1995, however, medical information scientists began to collaboratively design a graphical HTML "front end" to the clinical database, making its contents accessible over the Internet by using standard browsers such as Netscape Navigator or Microsoft Internet Explorer. Security for this Web-based medical record allows only authenticated users to access the system, which stores the records of 612,000 patients in more than 500 tables and 624,000,000 rows, comprising a total of 115 gigabytes of data. In this full-scale production system, each day 1300 users generate, on average, over 39,000 Web-page "hits." (11) This system has evolved into the standard electronic medical record at our institution.

The Clinical Practice Guideline

The guideline to be used in our study--for the primary prevention of coronary artery disease--was developed at Group Health Cooperative, a local group-model HMO. It was implemented for its 700,000 consumers in Washington State in 1996 and was updated through a systematic review of evidence in August 1998. The guideline algorithms take into account patient age, sex, blood pressure, diabetic status, smoking status, cholesterol levels, evidence of left ventricular hypertrophy, and family history of premature coronary artery disease to calculate 5-year risk for symptomatic cardiovascular disease. (12) Providers using this guideline receive customized recommendations for frequency of lipid screening and intervention, as well as customized patient education pieces that can be used as a foundation for shared decision making. However, the guideline is not part of Group Health Cooperative's electronic medical record; this inhibits use of the guideline at the point of care and limits researchers' ability to measure its effect on outcomes.

Integration into the Electronic Medical Record

In contrast, the guideline at the University of Washington will be integrated throughout the medical record. Providers who access the records of eligible patients in the intervention group will receive assessments of cardiovascular disease risk and recommendations for management derived from the guideline in all relevant aspects of the medical record. Guideline assessments will appear when providers review laboratory data, medication regimens, recorded vital signs, or preventive intervention reminders provided by clinical reminder software that runs against the clinical database. In addition, a new screen that integrates all clinical data related to the patient's problem (view-integrated-by-problem screen) (Figure 1) will be created that consolidates clinical data and guideline-derived recommendations for cardiovascular disease prevention for diabetic patients. Providers will have the opportunity to review the presented recommendations and print customized patient education pieces. The new screen will save providers time by consolidating clinical data and recommendations for management in a single view. In contrast, as illustrated in Figure 2, users of the standard electronic medical record would have to access multiple clinical data screens, a separate reminders screen, and separate databases for the written guideline or would have to retrieve articles that provide evidence to support the guideline. The new screen prototype has been developed by using an interface engineering process in collaboration with the University of Washington Human Interface Technology Laboratory. We used an iterative development process in creating the guideline-enhanced record; in this process, physicians are observed giving care, information deficits are noted, a prototype is developed, and refinements are made in a stepwise fashion after the prototype is presented to providers for feedback. (13)

Physician Participants and Patient Sample

All providers of adult primary care at three University of Washington ambulatory medical clinics will participate. This includes 53 faculty physicians and 79 resident physicians with an average of 275 and 125 eligible patient visits/month, respectively. The 1116 patients with diabetes who are regular clinic users and who fill prescriptions at University of Washington pharmacies will be included. The average age of this population is 53.5 years; 53.8% of the participants are male, and 46.1% are white. The mean low-density lipoprotein (LDL) cholesterol level for this sample, as measured in the past year, is 121.3 mg/dL.


Table 2 summarizes process and outcome measures to be assessed and the source of each measure. Because a majority of the care given to patients at these sites is documented in the Web-based medical record, the authors will use this to assess many of the outcome measures according to the following categories.

Do Providers Like the Intervention?

Provider surveys will be used to assess the utility of the intervention before and after implementation. Providers who see diabetic patients in the intervention study will be questioned about their use of the intervention and their perceptions of the utility of the intervention. In assessing utility, they will be asked to compare their satisfaction with the guideline-enhanced record versus standard care. For example, physicians may be asked to rate their confidence in assessing coronary artery disease risk and recommending appropriate therapy in patients when the guideline-enhanced record is used compared with the standard medical record.

Do Providers Use the Intervention?

To assess this aspect of the intervention, the number of times the medical record system is accessed and the activity within the record will be monitored. Measures will include the percentage of patient visits in which the guideline is used, how often cardiovascular disease risk is reviewed, how often customized patient educational materials are generated, and the time (in minutes) that providers using the system spent reviewing customized recommendations.

Does the Intervention Improve Clinical Outcomes?

The primary prevention of coronary artery disease is best guided by knowing a patient's baseline cardiac risk. (14-17) Unfortunately, providers typically do not estimate risk correctly without the aid of specific tools; as a result, lipid screening and intervention may be inappropriate. (18-21) The goal of the evidence-based guideline is to improve the assessment of cardiovascular disease risk to maximize the benefit of lipid screening and lipid-lowering drugs. Several process and outcome measures will be collected continuously over the 12-month baseline and 12-month intervention periods, as summarized in Table 2. The primary end point will be mean change in 5-year risk for cardiovascular disease. The impact of the electronic guideline will be evaluated by using three prevention measures that are part of the evidence-based clinical practice guideline: LDL cholesterol level and cardiovascular disease risk at initiation of lipid-lowering therapy, lipid screenings per year at each risk level, and mean risk for cardiovascular disease before and after intervention. In addition, we will follow process and outcome measures that are currently prompted for by the medical record system as part of usual care: frequency of HbAlc screening and mean HbAlc value, frequency of proteinuria screening, smoking assessment frequency, and mean blood pressure.

Possible Study Designs

Level of Randomization

As shown in Table 3, several methods for selecting units of randomization are possible. Authors working with computerized reminder systems have advocated for the use of practices or clinics as the unit of randomization, in part because these office systems are typically implemented throughout a practice. (22) At the University of Washington we have an office system that could conceivably be implemented at the patient, provider, or clinic level. In our system, randomization by clinic would produce dramatically different clinical settings and patient populations, threatening the validity of the trial. Thus, here we focus on the choice between randomization by provider and randomization by patient.

Randomization at the provider level has several drawbacks. First, it may be difficult to obtain comparable groups given a limited number of providers. Provider attributes, such as willingness and ability to use electronic information systems, are likely to differ between the two groups. Furthermore, because providers work at geographically distinct clinics, patient characteristics are likely to differ between the two groups. From an analysis perspective, randomization at the provider level also introduces complexity by adding another correlation structure to the data; the outcomes are correlated by provider and statistical power to detect that an effect is reduced (for more on randomization by groups, see Primer). (23) Second, it may be difficult to maintain the treatment assignment. Providers sometimes share a single log-in session, even when timed lock-outs are implemented. Patients are cared for by multiple providers (part-time providers are common in our academic setting), making it likely that some patients would receive care in both groups.

Randomization by patient assures that all of a patient's care will be exposed to the intervention, guarding against cross-coverage that would dilute the intervention's effect. This produces an additional advantage--the reduction of bias due to differences in patient panels or provider attributes. In this case, each provider has patients in each group and serves as his or her own control with respect to adherence to the guideline.

An important disadvantage to randomization by patient is the possibility of carry-over, or contamination. This would happen if, for example, providers learned about the guideline through intermittent use of the view-integrated-by-problem screen and applied this knowledge to control patients. This concern has been expressed in several other studies that used randomization at the patient level for health services interventions. (24-26) However, the guideline selected for use in this trial requires complex calculations not reasonably achievable without the use of a structured system. The recommendations are also complex. These factors argue against significant carryover.

Proposed Approach

We propose a randomized, controlled trial in which patients serve as the units of randomization and analysis. Across three clinic sites, eligible patients will be randomly assigned to usual care or intervention groups. Process and outcome measures will be collected continuously over 12-month baseline and intervention periods. Independent-sample t-tests and longitudinal analysis will be used to examine whether significant improvements occur in the intervention group relative to the controls. The study design is illustrated in Figure 3.

An intervention as complex as the one presented here raises questions about the impact of the implementation itself versus the guideline. A paper-based version of the guideline presented to providers at the time of the patient visit may have as much effect as a version integrated at the level of the electronic medical record. It is also possible that awareness of cardiac risk factors will be increased by integrating diabetic status information on a single screen. An approach to measuring the effect of increased information availability and documentation requirements might be to give providers paper-based materials immediately before seeing a patient in the office, regardless of study group. As an alternate approach, a third intervention group of the study could be created: In this group, the medical record would feature enhanced display of diabetic measures but no guideline recommendations would be given. These additions to the design were ultimately not adopted for several reasons. First, it is unlikely that any clinical entity would have the resources to prepare customized, paper-based versions of guideline recommendations for individual patient visits at the point of care. Second, it is unlikely that an organization seeking to improve outcomes would implement a guideline within a medical record without consolidating key information in a single display. Finally, the accumulated literature on providers' adherence to guidelines shows that only the most tightly integrated and well-executed approaches to guideline implementation can change physician behavior.


The positive effect of computerized reminders at the point of care has been shown at the University of Washington and elsewhere. What has yet to be demonstrated is the ability of information systems to help providers synthesize clinical data in ways that are consistent with evidence-based approaches to care. (27) Open-source medical record systems provide many possibilities for designing interventions and measuring the effectiveness of these interventions.

We expect that an integrated clinical practice guideline that meets the needs of clinicians will improve the process and outcomes of care. The path to successful implementation, however, involves many variables, from the concordance of a new guideline with established standards of care to the accessibility of the guideline within an electronic medical record. After implementation, measurement of outcomes may be difficult. Finally, various threats to validity will occur throughout the design and execution of such a study. These issues can be successfully minimized to test the effect of an intervention that is portable across any health care institution that invests in Web-based technology.

Take Home Points
  • The technology now exists to integrate complex practice guidelines into an electronic medical record and to measure their effectiveness in terms of intermediate and long-range outcomes.
  • Design challenges exist in the measurement of guideline effectiveness. These include determining the level of randomization, controlling for contamination, and measuring appropriate processes and outcomes with available clinical data.
  • In addition to study design, physician use and satisfaction with integrated guidelines must be measured because they may significantly affect guideline effectiveness.


1. Field M, Lohr K. Clinical Practice Guidelines: Directions for a New Program. Washington, DC: National Academy Pr; 1990.

2. Greco PJ, Eisenberg JM. Changing physicians' practices. N Engl J Med. 1993;329:1271-3.

3. Grimshaw JM, Russell IT. Effect of clinical guidelines on medical practice: a systematic review of rigorous evaluations. Lancet. 1993;342:1317-22.

4. Goldberg HI, Neighbor WE, Cheadle AD, Ramsey SD, Diehr P, Gore E. A controlled time-series trial of clinical reminders: using computerized firm systems to make quality improvement research a routine part of mainstream practice. Health Serv Res. 2000;34:1519-34.

5. Griffin S, Kinmonth AL. Diabetes care: the effectiveness of systems for routine surveillance for people with diabetes. Cochrane Database Syst Rev. 2000; CD000541.

6. McDonald CJ, Overhage JM, Tierney WM, Abernathy GR, Dexter PR. The promise of computerized feedback systems for diabetes care. Ann Intern Med. 1996;124:170-4.

7. McDonald CJ, Overhage JM. Guidelines you can follow and can trust. An ideal and an example [Editorial]. JAMA. 1994;271:872-3.

8. Davis DA, Taylor-Vaisey A. Translating guidelines into practice. A systematic review of theoretic concepts, practical experience and research evidence in the adoption of clinical practice guidelines. CMAJ. 1997;157:408-16.

9. Duff L, Casey A. Implementing clinical guidelines: how can informatics help? J Am Med Inform Assoc. 1998;5:225-6.

10. Shiffman RN, Brandt CA, Liaw Y, Corb GJ. A design model for computer-based guideline implementation based on information management services. J Am Med Inform Assoc. 1999; 6:99-103.

11. Goldberg HI, Tarczy-Hornoch P, Stephens K, Larson EB, LoGerfo JP. Internet access to patients' records [Letter]. Lancet. 1998;351:1811.

12. Anderson KM, Odell PM, Wilson PW, Kannel WB. Cardiovascular disease risk profiles. Am Heart J. 1991;121: 293-8.

13. Brown M. Improving access to patient records: redesigning the provider/EMR interface. Master of Science Thesis. Seattle: University of Washington; 2000.

14. Screening for high blood cholesterol and other lipid abnormalities. In: Guide to Clinical Preventive Services: Report of the U.S. Preventive Services Task Force. 2nd ed. Baltimore: Williams & Wilkins; 1996:15-39.

15. Ramachandran S, French JM, Vanderpump MP, Croft P, Neary RH. Using the Framingham model to predict heart disease in the United Kingdom: retrospective study. BMJ. 2000;320:676-7.

16. Robson J, Boomla K, Hart B, Feder G. Estimating cardiovascular risk for primary prevention: outstanding questions for primary care. BMJ. 2000;320:702-4.

17. Wallis EJ, Ramsay LE, Ul Haq I, et al. Coronary and cardiovascular risk estimation for primary prevention: validation of a new Sheffield table in the 1995 Scottish health survey population. BMJ. 2000;320:671-6.

18. Friedmann PD, Brett AS, Mayo-Smith MF. Differences in generalists' and cardiologists' perceptions of cardiovascular risk and the outcomes of preventive therapy in cardiovascular disease. Ann Intern Med. 1996;124:414-21.

19. Grover SA, Lowensteyn I, Esrey KL, Steinert Y, Joseph L, Abrahamowicz M. Do doctors accurately assess coronary risk in their patients? Preliminary results of the coronary health assessment study. BMJ. 1995;310:975-8.

20. Smith GD, Song F, Sheldon TA. Cholesterol lowering and mortality: the importance of considering initial level of risk. BMJ. 1993;306:1367-73.

21. Yudkin JS. How can we best prolong life? Benefits of coronary risk factor reduction in non-diabetic and diabetic subjects. BMJ. 1993;306:1313-8.

22. Hunt DL, Haynes RB, Hanna SE, Smith K. Effects of computer-based clinical decision support systems on physician performance and patient outcomes: a systematic review. JAMA. 1998;280:1339-46.

23. McDonald CJ, Hui SL, Zhou XH. Analyzing computer-based decision support systems [Letter]. Ann Intern Med. 1994; 121:469.

24. Nilasena DS, Lincoln MJ. A computer-generated reminder system improves physician compliance with diabetes preventive care guidelines. Proc Annu Symp Comput Appl Med Care. 1995:640-5.

25. McDonald CJ, Hui SL, Smith DM, et al. Reminders to physicians from an introspective computer medical record. A two-year randomized trial. Ann Intern Med. 1984;100:130-8.

26. Katon W, Von Korff M, Lin E, et al. Collaborative management to achieve treatment guidelines. Impact on depression in primary care. JAMA. 1995;273:1026-31.

27. Montgomery AA, Fahey T, Peters TJ, MacIntosh C, Sharp DJ. Evaluation of computer based clinical decision support system and risk chart for management of hypertension in primary care: randomised controlled trial. BMJ. 2000;320:686-90.


Ted A. Eytan, MD, MS, MPH, Group Health Cooperative, 521 Wall Street, ACC-2, Seattle, WA 98121; telephone: 206-448-5523; e-mail:


The authors acknowledge the contributions of Matthew Handley, MD (Group Health Cooperative), and Matthew Brown, MSIE (University of Washington Human Interface Technology Laboratory).