ACP American College of Physicians - Internal Medicine - Doctors for Adults

Effective Clinical Practice

ORIGINAL ARTICLE

Can Evidence Change the Rate of Back Surgery? A Randomized Trial of Community-Based Education

Effective Clinical Practice, May/June 2001

Harold I. Goldberg, Richard A. Deyo, Vicky M. Taylor, Allen D. Cheadle, Douglas A. Conrad, John D. Loeser, Patrick J. Heagerty, Paula Diehr

For author affiliations, current addresses, and contributions, see end of text.

Context. Timely adoption of clinical practice guidelines is more likely to happen when the guidelines are used in combination with adjuvant educational strategies that address social as well as rational influences.

Objective. To implement the conservative, evidence-based approach to low-back pain recommended in national guidelines, with the anticipated effect of reducing population-based rates of surgery.

Design. A randomized, controlled trial.

Setting. Ten communities in western Washington State with annual rates of back surgery above the 1990 national average (158 operations per 100,000 adults).

Participants. Spine surgeons, primary care physicians, patients who were surgical candidates, and hospital administrators.

Intervention. The five communities randomized to the intervention group received a package of six educational activities tailored to local needs by community planning groups. Surgeon study groups, primary care continuing medical education conferences, administrative consensus processes, videodisc-aided patient decision making, surgical outcomes management, and generalist academic detailing were serially implemented over a 30-month intervention period.

Outcome Measure. Quarterly observations of surgical rates.

Results. After implementation of the intervention, surgery rates declined in the intervention communities but increased slightly in the control communities. The net effect of the intervention is estimated to be a decline of 20.9 operations per 100,000, a relative reduction of 8.9% (P = 0.01).

Conclusion. We were able to use scientific evidence to engender voluntary change in back pain practice patterns across entire communities.

Take Home Points

Much effort has been expended on the development and distribution of clinical practice guidelines over the past decade. (1, 2) Some believed that physicians and health systems would quickly and easily move practice into compliance with available scientific evidence. However, empirical studies were later surprisingly consistent in demonstrating that guidelines in isolation failed to produce voluntary changes in practice patterns. (3-5) To increase guideline effectiveness, a growing list of adjuvant educational strategies continues to be evaluated. (6) Controlled trials of interventions as diverse as continuing medical education (CME) sessions, detailing techniques borrowed from the pharmaceutical industry, performance feedback, interactive videodisc technologies, and the use of respected opinion leaders have been conducted. (7-11)

Although the range of disease conditions, provider types, patient populations, and clinical behaviors studied makes comparisons across trials difficult, recent reviews have labeled overall findings as being "mixed." (12-15) Analogies are often made to stepped-care drug prescribing: Although no single educational "magic bullet" exists, guideline recommendations are more likely to be adopted when multiple strategies are used in combination. Noting the general ineffectiveness of approaches appealing purely to reason, however, others have counseled that more attention should be paid to the role of such social influences as habit, the beliefs of peers, and the existence of community norms. (16-18)

We previously reported that rates of back surgery across Washington State were well above the U.S. average, and the U.S. average, in turn, was the highest in the developed nations studied. (19-21) We acknowledged that high rates alone were not necessarily indicative of suboptimal practice. However, variations in rates of back surgery do not appear to be related to the prevalence of spine disease. (19) They are more often found to be related to nonbiologic factors, such as provider supply and physician uncertainty about indications. An expert panel convened by the Institute of Medicine concluded that "surgery for chronic back pain is overused and often misused." (22) Thus, we anticipated that adoption of the conservative, evidence-based approach to low-back pain recommended in guidelines developed by the Agency for Health Care Policy and Research would reduce local surgical rates to a level closer to the national average. (23)

To implement the guidelines, we chose to conduct a randomized, controlled trial of a multifaceted package of educational interventions intended to synergistically modify community practice norms. Although they focused on surgeons themselves, activities also involved primary care physicians, patients, and hospital administrators. The trial addressed the question of whether rates of low-back surgery would decline among residents of exposed communities.

Methods

Site Selection

To select trial communities, we defined the geographic boundaries of 87 Washington State communities based on their constituent hospital market areas (Codman Research Group, Lebanon, NH). Communities located more than 150 miles from the University of Washington in Seattle were excluded to limit traveling distances both for research staff coordinating community-based activities and for providers who came to meetings held in Seattle. We excluded communities with annual rates of back surgery below the 1990 national average (158 per 100,000 adults) and those that had fewer than 100 persons receiving back operations each year between 1987 and 1989. To ensure that the relevant health professionals worked closely together and were involved in the study, we also eliminated sites where most operations performed on residents were referred to surgeons practicing outside the community.

Reasoning that our ability to influence practice might vary by community size, the remaining 10 sites eligible for inclusion were stratified as being urban, suburban, or rural on the basis of their 1990 census populations. Within strata, sites were randomized by coin flip to study or control status. This resulted in one urban, three suburban, and one rural community being assigned to each trial arm. Figure 1 summarizes the inclusion and exclusion criteria and design of the study.

Baseline demographic, health system, and back-pain characteristics were measured, including those variables previously found to be significantly related to variation in county rates of low-back surgery across Washington State. (24)

Interventions

Details of trial interventions and their implementation have been reported elsewhere. (25-27) Planning groups comprising hospital and provider representatives were convened in each of the five study communities to consider data on surgical rates and variability and a menu of potential educational activities proposed by investigators. Group members were invited to tailor these activities to their respective community's unique needs and cultures, as well as suggest other activities they believed would be useful locally. (28) As a result of this planning intervention, six additional active interventions were selected, which targeted specific constituencies involved in decisions on back surgery (Table 1).

Between two and four influential "high-volume" orthopedists and neurosurgeons, defined as those performing at least 25 low-back operations per year, were invited from each community to attend regional "study group" meetings to consider observed variation in surgical rates and its implications. (29) Because these physicians were respected opinion leaders, it was expected that group members would share the information obtained with colleagues in their home communities. With the concurrence of these surgeons, an outcomes management effort was initiated to generate more detailed information on provider-specific styles of practice and surgical results. (30) To accomplish this goal, physicians and patients completed questionnaires at baseline and at 6 months following surgery. The questionnaires included information on examination findings, diagnostic test use, operative complications, symptom relief, and return to work. To provide feedback in a peer-comparison format, coded summaries were prepared by project staff and periodically mailed to participating surgeons.

Two complementary interventions targeting primary physicians were undertaken. Inside each community, CME conferences were scheduled that emphasized minimal bed rest, avoidance of early imaging, and appropriate criteria for surgical referral. Attendees were given patient education brochures and office wall charts illustrating the limited value of spinal radiographs. (31) A generalist opinion leader was recruited from each community to help develop and distribute academic detailing materials based on the content of these lectures. (32) Meant to simulate drug advertisements, all three detailing sheets had an eye-catching headline and cartoon to highlight essential messages. Complete clinical information and supportive references were printed on the reverse side.

Each study community was offered the use of computer hardware and interactive videodiscs to implement a shared decision-making program for patients who were surgical candidates. (33) This program incorporated outcome data from published studies as well as interviews with patients who had experienced either favorable or disappointing results of various medical and surgical treatment options.

Small discussion groups of key administrative personnel were formed at participating hospitals to examine comparative financial data from all 10 trial communities as part of a local consensus process. (34) The need for administrators and clinicians to work collaboratively in improving cost-effectiveness was stressed. Led by an economist investigator, the discussion groups had topics ranging from the increased lengths-of-stay associated with fusion operations to the posting of negative estimated profits per hospitalization for back pain during the previous calendar year in four of the hospitals. Details regarding the degree of participation, or penetrance, for each of the individual intervention components are included in Table 1.

Outcome: Low-Back Surgical Rate

Data on surgical procedures were obtained from the Comprehensive Hospital Abstract Reporting System (CHARS), a computerized registry of all nonfederal hospital discharges occurring within Washington State. Maintained by the Department of Health, these data become available 18 to 24 months after the close of each calendar year. The trial, divided into 30-month baseline and intervention periods, was conducted over the 5-year period from July 1, 1989, to June 30, 1994. Surgical rates could not be assessed beyond this time because of the advent of ambulatory discectomy procedures not included in our inpatient registry. (35) To identify patients undergoing surgery for mechanical back problems in the lumbosacral region, we applied an algorithm that excluded admissions associated with nonmechanical causes of pain, such as malignant conditions, infections, and major trauma. (36) The population of trial communities in even-numbered years was estimated using data from the 1990 census (Strategic Mapping, Santa Clara, CA). Values for odd-numbered years were estimated by interpolation.

Each community's experience was then expressed as a quarterly surgical rate (the number of age- and sex-adjusted procedures performed on residents per 100,000 adults). Because procedures were attributed according to place of residence (as opposed to where surgeries were performed), cases referred in or out of trial communities neither inflated nor lowered rates artificially.

Analysis

The trial used a pretest-posttest experimental design. Although randomization assures even-handedness in assignment to treatment groups, the very small number of units typically available for randomization in studies of community-level interventions can lead to baseline differences in prognostic or dependent variables or inadequate statistical power. (37) To address both issues, we performed a controlled time-series trial analysis. (38) Here, longitudinal observations from parallel study and control interrupted time-series experiments are subtracted to produce a "difference" interrupted time-series experiment. (39) The trial protocol was approved by the University of Washington's Institutional Review Board. (40)

Means for the five study and five control communities were computed and plotted as two time series. Each time series comprised 10 baseline (third quarter of 1989 to the fourth quarter of 1991) and 10 intervention-period observations (first quarter of 1992 to the second quarter of 1994). To estimate the net impact of the intervention package, each control time-series observation was subtracted from the corresponding value in the study time series to generate a third time series representing the difference between group means over time. Because the individual intervention components had been serially implemented for logistic reasons, we hypothesized that effects of the entire package would be expressed as a gradual linear trend. Therefore, a hinged least-squares regression was fitted to the difference time series that included time (quarters numbered from 1 to 20) and the interaction between time and intervention period. This interaction term was used to test whether the rate of change in mean difference between the study and control communities--that is, the slope of the "difference regression"--had significantly changed between the baseline and intervention periods. The regression was estimated using Econometric Views software (Quantitative Micro Software, Irvine, CA). Correlogram and Dubin-Watson statistics were calculated to test for serial correlation. Because the correlation was neither large nor statistically significant, no adjustments used in autoregressive models (e.g., lagged terms) were made when fitting the regression. All tests of significance employed an a-level of 0.05.

Results

Table 2 shows that the populations of the study and control communities had similar demographic characteristics. Notably, there were no important differences in 3 variables previously found to be related to geographic rates of low-back surgery: proportion of workforce in lumber/construction or transportation (positively correlated with back surgery); proportion of households receiving foodstamps (positively correlated); and hospital bed occupancy (negatively correlated).

Plots demonstrating changes in rates of low-back surgery over time are included in Figure 2. The implementation of the intervention package is superimposed over the trial timeline. The difference between study- and control-group means over time is plotted in Figure 3. Study- and control-group rates were minimally divergent over the baseline period, as confirmed by the slightly positive slope of the difference regression during this time. Without intervention, the two groups would have been expected to continue to diverge by 0.85 operations per 100,000 adults each quarter. Instead, mean surgical rates began to converge as educational activities were initiated, generating the larger, negative slope observed in the difference regression over the intervention period (0.85 vs. -1.24; P =0.01).

Thus, rather than increasing, on average, by 8.5 operations per 100,000 adults over the second half of the trial, study group rates actually declined relative to those in the control communities by 12.4, producing a net intervention effect (equal to 10 times the change in the slope of the difference regression) of 20.9. Indeed, at trial's end, the mean rate in the study group was below that of the control communities, a reversal of their historic position throughout the entire baseline period. To relate this magnitude of decline to previous performance, we divided the net intervention effect of 20.9 by 235, the study group's average annual rate over the entire baseline period. This yielded a relative decline from preintervention rate levels of 8.9%.

Discussion

We sought to determine the effect of a package of educational strategies aimed at making the care of back pain more evidence-based. We anticipated that doing so would reduce population-based rates of low-back surgery in communities with rates above the national average. To invoke social influences, the effort enlisted primary physicians, patients, and hospital administrators--as well as the surgeons controlling the decision to operate on patients once a referral is made. Decisions to perform elective surgery involve complex interactions within and between these clinical constituencies. To bring rational forces to bear on the decision-making process, content emphasized the scientific underpinnings of the conservative approach advocated by national guidelines made publicly available in December of 1994, just after the trial's conclusion. Because members of our research team were involved in preparing the guidelines, intervention content mirrored guideline content. Together, the interventions resulted in a decline in surgical rates of 8.9%. The decline was not attributable to unintended changes in the numbers of surgeons practicing locally. The average number of spine surgeons practicing in study versus control communities each quarter was persistently higher for the study group (approximately four versus three) over the trial's tenure, which may provide a partial explanation for these communities' higher rates of surgery at baseline. The decline in surgery also did not occur at the expense of quality as measured by 1-year reoperation rates. This measure was also unchanged across baseline and intervention periods in both groups.

The magnitude of this result should be interpreted in context. Occurring in an overwhelmingly fee-for-service environment, the decline ran counter to prevailing financial incentives. This suggests that results may be conservative compared with those that would be obtained if the intervention were implemented today. The percentage of the population in managed care has increased in Washington State from 16.4% to 35.6% over the past decade, which raises awareness of the importance of cost-effectiveness. (41, 42) Typical of effectiveness studies in the community setting, the penetrance achievable by various intervention components was necessarily low. For example, because of resource limitations, only 48% (12 of 25) of surgeons meeting our volume criterion attended group meetings. By the trial's conclusion, although 3515 surgeries were actually performed in the five study communities during the intervention period, only 247 surgical candidates had viewed the videodisc presentation. This also suggests that our results may be conservative. Yet, whether the effect size would increase if the same intervention were applied today to a prepaid, managed care population is a question for future research. One can only speculate that this might be the case, given a better alignment of financial and professional incentives, as well as an ability to directly involve a greater percentage of providers and patients in educational activities.

Still, even a moderate effect in percentage terms assumes increased importance in absolute terms when applied over a large geographic area. The fact that 3515 observed procedures represented an 8.9% reduction from the number that would be expected without intervention, suggests a total "savings" of 343 surgeries. The total cost of the intervention was approximately $40,000 per study community per year. Some of the expense of developing and pretesting components, such as the videodisc and detailing materials, however, would not be incurred in implementing the package elsewhere.

Our use of repeated-measure statistical techniques warrants comment. In addition to accounting for the baseline imbalances we encountered, the use of a controlled time-series trial analysis seems to have increased precision. An 80% power estimate performed during trial planning indicated that a decline in surgery rates of nearly 40% would have been required to reach significance if we had used a traditional mixed-model analysis-of-variance approach with single pretest and posttest observations. (43) The effect sizes of guideline program's attempt to reduce performance of an overused procedure are small compared with efforts seeking to substitute treatments or increase underutilized ones. Thus, steadfast application of a nonlongitudinal analysis would have probably resulted in the commission of a type II error. However, because so few sites were involved--as is the case with many group-level program-evaluation trials that focus on internal validity--caution should be taken in generalizing results beyond the particular communities studied. As computerized registries and repositories become more commonplace, the longitudinal datasets they contain will become increasingly accessible. It will be interesting to see if controlled longitudinal analyses of various kinds are applied to other questions previously thought unexaminable because of methodologic limitations imposed in having only very small numbers of clusters, such as communities or clinics, to affordably compare. (44) Finally, several limitations of our trial warrant explication. The design precludes meaningful assessment of the individual intervention components. Readers are referred to the many efficacy trials of solitary educational interventions included in the referenced reviews. Our findings also cannot address the relative effectiveness of noneducational strategies that might be used to improve guideline compliance, such as changes in health regulations, financial incentives, and local capacity. (45, 46) Because the advent of ambulatory discectomy in Washington State made it impossible to track statewide rates of back surgery after 1994, we were unable to evaluate what happened after the intervention's conclusion. Because the only outcome available from the CHARS registry was the crude quality indicator of reoperation rates, we acknowledge that our trial cannot address whether reducing surgical rates improved functional outcomes. One recent study does, however, provide evidence that better outcomes are associated with lower population-based rates. Compared with the 60% of patients in the highest-rate area in Maine who exhibited marked or complete relief of pain, 79% of patients in the lowest-rate area were equally improved at 4 years after surgery. Improvement in Roland disability scores, satisfaction, and quality of life were also significantly greater among patients in the lower-rate area. (47)

In conclusion, there is growing consensus that clinical practice should become increasingly based on rigorous scientific evidence if the quality and cost-effectiveness of the nation's health care is to be improved. (48) We were able to use such evidence to engender voluntary change in back-pain practice patterns across entire communities.

Take Home Points
  • Because surgery for chronic back pain is overused, we anticipated that adoption of national guidelines for low-back pain would reduce surgical rates in western Washington State.
  • We conducted a randomized, controlled trial among 10 communities of a multifaceted package of educational interventions intended to synergistically modify community practice norms.
  • The intervention included surgeon study groups, primary care CME conferences, administrative consensus processes, videodisc-aided patient decision making, surgical outcomes management, and generalist academic detailing, which were serially implemented over a 30-month intervention period.
  • The net effect of this package was approximately a 9% reduction in the rate of low-back surgery in the five intervention communities.
  • Guidelines are more likely to be adopted when attention is paid to the role of social influences, such as the beliefs of peers and the existence of community norms.

References

1. Audet AM, Greenfield S, Field M. Medical practice guidelines: current activities and future direction. Ann Intern Med. 1990;113:709-14.

2. VanArmringe M, Shannon TE. Awareness, assimilation and adoption: the challenge of effective dissemination and the first AHCPR-sponsored guidelines. QRB Qual Rev Bull. 1992;18:397-404.

3. Kosecoff J, Kanouse DE, Rogers WH, McCloskey L, Winslow CM, Brook RH. Effects of the National Institutes of Health consensus development program on physician practice. JAMA. 1987;258:2708-13.

4. Lomas J, Anderson GM, Domnick-Peirre K, Vayda E, Enkin MW, Hannah WJ. Do practice guidelines guide practice? The effect of a consensus statement on the practice of physicians. N Engl J Med. 1989;321:1306-11.

5. Lomas J. Words without action? The production, dissemination, and impact of consensus recommendations. Ann Rev Public Health. 1991;12:41-65.

6. Goldberg HI, Cummings MA, Steinberg EP, et al. Deliberations on the dissemination of PORT products: translating research findings into improved patient outcomes. Med Care. 1994;32:JS90-JS108.

7. Maiman LA, Becker MH, Liptak GS, Nazarian LF, Rounds KA. Improving pediatricians' compliance-enhancing practices. A randomized trial. Am J Dis Child. 1998;142:773-9.

8. Soumerai SB, Salem-Schatz S, Avorn J, Casteris CS, Ross-Degnan D, Popovsky MA. A controlled trial of educational outreach to improve blood transfusion practice. JAMA. 1993;270:961-6.

9. Johnson CC, Martin M. Effectiveness of a physician education program in reducing consumption of hospital resources in elective total hip replacement. South Med J. 1996;89:282-9.

10. Barry MJ, Cherkin DC, Chang Y, Fowler FJ, Skates S. A randomized trial of a multimedia shared decision-making program for men facing a treatment decision for benign prostatic hypertrophy. Disease Management and Clinical Outcomes. 1997;1:5-14.

11. Gifford DR, Holloway RG, Frankel MR, et al. Improving adherence to dementia guidelines through education and opinion leaders. A randomized, controlled trial. Ann Intern Med. 1999;131:237-46.

12. Grimshaw JM, Russell IT. Effect of clinical guidelines on medical practice: a systematic review of rigorous evaluations. Lancet. 1993;342:1317-22.

13. Oxman AD, Thomson MA, Davis DA, Haynes RB. No magic bullets: a systematic review of 102 trials of interventions to improve professional practice. CMAJ. 1995;153:1423-31.

14. Davis DA, Taylor-Vaisey A. Translating guidelines into practice. A systematic review of theoretic concepts, practical experience and research evidence in the adoption of clinical practice guidelines. CMAJ. 1997;157:408-15.

15. Solomon DH, Hashimoto H, Daltroy L, Liang MH. Techniques to improve physicians' use of diagnostic tests: a new conceptual framework. JAMA. 1998;280:2020-7.

16. Mittman BS, Tonesk X, Jacobson PD. Implementing clinical guidelines: social influence strategies and practitioner behavior change. QRB Qual Rev Bull. 1992;18:413-22.

17. Conroy M, Shannon W. Clinical guidelines: their implementation in general practice. Br J Gen Prac. 1995;45:317-75.

18. Stross JK. Guidelines have their limits. [Editorial] Ann Intern Med 1999;131:304-6.

19. Cherkin DC, Deyo RA, Loeser JD, Bush T, Waddell G. An international comparison of back surgery rates. Spine. 1994;19:1201-6.

20. Taylor VM, Deyo RA, Cherkin DC, Kreuter W. Low back pain hospitalization. Recent United States trends and regional variations. Spine. 1994;19:1207-12.

21. Taylor VM, Deyo RA, Goldberg H, Ciol M, Kreuter W, Spunt B. Low back pain hospitalization in Washington State: recent trends and geographic variations. J Spinal Disord. 1995;8:1-7.

22. Institute of Medicine. Overuse of surgery in chronic back pain. In: Osterweis M, Kleinman A, Mechanic D, eds. Pain and Disability: Clinical, Behavioral and Public Policy Perspectives. Washington DC; National Academy Press; 1987:204.

23. Bigos S, Bowyer O, Braen G, et al. Acute low back problems in adults. Clinical Practice Guideline No. 14. AHCPR Publication No. 95-0642. Rockville, MD: Agency for Health Care Policy and Research, Public Health Service, U.S. Department of Health and Human Services. December 1994.

24. Volinn E, Mayer J, Diehr P, Van Koevering D, Connell FA, Loeser JD. Small area analysis of surgery for low-back pain. Spine. 1992;17:575-81.

25. Taylor VM, Goldberg HI, Deyo RA, et al. Modifying community practice styles: the back pain outcome assessment team information dissemination effort. Journal of Continuing Education in the Health Professions. 1996;16:203-14.

26. Spunt BS, Deyo RA, Taylor VM, Leek KM, Goldberg HI, Mulley AG. An interactive videodisc program for low back pain patients. Health Educ Res. 1996;11:535-41.

27. Taylor VM, Deyo RA, Ciol M, et al. Patient-oriented outcomes from low back surgery: a community-based study. Spine. 2000;25:2445-52.

28. Sorensen G, Emmons K, Hunt MK, Johnston D. Implications of the results of community intervention trials. Annu Rev Public Health. 1998;19:379-416.

29. Keller RB, Soule DN, Wennberg JE, Hanley DF. Dealing with geographic variations in the use of hospitals. The experience of the Maine Medical Assessment Foundation Orthopaedic Study Group. J Bone Joint Surg. 1990;72:1286-93.

30. Ellwood PM. Shattuck lecture--outcomes management. A technology of patient experience. N Engl J Med. 1988;318:1549-56.

31. Browner WS, Baron RB, Solkowitz S, Adler LJ, Gullion DS. Physician management of hypercholesterolemia. A randomized trial of continuing medical education. West J Med. 1994;161:572-8.

32. Soumerai SB, Avorn J. Principles of educational outreach ("academic detailing") to improve clinical decision making. JAMA. 1990;263:549-6.

33. Barry MJ, Fowler FJ, Mulley AG, Henderson JV, Wennberg JE. Patient reactions to a program designed to facilitate patient participation in treatment decisions for benign prostatic hyperplasia. Med Care. 1995;33:771-82.

34. Putnam RW, Curry L. Impact of patient care appraisal on physician behavior in the office setting. CMAJ. 1985;132:1025-29.

35. Wohns RNW, Robinett RD. Day surgery for lumbar microdiskectomy. Experience with 60 cases. Ambulatory Surg. 1996;4:31-3.

36. Cherkin DC, Deyo RA, Volinn E, Loeser JD. Use of the international classification of diseases (ICD-9-CM) to identify hospitalizations for mechanical low back problems in administrative databases. Spine. 1992;17:817-25.

37. Koepsell TD, Wagner EH, Cheadle AC, et al. Selected methodologic issues in evaluating community-based health promotion and disease prevention programs. Annu Rev Public Health. 1992;13:31-57.

38. Goldberg HI, Neighbor WE, Cheadle AD, Ramsey SD, Diehr P, Gore E. A controlled time-series trial of clinical reminders: using computerized firm systems to make quality-improvement research a routine part of mainstream practice. Health Serv Res. 2000;34:1519-34.

39. Diggle PJ, Liang KY, Zeger SL. Analysis of Longitudinal Data. Oxford: Clarendon Press; 1994:72-7.

40. Goldberg HI, McGough H. Testing the implementation of clinical guidelines. IRB: A Review of Human Subjects Research. 1991;13:1-7.

41. Marion Merrell Dow Managed Care Digest. HMO Edition. Kansas City, MO: Marion Merrell Dow; 1991.

42. HMO-PPO/Medicare-Medicaid Digest. Kansas City, MO: Hoechst Marion Roussel; 1999.

43. Koepsell TD, Martin DC, Diehr PH, et al. Data analysis and sample size issues in evaluations of community-based health promotion and disease prevention programs: a mixed-model analysis of variance approach. J Clin Epidemiol. 1991;44:701-13.

44. Goldberg HI, Horowitz CR. Musings on using evidence to guide CQI efforts toward success: the computerized firm system as primary care microunit. Jt Comm J Qual Improv. 1999;25:529-38.

45. Eisenberg JM. Doctors' Decisions and the Cost of Medical Care. Ann Arbor, MI: Health Administration Press Perspective; 1986.

46. Wennberg JE, Freeman JL, Culp WJ. Are hospital services rationed in New Haven or over-utilised in Boston? Lancet. 1987;1:1185-9.

47. Keller RB, Atlas SJ, Soule DN, Singer DE, Deyo RA. Relationship between rates and outcomes of surgery for lumbar disc herniation and spinal stenosis: is more better? J Bone Joint Surg. 1999;81:752-62.

48. Sackett DL, Richardson SW, Rosenberg W, Haynes RB. Evidence-Based Medicine: How to Practice and Teach EBM. New York: Churchill-Livingstone; 1997.

Acknowledgments

The authors thank research assistants Scotte Cooper, Mark Leek, Lori Nordgulen, and Brandi Spunt; William Kreuter and Marcia Ciol for their help with data analysis; and Cheryl Tabachnick, Mary Mullen, and Sandra Coke for their invaluable secretarial assistance.

Grant Support

By grants HS06344 and HS08194 from the Agency for Health Care Policy and Research, Rockville, Maryland.

Correspondence

Harold I. Goldberg, MD, Division of General Internal Medicine, Harborview Medical Center, 325 Ninth Avenue, Box 359780, Seattle, WA 98104; telephone: 206-341-4433; fax: 206-731-8247; e-mail: hig@u.washington.edu.