Effective Clinical Practice
Effective Clinical Practice, November/December 2000.
Timothy P. Hofer, Eve A. Kerr, Rodney A. Hayward
For author affiliations, current addresses, and contributions, see end of text.
Context. Launched by the Institute of Medicine's report, "To Err is Human," the reduction of medical errors has become a top agenda item for virtually every part of the U.S. health care system.
Objective. To identify existing definitions of error, to determine the major issues in measuring errors, and to present recommendations for how best to proceed.
Data Source. Medical literature on errors as well as the sociology and industrial psychology literature cited therein.
Results. We have four principal observations. First, errors have been defined in terms of failed processes without any link to subsequent harm. Second, only a few studies have actually measured errors, and these have not described the reliability of the measurement. Third, no studies directly examine the relationship between errors and adverse events. Fourth, the value of pursuing latent system errors (a concept pertaining to small, often trivial structure and process problems that interact in complex ways to produce catastrophe) using case studies or root cause analysis has not been demonstrated in either the medical or nonmedical literature.
Conclusion. Medical error should be defined in terms of failed processes that are clearly linked to adverse outcomes. Efforts to reduce errors should be proportional to their impact on outcomes (preventable morbidity, mortality, and patient satisfaction) and the cost of preventing them. The error and the quality movements are analogous and require the same rigorous epidemiologic approach to establish which relationships are causal.
The medical error movement was announced to great fanfare with the report from the Institute of Medicine (IOM), "To Err is Human: Building a Safer Health Care System." (1) It was presented with drama, "Betsy Lehman died of a drug overdose"; alarm, "These horrific cases are just the tip of the iceberg"; and an assertion of lack of previous attention, "Yet silence surrounds this issue. . .The status quo is not acceptable and cannot be tolerated any longer." The chosen anecdotes reflect an emphasis on events that are clearly boneheaded, often fatal, flaws. There follows a plea to the medical profession to remember its promise to "do no harm" and that "at a very minimum, the health system needs to offer that assurance and security to the public." The clear implication is that even if health care providers are not able to help people, they should not kill patients at a rate (estimated at 44,000 to 98,000 per year) that each year exceeds the entire loss of U.S. servicemen over a decade of war in Vietnam.
The IOM report, which is echoed in many other articles describing the error movement, (2, 3) presents attention to error as a new and different approach to improving care. Indeed, through frequent references to the well-worn phrase from the Hippocratic oath, "First, do no harm," the proponents of the medical error movement imply that eliminating errors should come "first," before anything else on our agenda. However, the definition of medical error offered by the IOM is much more inclusive than the examples that are principally used to motivate the movement. Drawing from a cognitive psychology literature that analyzes industrial and transportation accidents, error is defined as "the failure of a planned action to be completed as intended (i.e., error of execution) or the use of a wrong plan to achieve an aim (i.e., error of planning)." This definition, in the broadest sense, would include much more than the anecdotal randomly removed legs and catastrophic drug overdoses.
So what exactly is medical error? How does the search to identify and remove error differ from the widespread efforts over the past 15 to 20 years to monitor, profile, and improve quality of care? Does the focus on eliminating error provide us with a way to substantially hasten improvement in health care? The answers to these questions are critical to successfully and productively harness the momentum generated by this new movement.
Definitions of Error
A patient scheduled for an amputation of the right leg has the left leg removed.
A patient is discharged from the hospital after myocardial infarction without having a b-blocker prescribed.
A hospitalized patient with multiple medical problems dies of cardiac arrest. The endotracheal tube inserted during the resuscitation is found to be in the right bronchus.
While waiting for correction of coagulopathy, a patient with overwhelming infection, multiorgan failure, and pleural effusion dies before having thoracentesis to check for empyema.
Which of these incidents represent a medical error? Is there a difference between them? If so, which types contribute to the alarming estimates of medical errors in our health care system? It seems fairly easy to label the first two vignettes as errors, but why is the first vignette so much more shocking than the second? The second, from a population perspective, is far more common and likely to result in death. Yet the first error appears to be more of an individual, immediate tragedy, and the second is lost in a population statistic. There are many examples of this apparent paradox, such as the public mobilization to help in the individual tragedy of a little girl who falls in a well versus our inattention to the population tragedy of AIDS in sub-Saharan Africa.
The third and fourth vignettes are more complex. The third vignette represents a conspicuous error in the intubation of a patient. However, this failure almost certainly had no effect on outcome, given the survival rate of hospitalized patients who have cardiac arrest. Do failed interventions represent an error when they do not affect the outcome? Finally, in the fourth case, it may be debatable in this clinical setting whether the intervention will improve the outcome (should pleural effusion be sampled even in the setting of coagulopathy?). Furthermore, the outcome clearly could affect an assessment of what the error was. If the patient died without having the procedure, the omission might be labeled an error. If the patient died of a bleeding complication after thoracentesis, the decision to do the procedure might be considered an error.
How Have Medical Errors Been Defined?
Table 1 shows the major studies used to estimate the rate of medical errors, (4-11) none of which were designed to study error. The Harvard Medical Practice Study in New York State and its successor in Colorado and Utah were designed to study medical malpractice and the potential costs of tort reform. (4-7) As part of this study, the investigators identified adverse events that were caused by negligence, defined in strict legal terms as care that did not adhere to the community standard. The primary outcome measured was an adverse event caused by medical care. A secondary assessment determined whether the adverse event was negligent (e.g., due to substandard care). The reliability of the assessment of negligence was poor. Negligent adverse events were referred to as preventable, although preventability was not directly measured. The other population-based study that is used to describe medical error rates is the Quality in Australian Health Care Study. (8) The principal measured outcome of this study was again adverse events caused by medical care according to a definition that was virtually identical to that of the Harvard Medical Practice Study.
In separate reports that followed these studies, the adverse events were classified into categories of errors that "could have caused" the adverse event, but there was no assessment of the degree to which the reviewers felt that the error was causally related to the adverse event or the reliability of the reviewer classification. (9, 12) Preventability was quantified, but few data were provided about the reliability of the assessment of preventability or what the reviewers meant when they said an event was "preventable." Although some of the investigators from these studies clearly feel that any preventable adverse event represents an error, (12) others have noted that their assessments of preventability were not generalizable (given that they were done by the investigators) and that "not all preventable adverse events were blunders." (13)
Another often-cited study on adverse drug events that uses definitions and inferences about errors that parallel those in the Harvard Medical Practice Study is confined to drug-related events. (10) This study assessed adverse drug events and as a secondary measure, preventable adverse drug events. The investigators variously defined preventable adverse drug events as "due to an error or. . .preventable by any means currently available" (10) and more simply, as all preventable adverse events being due to errors. (11)
Several small studies that rely on participant observation or self-report are cited to illustrate much higher rates of errors than the rates that are documented by looking only at adverse events. (14-16) These studies often use the much broader definitions of errors favored by the IOM report, such as "a deviation from standard conduct, as well as addition or omission of actions relating to standard operational instructions or routines of the unit." (14) This includes, for example, incorrectly adding a patient's fluid input and output figures on the ICU flow sheet. Although these are some of the only studies that have actual error as the primary measured variable, none of these studies examine the reliability of the error measurement or the causal link between the errors and adverse outcomes.
Other studies sometimes discuss errors in anecdotal cases, (17) report types of adverse events without attribution of error, (18-23) or measure other specific subsets of preventable adverse events: cardiac arrests (24) and preventable deaths. (25)
In short, virtually all of the empirical evidence about errors is from studies that first identify bad outcomes (called adverse events). Error is defined principally as a problem in the process of care. It is never measured explicitly but is one of several criteria reviewers use to judge an adverse event as being preventable or negligent. A subset of adverse events can be judged preventable, although preventability has not been rigorously measured. Whereas preventable or negligent adverse events are described as a subset of errors producing a lower bound to estimates, the conceptual definitions of these terms overlap to an unknown degree as shown in Figure 1. We have found no studies that were designed to directly measure the strength of the relationship between errors and adverse events and few that even refer to the need to do so.
The IOM report and other publications that discuss error in medicine at a theoretical or conceptual level refer repeatedly to nonmedical literature on human error. This literature grew out of the study of catastrophic industrial and transportation accidents and migrated into the medical literature. (1, 12, 26-28) The principal sources cited in arguing for what is called a systems approach or human factors engineering approach to medical errors are James Reason, a psychologist who wrote a book called Human Error, and Charles Perrow, a sociologist who published a book called Normal Accidents. The basic premise of both books is that complex technological systems produce the conditions that allow errors to occur and that attention to system design is more productive than blaming the individuals who are ultimately responsible. (29-32) The principal recommendations adopted by the IOM from this literature are to use a comprehensive approach to discovery, analyze all errors regardless of whether they result in accidents, and redesign the system to eliminate the errors.
Although the premise of this research seems sensible, we should point out several caveats. First, the
evidence that this approach works and is efficient is equivocal. The standard proof involves a comprehensive examination of the system associated with the causes of the catastrophic accident (Three Mile Island, the Challenger explosion, various train or airplane crashes) followed by re-engineering and observing that no further accidents have occurred (for a while). Of course, these accidents occur at such a low rate that it is often difficult to infer much about whether the process has improved safety. The examples are usually anecdotal, and cause and effect are generally inferred from a case study (called a "root cause analysis"). Perrow's point that these root cause analyses are "profoundly compromised. . .We do not know what to look for in the first place, and we jump to the most convenient explanations (culture or bad conditions) in the second place" is usually ignored in the industrial and medical literature that draws on his work, (32) as is Reason's statement that "Some concerns need to be expressed about the theoretical and practical utility of this ever-spreading quest for contributing factors." (33)
A second caveat is contained within Perrow's book itself. The title, Normal Accidents, is a term he uses to describe inevitable accidents. In fact, his writings reflect a deep skepticism that errors and consequent accidents can be eliminated. His book illustrates how difficult it is to classify and manage errors and gives numerous examples in which it cannot be done. In the end, his argument is that for many complex systems, we must weigh the benefits against the potential catastrophes. If the catastrophes are worrisome enough, we should get out of that business (nuclear power being his primary example). This gloomy assessment represents one pole of philosophical debate. At the other end are the members of the High Reliability Organization Project at the University of California at Berkeley, a group that tends to believe that you can always "find it and fix it." (34, 35) This group represents the optimist site of error management that the IOM appears to have adopted.
Finally, and most important, both Perrow and Reason point out that the cure can be worse than the disease. Reason notes that "An unquestioning belief in the attainability of absolute safety can seriously impede the achievement of realizable safety goals" and that defenses and safeguards can themselves cause catastrophic breakdowns of systems. (36) Likewise, Perrow cautions that attempts to fix some complex systems may simply add to complexity, thereby increasing rather than reducing accidents. (32)
What Are the Big Issues in Describing Medical Errors?
In an example of latent errors from his book, Perrow gives an account of a series of mishaps that result in his missing an important appointment. These include oversleeping, difficulty making coffee, leaving the car key in the house, locking himself out of the house, coincidentally having lent out his spare key (usually kept in the bushes), a bus strike, and lack of taxis due to the bus strike. He then asks for the cause of this foul-up to be picked from a list of errors and answers as follows:
The best answer is not "all of the above" or any one of the choices, but rather "none of the above.". . . The cause of the accident is to be found in the
complexity of the system. That is, each of the
failures--design, equipment, operators, procedures, or environment--was trivial by itself." (31)
In this example, how do we measure the error? If (trivial) failures occur everywhere and all the time, then whenever there is an adverse event, we will find lots of these latent errors even when they played little or no role in producing the adverse event. Furthermore, it will be difficult to find and measure these errors precisely because of their multiplicity, apparent triviality, and unclear consequence. In this way, the extraordinarily broad definitions of error favored by those in the error movement complicate measurement.
The principal method used to measure problems in health care further contributes to the measurement problem. All of the literature to date has used some form of implicit assessment to identify adverse events and errors whether they use self-report, (14, 16) observers, (14, 15) physician review, (4, 6, 8, 10, 25) or research investigator review. (5, 7, 13, 14) The low reliability of these assessments is a major issue in using error analysis to improve health care. In fact, the Harvard Medical Practice Study has shown that it is even hard for reviewers to decide if an adverse event has occurred. Interrater reliability of physician reviewers in this study was 0.6 for whether an adverse event had even occurred, implying that almost half of the measurement was composed of noise. The authors do not report the reliability for the subsequent decision, whether the adverse event was preventable, or whether it "could have been caused by a reasonably avoidable error." (5) However, assessing negligence is a task of similar difficulty, and the reliability of even two reviewers for that assessment was 0.24, implying that 75% of the measurement was composed of noise.
What is the implication of all this measurement noise? It seriously undercuts the premise of case-based error analysis proposed by the human factors engineers. With interrater reliabilities of 0.2 to 0.3, to support an investigation and response to the potential causes of an error, several independent reviews of a single case must be done even to be sure that an error occurred. In most cases, reviewers cannot agree on whether a bad outcome occurred or whether it was caused by substandard care, much less what exactly caused it. Fortunately, the obvious and disastrous errors often cited in anecdotes are rare.
Cause and Effect, the Forgotten Principle
Suppose we identified a series of blood transfusion reactions and found that a set of process problems labeled as errors had occurred in 60% of patients who had reactions. Now, suppose that in transfusions in which no reaction occurred there was also an error rate of 60%. Can we argue that the errors caused the adverse event? Can we infer that by engineering out the errors, transfusion reactions would be eliminated? It is clear that we cannot.
The paradox of the error movement is that while catastrophic outcomes are used prominently to motivate a search for problems in the process of care, there is often no attention given to even the preliminary epidemiologic evidence that would normally be expected when inferring a relationship between cause and effect. By way of contrast, in much of the mainstream quality-of-care literature, the relationship between structure and process on the one hand and outcomes on the other is discussed extensively as an essential part of quality care assessment. (37-41) In the error literature, not only does a tendency exist to quickly accept that the association is synonymous with causation, often no attempt is made to establish that there is an association between the "error" and the outcome. The argument that the complexity of a system makes it difficult to trace a direct relationship between many trivial failures and an outcome can be used to explain any failure to find evidence for cause-and-effect relationships.
The error movement clearly identifies a global, top-down management intervention to change organizational culture as one of the principal tools in promoting safety and improving care. (30, 42) As stated in the keynote address from the first Enhancing Patient Safety Conference, Anaheim, California, "Let me suggest the outlines of the steps we should be taking now on the meta-system--the management system in which the [workers who strive for the safety] of our future will either thrive or be silenced." (43) This holistic philosophical approach to improving care differs fundamentally from a probabilistic approach that demands a more narrow and quantifiable measure of cause and effect. The holistic approach is inspirational and simple to present, although difficult to carry out, and may be the best we can do in cases where adverse events are catastrophic but too rare or difficult to study in a rigorous empirical fashion. On the other hand, to those in medicine who laboriously conduct and evaluate trials that are carefully designed to demonstrate the benefits of clinical interventions, the loose standards of proof used to infer causality in the error literature are bemusing at best.
How Should We Proceed?
In determining how to respond to the IOM report and how to convert this enthusiasm into a constructive agenda, we must address the lack of a workable operational definition of error, the imprecision in the measurement and conceptual overlap of many of the terms (such as adverse events, preventability, and substandard care), and the consequent lack of good estimates of the magnitude of the problem. In short, we still lack a good answer to the question, "What is an error?" that we asked at the beginning of this paper. To move forward, we suggest the following steps.
Redefine Error as a Failure Clearly Linked to an Adverse Event
Is error inherent in a process of care, in an outcome, or both? The IOM definition would suggest a focus on process (e.g., a plan not carried out as intended)--there is no mention of outcomes. Can we define an error independent of outcome? Quality researchers have also found their attention meandering between process and outcome over the years but keep returning to Donabedian's definition that quality of care is a conceptual entity represented by the entire continuum from process to outcome and not by either one independently. (44) We argue that error must be defined in the same way. A failure of a structure or process is an indication of error only to the extent that it prevents maximizing the outcomes of interest to the patient. As such, we do not directly measure error. We measure specific attributes of structures, processes, or outcomes, and we infer error through an argument that rests critically on the strength of the link between structure, process, and outcome. (39, 44, 45) It is this argument that is largely missing from the existing literature on medical errors.
Establish Priorities among Errors
With this definition we can begin to define, measure, and quantify the attributes of errors and establish priorities. A very small subset of errors demands attention because its existence undermines both the public's and the profession's confidence in the whole system. What distinguishes these errors? They represent egregious failures of a structure or process of care that directly results in a bad outcome. Recalling the four vignettes from the beginning of the paper, we can plot these scenarios along the two dimensions of causality and egregiousness (Figure 2). Clearly, removal of the wrong leg has this high degree of causality and egregiousness. But we know that this type of error is rare, in large part because of the consistent and widespread lack of agreement between reviewers of the literature on whether any individual adverse event was caused by substandard care. (4, 46-49)
What should we do about the vast majority of errors, the murky mess in which we suspect that things could have turned out better? We need to focus on these errors on the basis of a different approach: the likelihood that they cause serious harm, how easy it is to prevent them, and a rigorous assessment of potential adverse consequences of the changes required. Rather than a crusade against all errors, we argue for a focused and targeted approach. This focus cannot be achieved by case study and stories, although a few of these may suggest a starting point. Establishing priorities ensures that our resources will not be wasted. It also allows us to better quantify the magnitude of the error problem.
Apply a Rigorous Epidemiologic Approach to the Assessment of Medical Error
Our refined definition of error pushes us away from anecdote, hindsight, and "sloganism" toward a rigorous epidemiologic evaluation of how best to identify and prevent errors that were previously advocated for profiling and improving quality. (39, 44, 45) Returning to our question about the differences between the quality movement and the error movement, we find that in both cases we need to measure adverse events or outcomes and rigorously trace the relationship to a specific set of modifiable structures or processes of care. In the end, we argue that if implemented appropriately, the error movement differs little from the quality movement. Many studies have documented significant quality problems in U.S. health care, and the potential benefit of the "error movement" is to promote efforts to address these problems. We define errors synonymously with poor-quality care and caution that much of the enthusiasm for focusing on errors results from the attention commanded by talking about bad outcomes and a misplaced hope that through a focus on a new entity called "error," we can cast off the complexities and difficulties we know to be involved in measuring and improving quality.
|Take Home Points
1. Kohn LT, Corrigan J, Donaldson MS, Institute of Medicine (U.S.) Committee on Quality of Health Care in America. To Err Is Human: Building a Safer Health System. Washington, DC: National Academy Pr; 1999.
2. Weingart SN, Wilson RM, Gibberd RW, Harrison B. Epidemiology of medical error. BMJ. 2000;320:774-7.
3. Reinertsen JL. Let's talk about error. BMJ. 2000;320:730.
4. Brennan TA, Leape LL, Laird NM, et al. Incidence of adverse events and negligence in hospitalized patients. Results of the Harvard Medical Practice Study I. N Engl J Med. 1991;
5. Leape LL, Brennan TA, Laird N, et al. The nature of adverse events in hospitalized patients. Results of the Harvard Medical Practice Study II. N Engl J Med. 1991;324:377-84.
6. Thomas EJ, Studdert DM, Burstin HR, et al. Incidence and types of adverse events and negligent care in Utah and Colorado. Med Care. 2000;38:261-71.
7. Thomas EJ, Orav EJ, Brennan TA. Hospital ownership and preventable adverse events. J Gen Intern Med. 2000;15:211-9.
8. Wilson RM, Runciman WB, Gibberd RW, Harrison BT, Newby L, Hamilton JD. The Quality in Australian Health Care Study. Med J Aust. 1995;163:458-71.
9. Wilson RM, Harrison BT, Gibberd RW, Hamilton JD. An analysis of the causes of adverse events from the Quality in Australian Health Care Study. Med J Aust. 1999;170:411-5.
10. Bates DW, Cullen DJ, Laird N, et al. Incidence of adverse drug events and potential adverse drug events. Implications for prevention. ADE Prevention Study Group. JAMA. 1995;274:29-34.
11. Leape LL, Bates DW, Cullen DJ, et al. Systems analysis of adverse drug events. ADE Prevention Study Group. JAMA. 1995;274:35-43.
12. Leape LL. Error in medicine. JAMA. 1994;272:1851-7.
13. Brennan TA. The Institute of Medicine report on medical errors-could it do harm? N Engl J Med. 2000;342:1123-5.
14. Donchin Y, Gopher D, Olin M, et al. A look into the nature and causes of human errors in the intensive care unit. Crit Care Med. 1995;23:294-300.
15. Andrews LB, Stocking C, Krizek T, et al. An alternative strategy for studying adverse events in medical care. Lancet. 1997;349:309-13.
16. O'Neil AC, Petersen LA, Cook EF, Bates DW, Lee TH, Brennan TA. Physician reporting compared with medical-record review to identify adverse medical events. Ann Intern Med. 1993;119:370-6.
17. Feldman SE, Rundall TG. PROs and the health care quality improvement initiative: insights from 50 cases of serious medical mistakes. Med Care Rev. 1993;50:123-52.
18. Schimmel EM. The hazards of hospitalization. Ann Intern Med. 1964;60:100-10.
19. Steel K, Gertman PM, Crescenzi C, Anderson J. Iatrogenic illness on a general medical service at a university hospital.
N Engl J Med. 1981;304:638-42.
20. Davis JW, Shapiro MF, Kane RL. Level of care and complications among geriatric patients discharged from the medical service of a teaching hospital. J Am Geriatr Soc. 1984;32:427-30.
21. Becker PM, McVey LJ, Saltz CC, Feussner JR, Cohen HJ. Hospital-acquired complications in a randomized controlled clinical trial of a geriatric consultation team. JAMA. 1987;257:2313-7.
22. Jahnigen D, Hannon C, Laxson L, LaForce FM. Iatrogenic disease in hospitalized elderly veterans. J Am Geriatr Soc. 1982;30:387-90.
23. Reichel W. Complications in the care of five hundred elderly hospitalized patients. J Am Geriatr Soc. 1965;13:973-81.
24. Bedell SE, Deitz DC, Leeman D, Delbanco TL. Incidence and characteristics of preventable iatrogenic cardiac arrests. JAMA. 1991;265:2815-20.
25. Dubois RW, Brook RH. Preventable deaths: who, how often, and why? Ann Intern Med. 1988;109:582-9.
26. Chassin MR. Is health care ready for Six Sigma quality? Milbank Q. 1998;76:565-91.
27. Bates DW, Gawande AA. Error in medicine: what have we learned? Ann Intern Med. 2000;132:763-7.
28. Leape LL, Berwick DM. Safe health care: are we up to it? BMJ. 2000;320:725-6.
29. Reason JT. Human error. New York: Cambridge Univ Pr; 1990.
30. Reason J. Human error: models and management. BMJ. 2000;320:768-70.
31. Perrow C. Normal accidents living with high-risk technologies. New York: Basic Books, 1984.
32. Perrow C. Normal accidents living with high-risk technologies. 2nd ed. Princeton, NJ: Princeton Univ Pr; 1999.
33. Reason JT. Managing the risks of organizational accidents. Brookfield, VT: Ashgate; 1997.
34. Zuckerman L. Is complexity interlinked with disaster? New York Times. 1999 Dec 11;Sect. B:11.
35. Pool R. Beyond engineering: how society shapes technology. New York: Oxford Univ Pr; 1997.
36. Reason J. Safety paradoxes and safety culture. Injury Control and Safety Promotion. 2000;7.
37. Donabedian A. The definition of quality and approaches to its assessment. Ann Arbor, MI: Health Administration Pr; 1980.
38. McAuliffe WE. A validation theory for quality assessment. In: Pena JJ, Haffner AN, Rosen B, Light DW, eds. Hospital Quality Assurance: Risk Management and Program Evaluation. Gaithersburg, MD: Aspen; 1984.
39. Eddy DM. Performance measurement: problems and solutions. Health Aff (Millwood). 1998;17:7-25.
40. Jencks SF. Measuring quality of care under Medicare and Medicaid. Health Care Financ Rev. 1995;16:39-54.
41. Brook RH, McGlynn EA, Cleary PD. Quality of health care. Part 2: measuring quality of care. N Engl J Med. 1996;335:966-70.
42. Roberts K. Organizational change and a culture of safety. In: Proceedings of Enhancing Patient Safety and Reducing Errors in Health Care. Chicago, IL: National Patient Safety Foundation at the AMA; 1999.
43. Berwick D. Taking action to improve safety: how to increase the odds of success. In: Proceedings of Enhancing Patient Safety and Reducing Errors in Health Care. Chicago, IL: National Patient Safety Foundation at the AMA; 1999.
44. Donabedian A. Quality assessment and assurance: unity of purpose, diversity of means. Inquiry. 1988;25:173-92.
45. Hofer TP, Bernstein SJ, Hayward RA, DeMonner S. Validating quality indicators for hospital care. Jt Comm J Qual Improv. 1997;23:455-67.
46. Goldman RL. The reliability of peer assessments of quality of care. JAMA. 1992;267:958-60.
47. Rubenstein LV, Kahn KL, Reinisch EJ, et al. Changes in quality of care for five diseases measured by implicit review, 1981 to 1986. JAMA. 1990;264:1974-9.
48. Hayward RA, Bernard AM, Rosevear JS, Anderson JE, McMahon LF Jr. An evaluation of generic screens for poor quality of hospital care on a general medicine service. Med Care. 1993;31:394-402.
49. Rubin HR, Rogers WH, Kahn KL, Rubenstein LV, Brook RH. Watching the doctor-watchers. How well do peer review organization methods detect hospital care quality problems? JAMA. 1992;267:2349-54.
50. Brennan TA, Leape LL. Adverse events, negligence in hospitalized patients: results from the Harvard Medical Practice Study. Perspect Healthc Risk Manage. 1991;11:2-8.
51. Brennan TA, Hebert LE, Laird NM, et al. Hospital characteristics associated with adverse events and substandard care. JAMA. 1991;265:3265-9.
52. Feldman SE, Roblin DW. Medical accidents in hospital care: applications of failure analysis to hospital quality appraisal. Jt Comm J Qual Improv. 1997;23:567-80.
Timothy P. Hofer, MD, MSc, PO Box 130170, Ann Arbor, MI 48113-0170; telephone: 734-930-5100; fax: 734-930-5159; e-mail: firstname.lastname@example.org.