Effective Clinical Practice
Effective Clinical Practice, November/December 2000.
Harold C. Sox, Jr, Steven Woloshin
For author affiliations, current addresses, and contributions, see end of text.
Context. The Institute of Medicine (IOM) report on medical errors created an intense public response by stating that between 44,000 and 98,000 hospitalized Americans die each year as a result of preventable medical errors.
Objective. To determine how well the IOM committee documented its estimates and how valid they were.
Methods. We reviewed the studies cited in the IOM committee's report and related published articles.
Results. The two studies cited by the IOM committee substantiate its statement that adverse events occur in 2.9% to 3.7% of hospital admissions. Supporting data for the assertion that about half of these adverse events are preventable are less clear. In fact, the original studies cited did not define preventable adverse events, and the reliability of subjective judgments about preventability was not formally assessed. The committee's estimate of the number of preventable deaths due to medical errors is least substantiated. The methods used to estimate the upper bound of the estimate (98,000 preventable deaths) were highly subjective, and their reliability and reproducibility are unknown, as are the methods used to estimate the lower bound (44,000 deaths).
Conclusion. Using the published literature, we could not confirm the Institute of Medicine's reported number of deaths due to medical errors. Due to the potential impact of this number on policy, it is unfortunate that the IOM's estimate is not well substantiated.
The Institute of Medicine (IOM) report on building a safer health system (1) created an intense public response by stating that the number of deaths due to preventable error in the United States is between 44,000 and 98,000 per year. The report cited two studies, one based on hospital discharges in New York in 1984 (2) and the other based on discharges in Colorado and Utah in 1992. (3) The federal government and professional organizations responded to the public outcry by pledging to reduce the number of errors in health care. Now, months after the release of the report, some are asking if the number of preventable deaths is really as large as the IOM report claims it to be. (4) By closely examining the two studies and the IOM committee's report, we had hoped to answer the question "How many deaths are due to preventable medical error?" Even after expanding our search to include related published articles, we are unable to answer this question. Without further examination of the original medical records, we doubt that anyone can.
Opportunities To Go Astray in Calculating the Number of Preventable Deaths Due to Error
The IOM committee did not show how it calculated the number of deaths due to preventable errors; as a result, any attempt to reproduce this calculation involves speculation about how the committee proceeded. Speculation is a necessary hazard for those, like ourselves, who want to see whether the IOM committee's estimates could be accurate. We organized our analysis by considering the data that would be necessary to calculate the number of Americans who die as a result of medical errors (Figure 1).
To judge whether the IOM committee could have accurately estimated the deaths due to preventable error, we examined the validity of the data required to make this calculation. In the following paragraphs, we consider each part of the calculation, other than the total number of admissions, which we assume to be correct. We summarize our findings in Table 1.
Proportion of Admissions with an Adverse Event
". . . adverse events occurred in 2.9% to 3.7% of hospitalizations"
The IOM stated that "adverse events occurred in 2.9% and 3.7% of hospitalizations," quoting the Utah-Colorado and New York State articles, respectively. (2, 3) The two studies used the same methods, which contained many checks to avoid misclassification of events (see the Appendix for a detailed summary of the two studies). Briefly, the search for adverse events began with nurses screening hospital records to detect findings that met at least one criterion for an increased risk for an adverse event; one of these criteria was death. Next, physicians reviewed screen-positive records for evidence of an adverse event. Both studies gave the reviewers a brief, broad definition of an adverse event--"injury caused by medical management (rather than by the underlying disease) and that prolonged the hospitalization, produced a disability at the time of discharge, or both"--and both relied on the reviewers' expert clinical judgment to decide whether an adverse event had occurred. When reviewers identified a possible adverse event, they used a 6-point scale to indicate their level of confidence that the occurrence was in fact an adverse event. The minimum level of confidence required to qualify as an adverse event was a score of at least 4 on the scale of 6. Interrater agreement about the presence of an adverse event (based on a sample of charts) was moderate, with a k statistic of 0.6 in one study (2) and 0.4 in the other. (3)
In short, the two cited studies clearly substantiate the IOM statement about the frequency of adverse events. The care that the authors took in these two studies is a model for studies involving subjective judgments. It seems unlikely that either study miscounted adverse events to a serious degree in either direction. If anything, their procedures would underestimate the number of adverse events for two reasons: They included only events that had been documented in the medical record, and they studied only hospitalized patients.
Proportion of Preventable Adverse Events
". . . over half of these adverse events resulted from medical errors and could have been prevented"
The two studies cited by the IOM committee do not themselves substantiate the committee's assertion about the proportion of adverse events resulting from preventable errors. In fact, neither of the original studies defined preventable adverse events or medical errors, and the authors never tried to estimate the number of preventable events. In a 1993 reanalysis of the New York study, (5) two of the original authors reviewed summaries of every adverse event in the study and classified the events as preventable (69.6%), potentially preventable (6.0%), or not preventable (24.4%) (see Table 1 for complete definitions). Unfortunately, they did not measure the interrater reliability of these judgments (the authors believed that an assessment was unnecessary since interrater reliability had been good in a similar but unpublished study). In a 1999 reanalysis of the Colorado-Utah study, two study investigators, including one of the authors of the New York study, reviewed summaries of all adverse events and judged about half to be preventable. (6) Although the interrater reliability was excellent, it is important to emphasize that the physicians judging preventability were examining a summary of the events and not the medical record itself. It is unlikely that the same degree of reliability would be achieved if reviewers were faced with the entire medical record (which contains much more information than the summary--most extraneous, but some undoubtedly germane); this concern was recently expressed by one of the original study authors. (7)
The two reanalyses found approximately the same proportion of adverse events to be preventable (50% to 70%). However, only three physicians were involved in making these judgments, and one of them participated in both studies. It will be important to confirm their findings by having other physicians decide when an adverse event was preventable. Table 2 reproduces three examples from authors of the Colorado-Utah study (6) and allows readers to ask themselves what constitutes a preventable adverse event.
Proportion of Preventable Adverse Events That Led to Death
". . . at least 44,000 Americans die each year as a result of medical errors . . . the number may be as high as 98,000"
The IOM committee did not show its calculations and assumptions but simply cited the original reports when it provided its estimate of 44,000 to 98,000 preventable deaths. (2, 3) Neither of the source studies attempted to define when a death resulted from a medical error. In fact, we could not find either figure in any of the major publications cited by the IOM committee.
How, in fact, did the IOM committee arrive at its estimate of deaths from "preventable medical errors"? A reanalysis of the New York data by Leape and colleagues, (5) published in 1993, describes the process used to estimate the upper bound of the range of deaths due to preventable errors cited in the IOM report. Extrapolating from the 13,451 adverse events associated with death in New York State to the U.S. population, they state that 198,000 adverse events led to death in 1984. They judged that 78% of cases in which an adverse event was the proximal cause of death were preventable. However, as they pointed out, a preventable adverse event does not necessarily mean a preventable death, since many of these patients would have died even if the adverse event had not occurred. Leape and colleagues then stated that "We estimate that about one-half of the deaths caused by iatrogenic AEs [adverse events] were preventable," by which they appear to mean deaths that would not have occurred without the adverse event. This statement is the clearest description of how the authors of these two studies decided that an adverse event caused a death. Leape and colleagues do not explain how they decided which adverse events led to a death that would not have occurred otherwise, and the reader is left to speculate that it was a well-informed guess based on their reading of the case summaries of patients who died. Earlier in this article, the authors calculated that 198,000 adverse events were associated with death in the United States in 1984, the year of their study. Half of 198,000 is 99,000 deaths, close to the number cited by the IOM committee.
The route to the lower bound of the estimate of deaths due to preventable error is shorter and somewhat less circuitous, but it still ends without reaching its destination. There are two pertinent articles. In the first, the authors estimate the number of patients who sustained a preventable adverse event and the number of deaths in these patients. (6) This article does not extrapolate to the entire U.S. population. In the second article, Thomas (3) extrapolated from the study sample in Utah and Colorado to the entire United States to estimate 64,809 deaths in patients with adverse events and 24,979 deaths in patients with adverse events due to negligence in 1992. This article does not cite an estimate of the number of preventable adverse events, and neither article contains an estimate of the number of deaths caused by preventable adverse events. These studies do not explain how the IOM committee could have calculated the lower bound of the number of deaths caused by preventable error.
In evaluating the studies discussed in this section, it is important to emphasize the highly subjective nature of a judgment that a death resulted from an adverse event. The Methods sections of both studies are entirely silent on the methods that the authors used to attribute a death to an adverse event. It is not clear, for example, whether the authors had a rule by which they routinely counted an adverse event in a patient who died as causing or contributing to the death. In 1999, one author of the New York study reviewed the records of patients who had a negligent adverse event and died (88% of the deceased patients who had an adverse event). (8) This author states that 14% of those patients were severely ill and that the adverse event tipped the balance toward death. In the remaining 86%, the error was a major factor leading to the patient's death. This report was disappointing to us because the author did not state criteria for attributing death to the adverse event. Furthermore, he did not engage a second physician to perform duplicate reviews of each case (with a measurement of interrater reliability) or, better still, excuse himself altogether and assign the reviews to physicians who were not authors of the study. In their original studies, the authors of the New York and Utah-Colorado studies used panels of physicians to identify cases in which adverse events occurred. They took great pains to maximize the accuracy of the physicians' judgments. In view of the public attention given to preventable deaths, they need to apply the same careful methods to the task of deciding when an adverse event caused a death.
In summary, the source of the upper bound of the IOM committee's estimate of deaths due to preventable error appears to be two investigators' estimate of the proportion of fatal adverse events in which the patient would have survived except for the adverse event (a good definition of "preventable death"). These authors did not describe any method for making this judgment, and it is not clear whether they based their estimate of "about half" by summing over a series of case-by-case judgments or whether they made a subjective assessment of the entire body of cases. We could not determine how the IOM committee estimated the lower bound of the proportion of preventable adverse events that caused a death.
If we have correctly described the situation, Americans do not have a credible estimate of the number of deaths caused by medical error. Without such an estimate, it is impossible to make an informed policy decision about how many of our limited resources should be devoted to reducing errors as opposed to other competing health needs. Our analysis shows that the IOM committee's key finding, the estimated number of preventable deaths caused by medical error, depends on (and would be highly sensitive to) the least well-substantiated measurement to come from the New York study and the Utah-Colorado study: the proportion of adverse events that led to a death that would not have occurred without the adverse event. In our view, the IOM committee has not completed its work. The committee should describe its calculations, make clear its assumptions, explain the limitations of the data that it used, and by some means, communicate these facts to the U.S. public. The authors of the New York study and the Utah-Colorado study also have some work to do. They need to measure, with the same methodologic rigor that characterizes their original studies, (2, 3) the number of cases in which an adverse event caused death in a patient who otherwise would have survived. Then, and only then, will the U.S. people have a credible estimate of the number of deaths caused by medical error.
The public's response to the published number of deaths due to medical error, 44,000 and 98,000, has given these numbers a life of their own, one that neither the authors of the studies nor the IOM committee may have anticipated. Our analysis is a sharp reminder that a scientific report, particularly one that raises serious concerns about the quality and safety of hospital care, must carefully describe the methods used to derive its findings. * The Institute of Medicine (IOM) report on medical errors stated that between 44,000 and 98,000 hospitalized Americans die each year as a result of preventable medical errors.
|Take Home Points
1. Kohn LT, Corrigan JM, Donaldson MS. Institute of Medicine (U.S.) Committee on Quality of Health Care in America. To Err Is Human: Building a Safer Health System. Washington, DC: National Academy Pr; 2000.
2. Brennan TA, Leape LL, Laird NM, et al. Incidence of adverse events and negligence in hospitalized patients. Results of the Harvard Medical Practice Study. N Engl J Med. 1991;324:370-6.
3. Thomas EJ, Studdert DM, Burstin HR, et al. Incidence and types of adverse events and negligent care in Utah and Colorado. Med Care. 2000;38:261-71.
4. McDonald CJ, Weiner M, Hui SL. Deaths due to medical errors are exaggerated in the Institute of Medicine report. JAMA. 2000;284:93-5.
5. Leape LL, Lawthers AG, Brennan TA, Johnson WG. Preventing medical injury. QRB Qual Rev Bull. 1993;19:144-9.
6. Thomas EJ, Studdert DM, Newhouse JP, et al. Cost of medical injuries in Utah and Colorado. Inquiry. 1999;36:255-64.
7. Brennan TA. The Institute of Medicine report on medical errors--could it do harm? N Engl J Med. 2000;342:1123-5.
8. Leape LL. Institute of Medicine medical error figures are not exaggerated. JAMA. 2000;284:95-7.
Dr. Woloshin is supported by a Veterans Affairs Career Development Award in Health Services Research and Development and a New Investigator Award from the Department of Defense Breast Cancer Research Program (DAMD17-96-MM-6712).
Harold C. Sox Jr., MD, Department of Medicine, Dartmouth-Hitchcock Medical Center, Lebanon, NH 03756; telephone: 603-650-7684; fax: 603-650-6122; e-mail: email@example.com.
*Answers to quiz in Table 2: A, preventable; B, preventable; C, not preventable.