Effective Clinical Practice
Effective Clinical Practice, January/February 2002
Stacey L. Sheridan, MD, MPH, Michael Pignone, MD, MPH
For author affiliations, current addresses, and contributions, see end of text.
Context. Although the ability to work with numbers is important to the practice of medicine, little is known about physician numeracy (basic skill with numbers).
Objective. To test medical students' numeracy and how it relates to the ability to interpret risk-reduction information.
Design. Randomized, cross-sectional survey.
Sample. 62 first-year medical students at the University of North Carolina at Chapel Hill medical school who attended a risk-communication seminar and had usable survey data (46% of the 134 students who received the survey).
Intervention. Students were given information about the baseline risk for developing a hypothetical disease and were randomly assigned to one of four risk-reduction presentations--relative risk reduction, absolute risk reduction, number needed to treat (NNT), or a combination of these three formats--about how two drugs would reduce this risk.
Outcome Measures. Number of correct answers to three numeracy questions (stating that 500 heads would be expected in 1000 coin flips; converting "1% of 1000" to 10; and converting "1 in 1000" to 0.1%). Correct data interpretation was judged with two tasks: a comparative task (i.e., state which drug provides greater benefit) and a quantitative task (i.e., calculate how much one of the drugs reduces disease risk).
Results. 77% of students answered all three numeracy questions correctly; 18% answered two correctly; and 5% answered one or none correctly. While 90% correctly stated which drug worked better, only 61% accurately interpreted the quantitative data. The ability to interpret data varied with numeracy: 71% of students who answered all three numeracy questions correctly also accurately interpreted the quantitative data, compared with 36% who answered two questions correctly and 0% who answered one or no questions correctly (P < 0.01). Correct quantitative interpretation was lower with the NNT format than with the other three formats (25% vs. 75%; P = 0.01).
Conclusions. Almost one quarter of first-year medical students in our study had trouble performing basic numerical tasks. Those who had trouble also seemed to have difficulty interpreting medical data. This difficulty seemed to be exacerbated by presenting data in the NNT format.
The ability to work with numbers is important for physicians, who are constantly interpreting test results, calculating drug dosages, and making treatment decisions. In addition, with the growing emphasis on evidence-based practice, physicians are increasingly expected to consult a medical literature that is replete with numbers to inform their clinical activities. Despite this, little is known about doctors' skill with numbers (numeracy).
Although low numeracy has been shown to be prevalent in the general population (1) and in selected patient populations, (2-4) with the exception of a single letter to the editor (5) we know of no attempt to assess physician numeracy. How numeracy skills relate to the ability to interpret medical data is also unknown.
In this study, we assessed the numeracy skills of first-year medical students at a major U.S. medical school and examined how these skills related to the ability to interpret data on treatment benefit.
Design and Sample
In 1999, we conducted a survey of first-year medical students at the University of North Carolina who were attending a required seminar on risk communication. This seminar discussed only qualitative dimensions of risk, such as the timing of risk, permanence of risk, and differing preferences for risk. No formal quantitative instruction was given.
One hundred fifty-nine students attended the risk-communication seminar, and 149 were approached at its conclusion to participate in our study. Seventy-seven students completed the self-administered questionnaire, but 15 questionnaires were excluded from the present analysis (which focuses on quantitative interpretations) because students received only qualitative presentations of risk information. This reduced our total eligible questionnaires to 134 and gave us a response rate of 46% (62/134).
Our 5-page, 20-item questionnaire (available on request from the authors) was informally pilot-tested for clarity among clinical research fellows at the University of North Carolina. In addition to demographic information (age, sex, race, occupation before medical school), we also asked about pastimes that require the use of risk concepts, such as gambling or playing the lottery, since such experience might influence their ability to interpret risk information.
We assessed numeracy by using a three-question numeracy scale adapted from Schwartz and colleagues. (2) Blank lines (i.e., ___ out of 1000 persons) were provided for responses. The questions and correct answers are given in Table 1. Blank responses were considered to be incorrect. We also asked students to rate their own skill with numbers.
Ability To Interpret Treatment Benefit
Figure 1 provides an overview of how we presented data on treatment benefit and how we assessed whether students correctly interpreted the data.
Students were given data about the risk for developing a hypothetical disease and were randomly assigned to receive information about two treatments that would reduce this risk in one of four risk-reduction formats--relative risk reduction (RRR), absolute risk reduction (ARR), number needed to treat (NNT), or a combination of these three formats (COMBO).
Students were asked to interpret data in two ways: in a comparative task (i.e., state which drug provided greater benefit) and in a quantitative task (i.e., calculate how much one of the drugs reduced disease risk). For the comparative task, students were asked to circle the correct answer. Response choices included "A is more effective than B," "B is more effective than A," "A and B are equally effective," and "Don't know." For the quantitative task, the students were asked to fill in their answer on a blank line.
To examine the relationship between numeracy and data interpretation, we used chi-square tests for categorical variables and t-tests for continuous variables. Fisher exact tests were used when comparisons involved a small number of participants (< 5). Similar bivariate analyses were used to determine the relationships between risk-reduction formats and the ability to provide correct comparative and quantitative data interpretations.
Forty-two percent of students were male, 76% were white, and the median age was 24 years. Twenty-seven percent reported that they had a pastime that required the use of risk concepts (Table 2).There were no differences in student characteristics among the four intervention groups.
Although 94% of students considered themselves good with numbers, only 77% answered all three numeracy questions correctly. Eighteen percent answered two questions correctly, and 5% answered one or no questions correctly. Students had more difficulty converting 1 in 1000 to 0.1% and 1% to 10/1000 (10% answered each of these questions incorrectly) than in stating that 500 heads would be expected in 1000 coin flips (4% answered this question incorrectly).
Numeracy and Interpreting Treatment Benefit
Although 90% of students correctly stated which drug worked better, only 61% correctly interpreted the quantitative data. Students' numeracy was associated with correctly interpreting data both comparatively and quantitatively (Figure 2). Ninety-four percent of students with three correct numeracy answers provided a correct comparative interpretation, compared with 91% with two correct answers and only 33% of students with one or no correct answers (P = 0.03). Similarly, 71% of students with three correct numeracy answers correctly interpreted the quantitative data, compared with 36% with two correct answers and 0% with one or no correct answers (P < 0.01).
Students who considered themselves to be good with numbers performed better than those who rated themselves as poor with numbers. Of the students who considered themselves to be good with numbers, 91% had correct comparative interpretations compared with 75% of the students who considered themselves to be poor with numbers (P > 0.2). These findings were similar for the quantitative interpretation (66% correct vs. 0% correct; P < 0.01).
Risk-Reduction Format and Interpreting Treatment Benefit
Figure 3 shows that there was no difference in correct comparative interpretations among the different risk-reduction formats. Correct quantitative interpretations varied, however, according to the risk-reduction format the students received. Students had significantly more difficulty with quantitative interpretation when NNT was used to express treatment benefit: 25% had a correct interpretation with NNT compared with 75% in all other risk-reduction presentations (P = 0.01).
Although the ability to work with numbers is important for physicians, almost a quarter of the first-year medical students in our study had trouble performing basic numerical operations. These students also had difficulty interpreting data about the benefit of treatment, particularly when it was presented as NNT.
We are aware of only one other study of medical student and physician numeracy in the literature. (5) That study, which was published as a letter, reported that 20% of a convenience sample of 45 physicians, nurses, doctorate faculty, and medical students gave incorrect answers to two or more of six basic numeracy questions. Although the results were not stratified by profession or level of training, they do support a conclusion that at least some health professionals have trouble performing basic numerical operations.
Two important limitations to our study need to be acknowledged. First, this is a relatively small study of first-year medical students. It is unclear whether problems with numeracy or difficulties interpreting data will persist through medical training. The clinical epidemiology curriculum, which is taught in many medical schools in the second year, provides an opportunity to refresh basic math skills while learning about the interpretation of medical data. Students completing this curriculum, and residents and faculty who are already in clinical practice, may perform very differently on our survey.
Second, our response rate was only 46%. Students who did not complete surveys may have performed differently from those who did. We suspect, however, that nonparticipants were less confident in their quantitative abilities; thus, we believe our findings may be an optimistic estimate of such abilities in first-year medical students.
Our findings also suggest that the NNT format may be especially difficult for medical students. This may be because the more effective treatment is represented by a smaller NNT, which is different from other risk-reduction formats in which the larger number represents the more effective treatment. In addition, using NNT to calculate event rates requires several potentially confusing steps. Educators may, therefore, need to spend extra time ensuring that students learn how to interpret NNT. Alternatively, educators may want to encourage the use of simpler risk-reduction formats.
Since limited ability to interpret medical data could contribute to such medical errors as misdiagnosis, inappropriate screening and treatment, or inaccurate communication with patients, it is important to recognize such difficulties early. Our findings suggest that medical educators should remain alert for students who have trouble with numbers. Students who believe that they are poor with numbers will probably benefit from additional instruction about how to interpret medical data. Others may also benefit if educators devote increased attention to developing students' numeracy skills as part of the medical curricula.
|Take Home Points
1. Kirsch IS, Jungeblut A, Jenkins L, Kolstad A. Adult literacy in America: a first look at the National Adult Literacy Survey. Washington: National Center for Education Statistics; 1993.
2. Schwartz LM, Woloshin S, Black WC, Welch HG. The role of numeracy in understanding the benefit of screening mammography. Ann Intern Med. 1997;127:966-72.
3. Woloshin S, Schwartz L, Byram S, Fischhoff B, Welch H. A new scale for assessing perceptions of chance: a validation study. Med Decis Making. 2000;20:298-307.
4. Woloshin S, Schwartz LM, Moncur M, Gabriel S, Toteson ANA. Assessing values for health: numeracy matters. Med Decis Making. 2001;382-90.
5. Estrada C, Barnes V, Collins C, Byrd J. Health Literacy and numeracy. [Letter] JAMA. 1999;282:527.
The authors thank Drs. Donald Pathman, Carmen Lewis, Greg Randolph, Katrina Donahue, Christopher Cox, and Pamela Christy Parham-Vetter for their critical review of several drafts of this manuscript.
Supported by a grant from Lineberger Comprehensive Cancer Center at the University of North Carolina at Chapel Hill.
Stacey L. Sheridan, MD MPH, Division of General Medicine and Clinical Epidemiology, 5039 Old Clinic, CB#7110, Chapel Hill, NC 27599-7110; telephone: 919-966-2276; fax: 919-966-2274; email: firstname.lastname@example.org.