ACP American College of Physicians - Internal Medicine - Doctors for Adults

Effective Clinical Practice

EDITORIAL

Are We Overvaluing Performance Measures?

Effective Clinical Practice, September/October 2000.

Vijan Sandeep, MD

For author affiliations, current addresses, and contributions, see end of text.

The focus on demonstrating high-quality medical care is intense and growing. The current trend is to document quality of care through measurement of compliance with "evidence-based" performance measures. Typically, these measures are the proportion of patients with a particular disease who meet a certain criterion: for example, the proportion of patients with coronary artery disease who have low-density lipoprotein (LDL) cholesterol levels below 100 mg/dL or the proportion of patients with diabetes who have an annual eye examination. The underlying assumption is that for a particular condition, treating all patients the same is equivalent to high quality of care. This simplifying assumption is necessary because of the difficulty in defining and measuring performance in a complex (and often fragmented) system.

Despite the use of simple measures, many health care systems are far from achieving optimal performance. Quality improvement efforts based on guideline production and dissemination have been only marginally successful. (1) Because health care systems are investing substantial effort and money to do better on performance measures, (2) this special issue of ecp is devoted to the topic. In it, you'll find articles on improving hospital performance in community-acquired pneumonia (3) and screening performance in breast cancer. (4) Here, I focus on the two articles that address performance in two chronic diseases: diabetes (5) and coronary artery disease. (6)

Nyman and colleagues (5) implemented a diabetes guideline by using an educational intervention and an electronic ordering support system. The study had no control arm, but the investigators demonstrated a 0.7% absolute decrease in hemoglobin A1c (Hb A1c) concentration and improvements in rates of eye, kidney, and foot screening. Siskind and colleagues (6) used an automatic prescription method to help manage lipid levels in patients with coronary artery disease. This intervention essentially took cholesterol management out of the hands of the primary provider and transferred it to a dedicated lipid management team. The study showed a significant improvement in the proportion of patients who achieved target LDL levels (72% in the intervention arm and 43% in the control arm). Although these two studies are innovative examples of how more intensive interventions can improve compliance with performance measures, the measures themselves have weaknesses that are often overlooked as we rush forward in our attempt to improve quality.

Intermediate- and Long-Term Outcomes Are Weakly Related

Most of the current performance measures represent intermediate outcomes (also known as process measures). While improvement in these measures is expected to lead to reductions in long-term adverse events that materially affect patients, the magnitude of improvement is not nearly as dramatic as we would hope. For example, the intervention studied by Nyman and colleagues (5) led to a 0.7% absolute reduction in Hb A1c concentration. In the United Kingdom Prospective Diabetes Study, (7) a 0.9% reduction in Hb A1c concentration was associated with an absolute reduction in photocoagulation risk of 3.1 per 1000 patient-years. Unfortunately, this improvement in glycemic control and retinopathy risk did not translate into significant reductions in risk for visual loss, even though the study followed nearly 4000 patients for an average of 10 years. Thus, the quality improvement intervention reported by Nyman and colleagues improved intermediate outcomes, but the benefit to patients may not become clear for many years, and its magnitude is likely to be relatively small. Although any amount of patient benefit is important, the expected magnitude of benefit and the investment required to achieve it must be carefully examined before interventions to achieve performance goals are adopted on a broad scale.

Dichotomous Measures Oversimplify the Relationship between Process and Outcome

Most performance measures are based on dichotomous outcomes. Success is therefore limited to reaching a specific goal, such as an LDL cholesterol level less than 100 mg/dL or an Hb A1c concentration less than 7%. However, setting these specific goals has unfortunate effects. Consider, for example, a patient with coronary artery disease. A reduction in LDL cholesterol level from 201 to 101 mg/dL should lead to substantial long-term benefit. Despite this potentially dramatic benefit, our simplistic performance standard terms this patient a "failure." In contrast, a patient whose LDL cholesterol level is reduced from 105 to 100 mg/dL, which should produce at best minimal benefit, is termed a "success."

This issue is highlighted in the study by Siskind and colleagues. (6) Their intervention was highly successful in terms of meeting performance standards, but the difference in mean LDL cholesterol level was less pronounced (111 mg/dL in the control group vs. 95 mg/dL in the automatic prescription group). The necessity of getting LDL cholesterol levels in all patients to the absolute target of 100 mg/dL, particularly when the control group had levels only slightly above this target value, is questionable. In fact, the major randomized trial of lipid lowering in coronary heart disease did not reach this goal. (8) Put simply, the incremental benefit of a reduction in LDL cholesterol level from 111 to 100 mg/dL is likely to be fairly small and must be weighed carefully against the incremental costs of any intervention.

Universality Can Lead to Inefficiency

Not all patients are the same. Treating them as such not only minimizes autonomy but is also a recipe for inefficiency. Analyses of randomized trials suggest that variability in individual risk, and therefore of individual benefit from interventions, is profound. Attempting to reach the same treatment goals for all patients is not only difficult and costly but can also, in the extreme, be harmful to patients who have a low baseline risk for adverse outcomes. (9, 10) A more efficient (and in some instances safer) alternative is to focus efforts on patients who are at high risk for adverse outcomes. The efficiency of targeting has been demonstrated for such common conditions as hyperlipidemia, hypertension, and glycemic control in diabetes. (11-14)

The reasons behind the efficiency of targeting are shown in Figure 1. The risk for the adverse outcome without treatment is denoted by the upper line. The risk for the outcome after a constant reduction in the risk factor (e.g., a 20-point reduction in LDL cholesterol level or a 2-point reduction in Hb A1c concentration) is denoted by the lower line. Thus, the shaded area is the benefit of treatment. The lower the baseline risk, the less benefit achieved with treatment--even though the absolute level of reduction in the risk factor is equivalent. In cases where treatments have substantial adverse effects, the lines often cross at low levels of risk, so that the low-risk patients are actually harmed by interventions. These relationships dictate that high-risk patients derive most or all of the benefit of any intervention, and targeting these patients for aggressive management will provide much more benefit than insisting on aggressive management for low-risk patients. Health systems would be much more efficient and still achieve most of their potential benefit if quality improvement systems that target these highest-risk patients were developed.

A More Balanced View of Performance Measures

Despite these limitations, we should not discount the potential benefits of setting and achieving performance standards. The quality of medical care can be improved in many areas, and performance measures are a potentially important method of achieving selected goals. However, we must set goals based on a better understanding of the actual benefits and costs of achieving them. Our optimism about these measures must be tempered by healthy skepticism and a realization that performance measures come with financial and personal costs to patients, providers, and health care systems.

Finally, we should also ensure that the expectations for these efforts are reasonable. Recently, manufacturing companies, such as General Electric, Motorola, and Daimler-Chrysler, have entered into partnerships with health care organizations to try to improve the quality (and control the costs) of health care. (15, 16) Such terms as "six-sigma," which represents an "allowable" number of errors in manufacturing processes, are now entering the medical lexicon. (16) However, while quality standards and performance measures have led to substantial improvements in the quality of manufactured goods, it is less clear that the same concepts can be applied to medicine. The complex interaction between patient preferences, the provider-patient relationship, and biological risk is by necessity minimized or even ignored when paternalistic universal performance standards are put in place. And whereas the performance of an automobile may be largely defined by the design and manufacturing process, we must acknowledge that the health of human beings is only minimally defined by their medical care.

References

1. Schuster MA, McGlynn EA, Brook RH. How good is the quality of health care in the United States? Milbank Q. 1998;
76:517-63.

2. Eddy DM. Performance measurement: problems and solutions. Health Aff (Millwood). 1998;17:7-25.

3. Estrada CA, Unterborn JN, Price J, Thompson D, Gibson L. Judging the effectiveness of clinical pathways for pneumonia: the role of risk adjustment. Eff Clin Pract. 2000;4:221-8.

4. Fishman P, Taplin S, Meyer D, Barlow W. Cost-effectiveness of strategies to enhance mammography use. Eff Clin Pract. 2000;4:213-20.

5. Nyman MA, Murphy ME, Schryver PG, Naessens JM, Smith SA. Improving performance in diabetes care: a multicomponent intervention. Eff Clin Pract. 2000;4:205-12.

6. Siskind A, Johnson M, Qureshi A, Solow B, Chesebro D, Oldham N. The impact of automatic prescription on reducing low-density lipoprotein cholesterol levels. Eff Clin Pract. 2000;
4:240-6.

7. Intensive blood-glucose control with sulphonylureas or insulin compared with conventional treatment and risk of complications in patients with type 2 diabetes (UKPDS 33). UK Prospective Diabetes Study (UKPDS) Group. Lancet. 1998;352:837-53.

8. Randomised trial of cholesterol lowering in 4444 patients with coronary heart disease: the Scandinavian Simvastatin Survival Study (4S). Lancet. 1994;344:1383-9.

9. Califf RM, Woodlief L, Harrel FE Jr, et al. Selection of thrombolytic therapy for individual patients: development of a clinical model. Am Heart J. 1997;133:630-9.

10. Rothwell P. Can overall results of clinical trials be applied to all patients? Lancet. 1995;345:1616-9.

11. Garber AM. Cholesterol screening should be targeted. Am J Med. 1997;102:26-30.

12. Prosser LA, Stinnett AA, Goldman PA, et al. Cost-effectiveness of cholesterol-lowering therapies according to selected patient characteristics. Ann Intern Med. 2000;132:769-79.

13. Browner WS, Hulley SB. Effect of risk status on treatment criteria. Implications of hypertension trials. Hypertension. 1989;
13:I 51-6.

14. Vijan S, Hofer TP, Hayward RA. Estimated benefits of glycemic control in microvascular complications in type 2
diabetes. Ann Intern Med. 1997;127:788-95.

15. Daimler-Chrysler takes steps to help health-care providers improve service. Wall Street Journal. December 14, 1999.

16. Chassin MR. Is health care ready for Six Sigma quality? Milbank Q. 1998;76:565-91.

Correspondence

Sandeep Vijan, MD, MS, Veterans Affairs Health Service Research and Development, P.O. Box 130170, Ann Arbor, MI 48113-0170; telephone: 734-930-5100; fax: 734-930-5159; e-mail: svijan@umich.edu.