Giving Doctors (Teachers) Grades
NY Times Opinion by Dr. Sandeep Jauhar 7/22/15 entitled “Giving Doctors Grades”
The original opinion has been copied and pasted below. My additions are found inside bolded, italisized parenthesis (like this). Enjoy!
ONE summer day 14 years ago, when I was a new cardiology fellow, my colleagues and I were discussing the case of an elderly man with worsening chest pains who had been transferred to our hospital to have coronary bypass surgery. We studied the information in his file: On an angiogram, his coronary arteries looked like sausage links, sectioned off by tight blockages. He had diabetes, high blood pressure and poor kidney function, and in the past he had suffered a heart attack and a stroke. Could the surgeons safely operate?
In most cases, surgeons have to actually see a patient to determine whether the benefits of surgery outweigh the risks. But in this case, a senior surgeon (administrator), on the basis of the file alone, said the patient (student) was too “high risk.” The reason he gave was that state agencies monitoring surgical (test score) outcomes would penalize him for a bad result. He was referring to surgical (educational) “report cards,” a quality-improvement program that began in New York State in the early 1990s and has since spread to many other states.
The purpose of these report cards was to improve cardiac surgery (teaching) by tracking surgical (test score) outcomes, sharing the results with hospitals (other schools) and the public, and when necessary, placing surgeons (teachers) or surgical programs (schools) on probation. The idea was that surgeons (teachers) who did not measure up to their colleagues would be forced to improve.
But the report cards backfired. They often penalized surgeons (teachers), like the senior surgeon (lead teacher) at my hospital, who were aggressive about treating very sick patients (hardest to reach students) and thus incurred higher mortality rates (lower test scores). When the statistics were publicized, some talented surgeons (teachers) with higher-than-expected mortality statistics (lower test scores) lost their operating (teaching) privileges, while others, whose risk aversion (test prep curriculum) had earned them lower-than-predicted rates (higher test scores), used the report cards to promote their services in advertisements.
This was an insult that the senior surgeon (lead teacher) at my hospital could no longer countenance. “The so-called best surgeons (teachers) are only doing the most straightforward (test prep) cases,” he said disdainfully.
Research since then has largely supported his claim. In 2003, a study published in the Journal of Political Economy compared coronary bypass surgeries in New York and Pennsylvania, states with mandatory surgical report cards, with the rest of the country.
It found a significant amount of cherry picking in the states with mandatory report cards: Coronary bypass operations were being performed on healthier patients, and the sickest patients were often being turned away, resulting in “dramatically worsened health outcomes.”
“Mandatory reporting mechanisms,” the authors concluded, “inevitably give providers the incentive to decline to treat more difficult and complicated patients.”
Surveys of cardiac surgeons in The New England Journal of Medicine and elsewhere have confirmed these findings. And studies from 2005 and 2013 have shown that report cards on interventional cardiologists who perform angioplasty procedures are having similar results.
Surgical report cards are a classic example of how a well-meaning program in medicine can have unintended consequences. Of course, formulas have been developed to try to adjust for the difficulty of surgical cases (student behavior) and level the playing field. For example, a patient undergoing coronary bypass surgery who has no other significant diseases has an average mortality risk of about 1 percent. If the patient also has severe kidney dysfunction and emphysema, the risk of death increases to 10 percent or more. However, many surgeons (teachers) believe that such formulas still underestimate surgical risk (teaching & learning difficulty) and do not properly account for intangible factors, such as patient frailty (childhood trauma or abuse).
The problem is compounded by the small number of operations — no more than 100 per year — that a typical cardiac surgeon performs. Basic statistics tell us that the “true” mortality rate of a surgeon is not what you measure after a small number of operations. The smaller the sample, the greater the deviation from the true average.
Report cards were supposed to protect patients by forcing surgeons to improve the quality of cardiac surgery. In many ways they have failed on this count. Ironically, there is little evidence that the public — as opposed to state agencies and hospitals — pays much attention to surgical report cards anyway. A recent survey found that only 6 percent of patients used such information about hospitals or physicians in making medical decisions.
It would appear that doctors, not patients, are the ones focused on doctors’ grades — and their focus is distorted and blurry at best.