Elizabeth Warren and the Terrible, Horrible, No Good, Very Bad, Utterly Misleading Bankruptcy Study
Megan McArdle Jun 4 2009, 4:21 PM ET
Tweet
More
Elizabeth Warren has another study out showing that medical expenses contribute to more than half of all bankruptcies--indeed, this time, it's 70%, up from the 50% she found in 2001.
Now, it is possible that this is true. The fact that it seems to disagree with every other study I've ever read that is not authored by Elizabeth Warren, and also, the self-reports of the people in her study (only about a third of whom attribute their bankruptcy to a health problem) could just be a fluke. It doesn't necessarily mean that it's wrong.
Yet upon closer examination, it turns out that it is not just wrong, but actively, aggressively wrong. Warren and her co-authors have obscured important and obvious facts that call the integrity of the work into serious question.
The text itself raises a huge red flags. It's hard to believe that more than half of people who have been pushed into bankruptcy by a medical issue don't understand this fact. Perhaps they are not the brightest bulbs on the Christmas tree, but could it really be true that most people catapaulted into a financial crisis by their medical bills don't even notice that health care expenses are their main problem?
My radar is further engaged by the fact that they're implying a really astonishing surge in medical-bill-driven bankruptcies, in a healthcare environment that just didn't change all that massively. Their study opens:
As recently as 1981, only 8% of families filing for bankruptcy did so in the aftermath of a serious medical problem. By contrast, our 2001 study in 5 states found that illness or medical bills contributed to about half of bankruptcies.
Since then, the number of un- and underinsured Americans have grown, health costs have increased, and Congress tightened the bankruptcy laws.
In those six years, the percentage of uninsured families ground upward, and health care cost continued to rise at about twice the rate of inflation. But a 2.5% real annual increase in the cost of a budget item that accounts for something like 5% of annual household expenditures shouldn't make the bankruptcy stats jump that much.
Perhaps there was a big increase in the volatility of those expenditures, with the average growing slowly, but a larger number of people being hit by truly massive bills? Perhaps, but I'm aware of no data that show it. Yes, people complain about deductible increases and more cost-shifting, but a $500 or $1000 increase in the annual deductible won't tip any family into bankruptcy, and the complaints about denied claims go back long before 2001. For this to be causing such a huge surge in bankruptcies, health care companies would have had to discover some extraordinarily clever new way to deny people health care benefits without being sued, or fired by the companies who buy their insurance. If they have, it hasn't been a prominent feature in the recent health-care debate. As far as I know, they're still using the same old strategy of outlasting and/or confusing their patients.
Yet Warren, et al. claim their current results both show a dramatic increase, and are in robust agreement with their earlier study. How could steadily, moderately rising medical bills, a roughly static business and legislative environment, and a small increase in the uninsured, possibly have driven up bankruptcies so massively?
Answer: they didn't. What Warren et. al. neglect to mention is that bankruptcies fell between 2001 and 2007. In fact, they were cut in half. Going by the numbers Warren et. al. provide, medical bankruptcies actually fell by almost 220,000 between 2001 and 2007, a fact that they not only fail to mention, but deliberately obscure.
Are Warren, et. al. unaware that bankruptcies fell by half? No bankruptcy analyst could possibly be unaware of this fact; it has been the most talked-about phenomenon in the bankruptcy area since the 2005 law was passed. Moreover, they're clearly familiar with the filings data, because they use it to make their point:
The number of filings spiked in mid-2005 in anticipation of the new law, then plummeted. Since hten, filings have increased each quarter. They are likely to exceed one million households in 2008, representing about 2.7 million people.
What's left out here? That in 2001, 1.45 million households filed for bankruptcy. In 2007, that number was 727,167. Had their paper done the basic arithmetic, readers would easily have seen that their own numbers imply a decrease in medical bankruptcies, from about 750,000 to slightly over 500,000. Yet their paper does not merely ignore this fact; it uses language that seems deliberately designed to conceal it. I invite any of my readers to scan the paper for any hint that medical bankruptcies had fallen significantly over 6 years.
This is elementary social science. A huge change in the composition of your sample needs to be noted. It certainly should not be artfully disguised. If the 2005 bankruptcy form made it more difficult to file bankruptcy, the people who still file bankruptcy will largely be those who are forced to it by events totally beyond their control. Medical bankruptcies seem to fill that bill.
Yet even so, their own work shows medical bankruptcies falling in the years between 2001 and 2007, which would seem to invalidate, not support, the claim that half of all bankruptcies in 2001 were driven by medical events beyond the household's control.
Elementary googling reveals that the two doctors who co-authored this study are prominent spokespeople for Physicians for a National Health Program, and thus have an obvious agenda, one that Elizabeth Warren has not been shy about sharing. The American Journal of Medicine, which published this study, seems to have flunked Peer Review 101--I sure hope they're more careful about controlling for background conditions when they're talking about cures for cancer Also wearing duncecaps are the journalists who are already uncritically parroting it.
There is, of course, a large amount of terrible advocacy masquerading of social science out there, and too many journals and journalists abet it. But this is particularly troubling because Elizabeth Warren is now in charge of overseeing the TARP program for Congress. What other inconvenient facts is she shielding us from?
theatlantic.com
...This paper was deeply, deeply flawed, and all in ways that suggested--as this discussion does--that rising medical bills were causing an increase in bankruptcies. I'll try to hit the highlights:1. I'm pretty sure that the metric they used this time around--10% of income or $5,000 would still make Patty Barreiro's bankruptcy a "medical bankruptcy", something which has never been asserted by anyone. The proximate cause of that bankruptcy was (according to Andrews) a lawsuit and an income crunch; the medical bills were simply added in.
2. The response rate on their survey was only 20%. Given the deep shame surrounding bankruptcy, you have to worry that they got an unrepresentative sample. And how is that sample most likely to be unrepresentative? Well, one pretty likely way is that people who went bankrupt through no fault of their own--folks who got whacked by large and unpayable medical bills or a business closure--were more likely to respond than the people with drug or alcohol problems, profound depression that left them unable to work, compulsive gambling issues, and so forth.
3. The authors have an odd tendency to ignore what the respondents themselves say. 32% of those surveyed about their 2007 bankruptcies--not 62%--reported that "medical problem of self or spouse was reason for bankruptcy." If anything, I would expect this to be overreported, since this is one of the few reasons for bankruptcy that does not trigger shame in our society; there should be a tendency for people to overemphasize the role that illness played in bankruptcies, which are often multifactorial and involve many different kinds of bills.
Warren, et al argue that they need to correct for the fact that someone may mortgage a home to pay for an illness, and then report the reason for their bankruptcy as "unable to pay for mortgage". Fair enough, but if you are going to try to correct for that sort of thing, you also need to try to correct for bias in the other direction. They only make adjustments that ratchet the results upwards.
More troubling, when you look at the percentage of people who self report that illness played a role in their bankruptcy, it hasn't changed that much. In the 2001 study, "illness or injury" is cited causally in just over 28% of cases. In the 2007 study, it's 32%. That's about a 10% increase, far less dramatic than they claim, and given how small their sample is, not particularly compelling. This should have induced far more caution in describing the changes than they claim.
4. Their methodology is quite explicitly designed to capture every case where medical bills, or medical loss of income, coexist with some other causal factor--but the medical issues are then always designated as causal in their discussion. As the case of Patty Barreiro indicates, this is simply not correct. Allow me to channel one of last year's commenters:
First, she redefines medical-related bankruptcy as having debts of $5000 or 10% of family income. She packages this as conservative, but for dual-unemployed families (income zero), ANY medical debt is therefore medical-related. She has a series of OR conditions which are likely to inflate cases where a person has been sick but where it ISN'T related to their bankruptcy.
Then she runs logistic regression to predict medical bankruptcy (at least as she defines it). It's not totally clear to me what variables she used or the order they were entered-- forward stepwise basically leaves it to the computer. All those other predictors that were skipped need to be declared. There are tricks with collinearity you can pull with stepwise (or, really any regression) to inflate or deflate coefficients. I also don't see a test for homoscedasticity.
Most importantly, though, we don't see the questionnaire she used. I don't see any checks for common methods bias, and not knowing her operationalizations, we have to basically take them on faith. She also doesn't establish comparability with her 2001 study, which is the whole point of her paper.
With flaws like this, I'm surprised that the study made it through peer review. I don't see a fatal flaw, but the reviewers would be sure to demand these kinds of methodological checks, and at six pages, there's plenty of room for that last half-page.
She seems to define medical-related as "bankruptcies which occur due to medical conditions, have an impact on medical conditions, or occur adjacent to medical conditions". If I calculate "medical-related sunspots", would I get a similar high percentage of the population? This limits its utility for anything other than a stat that can be pasted out of context into a flyer or campaign mailing.
. . .
I should point out that by collecting three very different kinds of "medical-related" bankruptcy into one predictor, she makes it totally meaningless. People teetering on the edge of bankruptcy have ALL sources of debt very high. Delinquent housing debts, credit card debts, mall boutique debts, and utility bills. That's why they're filing for bankruptcy.
You're supposed to use controls and good operationalizations to eliminate those problems. That's why we use regression in the first place. But she doesn't use it that way, in fact she goes out of her way to avoid using it; and where she does use it, she employs stepwise regression, which explicitly doesn't help with this problem. We also don't know her questionnaire because she didn't include it, even in an appendix.
So the real litmus test for this is: if I go back to her sample and use her methodology to find the proportion of credit card debt-related bankruptcy, I would probably ALSO get 80% or 90%. Then, I could go back again and show that automobile-related bankruptcy is another 40%-60% of bankruptcies.
When you strip out the statistical obfuscation, you end up with a couple simple correlations between year and another variable that can best be described as "we don't know what this is".
5. As I discussed at the time, early 2007 is a terrible, horrible, no good, very bad time to do any sort of study on bankruptcy. In 2005, Congress passed a bankruptcy reform bill that (among other things) tightened up the procedures for filing Chapter 7, the kind of bankruptcy where you just liquidate all your debts and don't have to go on a payment plan. This tightening was real, but rather exaggerated in the popular press (the overwhelming majority of people who need to file bankruptcy will have no problem qualifying for a Chapter 7). Perhaps for that reason, even bankruptcy experts were stunned by the number of people who rushed to file before the six month grace period in the law was up. They were also stunned by the magnitude of the decline, and how long bankruptcies took to start trending back towards previous levels. This graph from Credit Slips shows what it looked like:
 The study by Warren et. al. was done in early 2007, just as the rebound was gaining steam. There's quite a lot of reason to think that the sample was badly skewed. Many bankruptcies are long slow train-wrecks, the problems accumulating over a number of years as spending persistently outpaces income. A pretty convincing paper argues that the single best predictor of bankruptcy is simply how much debt you've accumulated--not income, job loss, divorce, or what have you. People who declare bankruptcy tend to have nicer stuff than others at the same income level.
But that paper also shows that medical events have independent predictive power. And medical events are much less of a foreseeable problem. If you're a plumber who has a stroke, you may well end up in bankruptcy simply because you lose income while you can't work (the medical bills may or may not play a large causal role).
So one worries that the people who rushed to declare bankruptcy before the 2005 change were the people with accumulating problems. One year out, it's reasonable to be concerned that the sample will consist mostly of people who, for one reason or another, couldn't see it coming--business failures and people with health problems being the obvious two candidates.
Now you can argue that this might not have been a problem. And indeed, maybe it wasn't. For that matter, maybe it biased the sample the other way--towards undercounting the increase since 2001. But the question remains: why do the study then? Why not wait a little while? Even another year would have substantially reduced this issue, as the people with accumulating financial problems re-entered the sample.
The authors deal with this problem in a thoroughly unconvincing way:
BAPCPA's effects appear nonselective. Current filers differ from past ones mainly in having struggled longer with their debts7. New restrictions fall equally on medical and nonmedical bankruptcies, with no preferences for medical debts or sick debtors. It is implausible to ascribe the growing predominance of medical causes of bankruptcy to BAPCPA.
Conversely, there is ample evidence that the financial burden of illness is increasing.
The paper they cite doesn't actually say what they imply; it addresses unsecured debt and income, not medical debt specifically. All bankruptcy is highly correlated with the level of unsecured debt, because that's the kind of debt you can most easily discharge in bankruptcy. Its major finding is that consumers who file bankruptcy have a higher level of absolute debt than in the past--something they cite as "consistent with the view" of a lengthier struggle (as is a modest rise in the percentage of people citing that they struggled with bills for at least two years before filing). Moreover, they specifically argue that their data challenges the thesis of the 2001 paper:
Yet the Sullivan-Warren-Westbrook model also cannot explain our findings on its own. It does not predict the sharp decline in bankruptcy filings we see in our data. Because there have been no dramatic improvements in health care delivery, employment prospects or family stability, their model would predict constant, not diminished, rates of filing.
6. Bringing me to my next point: the paper thoroughly obscures the point that by their own calculations, the number of medical bankruptcies fell quite dramatically between 2001 and 2007. This is, to put it mildly, sort of a problem for the thesis that exploding medical bills are shoving people into bankruptcy. They avoid dealing with this problem by citing only proportions, never absolute numbers--even thought they use absolute numbers freely whenever they are eye-popping.
Of course BAPCA put downward pressure on all bankruptcies, but medical bankruptcies should be among the least amenable to prior planning. You can try to sell the house, decide to struggle on in a bad marriage, or go to Gambler's Anonymous--but you can't decide not to have a heart attack this year. If medical bills really are soaring as dramatically as the authors imply, we should have seen an absolute rise between 2001-2007.
7. The authors dramatically underweight the role of income loss resulting from illness in their discussion. They also sort of failed to mention it when talking to the media. When I wrote my rather fiery critique last year, a lot of people said I should really criticize the media for screwing up the story. But every interview I read of the authors, particularly Woolhandler and Himmelstein of Physicians for National Health Care, basically put all the emphasis on medical bills. If the press got it wrong, it's because the authors went out of their way to put an extreme spin on their findings.
8. There are a bunch of other fiddling red flags, like the fact that there's no difference in the percentage of "medical" and "non-medical" bankruptcies where the family lacks insurance at the time of bankruptcy. They see this as evidence of underinsurance. I see this as frankly unbelievable, and at the very least, in need of much better explanation than they offer. There's a slight difference in the rate of lapsed coverage at some point during the prior two years, but it's still so small as to suggest that there are a lot of "medical" bankruptcies in their sample with some other real cause.
Does this persistent tendency to choose odd metrics that inflate the case for some left wing cause matter? If Warren worked at a think tank, you'd say, "Ah, well, that's the genre." On the other hand, you'd also tend to regard her stuff with a rather beady eye. It's unlikely to have been splashed across the headline of every newspaper in the United States. Her work gets so much attention because it comes from a Harvard professor. And this isn't Harvard caliber material--not even Harvard undergraduate.
So I think it matters on two levels. One, it matters how we evaluate the work--and I've been disappointed at how uncritically some people I really respect have been willing to accept the 2001 and 2007 findings. Not because I don't think that there are medical bankruptcies in the United States; I do! I think medical bills are certainly the primary cause of some of those filings, though I don't know how many, and I've also been writing about bankruptcy long enough to know that assigning any one cause to the ultimate financial meltdown is, in many cases, impossible. (If you have nice consumer goods and no health insurance, does a car accident count as a "medical bankruptcy" or a budgeting deficiency? If you lived right up to the edge of your income, was a job loss, or your spending pattern, to blame? How you answer these questions depends on a large number of prior value judgments that are hotly contested in our society.)
It matters that we get this stuff right. I am among the majority who would like to see bankruptcies reduced in this country, and we're not going to be very effective at that if we run around thinking we can cure 2/3 of them by putting a national health care system in place, when in reality a third or less have any strong causal relationship with medical bills. Obviously, this was also held out as an argument for PPACA, making an implicit promise to the American people which I believe to be false.
But it also matters because a large part of Warren's prominence comes from the fact that she's an academic. If she came from . . . well, the sort of think tank that publishes this sort of advocacy science . . . she would have considerably less glamor, and power.
And perhaps it mattes most of all because this woman is now under consideration to head a powerful new agency. If this is how she evaluates data, then isn't that going to hamper her in making good policy? If we're going to have a consumer financial protection agency, I want one that has a keen eye to the empirical evidence on consumer welfare--not one that makes progressives most happy by reinforcing their prior beliefs.
* I've actually seen reports of a few cases where people declare bankruptcy over a couple thousand worth of debt, and I don't understand why the lawyers who filed those cases aren't in jail. You don't file bankruptcy over a debt that can be cleared by working night shifts at Pizza Hut for a month--if you're too infirm for a part time job, you probably don't have a salary they can garnish or assets too large to hide under your mattress.http://www.theatlantic.com/business/archive/2010/07/considering-elizabeth-warren-the-scholar/60211/ |
