Nate Silver Did Not Get the Election Wrong

Nate Silver Did Not Get the Election Wrong

71% is not 100%. Statistics is not magic. Stop trying to burn Nate Silver at the stake.

In the wake of the 2016 U.S. Presidential election, journalists left, right, and center lamented what they described as the failed predictions of election forecasters like FiveThirtyEight and the New York Times’s The Upshot. Nate Silver, who founded FiveThirtyEight following his perfect forecast of the 2012 election and near-perfect forecast of the 2008 election, bore the brunt of these criticisms. Newsweek chastised him for defending his “erroneous prediction” of a Clinton win. Michael Braun, in a Medium article, claimed that Nate got the election “spectacularly wrong” and inferred a bleak future for FiveThirtyEight. Even The Economist got in on the fray, albeit in their notoriously even-handed manner.

The biggest problem with these analyses is their most basic premise: that Nate Silver made a prediction. Any forecaster worth their salt is careful to refer to their output as forecasts, and Nate is no exception. Forecasts are inherently uncertain, and that uncertainty manifests itself in the probability distribution reported by each of these data journalists, including FiveThirtyEight. If you scroll down on the website’s final 2016 forecast, you find the following chart:

Win Distribution

When laypeople—and even statisticians!—think about one-off events like elections, we are tempted to view them as deterministic: they happen once, and the result seems predictable in hindsight. The idea behind probability distributions like the two above is that, if the event (i.e., the election) were to happen an infinite number of times, then each outcome (in this case, the electoral vote count) would occur a certain percentage of the time, with each outcome’s percentage represented by the height of each bar on the distribution.

In real life, we only witness a single draw from this probability distribution. Critics who pan forecasters’ results as “mispredictions” implicitly operate under the assumption that the true distribution of outcomes is a single bar with a height of 100%. That isn’t an assumption that any forecaster makes or would ever choose to make. It’s an assumption that lies outside of the epistemology of statistics.

For events that are truly random, such as the flip of the coin, the distribution of outcomes is unchanging and can be exactly determined. In election modeling, the distribution is instead an inference of the probability of certain outcomes based on a combination of polling data and past election results. Under perfect methodological practices, the inferred distribution converges to the “true” distribution as the sample size increases. The exact and estimated distributions operate in essentially the same way. If I told you that you have a 70% chance of rolling less than an eight on a ten-sided die and you rolled a nine, you wouldn’t pen an article saying that I got the die roll spectacularly wrong. Nate silver told us that there was a 71% chance of a Clinton Presidency, and we rolled a nine.

FiveThirtyEight Was Relatively Bearish on a Clinton Win

There is a case to be made that some election forecasters, such as The Upshot, called the election incorrectly, even within the bounds of statistical uncertainty. Many forecasters other than FiveThirtyEight had the probability of a Clinton win above 90%, where Nate Silver put Barack Obama’s odds at re-election in 2012, an outcome on which he was willing to bet $1,000.

Nate Silver and FiveThirtyEight were not confident about Hillary Clinton’s odds, even when their model had her favored to win all swing states plus Arizona. Throughout the race, Nate insisted that, though the probability of a Clinton win roughly mirrored Obama’s 2012 odds, the calculations behind the probability told two different stories: Obama held a small but stable lead over Romney, while Clinton held a large but volatile lead over Trump. In a race where a zero-percent-plus-one vote margin is all that you need to win every one of a state’s electoral votes, stability is more assuring than magnitude.

Nate was also one of the few members of the media that specifically and confidently singled out the Midwest as a potential opportunity for Donald Trump, and criticized legacy media’s dismissal of Trump’s campaigns in Michigan and Wisconsin. As early as 2015, Nate penned an article titled, “There Is No Blue Wall,” and on November 1st, he outlined all of Trump’s paths to victory under a 2 point or less popular vote margin in an article titled “Yes, Donald Trump Has a Path to Victory.”

On the eve of the 2016 election, FiveThirtyEight released a final, pre-election podcast that discussed their model’s final results. When the host asked Nate about Pennsylvania and the model’s 75% odds of a Clinton win, he answered with what seems now to be an almost perfect call:

“My theory is that it’s more plausible for Trump to win Pennsylvania and some other rust belt states like Michigan than people infer, and [they make] these kind of sophomoric arguments like, ‘Oh, well, she’s been ahead in all the polls.’ But that doesn’t really matter much if you happen to be one point ahead or two points ahead and it’s close. And you understand the scenario by which she loses a state like Michigan or Pennsylvania is, you know, those white working class voters, those union workers go further to Trump, and she doesn’t turn out her African-American base. If both of those things happen — so you’re flipping a coin twice, 25 percent chance — then I think she could be in jeopardy in one of those states. People are a little smug about saying, ‘how dare Trump campaign in Michigan, that shows how desperate he is.’”

Given the final outcome of the election, the model’s call, and Nate’s explanation of it, was quite accurate. There was a swing toward Trump in the last week of the election, following the Comey letter, and nearly all voting in Pennsylvania occurs on the day of the election. Nate insisted that a 75% probability isn’t enough to withstand two adverse events, and it wasn’t. Would you be surprised if you flipped a coin twice and got heads each time?

Blame the Pollsters—But Not Too Much

I am clearly partial to FiveThirtyEight’s coverage. However, there are reasonable criticisms to be levied. Nevada, Colorado, and New Hampshire were all forecasted to swing for Trump before Michigan and Wisconsin, yet they went for Clinton (in Colorado, by five points!). Meanwhile, Wisconsin was forecasted to swing for Clinton by five points.

This is less of an indictment of FiveThirtyEight’s methodology than of the distribution and methodology of polling companies. Contrary to the belief held by certain unnamed individuals, FiveThirtyEight does not conduct any of their own polls; they simply aggregate the polls taken by other organizations. Forecasters use models that are constrained by the frequency and quality of polls that independent companies choose to conduct.

The quality of polls overall has declined significantly over the past twenty years. According to a study by Pew, response rates in telephone surveys declined from 36% in 1997 to just 9% in 2016. Pollsters remedy the limited and less random samples they now have by demographically weighting their respondents according to a model of the electorate. However, demographics are merely estimated in non-census years (which is most of them), and the demographic composition of voter turnout is highly fluid. Many pollsters get their models wrong or systemically over-estimate the odds of one party. FiveThirtyEight remedies the variability in sample size and poll quality by weighting historically reliable pollsters more heavily than unreliable ones and adjusting outliers to concurrent trends, but reliable pollsters are scarce and subject to the same constraints as unreliable ones.

Polling issues faced by forecasters are quantitative as well as qualitative: Even a poll conducted with a 100% response rate using a perfectly random sample will have larger errors if its sample is small. A sample of 300 respondents has a wider error than one with 600 or 1,200 respondents.  Poll aggregation essentially pools the samples of each poll into one large sample, so states with fewer polls conducted would have weaker forecasts. Colorado and Ohio were polled more than Michigan and Wisconsin; unsurprisingly, the latter two had larger errors.

Flawed but Critical

This may seem like a frivolous hill to fight and die on. I am not a data journalist myself, and it should not matter to me whether other people take the same things seriously that I do. But the anti-intellectual backlash to polling has costs. Since the election, Trump administration officials and surrogates have repeatedly dismissed negative approval ratings, citing election poll inaccuracy (even though national polls were fairly accurate). During the U.K.’s 2017 election, media outlets cited polling errors in the U.S. and U.K. as a justification for cutting their polling budgets and “talking directly to voters.”

It would be a step in the direction of good journalism to move resources away from horse-race polling and into high-quality field efforts. It is not good journalism to take resources away from horse-race polling and devote them to horse-race election coverage. A trend in national election coverage is to interview prototypical voters in certain areas that confirm conventional narratives of that election. Take for example, the New York Times’s articles from March 12th and June 2nd, in which they interview the same Ohio small business owner about Donald Trump and conclude that the President has not lost his core constituency.

This sort of journalism is not an alternative to polling. It is a crumby version of polling, which uses a miniscule sample and an array of narrowed-down survey questions. It does not serve to inform readers about the diversity of views in a broad set of regions, but perpetuates pre-existing narratives. On the subject of coal miners, reporters travel to towns in Kentucky and West Virginia, and they ask leading questions about jobs, often failing to provide the big picture to their readers.[1]

Polling is imperfect and necessarily fails to capture the nuance that longform reporting provides, but it informs the public about the prevalence of certain views. Without polling, we would not know that a majority of Trump voters cited voting against Clinton as their primary motivator, or that anti-immigrant beliefs are a global phenomenon. We would not know that the Congressional Republicans’ current health care proposal is unpopular with almost every demographic group.

This polling data provides journalists and policy-makers with an idea of where to direct their attention and resources. With a political and media machine that is focused in an isolated region that contains a small proportion of the population, far from the two most populous states in the country, that polling data is necessary. It is not gospel, and it should not stand alone, but without it, would we really gain more than we lose?

The efficacy of a forecaster’s model does not depend on her binary predictive record. A poll aggregator who correctly forecasts the winner with 50.1% or greater chance of winning every single time has a worse model than somebody who misses one or two but predicts strong winners with greater precision, as Nate did in 2008 and 2012. That would be true even if polling methodology were perfect. Data journalism is in its infancy and has a bright future. The state of conventional journalism, on the other hand, is up for debate.

[1] I edited out a paragraph about coal economics and politics in the United States. If you are interested in this subject, please contact me. I am considering it for a future post.

The GOP’s Obamacare Imbroglio

The GOP’s Obamacare Imbroglio

Imbroglio (noun): 2.c:  a violently confused or bitterly complicated altercation

After seven years of fierce opposition to the Patient Protection and Affordable Care Act, or “Obamacare,” the national Republican Party is presented with the opportunity to do with it as they please, having been granted single-party rule in the White House and both chambers of Congress. Senate Majority Leader, Mitch McConnell, and House Speaker, Paul Ryan, have vowed that Obamacare repeal is the first item on their agenda, and both the House and Senate held true to this promise by passing the first step in repeal.

For all of the coverage that this issue receives, few Americans understand the ACA, and fewer are likely to understand the complexities of the repeal process. These complexities are politically consequential for Congressional Republicans, and, depending on the actions they take in response, could result in an economic disequilibrium with negative outcomes for many low-income and middle-class Americans.

The next two sections provide context for the ACA’s passage and explain what the ACA is. If you are familiar with these, feel free to skip ahead for a maybe-hot take on the GOP’s imbroglio.

Why Did Obamacare Happen in the First Place?

Legislation is rarely enacted without a crisis, and the ACA is no exception. Prior to the ACA’s passage and implementation, the uninsured rate was unsustainably high, and health insurance premiums consistently rose faster than the rate of inflation.

uninsured-rate

The uninsured rate was so high primarily because of the design of the United States health insurance industry. An overwhelming majority of insured Americans received their health insurance from their employers, thanks to incentives in the tax code for employers to offer health insurance in lieu of wage increases. Americans who could not receive employer-sponsored insurance largely went uncovered. Insurers required buyers to provide mountains of health records and personal information, often denying coverage to those with any sign of poor health. The video below of Vlogbrothers’s John Green buying health insurance on the pre-Obamacare individual market sheds light on the type of Byzantine bureaucratic navigation necessary to obtain coverage under the former status quo.

 

The pre-Obamacare health insurance status quo was simply untenable. Insurers could (and did) deny coverage to willing buyers with pre-existing conditions, and health care costs were rising inexorably. Many sick people were being left without care through no fault of their own. Aside from the (many) moral arguments for government intervention, the structure of the health insurance industry was economically destructive. In 2009, more than 60% of bankruptcies were caused by medical bills. The evidence of whether or not medical bankruptcies declined under the ACA is mixed, but other measures of medically-driven financial distress give reason for optimism. That was the world before Obamacare, and that is the world that would be returned to if Obamacare were repealed and not replaced.

What is Obamacare?

The complexities of the repeal process are impossible to describe without describing the fundamental composition of the Affordable Care Act (ACA). The ACA is, in the most general sense, a series of business and personal taxes and mandates, combined with a set of regulations on the U.S. health insurance industry and an expansion of the Medicaid program. This core provisions of the ACA are: banning private health insurers from discriminating on the basis of pre-existing conditions, the individual mandate, and health insurance subsidies. There are also many provisions of the act that are controversial and consequential but not structurally important. To name a few and provide references to information on them: the contraceptive mandate, employer mandate, Cadillac tax, and medical device tax.

The three core provisions of the ACA comprise a three-legged stool: the policy cannot stand without all three legs. The first leg, which galvanized the reforms, is the requirement that private insurers cover buyers with pre-existing conditions. This provision is self-justifying: if somebody does not receive insurance from her employer, was born with ulcerative colitis, and wants to purchase health insurance, then she should be able to. However, instituting this reform on its own would allow (and, to some degree, incentivize) people to abuse the health insurance system. If insurers cannot refuse coverage to you or charge you more for pre-existing conditions, then what is to stop you from staying uninsured until you get into an accident or get sick?

The result of such a standalone policy is hyperbolically named a health insurance death spiral: healthy folks who recognize that they don’t need to buy insurance choose to drop out of the market, and their absence makes the insurance pool costlier to the insurer. The insurer raises premiums to cover the higher-cost pool, which prompts more healthy folks to drop out. Follow this trend to its logical conclusion, and the result is an even higher uninsured rate and completely unaffordable health insurance premiums.

To buttress against the death spiral, the second leg of the stool, the individual mandate, requires that all Americans buy health insurance. The ACA achieves this in the form of a tax that must be paid only by those who are uninsured. However, many lower-income individuals and families simply cannot afford even the slimmest of health insurance plans. Ergo, the third leg of the stool: health insurance subsidies that phase out as income rises.

The ACA provides these subsidies via non-refundable tax credits. Unfortunately, families must make enough money for the tax credit to apply, leaving families with no taxable income without subsidies. The ACA insures them by expanding Medicaid to cover them. However, some states chose not to accept the Medicaid expansion, leaving such families uninsured.

The GOP’s Imbroglio

The Republican Party has put the Affordable Care Act at the center of their role as an opposition party ever since its passage. This tactic won them seats in the 2010 elections, and the 2012 Republican Presidential nominee pledged to repeal the law. Two of the three legs of the stool, the individual mandate and insurance subsidies, were nearly struck down by the Supreme Court. Six years after the law’s passage, with over 20 million Americans relying on it for coverage, the Republican President-Elect campaigned on repealing the ACA. Congressional Republicans would abandon their ideological raison d’être of the past six years if they reneged on the promise to repeal Obamacare, and they would suffer political consequences for tinkering at its margins or making a tepid effort to do so.

A core problem for the GOP lies in the attacks they have leveled on the ACA. Speaker Ryan’s recent comments at a news conference provide a convenient overview of GOP criticisms of the law: buyers have too few choices with too high of costs, premiums are rising too quickly, and the deductibles of available plans are too high. Ryan also abstractly references freedom, presumably referring to the individual mandate, which was historically supported by Republicans, but became unpopular during the ACA’s passage under the premise that it limits personal freedom.

GOP leaders bind themselves by simultaneously criticizing high-deductible plans and rising premiums. Rising health care costs and high-deductible plans are inextricably related, a fact that is not lost on Congressional Republicans. A primary factor in the U.S.’s abnormally high health care costs is that the insurance system creates what is referred to in economics as a principle-agent problem: Americans with health insurance typically choose not to “shop around” for health care services because their insurer pays their cost. Therefore, demand for health care services is higher and less elastic than it would be if they paid for the services themselves. Principle-agent markets lead to higher overall demand, resulting in higher prices. Health care buyers with high-deductible plans, however, will try to avoid paying their full deductible, making them more selective with their care. The empirical results of this are mixed, but by criticizing both health care inflation and the prevalence of high-deductible plans, Speaker Ryan and the Congressional GOP are betting on not being held to account for their rhetoric over the ACA’s outcomes.

In reality, Republican leaders dislike Obamacare because it expands the role of government, taxes investments and high-income earners, and was passed by a popular Democratic President that they sought to defeat. To be fair, they have reason to worry about the expansion of government spending under Obamacare: in recent history, health care costs consistently rose faster than inflation and continue to do so under the ACA. The cost of insurance subsidies and the Medicaid expansion are bound to increase over time, as well.

med-inf-cpi

These criticisms of the ACA are problematic for a now-governing party. The only known replacement plans that address the GOP’s concerns would expand the role of government more than the ACA does. Health insurance could be more affordable if the subsidy program were expanded, and a public option would preclude the incidence of monopoly markets. A single-payer system would lower costs, remove the need for premiums, and minimize deductibles. It is also unclear that the ACA had any impact on insurance premiums or their rate of their growth. The chart below, from a Forbes article released last fall, provides little evidence that the ACA had any significant effect on health insurance premiums. This is not surprising; the law aimed to provide universal coverage, not to control costs.

premium-increases-under-aca

Other than expanding the use of high-deductible plans, the only mechanism for reducing health care costs is consolidating the market power of U.S. insurers (including Medicare and Medicaid) and allowing them to bargain health care prices downward. This cannot be done without an unprecedented expansion of government influence in the health care industry, an aspect of the ACA that the GOP also criticizes. This is why health care costs are lower in countries with single-payer health insurance: the government, insuring every single citizen, uses its significant market power to bargain for lower health care prices (a coherent explanation of this process is available on this episode of Vox’s policy podcast).

What Does the GOP Do Now?

The Senate voted 51-48 on January 12th in favor of legislation that has been unhelpfully referred to as a “step” toward Obamacare repeal. The following night, the House followed suit in a 227-198 vote. They did so through the budget reconciliation process, which allows Congress to vote on legislative changes to spending, revenues, or the debt limit without facing the risk of a filibuster in the Senate (which would require a 60-vote threshold for passage). This process prevents Congress from actually repealing the ACA or halting many of its provisions, notably the requirement that insurers cover pre-existing conditions (the first leg of the stool!). The reconciliation bill defunds health insurance subsidies, repeals the individual mandate, ends the Medicaid expansion, and eliminates most of the law’s various taxes.

If President Trump signs the partial repeal into law after he takes office on January 20th, then only one leg of the ACA’s policy stool will remain. Some 22 million Americans will lose their health insurance, and America will stare into the face of the insurance death spiral, because the requirement that insurers cover pre-existing conditions will remain. In order to fully repeal the ACA and/or enact a replacement plan, Congress will need to pass a bill by conventional means, with 60 votes in the Senate (only 52 senators are Republican). Even if Congressional Republicans could agree on a policy that satisfies their purported priorities for health care reform, they would need to either abolish the filibuster (referred to as the nuclear option for a reason) or win eight Democrats over in addition to keeping the entire Republican Senate caucus in line.

Nothing written above is news to Republican leaders and is probably known by most, if not all, rank-and-file Republican members of Congress. Why, then, would Congressional Republicans pass a piece of legislation that they know will send the American health insurance market into a death spiral, which is sure to result in their electoral defeat?

Possibility #1: They believe Donald Trump will veto the budget reconciliation bill

On Wednesday, the day before the Senate passed a repeal via budget reconciliation, President-Elect Trump stated in a press conference that Obamacare would be repealed and replaced “essentially simultaneously.” The following day, Ipsos released a poll showing that only 14% of Americans think that Obamacare should be repealed without a replacement. It is possible that Congressional Republicans see a Trump veto as an opportunity to placate their base without throwing the health insurance market into a death spiral. Trump is the most credible figure with the GOP base to turn down the option of repeal via reconciliation, while Congressional Republicans are more likely to be seen as sell-outs or weak if they vote against any form of Obamacare repeal.

Possibility #2: They plan to make cosmetic changes to Obamacare and cut their losses

This possibility is an amendment to the first one. After facing a Presidential veto and knowing that they will never get 60 votes in the Senate, Republicans in Congress trim the ACA’s spending measures, repeal most of its tax provisions, and make other trivial changes to the legs of the ACA’s policy stool. They pass the measure via budget reconciliation, rebrand the reforms, and hold press conferences lauding themselves for “repealing and replacing Obamacare,” hoping to convince a large enough share of their base to avoid widespread losses in subsequent primary and general elections.

Possibility #3: They plan to use the budget reconciliation bill as a bargaining chip

The GOP Congress has put itself in a position where choosing not to repeal the ACA via budget reconciliation would result in certain electoral defeat, either in Republican primaries or the general election, or both. Repealing the ACA without a replacement plan, causing 22 million Americans to lose their health insurance, and throwing the insurance market into the death spiral is also likely to be unpopular. However, Congressional Republicans could minimize their electoral losses in the latter situation by blaming Democratic intransigence for the failure of replacement efforts.

By repealing all key provisions of the ACA other than the pre-existing condition coverage mandate, Congressional Republicans light the fuse of an economic (and therefore political) bomb which every member of Congress must attempt to defuse in order to face their constituents in the following election. Congressional Republicans then introduce a bill that fully repeals the ACA and replaces it with one of their various proposals. This puts Congressional Democrats in a catch-22: if they kill the replacement bill, then Republicans have a strong talking point against them in coming elections when the death spiral and skyrocketing uninsured rate become the central issue in the 2018 and 2020 elections. If they support the replacement bill, they alienate their base and dismantle a key piece of their popular former President’s legacy.

This maneuver is, of course, highly risky. However, the Congressional Republicans’ imbroglio requires them to take significant risks in order to retain power after the 2018 midterms and 2020 Presidential election. The political acuity of this possibility leads me to believe that it is the likeliest. The complexity of repeal via budget reconciliation and the risks associated with it lends the issue well to obfuscation, which is beneficial to parties looking to hold power in low-turnout midterm elections amid a media environment that allows party hacks to choose how the issue is presented to partisan consumers.

In my opinion, a partial repeal without replacement, while politically savvy, presents the most risk to America’s most vulnerable families. The ACA is far from perfect: many Americans are left uninsured, it leaves long-term cost growth untouched, and its bureaucratic design creates barriers for many of the people it was created to help. However, no alternative that is acceptable to Republican leaders or their base will solve any of these problems. The best plausible outcome, in my view, is that the GOP’s repeal efforts die a quiet death under Presidential veto, and this issue fades from their slate of policy priorities. Here’s to hoping.