Why Government Fails So Often: And How It Can Do Better (10 page)

Another reason why retrospective assessment is so important is that the alternative is to rely on officials’ post-hoc accounts of why programs did not meet expectations—for example, that the program did not have enough money to do the job right or was administered by unsympathetic officials. Although such claims may be true in a particular case, their implausibility as a general matter becomes evident once one understands the deep structural reasons for failure. These reasons, detailed in subsequent chapters, often combine to overdetermine program failure.

Tetlock’s research has established empirically that experts are poor at predicting future events. Indeed, experts are actually worse at prediction than nonexperts, including nonexperts who choose more or less at random. In his experiments, both experts and nonexperts were highly intelligent, well-educated individuals who were asked to make more than 27,000 predictions over a large number of topics, regions, and time periods. The main difference was that experts were specialists in the areas in which predictions were sought while nonexperts were not.
³²
These findings are consistent with what many
other observers have concluded on the basis of less systematic experience and observation. Examples include elite law professors predicting Supreme Court decisions,
³³
psychologists diagnosing mental illness and dangerousness,
³⁴
and movie executives predicting film hits.
³⁵

Tetlock ascribes the large observed difference in predictive accuracy to the different cognitive styles that, drawing on Isaiah Berlin’s famous distinction, he ascribes to expert “hedgehogs” (those who know one big thing and try to integrate the diversity of the world into a comprehensive and parsimonious vision) and to nonexpert, more eclectic “foxes” (those who know many small things and try to improvise solutions on a case-by-case basis).
³⁶
Tetlock finds that who experts are (i.e., taking into account professional background and status) and what they think (i.e., whether they be liberal or conservative, realist or institutionalist, optimist or pessimist) are less important than
how
they think—their reasoning style. Specifically, he finds that “the tentative, balanced modes of thinking favored by foxes” predict better than “the confident, decisive modes of thinking favored by hedgehogs.” Among other differences, hedgehogs are slower to change their minds as much as they should when they err. This is partly because they employ “belief system defenses” to avoid such revisions, including the belief that bad luck explains their predictive errors more than good luck explains their predictive successes: “their defiant attitude was ‘I win if the evidence breaks in my direction’ but if not, ‘the methodology must be suspect.’ ” Both modes of thinking exhibit certain characteristic advantages and disadvantages, Tetlock finds, but “[t]he dominant danger remains hubris, the mostly hedgehog vice of close-mindedness, of dismissing dissonant possibilities too quickly.”
³⁷
More recent analyses by Nate Silver and Nassim Nicholas Taleb reach similar conclusions about experts’ overconfidence in their predictions and their systematic underestimation of uncertainty.
³⁸

Quite apart from Tetlock’s theory of the specific cognitive causes of predictive failures, many other studies have demonstrated how weak most predictive models are when they seek to forecast complex
nonmarket phenomena.
³⁹
As we shall see in later chapters, the success of federal programs depends on accurately predicting the interactions of autonomous individual actors and voters, bureaucracies, markets, other levels of government, and other factors. And, as we shall further see, these predictions are especially prone to error.

8. Ending a failed policy is a kind of policy success
. Occasionally, government officials acknowledge that one of their long-standing policies has failed and should be abandoned. Such confessions of governmental error are to be strongly encouraged, of course, but they are usually belated and exceedingly rare. One example, discussed in
chapter 11
, is the Airline Deregulation Act of 1978, which repealed a forty-year old regulatory cartel that was egregiously inefficient, replacing it with a competitive system that has benefited the vast majority of consumers immensely.
⁴⁰
Another example is the military’s repeal of its “don’t ask, don’t tell” rules in September 2011. No less an expert on the subject than congressman Barney Frank supported the original “don’t ask, don’t tell” policy because he was convinced that it was a step forward and the best policy reform that could then be obtained. The fact that the military now believes that its policy was wrong is indeed welcome.
⁴¹
Gay policy entrepreneurs’ vision, skill, persistence, and courage over a twenty-year period in agitating for the repeal of a misguided, unjust regime are remarkable and admirable. Compare this with the speed with which private firms that anger their customers often respond quickly to regain them—for example, the banks that had to revoke their debit card and swipe fees once consumers rebelled.
⁴²
Occasionally, a legislative policy is quickly reversed when it proves very unpopular; the Medicare Catastophic Coverage Act of 1988, repealed less than eighteen months after enactment, is an example.
Chapter 6
discusses policy rigidity in greater detail.

9. Policymaking demands appropriate organizational analysis
. The success of particular programs will depend on the kinds of organizations in which they are embedded and the ways that their task structures constrain and shape their behavior. The late political scientist James Q. Wilson explored these design considerations in some
detail, which I consider in
chapter 7
, on policy implementation, and in
chapter 10
, on bureaucracy.

10. Policy assessment requires an appropriate time frame
. Historical contingency, timing, and plain luck may affect a policy assessment. A program that seems effective at time A may seem ineffective at time B, and vice versa. Hindsight, of course, is 20–20, but myopia is far more common. The time frame constraint has three aspects. First, after a program is enacted, its opponents may succeed in undermining its effectiveness. Embedding a policy so that it can resist attack is difficult; the 1986 tax reform failed to do so, and “Obamacare” faces an even stiffer challenge given unanimous Republican opposition in the Senate and the law’s reliance on implementation by many recalcitrant states.
⁴³
Only time can tell whether this will happen.

Second, the program being analyzed must have been in effect for a sufficient length of time to demonstrate both its desirable and undesirable effects. If, for example, one were assessing the government’s 2009 bailout of General Motors solely in light of its consequences—that is, not in terms of whether such a policy is desirable in principle or in terms of its possible future precedential effect—then one should wait long enough to determine whether the company can be restored to profitability without huge losses to the government. (This will evidently be a long time. In September 2012, the U.S. Treasury, fearing that it would suffer an immense loss, once again rejected GM’s pleas to sell its remaining stake in the company.
⁴⁴
) At the same time, the bailout of American International Group was successful by many accounts,
⁴⁵
although the government’s cost-benefit calculations do not factor in the below-market rates that it charged the company (and other recipients of large bailout loans) at a time when liquidity was almost impossible to obtain otherwise.
⁴⁶
In the case of Head Start (discussed in
chapter 6
), the policy’s goal is to improve long-term education, social, and health outcomes for participant children as they reach adulthood, so assessment of its performance requires a suitable time frame to permit that reckoning. Some policies—the Safe
Streets and Crime Control Act of 1968, for example—may have far-reaching effects long after they are repealed.
⁴⁷

Third, the time frame should allow for the possibility that a policy that makes sense at time A no longer does at time B. The alternative minimum tax, for example, may have been a sound and successful policy when it was enacted in 1982 (a predecessor minimum tax was enacted in 1969), but a fair assessment today would properly consider the fact that because of inflation and bracket creep, it now applies it to millions of middle-income taxpayers who Congress never intended to subject to it.

11. Policy assessment requires competent and objective assessors
. This precept is obvious, yet needs some elaboration. Any assessing organization has some characteristic perspective or orientation about government programs, even if it does not have any manifest political or ideological ax to grind.
*
Program evaluations are often costly and difficult to conduct, and none is likely to be immune from some criticism, so we must take them as we find them and then try to assess the assessments by applying to them professionally refined principles of analytical rigor with as much objectivity as possible. This can be challenging for outsiders not intimately familiar with the program data, but there really is no alternative if we take seriously the ancient poet Juvenal’s query
quis custodiet ipsos custodies—
“who will guard the guards?”

12. The well-designed randomized controlled experiment is the gold standard for assessment
. This methodology, discussed in
chapters 11
and
12
, is the only way to justify confidence on the part of policy makers that they know what they think they know about the effects of actual and proposed policies. But although it is now used to test policy innovations in business, political strategy, and many other areas, policy makers seldom use it.

13. Policy assessments must take most of the existing social and institutional context as given
. The assessment, to be useful, must take certain existing conditions as a baseline and then try to predict how outcomes would change if we altered some of those baseline conditions. We may consider some of these baseline conditions suboptimal in the sense that society would arguably be better off today had it made different choices in the past—indeed, if we didn’t consider the status quo suboptimal, we probably wouldn’t conduct the policy analysis in the first place—but we must hold them constant. For example, our society would be very different today—perhaps better in some respects, worse in others—had the United States invested in creating better rail networks instead of using the resources to build a national highway system. Nevertheless, when we consider policy proposals today on mass transit, we must take the highway system and its consequences as a given for purposes of conducting the assessment. This is not conservative; it is simply necessary for a coherent, accurate policy assessment. The policy implications of such an analysis, of course, may be conservative, radical, or something else altogether.

14. Avoid the “Nirvana fallacy.”
When one concludes that a policy has failed, one usually has in mind a better, more effective, reformed version of that policy. Fortunately, such reforms often do improve the program’s performance. In making such comparisons, however, policy makers should not idealize the new version, for it will be vulnerable to many of the same structural factors that have promoted failure in the existing policy. Economist Harold Demsetz has called this seductive idealization “the Nirvana fallacy”: viewing the policy choice as if it were one between an ideal program and the existing, flawed one.
⁴⁸

In conclusion, policy success or failure is
not
simply in the eye of the beholder. We have seen that good policy assessment rests upon a number of well-established, relatively uncontroversial criteria that are certified in analytic methodology and routinely used in government practice. Although the application of these criteria to particular policies might be contested, it turns out that the
results
of
such assessments are
also
remarkably consistent—and consistently negative. Winston, who as noted in
chapter 1
claims to have read all of the studies, finds,

Notwithstanding the potential for methodological disputes to arise when microeconomic policies are evaluated, my assessment of the empirical evidence reveals a surprising degree of consensus about the paucity of major policy successes in correcting a market failure efficiently. In contrast to the sharp divisions that characterize debates over the efficacy of macroeconomic policy interventions, I found only a handful of empirical studies that disagree about whether a particular government policy had enhanced efficiency by substantially correcting a market failure…. Generally, my fundamental conclusions are not influenced by studies that use a particular methodology. In fact, researchers who used vastly different techniques to assess specific policies often reached very similar conclusions.
⁴⁹

Nor is Winston alone in this conclusion. Derek Bok’s careful review of the evidence concurs: “Again and again, in field after field, the operative legislation is burdened by unrealistic objectives, inadequate funding, clumsy implementing machinery, and poor targeting of funds. The costs in terms of waste, frustrated expectations, and harmful side effects are virtually incalculable.”
⁵⁰