On Aid and Growth and World Bank Research

"The influence of Easterly’s work has been equaled only by a series of papers and reports that use cross-country evidence to study how globalization affects poverty in countries with and without good policies. The paper by Craig Burnside and David Dollar published in the American Economic Review in 2000, “Aid, policies, and growth,” has currently 743 cites according to Google Scholar. Contrary to Easterly’s arguments, this paper, which argues that aid is effective in countries with good policies, has become the orthodoxy for those who are in favor of aid, and is cited in many prominent Bank documents. Dollar’s widely cited (893 cites on GS) paper with Aart Kraay on “Growth is good for the poor,” needs neither abstract nor summary. Another paper by Dollar and Kraay, in the Economic Journal in 2004, argues that countries that used large tariff cuts to open their trade to the beneficial effects of globalization have seen more poverty reduction than those that have not. Many of these arguments are brought together in a 2001 Policy Research Report on Globalization, growth, and poverty written by Dollar and Paul Collier. All of this work has had an enormous influence on the intellectual debates about globalization and poverty reduction and, to many around the world, it is seen as defining the World Bank’s position on these issues, as well as establishing the Bank’s intellectual leadership in the globalization debate.

The panel agrees that this provocative research program has set out some stimulating research questions, and applauds the Banks initial efforts. At the same time, however, we see a serious failure in the checks and balances within the system that has led to Bank to repeatedly trumpet these early empirical results without recognizing their fragile and tentative nature. As we shall argue, much of this line of research appears to have such deep flaws that, at present, the results cannot be regarded as remotely reliable, much as one might want to believe the results. There is a deeper problem here than simply a wrong assessment of provocative new research results. The problem is that in major Bank policy speeches and publications, it proselytized the new work without appropriate caveats on its reliability.

Unfortunately, as one reads the research more carefully, and as new results come in, it is becoming clear that the Bank seriously over-reached in prematurely putting its globalization, aid and poverty publications on a pedestal. Nor has it corrected itself to this day. We wish to emphasize that we, too, believe that countries with good policies and institutions are far more likely to benefit from aid than, say, countries with deep corruption and poor governance where aid can delay reform rather than enhancing it. There is a strong theoretical presumption in favor of this commonsense dictum. However, it is very unclear empirically where the line can be drawn, or which policies matter, and in our view, the jury is very much still out on any quantitative assessment of the issue. Nor does the panel challenge the Bank’s need to mount strong arguments in favor of its policies. Our problems are with the way that Bank research was used in the process, given the great credibility attached to what the Bank says.

Some of the Bank’s research on globalization, aid, policy, and poverty, was read by Francesco Caselli (LSE). He reviewed the Bank’s 1998 Policy Research Report Assessing Aid, written by Dollar and Lant Pritchett and which makes heavy use of the Burnside and Dollar argument that aid reduces poverty in countries with good policies. (The Burnside and Dollar results were then available as a 1997 Working Paper.) The argument is packaged for the broader policy community, and is used to argue for focusing aid on countries where there is both poverty and good policy. But as Caselli notes, subsequent studies, including a state-of-the-art study from the IMF that, although still within the tradition of cross-country regression work, is methodologically stronger than earlier work, have shown that the Burnside and Dollar results are not robust. It is possible to argue that the authors of Assessing Aid were simply unlucky, and that they could not to know that the ground on which they chose to take their stand was so deeply undermined. But the panel takes a somewhat different view. There is at the very least a good argument that it should have been clear from the outset that the evidence could not bear the weight that was placed by it in the arguments about, and justification for, Bank policy.

In spite of having been published in the American Economic Review, the Burnside and Dollar paper is unconvincing. The analysis uses an index of policy that combines the government surplus, the inflation rate, and an openness measure, at least two of which are measures of outcomes, not of policies (as is indeed recognized in later work by Collier and Dollar). It is also clear from the way in which this index is constructed that the results are not robust; attempts to work with all three measures fail, as does a principal components index, and the final index is constructed using a regression of growth on policy that is at best arbitrary, and at worst appears to be inconsistent with the main equation of interest. But this issue is dwarfed by the specter that haunts all of this literature, that external aid is not only a determinant of economic performance, but is determined by it. Burnside and Dollar conform to the previous (and subsequent) literature by using an instrumental variable technique, but this is a chicken-and-egg problem that is not readily resolved by mechanical means. In particular, it would require an unusually generous suspension of disbelief (even for cross-country regression analysis) to accept the identification assumption that the size of a country’s population, the (one period lag) of the share of its imports that come in the form or arms, and whether or not it is Egypt have no affect whatever on their economic growth rates except in so far as they affect its receipts of foreign aid. Again, we are not arguing that the Burnside and Dollar paper is weaker than most of the literature on aid effectiveness, but we are arguing that its results provide only the weakest of evidence for their central contention, that aid is effective when policies are sound.

The Bank did not appear to recognize the weakness of this evidence. Not only did it form the basis for the PRR Assessing Aid, but its results were built upon in a series of papers by Collier and Dollar that were published between 2001 and 2004 in the Economic Journal, in the European Economic Review, and in World Development. These papers use an arguably improved indicator of the quality of economic policy (derived as an average of scores by Bank staff on a number of dimensions) but they make no attempt to deal with the chicken and egg problem, arguing that because Burnside and Dollar’s results were very much the same whether or not they used instrumental variables, there is no need to worry. So their results, which go into the further Bank documents cited below, are derived by ordinary least squares regression. It is not clear why such an argument would hold for a different data set and different variables but, in any case, it is founded on the scarcely credible assertion that the identification assumption in Burnside and Dollar are valid. The Bank then built on this second round of work, giving it extended prominence in a 2002 book The Case for Aid, which brings together a speech by President Wolfensohn, a paper summarizing the Monterey consensus by Chief Economist Nicholas Stern, and a monograph-length substantial analysis “The role and effectiveness of development assistance,” by Ian Goldin, Halsey Rogers, and Stern, written for and presented at the Monterey conference. The monograph brings together evidence from a number of sources, but it contains detailed calculations of the effectiveness of current Bank aid, now directed to poor countries with good policies, as opposed to aid that is not so directed, including some of the Bank’s own previous lending.

We think that the Bank was unwise to place so much weight on one paper whose evidence is so unconvincing. At the same time, there was other work being done by Bank researchers, particularly by Easterly, that did not find aid effective, even conditional on good policies. That work shares many of the problems that are inherent in trying to use the cross-country evidence to make a solid inference about the effectiveness of aid. But the Bank reports prepared for Monterrey did not present a balanced picture of the research, with appropriate reservations and skepticism, but used it selectively to support an advocacy position. Once again, we emphasize that we do not think that the research was unusually weak relative to the literature. Nor do we challenge the appropriateness of the Bank’s making the best possible case for its policies. But once the evidence is chosen selectively without supporting argument, and empirical skepticism selectively suspended, the credibility and utility of the Bank’s research is threatened.

There is a similar set of issues with the paper “Growth is good for the poor” which is sometimes used to argue that, in the presence of economic growth, explicit anti-poverty measures are redundant. Yet, here too, there are serious questions about whether the conclusion is really supported by the evidence. Their measure of the incomes of the poor (the average per capita income in the bottom fifth of the population) is derived from aggregate national income using either estimates of the share of the bottom quintile from surveys, or from estimates of Gini coefficients of income inequality together with the assumption that incomes are distributed according to the lognormal distribution. The problem is that many of the estimates of the income shares and of the Gini coefficients are quite imprecisely measured and, when the data are uninformative about the true level of inequality, Dollar and Kraay’s procedure guarantees that, on average, the incomes of the poor will track average income. If the Gini coefficients were random numbers, the conclusion would be guaranteed. So, in the end, we do not know how much of the result is genuine, and how much is driven by errors in the data.

In this case too, there was a very different view in other Bank research, in this case by Branko Milanovic, who was providing extensive empirical evidence of increasing income and consumption inequalities in the world, and taking a much more jaundiced view of the benefits for the poor of growth and of globalization. Milanovic’s results have been criticized by others, and the panel takes no view on the issue, but there is certainly no consensus that his findings are incorrect. Yet once again, the official position of the Bank gave selective prominence to one set of views (for example, in the Monterrey document), although it is does not appear to the panel that one set of results is any stronger than the other."
-Evaluation of Bank Research

Related;
Aid, Policies, and Growth data set
Aid, Policies, and Growth: Revisiting the Evidence
Aid, Policy and Growth: A Threshold Hypothesis
Ten Reforms to Make the World More Conducive to Development
Globalization’s Assassin