Rescuing Knowledge at the Knowledge Bank

The Economist comments about the recent review of World Bank research by an esteemed group of economists chaired by Angus Deaton;

"But if they were withering about this “long tail” of inconsequential work, they were much ruder about the way more striking research findings were used by the bank's top brass. They accuse the leadership of taking “new and untested results as hard evidence that its preferred policies work”; and of then using this research to “proselytise on behalf of bank policy”. This has a cost, the inspectors point out. “Placing fragile selected new research results on a pedestal invites later recrimination that undermines the credibility and usefulness of all bank research.”

In the main, the academics levelled this charge at the bank's big-think studies of globalisation. Comparing scores of countries in one statistical sweep, these studies concluded that the poor do not get left behind by growth; that poverty falls when tariffs are cut; and that aid works, but only if the governments that receive it pursue “sound” policies. These results, especially the last, have been enthusiastically promoted by the bank's leaders. The underlying propositions are probably true.

It is inconvenient, then, that the results tend to fall off their pedestals when prodded or poked a little. In 1997, for example, Craig Burnside and David Dollar published a paper showing that aid results in faster growth, but only in countries with stable prices, lots of trade and prudent budgets. The paper eventually appeared in the American Economic Review, as well as in a bigger research report the following year on “Assessing Aid”. This newspaper welcomed the work, and its message helped to fire the new enthusiasm for aid shown at the UN's Monterrey Conference in 2002.

Unfortunately, the study, like many in the genre, proved fragile. Critics have shown that adding four more years to the sample (and plugging some other gaps) undermined the result. Using more plausible measures of aid and policies had the same effect. And the paper did not convincingly disentangle cause and effect: perhaps faster growth attracts more aid money from donors keen to back a winner.

According to François Bourguignon, the bank's chief economist since 2003, “Assessing Aid” was championed by the bank's leaders because it was “convenient”. The work confirmed a plausible intuition they already held. “It was a very good piece of research the bank's management could use for its advocacy and it was difficult to resist.”

This temptation was easy to understand. The bank spent much of the 1990s under attack from critics on the left and right, claiming that trade was akin to imperialism, growth bypassed the poor, or that aid was money down a rat hole. Unsurprisingly, it sent in its best economists to lance its detractors...

True to this intellectual fashion, the evaluators urge the bank's economists to immerse themselves in the bank's work “in the field”. They also call on them to unbutton their lips, crying foul if their superiors over-egg (or ignore) their results. After all, the best way for the bank to disarm its dafter critics is to replace them with more sophisticated ones. That is a role its shrinking band of economists should be encouraged to play."

Some excerpts from the report;

“The use of consultants was uneven and apparently haphazard between Bank researchers and Bank projects, and it appears as if some groups or individual researchers are much more successful than others in selecting consultants, or better at working with good consultants. It seems that weaker researchers tend to select weaker consultants as in Hirschman’s law of squares which posits that first rate researchers make first rate appointments; second rate researchers make fourth rate appointments, and so on.” p. 70-1

"Daron Acemoglu, who admittedly uses a very high standard of comparison, finds that almost all that he read was well below academic best practice. The work was typically well-written, usually aware of the relevant literature, and occasionally addressed important and neglected issues, particularly in the books and flagships which were clearly doing a good job of dissemination. Much of the work was flawed by “serious deficiencies in terms of empirical work or conceptual framework.” Like several of our evaluators, Acemoglu feels that there are many talented researchers in the Bank, but that the work does not always reflect those talents nor use them to the best purpose."-p.102

Related;
Evaluation of Bank Research
Rescuing the World Bank
Roads out of poverty? assessing the links between aid, public investment, growth, and poverty reduction
Assessing aid - what works, what doesn't, and why