The Millennium Villages project (MVP) is an ambitious program to break targeted African communities free from extreme poverty and thereby demonstrate how poverty in general can be eradicated. Run by the United Nations Development Programme [*] and Columbia University’s Earth Institute—headed by renowned economist Jeffrey Sachs—the MVP began its pilot phase in a Kenyan village in 2004 and has grown to include numerous rural village clusters in nine other countries. It hopes to scale up its approach across much of Africa, and has plans on other continents as well.

The MVP’s strategy employs an intense, multi-pronged aid package for agriculture, education, health, and infrastructure—with a total cost of about $1.5 million per village cluster spread out over five years. It seeks to disrupt, all at once, the factors that together create the conditions for destitution. The MVP treatment simultaneously deploys a variety of aid-financed interventions, including fertilizer, anti-malaria bed nets, and school meals, among many others. Where a single intervention might leave villagers trapped in poverty by other problems, the MVP asserts that its simultaneous execution of several interventions can break villages out of poverty traps for good.

Although its aims are noble and its leaders well-intentioned, the MVP suffers from a glaring omission: it is not conducting an objective and independent evaluation of its impacts. The MVP downplays the importance of impact evaluation, since it accepts each element of its intervention package as already “proven.” According to the MVP’s view, as long as basic science has already shown that fertilizer can raise crop yields, bed nets reduce malaria transmission, and so on, there is no need to prove that its multi-pronged package of interventions can, in context, do what it claims. For example, it claims that this package can spark “self-sustaining economic growth” in isolated rural villages.

By ignoring the crucial step of objectively and independently evaluating its interventions’ impacts in context, the MVP cannot assure its donors that it has succeeded in achieving the promised development impact. Skipping this step means possibly wasting scarce aid money on one intervention, while a more effective one goes ignored.

Impact evaluation is a set of applied scientific tools to measure exactly how a particular community improved due to a given anti-poverty project, relative to how that community would have faired without that project. With so many solutions on offer for eliminating global poverty—population control, girls’ schooling, debt forgiveness, high-yielding seeds, free trade, microcredit, property rights, public-private partnerships, and information technology, to name a few—impact evaluation provides a basis for adjudicating among them and determining which package of solutions is most effective for a particular context. What science has shown to be effective generally may not work in a particular milieu; what succeeded in a Malawian village, for example, may not work in Burundi.

Despite its promise, there is surprisingly little impact evaluation of this kind performed in global-development circles, but there could easily be more. The MVP is no exception.

Since the MVP views the “solution to extreme poverty” as essentially technological, it is concerned primarily with how to deliver technology. But this is profoundly problematic as development policy. A clear scientific finding in one context may tell us little about what makes a sensible development project.

The Millennium Villages project cannot assure its donors that it has achieved the promised development impact.

Take fertilizer. The basic scientific result is that fertilizer raises crop yields. But, first, this does not mean that any given aid package that includes fertilizer can end poverty and unleash self-sustaining economic growth in a small village, even in combination with other efforts. Second, that finding does not mean that any given project promoting fertilizer use will actually influence use. There may be contextual reasons why farmers in a given place and time dislike fertilizer. Third, numerous forces other than the project can affect crop yields: that crop yields went up after a village received fertilizer does not mean that the fertilizer was primarily responsible, or that lack of fertilizer is the binding constraint on crop yields in villages elsewhere.

Just as the soundness of the basic science does not prove the success of an intervention, so too does in-house, qualitative evaluation obscure results. Yet this has been the MVP’s approach.

Together with my co-author, Gabriel Demombynes of the World Bank’s Kenya office, I have shown how wrong a project can go when it skips rigorous impact evaluation. Last year, the MVP issued a report (PDF) showing improvement in various development indicators at its sites. This work, described by the project’s leaders as a “major scientific report,” shows increased cell-phone ownership, access to improved drinking water, and crop yields. The report describes all of these advancements and many others as the “impacts” of the intervention.

In one instance, the report describes an increase in mobile phone penetration from 4 percent to 30 percent as among the project’s “biggest impacts” in the Ghanaian village of Bonsaaso.

Yet the same changes have been going on all around Bonsaaso, in areas untouched by the project.

Considering all of the indicators and countries that public data allow, Demombynes and I have found that the MVP claims as its “impacts” several increases in indicators that are on the rise throughout the regions and countries where their sites are located—that is, in places untouched by the project.

In some cases, increases away from the intervention sites have been smaller than increases at the intervention sites, suggesting that the MVP is having an effect. But while the MVP does intend to go beyond before-and-after analysis and observe outcomes at “comparison” villages that did not receive the intervention, its approach falls short of genuine impact evaluation. For example, it is unclear how the current intervention sites were chosen, which makes it difficult to know if they were unusually likely to respond well to the project. And the MVP only chose comparison villages and started to collect data on them years after the project began, rendering them an unreliable control group.

Though development organizations frequently make overly optimistic claims about their impacts, the MVP is supposed to be different. Its publications, including the report referenced above, are prominently labeled as a product of Columbia University—one of the world’s leading institutions of scientific research. The MVP often claims unique credibility due to its ostensible focus on science. And its special status as a flagship initiative of the United Nations [*] gives it special responsibilities in this regard. We have a right to expect higher standards.

United Nations Development Programme Administrator Helen Clark and economist Jeffrey Sachs / Maureen Lynch, UNDP / Wikimedia Commons

There are many possible ways to evaluate the MVP’s impact more credibly. One way depends on the kind of analysis mentioned above: comparing the intervention sites to trends happening around those sites. An even better way, feasible but more demanding, requires a randomized treatment design that has become standard in the research literature of development economics. When money is available to intervene at several more sites—only twenty would be needed, far fewer than the MVP ultimately intends—an independent agency should randomly pick which sites receive the intervention. It would pick one from each of twenty matched pairs of candidate sites. That agency should then follow both members of each pair for several years after the intervention ends, to evaluate long-term impact.

Such engaged, rigorous, objective experimentation would resolve the problems with settling for basic science and technology. It would judge the effect of the project against its goals, clearly document that the project was their cause, and be free of potential conflicts of interest. And as we have shown, the price would be less than a sixth of the intervention cost per village—a tiny investment that is surely worthwhile if the intervention is going to spread across the continent, affecting millions of hopeful Africans.

Rigorous impact evaluation has many limitations. It can be hard to know if the effect measured in a small experiment would hold in a much larger intervention, hard to know on what factors the measured effect depends, and hard to measure all of the desired impacts comprehensively. These issues are now the subject of intense debate among the best minds in development economics. Whatever the standard, it is apparent that rigorous and independent impact evaluation should not be required for every project. Few would claim, for example, that emergency shipments of drinking water after an earthquake require rigorous evaluation.

But careful impact evaluation should be used when the benefits outweigh the costs. This is likely to happen when, first, the cost of properly collecting evidence is relatively low; second, a policy decision is not needed more quickly than rigorous analysis can be performed; third, the consequences of applying the wrong policy are particularly bad; fourth, resource constraints make it impossible for everyone to receive the intervention at once; fifth, strong interests on different sides of a policy decision necessitate an objective criterion of success; and finally, the carefully evaluated setting is similar enough to the scaled-up setting.

Many development projects, including the MVP, meet all of these conditions, so their impacts should be carefully and independently evaluated. A model in this regard is Mexico’s PROGRESA (the Education, Health, and Nutrition Program)—later renamed Oportunidades (Opportunities)—which rewarded low-income families with cash for enrolling children in school. Its initial phase was built on a careful, randomized impact evaluation executed by researchers outside the Mexican government. After this independent evaluation demonstrated large positive impacts, similar policies were greatly expanded in Mexico, survived a major change of government, and spread throughout Latin America. In many of the countries where the idea was later applied, it was rigorously evaluated anew, in context, and independently.

The challenges of impact evaluation are many and can frustrate the development of urgently needed delivery technologies in a world in which aid funding is scarce and has countless plausibly beneficial uses. Yet it is essential to assuring that promised aid is in fact doing more good for more people. The long, troubled history of efforts to help the poor suggests that we have much to learn.

UPDATE (June 23, 2011): Since publication of the article, a spokesperson for the United Nations Development Programme (UNDP) has told us the following:

Boston Review’s sources of information regarding the relationship between UNDP and the MVP included public websites, such as the Millennium Villages website, that identified UNDP as the “lead implementing partner” of the project. As of June 23, the Millennium Villages website has been corrected.