This page provides a brief introduction to the concepts of impact and impact evaluation and an overview of the work of the Network, in particular through the Network of Networks on Impact Evaluation (NONIE).You can find completed impact evaluations on DEReC or follow these links to resources on impact evaluation.

 

What is impact evaluation?

Impact is one of the five core evaluation criteria for assessing development results, along with relevance, efficiency, effectiveness and sustainability. Impact is defined  by the DAC as: "Positive and negative, primary and secondary long-term effects produced by a development intervention, directly or indirectly, intended or unintended" (OECD DAC 2002).

 

In international development, many evaluations include impact as one of the criteria used to evaluate a development project or programme. The term impact evaluation is used to indicate specific types of evaluations that are primarily concerned with final results of interventions (programs, projects, policy measures, reforms) on the welfare of communities, households, and individuals. Impact evaluation is one tool within the larger toolkit of monitoring and evaluation, including broad program evaluations, process evaluations, ex-ante studies, thematic evaluations, participatory assessments, etc. (NONIE Guidance, 2009).

Guidance on Impact Evaluation (NONIE) Highlights key conceptual and methodological issues in impact evaluation, covering such topics as delimitation, intervention theory, attribution, and combining methods. It also presents an introduction to participatory approaches to impact evaluation and assessing impact for complex interventions.

 

Impact evaluations are interested in effects caused by a development intervention, or, to use evaluation language, in attribution. This means going beyond describing what has happened to look at causality. Evaluating impact will, therefore, often require a counterfactual, or an assessment of the effects the development intervention has had, compared to what would have happened had the intervention not taken place.

 

However, interest in attributing results does not mean that a single set of analytical methods should be used above all others in all situations. In fact, the internationally agree NONIE Guidance on Impact Evaluation underlines that no single method is best for addressing the variety of evaluation questions and aspects that might be part of impact evaluations. Particular methods or perspectives complement each other in providing a more complete “picture” of impact. The most appropriate and useful

Resources on impact evaluation Several international institutions are now producing high quality impact evaluations: Useful resources on impact evaluation

methodology should be selected based on the specific questions or objectives of a given evaluation (as described in the DAC Quality Standards for Development Evaluation, 2010).

 

Rising Interest in Impact Evaluation

Impact evaluation is now high on the development agenda. The increased importance of this type of evaluation is linked to the focus on outcomes, as embodied in the Millennium Development Goals, and the need to demonstrate to donor nations the impact of the development projects they help finance. Driven by the results agenda, there is increased international attention by policy makers, practitioners, and evaluators to impact evaluation. Managing for development results implies a number of changes in the way interventions are designed, implemented, monitored, and managed. The role of evaluation in this context is to assess results in a credible and independent fashion, contribute to learning and accountability, and for effective policy decisions and programme improvement.

 

The Network of Networks for Impact Evaluation, or NONIE (see below), has made considerable progress in expanding awareness of impact evaluation and developed guidance. The World Bank Group's Development Impact Evaluation Initiative (DIME) has supported a number of impact assessments. A new international agency, 3IE, or the  International Initiative for Impact Evaluation, was created with the support of DAC members and others to fund and strengthen impact evaluations. A number of other initiatives in and beyond the donor community are looking at assessing impacts in specific sectors or areas of development cooperation. Visit DEReC to browse DAC member impact evaluations (use the title or keyword search) and for links to other impact evaluation databases.

 

The Value of Impact Evaluation

Impact evaluation can, if done well under the right circumstance, be a credible and useful tool for understanding development results and the role of development cooperation programmes in supporting change.

 

International experiences with impact evaluations highlight some of the ways this approach can be useful to development partners (Independent Evaluation Group Room Doc 3/A, 2008):

• Supporting or questioning a program: Impact evaluations demonstrated the effectiveness of conditional cash transfer programs on school attendance, children’s visits to clinics, and consumption poverty at a time when these programs were being dismissed by development practitioners. On the other hand, impact evaluations have also shown that some development projects, while popular, do not have the intended impacts.

 

• Tracking cross-sectoral benefits: Some of the strong development impacts come from interactions across sectors. An impact evaluation showed the linkage between a nutrition program in Bangladesh and secondary schooling for girls. Evaluation of the Mexican piso firme project that upgraded dirt floors to cement in slum housing revealed that the intervention also reduced the incidence of diarrhoea, anaemia, and parasitical infection in children. More recently impact evaluation has shown the education and health effects of rural electrification projects, even though the original emphasis of these projects was on infrastructure.

 

• Helping to depoliticize decision making: Impact evaluations can help to focus the support for programs on the basis of evidence of the benefits, depoliticizing decisions to some extent. By building an evidence-based case for a programme, they make it difficult for politicians to discontinue their support for the intervention after a change of government. This feature has important implications for decisions within development agencies and bureaucracies as well.

 

Of course, impact evaluation has important limitations and is not easily applied in the case of a good number of development programmes or policy situations. Impact evaluation is not intended to supplant or displace other evaluation tools. Furthermore, full-scale impact evaluation can be costly. This is one reason why impact evaluation cannot reasonably be expected to be done for more than a relatively small percentage of operations. It is important, therefore, to select subjects carefully so as to build a collection of policy-relevant knowledge.

 

There are cases where a full-blown, rigorous impact evaluation isn't necessary and where attribution is obvious. For example, where the installation of a pump in a village reduces the amount of time villagers spend walking to get water, a scaled-down impact analysis, namely a simple before-after analysis, without a control group, would be sufficient.

 

 

NONIE: the Network of Networks on Impact Evaluation

 

NONIE meeting, 19-20 April 2012, Paris, France www.nonie2012.org

• Read the full meeting report

The DAC Network on Development Evaluation has joined forces with three other major networks to form NONIE (the Network of Networks on Impact Evaluation). In addition to the DAC Evaluation Network, NONIE includes the UN evaluation group (UNEG), the multilateral development banks Evaluation Cooperation Group (ECG) and the IOCE, the main evaluation association grouping country evaluation associations. The groups meet to share experiences and discuss how to improve evaluation practice to better capture impacts, as well as strategies for using impact evidence in development policy and programming.   

Together these Networks have developed a Guidance on Impact Evaluation

NONIE was formed to promote quality impact evaluation. NONIE fosters a programme of impact evaluation activities based on a common understanding of the meaning of impact evaluation and approaches to conducting impact evaluation. NONIE focuses on impact evaluation and does not attempt to address wider monitoring and evaluation issues.

To this end NONIE aims to:

• Build an international collaborative research effort for high-quality and useful impact evaluations
  as a means of improving development effectiveness.
• Provide its members with opportunities for learning, collaboration, guidance, and support,
  leading to commissioning and carrying out impact evaluations.
• Develop a platform of resources to support impact evaluation by member organizations.

 

Past Work on Impact Evaluation

 

3IE Proposal PowerPoint presentation  (November 2008)

NONIE update PowerPoint presentation ((November 2008)

 

Impact evaluation (NONIE) - Room Doc 3/A (February 2008)

Status and Future of Impact Evaluation - Room Doc 3/B (February 2008)

Draft NONIE statement on impact evaluation - Room Doc 3/C (February 2008)

 

Report on work by NONIE -  Room Doc 6 (June 2007)

 

International Workshop on Impact Evaluation for Development (November 2006) Hosted by the World Bank and the DAC Evaluation Network

 

Back to areas of work