Distribution by Region

Distribution by Organisation Type, Document Type, Region and Year

Impact occurs mostly in documents published in Europe, followed by North America, Asia, Africa and Oceania with comparatively smaller contributions. Overall, the top five contributors in terms of occurrences are NGO, IGO, NGO_Fed, RC and Net organisations.

NGO, NGO_Fed, RC and Net documents provide the greatest number of occurrences, primarily from activity reports published in Europe. Occurrences from IGO were mostly obtained from general documents published in Europe.

Impact ...

• is a/an

  • effect/result of development interventions

    • positive & negative

    • primary & secondary

    • long-term

  • area of performance assessment

  • OECD/DAC criterion

• may be defined uniquely by each organisation

  • a narrative, not a number

  • benefits & outcomes of intervention

  • difference between intervention & status quo

• is sometimes called humanitarian impact

  • which can refer to

    • interventions

    • disasters, climate change...

• has dimensions

  • area

    • nature

• • • human systems

• • manner

• • probability

• time frame

• value (positive vs. negative)

• scale (significant vs. insignificant)

• location

• quantification (being measurable or not)

• is quantified using measurements, indicators & ratings

  • including positive & negative change

  • with qualitative & quantitative data

  • with global and contextual data

  • which change over time

  • which have usability limits

• is quantified in assessments, evaluations & reports

  • which have term variants

  • which have differing usages

• • which can be offered as a service

  • which include objectives

• • which have many subtypes, e.g.

• • which is overseen by networks & bodies, e.g.

• • which has various activities & models, e.g.

• is sometimes funded via impact investment

• is related to & sometimes confused with

  • output

  • outcome

  • effectiveness

• lacks consensus regarding its definition

• the monitoring of which

  • requires better measurement systems

  • requires more & better partnerships

  • requires more industry-wide published data

  • is hampered by assessments being too one-dimensional

  • is hampered by ad hoc assessments, lack of rigour, methodological flaws

  • is stagnant due to lack of incentive & reluctance to improve assessment standards

  • is fundamentally challenged by the difficulty of attributing cause & effect

In the context of development results, impact is defined as: positive and negative, primary and secondary long-term effects produced by development interventions, directly or indirectly, intended or unintended (OECD, 2010). 5.

An impact is defined as the enduring result and differs from a result assessed immediately after an activity is completed. In addition, impacts can have different levels. Individuals who through Läkarmissionen's support can advance to a new future is one type of impact, another is how well the local community develops in relation to its starting point, and a third is the degree of change in official attitudes to individual vulnerable groups.

Impact is not a number, it’s a narrative of how change has happened, and our role in helping to achieve that change. This is a really big shift in how we approach impact, and are held accountable for results

What does 'impact' mean at Save the Children?

Bidjan: It’s defined as the benefits and the outcomes that have been made through an intervention, directly or indirectly. But we need to keep two important things in mind: First, impact isn’t just what we find at the end through monitoring and evaluation.

What do we mean by impact? Many organizations measure “outputs”: how many people they’ve fed, how many textbooks they’ve provided. Oxfam seeks “impact,” not just outputs. This means we address the root causes of poverty and social injustice, not just symptoms.

A constant unifying theme for EWB’s overseas operations, and indeed our whole organisation, is a culture of impact. From co-CEO to brand new volunteer, we are always self-evaluating based on our desired impact. To us, impact means putting ourselves in the shoes of our beneficiaries and ensuring that their interests are at the heart of our programs.

Our Goal is to increase our impact in achieving safety, justice and dignity in communities threatened by conflict and disaster. Impact is the difference between what would happen with action and what would happen without it.

• areas of performance assessment

  • relevance, effectiveness, efficiency, sustainability

• OECD/DAC criteria

  • appropriateness, connectedness, coherence, coverage, efficiency, effectiveness

• types of development results

  • output, outcome

Most have left home due to security concerns and the humanitarian impact of the conflict (lack of water, food, medicines).

Activities are concentrated where the humanitarian impact is greatest in order to: enable safe access to water, shelter and food for remote communities, and assist the safer return of refugees and internally displaced persons to their homes;

• area

  • nature

    • climate change impact

    • disaster impact

    • environmental impact

    • impact of natural disasters

    • impact of natural hazards

  • human systems

    • development impact

    • economic impact

    • financial impact

    • health impact

    • human impact

    • humanitarian impact

    • impact of armed conflict

    • impact of high food prices

    • programme impact

    • social impact

    • socio-economic impact

• manner

  • cumulative impact

  • collective impact

  • overall impact

  • direct impact

  • indirect impact

  • wide impact

  • sustainable impact

  • material impact

  • psychological impact

• probability

  • expected impact

  • likely impact

  • possible impact

  • potential impact

• time frame

  • annual impact

  • immediate impact

  • long-term, lasting impact

• value

  • negative impact

  • positive impact

  • desired impact

  • adverse devastating, detrimental impact

• scale

  • significant

    • big impact

    • disproportionate impact

    • great impact

    • high impact

    • huge impact

    • important impact

    • considerable impact

    • major impact

    • maximum impact

    • most impact

    • profound impact

    • serious impact

    • severe impact

    • significant impact

    • strong impact

    • real impact

  • insignificant

    • limited impact

    • little impact

  • location

    • local impact

    • regional impact

    • global impact

  • quantification

    • measurable impact

We define impact indicators in four key categories. Such systemic change is a high bar, but that’s what shifting power demands. These are the indicators for our West African oil-drilling and gold-mining program:

Individual and collective voice and influence

• Number of formal actions filed by mining-affected populations relative to the total number of violations recorded [....]

Social practice

• Percentage of official complaints from villages filed with the judiciary or mines on which the government or companies take action [....]

Government and private sector policies

• Rating of the government’s fiscal transparency [....]

People’s well-being

• Rates of violence and intimidation across the region (per population affected) [....]

To measure our impact, we continuously monitor programme quality and evaluate our outputs using a range of quantitative and qualitative research methods. We understand the challenges in attributing impact to specific education interventions, and so we triangulate quantitative and qualitative data with observations in order to gain a holistic view.

Quantitative Tools

• Student surveys

• Classroom observations

• In-app progress monitoring

• Academic test

Quantitative Tools

• Student Focus Group Discussions

• Senior Leadership Interviews

Local data are collated and fed to international databases by intermediaries. Not only do intermediaries lack a standard set of definitions to organize their data, but they might be tempted to exaggerate or suppress data for professional, political or economic advantage. Each key indicator of disaster impact faces its own limitations. Mortality is the 'cleanest' indicator of disaster loss. But even here the distinction between deaths and people missing creates uncertainty, with some countries requiring that people be declared dead after they have been missing for 12 months.

To date, the measurement of social impact is still largely focused on inputs and outputs – for example, the number of children educated. The measurement of outcomes is much more difficult and requires specifically tailored approaches that can meet the needs of investors while not overburdening the social venture. The development of standard social impact measurement systems will be important for further engaging mainstream investors (HM Government, 2013).

Impact monitoring: Increasingly, the assessment of impact (the wider effects of interventions in the short to medium term, positive or negative, intended or unintended) is viewed as both feasible and essential for humanitarian response. Impact assessment is an important emerging field, linking particular humanitarian contributions to changes in populations and the context that are complex and interrelated. The affected people are the best judges of changes in their lives; hence outcome and impact assessment must include people’s feedback, open-ended listening and other participatory qualitative approaches, as well as quantitative approaches.

Impact evaluation – the measurement of long-term changes in conditions attributable to interventions – is a deservedly prevalent topic in humanitarian evaluation discourse today (Stoddard, 2011). However, impact evaluation is difficult, not least because it can be very hard to attribute particular changes to particular interventions, and there are very few examples of true outcome/impact evaluation of humanitarian assistance (see section 4.3.9 below, Monitoring and evaluation).

Social impact investment is the use of public, philanthropic and private capital to support businesses that are designed to achieve positive, measurable social and/or environmental outcomes together with financial returns (OECD, 2015c).

The social impact investment ecosystem is complex Social impact investments can be made across countries, sectors and asset classes and can produce a wide range of returns (Bridges Ventures, 2009).

According to the DAC definition, outcomes are the likely or achieved short-term and medium-term effects of a project's output, while impacts are the positive and negative, primary and secondary, intended and unintended long-term effects of a project.

This type of evaluation examines effectiveness (impacts and outcomes) of programmes.

Outcomes are defined in this context as tangible results arising from people's action to bring about change. Outcomes can range from initial or short-term results such as people achieving improved knowledge or skills in campaigning against gender based violence, through to more significant / longer term results such as these groups successfully taking action in reducing the incidence of early marriages in their programme area.

Arguably, the humanitarian sector is used to reporting on outputs but ill equipped to "define outcomes clearly, quantify and measure them".

Development results are defined as the output, outcome or impact (intended or unintended, positive and/or negative) of a development intervention (OECD, 2010).

The chart below represents the distribution of impact between 2005 and 2019 in terms of the number of occurrences and relative frequency of occurrences. It also allows you to view the distribution across Regions, Organisations and Document types.

Furthermore, given the diversity of operational contexts and their dynamic nature (sudden onset; protracted crises; natural disasters; conflict related; complex political; food and nutrition insecurity; etc.) there is no real consensus around the type of evidence that is acceptable or what 'impact' means in a humanitarian context. Nonetheless, pressure for an improved evidence base and analysis of the impact of humanitarian assistance has grown in line with the increase in resources allocated to the sector, and a broader focus on results, performance and overall value for money in the public sectors of donor governments.

Nevertheless, in many cases, maximizing impact means working closely with and through others – both within and outside the United Nations.

At the same time, multiplying our impact means operating in close partnership with our colleagues at home and abroad.

However, despite the increased attention impact assessment is attracting, there is still a dearth of published information on the impact of humanitarian interventions.2 There are many reasons for this, not least the methodological and ethical difficulties involved in obtaining control groups in emergency contexts, and the fact that no single agency is responsible for assessing the impact of different types of nutrition and food security intervention. Well-conceived FSIS could theoretically provide the vehicle for systematic impact assessment.

The main problem emerging from the review of evaluative reports was not only the limited quality of evidence, but the limited quantity of reports seeking to assess impact. Most evaluative reports focus on operational challenges linked to managerial issues, the timeliness of the response, co- ordination and staffing. The question of impact is only addressed at the margins.

The opportunities for long-term benefits in technology transfer, capacity-building and mainstreaming environmental practice should be realised. Drought preparedness is a priority. 6. Analysis in humanitarian programming in Darfur needs to make stronger linkages between conflict, protection, livelihoods and environment. Impact assessments in each area need to refer to the others.

Despite the gradual embedding of evaluations within the system over the last decade, the ALNAP Evaluative Reports Data Base shows that, in practice, there are still too few evaluations of impact. While new guides and frameworks have been written and new initiatives set up, humanitarian impact assessment is still ad hoc, rather than systematic. There are good reasons for this, including genuine difficulties around methodology, but there is also a lack of incentives for agencies to tackle this seriously.

Similarly, assessment of the impact of both external interventions and local responses to protracted crises has improved, but many donors and agencies are still reluctant to invest in impact assessment, as well as in response analysis, to the required extent. Impact assessment, monitoring and evaluation systems and learning and accountability mechanisms all need to be strengthened if we are to improve the way in which we respond to food security in protracted crises.

Data identifying specific barriers to inclusion can generate 'policy-actionable' information. Rigorous impact assessments (ex-ante and ex-post) also need to rely on clear and agreed indicators on social inclusion at national and local levels. The local level is particularly critical as often this is where the data gap is binding.

The disproportionate vulnerability of women (and children) to hazards, but also to exploitation during the social disruption that follows disaster, has not been adequately factored into disaster planning. For example, disaster impact assessments are not routinely disaggregated by gender..

There is more acceptance of the idea that impact evaluation is important. Approximately 25% of evaluations in the sample make some reference to ‘impact’. There is no standard treatment of the criterion however, and in a significant number of cases the concept is conflated with that of effectiveness. Where there is an attempt to measure impact, caveats about the problem of attributing outcomes to specific inputs are usually included. There are a few examples (including those by IRC and HSNP) of randomised controlled trials, using a control population adjacent to the area of intervention. Such evaluations tend to be university- sponsored studies with full academic/scientific rigour. But others point out that it is costly to establish a baseline and track progress against it.

On the whole, impact evaluations have been inadequate tools for the task in hand due to, among other things, a lack of baseline data and compressed timeframes. Impact monitoring has not worked and research programmes are prone to methodological problems, including the seemingly intractable problem of how to attribute impact in a multi-variable environment.

The principle refers to the publication Contribution to Change,11 which provides the steps for implementing this approach, and notes how this can overcome the challenge that 'Existing impact evaluations often focus on outputs achieved ... they tend not to look at the contribution of interventions towards the overall process of recovery'

Challenges to leveraging robust impact evaluations include:

• Short-term humanitarian funding cycles, which restrict timelines for scientific evaluation (e.g. randomised trials featuring waitlisted control groups to assess the added value of interventions; longer periods of follow- up observation to test for sustained impacts).

• Slow academic timelines, as publishing research often involves extensive data analysis and lengthy peer review processes of critical evaluation.

• Deep engagement to create trust and build strong academic–humanitarian community partnerships (moving from processes of ‘cultural adaptation’ to ‘cultural engagement’).

• Identification of the ‘active ingredients’ or key elements of specific interventions, in order to understand what works, for whom, why and for how long, in specific contexts (moving beyond the basic question of ‘what works’ in mental health and psychosocial programming).

• Sustainability of partnerships over time, to foster new ways of working (funds are rarely geared to sustaining dialogue in order to achieve a more effective dissemination of findings).

For social impact investment to become a credible bridge to a private sector focus on bottom-of-the-pyramid populations, it needs to address three challenges.

Align incentives for social and financial goals [....]

Change the economics of reaching bottom-of-the-pyramid populations [....]

Cultivate top-tier, on-the-ground investment and entrepreneurial talent [....]

Key recommendations for getting social impact investment right

●Advance knowledge of social impact investment instruments and their applicability in the context of the 2030 Agenda, in a variety of sectors and across different country settings.

●Promote international research, data collection, case studies and the development of indicators on social impact investment [....]

[See source document for full list]