Close

Outcome harvesting Outcome harvesting begins by identifying changes (whether intended or not, desirable or not) and then works backwards to find out whether and how the intervention being evaluated is likely to have contributed to them. This approach has the merit of not requiring a theory of change; indeed, it can be used to develop one. It is particularly useful when cause-and-effect relationships are not well known. For example, to evaluate a series of local projects aimed at combating gender stereotypes, the starting point is concrete examples of changes observed by the programme promoters. Evidence of a link with the intervention is gathered through specific questioning to check the plausibility of the contribution. A theory of change can be developed to synthesise the results.

Randomised Controlled Trial This approach compares, in the context of an experiment, two groups of people drawn at random from the eligible population; one benefiting from the intervention, the other not. These groups are observed with regard to an indicator measuring expected changes. The impact is the difference between the value observed for the group with the intervention and the group without the intervention. For example, to evaluate an inclusion programme, the 18-month return to employment rate of those selected to participate in the programme will be compared with that of the other group who were not included in the programme.

Difference-in-Differences (DiD) In the absence of matching, a group sufficiently similar to the beneficiary group is formed and observed on a given indicator.  The impact is estimated by subtracting the difference between the two groups after the intervention from the difference observed before the intervention. This operation aims to neutralise the difference between the groups. It is best to be able to check the evolution of this indicator before the intervention to ensure that the two groups indeed behave similarly. For example, to evaluate the effect of a business support programme, reasonably similar businesses in terms of activity and turnover in the three years preceding the start of the intervention will be identified, in order to be able to compare the evolution of turnover one year after receiving the aid..

Quasi-experimental As in the experimental method, two groups are observed with regard to an indicator to estimate the impact. As the evaluation is carried out ex post, a "control" group made up of "twins" of the beneficiaries must be reconstituted, identified from internal or external databases. Large, comprehensive databases are required to carry out such an evaluation. For example, a programme helps companies to recruit. A comparison group is formed ex post from Chamber of Commerce data to compare net job creations 12 months after receiving the assistance. As many factors may explain why a company applied for the assistance, a propensity score is calculated to form a comparison group. It refers to the probability of participating in terms of observable firm characteristics.

Longitudinal studies An indicator is measured before and after the introduction of an intervention. If a break in the trend is observed and cannot be attributed to another factor, it will be attributed to the intervention. Note that this approach allows an impact to be identified, not measured: in the mid-term, other factors, particularly contextual ones, may explain the changes observed. This method can be used to evaluate the effect of a congestion charge on air pollution. The longitudinal measurement of the pollutants and their main explanatory factors makes it possible to verify that the introduction of a toll caused a break that cannot be explained by other factors. On the other hand, it does not make it possible to attribute the measured difference to the toll.

Most Significant Change The Most Significant Change For example, to evaluate a series of local projects aimed at combating gender stereotypes, the starting point is concrete examples of changes observed by the programme promoters. Evidence of a link with the intervention is gathered through specific questioning to check the plausibility of the contribution. A theory of change can be developed to synthesise the results.

Impact Evaluation Approach Tree Hello and welcome to this tree of impact evaluation approaches. To navigate the tree, start with the impact questions (top), check the main conditions for the use of the different approaches, and then consider the methods available to the evaluator. Click on each box to find out more. This tree is freely reusable (except for commercial use) provided that the source is acknowledged: "Quadrant Conseil, 2017 - www.quadrant.coop". Graphic Design: Atelier Beau/Voir

Qualitative Comparative Analysis A limited number of factors ("conditions") that can explain the success or failure of the intervention, known in advance (possibly through an initial collection phase), are systematically recorded on a number of cases (usually 20 to 50). Statistical analysis allows for the identification of patterns explaining success or failure and whether certain conditions are necessary or sufficient to achieve the expected effect. For example, in order to evaluate in what cases train services can contribute to reduce car use among residents, the evolution of the modal share in different areas is first systematically qualified using statistical data. Then, the main conditions (internal or external) that can explain a positive or negative evolution of the modal share are identified and then characterised through specific data collection. Finally, the analysis aims at identifying whether some of these factors are individually or collectively necessary or sufficient to lead to the expect outcome of reducing car use.

Simulation If two groups cannot be compared, the comparison is simulated. A multivariate model is developed whose outcome is the expected change. The impact is estimated by comparing the results obtained with and without the intervention. Caution: the quality of the model and its transparency are essential for the credibility of the results. For example, to evaluate the effect of a tax relief system for the construction of housing, a model is constructed that simulates the construction of housing without tax assistance. The housing created are estimated by comparing the number of housing simulated excluding the intervention with those actually built.

Regression discontinuity When matching is impossible, two groups that are very close but located on either side of a fixed threshold are compared and observed with respect to an expected impact indicator. The impact is estimated using the difference in value of this indicator between the two groups.  For example, students can access a scholarship if their household revenue is under 15,000 euros per year. This threshold is arbitrary: to estimate the effect of the scholarship, one can compare the academic results obtained by those whose income is just below and just above this threshold, as long as their observable characteristics remain close.

Configurational Cross-case Analysis A number of contexts in which the intervention being evaluated is implemented are studied in depth, and analysed using a specific method to identify the factors that explain the success or failure situations. "Screening & scoping" is an example of such a method. This approch is more flexible than QCA and better adapted to complex situations, but also less detailed. For example, in order to evaluate a regional system of Health Centres, These are first characterised using a series of criteria likely to explain their success or failure with regard to one or more expected effects. A typology is used to identify typical cases, which are studied in greater depth. The main hypotheses as to the factors for success or failure are explored in greater depth leading, if possible, to configurations of drivers associated to expected changes.

Regression Analysis Potential success or failure factors (independent variables) are correlated with expected changes (dependent variable) over a large number of observations. However, for the impact to be proven, competing explanations must have been tested and rejected. This is the principle of many so-called "Big Data" impact evaluations. For example, in order to identify the cases in which an adult training programme best enables a return to employment, a survey is conducted among the beneficiaries. A statistical analysis allows the identification of the characteristics of these people most frequently associated with the expected outcome.

Contribution Analysis Contribution analysis is a 6-step process to systematically test the different stages of the theory of change of the intervention being evaluated. For each step, it indicates whether or not the expected changes have occurred, and identifies in a contribution story the main direct and indirect contributions to these changes, including the intervention being evaluated. For example, to assess the contribution of a research centre to decision-making on a given subject, one would first identify the ways in which this centre (or others) could contribute to the decision. Then the analysis will aim first to qualify what decisions were taken and how; then look at the evidence (dis)confirming the contribution claims; at the evidence of other contributions; and finally explain these contributions.

Process Tracing Process tracing is the systematic analysis of all mechanisms that can explain an expected change using a series of empirical tests. The probability of the contribution can be accurately estimated using a Bayesian inference approach. Beware that process tracing is usually applied to a single case, due to its degree of depth. For example, in order to evaluate France's contribution to a European decision, the mechanisms by which French actors were able to interact with their counterparts and the European Union to assert their positions, as well as other explanations that may lead to a similar outcome, are first identified in detail. Empirical tests are then used to test each of these mechanisms.

Congruence Analysis Congruence analysis systematically identifies the social theories that can explain an observed change (intervention-related or not) and checks whether these explanations complement, duplicate or compete with each other. It results in the identification of social theories (primary and secondary) to explain the observed changes. For example, in order to evaluate an anti-discrimination programme in the workplace, one would start by identifying the main theoretical frameworks that can explain discrimination (and the means to fight it): discrimination "by taste", by interest, statistical discrimination. These frameworks are applied to identify hypotheses that should hold if the theory works, and possibly empirical tests accordingly. The analysis allows theories to be ranked according to their explanatory power in the case under evaluation.

Concept Mapping Concept mapping of impacts is a method for identifying and weighting the different potential impacts of a programme. Potential impacts are identified from a literature review or the views of experts or stakeholders. They are then weighted and grouped, and these groupings are then projected onto a map. For example, to identify the impacts of a tramway on the area it passes through, a literature review and interviews will identify all plausible impacts. A group of expert stakeholders then weighs up these impact statements in response to two questions: is the impact proven? Is it desirable? They are also grouped by the participants. These groupings are projected in the form of a map. The results provide an initial overview of the likely impacts of the tramway, which can then be examined in greater detail.

Realist Evaluation In realist evaluation, the intervention is an opportunity that stakeholders decide to seize or not. The reasoning by which they make this decision is called a mechanism. The aim is therefore to identify these mechanisms and to compare their capacity to explain the changes observed in a variety of contexts studied. For example, in order to evaluate a cultural mediation programme, we begin by identifying the mechanisms that can explain why and how social workers would use this programme, and why and how the targeted social groups would use it in turn. These mechanisms are then studied on a number of deployments of the programme to verify for whom and in which contexts they are activated.

Ethnographic monographs When the complexity or diversity of cases is such that it is very difficult to generalise, ethnographic monographs make it possible to identify the impact in specific cases. These monographs are generally based on fieldwork lasting several months. A synthesis may allow lessons to be drawn, but generally without any possibility or ambition of generalisation. For example, to evaluate a policy fighting illegal work, ethnographic monographs can be used to understand how labour inspectors use the legal arsenal and existing directives, the priority they give to this subject, etc. The evaluator will endeavour to follow the teams concerned over time to better understand the driving forces behind their practices.