Taking Oxfam’s evaluations to another level

Methodology

The best evaluations of development programmes combine qualitative and quantitative data. But working out how to use the data harmoniously and adapt research to fit the latest findings is not straightforward. Here Simone Lombardini, Global Impact Evaluation Adviser, explains how Oxfam went about measuring impact on women’s empowerment in Pakistan recently.

Last week I had the pleasure to present the recently published evaluation report ‘Women’s Empowerment in Pakistan: Impact evaluation of empowering small scale producers in the dairy sector‘ at the IDEAS’s Global Assembly conference in Bangkok. The conference brought together evaluators from all over the world and experts in development evaluations to share their work and
experience. There was interest in the evaluation Oxfam conducted in Pakistan as it provides an honest overview of the strengths and limitations of Oxfam’s efforts to conduct a rigorous impact evaluation using a quantitative impact evaluation design combined with a qualitative component.

an honest overview of the strengths and limitations of Oxfam’s efforts to conduct a rigorous impact evaluationOxfam has been seeking to understand the extent to which its projects have contributed to women’s empowerment since 2011 by conducting effectiveness reviews, robust impact evaluations of its development projects. The approach used for these evaluations continually evolves and improves to match the commitment to conduct these
evaluations in a way that is rigorous, but also pragmatic and proportional
.

The effectiveness reviews have mainly used quasi-experimental impact evaluation designs. Whilst these tools provide us with robust and quantifiable evidence of impact which can be generalised for the entire project, they have a limited ability to unpack why the projects did or did not have an impact. For this reason, in the recent evaluation of the empowering small scale producers in the dairy sector project in Pakistan,
we combined the rigorous quantitative component with additional qualitative methods.

Measuring women’s empowerment

In order to measure women’s empowerment, Oxfam has developed a multi-dimensional framework containing a range of indicators intended to capture the characteristics of an empowered woman in the socio-economic context under analysis. While the dimensions of empowerment remain constant, because women’s empowerment is considered context specific, the specific characteristics and relevant indicators of empowerment are defined differently in each evaluation. The framework recognises three possible levels where change can take place: at a personal
level, in relationships with others, and changes in the broader environment.  These indicators are combined into a composite index, providing an overall measure for empowerment which can be used to estimate the project’s impact the specific characteristics and relevant indicators of empowerment are defined differently in each evaluation

the specific characteristics and relevant indicators of empowerment are defined differently in each evaluationThe aim is to identify a holistic definition of empowerment, which might be wider than the specific objectives of the project under review. This helps to maximise learning, allowing Oxfam to investigate the intersectionality between the dimensions and characteristics, and to consider both intended and unintended impacts of the project.

For the effectiveness review in Pakistan eighteen different indicators were selected and defined during a three day workshop with project staff, partner organisations, and local consultant experts. Then, in order to assess the framework’s validity, the indicators were discussed and analysed by the women participating in the project during focus group discussions.

The qualitative-quantitative approach – what to improve next

The quantitative element involved comparing, with propensity score matching, 300 randomly selected women among the project participants and more than 500 randomly selected women in 25 comparison villages. This was combined with qualitative components consisting of a literature review, six key informant interviews and ten focus group discussions. 

While the quantitative findings provided evidence that the project led to an increase in the overall women’s empowerment index, particularly in: self-confidence, personal autonomy, group participation, proportion of household personal income, control over time, and safety of movements outside the house. The qualitative findings mostly confirmed the measurement tool that was developed, and also provided additional tools for better understanding the strengths and weaknesses of the project. 

While we believe we are going in the right direction with this approach, this pilot didn’t go perfectly to plan and there is clearly space for improvement. We hoped that the qualitative data could be used to tailor the survey instrument, inform decisions on cut-offs and weighting during analysis, and support interpretation and/ or investigate some of the qualitative findings that emerged.  In practice, the qualitative component took place simultaneously with the survey data collection process. This meant that in the end, the findings from the qualitative component could only be
used to help us to better understand context and interpret survey data. 

Where next?

This evaluation shows Oxfam’s commitment to continuously improve our evaluations and our programmes (Read the final Pakistan effectiveness review to find out what we discovered).  And we remain committed to continuing to trying to embed more qualitative enquiries into those effectiveness reviews which take a quasi-experimental approach to increase the credibility of findings and maximise learning.  

We are now moving from paper surveys to digital surveys wherever possible, which will help strengthen the integration of the qualitative and quantitative approaches even further.  It means that some information from the survey can be processed instantly, and then used to inform focus group discussions.

Read more

Photo: Women involved in the Raising Her Voice programme in Pakistan, 2013. Credit: Irina Werning/Oxfam

Author: Simone Lombardini
Archive blog. Originally posted on Oxfam Policy & Practice.