Rising to the challenge: Measuring an expanding concept of resilience in Oxfam’s impact evaluations

Jonathan LainMethodology, Real Geek

Resilience has become central to the development sector in recent years, but conceptually it is still in a state of flux. In this blog post, we explore some of the new challenges associated with measuring the expanding concept of resilience in Oxfam’s impact evaluations or ‘Effectiveness Reviews’.

Resilience keeps getting bigger. What started off as an ambitious attempt to meld humanitarian and long-term development work, has become a huge cross-cutting entity in its own right. Resilience is no longer just about ‘bouncing back’. It now includes a host of risk management strategies, adaptation… and even talk of transforming the systems and structures that create vulnerability. Although moving the conceptual goalposts is not a problem in itself, it presents a challenge to any efforts to measure resilience.

While there are a host of conceptual and technical challenges associated with measuring resilience, we’ll focus here on Oxfam’s efforts to measure ‘transformative capacity’ in the Effectiveness Reviews (our impact evaluations) – here are some of the specific challenges, some of our (partial) fixes, and more…

The Background

Oxfam defines resilience as ‘the ability of women and men to realise their rights and improve their wellbeing, despite shocks, stresses and uncertainty’. For measurement purposes, we don’t simply wait for shocks and stresses (such as droughts, floods, and conflict) to occur and then check the progress of wellbeing. Instead, we identify context-specific drivers of resilience: characteristics that we think determine people’s ability to thrive despite shocks, stresses and uncertainty in the future. This allows us to evaluate projects shortly after their completion, regardless of whether a shock has occurred or not. Over the years, we’ve found several ways of identifying contextually relevant indicators of resilience from complex conceptual resilience theories.

Using this approach, the number of possible resilience indicators gets really big, really fast – in a typical evaluation, we might measure 30 variables. For a long time, Oxfam broke resilience down into five bite-sized dimensions to try and make this measurement approach more manageable, but as the organisation’s programme learning has evolved (see Oxfam’s new resilience programming guidelines), so too has the measurement approach. Now, resilience in programmes is understood in terms of three capacities: absorptive, adaptive, and transformative. The last of these three has proved especially tricky for us bean counters in the evaluation team.

What is transformative capacity?

Transformative capacity is defined as the capacity to change the systems that create risk, vulnerability and inequality

Transformative capacity is defined as the capacity to change the systems that create risk, vulnerability and inequality. This is about more than just stability or flexibility. It is about long-term transformation of political systems, protection of eco-systems, gender dynamics, values, beliefs and epistemologies – the structures that determine who will face shocks and stresses 10, 20, 30 years into the future and beyond. Oxfam is committed to measuring what’s important, and transformative capacity is clearly important. So, what’s the issue with measuring transformative capacity?

It’s too big…

Oxfam’s large-N Effectiveness Reviews use ‘counterfactual analysis’ to estimate projects’ impact. We compare the situation of project beneficiaries with other non-beneficiaries (the so-called ‘comparison group’), and mop up observable baseline differences with propensity score matching. This methodology relies on collecting data on a large number of units – typically we’ve gathered information at the household level. Meanwhile, transformative capacity often operates at the level of the system – institutions or networks of actors and agents for the community, the district, or even the whole country. But we’d struggle to make valid comparisons at this high a level, as would be too few units to enable comparison. Evaluating systems-level change requires different evaluation approaches…

It’s too small…

Just because transformative capacity relates to systems and long-term change, doesn’t mean it needs to be about big stuff. Intra-household power dynamics, including the interactions between women and men, are likely to be vital drivers in the long-run. Measuring these concepts is no easy task…

It’s too far away…

Perhaps the most fundamental issue with capturing transformative capacity using our measurement framework is that is really stretches the logic of focussing on drivers or characteristics of resilience. For our indicators to be valid, we need to be confident in our assumptions about which drivers or characteristics of resilience will truly determine wellbeing in the future. But if these assumptions are operating over 10, 20, 30 years or more, we may be on shaky ground…

So, what can we do?

If we want to know about systems-level change, then the level of the system is where we should collect and analyse our data 

Firstly, quantitative counterfactual evaluations aren’t going to be enough on their own anymore. If we really want to know about systems-level change, then the level of the system is exactly where we should collect (and analyse) our data. Oxfam has a long history of applying theory-based evaluation designs (see, for example, our work on ‘Process Tracing‘ ), so these all need to come into play. But, let’s not throw the baby out with the bathwater! We want these theory-based evaluation approaches to go alongside our existing quantitative methods, not replace them (which is another blog post in itself). Many important aspects of absorptive, adaptive capacity, and even some elements of transformative can be captured by household surveys, and our quasi-experimental methods remain a valid way to try and estimate projects’ household- or community-specific effects.

Secondly, the devil will remain very much in the detail, for capturing intra-household issues. This is not only about choosing the right questions and wording them carefully, but also finding ways to robustly sample different members of the household for interview.

Finally, let’s find ways to test and revise the assumptions behind our resilience indicators. Oxfam has started doing this already by meta-analysing the existing data from doing five years of Effectiveness Reviews (watch this space). But the key is maintaining and improving links to local expertise, rather than making top-down guesses about how to best interpret quantitative results. This is something we’ve also been trying, by ‘socialising’ initial quantitative findings with local community groups and government representatives while the fieldwork is ongoing. (All made possible by mobile data collection, of course…)

If you’ve struggled with – or better still overcome – similar challenges with measuring resilience as it continues to expand, please let us know or use the comments section below.

Author

Charikleia Poucha