Marta Arranz introduces the challenging topic of measuring indirect beneficiaries as part of Oxfam’s efforts to better measure influencing work.
As someone who works on M&E of influencing, I am interested in how programmes and campaigns can estimate who and how many people they actually benefit, particularly those who benefit indirectly, without being directly engaged with project activities.Don’t miss this really great Dilbert cartoon, does it ever feel like this?
Why are we doing this?
Between us, in a previous (non-Oxfam) role I was part of a well intended (and unsuccessful) group of M&E folks lobbying senior leadership to lighten the corporate reporting requirements so programmes didn’t have to annually “guesstimate” the number of indirect beneficiaries they had reached. In my opinion, the exercise was very resource intensive, mainly served ‘upwards’ accountability purposes and didn’t give much in return to programmes. With limited monitoring and evaluation (M&E) resources, I’d rather see M&E capacity producing good quality evidence for learning and improvement rather than producing estimates that risked becoming vanity metrics.
Then I arrived at Oxfam, and in the first week of my new job I was asked to develop an approach to better measuring indirect beneficiaries, as part of Oxfam’s Global Output Reporting process. Karma or irony?
However, after a few good conversations, I am now fully convinced that it’s worth investing in efforts to measure indirect beneficiaries. The drive for developing a measurement approach is much more focused on programme quality than on organisational accountability. By improving the measurement of indirect beneficiaries at different stages of the results chain, we’ll be better able to validate or adjust our plans as we test our theory of change and the assumptions we make during the design. Measuring indirect beneficiaries therefore becomes a means of retaining a focus on impact (changes in peoples’ lives) and on impact “at scale”.
The challenges
Influencing: Long-term sustainable solutions to poverty and injustice demand that we address its root causes and initiate changes in power relations, social norms, structures and systems. The new generation of programmes acknowledge the complexity of the context in which they are operating and seek systemic change at scale, often through more deliberate use of influencing approaches. When Oxfam serves as a convenor, broker or catalyst (rather than direct service provider), drawing out and evidencing the causal pathway between Oxfam’s influencing contributions and (intended) benefit to beneficiaries is more challenging, but as important and necessary.
The term “beneficiaries” is contested – Scott Chaplowe explores this issue well on his blog and there have been lively debates on the pelican group. I agree, it has a post-colonial connotation that is at odds with a human rights based development approach, but I find the alternatives equally unsatisfying for different reasons. Clients, users, citizens, participants, change agents… The issue for me is not as much the term we use to refer to those benefiting from the intervention, as what is the role they (as agents) play in that change process – but that is a topic for another day!
Output, outcome or impact level indicator? In Oxfam’s Global Output Report, the indicators “# of direct and indirect beneficiaries” are output level metrics that programmes report against on an annual basis. This exercise historically focused on ‘direct beneficiaries’, where output level benefit is assumed by direct reception of goods, services or skills. To understand who is indirectly benefitting from the intervention, however, we need to look at a higher level (outcome or impact). How do we go beyond output level measurement in an output reporting framework?
Level of rigour, level of investment – We need an approach that can be integrated in regular monitoring and evaluation frameworks and adopted across a range of programmes. Highly complex methodological proposals that require advanced technical skills won’t do. What is the right balance between level of rigour and the level of investment? Is there necessarily a trade-off between technical complexity of method and quality of output? (Perhaps not)
What we are exploring
We are testing a theory-based approach that aims to identify people (or groups of people) that the intervention intends to benefit at the different stages of the intervention logic. To do this, we follow the programme’s theory of change and for each step in the causal/results chain, the channels through which people will be reached and benefitted (still wrestling with definitional issues). In influencing, very often the direct recipient of services or goods and/or those participating in the activities are not the ultimate beneficiaries, but enablers or intermediate agents. They enable a transfer resulting in change further along the results chain – affecting those who ultimately want to benefit. Programmes will need to identify what scaling up strategy they are using and we are exploring a typology based on Elliott, Hitchens and Nippard’s (2014) Adapt, Adopt, Replicate, Respond model. We also want the approach to include some practical guidance on measurement options and example of sampling strategies that can be used by programmes to develop their own.
Interested in more? Watch this space, there is more coming soon. Please comment in the section below and share your thoughts and reactions. And get in touch! I’ll be in Atlanta for Evaluation 2016 and I look forward to meeting with others interested on this challenge.
Author
