Katrina Barnes introduces a new analysis that brings together over 100 impact evaluations of Oxfam projects between 2011 and 2021 – and sets out how we are reimagining the way we define and measure “impact” to better reflect the priorities of people we work with.
In 2011, Oxfam Great Britain began conducting rigorous impact evaluations on a random sample of our longer-term projects and making the results public. These evaluations – and the decision to openly publish their findings – were, at the time, viewed as quite a novel approach.
In what we hope is a move towards even greater transparency, this week we release a new report, Oxfam’s Impact: A Decade of Insight, that shares findings from a systematic review of more than 100 evaluations between 2011 and 2021.
As well as looking at our overall impact, the report focuses on the lessons learned from 67 evaluations that can inform our future work in three key areas: tackling extreme vulnerability (ie supporting people in “fragile contexts”, affected by armed conflict, violence, disaster and/or chronic crisis), valuing women’s work, and climate justice.*
‘We will be looking at fundamental changes to the way we evaluate: ensuring communities, partners and teams we work with play a much bigger role in setting the evaluation agenda.’
The headline insights? We found that 82% of programmes had a positive impact in the lives of all or at least some of the people we worked with. That includes 100% of projects in urban and 75% in rural settings.
In this blog, we look at key findings and insights in three areas Oxfam GB will focus on in future – and also, crucially how we will be changing the way we evaluate to better guide our work.
1. Tackling extreme vulnerability
By 2030, an estimated 86% of people facing extreme poverty will live in the most fragile places on earth, affected by armed conflict, violence, disaster and/or chronic crisis. Oxfam evaluated projects in such contexts across Asia, Africa, and the Middle East that supported people through:
- increasing agricultural sustainability and productivity;
- improving livelihood practices and market systems;
- building, rehabilitating and managing infrastructure (such as water access points or land);
- distributing cash or food/hygiene items; and
- advocating to protect human rights in times of crisis, both nationally and internationally.
Most projects had a positive impact: 65% of projects led to increased household income and wealth – surprisingly higher than similar projects in less fragile contexts. And 70% of projects working towards community resilience led to positive impact.
We found that projects had the greatest impact when they supported community groups, or brought people together. These did better than projects that focused solely on building skills or creating awareness, which did not consistently lead to a positive impact on people’s livelihoods or their resilience. This may reflect the environment in which
extreme vulnerability occurs, where political turmoil or repeated conflict, rather than a lack of knowledge or skills, inhibit people’s ability to make changes. In some cases, positive outcomes were subsequently undone by re-emerging conflict, climatic disasters, community evictions or displacement. These findings reinforce the need to work on addressing longer-term, structural and political causes of instability.
2. Valuing women’s work
The unpaid and informal work of millions of women is largely unobserved and hugely undervalued. This injustice keeps women at the margins of economies and societies and tackling it will be a central focus for Oxfam GB in the coming years.
Projects we evaluated relevant to this future priority looked to support women through:
- strengthening the skills and capacities of women, in relation to economic empowerment, such as agriculture or business skills;
- bringing women together in groups;
- improving women’s market access;
- increasing access to credit;
- supporting women’s leadership for change; and
- raising awareness on women’s rights, social norms, family planning, early marriage and preventing gender-based violence.
The report reveals 80% of projects addressing broad social and systemic barriers to women’s empowerment had a positive impact, with work to build coalitions of women particularly successful.
We saw particularly strong impact on women’s own beliefs and attitudes, as projects supported them to join community groups, take leadership positions, and raise community awareness about care work or their rights around land ownership.
But projects focusing on women’s economic status did less well: only 45% increased household income. It seems improvements in, for example, skills did not translate into income because of social and systemic barriers. Social expectations affect all aspects of women’s lives. Whilst all projects targeted women, and often included specifically tailored activities, project activities were not always sufficient to overcome these patriarchal barriers or address women’s specific needs. This key finding suggests a need to better understand and address gendered social norms within each specific context we work in.
Marginalised and minority groups also seem to be missing out – with findings showing they are sometimes overlooked due to inequalities in voice and agency.
So going forward our focus will shift: as well as supporting women into paid work, if that’s what they choose, we will work to change the broader system, targeting attitudes and narratives about women’s economic value, and how governments value and measure unpaid care in their economies. We will also focus on those in the most precarious situations, such as migrant workers and women in the informal economy.
3. Climate justice
Tackling the climate emergency is going to be a central plank of Oxfam GB’s future work. The projects analysed focused on supporting communities in Africa and Asia to become more resilient to climate change through:
- strengthening and diversifying agricultural practices so they can be sustained in the face of climate shocks;
- awareness-raising and taking action on climate change adaptation;
- developing and raising awareness of community-based early warning systems;
- bringing community members together for collective action; and
- influencing policymakers.
These projects had mixed results. 85% of those which sought to build resilience to climate shocks did so. Interestingly, the small number of projects targeting policy change all had a positive impact, but we saw less impact from projects seeking to raise awareness of climate crisis with communities – possibly because communities are already very conscious of the crisis and have often already developed their own adaptation strategies.
One approach to climate justice is to support diversified and climate-resilient livelihoods. However, like the projects under Valuing Women’s Work, we found that while
our livelihoods work led to increased skills – and in half of the projects led to diversified incomes or greater production – it did not consistently increase household income.
So in future, Oxfam will focus on connecting policy makers with activists, community leaders and those most affected by the climate crisis; on making sure that polluters take responsibility for their actions; that climate financing is available; and that the people most impacted co-create just and transformative solutions. We will no longer focus on increasing incomes at a household level: instead we will work with partners and communities on systemic solutions, pushing governments and corporations in the Global North to act and take responsibility, while ensuring communities in climate-vulnerable regions lead just and transformative solutions.
Changing how we work must include changing how we evaluate
Defining success and measuring impact has historically been controlled by organisations based in the Global North. So we will be looking at fundamental changes to the way we evaluate: ensuring those engaging in the work, partners and teams we work with play a much bigger role in setting the evaluation agenda. In this new approach, diverse partners – such as women’s rights organisations, academia, businesses and grassroots organisations – will be in the drivers’ seat on the development of evaluation questions, as part of the analysis and in owning and how we use evaluation findings.
A second fundamental shift will be looking much more at systems change rather than just individual projects. Assessing standalone projects by looking at change at an individual or household level fails to capture systemic change, so our approach to learning will look much more for signs that broader systems are changing – across portfolios of work. We will also no longer randomly select projects for scrutiny: instead, we will prioritise assessing innovative or strategic priority areas of work and contexts as we put more emphasis on learning.
Our future? Holistic, networked – and led by the people we work with
Publishing these findings is part of our accountability to our supporters and the people we work with around the world, but we also hope these lessons will benefit the wider development sector.
This report supports a call for holistic approaches that shift entrenched gender norms, and the power of networks and coalitions to drive change at a local and national level. It is also a reminder that impact is highly nuanced, complex (see our case study of some work in Iraq), not necessarily linear, takes time and must be built to last: investing in longer-term partnerships with local organisations and communities is going to be the only way Oxfam and organisations like us really drive profound and systemic change.
Author
Read the report: Oxfam’s Impact: A Decade of Insight
Find out more: Want more on Oxfam’s future direction? Read our CEO’s blog written to mark our 80th anniversary: “As Oxfam turns 80, here are three big ideas that I think will shape its future…”
*The impact report does not cover rapid humanitarian responses, which we assess using different approaches, and focuses on work delivered all over the world by Oxfam’s British affiliate, which is part of a global confederation).