What overall impact are Oxfam’s women’s empowerment projects having? Simone Lombardini and Kristen McCollum attempted to answer this question by conducting a meta-analysis of our effectiveness review evaluations. Here Simone shares the key findings.
After over six years conducting impact evaluations of women’s empowerment projects, I am very pleased to share our lessons from a recent meta analysis. ‘Using internal evaluations to measure organisational impact: a meta-analysis of Oxfam’s women’s empowerment projects’ is available for download from the Journal of Development Effectiveness. A slightly earlier free access version can be found on Oxfam Policy and Practice.
If you’ve been following our progress since 2011, you’ll may be familiar with our measurement framework for women’s empowerment, which attempts to capture empowerment characteristics at a personal, relational and environmental level. Last year, we shared a ‘How To’ Guide To Measuring Women’s Empowerment, which I hope will help others build on what we’ve learned.
Our impact evaluations of large-scale development projects, named Effectiveness Reviews, aim to better capture and communicate Oxfam’s impact on women’s empowerment and promote evidence-based learning in this complex area.
With a wealth of 16 impact evaluations of randomly selected Oxfam projects, combined with a consistent measurement framework, and the commitment to publish each study regardless of whether the results are positive or negative, we conducted a meta-analysis with all the effectiveness reviews conducted under the thematic area of women’s empowerment over the last six years.
The results provide evidence that overall, Oxfam’s development projects have a positive and statistically significant effect on women’s empowerment (the estimated overall effect size on the women’s empowerment index is 0.32 standard deviations). The analysis attempted to explain heterogeneity by conducting subgroup analysis by project type, regional implementation and projects’ characteristics, without finding robust evidence. We hope that with additional impact evaluations in the future we will able to further explore the project characteristics associated with higher levels of empowerment.
‘There is evidence that the projects had an overall positive and significant impact in changing women’s opinions about their economic role’There is evidence that the projects had an overall positive and significant impact in changing women’s opinions about their economic role. However, there appears to be no evidence of increasing women’s contribution to household income, nor of overall positive changes in power dynamics within the household. While this is a disappointing finding, many of our staff might say it’s not a hugely surprising one. Oxfam’s current strategy is to promote a more holistic approach to women’s empowerment by working across different dimensions of women’s lives including personal, political, social and economic aspects. We believe this meta-analysis provides a useful contribution in this area.
There is evidence that the evaluated projects increased women’s participation in community groups and women’s influencing in the community. This suggests that there were positive changes in the power dynamics between women and groups, and the communities.
‘the meta-analysis picked up an overall negative impact on the indicator measuring women’s experience of violence’However, the meta-analysis picked up an overall negative impact on the indicator measuring women’s experience of violence (which did not come through in the individual impact evaluations). It’s unclear to us if this is explained by measurement issues – for example, with more empowered and interconnected women being more willing to report violence. It may also show a negative effect of efforts to change power dynamics – that women involved in changing power dynamics also experience an increased risk of violence. Regardless of the answer, these results strengthen the argument that all programmes should be investing in more accurately detecting and preventing any possible unintended negative impacts on women’s experience of violence.
This meta-analysis allows us to ask and answer questions that single studies cannot. Consistent attention to women’s empowerment and a strong and consistent internal evaluation system have been key to enabling this.
In the coming weeks we will be publishing blogs on the meta-analyses for other thematic areas: resilience, livelihoods, and good governance. To read these watch this space and sign up for our Real Geek email alerts.
Author
