Charles is an IDP living in an abandoned medical centre with 21 other families in the DRC. Credit: Suzi O'Keefe/Oxfam

Leaving no one behind in our evaluation practice

Methodology, Real Geek Leave a Comment

Stephen Porter reflects on the importance of understanding why people are left behind in development practice and acknowledging what goes unseen and unsaid in evaluations.

Charles is an IDP living in an abandoned medical centre with 21 other families in the DRC. Credit: Suzi O'Keefe/Oxfam

Charles is an IDP living in an abandoned medical centre with 21 other families in the DRC. Credit: Suzi O’Keefe/Oxfam

Leave no one behind is a call for action within the Sustainable Development Goals. The Goals are meant to be met for all people, especially groups who are marginalized and vulnerable. In the Sustainable Development Goals Report in 2016 it was recognized that data is sparse on populations who are left behind. Often getting more data is treated as a technical affair; we just need more resources and expertise.

In international development the voices and perspectives of the people we are meant to serve remain invisible if we have no data about them. People continue to be left behind because they remain invisible at country, regional and international levels. This is a failure of valuing of voice and perspective rather than a technocratic oversight.

I am an evaluator in Oxfam. A major part of my job is to provide systematic, accessible, and objective evidence about the value of the work we conduct, whether people live equitably and free from the injustice of poverty.  When conducting evaluations we often seek to apply tools to improve the quality and use of evaluations in an ethical manner. We work to well-developed evaluation standards (such as the JCSEE Program Evaluation Standards and the OECD Quality Standards for Development Evaluation), but even with these we do not tackle the issue of invisibility head-on.

While I recognized issues of visibility in evaluation practice my thoughts remained amorphous until I was invited to a panel at DC student’s consortium on evaluation entitled ‘What is Evidence? Towards Social Equity in Evaluation and Policy,’ this triggered me to start to get ideas out in a more coherent form. In thinking about bottlenecks in my own practice, I reflected upon literature, specifically novels, that had shaped my own journey in understanding injustice and found that the nature of invisibility was often explored.

The three novels that I was drawn to relate to three countries that I have called home, namely:

  • South Africa – J.M. Coetzee’s Waiting for the Barbarians;
  • the United Kingdom – George Orwell’s 1984;
  • the United States – Ralph Ellison’s Invisible Man.

In all of these novels the invisibility, injustice and abuse is extreme. It is instructive to have an extreme to hold your own practice against as a warning and to help understand consequences of failures in practice.

Waiting for the Barbarians

An allegory for apartheid South Africa, Waiting for the Barbarians focuses on a nameless male character working through his own complicity in cruel acts and injustice. In this book the invisibility is of ‘the other’. The Barbarians remain on the periphery of the story rarely encountered except in situations of subjugation, snatched sightings and hearsay. In evaluation processes there is also often an invisible other. A group on the periphery that is known, but does not participate as stakeholder or is ignored in data collection and analysis.

A lament in evaluation reports or processes is often a lack of participation. The remedy being to be simply ‘more participatory’. The issue in the evaluation might not be of neglect of participation, but of power that prevents participation. Some people are invisible in evaluations because they are a perceived threat, neglected or they compete for resources. There is a need to recognize the other is invisible and left behind not just because of defective process, but because of choices involving power that need effort to overcome. 

There is a need to recognize the other is invisible and left behind not just because of defective process, but because of choices involving power that need effort to overcome.

1984

The novel 1984, a warning about authoritarianism, has many forms of invisibility about the individual and of memory. The one that caught my eye is double think, the acceptance of competing contradictory beliefs. War is peace, ignorance is strength, and freedom is slavery. Double think is a challenge to evaluators to get and maintain perspective. Sometimes reading evaluation reports conducted in authoritarian states truth can become invisible. Ethnic tensions that simmer beneath the surface are not mentioned, statistics are re-purposed to remove discussions of inequality and the line of the ruling party is represented without question. People are left behind in development because the state wills it and covers it up. Sometimes these issues arise because it is necessary to do evaluations in imperfect settings and to tell the truth in a way that it can be heard, sometimes it could be evidence of double think, the turning of repression, in an evaluation report, into success.

Invisible Man

Invisible Man is a story of racism in America in which people simply refuse to see a man because of the color of his skin renders him invisible. This form of invisibility comes into play when, as an evaluator, you work in a context and glimpse something that you cannot fully appreciate or you cannot even see a phenomenon. A social asset in a community that enables resilience is ignored, an organizational practice that puts children in harm’s way is misrepresented. People are left behind because our frames of reference mean they cannot be seen, even when they are in front of you. This seems to require more than hiring evaluators who know a context. Not being able to see something is also rooted underlying prejudices, which require personal evolution to confront.

In the agreed agenda for the Sustainable Development Goals, country-led evaluation is to play a role in the follow-up and review processes. We need evaluation practice, in Oxfam and international development that contributes to organizational impact and sustainability in a manner that helps us not to leave people behind. We need to recognize that the systems we have built and standards we apply for evaluation practice do not always sufficiently value the voice and perspectives of populations left behind. While they remain invisible we cannot effectively overcome injustice.

Although this is a broader issue for the international development system, evaluators have a role to play in confronting and changing practice. Evaluators are after all a group whose job it is to systematically and objectively verify the value of the work we undertake and help those in power, no matter how well-meaning, to see truths about why we are leaving people behind. Understanding how our practice is exercised and encourages invisibility is a starting point for evaluators to understand where we contribute to injustice, perhaps unwittingly.

Author
Stephen Porter

Stephen Porter

Stephen Porter is currently the Director for the Learning, Effectiveness and Accountability Department at Oxfam America. Stephen has a range of experience in development practice, including academic, donor and experience applying a rights-based approach to evaluation. Previously, Stephen was Evaluation Advisor for Market Development at the UK Department for International Development and was the Director of the Center for Learning on Evaluation and Results for Anglophone Africa at the University of Witwatersrand. He holds an MPhil in Public Policy from the University of Cape Town. Stephen has eleven journal and book chapters published on the topic of evaluation systems development.