In the latest for our Real Geek series, Ruth Gripper from NPC delves into the latest innovations for measurement and evaluation.
If I asked you what were the most exciting developments in measurement and evaluation today—things that are going to shape practice in the years to come—could you pick a list?
Exploring innovations in measurement and evaluation
Call it brave or foolhardy, that’s what NPC, with support from partners including Oxfam GB and DFID, sought to develop. Here is our list of eight, and why we think they’re important. NPC works with all kinds of (predominantly domestic) charities to help them improve their effectiveness; as such, we are passionate advocates for the role of evaluation in helping charities to learn and improve. Not all organisations have in-house expertise or the resource to keep track of new developments in other settings, so we wanted to highlight opportunities, to inspire others, and to help them consider the implications for their own work.
We spoke to experts from around the world—people like David Ameyaw, John Gargani, and Jo Puri—to get their views on the latest trends and what they meant for the social sector. Ultimately though the choice was ours.
Half of the eight innovations we highlight relate to technical developments and advances in the way we generate, collect, share and curate data. These developments are allowing us to gather data in remote locations, identify trends and patterns using big data, and link datasets to gain a richer picture of what is going on. What really excites me though is the shifting thinking in what evaluation is for and how it should be practiced, so I want to pull out three trends highlighted in the report.
User voice: who’s in the driving seat?
Historically evaluation has been seen as something done primarily for the benefit of donors—a tedious but necessary part of accountability to funders. We have seen this at NPC in our work with charities. Increasingly, however, the language of accountability is shifting—with accountability to the intended beneficiaries of an intervention being at least as important as accountability to the individual or organisation who paid for it. This is evident at Oxfam, for example in the work on capturing informal feedback at Za’atari refugee camp. More widely we found user feedback being taken more seriously as a critical component of evaluation data, and teams looking at the whole evaluation cycle with fresh eyes—asking how the process itself can benefit participants, rather than simply ‘extract’ their views.
Theory-based evaluation: because life is complicated
Life is complicated, and so too are the questions we seek to answer. What works within a complex system? How can we even tell? Why does it work, and will it work elsewhere? All of these are important questions, but figuring out the answer isn’t easy. What we have seen, though, is a greater willingness to tackle them—based on the recognition that we need to address them if we are going to make progress. With this comes an acceptance that in some circumstances learning and partial knowledge, rather than proof, might be the best we can hope for.
Impact management: I want insight, and I want it now!
Gone are the days when a gold standard post hoc evaluation delivered three years after the programme finished was good enough. Fed up of being told three years too late that something isn’t working—and influenced perhaps by Silicon Valley’s ‘fail fast’ and ‘minimum viable product’ approach to software development—programme managers are today more interested in learning in real-time (or close to), and course-correction during the life of a programme.
So, over to you
Of course, we recognise that using big data for measurement and evaluation is not a realistic short-term aim for most organisations. And it’s worth noting that although some of the innovations we have chosen are new ideas and technologies, others are renascent, or new technologies being applied to old ideas and problems. But in our view, they offer opportunities to expand the measurement and evaluation toolkit—improving our understanding of social interventions and thereby increasing our effectiveness, something that is ever more critical in difficult times.
So do take a look at the paper, share with your peers, and let us know what you think. We’ve made our selection and it’s out there in the world—what would yours be?
The full list of NPC’s top picks with accompanying case studies and our thoughts on what next can be found in Global innovations in measurement and evaluation.
Author
