Tackling the evaluation challenge – how do we know if we're effective?
Jennie Richmond Deputy Head, International Programmes
10th Oct 2012
As Oxfam publishes its first batch of Project Effectiveness Reviews, Jennie Richmond, Head of our Programme Performance and Accountability team discusses the evaluation challenge facing NGOs.
A couple of weeks ago, William Savedoff of the Centre for Global Development, blogged on the challenges facing NGOs in trying to step up to the challenge of impact evaluation. "They are everywhere", he claimed, and yet NGOs, who have relatively fewer resources at their disposal for such processes than government or multilateral counterparts, are struggling to know how to tackle "the evaluation challenge". Savedoff gives three examples of
organisations that are taking creative approaches to assessing their own impact: the Inter-American Foundation, GlobalGiving and Oxfam GB.
Today I get to tell you a little more about the approach taken by Oxfam, as we publish the first of a set of 26 project effectiveness reviews undertaken during 2011-12. Effectiveness reviews are used by Oxfam to try and trace its own part in development change processes, using a range
of methods. The projects that were evaluated for impact were randomly selected from a total of 400 or so mature projects. The approach we have taken is explained in detail in a blog by Karl Hughes, Oxfam's Programme Effectiveness Team Lead - Can we demonstrate effectiveness without bankrupting our NGO and/or becoming a randomista? Today Karl provides an update on our progress. As Karl points out, the reviews provide a mixed picture of the impact we are having on
the ground - some clear evidence of strong impact and some very challenging areas for improvement highlighted.
So, the challenge for NGOs, as Savedoff highlights, is determining the appropriate levels of investment and effort to assess the impact of their work. It is a very topical theme in the development sector right now, as we all try to find ways of demonstrating our results to the public, donors and the people we work with, and as we seek to learn more about what works, where and how. It is essential to first generate impartial and robust evidence on how we are doing, in order to take the action needed to adjust our programmes and make sure we are having the best possible impact.
The silver bullet is to ensure that external accountability purposes do not out-weigh the potential to use this evidence for learning and improvement.
We have spent a great deal of time during the last year thinking through how we can balance the dual purposes of accountability and learning through our Effectiveness Reviews. The two objectives are not perfectly complimentary, and sometimes require quite different approaches to the implementation of evaluations. In order to be serious about drawing the learning from the set of reviews we have produced, we have had to modify our original approach and reorient the way we use our people capacity - factoring in far greater chunks of time
for internal communications, working closely with local teams to reflect on the findings from the reports, investing energy in working with management to ensure that clear plans for follow-up are articulated and then implemented. This has meant that, at this juncture, we have had to slightly scale-down our original ambition of the numbers of studies we are able to deliver (we have adapted plans for 2012-13 from 42 to 30) in order to ensure that we are fully using the rich material in the evaluations and maximising the improvements we can make to how we implement our projects.
Worth noting, then, that even a large NGO like Oxfam has serious limits to the capacity we have available for impact evaluation. The silver bullet is to get an appropriate balance of investment versus delivery, and to ensure that external accountability purposes do not out-weigh the potential to use this evidence for learning and improvement.
So, in that spirit, we are really looking forward to hearing the views of collaborators and critics on our approach, and hope to learn how to improve it further through this discussion.
Oxfam's Effectiveness Reviews
Oxfam's work on Monitoring, Evaluation, Accountability and Learning