Blue Flower

Laws Of Evaluation

Probably the most cautionary thing to read for any evaluator before beginning their evaluation is Peter Rossi's classic 1987 work on the subject.

Rossi is most famous for his 'metallic laws', a set of rules of thumb relating to evaluation. There's an iron one, a stainless steel one, a brass one and a zinc one. None of them really laws any more than Murphy's Law, Godwin's Law or Michel's Iron Law of Oligarchy, but they are no less useful because of that.

Slightly tongue-in-cheek though all the laws may be, the steel one especially is a necessary caution against wildly overoptimistic hopes in relation to prospective evaluation results, especially when undertaking the kind of economic evaluation (cost-benefit analysis, Social Return on Investment and so on) that tends to result in an '£x of value was created per £x invested in the project/programme' calculation.

(We should make clear that SERC has nothing against these approaches, and some of our members have a great deal of expertise in producing exactly this kind of analysisthere are some circumstances where projects can produce very impressive numbers.  But it is a technique open to misuse by the unscrupulous or inexpert, so any figures derived from it should always be treated to a close examination of the methodology used to reach them.)

The steel 'law' runs as follows:

The Stainless Steel Law: the better designed the impact assessment of a social program, the more likely is the resulting estimate of net impact to be zero.

While almost all the projects we look at do produce a positive return, in terms of creating more value than they consume, the numbers do vary significantly, and Rossi's law should arm any evaluator or evaluation reader with the necessary scepticism to be sure a figure is genuine. 

A very good summary of the reasoning behind this and the other metal laws is available on Gwern.net, which covers sources like progamme failure, such as problem theory, programme theory and implementation failure – a must read in our view.

 

 

 

Process Tracing

Process tracing is a technique for thinking about whether a change has happened because of an intervention, and perhaps even could only have happened because of that intervention.

It is most famous for a) its four levels of tests used to trace the process by which a project has caused change and b) being a bit like the techniques famous literary detectives use, particularly Sherlock Holmes in 'The Adventure of Silver Blaze'.

The four levels of test are as follows:

'Straw in the wind' – a test which affirms that a given hypothesis (say that someone may be involved in a crime because they know the victim) might be relevant, but does not confirm it (e.g. as other suspects might know the victim as well).

'Hoop' – a test which a hypothesis would have to pass for it to be relevant (say that someone involved in a murder would have to have a means, such as a murder weapon, of carrying it out) but which doesn't confirm the hypothesis is correct (as the person might have had the object used as a weapon for entirely innocent reasons).

'Smoking gun' – a test which confirms a hypothesis (it was the butler that did it, the police officer saw him firing the weapon!), but which doesn't entirely eliminate all other possibilities (the pathologist's report found out later that the butler had actually missed and the victim died of a heart attack).

'Doubly decisive' – the namer of the four tests appears to have run out of metaphors here, but basically the final type of test is one which confirms a hypothesis and eliminates all possible other ones at the same time (e.g. the policeman caught the butler firing the gun as the victim was trying to run away from him).

The simplest way to see if these are useful tests to have in your mind when evaluating is probably to skip the slightly complicated formal explanation and just go straight to the story (which is fourteen pages long and reproduced in full starting from page 19 of this PDF.

Note that the illustrative examples to the tests above are made up, rather than drawn from the story, so as not to spoil it if you do want to read it.  If you can solve the case before Holmes does, process tracing is clearly for you!

(For the record, everyone at SERC still thinks it was the butler that did it despite what Holmes says.)

 

 

 

Realist Evaluation

Realist evaluation is an approach all projects or programmes should be aware of.

The standard approach of many funders and policymakers is on asking the non-realistic question "What works?", meaning "What works with all people in all circumstances?"). 

The only accurate answer to that question is "Well it depends...", as no project or programme works with all people in all places at all times. 

Realistic evaluation asks a different question: who did the project or programme work well with, who not, and what were the possible reasons in both cases?

The following one-page guide from Community Matters is the best starting point we've come across for it, and there's also the original paper from 1997 that launched the whole concept. (Don't be put off by the fact that it's not a shiny brand new theory these days – it's still barely known despite being the most fundamental starting point for any evaluation. But if you want something more up-to-date, the ODI guide to it is the most official one.)

 

 

 

The vast majority of projects decide in advance exactly what they're going to achieve, and list these achievements under headings like "outcomes" and "indicators".

This is good thing to do in itself - it's always good to have a plan - and is often a requirement of funding.

But too great a focus on outcomes and indicators can risk missing any unexpected changes your work has led to.  Outcomes and indicators also rarely say much about which of the changes that happened were the most important for the people you worked with.

Most significant change technique addresses this problem by focusing on the qualitative, storytelling aspect of evaluation that is so important for establishing the quality of a project's outcomes, as well as their quantity.

Our experience is that even if it is only used in the most, basic, stripped-down way by simply making sure to ask participants without any prompts what the MSC was for them, any project's learning will benefit from being familiar with it.

If you're interested in the fuller version of the technique, the original (and very accessible) guide to it is available from Capacity4dev among various other places.