Blue Flower

Laws Of Evaluation

Probably the most cautionary thing to read for any evaluator before beginning their evaluation is Peter Rossi's classic 1987 work on the subject.

Rossi is most famous for his 'metallic laws', a set of rules of thumb relating to evaluation. There's an iron one, a stainless steel one, a brass one and a zinc one. None of them really laws any more than Murphy's Law, Godwin's Law or Michel's Iron Law of Oligarchy, but they are no less useful because of that.

Slightly tongue-in-cheek though all the laws may be, the steel one especially is a necessary caution against wildly overoptimistic hopes in relation to prospective evaluation results, especially when undertaking the kind of economic evaluation (cost-benefit analysis, Social Return on Investment and so on) that tends to result in an '£x of value was created per £x invested in the project/programme' calculation.

(We should make clear that SERC has nothing against these approaches, and some of our members have a great deal of expertise in producing exactly this kind of analysisthere are some circumstances where projects can produce very impressive numbers.  But it is a technique open to misuse by the unscrupulous or inexpert, so any figures derived from it should always be treated to a close examination of the methodology used to reach them.)

The steel 'law' runs as follows:

The Stainless Steel Law: the better designed the impact assessment of a social program, the more likely is the resulting estimate of net impact to be zero.

While almost all the projects we look at do produce a positive return, in terms of creating more value than they consume, the numbers do vary significantly, and Rossi's law should arm any evaluator or evaluation reader with the necessary scepticism to be sure a figure is genuine. 

A very good summary of the reasoning behind this and the other metal laws is available on Gwern.net, which covers sources like progamme failure, such as problem theory, programme theory and implementation failure – a must read in our view.