When asked by the editor for a title for my CIPD book on evaluation I suggested – ‘The Last Word on Evaluation’ – not out of arrogance but because the mountain of literature and endless debate around the subject had never reached a conclusion.  Evaluation is essentially a simple subject made unnecessarily complicated by vested interests. Probably the biggest and most influential vested interest is the American Society for Training and Development (ASTD) who, whenever they tried to offer their members advice on this thorny subject, managed to get it so completely and so obviously wrong.

From the 1970’s the ASTD regarded Kirkpatrick’s 4-Levels as their standard:

1. Reaction to training (happy or smile sheets)
2. Testing learning
3. Applying learning in the workplace
4. Business impact

Let’s try this out by thinking of one salesperson going on a sales training course at a cost of $1000. When they return you take them through the 4 levels, starting with how they felt about the course…  – WRONG.  That way you end up at Level 4 with no way of assessing impact.

Let’s start again. Before you send them on the course you ask them how much they sell now ($1.2 million) and what the profit margin is (10%)?  Then afterwards, when you get to Level 4, you can ask again how much they are selling and you have a basis for gauging impact. The critical, pre-learning questions form the BASELINE and produce the simplest, most obvious and effective evaluation model with just 2 levels:

1. Baseline evidence
2. Business impact measured against the baseline data

- but you will quickly find in practice that this 1st, Baseline level will usually suffice because it is this one that adds all the value in what is now an enhanced, learning process. Individual learning starts when each individual knows their own Baseline.

The 4-Levels never captured this so, just as corporations started asking what the financial return (ROI) was on their training investment, Kirkpatrick’s obsolescence was becoming apparent.  So the ASTD decided to back a different horse in the 1990’s, but could not admit Kirkpatrick* was wrong, so adopted Jack Phillips’ model which just added another level – level 5 for the ROI calculation – which also made it look like an innovation – WRONG again.

Let’s go back to the beginning. You ask the sales trainee the BASELINE questions. You do the training and then at Level 4 you get an answer as to how much sales have increased (1%) and you know the cost ($1000) so you can do the net ROI calculation immediately – it’s 20%. There is no need for any level 5 – it doesn’t add anything and anyone with a calculator can work it out.  Under Phillips though you have to spend even more time and money making the numbers up and converting to $ because he doesn’t establish the relevant $ sign at the beginning.  Level 5 was always a figment of the ASTD’s collective imagination and that’s why the figures Phillips produced never convinced anyone who had a business head on their shoulders.

So the ASTD decided it needed more credibility and drafted in a labour economist  – Laurie Bassi – in a vain attempt to garner a more academic and quasi-scientific level of respectability.  Laurie tried to show the business impact from the billions of $’s ASTD members were spending on training.  Laurie did not know any more about evaluation than her predecessors though and, as an academic, used the only analytical tools she had, regression analysis to produce correlations, using retrospective data.  I guess she also did not have the benefit of first-hand experience of what it feels like to work in a training department in a large corporation; where trainers are often under pressure from managers to produce all sorts of stupid, knee-jerk programmes to cover up their deficiencies in people management and development.  Laurie is now heading up efforts to establish international standards by the other large American professional body, SHRM ( the Society for HRM) and the ASTD still makes it mandatory for anyone wanting to be their ‘partner’ to attend a Jack Phillips training programme.

Forty years should have been long enough for the Americans to get it right but so far, despite being an SBO (statement of the bleeding obvious), no ‘expert’ in the US has ever fully understood or acknowledged the crucial importance of the BASELINE level in evaluation and learning; whether it be sales training, management development, leadership or OD.  Starting with a Baseline makes learning evidence-based, it is a perfect application of evidence-based management.  Now, if you are looking for an indication of which corporations waste the most money on ‘training’ just ask them which model they use and where they start from.

For personal development linked to this topic visit the Consummate Professional Series or attend a workshop.

* The Kirkpatricks still refer to their model as ‘the 4-levels’, although son Jim now tries to make up for the absence of ROI with something called ROE (return on expectations).  This is intended as a sort of ROI-lite, except ‘expectations’ are not necessarily couched in $ terms and no ‘return’ can actually be calculated because ROE is not a mathematical or financial formula.  In short, ROE can mean whatever you want it to mean.