When asked by the editor for a title for my CIPD book on evaluation I suggested – ‘The Last Word on Evaluation’ – not out of arrogance but because the mountain of literature and endless debate around the subject had never reached a conclusion. Evaluation is essentially a simple subject made unnecessarily complicated by vested interests. Probably the biggest and most influential vested interest is the American Society for Training and Development (ASTD) who, whenever they tried to offer their members advice on this thorny subject, managed to get it so completely and so obviously wrong.
From the 1970’s the ASTD regarded Kirkpatrick’s 4-Levels as their standard:
- Reaction to training (happy or smile sheets)
- Testing learning
- Applying learning in the workplace
- Business impact
Let’s try this out by thinking of one salesperson going on a sales training course at a cost of $1000. When they return you take them through the 4 levels, starting with how they felt about the course… – WRONG. That way you end up at Level 4 with no way of assessing impact.
Let’s start again. Before you send them on the course you ask them how much they sell now ($1.2 million) and what the profit margin is (10%)? Then afterwards, when you get to Level 4, you can ask again how much they are selling and you have a basis for gauging impact. The critical, pre-learning questions form the BASELINE and produce the simplest, most obvious and effective evaluation model with just 2 levels:
- Baseline evidence
- Business impact measured against the baseline data
- but you will quickly find in practice that this 1st, Baseline level will usually suffice because it is this one that adds all the value in what is now an enhanced, learning process. Individual learning starts when each individual knows their own Baseline.
The 4-Levels never captured this so, just as corporations started asking what the financial return (ROI) was on their training investment, Kirkpatrick’s obsolescence was becoming apparent. So the ASTD decided to back a different horse in the 1990’s, but could not admit Kirkpatrick* was wrong, so adopted Jack Phillips’ model which just added another level – level 5 for the ROI calculation – which also made it look like an innovation – WRONG again.
Let’s go back to the beginning. You ask the sales trainee the BASELINE questions. You do the training and then at Level 4 you get an answer as to how much sales have increased (1%) and you know the cost ($1000) so you can do the net ROI calculation immediately – it’s 20%. There is no need for any level 5 – it doesn’t add anything and anyone with a calculator can work it out. Under Phillips though you have to spend even more time and money making the numbers up and converting to $ because he doesn’t establish the relevant $ sign at the beginning. Level 5 was always a figment of the ASTD’s collective imagination and that’s why the figures Phillips produced never convinced anyone who had a business head on their shoulders.
So the ASTD decided it needed more credibility and drafted in a labour economist – Laurie Bassi – in a vain attempt to garner a more academic and quasi-scientific level of respectability. Laurie tried to show the business impact from the billions of $’s ASTD members were spending on training. Laurie did not know any more about evaluation than her predecessors though and, as an academic, used the only analytical tools she had, regression analysis to produce correlations, using retrospective data. I guess she also did not have the benefit of first-hand experience of what it feels like to work in a training department in a large corporation; where trainers are often under pressure from managers to produce all sorts of stupid, knee-jerk programmes to cover up their deficiencies in people management and development. Laurie is now heading up efforts to establish international standards by the other large American professional body, SHRM ( the Society for HRM) and the ASTD still makes it mandatory for anyone wanting to be their ‘partner’ to attend a Jack Phillips training programme.
Forty years should have been long enough for the Americans to get it right but so far, despite being an SBO (statement of the bleeding obvious), no ‘expert’ in the US has ever fully understood or acknowledged the crucial importance of the BASELINE level in evaluation and learning; whether it be sales training, management development, leadership or OD. Starting with a Baseline makes learning evidence-based, it is a perfect application of evidence-based management. Now, if you are looking for an indication of which corporations waste the most money on ‘training’ just ask them which model they use and where they start from.
For personal development linked to this topic visit the Consummate Professional Series or attend a workshop.
* The Kirkpatricks still refer to their model as ‘the 4-levels’, although son Jim now tries to make up for the absence of ROI with something called ROE (return on expectations). This is intended as a sort of ROI-lite, except ‘expectations’ are not necessarily couched in $ terms and no ‘return’ can actually be calculated because ROE is not a mathematical or financial formula. In short, ROE can mean whatever you want it to mean.
Pingback: All you need to know about training evaluation in about 700 words - Evidence-Based HR - Member Blogs - HR Blogs - HR Space from Personnel Today and Xpert HR
Pingback: Evaluating Training – Will We Ever Do It Well? « Next Practices
Hi Paul,
I couldn’t agree more. We now work in an economy where most company spend needs to be justified and reported on. If you don’t understand where you are before training, how will you ever report back the impact.
By understanding the skills and behaviours the organisation requires from its employees, you can quite easily measure the success of your programmes, against the development of capabilities.
A large number of business leaders are complaining about the lack of talent in their industries, so it is vital they retain, attract and develop the best people. Only by continually monitoring the organisational skills required is it possible for succession planning, ensuring the right individuals receive the right training at the relevant time.
I do not think Kirkpatrick’s 4-Levels have to be obsolete, as long as you ask the questions before and after.
my problem with Evaluation and ROIs remain the same. It’s all correlation. How can the argument of correlation vs causation be dealt with?
The Baseline sorts out the causation problem at the only time it can be resolved – at the beginning. Take a leadership programme – the Baseline asks ‘what is the problem’, ‘what’s causing it’ and ‘is this the best solution’? Then you design leadership development. Baseline, evidence-based learning is never a solution looking for a problem. You might also like to take a look at “EBM Lesson 5 Finding the root cause of people problems” or “EBHR Managers know better than to rely on correlations”
Thanks for the question.
best regards
Paul
Hi Paul, thanks for the post. Gathering baseline data is certainly essential – in my opinion particularly if you’re considering changing the training (either content and/or the delivery platform). Without the baseline data it is pretty much impossible to subsequently determine what benefits you have generated as a consequence of the change. If you have data you can consider running pilot projects, amending training in one area, assessing the results, then making informed data based decisions on how/if you roll out those changes for other aspects of training.
I seem to be missing something. A scientist would point out that baseline data won’t tell you anything meaningful unless you are able to control for exraneous variables: we may find that training is correlated with performance gains, but we won’t be able to say confidently if it contributed to them (the point made above). And indeed this is precisely what happens ‘We saw improved sales results but obviously we can’t attribute these to training alone – what with everything else going on’. But a scientist might also point out that it was for this reason that the ‘independent measures design’ was created: identify two groups of sales people who are – as good as you can make them – equal; give one group the training and not the other. Only in this way will you have a quantifiable degree of confidence regarding the causal relationship. But we don’t do this in training. So, so as far as I am aware there has never been any real question regarding how to establish e ROI of training, only lots of interesting ones about why we don’t.
Hi Nick,
I’m not a scientist, I’m a manager and maybe this is one of the differences – EBM is not about proving anything, just providing better evidence for better management decisions – see What is EBM . I’ll be addressing the academic research community’s efforts in EBM in a future post – in the meantime you can see some of my interactions with them (and join up if you wish) and you might also want to read ‘Managers with ODD attitudes‘.
Thanks for the comment
best regards
Paul
Hi Paul,
It’s great to see so many thoughtful contributions on this subject. I’d like to follow up on Nick’s point about having a control. Essentially, for any initiative, we’d ideally like to have evidence that tells us whether the intervention improves performance, makes no difference or has a negative impact (all of which have associated financial impacts).
Where the intervention has a financial and/or opportunity cost, whether it’s a new performance management system or a new learning and development initiative, what reason could there be for not having a pilot group and a control?
There must be something in the air, because I wrote about using control groups just the other day.
Using a control group is straightforward and easy to do, so why do we not see that approach used more often? Is it that people working in L&D are less likely to have a scientific background?
Thanks for kicking off an interesting discussion.
Owen
Hi Owen,
Thank you for your comments. The method I advocate is a personal Baseline for every individual, which obviates the need for a control group. I don’t want to create work so my standard response to control groups (see ‘Appendix 1 – The top 20 most common questions…’ in my book ‘Evaluating the ROI from Learning‘) is:-
“Q.3. Should I design a control group experiment?
A. A control group in training is a group of employees who do not receive the training that everyone else receives. The basic idea is to be able to compare the results both with and without the influence of training. If you have read the answers to 1 and 2 above (strongly advising against Phillips’ idea of separating out the effects of training) you can probably guess that I do not normally recommend the use of control group experiments. The first reason is you cannot normally keep an organisation still while you try to observe the change in one variable. Organisational life does not work like that: we do not have the time and we have to cope with a highly dynamic situation. Variables do not change one at a time, they are constantly changing. The second reason is more practical. While you are running a control group that group are not performing as well as they could with the training. You might be proving a point but don’t expect the CEO to pat you on the back for missing a business improvement opportunity. The third reason though comes back to designing in evaluation. If you do this you do not need a control group.”
best regards
Paul
Hi Paul,
I’m all for taking a pragmatic approach, but I’d query the logic behind your second reason: “While you are running a control group that group are not performing as well as they could with the training”.
There’s an inbuilt assumption here that the training will result in improved performance. Isn’t that what we’re trying to ascertain?
What if the training results in worse performance? What if the new sales technique/performance management approach/incentive scheme has a negative impact on performance? Rolling out to anyone without piloting and comparing to a control could simply ensure that the performance of everyone gets worse.
If we could be certain that the training results in better performance there wouldn’t be a problem. Unfortunately, we can’t be sure of that for many HR initiatives.
I may be wrong, but that seems to be the assumption built into your previous answer.
Hi Owen,
yes you’re right – a built-in assumption of improvement just as a marketing director would do for a marketing campaign or a production director introducing Six Sigma – but also a clear-headed assessment of the probability of success. It would be very odd to spend money on training if there were no assumption of likely improvement wouldn’t it – except that’s precisely what the BBC did. If I was unsure though, but thought the opportunity for value was big enough, I might run a pilot to test these assumptions but I would still not recommend a control group except in very exceptional circumstances.
best regards
Paul
Just a note that “ask the sales trainee” this and that is great, but asking the supervisors of those sales trainees is worth considering. After all, those are probably the people who decide in the end whether the company is getting a return on their investment in training, they need to see results.
Hi Will
Your suggestion is not just worth considering – it’s a must – and why stop with involving supervisors, why not go as high up as possible to gain sponsorship and encouragement. This is where evidence-based learning actually fosters evidence-based management throughout the organisation. Thanks for the comment.
best regards
Paul
Pingback: “It’s all right as long as we are all wrong.” |