Tag Archives: training evaluation

The last word on Learning Needs Analysis (LNA)

Posted on by
Share

Adopting an evidence-based approach to learning and development is not new – far from it.  Trainers just have to do what they have always been taught to do – learning needs analysis (LNA) and evaluation.  These two crucial steps are inseparable – two sides of the same coin – the yin and yang of individual and organisational learning.  The purpose of the original TNA (training needs analysis) was to provide the evidence-base to justify spending money on training: the purpose of evaluation was to ensure the money was invested wisely.  In other words LNA is an individual needs analysis as part of a business investment appraisal process.

As we saw earlier though trainers have never had a workable model to deal with the evaluation aspects of evidence gathering.  Now we have to explore whether there is any guarantee either that they will bother to undertake a proper, evidence-based, needs analysis.  Visit any training provider (me included) offering training courses or workshops, or any in-house L&D department offering its own menu of programmes, and you will witness ‘learning solutions’ (sic) being offered without any analysis of the learning needs of those who might participate.  Any professional who supports this will have to admit that they are breaking all their own rules.  It should be outlawed and, if I am as guilty as anyone else, what should I do about it?

Well, first let me declare that I fully accept the crucial importance of LNA in principle even if I don’t always do it in practice.  That might sound hypocritical but how many ‘learning professionals’ would at least be willing to admit the same?  Second, I am bringing it to your attention, and anyone else who cares to read this, in an attempt to increase the incidence of proper LNA.  Third, I am prepared to make myself very unpopular by exposing bad practice and the stupid organisational behaviour it produces.  Fourth, I have at least offered part of the solution already with a very simple Baseline Model and fifth, I am now going to complete the set with the last word on what learning needs analysis really means. There is probably a sixth and a seventh but what have you been doing about it?

Let’s re-visit what we were taught about the old-fashioned TNA. If you want to train someone in customer service, say in the hotel trade, the business has to decide what standard of service it wants to provide (e.g. every customer should be greeted with a friendly smile and asked ‘can I help you?’).  This is OK for basic training and it would be easy to do a spot check (evaluate) that the training is being applied. If the hotel wants to differentiate itself from its competitors though, by raising its own standards, the trainer needs to gather more evidence before designing any training.

  • ‘What do higher standards look like – would it be speedier response times at the desk?
  • ‘Are we measuring that already?’
  • ‘So what if we are speedier, does that produce more value in $’s?’

So far so professional but move away from basic training and soon the rules of the game, the principles that mark out the professional, are jettisoned remarkably quickly.  Yet the very reason TNA started to become known as LNA was because we all eventually realised that training was just input (e.g. sending people on generic management modules) and learning had to be applied if it was to produce a beneficial output (i.e. more satisfied, paying customers).  Somewhere along this line though LNA forgot its TNA roots and became a much broader concept; to the extent that many ‘learning professionals’ today have started suggesting that no LNA is needed because ‘all learning is good’.

This is not a principle I have ever been taught and my 30 years of experience clearly demonstrates this is patently untrue.  If you subscribe to this notion why do we still pay for training these days when so much learning can be had  for free?  Conversely, if all learning is good, go and ask your board for a 1000% increase in the training budget and see what response you get.  It doesn’t stand up well does it?  Yet this fuzzy logic has been exploited by the technology-obsessed, ‘e-learning’, social media advocates who do not know how to justify what they are doing.  Often business needs are forgotten altogether and there is no attempt at Baseline evaluation.  In effect, the ‘new’ training providers have been taking several steps backwards for years, moving further and further away from the fundamental tenets of learning whilst trying to pretend they are transforming the way we learn (why do you think they never chose to call it e-training?).

There is a simple solution.  Having admitted my own guilt, the first thing I try to do is make up for the lack of individual, learning needs analyses whenever I can.  If you have not already read What is EBM or The Case for EB-HR then I would advise you to do so now because that is exactly what these pages were designed to do; to make sure you are in the right place.

If we should ever meet in a classroom and you are not sure why you are there I would ask whether you want to come back when you know and remind you that you need to come back with your own Baseline, because you can only start from where you are, regardless of where the rest of the class might be.  If you show no interest in sticking to these rules I would recommend that you do not stay, not just because you are unlikely to learn very much but because I am passionate about learning and I want you to be passionate as well.  By the way, there is no such thing as a ‘generic’ or ‘group’ LNA – we can only learn as individuals – even though we are all under pressure to conform to group behaviour.

One of the key advantages of analysing an individual’s learning needs is that we want to avoid upsetting, or getting in the way of, those who are genuinely there to learn.  Maybe that is why very few trainers are inclined or even allowed to perform a proper LNA – it might be too embarrassing to realise just how inappropriate and ineffective a lot of training is; whether it is in the classroom or online.

The last word? LNA is not an option – it’s a professional duty.

For personal development linked to this topic visit the Consummate Professional Series

Share

All you need to know about training evaluation in about 700 words

Posted on by
Share

When asked by the editor for a title for my CIPD book on evaluation I suggested – ‘The Last Word on Evaluation’ – not out of arrogance but because the mountain of literature and endless debate around the subject had never reached a conclusion.  Evaluation is essentially a simple subject made unnecessarily complicated by vested interests. Probably the biggest and most influential vested interest is the American Society for Training and Development (ASTD) who, whenever they tried to offer their members advice on this thorny subject, managed to get it so completely and so obviously wrong.

From the 1970’s the ASTD regarded Kirkpatrick’s 4-Levels as their standard:

  1. Reaction to training (happy or smile sheets)
  2. Testing learning
  3. Applying learning in the workplace
  4. Business impact

Let’s try this out by thinking of one salesperson going on a sales training course at a cost of $1000. When they return you take them through the 4 levels, starting with how they felt about the course…  – WRONG.  That way you end up at Level 4 with no way of assessing impact.

Let’s start again. Before you send them on the course you ask them how much they sell now ($1.2 million) and what the profit margin is (10%)?  Then afterwards, when you get to Level 4, you can ask again how much they are selling and you have a basis for gauging impact. The critical, pre-learning questions form the BASELINE and produce the simplest, most obvious and effective evaluation model with just 2 levels:

  1. Baseline evidence
  2. Business impact measured against the baseline data

- but you will quickly find in practice that this 1st, Baseline level will usually suffice because it is this one that adds all the value in what is now an enhanced, learning process. Individual learning starts when each individual knows their own Baseline.

The 4-Levels never captured this so, just as corporations started asking what the financial return (ROI) was on their training investment, Kirkpatrick’s obsolescence was becoming apparent.  So the ASTD decided to back a different horse in the 1990’s, but could not admit Kirkpatrick* was wrong, so adopted Jack Phillips’ model which just added another level – level 5 for the ROI calculation – which also made it look like an innovation – WRONG again.

Let’s go back to the beginning. You ask the sales trainee the BASELINE questions. You do the training and then at Level 4 you get an answer as to how much sales have increased (1%) and you know the cost ($1000) so you can do the net ROI calculation immediately – it’s 20%. There is no need for any level 5 – it doesn’t add anything and anyone with a calculator can work it out.  Under Phillips though you have to spend even more time and money making the numbers up and converting to $ because he doesn’t establish the relevant $ sign at the beginning.  Level 5 was always a figment of the ASTD’s collective imagination and that’s why the figures Phillips produced never convinced anyone who had a business head on their shoulders.

So the ASTD decided it needed more credibility and drafted in a labour economist  – Laurie Bassi – in a vain attempt to garner a more academic and quasi-scientific level of respectability.  Laurie tried to show the business impact from the billions of $’s ASTD members were spending on training.  Laurie did not know any more about evaluation than her predecessors though and, as an academic, used the only analytical tools she had, regression analysis to produce correlations, using retrospective data.  I guess she also did not have the benefit of first-hand experience of what it feels like to work in a training department in a large corporation; where trainers are often under pressure from managers to produce all sorts of stupid, knee-jerk programmes to cover up their deficiencies in people management and development.  Laurie is now heading up efforts to establish international standards by the other large American professional body, SHRM ( the Society for HRM) and the ASTD still makes it mandatory for anyone wanting to be their ‘partner’ to attend a Jack Phillips training programme.

Forty years should have been long enough for the Americans to get it right but so far, despite being an SBO (statement of the bleeding obvious), no ‘expert’ in the US has ever fully understood or acknowledged the crucial importance of the BASELINE level in evaluation and learning; whether it be sales training, management development, leadership or OD.  Starting with a Baseline makes learning evidence-based, it is a perfect application of evidence-based management.  Now, if you are looking for an indication of which corporations waste the most money on ‘training’ just ask them which model they use and where they start from.

For personal development linked to this topic visit the Consummate Professional Series or attend a workshop.

* The Kirkpatricks still refer to their model as ‘the 4-levels’, although son Jim now tries to make up for the absence of ROI with something called ROE (return on expectations).  This is intended as a sort of ROI-lite, except ‘expectations’ are not necessarily couched in $ terms and no ‘return’ can actually be calculated because ROE is not a mathematical or financial formula.  In short, ROE can mean whatever you want it to mean.


Share

Evaluation of learning – 3 simple rules of the game

Posted on by
Share

When is a profession not a profession?  When it fails to follow its own professional advice.  Imagine a surgeon saying – ‘Yes, I know the operating theatre should be a sterile environment but hey, what are a few bacteria among friends?’  Well that is exactly the sort of attitude adopted by the vast majority of training ‘professionals’ towards the subject of evaluation. They know their own professional standards demand they evaluate – as a very specific requirement in the learning needs, design, delivery, and evaluation cycle – but studiously avoid doing so.

So let’s try re-phrasing this.  If we said that trainers do not collect any evidence that their methods work we might expect them to be operating in very ‘dirty departments’.  Of course, if you’re teaching a bricklayer to lay bricks this does not matter too much because you at least have the evidence that the wall is still standing (or not).  However, as soon as you move away from physical evidence (e.g. soft skills, talent development, diversity, leadership) who knows what ‘nasty bacteria’ the trainer might be cultivating in this breeding ground?

So why do trainers still resist evaluation so fiercely?  If they don’t like it why don’t they just change the syllabus? Well because they can’t.  Evaluation is not an optional extra, it is a fundamental element in learning theory – it is the feedback loop that reinforces the right behaviour. Think of the consequences of not doing evaluation – surgeons operating on patients without checking survival rates; rail companies not checking whether train drivers obey red signals; handing over your car keys to someone without a driving licence.  All of these have happened in real life and it is only when people die that we decide we have to act to plug the gaps.  So how can we continue to call some people ‘managers’ or, worse still, ‘leaders’ without any feedback that reassures us they know what they are doing?  If we don’t the potential consequences will be just as disastrous.

In my experience there are two broad reasons why trainers’ practices fail to live up to any proper, professional standards. First, many ‘trainers’ are not professionally trained; so either do not understand the important role of evaluation or just dismiss it as unnecessary.  Second, even if they attend a ‘professional trainer’s course’  the people teaching them don’t know the rules of evaluation or how to apply them. So what is the answer?

Actually it is extremely simple, but not easy, because it demands that we set the same standards as the surgeon, with no compromise allowed. The standard for anyone purporting to be a professional, evidence-based trainer, can be expressed in terms of 3 sacrosanct rules.

  1. Accept that all learning (and therefore training and development input) has to be based on a closed-loop feedback cycle. So if you are supposedly ‘developing managers’ you need to reach a point where you can say they have been developed effectively.
  2. Agree what ‘developed effectively’ means, in advance (not after the event as most evaluation models suggest) with each of the trainees.  This has to be defined in terms of an output related to the role for which the person is being developed.  For example, an ‘effective negotiator’ needs to spell out what that means – higher margins, more contracts, more durable customer relationships?
  3. Realise evaluation always means value – that’s a financial matter.  If there is no anticipated financial benefit to the business then there is no justification for spending any money on pure ‘training’ activity (defined as an input with no expected output).

All of these rules, yes they are unequivocal rules, are broken on a regular basis in every organisation I have ever worked with.  Yet when they are fully understood, and applied consistently, they become the founding principles of a true learning organisation.  Unlike the surgeon though, this is not intended to create a sterile environment, quite the opposite, it will be the most fertile ground you could ever possibly imagine in which beneficial learning will flourish.

For personal development linked to this topic visit the Consummate Professional Series

Share

Is there any ROI in ‘learning’ technology?

Posted on by
Share

How times have changed – or maybe not? Having just spoken at the CIPD’s ‘HRD 2011’ on the topic of ‘How accurate and necessary is ROI for L&D?’ (based on my CIPD book) I was trying to compare it to my last big CIPD conference session (Scottish 2006), entitled “The ROI from Human Capital”, when I suddenly realised I had been typecast.

So what has changed?  Well obviously not the subject matter – no I was thinking more along the lines of the application of technology to individual and organisational learning.  For a start, this was the first conference I have ever spoken at where apparently my every utterance was being tweeted, every couple of minutes, such as -

“The purpose of evaluation is to establish evidence that your organisation is creating value by learning.” (11.41 precisely)

- and I ask myself, is this where the technology was meant to take us?  Is this what the great technological revolution was all about?  All the organisations I know, who have been relentlessly moving towards something they call ‘e-learning’, is this what they had in mind – twittering?  Is the great white hope of social media the answer to anything?

If you do follow tweeters what were you supposed to make of these disconnected, 140-character snippets?  If you look at Lesson 7 I will at least provide one practical lesson that you can take away and use.  Then when you get stuck – comment here, drop me a line, call, or email and I will offer a possible answer to your question.  Now, out of all of that activity, which bit did the technology help with?  It can certainly disseminate data faster to more people.  What it doesn’t seem able to do is offer an answer to the most difficult problems in individual and organisational learning – how we discriminate between what is worth knowing, and what isn’t, and then how we manage to apply what we have learned in a human organisation that doesn’t necessarily want to learn.

Of the 140+ people, who crammed themselves into the seminar room on Wednesday, I wonder what they learned from my session? More importantly, if they learned anything at all, how have they applied it?  What I hope they took away was this (my first slide – tweeted at 11.25 precisely) -

“ROI doesn’t have to be accurate and isn’t always necessary”

This was not only a straight answer to a straight question but one I could only offer, with absolute confidence and 20 years experience, because I know ROI isn’t the main issue here.  The main issue in learning is human relationships.  There are huge barriers to learning in the human condition, especially humans drawn together, willingly or otherwise, in organisations.

The IT industry have always told us that ‘information is power’ but they are wrong.  Partly because they usually provide data, not information (the data has to be processed by a human brain for it to be called information) but even if they do manage to get that far they should have realised that it is not information that is powerful but knowledge and wisdom.  Moreover, it is a reluctance to share knowledge, because it is so powerful, that stops organisations learning what they need to know: those with power are usually reluctant to let it go.  People play politics in preference to playing the learning game and trying to get someone to admit their ignorance, the first step towards enlightenment, usually means they fear losing face.  So you had better have superb skills in helping people to feel good about themselves while they are learning.

So where does ROI fit in with all of this – well you should have come to hear me speak instead of reading tweets or, failing that, you might learn something from reading Lesson 7, or even my book, but none of this beats developing a proper, warm blooded relationship where we can continuously learn from each other in a safe, constructive and mutually supportive environment – and if you already reside in such a place then you are truly blessed.

For personal development linked to this topic visit the Consummate Professional Series

Share