Tag Archives: evaluation

Trainers who rely on happy sheets should at least learn to use them properly

Posted on by
Share

The level of ignorance about evaluation amongst the training community is alarming and it is actually getting worse, rather than better, as more and more self-proclaimed experts come onto the scene offering unhelpful and unnecessary ‘new’ models that just add further confusion. Evaluation is a very simple subject if you keep it simple. It is not about ‘proving’ that training works or justifying training spend but it plays a crucial part in the learning cycle; providing that all-important feedback loop.  However, because most trainers misunderstand its role they either over-complicate it or under-estimate its importance.

You don’t need to know much about training evaluation to know what happy (or smile) sheets are – they are the most simplistic and rudimentary form of evaluation and are very unreliable – but if you persist then you really need to know what you are doing if you are to produce any meaningful evidence from them or learn anything in the process. Strictly speaking, happy sheets are not evaluation* at all but let’s keep things simple for now.  To illustrate my point I am going to bear my soul – or at least share with you some painfully honest ‘happy sheet’ feedback from a very short, mini-workshop I ran recently; although it wasn’t me who chose to hand out happy sheets.  Funnily enough, the subject was evaluation, for a group of trainers who should already have known how to use evaluation if they had received the right professional development; but that is another story as well.

Now, like any ordinary, sentient, human being it is very easy to be flattered by positive responses and not to want to pay too much attention to negative feedback. These are very natural, understandable reactions but there is not much point handing out happy sheets unless you are prepared to follow through on both the good and the bad. If happy sheets have any use at all they have to be part of an evidence-based, highly professional learning methodology: so how would that be different to a conventional approach?

First, the evidence-based trainer would not bother handing out happy sheets unless they had already gauged, before the event, the learner’s own baseline – how much do they already know and what else do they need to know?  They would also need to establish the potential value of the learning experience (in $’s) beforehand because that will significantly influence the trainee’s motivation to learn. Without adopting this Baseline step there is no evidence base for the training and no way to gauge the trainee’s expectations.

Two, having ‘happy’ trainees at the end of a training session is not evidence of success, or value in $’s, which is why* happy sheets are not evaluation: no professional trainer would ever pretend otherwise.

Three, a professional trainer does not regard having ‘unhappy’ trainees at the end of a training session as failure.  Any ‘session’ is only one step along the learning process. The unhappy sheet only reveals one thing (assuming they needed training in the first place) and that is their ‘problem’, which is the organisation’s problem, is still to be resolved.  If the trainer cannot deal with it someone else will have to.

Now for the painful bit. In my session the feedback ranged from “excellent” to “really poor”, with one participant feeling it was “nothing new” – a criticism that was unintentionally a great compliment.  So what does an evidence-based trainer make of that?  How can the same session, from the trainer’s perspective, be both excellent and really poor at the same time?  There is a very simple and extremely old answer to that one: as Ralph Waldo Emerson would say – “Tis the good reader that makes the good book” and only the receptive learner can make the good training session.

Share

The Rise and Fall of America’s management ‘empire’.

Posted on by
Share

There are many, many things that I have come to admire about the ‘American Way’ over the years but human capital management isn’t one of them (nor the subtlety of their automotive designs).

Having experienced, at first hand, the style of people management methods amongst some of the biggest American corporations (Exxon, Ford, GE, GM, IBM, Microsoft, Oracle, Texaco) it always struck me that any success they achieved was in spite of, rather than because of, their people management.  The evidence to support this view has been growing steadily over the years (and is presented throughout this series) but to get to the heart of what is wrong in American management one only has to look at America’s inability to learn from its own mistakes.

This is most clearly manifest within the very institution that should be promoting learning – the ASTD (American Society for Training & Development) – which always based evaluation of learning on a poorly designed model from 1959 – Kirkpatrick’s 4 levels.  When the design flaws apparent in this model were exposed, during the TQM revolution of the 1980’s, what did ASTD do?  Instead of admitting they had failed, and returning to the drawing board, they moved into hyper-over-engineering  mode and bolted on another superfluous ‘tailfin’* (Jack Phillips’ fatuous ‘level 5’) in the hope that the new look might deceive corporations enough for them to continue employing their members.

So far, American management has fallen for it.  That does not worry me unduly and should please managers from competing countries.  No, what really concerns me is that the ASTD is now trying to force perfectly sensible learning and development people, from around the world, into following its asinine lead. You don’t need to be a historian to realise that all empires go through a natural, rise-and-fall cycle and there is a now a big question mark over the West’s future but is this just another tell-tale sign of it entering its descendency phase.  By the way, the word ‘descendency’ does not appear in either American or English dictionaries.

It is equally well documented that the grieving process tends to follow five phases of denial, anger, bargaining, depression and, finally, acceptance.  So you might think that the US is already well into the bargaining phase; as evidenced by its internal, political wrangling over its inability to tackle its huge debt, but actually America is still firmly in denial. They are still trying to convince themselves and the world that HCM is something they are actually quite good at.  This is why, if you want the ASTD’s blessing, you will have to attend one of Phillips’ garage workshops on how to add an enormous and costly tailfin to your ‘Mini’ (or whatever model you drive).  It might look ridiculous and not fit very well but you’re stuck with it because it only comes in one size – ROI.

You would be forgiven for thinking that, as a Brit, I am being partisan here if it were not for the fact that I am the first to admit that we are no better at HCM in the UK.  We might take a less ostentatious, more thoughtful, approach but thoughts don’t amount to a hill of beans.  To experience minds that are completely open to new ways of addressing the human dimension of large corporations (and Governments) you need to travel much further East; to a very different, underlying philosophy.  We have been here once before, when the Japanese taught the West a few lessons about how to manufacture efficiently, but there was always so much more that we needed to learn – not least of which was some humility.

When I teach in the East** myself I know they don’t have all the answers either but their great strength is that, unlike the Americans, they don’t try to pretend that they do – and neither do I.  Empires that are built on hubris crash and burn for the very same reason.  When you have believed that you are the best for so long you tend to breed people who either believe their own hype or, worse still, are too frightened to challenge it – why do you think Hitler employed Goebbels?  When hype trumps reality learning ceases and the problems really start.

I still think early reports of the West’s demise are, in the words of one wise American – Mark Twain – greatly exaggerated, but my own prediction would be that the next, most sustainable, management empire is likely to be founded on clear evidence that management is learning to best serve society – not some ugly tailfin.

*Many other evaluation models have emanated from the US over the last 30 years or so – all of them adding unnecessary paraphernalia and gadgets rather than focusing on super-charging the engine.

**Beyond Evaluation & ROI

Share

I’m a reflector, completer finisher, ENTJ, inspirer – what are you?

Posted on by
Share

What is it with psychologists that they want to pigeonhole everybody – all 6 billion of us? Here is just a very small sample of the sort of ‘bump feeling’ that I have personally been subjected to over the years: -

Peter Honey tries to tell me we have 4 ‘learning styles’ including the ‘reflector’  while

Belbin’s theory of Team Roles fails to convince me that there are only 9 roles including the ‘completer finisher’ and

Myers Briggs Type Inventory (depending on which day of the week it was) last told me that, of the 16 options available, I was an ENTJ

Now there is something called the Insights Discovery Full Circle Profile which has 8 segments, one of which is ‘inspirer’.

Obviously the question that we have to ask is where is the evidence that any of this stands up to scrutiny? But before we do that let us not under-estimate the validity of our own conclusions, drawn from our own experiences of actually meeting, relating to and working with our fellow human beings over many years.  In my own case (55 years and counting) I can honestly say that, in my entire life, I have never met two identical human beings (even ‘identical’ twins are not identical). Fortunately this personal view is now backed up by what we know about DNA.

Consequently I have my own hypothesis (the theory is as yet untested) that if we took all of the psychological tools, instruments, profiles and tests available on the entire global market, and analysed the make up of every single human being on the planet, we would come to the conclusion that the permutations of personality types available would precisely match the size of the world’s population (just the four mentioned above produce 4608 permutations).  We are all different; we don’t behave in a consistent way every day at work; we might not even follow our own preferences (because of the organisational pressures on us) and often we will not be allowed to bring our entire personality to bear anyway (are you allowed to be as creative as you would choose?).

Of course the authors of these products will have concocted their own theories and have testimonials from happy customers to support their methods but what they don’t seem to offer is any evidence.

Having taught and trained many, many people over the last 30 years myself, and as an evaluation specialist, I have never ever regarded happy or smile sheets (known as level 1 questionnaires in the trade) as evidence of anything.  Happiness is not evidence of learning and unhappiness is not evidence of failure (learning is often initially painful and starts with resistance). Even if I make participants take a test (level 2) and they can tell me what Belbin’s 9 team roles are does not mean they know what to do with them. Even a level 3 visit (to see if they are applying them in the workplace) might only reveal them putting labels on hapless managers and does not offer evidence that having a defined role, in a particular team dynamic, at a particular time and place, produces better team results.

There is not much point waiting for convincing evidence to be provided either.  Even if these tools appear to offer some benefits they could equally be doing some harm; confusing people and altering their behaviour artificially and to what end? The only end that matters is an improvement in the value the organisation can create. Without evidence that points in this direction all of these approaches have no more validity in an organisational setting than an astrologer telling you that your star sign is in Uranus.  Moreover, giving people too many labels can send them into a spiral of hyper self-consciousness which inevitably leads to them disappearing up their own black hole.  As one of Insights’ own customers from the Nationwide Bank demonstrates when he testifies: -

“I was amazed at the accuracy of the initial Insights Discovery Personal Profile. To now take the exercise a stage further by consulting with peers and customers is excellent in giving all-round unbiased feed-back.”

Share