Category Archives: Evaluation & ROI

Evaluation is a crucial element in evidence-based management but it does not have to be a bureaucratic chore or paperchase. ROI is a simple financial formula that should be used selectively and wisely to make the best use of it.

Track records of the 5 management talents

Posted on by
Share

Oakland Athletics Josh Reddick watches his home run off Minnesota Twins

Michael Lewis’s book Moneyball stopped me in my tracks as soon as I read about the ’5 talents’ of baseball.  It forced me to ask myself two very important questions. One, did we ever get to grips with what the ‘5 talents’ of management might be?  Two, has conventional management been made worse by traditional talent selection and management development?  To answer these questions I decided to have a stab at what the equivalent, minimum, complete set of ’5 talents’ might be for a manager?

Here’s the list I came up with, in sequential order: -

  1. Analytical objective setting
  2. Decision making
  3. Planning and organising
  4. Delegating
  5. Monitoring, checking, feedback, listening

Now, before anyone jumps down my throat and says these are not ‘talents’ or I seem to have combined several different skills into one, or whatever other criticism you might want to throw at me, let me just make a very simple point.  Whatever 5 you choose as your minimum the reason we are doing this is to go and check how good our managers are.  If they do not have the ‘minimum’, or have a reasonable prospect of getting there sometime soon, they should not be a manager.  If you do not accept this then the whole exercise is a waste of time anyway, regardless of which ones we choose or whether we call them skills, talents or competencies.

To answer my own question I will start with number 1 on my list – we can always change or adapt it can’t we?  Isn’t this better than a dictionary of 4 zillion competence statements with different descriptors and different levels for each?  Hasn’t conventional management development and ‘talent management’ completely bamboozled everyone by over-complicating what is essentially a simple set of disciplines?  Should management be any more complicated than running a baseball team?  Shouldn’t we just manage people according to what they are particularly good at but, when it comes to the role of ‘manager’, accept it is a bit of an all-rounder that requires an all-round combination?

If we consider the management sequence should it not always follow the one above?  From analysis, to objective, to decision, to planning, to delegation, to action and finally through feedback (and learning) all the way back to the beginning again?  Apply it to any managerial position you like.  Managing does not start with a decision or an objective; it starts with an active mind being analytical.  Whatever a manager is told to do by their boss, whatever task they have been set, whatever performance targets they have to meet, they should use their analytical power to check that what they are being asked to do is not only plausible and viable but of the right priority?  An unthinking manager is an oxymoron (so is a non-evidence-based manager).  Only when their own analysis of their task makes sense to them can they explain the sense of it to the people they manage.  That is why analysis is always first on the minimum list but so often skipped altogether in practice.

Only from there can the other talents come into play.  Managers should not dither or prevaricate, even though the majority of managers are afflicted by that ailment at some time or another.  A point is always reached where a decisive action has to be taken (someone should remind EU ministers and bureaucrats of this).  Only when a decision has been taken can planning and organising begin.  This will require the delegation of tasks and responsibilities to the most appropriately skilled people.  Then by monitoring, listening and taking feedback they can check what progress is being made and manage accordingly.

We could still debate whether these are skills or talents if you want to.  Pragmatically though, if they are genuinely talents, then they cannot be taught, just as I could never be taught how to pitch a baseball to the standard required of the Oakland A’s.  So let’s not pretend otherwise.  If they are skills then rather than focus on developing a skill (or competence) let’s just concentrate on some stats, over time, and the more stats the better.  As long as they produce a meaningful track record to distinguish between those who give an impression of a being a manager and those where supporting evidence starts to stack up.  Regardless of the talents or skills that any individual might possess, Billy Beane’s experience should teach us that the only track record evidence worth looking at is what ends up on the score sheet.

Share

Trainers who rely on happy sheets should at least learn to use them properly

Posted on by
Share

The level of ignorance about evaluation amongst the training community is alarming and it is actually getting worse, rather than better, as more and more self-proclaimed experts come onto the scene offering unhelpful and unnecessary ‘new’ models that just add further confusion. Evaluation is a very simple subject if you keep it simple. It is not about ‘proving’ that training works or justifying training spend but it plays a crucial part in the learning cycle; providing that all-important feedback loop.  However, because most trainers misunderstand its role they either over-complicate it or under-estimate its importance.

You don’t need to know much about training evaluation to know what happy (or smile) sheets are – they are the most simplistic and rudimentary form of evaluation and are very unreliable – but if you persist then you really need to know what you are doing if you are to produce any meaningful evidence from them or learn anything in the process. Strictly speaking, happy sheets are not evaluation* at all but let’s keep things simple for now.  To illustrate my point I am going to bear my soul – or at least share with you some painfully honest ‘happy sheet’ feedback from a very short, mini-workshop I ran recently; although it wasn’t me who chose to hand out happy sheets.  Funnily enough, the subject was evaluation, for a group of trainers who should already have known how to use evaluation if they had received the right professional development; but that is another story as well.

Now, like any ordinary, sentient, human being it is very easy to be flattered by positive responses and not to want to pay too much attention to negative feedback. These are very natural, understandable reactions but there is not much point handing out happy sheets unless you are prepared to follow through on both the good and the bad. If happy sheets have any use at all they have to be part of an evidence-based, highly professional learning methodology: so how would that be different to a conventional approach?

First, the evidence-based trainer would not bother handing out happy sheets unless they had already gauged, before the event, the learner’s own baseline – how much do they already know and what else do they need to know?  They would also need to establish the potential value of the learning experience (in $’s) beforehand because that will significantly influence the trainee’s motivation to learn. Without adopting this Baseline step there is no evidence base for the training and no way to gauge the trainee’s expectations.

Two, having ‘happy’ trainees at the end of a training session is not evidence of success, or value in $’s, which is why* happy sheets are not evaluation: no professional trainer would ever pretend otherwise.

Three, a professional trainer does not regard having ‘unhappy’ trainees at the end of a training session as failure.  Any ‘session’ is only one step along the learning process. The unhappy sheet only reveals one thing (assuming they needed training in the first place) and that is their ‘problem’, which is the organisation’s problem, is still to be resolved.  If the trainer cannot deal with it someone else will have to.

Now for the painful bit. In my session the feedback ranged from “excellent” to “really poor”, with one participant feeling it was “nothing new” – a criticism that was unintentionally a great compliment.  So what does an evidence-based trainer make of that?  How can the same session, from the trainer’s perspective, be both excellent and really poor at the same time?  There is a very simple and extremely old answer to that one: as Ralph Waldo Emerson would say – “Tis the good reader that makes the good book” and only the receptive learner can make the good training session.

Share

Why is HR so expensive?

Posted on by
Share

I bet that is a question that every CFO has asked themselves a million times and at a time when they would dearly love to decimate HR they are probably no nearer to finding an answer than they ever were.

Well, based on my own experience of measuring and assessing HR and L&D functions over the last 20 years, my rough estimate of how much could be saved in the majority of HR and L&D departments – and the bigger the organisation the larger the opportunity – is a straight 50%, with no loss in human capital management effectiveness.

For any CFO’s listening here is a list of some areas you might want to look at.

  • HR and L&D do not act like value adders – they only measure activity (vacancies filled) and inputs (people) not outputs (performance). Otherwise they are no different to anyone else (including accountants) and will want to preserve their jobs and empires by creating work for themselves. So it is worth checking how much of that work is absolutely necessary or adding value?  If it doesn’t fit neatly under one of those headings it has no purpose: so you can start your cost savings immediately without fear of doing any damage.
  • HR people generally tend to work to the letter rather than the spirit of the law because they are risk averse.  Others have a personal agenda that could be doing as much harm as good – dealing with symptoms rather than causes – namely, trying to smash the glass ceiling, stress management, counselling etc.  When they tell a line manager they can’t do something because of the law, or regulations or contractual terms ask them how many times their anticipated problem has ever arisen?  If they don’t have any relevant data they are replacing reasonable risk management with an unnecessarily expensive alternative – perfect legality.
  • Probably the most obvious place to start looking for unnecessary cost comes in the form of HR hand-holding line managers in grievance or disciplinary issues or, worse still, constantly chasing them to do what good managers should already be doing – decent performance reviews and personal development plans. There are two cost saving opportunities here.  First, fire HR people who design policies whose benefits are not obvious to the line and second, fire line managers who are not capable of holding down their job on their own.
  • Compensation and benefit schemes are generally over-complicated these days and have been heading in only one direction (until recently) and for what?  Is there any evidence that they achieved a higher level of performance or retention?  Either way, it is a no-brainer that the cost of complicated salary, reward and benefit schemes tends to rise faster than inflation and their administration is bound to be a relatively high cost operation.  Time to go back to simplicity and at least show some connection between reward and business return?

In short, HR is 50% more expensive than it needs to be simply because non-evidence based HR people cost twice what they are worth.

Share

The Rise and Fall of America’s management ‘empire’.

Posted on by
Share

There are many, many things that I have come to admire about the ‘American Way’ over the years but human capital management isn’t one of them (nor the subtlety of their automotive designs).

Having experienced, at first hand, the style of people management methods amongst some of the biggest American corporations (Exxon, Ford, GE, GM, IBM, Microsoft, Oracle, Texaco) it always struck me that any success they achieved was in spite of, rather than because of, their people management.  The evidence to support this view has been growing steadily over the years (and is presented throughout this series) but to get to the heart of what is wrong in American management one only has to look at America’s inability to learn from its own mistakes.

This is most clearly manifest within the very institution that should be promoting learning – the ASTD (American Society for Training & Development) – which always based evaluation of learning on a poorly designed model from 1959 – Kirkpatrick’s 4 levels.  When the design flaws apparent in this model were exposed, during the TQM revolution of the 1980’s, what did ASTD do?  Instead of admitting they had failed, and returning to the drawing board, they moved into hyper-over-engineering  mode and bolted on another superfluous ‘tailfin’* (Jack Phillips’ fatuous ‘level 5’) in the hope that the new look might deceive corporations enough for them to continue employing their members.

So far, American management has fallen for it.  That does not worry me unduly and should please managers from competing countries.  No, what really concerns me is that the ASTD is now trying to force perfectly sensible learning and development people, from around the world, into following its asinine lead. You don’t need to be a historian to realise that all empires go through a natural, rise-and-fall cycle and there is a now a big question mark over the West’s future but is this just another tell-tale sign of it entering its descendency phase.  By the way, the word ‘descendency’ does not appear in either American or English dictionaries.

It is equally well documented that the grieving process tends to follow five phases of denial, anger, bargaining, depression and, finally, acceptance.  So you might think that the US is already well into the bargaining phase; as evidenced by its internal, political wrangling over its inability to tackle its huge debt, but actually America is still firmly in denial. They are still trying to convince themselves and the world that HCM is something they are actually quite good at.  This is why, if you want the ASTD’s blessing, you will have to attend one of Phillips’ garage workshops on how to add an enormous and costly tailfin to your ‘Mini’ (or whatever model you drive).  It might look ridiculous and not fit very well but you’re stuck with it because it only comes in one size – ROI.

You would be forgiven for thinking that, as a Brit, I am being partisan here if it were not for the fact that I am the first to admit that we are no better at HCM in the UK.  We might take a less ostentatious, more thoughtful, approach but thoughts don’t amount to a hill of beans.  To experience minds that are completely open to new ways of addressing the human dimension of large corporations (and Governments) you need to travel much further East; to a very different, underlying philosophy.  We have been here once before, when the Japanese taught the West a few lessons about how to manufacture efficiently, but there was always so much more that we needed to learn – not least of which was some humility.

When I teach in the East** myself I know they don’t have all the answers either but their great strength is that, unlike the Americans, they don’t try to pretend that they do – and neither do I.  Empires that are built on hubris crash and burn for the very same reason.  When you have believed that you are the best for so long you tend to breed people who either believe their own hype or, worse still, are too frightened to challenge it – why do you think Hitler employed Goebbels?  When hype trumps reality learning ceases and the problems really start.

I still think early reports of the West’s demise are, in the words of one wise American – Mark Twain – greatly exaggerated, but my own prediction would be that the next, most sustainable, management empire is likely to be founded on clear evidence that management is learning to best serve society – not some ugly tailfin.

*Many other evaluation models have emanated from the US over the last 30 years or so – all of them adding unnecessary paraphernalia and gadgets rather than focusing on super-charging the engine.

**Beyond Evaluation & ROI

Share

A different brand of HR and L&D – actual results

Posted on by
Share

In the last post we looked at the empty rhetoric of ‘talent management’ but we could equally have asked to see the substance behind ‘employer branding’, ‘employer of choice’, ‘employee engagement’, ‘competence’, ‘HR business partners’, ‘e-learning’, ‘social media’ – the list goes on and on and it is almost as if we are moving further and further away from actually making a concrete connection between people management and business performance.

This is what ‘Team HR’ has become, a mere logo, a low value brand that is obsessed with appearance and style rather than substance.  ‘Team L&D’ hail from the same stable and struggle with the concept, never mind the practice, of putting a $ sign on their impact, preferring instead to concentrate on the ‘process’ and activity rather than business outcome.

If you want a perfect, bang-up-to-date example of what I mean you only have to read the cover story on this month’s HR Magazine from the American Society for HRM (SHRM) Vol. 56 No. 8 entitled “The Care and Feeding of High-Potential Employees” by Robert J. Grossman suggesting that “Up to one-quarter of your top talent might be fed up and thinking of leaving your organization.”

No doubt many SHRM readers will be fascinated by this apparent insight but my reaction is – are you sure you have identified the right problem?  It is not at all clear what this article is about.  It seems to be about hanging onto ‘talent’ but offers the reader no evidence that these “high potentials” are actually great performers. It also assumes that Eric G., as a “freshly minted Stanford MBA”,  must have been a real catch (does having an MBA guarantee that?) but only reveals that he relocated 5 times in 3 years.  One could reasonably deduce from this that he could not possibly have performed effectively in any of those roles. Have a read for yourself and see what you think.

This blog/book is a determined attempt to change the emphasis of HR, L&D and articles like this from process to evidence.  It is building a new brand of HR and L&D that is not concerned with vacuous logos such as ‘talent management’, ‘human capital’ or ‘analytics’ but a fundamentally different brand of management with clearly distinguishing features.  Right at the top of the list is probably the most obvious and simple point of all – measurable business needs have to drive HR and L&D – not the other way around.

The evidence to date is very clear – every survey of HR and L&D from around the globe will reveal remarkable consistency in approach, process and activity – not differentiation.  How can that be when there are literally thousands of different organisations, in different markets and at different stages in their life cycle?  All of them have their own unique problems and issues but what they get from HR and L&D are standardised answers.  Whereas the evidence-based manager asks what does the business really need?  What business need led to the hiring of people like Eric G, who seem to be difficult to satisfy?  How might the business suffer if a ‘lesser mortal’ were to be employed instead (and may already have been passed over)?  On the scant information offered by this article it appears many American corporations are still much more preoccupied with and obsessed by CV’s than they are with results and tangible value.

Share

Audits of HR and L&D are illogical and inadequate

Posted on by
Share

HR and learning and development (L&D) have come in for some serious criticism over the years, in terms of value for money.  If I felt they needed defending I would suggest that managing people in organisations is a much more complex affair than most operational managers realise or are prepared to admit.  Certainly HR administration can become very costly if not managed well.

It is not just the constantly changing landscape of employment legislation and employee rights or the shifting expectations of new generations of employees that make it so difficult.  Managing human performance involves constantly re-tuning and re-balancing a problematic, employment relationship that comprises a complex mix of mutually-dependent, contractual rights, responsibilities and rewards.  Most line managers I have ever worked with would love to be able to just get on with the task in hand without having to worry about any HR implications but it’s about time they accepted that they can’t.  HR and L&D require careful thought and conventional auditing practice does not tell us whether they are managing people well.  Furthermore, the audit profession at large has failed to acknowledge that human capital management represents a new and special case requiring a fresh perspective.

To put this assertion to the test I wrote a letter to several international, auditing journals last year asking their readers how they audit training spend.  This elicited several replies, all of which took my enquiry seriously and offered very professional, considered and constructive suggestions but none of them acknowledged that the advent of human capital management poses a different set of questions.  Almost hidden within the detail of their answers though were two unshakeable principles of auditing philosophy.

  • All organisational objectives – that means HR and learning objectives as well – have to be seen to be contributing to the main corporate objectives and
  • All objectives have to be measured in terms of financial impact

Trying to apply these principles in a world where ‘intangibles’ are now accepted as having a significant, albeit indeterminate, value is proving to be a real auditing challenge.  This extract from one auditor’s response clearly illustrates how accountants themselves are struggling with this: -

“Unfortunately many accountants have not been trained in the setting of non-financial objectives and are used to defining objectives in qualitative rather than quantitative terms, e.g. the best xxx, the biggest yyy, more effective zzz, etc.  The setting of non-financial objectives can, despite the best of intentions, soon become a game in which the participants revert to the old practices at the first opportunity.  It can take several cycles of the planning process before the setting of quantitative objectives becomes the accepted norm.  The absence of objectives expressed in measurable terms is itself a significant audit finding and associated recommendations may have to be repeated over 3-5 years before they are fully implemented.”

This also describes perfectly what has been happening to HR and L&D practitioners who have yet to accept that they need better measures of what they do. Whereas evidence-based HR and L&D professionals have no problem adopting the auditors’ most cherished principles.  If we insert the words ‘human capital’ for ‘xxx’ and ‘zzz’ why shouldn’t we be asked whether we have evidence that we helped the organisation to acquire the ‘best’ human capital or managed it to ‘best effect’?  If we do not take this challenge on ourselves then we will have to suffer auditors judging us with their conventional methods.  Auditing bodies, such as the UK’s National Audit Office, still try to shoehorn HR ‘auditing’ into their traditional methods by measuring what they can, activity and inputs, rather than what matters – value added.

This inevitably produces meaningless numbers such as ‘the number of HR people per 100 FTE’s’, ‘the number of training days per year’ or ‘the average cost of training’.  The NAO might try to convince themselves that these are indicators of “the organisation’s commitment to enhancing its capacity to deliver and improve” but they know only too well that illogical thinking produces absurd conclusions.  If spending = commitment then why not seek 100% commitment by spending 100% of employee time on training?

Auditors are not stupid people, far from it, but when their tools are just not sophisticated enough for the modern job required we can only expect them to start hitting everything with the same hammer.  In doing so their legendary, sharp-eyed and equally sharp-pencilled cost-cutting logic becomes twisted and distorted under the strain imposed by trying to audit human beings.  This leads them to develop at once both a fallacious and contradictory view of HR and L&D – as a ‘necessary evil’ cost that is also a positive sign of commitment: they cannot be both at the same time.

Do finance and auditing departments have an equally fixed idea of the perfect ratio of accountants and auditors for each 100 FTE’s?  If so, do they automatically assume that a lower ratio is good news or can they for once allow their intelligence some room for manoeuvre and admit it really depends on how much value their auditing ability adds?  If Arthur Andersen auditors had spent more time using their brains and doing their job properly, instead of shredding evidence, maybe Enron and even Andersen would still be around today?

Perhaps those of us in EB-HR and EB-L&D, who base everything we do on evidence of organisational impact, could teach auditors a thing or two about what indicators are necessary to assess the real value of great human capital management?

Share

The last word on Learning Needs Analysis (LNA)

Posted on by
Share

Adopting an evidence-based approach to learning and development is not new – far from it.  Trainers just have to do what they have always been taught to do – learning needs analysis (LNA) and evaluation.  These two crucial steps are inseparable – two sides of the same coin – the yin and yang of individual and organisational learning.  The purpose of the original TNA (training needs analysis) was to provide the evidence-base to justify spending money on training: the purpose of evaluation was to ensure the money was invested wisely.  In other words LNA is an individual needs analysis as part of a business investment appraisal process.

As we saw earlier though trainers have never had a workable model to deal with the evaluation aspects of evidence gathering.  Now we have to explore whether there is any guarantee either that they will bother to undertake a proper, evidence-based, needs analysis.  Visit any training provider (me included) offering training courses or workshops, or any in-house L&D department offering its own menu of programmes, and you will witness ‘learning solutions’ (sic) being offered without any analysis of the learning needs of those who might participate.  Any professional who supports this will have to admit that they are breaking all their own rules.  It should be outlawed and, if I am as guilty as anyone else, what should I do about it?

Well, first let me declare that I fully accept the crucial importance of LNA in principle even if I don’t always do it in practice.  That might sound hypocritical but how many ‘learning professionals’ would at least be willing to admit the same?  Second, I am bringing it to your attention, and anyone else who cares to read this, in an attempt to increase the incidence of proper LNA.  Third, I am prepared to make myself very unpopular by exposing bad practice and the stupid organisational behaviour it produces.  Fourth, I have at least offered part of the solution already with a very simple Baseline Model and fifth, I am now going to complete the set with the last word on what learning needs analysis really means. There is probably a sixth and a seventh but what have you been doing about it?

Let’s re-visit what we were taught about the old-fashioned TNA. If you want to train someone in customer service, say in the hotel trade, the business has to decide what standard of service it wants to provide (e.g. every customer should be greeted with a friendly smile and asked ‘can I help you?’).  This is OK for basic training and it would be easy to do a spot check (evaluate) that the training is being applied. If the hotel wants to differentiate itself from its competitors though, by raising its own standards, the trainer needs to gather more evidence before designing any training.

  • ‘What do higher standards look like – would it be speedier response times at the desk?
  • ‘Are we measuring that already?’
  • ‘So what if we are speedier, does that produce more value in $’s?’

So far so professional but move away from basic training and soon the rules of the game, the principles that mark out the professional, are jettisoned remarkably quickly.  Yet the very reason TNA started to become known as LNA was because we all eventually realised that training was just input (e.g. sending people on generic management modules) and learning had to be applied if it was to produce a beneficial output (i.e. more satisfied, paying customers).  Somewhere along this line though LNA forgot its TNA roots and became a much broader concept; to the extent that many ‘learning professionals’ today have started suggesting that no LNA is needed because ‘all learning is good’.

This is not a principle I have ever been taught and my 30 years of experience clearly demonstrates this is patently untrue.  If you subscribe to this notion why do we still pay for training these days when so much learning can be had  for free?  Conversely, if all learning is good, go and ask your board for a 1000% increase in the training budget and see what response you get.  It doesn’t stand up well does it?  Yet this fuzzy logic has been exploited by the technology-obsessed, ‘e-learning’, social media advocates who do not know how to justify what they are doing.  Often business needs are forgotten altogether and there is no attempt at Baseline evaluation.  In effect, the ‘new’ training providers have been taking several steps backwards for years, moving further and further away from the fundamental tenets of learning whilst trying to pretend they are transforming the way we learn (why do you think they never chose to call it e-training?).

There is a simple solution.  Having admitted my own guilt, the first thing I try to do is make up for the lack of individual, learning needs analyses whenever I can.  If you have not already read What is EBM or The Case for EB-HR then I would advise you to do so now because that is exactly what these pages were designed to do; to make sure you are in the right place.

If we should ever meet in a classroom and you are not sure why you are there I would ask whether you want to come back when you know and remind you that you need to come back with your own Baseline, because you can only start from where you are, regardless of where the rest of the class might be.  If you show no interest in sticking to these rules I would recommend that you do not stay, not just because you are unlikely to learn very much but because I am passionate about learning and I want you to be passionate as well.  By the way, there is no such thing as a ‘generic’ or ‘group’ LNA – we can only learn as individuals – even though we are all under pressure to conform to group behaviour.

One of the key advantages of analysing an individual’s learning needs is that we want to avoid upsetting, or getting in the way of, those who are genuinely there to learn.  Maybe that is why very few trainers are inclined or even allowed to perform a proper LNA – it might be too embarrassing to realise just how inappropriate and ineffective a lot of training is; whether it is in the classroom or online.

The last word? LNA is not an option – it’s a professional duty.

For personal development linked to this topic visit the Consummate Professional Series

Share

All you need to know about training evaluation in about 700 words

Posted on by
Share

When asked by the editor for a title for my CIPD book on evaluation I suggested – ‘The Last Word on Evaluation’ – not out of arrogance but because the mountain of literature and endless debate around the subject had never reached a conclusion.  Evaluation is essentially a simple subject made unnecessarily complicated by vested interests. Probably the biggest and most influential vested interest is the American Society for Training and Development (ASTD) who, whenever they tried to offer their members advice on this thorny subject, managed to get it so completely and so obviously wrong.

From the 1970’s the ASTD regarded Kirkpatrick’s 4-Levels as their standard:

  1. Reaction to training (happy or smile sheets)
  2. Testing learning
  3. Applying learning in the workplace
  4. Business impact

Let’s try this out by thinking of one salesperson going on a sales training course at a cost of $1000. When they return you take them through the 4 levels, starting with how they felt about the course…  – WRONG.  That way you end up at Level 4 with no way of assessing impact.

Let’s start again. Before you send them on the course you ask them how much they sell now ($1.2 million) and what the profit margin is (10%)?  Then afterwards, when you get to Level 4, you can ask again how much they are selling and you have a basis for gauging impact. The critical, pre-learning questions form the BASELINE and produce the simplest, most obvious and effective evaluation model with just 2 levels:

  1. Baseline evidence
  2. Business impact measured against the baseline data

- but you will quickly find in practice that this 1st, Baseline level will usually suffice because it is this one that adds all the value in what is now an enhanced, learning process. Individual learning starts when each individual knows their own Baseline.

The 4-Levels never captured this so, just as corporations started asking what the financial return (ROI) was on their training investment, Kirkpatrick’s obsolescence was becoming apparent.  So the ASTD decided to back a different horse in the 1990’s, but could not admit Kirkpatrick* was wrong, so adopted Jack Phillips’ model which just added another level – level 5 for the ROI calculation – which also made it look like an innovation – WRONG again.

Let’s go back to the beginning. You ask the sales trainee the BASELINE questions. You do the training and then at Level 4 you get an answer as to how much sales have increased (1%) and you know the cost ($1000) so you can do the net ROI calculation immediately – it’s 20%. There is no need for any level 5 – it doesn’t add anything and anyone with a calculator can work it out.  Under Phillips though you have to spend even more time and money making the numbers up and converting to $ because he doesn’t establish the relevant $ sign at the beginning.  Level 5 was always a figment of the ASTD’s collective imagination and that’s why the figures Phillips produced never convinced anyone who had a business head on their shoulders.

So the ASTD decided it needed more credibility and drafted in a labour economist  – Laurie Bassi – in a vain attempt to garner a more academic and quasi-scientific level of respectability.  Laurie tried to show the business impact from the billions of $’s ASTD members were spending on training.  Laurie did not know any more about evaluation than her predecessors though and, as an academic, used the only analytical tools she had, regression analysis to produce correlations, using retrospective data.  I guess she also did not have the benefit of first-hand experience of what it feels like to work in a training department in a large corporation; where trainers are often under pressure from managers to produce all sorts of stupid, knee-jerk programmes to cover up their deficiencies in people management and development.  Laurie is now heading up efforts to establish international standards by the other large American professional body, SHRM ( the Society for HRM) and the ASTD still makes it mandatory for anyone wanting to be their ‘partner’ to attend a Jack Phillips training programme.

Forty years should have been long enough for the Americans to get it right but so far, despite being an SBO (statement of the bleeding obvious), no ‘expert’ in the US has ever fully understood or acknowledged the crucial importance of the BASELINE level in evaluation and learning; whether it be sales training, management development, leadership or OD.  Starting with a Baseline makes learning evidence-based, it is a perfect application of evidence-based management.  Now, if you are looking for an indication of which corporations waste the most money on ‘training’ just ask them which model they use and where they start from.

For personal development linked to this topic visit the Consummate Professional Series or attend a workshop.

* The Kirkpatricks still refer to their model as ‘the 4-levels’, although son Jim now tries to make up for the absence of ROI with something called ROE (return on expectations).  This is intended as a sort of ROI-lite, except ‘expectations’ are not necessarily couched in $ terms and no ‘return’ can actually be calculated because ROE is not a mathematical or financial formula.  In short, ROE can mean whatever you want it to mean.


Share

Criteria for producing the Best HR Evidence.

Posted on by
Share

A recurring theme in this series, and a rather irritating obstacle to the further development of the evidence-based management movement, is the limitations of language – especially the one word that is meant to underpin everything – evidence.  What should be the movement’s greatest strength could prove to be its biggest impediment, unless we clarify it once and for all.

If that task were not difficult enough, we have the added complexity here of concentrating on evidence-based human resource management, which has its own data measurement problems.  Human data is different to most other types of organisational and performance data and just calling it ‘human analytics’ doesn’t suddenly transform it into evidence.  In order to break down this dual barrier there needs to be more precise use of language and clear criteria as to what constitutes valid evidence.

We should not lose sight of what we are trying to do here though – we define EBM as ‘making managerial decisions based on the best evidence available’ accepting that we do not live in a perfect world.  The best we can do therefore is aim for the highest probability that we are using the best evidence available.  Below is one set of pragmatic, working definitions (I don’t want any of my academic colleagues to get me bogged down in semantics) and you can add this to your favourite management checklists if it works for you.

1. Data, information and knowledge

First, let us make a distinction between 3 commonly used terms whilst adding a few human insights -

Data – straightforward facts, statistics and numbers that do not provide a basis for decision making – e.g. the population of the UK is 60 million, the earth is 93 million miles from the sun.

Human insights – some people think they can make decisions based on such raw data saying the UK is ‘overcrowded’ or ‘we have too many immigrants’.  Beware those who try to promote data as knowledge. Also, human error might have got the data wrong.

Information – a human being has processed all the relevant data in their head and started to draw some conclusions e.g. the population growth trend in the UK relative to available water resources, how long it would take to get to the sun – but it is still unusable in this format.

Human insights – Information only has meaning when processed by human beings and it is always unique to the individual. Once human beings are involved you can forget trying to regard this as a science.  An engagement survey score of 52% can be seen as both a success and a failure, depending on your perspective, and will not change the attitude or behaviour of the manager who does not perceive it as having value.

Knowledge – here we will use the simplest, purest definition – you actually know something to be true.  We know ‘1 + 1 equals 2’ because no one disputes the basis of the calculation, it is regarded as proof or perfect knowledge.  Better still, we can see, touch and experience it for ourselves.

Human insights – you are rarely, if ever, going to achieve a state of pure knowledge in EB-HR and genuine evidence-based managers do not pretend otherwise.  Beware anyone working in HR holding out the promise of proof.  Managers should openly admit their knowledge is imperfect and always accept they have to continuously learn (yeh, right), rather than blame, because imperfect knowledge inevitably produces imperfect decisions.

2. Defining Evidence

These definitions, on their own, might not appear to be particularly helpful though until we use them to define what evidence means to an evidence-based manager -

Evidence = Actionable Knowledge

This is really the only pragmatic definition.  Decisions have to be made, regardless of how well they are made.  At any point in time we just try to ensure we get as much knowledge as we can in order to act.  So, in the south east corner of the UK, an action plan is required to continue to provide water according to the population growth projections and what is ‘known’ about climate and weather pattern predictions.  Over time those decisions could turn out to be unsuccessful or appear to have been ‘wrong’ but taking no action at all is not an option.

What we need to do now is look at what data we might come across in the fields of HR, human capital and learning and how we can develop our own heightened awareness of what constitutes the best knowledge and evidence available and the skills to use them effectively.

3. Criteria for ‘best evidence’

Here is a simple list of questions to help you make intelligent choices about the quality of evidence presented to you and how you are prepared to use it.

Activity – avoid activity data like the plague. Probably the best example is ‘number of training days’ or ‘annual training hours per employee’.  PwC/Saratoga (see page 6) still insist that knowing the ratio of HR people to FTE’s is meaningful data – it isn’t. This type of data just tells you somebody is sitting somewhere, not what they are producing.  See also INPUT and CORRELATION below.

Causation – both Gallup’s Q12 “Proven Approach” and Watson Wyatt’s* (now Towers Watson) “Human Capital Index” refer to their statistics as PROOF that their methods work. No credible EB-HR Manager would dare to suggest this without establishing CAUSATION from the beginning: statistical regression mistakenly tries to do this after the event.  Regression is an aircrash investigation while EB-HR is aircraft design.  EB-HR managers have a much simpler, more practical and convincing way to deal with CAUSATION – they go and ask the people using Q12 or the HCI to show their original, causative hypothesis (e.g. which particular employees in this department will sell more if they become more engaged).  If they don’t operate in this way they failed to use the best evidence available.

Correlations – In the absence of CAUSATION the purveyors of very popular, off-the-shelf, HR ‘solutions’ substitute CORRELATIONS, which is the lazy statistician’s way of making up data to suit themselves in order to make a fast buck.  For example, suggesting there is data that shows the more a company spends on training the better it will perform.  Even if these spurious correlations are made to look statistically valid the dimmest, first grade student of statistics will remember they were taught not to trust CORRELATIONS**.  Instead, the evidence-based learning manager will make sure training is designed strictly in accordance with the principle of causation by only investing in training that is designed to deal with specific business issues.

Input – generally cost, time or manhours expended. It cannot be turned into useful INFORMATION until it is set against a corresponding OUTPUT (e.g. how much did all this produce in terms of cars, bank loans etc.)  HR departments and training managers often resort to measuring input data (happy sheets) simply because they don’t know how to measure outputs.  The worst ones think measuring inputs tells their bosses something about the way they work – yes it does, but not what they intended it to.

Output – The only thing that matters and produces value. Goods or services actually produced and sold, their costs reduced or quality improved and any extra revenue earned.  Any HR or training not focused on these outcomes is not only non-evidence-based but a waste of resources.

Performance – a rule-of-thumb definition of a performance measure is that you should know which way the measure should move, which is not as easy as it sounds.  For example, should your staff turnover or attrition rate go up or down?  Should your ratio of HR to FTE’s go up or down – well surely it depends on what they are doing? The key is for everyone involved to understand and agree what ‘good’ looks like.  Stupid measures or ratios encourage stupid behaviour: reducing your HR to FTE ratio could seriously damage the business if you lose people in HR who were adding a lot of value.

Qualitative – a confusing term – the best definition is to view it as the opposite of OBJECTIVE.  Some people use the word QUALITATIVE to mean SUBJECTIVE data (i.e. how much do you respect your boss on a scale of 1 to 5?) while others regard it as a measure of intangibles (i.e. engagement, empowerment etc.).  It only becomes tangible EVIDENCE (actionable knowledge) though when someone has a stab at making it OBJECTIVE by putting a potential value on it.

Quantitative – the meaning should be very obvious – it is simply data that has a number attached (the population is 60 million) – regardless of whether the number has any intrinsic validity, purpose or application.  It is often viewed, mistakenly, as the opposite of QUALITATIVE on the basis that any number or measure is preferable to an indistinct or subjective statement.  But some things are extremely important without any numbers attached – how about the question ‘do we have a financial regulation SYSTEM in place?’ It’s not quantitative, it’s not qualitative (it can be clearly demonstrated) but it’s probably the most valuable element in evidence-based human resource management.

For personal development linked to this topic visit the Consummate Professional Series Module 2 and Module 4

*Compare Towers Watson’s ‘proof’ with a contradictory statement from their own European Survey Report in 2000 – “(HCI) Demonstrates a very strong correlation between effective people practices and shareholder value…but on its own does not prove a causal link.”

**“It is important to state 3 caveats about regression models. First, correlation does not mean “cause”. The fact that one variable is correlated to another does not necessarily mean that one variable causes another…. Generally, you should conclude that one thing causes another only if you have some other good reason besides the correlation itself to suspect a cause-and-effect relationship. Second, keep in mind that these are simple linear regressions…. Finally, in multiple regression models, you should be careful of independent variables being correlated to each other…. regression modelling …is a useful tool, but proceed with caution.”

From “How to measure Anything – Finding the value of ‘intangibles’ in business” (2nd Ed.) Douglas W. Hubbard, Wiley, 2010

Share

Evaluation of learning – 3 simple rules of the game

Posted on by
Share

When is a profession not a profession?  When it fails to follow its own professional advice.  Imagine a surgeon saying – ‘Yes, I know the operating theatre should be a sterile environment but hey, what are a few bacteria among friends?’  Well that is exactly the sort of attitude adopted by the vast majority of training ‘professionals’ towards the subject of evaluation. They know their own professional standards demand they evaluate – as a very specific requirement in the learning needs, design, delivery, and evaluation cycle – but studiously avoid doing so.

So let’s try re-phrasing this.  If we said that trainers do not collect any evidence that their methods work we might expect them to be operating in very ‘dirty departments’.  Of course, if you’re teaching a bricklayer to lay bricks this does not matter too much because you at least have the evidence that the wall is still standing (or not).  However, as soon as you move away from physical evidence (e.g. soft skills, talent development, diversity, leadership) who knows what ‘nasty bacteria’ the trainer might be cultivating in this breeding ground?

So why do trainers still resist evaluation so fiercely?  If they don’t like it why don’t they just change the syllabus? Well because they can’t.  Evaluation is not an optional extra, it is a fundamental element in learning theory – it is the feedback loop that reinforces the right behaviour. Think of the consequences of not doing evaluation – surgeons operating on patients without checking survival rates; rail companies not checking whether train drivers obey red signals; handing over your car keys to someone without a driving licence.  All of these have happened in real life and it is only when people die that we decide we have to act to plug the gaps.  So how can we continue to call some people ‘managers’ or, worse still, ‘leaders’ without any feedback that reassures us they know what they are doing?  If we don’t the potential consequences will be just as disastrous.

In my experience there are two broad reasons why trainers’ practices fail to live up to any proper, professional standards. First, many ‘trainers’ are not professionally trained; so either do not understand the important role of evaluation or just dismiss it as unnecessary.  Second, even if they attend a ‘professional trainer’s course’  the people teaching them don’t know the rules of evaluation or how to apply them. So what is the answer?

Actually it is extremely simple, but not easy, because it demands that we set the same standards as the surgeon, with no compromise allowed. The standard for anyone purporting to be a professional, evidence-based trainer, can be expressed in terms of 3 sacrosanct rules.

  1. Accept that all learning (and therefore training and development input) has to be based on a closed-loop feedback cycle. So if you are supposedly ‘developing managers’ you need to reach a point where you can say they have been developed effectively.
  2. Agree what ‘developed effectively’ means, in advance (not after the event as most evaluation models suggest) with each of the trainees.  This has to be defined in terms of an output related to the role for which the person is being developed.  For example, an ‘effective negotiator’ needs to spell out what that means – higher margins, more contracts, more durable customer relationships?
  3. Realise evaluation always means value – that’s a financial matter.  If there is no anticipated financial benefit to the business then there is no justification for spending any money on pure ‘training’ activity (defined as an input with no expected output).

All of these rules, yes they are unequivocal rules, are broken on a regular basis in every organisation I have ever worked with.  Yet when they are fully understood, and applied consistently, they become the founding principles of a true learning organisation.  Unlike the surgeon though, this is not intended to create a sterile environment, quite the opposite, it will be the most fertile ground you could ever possibly imagine in which beneficial learning will flourish.

For personal development linked to this topic visit the Consummate Professional Series

Share