What’s the deal with ‘evidence-based practice’? Aren’t we all evidence based?

To an extent, yes - we all use evidence. But as we argue in our positioning paper, In search of the best available evidence, there are two big things we tend to get wrong when using evidence to inform decisions.

First, we’re often not great at gauging the quality of evidence we’re looking at. There is a well-established hierarchy of scientific evidence on cause-and-effect relationships. The ‘gold standard’ is randomised controlled trials, which carry a good deal more weight than, say, simple before-and-after studies, and far more than surveys run at one point in time. If we can take note of this hierarchy in looking at evidence, we are well on the way to making more reliable, better decisions.

Second, we tend to cherry pick evidence that supports our pet theory. It feels great when you find a piece of research that confirms what you long suspected. But barring the ridiculous, the chances are you’ll be able to find research – even good quality research – to back your opinion whatever it is. To find out if a technique is really worth replicating, we should look at the wider body of evidence. So sitting above the hierarchy of single studies, we have a ‘platinum standard’ of systematic reviews and meta-analyses.

Evidence-based HR means being anti-fad and willing to face uncomfortable truths. But it’s hugely important. Relying on weak evidence can feel politically expedient (staying in line with what your boss expects to see) or compelling (in tune with the zeitgeist, intuitively right), yet at its worst it gives little more than a 50% chance of success, the equivalent of flipping a coin. If a decision is important, it’s worth testing the available evidence in this light: how much more scientific is it than a coin toss?

There are plenty of thorny questions in evidence-based HR but the basic principles are pretty simple and more importantly, worth striving for. Our hope is that this forum will help put people these principles to work and grapple with the challenges. Thoughts?

Parents
  • Part of the challenge is that it can be hard to find practitioner-focused evidence. A lot of academic research is aimed at other academics (rather than people managers) and adopts a language that many HR professionals find hard to understand. Also, the research often follows fads and fashions, so we don't necessarily get the evidence base in the right areas of people management.
  • Yes I completely agree Charles. A lot of academic research is not practitioner-focused, and for good reasons. Similarly the language is not helpful - often not even to other academics! And fads and fashions are just as evident in academic research.

    On the other hand, academic research is just one source of evidence and like any source of evidence it needs to be judged for it's relevance and trustworthiness. And it's also about using the best available evidence. So even if the evidence isn't great it's still worth checking out.

    As you probably know, more and more academics are becoming concerned about scientific (mal)practices. There's a great summary here by John Antonakis where he discusses what he calls the five diseases of academic publishing:

    retractionwatch.com/.../
  • Sorry that was way too small!

  • I'd hate to speak for Jacques on the basis of a 20-minute seminar and 15-minute face-to-face. Looking over my notes, though, his point on Big Data seems to have been less that it was the answer to all our woes and more that HR practitioners are ignoring its potential, seeing it as a sales/marketing tool rather than a people tool. If I look at the other topics we covered, and particularly at the approach to employee-led performance appraisal, I'd say that the overall push was in the direction of, but not identical to, the EBP infographic.

    For example, he was quite a fan of psychometrics as a tool for employee development, but my understanding of the literature on that subject from practising academics in neurology and psychology (as opposed to former academics in those fields who've developed a product to sell) is that psychometrics are basically pseudoscience.

    However, when dealing with any consultant - especially one at the top of his game - you always have to remember that they too have a product to sell.

    Your point about the emphasis on practice may be semantic, but it's very important and not a perspective I'd thought of before.
  • What a great example. We often think a big question - like 'how do we improve learning in our organisation'? - needs a big answer - like 'a £1m LMS system'. There's a great book our by Owain Service from The BIT, that aims to debunk just that...

    www.behaviouralinsights.co.uk/.../

    I bet there is plenty 'small steps' work out there that never gets captured or shared. Especially when competing with the glossy campaigns of big corporate HR and their consultancies.

    Perhaps you could outline one of your experiments, Liam? What approach did you take to measuring your interventions and their impact on learning? What were your results?

    Jonny/Rob - the salesman in me thinks that retelling and heroising some of these stories/examples alongside tools and resources might be impactful.
  • James I find your arguments compelling - here and above ('how should we compete'? How can we be heard?'). I think part of the challenge is being evidence-based ourselves.

    Most of CEBMa resources, tools and trainings available are based on the hard science of how people learn and build skills https://www.researchgate.net/publication/260178378_The_Science_of_Training_and_Development_in_Organizations_What_Matters_in_Practice 

    So we may not sell evidence itself with overstatements, but can we sell the path to understand and use better quality evidence with confidence over specific outcomes? I think we could begin defining what outcomes we aim to achieve, and at what level of analysis - maybe beginning with individual-level?

  • James I find your arguments compelling - here and above ('how should we compete'? How can we be heard?'). I think part of the challenge is being evidence-based ourselves.

    Most of CEBMa resources, tools and trainings available are based on the hard science of how people learn and build skills www.researchgate.net/.../260178378_The_Science_of_Training_and_Development_in_Organizations_What_Matters_in_Practice

    So we may not sell evidence itself with overstatements, but can we sell the path to understand and use better quality evidence with confidence over specific outcomes? I think we could begin defining what outcomes we aim to achieve, and at what level of analysis - maybe beginning with individual-level?
  • Exactly. In the absence of strong or reliable evidence you can still proceed but, as you say, more cautiously than if that wasn't the case.
  • I agree with most but not the bit about defining outcomes. First you need evidence to define and identify problems (or opportunities)! In other words, I think that starting with 'outcomes' in mind (e.g., let's improve performance) can be very unhelpful and misleading. Clearly, those interested in evidence-based practice do not yet have compelling evidence that there is a 'problem' which EBP can help fix.
  • Thanks interesting points. I'm not sure I agree that psychometrics = psudoscience as there are so many different types of psychometrics with varying validity and reliabilty.

    Yes, it's ALL about practice and not about evidence as such. But it's also about doing stuff first to discover the evidence about what the real problems/opportunities may be and only then thinking about what you might do. Things like 'big data' are sold as sort of non-solutions to non-problems though of course data can help.
  • Thanks, Pietro – I’m pleased to hear that, I’ve been blown away by the work you and the team are doing at ScienceForWork.

    The paper on training and development is great. The area I’m most intrigued by is described as ‘individual differences’ - and in particular the components of self-efficacy, goal orientation and motivation. It’s an oversimplification, but I would position these as providing an intent to learn and practice EBHR. Until you have that, there’s little drive to use the tools and resources that are available.

    We know quite a lot about amplifying intent – take the messenger effect I mentioned above – we’re influenced by who communicates information; we’re more likely to act when that person or institution has authority, we trust them, and they're ‘like us’.

    There’s a problem with the latter. EBHR advocates will typically tick the first two boxes, but the approach feels very unfamiliar to an ‘everyday’ HR practitioner. The language is different and the expertise of an HR professional (and therefore their self-efficacy) may feel under threat or undermined (particularly when we spend so much time ‘debunking’ concepts that are so widely used in the community).

    Now debunking should of course continue, and the language of academia is here to stay. But how can you also make advocates of messengers like Liam, who are taking small, accessible steps in the field, and whom HR practitioners are able to relate to on a day-to-day basis…? How do you begin to hold early-adopter practitioners up as heroes? As pioneers, even?

    Just an example of course - and there’s far more than the messenger effect to take into account - but that’s where I’m going with the whole feasible and desirable thing. If we want to change the behaviour of HR professionals, we should use the science of behavioural change.
  • Thanks James.

    In my view, one thing to consider is that, when you start working in this way at least, your methodology doesn't need to be solid you just need to be discerning in how you work with the data you get. You can then use what you learn to develop a more robust methodology.

    For example, one simple thing we did was to flag MOOCS in our monthly L&D mail out using bitly.com to track click through. It was quite rough and ready but we got a sense of topics people were curious about and that online learning was worth pursuing still. We also saw a significant increase in people proposing online learning as a solution to learning needs in their personal development plan (PDP) this year. That might be due to various things but combining that data with anecdotal evidence and the click through data created a better indication that people are keen to know more and we have a (slightly) better sense of the kinds of things to focus our time on sourcing, and that we should also trial providing support to help people research and work with these learning resources themselves on the other.
    The question of how to measure use of external resources is still a challenge but in the process of writing this I've realised we can evolve our PDP process this year into a L&D survey which should help with that - so thanks for asking your question!
Reply
  • Thanks James.

    In my view, one thing to consider is that, when you start working in this way at least, your methodology doesn't need to be solid you just need to be discerning in how you work with the data you get. You can then use what you learn to develop a more robust methodology.

    For example, one simple thing we did was to flag MOOCS in our monthly L&D mail out using bitly.com to track click through. It was quite rough and ready but we got a sense of topics people were curious about and that online learning was worth pursuing still. We also saw a significant increase in people proposing online learning as a solution to learning needs in their personal development plan (PDP) this year. That might be due to various things but combining that data with anecdotal evidence and the click through data created a better indication that people are keen to know more and we have a (slightly) better sense of the kinds of things to focus our time on sourcing, and that we should also trial providing support to help people research and work with these learning resources themselves on the other.
    The question of how to measure use of external resources is still a challenge but in the process of writing this I've realised we can evolve our PDP process this year into a L&D survey which should help with that - so thanks for asking your question!
Children
  • Just possibly to stir the pot by linking:

    www.aconventional.com/.../is-evidence-based-hr-another-fad.html

    Just musing that a lot (most?) of management - including but not 'just' HRM - involves motivation of human beings, and, whilst I'm all for this being based on (social-) scientific inquiry and evidence etc I think usually the relevant / available evidence will tend to conflict and / or be inconclusive or arguable - simply because there are only in the present state of our knowledge about human motivation, there exists only a motley body of conflicting theories and by no means any adequate scientific model. Even if one existed, applying it to particular, unique, situations would raise many judgement calls too?

  • Liam, I completely agree with James, examples like yours show how practitioners really can be evidence based. Thanks for sharing. If you are up for it, a new post or blog recounting your journey in taking an EB approach to L&D would be brilliant. Email me if I can help j.gifford@cipd.co.uk

  • Hi David, I think the motivation issue is an important one, but no more so for this approach than for any other - any approach demands taking the time to build influence and engagement and I would argue that an evidence based approach his self-serving in that respect as it brings confidence and therefore control, which we know to be crucial to influence.

    What I sense from comments from practitioners about getting to grips with evidence-based practice is that they experience a kind of paralysis when it comes to getting started with working in this way.

    What i understand Rob, Johnny et al. to be advocating is a simple set of considerations for gathering evidence in the first place (e.g. whatever is available from various sources of evidence such as professional experience, organisational data...), and then another set for putting them into practice (e.g. identifying a clear set of goals with which to focus) combined with a self-aware attitude to analysing and presenting the data one obtains.

    I sense that practitioners are stymied by reading from the academic origins of this approach that there is a 'proper' way of approaching things, when actually there's a good element of just getting one's hands dirty and trying stuff out - essentially what organisations do anyway but with slightly (and increasingly) more method to the madness.

    I also think that the fashionable focus on 'big data' is part of the issue. It is overwhelmingly challenging to work with unless you have the resources of a tech co like Citymapper whose business model is focused on exploiting it.

    We need to get over that hurdle to move this forward. I think a great starting point is for HR practitioners to talk to colleagues in marketing, for example, who are used to working with both quantitative and qualitative data in their work, which like HR is not an exact science.
  • Thanks Johnny. Time's a little tight at the moment but I'd be happy to write a post.
  • Hi David,

    The blog reminded me of Adam Grant, who does a great line on the 'narcissism of small differences' - parties who share similar views but bicker relentlessly on the details, and therefore halt the progress of their shared goals.

    Evidence-based HR vs. 'Experimental HR' or 'Scientific HR' strikes me as a conceit - and a potentially damaging debate to hold in front of an already confused audience. I don't think anyone's suggesting there can be certainty in human behaviour; everything is inconclusive. But this model is about trying to reduce that empirically, and - as Rob says - typically finding out we didn't know as much as we thought along the way.

    You're right there are no perfect answers or models, and much will depend on context. But we can still strive to better, more informed ways to gather knowledge and make judgements in our workplaces.

    The scientific method has done us alright elsewhere - I hope we could all agree we should take from it everything we can.
  • Bloomin' Google - always one step ahead!

    Nice story about accessible EBHR...

    rework.withgoogle.com/.../