56

What’s the deal with ‘evidence-based practice’? Aren’t we all evidence based?

Jonny

| 0 Posts

CIPD Staff

25 May, 2017 14:55

To an extent, yes - we all use evidence. But as we argue in our positioning paper, In search of the best available evidence, there are two big things we tend to get wrong when using evidence to inform decisions.

First, we’re often not great at gauging the quality of evidence we’re looking at. There is a well-established hierarchy of scientific evidence on cause-and-effect relationships. The ‘gold standard’ is randomised controlled trials, which carry a good deal more weight than, say, simple before-and-after studies, and far more than surveys run at one point in time. If we can take note of this hierarchy in looking at evidence, we are well on the way to making more reliable, better decisions.

Second, we tend to cherry pick evidence that supports our pet theory. It feels great when you find a piece of research that confirms what you long suspected. But barring the ridiculous, the chances are you’ll be able to find research – even good quality research – to back your opinion whatever it is. To find out if a technique is really worth replicating, we should look at the wider body of evidence. So sitting above the hierarchy of single studies, we have a ‘platinum standard’ of systematic reviews and meta-analyses.

Evidence-based HR means being anti-fad and willing to face uncomfortable truths. But it’s hugely important. Relying on weak evidence can feel politically expedient (staying in line with what your boss expects to see) or compelling (in tune with the zeitgeist, intuitively right), yet at its worst it gives little more than a 50% chance of success, the equivalent of flipping a coin. If a decision is important, it’s worth testing the available evidence in this light: how much more scientific is it than a coin toss?

There are plenty of thorny questions in evidence-based HR but the basic principles are pretty simple and more importantly, worth striving for. Our hope is that this forum will help put people these principles to work and grapple with the challenges. Thoughts?

4520 views
  • Heather

    | 0 Posts

    CIPD Staff

    25 May, 2017 15:03

    Great stuff - looking forward to the discussion!
  • Enjoyed the challenge to this posting, I was nodding in agreement when I read...'first, we’re often not great at gauging the quality of evidence we’re looking at. '

    And then feeling the heat of my own tendencies when I read...'second, we tend to cherry pick evidence that supports our pet theory.'

    Fortunately I am pretty anti-fad so that got the casting vote!? 'Evidence-based HR means being anti-fad.'

    But the point of the posting (for me at least) is that evidence based practice is great in theory but we have to keep an eye out for those common pitfalls...
  • Charles

    | 0 Posts

    CIPD Staff

    26 May, 2017 11:37

    Part of the challenge is that it can be hard to find practitioner-focused evidence. A lot of academic research is aimed at other academics (rather than people managers) and adopts a language that many HR professionals find hard to understand. Also, the research often follows fads and fashions, so we don't necessarily get the evidence base in the right areas of people management.
  • Jonny

    | 0 Posts

    CIPD Staff

    26 May, 2017 11:55

    In reply to Stuart:

    Thanks Stuart. I do think these are things we can all fall prey to, so it's a question of calling them out and keeping ourselves in check (and there are methods we can use to do this). I think pretty much all of us are prone to fads, whether it's because we're attracted to the latest thinking, or because we like counterintuitive challenges to received wisdom. Equally though, we can be prone to taking received wisdom for granted - which can be just as misguided. In either case, we're following practices or using tools blindly, with no decent idea of whether they will do what we hope. Not so critical if you're spending £20 on a kitchen utensil, say, but really important if you're putting in place a major organisational change intervention. There are some nice descriptions of what makes a management fad by Danny Miller and colleagues e.g. hbr.org/.../spotting-management-fads
  • In reply to Jonny:

    Spot on! Many of us (OK me at least) are drawn to fads, but I've learnt to dial it back. Rob Briner uses a great metaphor of banana guards, you might want one, but do you really need one? The second question is a playful evidence based one...
  • In reply to Stuart:

    Hi Stuart - yes I agree that being evidence-based means being anti-fad until there is a good quantity of good quality evidence available that the 'fad' is no longer a fad.

    Of course, I'm guessing that the vast majority of fads are not ever found to be sensible.
  • In reply to Charles:

    Yes I completely agree Charles. A lot of academic research is not practitioner-focused, and for good reasons. Similarly the language is not helpful - often not even to other academics! And fads and fashions are just as evident in academic research.

    On the other hand, academic research is just one source of evidence and like any source of evidence it needs to be judged for it's relevance and trustworthiness. And it's also about using the best available evidence. So even if the evidence isn't great it's still worth checking out.

    As you probably know, more and more academics are becoming concerned about scientific (mal)practices. There's a great summary here by John Antonakis where he discusses what he calls the five diseases of academic publishing:

    retractionwatch.com/.../
  • Charles

    | 0 Posts

    CIPD Staff

    26 May, 2017 12:27

    In reply to Rob:

    And not just academic research but conferences as well, Rob blogs.lse.ac.uk/.../
  • Jonny

    | 0 Posts

    CIPD Staff

    26 May, 2017 12:44

    In reply to Charles:

    I also agree Charles. There is a challenge for practitioners to be evidence-based, but also most definitely a challenge for academia - to make research outcomes driven, relevant & accessible. These challenges will never disappear, even if we make decent progress. But one thing that helps connect the 2 domains is decent summaries of the best research for practitioner audiences. So rapid evidence assessments play a really important role as do sites like http://scienceforwork.com/
  • Charles

    | 0 Posts

    CIPD Staff

    26 May, 2017 13:01

    In reply to Jonny:

    To be fair to be academics, some of their research is stymied because they can't get employers to participate in their research. Though that may also indicate how relevant the research is for practitioners. Perhaps we should ask the HR community what 10 pieces of evidence would be most useful for them and then pass this on to the research community?
  • In reply to Charles:

    Interesting idea - I think even more useful would be to ask HR practitioners what are the 10 most difficult problems or questions they face in their everyday work. I think then an analysis of the nature of these problems might indicate what types of evidence might be most relevant or useful for these common practice problems/questions.

    I've been thinking about doing this for, er, only around 15 years or more I reckon. Maybe it's nearly time to get around to it with help from a large professional HR body perhaps?
  • Jonny

    | 0 Posts

    CIPD Staff

    28 May, 2017 10:17

    In reply to Rob:

    I hope we'll hear both in this forum. The challenges people are grappling with is an ideal starting point, but definitely good for practitioners to be talking about evidence too.
    What aspects of people management are most problematic for your organisation?
    On what basis are these identified as issues?
    What evidence would help you progress on these issues?
  • Jonny

    | 0 Posts

    CIPD Staff

    28 May, 2017 10:42

    In reply to Rob:

    Rob, to your point of doing this for 15 years, you've clearly had a lot of influence, so don't underestimate that. But clearly it's not easy to get traction with evidence-based practice. I think we have to reflect on this & ask why it's the case. For example:
    1. EBP can feel like a technical point of methodology and I think many people are happy leaving that to the experts. The line of thought of: surely you've got to at least trust the scientists.
    2. Practitioners need to feel they are central to the solution of EBHR, not just consumers of research. So as you suggest, to ask: what problems do you grapple with? But also: what would help HR in your org be more discerning of evidence, or make better use of it? And: what tools would you like to see to help?

  • In reply to Jonny:

    There's a great quote from Daniel Kahneman that I think goes some way to answering that question. He suggests evidenced based solutions will always struggle to land within the complexity of big business:

    "Experts who acknowledge the full extent of their ignorance may expect to be replaced by more confident competitors, who are better able to gain the trust of clients. An unbiased appreciation of uncertainty is a cornerstone of rationality--but it is not what people and organisations want."

    There's a difficult reality that follows for me - how much compromise (if any) should there be between progress and rigour/credibility? We can say none: we stick to the gold standard of RCT's and communicate with plainly presented objectivity. But we should recognise that the competition from non-evidence-based solutions is highly sophisticated and evolved to be 'sellable':

    Take the '10 questions' approach - the strength of non-EBHR is to answer them with everything we know people lean towards - certainty (this is the one answer that works), norms + loss aversion (everyone else is doing it, you're falling behind), messenger (this guru says it's right) etc etc. Understanding why it isn't easy to get traction with EBHR is partly about understanding why the alternative is so compelling...

    The question then evolves to 'how should we compete'? How can we be heard?' I imagine the subsequent debate to be in line with a respected academic writing a best-selling psychology book... what's the line?
  • In reply to Jonny:

    There's a great quote from Daniel Kahneman that I think goes some way to answering that question. He suggests evidenced based solutions will always struggle to land within the complexity of big business:

    "Experts who acknowledge the full extent of their ignorance may expect to be replaced by more confident competitors, who are better able to gain the trust of clients. An unbiased appreciation of uncertainty is a cornerstone of rationality--but it is not what people and organisations want."

    There's a difficult reality that follows for me - how much compromise (if any) should there be between progress and rigour/credibility? We can say none: we stick to the gold standard of RCT's and communicate with plainly presented objectivity. But we should recognise that the competition from non-evidence-based solutions is highly sophisticated and evolved to be 'sellable':

    Take the '10 questions' approach - the strength of non-EBHR is to answer them with everything we know people lean towards - certainty (this is the one answer that works), norms + loss aversion (everyone else is doing it, you're falling behind), messenger (this guru says it's right) etc etc. Understanding why it isn't easy to get traction with EBHR is partly about understanding why the alternative is so compelling...

    The question then evolves to 'how should we compete'? How can we be heard?' I imagine the subsequent debate to be in line with a respected academic writing a best-selling psychology book... what's the line?

    (Great thread btw!)