• Replies 56 replies
  • Subscribers 3 subscribers
  • Views 19247 views
  • Users 0 members are here
Options
Related

What’s the deal with ‘evidence-based practice’? Aren’t we all evidence based?

Jonny

To an extent, yes - we all use evidence. But as we argue in our positioning paper, In search of the best available evidence, there are two big things we tend to get wrong when using evidence to inform decisions.

First, we’re often not great at gauging the quality of evidence we’re looking at. There is a well-established hierarchy of scientific evidence on cause-and-effect relationships. The ‘gold standard’ is randomised controlled trials, which carry a good deal more weight than, say, simple before-and-after studies, and far more than surveys run at one point in time. If we can take note of this hierarchy in looking at evidence, we are well on the way to making more reliable, better decisions.

Second, we tend to cherry pick evidence that supports our pet theory. It feels great when you find a piece of research that confirms what you long suspected. But barring the ridiculous, the chances are you’ll be able to find research – even good quality research – to back your opinion whatever it is. To find out if a technique is really worth replicating, we should look at the wider body of evidence. So sitting above the hierarchy of single studies, we have a ‘platinum standard’ of systematic reviews and meta-analyses.

Evidence-based HR means being anti-fad and willing to face uncomfortable truths. But it’s hugely important. Relying on weak evidence can feel politically expedient (staying in line with what your boss expects to see) or compelling (in tune with the zeitgeist, intuitively right), yet at its worst it gives little more than a 50% chance of success, the equivalent of flipping a coin. If a decision is important, it’s worth testing the available evidence in this light: how much more scientific is it than a coin toss?

There are plenty of thorny questions in evidence-based HR but the basic principles are pretty simple and more importantly, worth striving for. Our hope is that this forum will help put people these principles to work and grapple with the challenges. Thoughts?

Parents
  • Part of the challenge is that it can be hard to find practitioner-focused evidence. A lot of academic research is aimed at other academics (rather than people managers) and adopts a language that many HR professionals find hard to understand. Also, the research often follows fads and fashions, so we don't necessarily get the evidence base in the right areas of people management.
  • in reply to Charles
    Yes I completely agree Charles. A lot of academic research is not practitioner-focused, and for good reasons. Similarly the language is not helpful - often not even to other academics! And fads and fashions are just as evident in academic research.

    On the other hand, academic research is just one source of evidence and like any source of evidence it needs to be judged for it's relevance and trustworthiness. And it's also about using the best available evidence. So even if the evidence isn't great it's still worth checking out.

    As you probably know, more and more academics are becoming concerned about scientific (mal)practices. There's a great summary here by John Antonakis where he discusses what he calls the five diseases of academic publishing:

    retractionwatch.com/.../
  • in reply to Mark
    I think academics should do more - the problem is we are completely disincentivized to do this. All that counts is publishing new papers and new research.

    And as for knowledge transfer partnerships these are often about conducting new research and new data collection - which is not what EBP is about. Rather, in relation to the scientific evidence part of EBP, we always start with systematic reviews of the existing evidence-base not new research.
  • in reply to Rob
    Thanks for clarifying that Rob, that's interesting.

    Feels like a gap does exist there though, pre-EBP, to be the catalyst for establishing more and good quality evidence.
  • in reply to Rob
    Totally agree with all.

    The better quality evidence you have the better informed your analysis and action is likely to be and the more progress you'll make.

    But how do you encourage practitioners to adopt that approach? By changing their minds? By giving them tools and resources? It's very 'System 2'...

    Should that great work be bolstered with the sorts of techniques management consultancies or national governments are using to sell us their ideas or change our behaviours? When we know these are more likely to have an effect?

    That's the compromise I'm referring to, and it's the bit I think will make academics nervous. Maybe rightly so, maybe not?
  • in reply to Robey
    Nice comments. Making better decisions is the key thing. And not every problem need a meta-analysis to solve, let alone big data. But there are other things we can do on a smaller scale.
    So, the 50% argument is basically descriptive, to say that technically speaking, some evidence is no better at predicting outcomes than the toss of a coin. It's to recognise that we all like to latch on to evidence that supports our pet theory, especially if it tells an intuitive story, but don't kid yourself about being evidence based if the evidence is not trustworthy.
    It's a very different point from marginal gains and what decisions are important. Gary Klein has done some really interesting work on this - he argues that some decisions probably ARE best made by the toss of a coin, if you've eliminated the no-nos and are quibbling over minutiae. I enjoyed this book of his wordery.com/the-power-of-intuition-how-to-use-your-gut-feelings-to-make-better-decisions-at-work-gary-klein-9780385502894 . Incidentally, Klein wrote a paper with & gets name-checked by Kahneman in T,F&S. He's good, even if his book has a corny title.
    Don't think Kahneman is a champion of instinct-led decision-making, tho. If anything, the opposite!
    One other thing I'd flag is the different sources of evidence - in particular there's published scientific research as well organisational data and expertise. Someone will be posting more on that here soon I think...
  • in reply to Rob
    I have a working example of 'reducing uncertainty'. I was recruited in part to progress our organisation's use of technology to support learning. The results of my initial research into its wider use by other organisations were highly inconclusive. There appears to be little reliable evidence to support the use of technology to enhance learning within organisations so I found myself drawing on reports from consultancies and literature from research on its use in academic contexts.
    This didn't stop us moving forward, it just meant my recommendation was to move forward with caution, namely to take small steps, gather data about it as we went and definitely not spend money on a bright new shiny LMS!
    After a year I've collected small amounts of data from a handful of experiments and other sources to identify what has worked well, and crucially what we could have reasonable success in trying to measure this year. From that I've made a proposal for how to progress over the next 12 months, which I'll do with a little (only a little) more certainty.
  • in reply to Jonny
    Exactly. Evidence-based practice IS ABOUT PRACTICE - not about evidence or research. In other fields I get the impression that researchers tried to push or impose 'their' research onto practitioners. That doesn't work. Need to start with practice, practitioners, and real problems.
  • in reply to James
    Hi James - yes - great point. Nobody 'disagrees' with evidence-based practice so, as you say, why is the alternative so compelling? I think in part its about the way practitioners get rewarded and the marketing ability of HR consultancies and others who push products and services onto HR.
  • in reply to Robey

    Hi Robey.  I'd love to know if Jacques Quinio is an evangelist for evidence-based HR in the way evidence-based practice is generally understood.  Does he just mean big data (which is not what evidence-based HR is about) or does he actually mean this:

  • in reply to Rob

    Sorry that was way too small!

  • in reply to Rob
    I'd hate to speak for Jacques on the basis of a 20-minute seminar and 15-minute face-to-face. Looking over my notes, though, his point on Big Data seems to have been less that it was the answer to all our woes and more that HR practitioners are ignoring its potential, seeing it as a sales/marketing tool rather than a people tool. If I look at the other topics we covered, and particularly at the approach to employee-led performance appraisal, I'd say that the overall push was in the direction of, but not identical to, the EBP infographic.

    For example, he was quite a fan of psychometrics as a tool for employee development, but my understanding of the literature on that subject from practising academics in neurology and psychology (as opposed to former academics in those fields who've developed a product to sell) is that psychometrics are basically pseudoscience.

    However, when dealing with any consultant - especially one at the top of his game - you always have to remember that they too have a product to sell.

    Your point about the emphasis on practice may be semantic, but it's very important and not a perspective I'd thought of before.
Reply
  • in reply to Rob
    I'd hate to speak for Jacques on the basis of a 20-minute seminar and 15-minute face-to-face. Looking over my notes, though, his point on Big Data seems to have been less that it was the answer to all our woes and more that HR practitioners are ignoring its potential, seeing it as a sales/marketing tool rather than a people tool. If I look at the other topics we covered, and particularly at the approach to employee-led performance appraisal, I'd say that the overall push was in the direction of, but not identical to, the EBP infographic.

    For example, he was quite a fan of psychometrics as a tool for employee development, but my understanding of the literature on that subject from practising academics in neurology and psychology (as opposed to former academics in those fields who've developed a product to sell) is that psychometrics are basically pseudoscience.

    However, when dealing with any consultant - especially one at the top of his game - you always have to remember that they too have a product to sell.

    Your point about the emphasis on practice may be semantic, but it's very important and not a perspective I'd thought of before.
Children
  • in reply to Robey
    Thanks interesting points. I'm not sure I agree that psychometrics = psudoscience as there are so many different types of psychometrics with varying validity and reliabilty.

    Yes, it's ALL about practice and not about evidence as such. But it's also about doing stuff first to discover the evidence about what the real problems/opportunities may be and only then thinking about what you might do. Things like 'big data' are sold as sort of non-solutions to non-problems though of course data can help.