What’s the deal with ‘evidence-based practice’? Aren’t we all evidence based?

Part of the challenge is that it can be hard to find practitioner-focused evidence. A lot of academic research is aimed at other academics (rather than people managers) and adopts a language that many HR professionals find hard to understand. Also, the research often follows fads and fashions, so we don't necessarily get the evidence base in the right areas of people management.

Yes I completely agree Charles. A lot of academic research is not practitioner-focused, and for good reasons. Similarly the language is not helpful - often not even to other academics! And fads and fashions are just as evident in academic research.

On the other hand, academic research is just one source of evidence and like any source of evidence it needs to be judged for it's relevance and trustworthiness. And it's also about using the best available evidence. So even if the evidence isn't great it's still worth checking out.

As you probably know, more and more academics are becoming concerned about scientific (mal)practices. There's a great summary here by John Antonakis where he discusses what he calls the five diseases of academic publishing:

retractionwatch.com/.../

Feels like this is on a trajectory already Rob.
I like the idea of seeking out the 10 most difficult problems or questions - maybe CIPD could survey members to determine.

Another idea for helping EBP support HR, would be for the world of academia to lend their support and services directly to HR practitioners 'in the field' through knowledge transfer partnerships.

Thanks Jonny, I haven't seen the Ted Talk - one for the commute home. I had seen Kahneman's mea culpa (#mancrush). What a guy ;)

I reckon you're spot on - appealing and do-able are great words (as are desirable and feasible from the Rubicon Model). I also reckon the tendency of evidence based peeps is to lean towards the latter (i.e tools and resources), and the tendency of less credible solutions is to lean towards the former (for obvious reasons).

So, 'appealing'... how far do you go? Likening 'test, learn and adapt' methodology to the agility of startup cultures that big corporates so admire? Tie up with publications like HBR? Go full on with branding academic research like this from Francesa Gino... hbr.org/.../let-your-workers-rebel.

I know this will make many academics squirm, but I also supsect that practioners subscribing to the continuum approach may sympathise. I saw a criticism of an IOPsych approach as 'old wine in new bottles' recently, and thought, 'if the old wine is good and the new bottles sell more of it, what's the problem?'

So maybe the 10 questions will work, but maybe that offers an alternative answer. What are the solutions that EBHR are most confident about? Which will be easiest to land in a big organisation? How can we repackage those in the sweet spot to be attractive to a practioner audience?

Another Kahneman fanboi, here. Great to see James raise (and Jonny acknowledge) his important contribution to this field, especially in commercial HR.

I recently sat down with Jacques Quinio of Manpower Group who's an evangelist for EBHR and has some great insights on how organizations can make practical use of big data, but I challenged him a bit on how much harder it is for SMEs to acquire people data on the scales necessary to make really evidence-based decisions. Both in terms of the quantity of data we have available and the time we have within which decisions have to be taken, we don't have that luxury. HR practitioners in SMEs, with small teams, most of whose time is occupied with transactional pratice, have to straddle the line between data-led and instinct-led decision-making and Kahneman is our guru for this.

Our instincts are fallible, but time spent considering the available evidence will inform and improve our instinctive decision-making.

I'd take issue with Jonny's assertion that "it gives little more than a 50% chance of success". Not on the 50% figure (I'll take that), but on the idea that HR decision are fail/success binary options. We aren't financial market traders for whom the buy/sell decision is a straightforward win/lose binary state. Our decisions are far more nuanced. Do we hire X or hire Y? If we make the "wrong" decision, we still end up with a qualified, capable employee (most of the time) but just one who might not have been as good as the alternative.

That's not to say that HR is incapable of making business-breaking decisions, but they are usually at the tail-end of a series of failures made at executive level (q.v. bhs).

As a parting thought, if we assume that EBHR is the pathway to the most effective decision making, to what extent will it therefore be possible for even the most nuanced and sophisticated HR decisions to be automated, given sufficient data of the appropriate quality?

Robey said:

"...if we assume that EBHR is the pathway to the most effective decision making, to what extent will it therefore be possible for even the most nuanced and sophisticated HR decisions to be automated, given sufficient data of the appropriate quality?"

That is a question, Robey.

I recall having this exchange about 'empathy' with Peter Cheese and (deleted blog).

So as we often argue when training and teaching is that EBP is definitely not about certainly but rather trying to reduce uncertainty. The more we find about something the more we realize (usually) that our we know less than we thought.

I don't think there's any need for a compromise between progress and rigor - EBP is about using the best available evidence and being clear about the quality of that evidence. So you can still do stuff without much if any good quality evidence - but the point is you know that's what you're doing.

And more generally, in relation to 'progress', clearly the more better quality evidence you have better-informed your analysis and action is likely to be and the more progress you'll make. What feels like progress may sometimes turn out to be going backwards.

I think academics should do more - the problem is we are completely disincentivized to do this. All that counts is publishing new papers and new research.

And as for knowledge transfer partnerships these are often about conducting new research and new data collection - which is not what EBP is about. Rather, in relation to the scientific evidence part of EBP, we always start with systematic reviews of the existing evidence-base not new research.

Thanks for clarifying that Rob, that's interesting.

Feels like a gap does exist there though, pre-EBP, to be the catalyst for establishing more and good quality evidence.

Totally agree with all.

The better quality evidence you have the better informed your analysis and action is likely to be and the more progress you'll make.

But how do you encourage practitioners to adopt that approach? By changing their minds? By giving them tools and resources? It's very 'System 2'...

Should that great work be bolstered with the sorts of techniques management consultancies or national governments are using to sell us their ideas or change our behaviours? When we know these are more likely to have an effect?

That's the compromise I'm referring to, and it's the bit I think will make academics nervous. Maybe rightly so, maybe not?

Nice comments. Making better decisions is the key thing. And not every problem need a meta-analysis to solve, let alone big data. But there are other things we can do on a smaller scale.
So, the 50% argument is basically descriptive, to say that technically speaking, some evidence is no better at predicting outcomes than the toss of a coin. It's to recognise that we all like to latch on to evidence that supports our pet theory, especially if it tells an intuitive story, but don't kid yourself about being evidence based if the evidence is not trustworthy.
It's a very different point from marginal gains and what decisions are important. Gary Klein has done some really interesting work on this - he argues that some decisions probably ARE best made by the toss of a coin, if you've eliminated the no-nos and are quibbling over minutiae. I enjoyed this book of his wordery.com/the-power-of-intuition-how-to-use-your-gut-feelings-to-make-better-decisions-at-work-gary-klein-9780385502894 . Incidentally, Klein wrote a paper with & gets name-checked by Kahneman in T,F&S. He's good, even if his book has a corny title.
Don't think Kahneman is a champion of instinct-led decision-making, tho. If anything, the opposite!
One other thing I'd flag is the different sources of evidence - in particular there's published scientific research as well organisational data and expertise. Someone will be posting more on that here soon I think...

I have a working example of 'reducing uncertainty'. I was recruited in part to progress our organisation's use of technology to support learning. The results of my initial research into its wider use by other organisations were highly inconclusive. There appears to be little reliable evidence to support the use of technology to enhance learning within organisations so I found myself drawing on reports from consultancies and literature from research on its use in academic contexts.
This didn't stop us moving forward, it just meant my recommendation was to move forward with caution, namely to take small steps, gather data about it as we went and definitely not spend money on a bright new shiny LMS!
After a year I've collected small amounts of data from a handful of experiments and other sources to identify what has worked well, and crucially what we could have reasonable success in trying to measure this year. From that I've made a proposal for how to progress over the next 12 months, which I'll do with a little (only a little) more certainty.

I have a working example of 'reducing uncertainty'. I was recruited in part to progress our organisation's use of technology to support learning. The results of my initial research into its wider use by other organisations were highly inconclusive. There appears to be little reliable evidence to support the use of technology to enhance learning within organisations so I found myself drawing on reports from consultancies and literature from research on its use in academic contexts.
This didn't stop us moving forward, it just meant my recommendation was to move forward with caution, namely to take small steps, gather data about it as we went and definitely not spend money on a bright new shiny LMS!
After a year I've collected small amounts of data from a handful of experiments and other sources to identify what has worked well, and crucially what we could have reasonable success in trying to measure this year. From that I've made a proposal for how to progress over the next 12 months, which I'll do with a little (only a little) more certainty.

What a great example. We often think a big question - like 'how do we improve learning in our organisation'? - needs a big answer - like 'a £1m LMS system'. There's a great book our by Owain Service from The BIT, that aims to debunk just that...

www.behaviouralinsights.co.uk/.../

I bet there is plenty 'small steps' work out there that never gets captured or shared. Especially when competing with the glossy campaigns of big corporate HR and their consultancies.

Perhaps you could outline one of your experiments, Liam? What approach did you take to measuring your interventions and their impact on learning? What were your results?

Jonny/Rob - the salesman in me thinks that retelling and heroising some of these stories/examples alongside tools and resources might be impactful.

Exactly. In the absence of strong or reliable evidence you can still proceed but, as you say, more cautiously than if that wasn't the case.

Thanks James.

In my view, one thing to consider is that, when you start working in this way at least, your methodology doesn't need to be solid you just need to be discerning in how you work with the data you get. You can then use what you learn to develop a more robust methodology.

For example, one simple thing we did was to flag MOOCS in our monthly L&D mail out using bitly.com to track click through. It was quite rough and ready but we got a sense of topics people were curious about and that online learning was worth pursuing still. We also saw a significant increase in people proposing online learning as a solution to learning needs in their personal development plan (PDP) this year. That might be due to various things but combining that data with anecdotal evidence and the click through data created a better indication that people are keen to know more and we have a (slightly) better sense of the kinds of things to focus our time on sourcing, and that we should also trial providing support to help people research and work with these learning resources themselves on the other.
The question of how to measure use of external resources is still a challenge but in the process of writing this I've realised we can evolve our PDP process this year into a L&D survey which should help with that - so thanks for asking your question!

Hi David, I think the motivation issue is an important one, but no more so for this approach than for any other - any approach demands taking the time to build influence and engagement and I would argue that an evidence based approach his self-serving in that respect as it brings confidence and therefore control, which we know to be crucial to influence.

What I sense from comments from practitioners about getting to grips with evidence-based practice is that they experience a kind of paralysis when it comes to getting started with working in this way.

What i understand Rob, Johnny et al. to be advocating is a simple set of considerations for gathering evidence in the first place (e.g. whatever is available from various sources of evidence such as professional experience, organisational data...), and then another set for putting them into practice (e.g. identifying a clear set of goals with which to focus) combined with a self-aware attitude to analysing and presenting the data one obtains.

I sense that practitioners are stymied by reading from the academic origins of this approach that there is a 'proper' way of approaching things, when actually there's a good element of just getting one's hands dirty and trying stuff out - essentially what organisations do anyway but with slightly (and increasingly) more method to the madness.

I also think that the fashionable focus on 'big data' is part of the issue. It is overwhelmingly challenging to work with unless you have the resources of a tech co like Citymapper whose business model is focused on exploiting it.

We need to get over that hurdle to move this forward. I think a great starting point is for HR practitioners to talk to colleagues in marketing, for example, who are used to working with both quantitative and qualitative data in their work, which like HR is not an exact science.

Thanks Johnny. Time's a little tight at the moment but I'd be happy to write a post.

Hi David,

The blog reminded me of Adam Grant, who does a great line on the 'narcissism of small differences' - parties who share similar views but bicker relentlessly on the details, and therefore halt the progress of their shared goals.

Evidence-based HR vs. 'Experimental HR' or 'Scientific HR' strikes me as a conceit - and a potentially damaging debate to hold in front of an already confused audience. I don't think anyone's suggesting there can be certainty in human behaviour; everything is inconclusive. But this model is about trying to reduce that empirically, and - as Rob says - typically finding out we didn't know as much as we thought along the way.

You're right there are no perfect answers or models, and much will depend on context. But we can still strive to better, more informed ways to gather knowledge and make judgements in our workplaces.

The scientific method has done us alright elsewhere - I hope we could all agree we should take from it everything we can.

Bloomin' Google - always one step ahead!

Nice story about accessible EBHR...

rework.withgoogle.com/.../