Teaching Evidence-Based Management

Teaching evidence-based management to undergraduates - part 2

• Season 1 • Episode 2

Share episode

This episode continues our discussion about teaching evidence-based management to undergraduates (listen to Part 1 if you haven't already!).

We discuss how to ensure that the next generation of managers is not just well-versed with the theories, but also equipped to make evidence-based decisions in the real-world, developing the skills they need to challenge the current way of doing things (with humility!). We bring together experienced teachers from renowned universities like Tilburg, Colorado College, James Madison, Carnegie Mellon and KU Leuven, to uncover the strategies and challenges in teaching evidence-based management to undergraduates.

Our guests share their experience of using pre and post-tests to measure the effectiveness of teaching and how to create an aspiration among students to continue using evidence-based practices beyond their education - tapping into the ways in which they already use multiple sources of evidence. We discover that an evidence-based approach is giving students a framework for navigating the often overwhelming amount of information, facts and truths available to them - helping to combat the increasing level of uncertainty and ambiguity they otherwise face in the world.

Join us as we unravel the complex world of teaching evidence-based management to undergraduates, filled with practical tips, expert advice, and much more.

Host:
Karen Plum

Guests:

Mentions:
Eric Barends, Managing Director of the Center for Evidence-Based Management

CEBMa Teachers Network

CEBMa’s Online Course Modules

Karen Plum:

Hello and welcome to the Teaching Evidence-Based Management podcast. This time we continue our look at teaching evidence-based management to undergraduates. If you haven't listened to part one yet, you might want to go back and play that one first. So this is the second part of our discussion about undergrads, for whom evidence-based management seems a critical part of helping them think differently.

Karen Plum:

Although most undergraduates, particularly those that study immediately after school, don't have experience of the world of work, evidence-based management seems to be a tool that can help prepare them for a role in that world, one which could certainly benefit from improvement in terms of how decisions are made. And although they may not fully appreciate the nature of how decisions are typically made by managers, they are living in a world where the amount of information, opinion and so-called facts, can be overwhelming. It's all too easy to Google any topic and leap on the first result, believing it to be accurate and reliable. Undergraduates are also not strangers to multiple sources of evidence. When I was making CEBMa's podcast series that accompanies the evidence-based management online course, Rob Briner, Professor of Organizational Psychology at Queen Mary University of London, pointed out that most of us, including undergrads, are used to using sites like TripAdvisor or TrustPilot before making a decision about a restaurant, for example. I'd asked him how normal, busy humans can make better use of evidence, and here's what he said.

Rob Briner:

How can normal busy humans use evidence better? Well, I think one of the main ways is they can think about the contexts and situations in which they do try and use more evidence better. My current favourite example possibly because we're kind of slowly, at least in some parts of the world, escaping lockdown is how you might do something like choose a restaurant. So if you're going to choose a restaurant say, for example, someone's going to buy you dinner - eating is really important to you, as it is to me and someone's offering to buy you a great dinner - I think you would observe a busy person making good use of evidence. They will probably look across multiple sources. So they wouldn't just look at maybe I don't know one particular guide book like Zagat, maybe they look at TripAdvisor, but they'd also be aware that all these sources are quite questionable. They may look at a guidebook, they may look at a website, they may look at other kinds of guides, but they'll typically look across multiple sources of evidence to say, okay, I don't know this city, someone's buying me dinner, I have to choose a restaurant. Where am I going to go? So multiple sources is one of the principles. Critically appraising, looking at the quality of that, being aware that, for example, all reviews may be biased that, for example, TripAdvisor might be biased.

Rob Briner:

But you're trying to triangulate, you're looking across these sources and also doing it in a semi-structured way, such that you might start to get the names of some places to go to have dinner and you start to write them down and you start to see if people agreed or not agreed when you looked across different kinds of reviews. So I think one way of getting people to use it in their busy everyday lives is to think, well, how important is this decision to you or to your organization or to whoever it is? And if it's important, I think we can all think of examples where we do take the use of evidence a bit more seriously and just I would say, just apply those principles to other kinds of decisions.

Karen Plum:

Interesting to hear Rob talk about emerging from lockdown. That seems such a long time ago, but his advice is still right on the money. Most people, undergrads included, are more evidence-based in their everyday lives than people used to be. We all have access to a world of different sources of information. The challenge is knowing which of those sources are trustworthy and reliable. So using this sort of everyday example helps to connect undergraduates to the evidence-based management approach. If you're evidence-based in your choice of restaurant, surely you'd want to use multiple sources of evidence when making decisions as a manager. It's generally felt that undergraduates have high expectations of what the world of work is going to offer them, and they want to make a solid contribution and to make a difference when they do join an organization. So learning the skills required to make better decisions and how to apply this approach seems to me to be a sound way to prepare them for the future.

Karen Plum:

I'm Karen Plum, a student of evidence-based management myself, and in this episode we dig a little deeper into how teachers approach this subject for undergraduates in the company of experienced teachers Brigitte Kroon from Tilburg University in the Netherlands. Christina Rader from Colorado College in Colorado. David Peterson from James Madison University in Virginia, Denise Rousseau from Carnegie Mellon University in Pennsylvania and Jeroen Stouten from KU Leuven in Belgium. You can find their details in our show notes. In part one we talked about how the teachers approach evidence-based management in their courses. We start part two by looking at how the teachers judge the success of their courses. Naturally, there are course requirements in terms of scores, grades or whatever, but do their courses have a broader impact on their students' attitudes, perspectives and, essentially, their way of thinking and approaching problems? To get us started, here's Denise.

Denise Rousseau:

If I might say one thing that I try is to do a pre and a post test separate from the modules that looks at first their self-identified level of expertise and their approach to solving a problem. And then post test self-identified expertise. - do they feel they've acquired experience and increased their level of expertise in evidence-based practice and then a parallel problem to solve? And one of my observations has been people who thought they were experts at time one are knocked down to intermediate by time two and feel successful about that. The other thing is that the far more elaborated task strategies in the post test for how to frame a question, how to gather information, how to - I like David's word reason - about what interpretation you would give. So just to try to do that.

Denise Rousseau:

I like to use that because it gives me information of how completely I was able to, through the process, knit an approach to problem solving together. The other thing is to know about their own self-identification, about level of expertise, because that's actually something I think feeding back to people about where we go from here, because most people in the post test will not say they're experts. Thank God, because I'm not an expert, but I do think the idea of creating aspirations for what I'll do after my schooling is done, is perhaps an important part of this process.

Karen Plum:

Yeah, excellent, anybody else?

Brigitte Kroon:

Yeah. So no, besides the normal testing and grading and seeing whether they hit the boxes, as well as comes to the learning objectives of sub courses, we evaluate qualitatively to see if students like it, if they do see that the things they learn in university make sense. And I did before I ran the, especially the course where we used the modules, I ran a pilot with a group of students and they were extremely enthusiastic and some of them said I know I finally understand why I'll do all these methods courses and why I need theory, and it really was also going to help me in my research and in practice. And we do only parts of the modules, but of course we do offer all of them. Another indication of its success is also the percentage of students who complete all the modules to also get the certificate, and in the beginning my expectations were not that high, but there are definitely a substantial percentage that engage in, that complete all the modules.

Brigitte Kroon:

I think, that indicates something.

Karen Plum:

Right, right. What about you Christina?

Christina Rader:

I was going to say what we're not doing, that I wish we were, which was being able to see whether they are using these skills once they leave. So because you know, like David was sharing, my favorite things are the anecdotes that I get back from students about how they used it. Can you share one? I have some students working for our career center, so they're still here on campus working for our career center and they took the course with me in two different semesters, so it's a set of four students and they're working on a project for how to help first and second year students come use the career center. And they said "we said we have to do this using evidence- based management and that's just you know, made my year.

Christina Rader:

They were using it immediately and yeah.

Jeroen Stouten:

Yeah, maybe just one thing about the last question. What I think students learn and we see them for quite a bit over time, is the more consistently used language that fits the appraisal of the evidence, and that's something they start off with not thinking about at all. So they would say they would read a qualitative research paper and say X as an effect on Y, and over time you'd see that they're more careful in the language that they use when they address certain findings and I think that is that's a sign that they picked up something - being more critical about what they've read and how they present it.

Karen Plum:

Yeah, well, as they learn, as I learned when I was doing the course, so often and I think that some of you said this earlier, we use terms interchangeably and we talk about terms that have no agreed definition and yet we all speak about them as if they did. And when you've tripped up on that and that's caused you a problem in your understanding or the results of the outcome that you've come up with, then you are going to naturally become more cautious over your use of language. And I think you're right Jeroen, it's a really useful skill to start developing, because you realize the importance of communication and the clarity. I mean, I'm a change manager and you know, you know you have to be clear with people, you have to be able to explain the why of the change and just to make sure that everybody's clear and on the same page, because if everybody in the change project is using different language, it's really confusing and then you don't take anybody on the change. Anyway.

Christina Rader:

I just thought before we go to the next thing, one to pull together what Jeroen said and David said. David was talking about the difficulty of effect sizes, and I loved the French fries example - you have a small, medium or large! And I think another thing that's a huge distinction for students to learn is there's a difference between the absence of an effect and the absence of evidence, or strong and weak. So in other words, there can be strong evidence for no effect, or that there can be no evidence of any effect, or there can be weak evidence for a strong, you know, for a large effect and just getting people's brains around, that is huge! Or your head spins.

Christina Rader:

then doesn't it? Absolutely especially because the way the media, which is where we get a lot of our findings that is just not part at all of what.

Karen Plum:

No, that's not a headline, is it? You've got to have a punchy headline even if it's wrong, even if there's no evidence for it.

Christina Rader:

So just distinguishing between the strength of the evidence and the actual effect is huge.

David Peterson:

On the metrics, on the how do we know, how do we judge success? I wanted to share this. I recently put my tenure packet together last year and I had to go through and calculate what am I, you know, what are my teaching scores and I compared them over time and to our department average. And we consider ourselves we're a balanced school, so teaching is very important. I think we have pretty good teachers. And I noticed the effect size on one of the questions, which was "this course has helped me understand problems and solutions in new ways", basically given me new ways to think. That score for my evidence- based courses was, speaking of effect sizes, a large effect - larger than the rest of our department, and it didn't matter what I assumed about the variance there, for computing the effect sizes, it was not just a large effect, but it was like a D of like two or something, and that didn't happen until I got the book. Before that I wasn't doing so well.

David Peterson:

When I got the book and started to focus on that framework, all of a sudden, compared to all my other courses and the rest of my department and we've about 800 management students so compared to that, this course was helping people to see problems differently by very, very, this is super sizing the effect now.

Karen Plum:

This is extra large fries then! Brilliant, I love that. Ok, so let's move on. Obviously, we know there are some teachers who'll be thinking about whether they should cover evidence-based management in their courses, and some that haven't even considered it yet. So what piece of advice would each of you give those teachers who aren't teaching evidence-based management yet, or perhaps those that are, but who are finding it a really tricky subject to cover? Christina, what do you think?

Christina Rader:

I think I have three pieces of advice. So one is what we call scaffolding, or supports, or - everything that they do, we give them a worksheet, or we give them something that helps them with the steps, because we're teaching them things that took us six to ten years to learn and we still don't know it. That's point number one. I think the second point would be, we have to let go of perfection. So again, we've spent six to ten years and we can see all the things that really stay in focus. Like if I get them saying 'association' instead of 'effect', I am just so happy, right. If I get them only drawing conclusions and recommendations that come from the evidence, I am so happy.

Christina Rader:

And then I think the third thing is just keeping connected for them, which they see pretty quickly, but not always, so, keeping connected for them, this has a real world application. Sometimes when you're deep in the CAT, so we do CATs and now we do just mini CATs because of the tight time frame we're on. But sometimes when you're deep in the CAT, the students can forget that they're studying something real world and it starts to feel just overwhelming. And so, coming back to the real world connections.

Karen Plum:

Wonderful. So who's next - David.

David Peterson:

I would do exactly what Christina said, to start, everything that she just said. I was trying to think of what I could add to that. I would say if you're getting into to teach this, go through the modules yourself, really - I have learned so much going through, especially when I first started to go through and took notes, and as there have been revisions, I have learned a lot about how to communicate. Like Christina was saying, we've spent years doing these things, but being able to do it isn't being able to communicate it, and so learning what the students, what their worldview is, what they do and don't understand in terms of terminology and in being able to translate the stuff that we do or that we're reading about into their language, first requires get in and really understand. There's a lot in those modules, and so I think the more that we can get in as educators and that's true for any topic, but this in particular, because it's so new and to the students. Brigitte what about you?

Brigitte Kroon:

I back up what David was saying about doing the modules yourself helps a lot. Also if you're new to evidence- based management. What I find difficult sometimes is that I have quite a number of skeptical colleagues who are debating that this is, of course, rational decision making. We're trying to put things in a process that is, in reality, change is much more complicated. So I'm kind of fighting all these prejudices continuously. I know the students like it, but I would say that the struggle to convince everybody else is sometimes hard.

Karen Plum:

Interesting. What about you, Jeroen?

Jeroen Stouten:

Well, I agree with what's been said. I think what I possibly could add is think about it as a decision that you're taking and the process helps you to give you lots of information to base your decision on. But at the end it's your decision. So if you were to look at that process, it's a learning experience for you, but also for the students, and it's your own discretion to make a decision that you feel comfortable with. So I don't know, I think it takes a little pressure off in the sense that there's not really one answer or one decision to take in a particular situation. So don't feel you have to find that what's it called, Holy grail, but make a decision you feel comfortable with the information you've got.

Karen Plum:

Right, okay.

Denise Rousseau:

I think that one of the important aspects of this it's related to the rationality concern, but that is that we use evidence in a social setting and so what we do and how we approach others or raise issues with others or ask questions, sort of needs to be socialized in a sense. We don't want to turn people into the evidence police because well, a) you know it's ineffective and b) it's disrespectful and that for our process that we're training people in to be of use, it needs to really take context and kind of the meanings that people ascribe to things and what's hard in the social setting into account, because evidence is not answers, it informs how we approach a solution or how we diagnose a question, and so a level of humility, which is often absent in the real world of non evidence- based decision making, but.

Denise Rousseau:

If we can't commit that sin of not being humble and respectful of the environment. And to help people learn how to raise questions and share information in a way that doesn't hit the other person on the nose! And that's a skill it's also - people have to want to do that, you know, and I see this as a democratic project to encourage participation and engagement and not just blind faith and authority or prior beliefs. But to, as any change project, one needs to be sensitive to you know what the stakeholders need, which also includes, I was listening to Brigitte's comment, our colleagues who in other courses are going to have students who ask questions they didn't ask before, that might be seen as a challenge by our fellow faculty members. So helping them ask questions in a way that is a genuine question and not an attack, I think it's really valuable.

Karen Plum:

That's respectful, a bsolutely. In terms of the teachers that you teach, Denise, is there anything else that you would add that you think is particularly pertinent for us to include in the episode?

Denise Rousseau:

Well, I thought that what David commented on about having frameworks and tools that people could use and adapt as part of the teaching, is really valuable. We're trying to gather this kind of information in a teacher's handbook that teachers are providing through the Center to share with each other, and I would say watch that space and participate in this community so we can share these kinds of ideas. There's a lot of really lovely pedagogy being developed and I'm really excited to learn about that and to help share it.

Karen Plum:

Super. Before we finish, is there anything that any of you want to say that you haven't had a chance to, that you think it would be valuable to include? Let me just go around - David, is there anything?

David Peterson:

Yeah, I'm looking at my list here. So when I got your email to participate in this, I asked my students and said, hey, first I asked them, should I do this? And they were all yes, yes, you should absolutely do this. I had already sent an email back to you saying I was going to that point. But I said, what should I shar e? I said our audience will be from this, will be other people who will be teaching this to undergraduates. What should I share to them?

David Peterson:

And we've covered some of the things that my students came up with, but a couple of things that we haven't is that, looking at their context, their world, there's a lot more information available nowadays and it's overwhelming. We talked about there anything that they've gotten through primary and secondary school in terms of information literacy, and they said it's all been piecemeal. It's been a little bit here, a little bit there. People talk about it, but they haven't ever gotten anything that really helps them sort through all that information. And so they said what this provides, and one of the things that we, as teachers of evidence- based management, can help communicate to them is that this framework, these modules, if you look at the big picture of how this all comes together - and I think it's what module two it's,

David Peterson:

We have an assumed problem and we ask questions to get at that and look for evidence - did we get the problem right? And then we have the preferred solution and those two words assumed and preferred are so important - preferred solution. And then we look at evidence and from multiple sources again and how well supported is that? That's it. It's that it really is that simple at the grand scale and they said that to them is what is really transforming their own personal lives. You know our students think estimates now are the anxiety is one of their biggest concerns in student mental health for undergraduates, and a lot of that comes from ambiguity and its overwhelming. How do I get through all the noise that's out there? So I'd like to start by telling them this right here is one of the answers to that is this can give you a process that's concrete and allows you to navigate all this ambiguity and assess it.

Karen Plum:

And that ambiguity, that anxiety is hugely draining on our brains, on our resources, isn't it - on our energy?

David Peterson:

Yeah. My students thought that was really helpful.

Karen Plum:

Yeah. Okay, were there any other points that they raised that you wanted to cover, David?

David Peterson:

They love the modules. They love the interactive - I mean surprise right, teaching someone something in an evidence- based way, works. They really love the interaction, the interactive nature of the modules and I use the learning dashboard a lot. They like that, that I'll use that before coming to class to say, okay, you got this, so we're not gonna cover that so much, but you struggled. And they like to see what other people struggle with and that there's some like oh, okay, it wasn't just me, I can see right there, we all really stunk at that part, so they like to see that.

Karen Plum:

It's another way to learn too, isn't it? To see what other people have taken from the material. Yeah, okay, so, Jeroen, anything that you feel that we haven't covered that you'd like to cover?

Jeroen Stouten:

Yeah, maybe just one thing. We talked a lot about the different steps of acquiring and appraising the evidence. I think what certainly helps for students and since they're studying this as a good point to start is have a good foundation of theory, because it helps them to make the steps easier. It helps them, to what David already said, to link concepts together, to link practical terms to scientific terms. They are not always the same or they may be understood differently, and knowing what these are, seeing relations and different types of literature, is also helpful to sort of know how to navigate the field and to research more effectively, I think.

Brigitte Kroon:

I'm going to first say thank you to Jeroen, because I have exactly the same perspective on this. I feel that it all starts with knowing your theories, and then you can find out where the evidence is. Because the theory will help you to understand the mechanism that links the X and the Y. And maybe, as a final thing, I really think that students forget if they have one course and then in the next course they don't remember what they learned in the previous course. So repetition throughout the program, if it's possible to include that, I think that's a strength. So we literally made one of the learning lines called evidence-based HR and consulting and that all the courses tied to that.

Karen Plum:

I was very struck by how the teachers reflected that it wasn't surprising that students found evidence-based management challenging. They had many years of experience practicing and teaching, and were still learning themselves. They stressed the importance of context for the students and it's obvious that until students start to work in an organization, they'll lack the understanding of why different problems are of concern to managers, why they find them challenging and why it's often quicker and easier to make a snap decision based on gut instinct. Particularly as in a lot of situations, managers don't actually have to live with the consequences of their decisions because things move on, or the next management fad or trend arrives, or it's their staff that pay the price of their poor decision-making in terms of their development, progress or well-being. I think everyone was amazed by David's teaching scores. His students said the evidence-based management course had helped them understand problems and solutions in different ways, giving them new ways to think. It seems that this is a real way to counter the feedback Brigitte receives from students that they learn nothing practical at university. What better skills than critical thinking, asking critical questions, diagnosing the actual problem rather than the assumed one, and identifying solutions based on evidence, and learning. T o do this in a respectful, inquisitive way, rather than attacking people. As Denise said, we don't want to turn people into the evidence police, but to help them become gentle, respectful advocates for a different way of thinking, with a healthy degree of skepticism and an unwillingness to accept things just because they're written in a textbook or stated by their teachers, famous academics or indeed by powerful managers and leaders when they reach the workplace. As Brigitte said, teachers in other faculties are finding themselves being asked questions they've never been asked before. Perhaps that's not comfortable for them, but it shows that the seeds are being sown.

Karen Plum:

Unless undergraduates learn how to think critically and approach problems differently, things in organizations won't change, and because the world is changing so fast, we need undergraduates to bring a new way of doing things with them when they enter the workplace. The other worry, as David said, is that students are keenly aware of the risks of fake news and of the overwhelming level of suspect so-called facts or alternative truths that do the rounds on social media every day. Evidence-based management is a way to help them navigate that terrain and look at the world differently. As such, it's more important than ever that undergraduates are exposed to an evidence-based approach, because university is a formative time and we have the opportunity to embed it before they're exposed to the status quo that exists in so many organizations. Let's get them while their minds are still open, before they start to form bad habits.

Karen Plum:

And that's the end of this two-part episode on teaching evidence-based management to undergraduates. Many thanks to Brigitte, Christina, David, Denise and Jeroen for their time and for their insights, which I hope will give everyone food for thought and reflection. My thanks also to Rob Briner for his great restaurant example back in 2021 and to Eric Barends, Managing Director of the Centre for Evidence-Based Management, for sharing his thoughts about undergraduates, which informed this episode. If you'd like to hear future episodes of Teaching Evidence-Based Management, you can like or follow the show. If you're interested in joining the CEBMa Teacher's Network, head to the website cebma. org and go to the Services page. Thanks so much for listening. See you next time. Goodbye.

Head Shot

Karen Plum

Host