What are we really teaching?

What are we really teaching when we teach evidence-based management?

Teachers often hope their students leave with more than knowledge of concepts and frameworks. But how do we know whether learning has changed the way people approach problems, make decisions, and reason through uncertainty? And what do students carry into practice once the course has finished?

This episode explores those questions through two pieces of research. The first examines the learning outcomes of students who completed CEBMa's online Evidence-Based Management course. The second looks at whether participants approached management problems differently after studying the evidence-based decision-making approach, using a think-aloud protocol analysis conducted as part of a wider research project at OzChild in Australia.

Along the way, the discussion considers learning versus performance, what teachers might notice when students begin to think differently, and whether evidence-based management is teaching more than a set of tools and techniques.

Host: 

Karen Plum

Guests: 

• Denise Rousseau, H J Heinz University Professor, Carnegie Mellon University, Pennsylvania, USA
• Eric Barends - Managing Director, Center of Evidence-Based Management
• Erin Czerwinski, Manager, Learning Engineering, The Simon Initiative, Carnegie Mellon University
• Dr Fiona Chatteur, Adjunct Senior Research Associate, COCA, Torrens University, Australia 

 Links mentioned in the episode:

• “Taking the course” episode of the Evidence-Based Management podcast (that accompanies the online course)
• Details about CEBMa's online course in Evidence-Based Management
• The e-Textbook on Evidence-Based Decision Making (2026): https://cebma.org/resources/cebmas-e-textbook-evidence-based-decision-making-for-organizations-public-policy-and-practice/ 

Contact for more information:

Eric Barends, Managing Director of the Center for Evidence-Based Management

Transcript

Copy Transcript Copied

What Are We Really Teaching?

Karen Plum 0:02

Welcome to the Teaching Evidence-Based Management Podcast. I'm Karen Plum, a student and devotee of the practice of evidence-based decision-making. And in this episode, I'm going to explore the question of what we're really teaching when we include evidence-based management or decision-making in our courses. Clearly, it isn't about learning and remembering information to be demonstrated in a final dissertation or exam. There may not even be anything monumental that marks the end of the course material with a fanfare and hearty congratulations. Oftentimes, the evidence-based element is part of a broader management program, which may include some testing or examination of the student's approach to the decision-making process, but isn't the core of what they'll need to demonstrate in order to graduate or complete their master's or doctorate. Not in all cases clearly, but perhaps for the majority. Recently I talked to a number of teachers who cover the evidence-based approach in their courses. It became clear to me that although they were dedicated to this topic, recognised the value of the approach, and were in many cases pioneering something without an enormous amount of support from their institutions, many harboured a nagging question. How do I know if what the students have learned will stay with them and be useful in their future organizational life? What signs are there that the teaching and learning experience has been successful? I think it's important to all of us in the evidence-based movement that the practice spreads, so that the quality of decision making improves over time. And therein lies the first challenge, because teachers may never see the fruits of this labour, which may only show up months or years into the future. The seeds sown during this period of study may have different opportunities to be used, depending on the organizational power, politics, hierarchy, and so on. All of that said, I want to start this episode by looking at the way the evidence-based approach is taught. There are a variety of options. Some use the CEBMa 15 module online course, originally based on the book written by Denise Rousseau and Eric Barends. The book's also now available as a fully interactive e-textbook. You may know this already, or this may be new news. Stay with me as I hope to share some background and information about how the course works, which might broaden your understanding of the learning science. We'll also learn more about how changes in reasoning can be assessed. Let's get going.

Learning Versus Performing For Grades

Karen Plum 3:27

Five years ago, as part of the original Evidence-Based Management podcast that accompanies CEBMa's online course, I talked to the authors, but also the learning scientists that designed the platform that hosts the course. The purpose was to explain to students the science behind the course, which is very much a learning by doing approach, which encourages a learning approach and not a performance approach. The aim was to help students understand the journey they were embarking on, and how, unlike many other courses, getting the right answers, or being disheartened by not getting them, wasn't the point of the course. Naturally, nobody was saying rejoice in being wrong, but the point was to explore and challenge ways of thinking. The course doesn't glory in how you perform, so much as gently move you towards an openness to being more comfortable with uncertainty, to embrace the complexity of using multiple sources of evidence, of evaluating the trustworthiness of the evidence used, and so on. The episode was a preparation for the learning experience, somewhat unlike many others the students of whatever level might have encountered before. To encourage them not to be disheartened, to realize that it's not an easy process, but there's huge value in learning the approach and embracing a different way of approaching decisions. If you've never heard the episode, there's a link in the show notes if you'd like to check it out. But to give you the idea, here's Carnegie Mellon Professor Denise Rousseau explaining the difference between learning and performance.

Denise Rousseau 4:42

What we've learned recently in learning science is that there's a big difference between learning and performance, that you have to learn to perform better. But many of our participants are used to trying to perform to get the answer. And that really isn't the point. The modules are designed to help people understand the fundamentals and practice under different conditions, different problem sets or different cases to use the concepts so that they get a chance to expand the connections and their understandings to actually make them work in the day-to-day. In order to do that, you have to have an open mind to what works where and why. And that is inherent in learning research is the idea of exploration, that mistakes are a way to get feedback. And that feedback isn't about the right answer, it's about the why.

Trustworthy Evidence Starts With Asking

Karen Plum 5:48

While the CEBMa course teaches students about the full process of collecting, evaluating, and combining four sources of evidence, not all university or college courses cover all of that material. The option to complete the full online course nevertheless exists for most students who have access to it, regardless of whether their university or college curriculum requires them to do so. So it seems worth sharing some of the thinking behind the online course, which was designed to prove that online learning could be as effective as face-to-face tutoring, as explained by Erin Czerwinski, Learning Engineer Manager at the Simon Initiative at Carnegie Mellon University. It turns out that it is.

Erin Czerwinski 6:33

We wanted to figure out how can we make or prove that online learning, if designed a certain way, could really be just as effective as face-to-face or even tutoring, one-on-one tutoring. And we did actually prove that out. And it's because we used what we knew about the science of learning and essentially, the principles that are in a book called How Learning Works that was written by a bunch of folks, but many of them were at Carnegie Mellon University at the time, and really taking what we learned in the research at Carnegie Mellon and distilled that out into actual design principles and ideas that any teacher or designer could use when building any kind of learning, actually, but it turns out these principles bear out just as well in face-to-face as in online learning. And what those are, essentially, you learn by doing, making sure that practice is tied to goals. We also pay careful consideration to the learning objectives, making sure that they're articulated in a way that are student-centered and measurable, so that you know learning when you see it, because you're measuring it that accurately.

Karen Plum 6:33

I guess one of the biggest learning moments for me when I took the course is that it taught me how to not just collect, but to evaluate the trustworthiness and robustness of each form of evidence. I couldn't simply grasp for what I thought was the right answer. I had to really think about why I'd drawn my conclusions, whether they were in fact right or wrong in terms of the totality of what I could have taken into consideration. As Denise said earlier, it's about the why. There's quite a lot of discomfort in recognising that the way you've been thinking and deciding all your life might have led you to make some pretty questionable decisions. Denise explains the problem.

Denise Rousseau 6:33

All of us have habits of mind about how we think about information, truth, what sources we respect, the opinion leaders who we grant credence to, the assumptions we make about what senior leaders know or don't. What is an expert? We have a lot of pre-existing beliefs. If we were blank slates, it would be a different world, but we are not. We come with many embedded concepts. And then you're exposed to evidence-based approaches that put great attention on figuring out two things. One is the trustworthiness of the information you possess. Trustworthy means, well, based on certain conditions about features in the science about controls, features in opinions and expertise about prior experience and the validity of the experience, qualities of reliability and validity in your own organizational data, you're learning across all of these different types of evidence, what makes it more or less reliable, more or less valid to test the claim you might make. And many people aren't used to doing that amount of work, thinking about what they think about. You know, is this really reliable? Is this something that really I should act on? And that's self-implicating. Second one that is challenging for people is becoming aware that very often when we're trying to make a decision or trying to make sense of information, we start too far down the path. One of the critical issues in evidence-based practice is the concept of ask. What questions are important to the problem or the situation that we face? What's the real problem we're trying to solve? For many of us, well, for all of us at some point in time, we think we know what the issue is. I want to fix this, I want to get this thing removed. But is what I'm focused on really the problem that needs to be solved.

Does The Online Course Work?

Karen Plum 10:54

Some years on from when the original course was launched, what's clear is that it hasn't remained the same. It's been reviewed in line with emerging information about what's worked for the learning process. The online course metrics show the learning scientists how students reacted and responded to the exercises and challenges that occur throughout the program. The course has also been revised to reflect changes such as the availability of AI tools when searching for appropriate academic research, and developments in thinking about the best ways to aggregate multiple sources of evidence. And most recently, the interactive e-textbook provides another way for students and teachers to interact with the course material. A little more about that later. So the course that teaches students how to take an evidence-based approach is, in itself, evidence-based and evolving, as you'd expect. And to answer the inevitable question, what's the evidence that the course modules are effective? Several studies have been looking into this. One study looked at nearly 900 bachelor and master's students in universities and business schools in North America, Australia and Europe, plus executive students and professionals taking part in development programs within their own organizations. The online course has a pre- and post-test built in to see what students have learned and retained at the end. While around 2,000 students took one or more of the modules over the time period, the research focused on those that completed at least 70% of the pre-test questions, and who also completed the corresponding questions at the end of the course. And although this was a convenience sample, the students turned out to actually be a very good representation of the wider student body that studied the course. Here's Eric to explain what was found when they looked at the difference between pre- and post-test results.

Eric Barends 12:57

So if we look at the effect size, the difference between the pre and post, that's a really, really high difference that suggests, and that I was absolutely pleasantly surprised by that finding, because there's so much information in those chapters, in those modules. And you can hardly expect from students that they recall all this, because what you learn in module two is probably out of your brain when you're at module nine. And then when you move to module 14, will you still remember what was there? So it's amazing how much how much stuff is there. So that means in terms of factual knowledge, what we teach in the modules, what is an effect size? How what kind of research design is this? Stakeholder values, that all that factual information kind of sticks in their brain. So the first step in in effectiveness research is of course, did they did do they have the information? Did they learn anything? And the second question is then, well, do they apply it? Because you can know it and you can learn it, but what do you do with it in practice? And maybe you don't do that.

Karen Plum 14:11

And for those of you that like effect sizes..

Eric Barends 14:14

So the effect size we call that a Cohen's d. That is a typical effect size that is about uh differences between pre and post or differences between group, but the Cohen's d was 1.17, and that is a lot because in educational interventions in general, the Cohen's d is about 0.4, so that's almost almost three times as high as average educational interventions. So that's that's indeed very high.

Karen Plum 14:48

So this is all about recalling the facts, the steps, the approach of evidence-based decision making, but also the pre- and post-tests are looking for evidence that students are exercising some judgment. They aren't simply yes or no answers. I was curious about whether the results applied equally to all types of students, from the undergraduates through to master's students and experienced managers. For Eric, the results here were probably the most surprising outcome of the research. The assumption had been that given the different life stages, experiences, and motivations of people taking the course, teaching different types of students would require different types of examples, so real-life examples as opposed to organizational ones, where knowledge of organizational life would be needed. And that's why the course has both. But..

Eric Barends 15:42

But the outcome is actually that there is no difference in effectiveness of the course when it comes to these groups. Both undergraduate students, 18-year-old, young people with no experience learn just as much as a person that has already 20 years of experience, maybe already an MBA, but now does this program to deepen their professional knowledge. What does that mean? It took us off guard, but that means apparently that they may have sometimes a little bit more knowledge to start with, so they score a little bit higher on the pre test. But apparently the whole idea of evidence-based decision making is a challenge for everyone. It's it's not when you work in organizations for a longer time, it's easier or you know more. No, it's you're exactly on the same level, and the struggles are exactly the same, and also the learning benefits are also the same for undergraduates or masters or even DBA students.

Karen Plum 16:54

I think that's such an interesting finding, that students struggle just the same with all these new concepts and approaches, and that the course is effective at teaching those aspects. The application of that knowledge is of course another matter and a subject for another podcast, but I found this as exciting as I suspect the researchers did. And for teachers using or considering the online course or the e-textbook, it's valuable evidence from an educational quality control perspective. These results could strengthen the case for including evidence-based decision making in management or change management or HR courses, for example. If you're interested in the course, there are some links in the show notes. And before we move on, a few words about the new e textbook that I've mentioned during this episode.

E-Textbook Options And Online Beats Lectures

Karen Plum 17:45

I was curious to know why it's been produced and how it differs from the full online course.

Eric Barends 17:51

We did that because we feel it is helpful if teachers, universities have choices. Obviously, it would be great if all universities in the world would use our online course modules, but there's not always the budget or the means or the possibilities to do that. So I think a second best is then an interactive e-text book, which is different than a paperback book with papers. It is really interactive, so it is very close to the online course. I mean, you could do both on your iPad and there's hardly a difference.

Karen Plum 18:29

Eric explained that the full online course allows universities to see how students have scored and where most mistakes are made, which can be helpful for teachers. For students, once the course is complete, they lose access to the online course modules, whereas they can download the e-textbook and always have access to it. It'll also benefit from being updated whenever a new version is available, just like a software update. So as Eric said, these are different options with their own distinct benefits. Turning back to the online course, Eric also told me that a comparison had been done between online learning and traditional teaching methods, the aspect that Erin spoke about earlier. A small randomized controlled trial was carried out with a group of students from NYU Wagner School of Public Service, focused on one of the modules, the acquisition of scientific evidence. So it was quite a technical activity, very hands-on for the students.

Eric Barends 19:32

The idea was that we would have several classes, uh, groups, and we would randomize them to two conditions. One is they do the online course on this specific module, how to search for research findings, and the other one would be me teaching, and I consider myself a brilliant stellar teacher. So I thought, let's raise the bar. I'm great at this. And it turned out that the students that did the online course were actually better and scored higher on the post-test than me teaching.

Karen Plum 20:06

For Eric, it reinforced the importance of students being able to practice on their own and to do the work privately. Naturally, it's important to say this was a small sample and only for one type of activity. But nevertheless, the online course compares favorably with conventional instruction. It seems positive in the sense that it's available for teachers to use as part of their courses.

The Hard Part Is Real-World Use

Karen Plum 20:33

What interests me beyond this is whether students are able to move beyond the pure understanding of process and concepts and to approach problems in an evidence-based way. Do they approach problems differently? Naturally, the two broad categories of students are in quite different situations. Those that are involved in day-to-day decision-making as part of a career, and those for whom taking an evidence-based approach may so far only apply to their understanding of types and quality of evidence, improving their personal decision-making. For this second category, early career students or inexperienced workplace practitioners, their opportunity is to cultivate curiosity and reflective judgment, as opposed to trying to change ingrained habits and reliance on their personal career experience and gut instinct. In a learning sense, the tension between performance and developmental thinking seems to be present no matter the type of student, such is our hunger for getting the right answers. In the 2021 podcast, Denise Rousseau commented on the tendency for experienced professionals to have quite firm views about their capabilities.

Denise Rousseau 21:48

Oh, that's always one of the challenges older professionals who have notions of their capabilities. I, for example, teach physicians, medical doctors, in evidence-based management because they're moving into managerial roles. And if they don't get 95% of the answers right on the evidence-based management modules, they go ballistic. So my job is to calm them down and say, what really matters here is that you're learning. Are you learning new things? Okay. Yeah. Can you reflect on what it is you're learning? Not so much what it takes to get a better score, but what's the basis of the answer and what makes it right? And why is that information you can use? That's sufficient. Our goals are lifelong learning as an evidence-based practitioner, and having the tools that come from the evidence-based management class positions you on that trajectory.

Karen Plum 22:52

So the big question for me is what happens to the way students think after they've studied the evidence-based approach? Do they reason differently? In a way, it's what any trainer would be curious about following a course. What stuck, what survived, what's going to be useful to those students in the future. And that often is lost as soon as the exam results are published and we rarely know any more, at least on any consistent basis. Teaching reasoning is different to teaching knowledge after all. Given that this is a question that occupies a lot of thought among evidence-based decision-making educators, I'm delighted to be able to share the results of a small

Measuring Reasoning With Think Alouds

Karen Plum 23:33

experiment in Australia which may provide not only some light and encouragement, but something that could be useful to teachers. Let me give you a little background about the research, which was commissioned by the CEO of Child Welfare Organization OzChild. The CEO is Lisa Griffiths, a passionate advocate of evidence-based practice, and contributor to several episodes of our Evidence-Based Management podcast. Having studied and practiced the approach within the organization herself, its practice spread as more leaders trained in taking an evidence-based approach. As Lisa said to me on one occasion, when your managers start to ask you, what's the problem you're trying to solve, you know things are changing. The research was trying to understand whether the learning delivered by the course translates into practice, in this case within OzChild. What particularly caught my eye was the work done as part of this mixed methods research to see whether those studying evidence-based practice showed any differences in how they think and reason. And I was interested because it's something I hear in conversations with teachers. How do we know that what we teach sticks and carries into practice? The research was carried out by the Centre for Organisational Change and Agility at Torrens University in Australia. And I talked to lead researcher Fiona Chatteur and asked her about the approach they use to identify changes in thinking and reasoning, what they call the "think aloud" protocol, which is analyzed by a protocol analysis. So what is that exactly?

Fiona Chatteur 25:17

It's basically a way of engendering reflection in action. So you're reflecting as you're as you're doing a task. The people are given a task. They are asked to vocalize their thinking out loud, which is often quite, if you've got anybody watching you, it's quite awkward, so we always turn off the cameras. Vocalise your thinking out loud, and that is then recorded and analyzed. And what it allows to happen is that you've got qualitative data, which is the think aloud, the spoken word, which can then be quantified. So you can quantify it into numbers and then do a statistical analysis on those vocalizations, which is fascinating.

Karen Plum 26:00

Fiona explained that what the participants say during the think aloud activity is captured and analyzed according to keywords and categories that are representative of whatever's being studied. In this case, the categories are aspects of evidence-based decision making. So how do the think aloud sessions work?

Fiona Chatteur 26:20

We were using a quasi-experimental approach. So we did all of the think alouds, uh, we did two groups, a control group and a intervention group, the group who did the course. All of them had a think aloud before the intervention. And then we waited eight months while they did their course. And then after they completed the course, we did another tranche of think alouds, both the control group and the intervention group, to see whether there was any difference.

Karen Plum 26:46

The groups were given a general management scenario, designed actually by Eric Barends and pre-tested with his own students. Different scenarios were used for the pre- and post-think aloud sessions, and the results were super interesting. There were differences in terms of the sources of data mentioned by the control group versus the course participants, but it went further than that.

Fiona Chatteur 27:10

There was more asking, there was more acquiring of data, there was more reference to scientific data, there was more reference to stakeholders and practitioners. And if you looked at the word clouds, you could actually see the they're talking about it more frequently. The frequency increases. So you look at it at the data and you go, actually, I can see, I can see in the word clouds that there's a difference. And it was quite, it was quite um, it's quite wonderful. I call it a change in thinking. A change of approach to the problem solving. They're relying on on more um talking to the staff and talking to the stakeholders and talking to the practitioners and and also looking for the evidence in um they referred to white papers, they referred to um evidence, you know, peer peer-reviewed research a lot more. Um so they're actually going back to the basics in evidence-based decision making.

What Changed In How People Decide?

Karen Plum 28:03

What stood out was how far towards reaching a decision each of the groups were prepared to go. The control group relied on organizational data and their experience to make a decision, whereas the course participants didn't want to move to a solution until they'd examined the data which they needed in order to make a decision. They were more cautious, and it resonated with work Fiona had seen on design research from the 1990s, where three stages were identified: novice, intermediate, and experienced designers.

Fiona Chatteur 28:39

The intermediate designers spent a lot of time gathering information, and it took them a lot to get into the implementation kind of field. And I think there was a correlation, I mean, thinking about it today, there's a correlation between that, that once you've you learned something and you're you know you've got a certain amount of skills, you're a bit cautious to step forward into the results because you're not you're not a n ovice where you don't know anything and you're not super experienced because you're new to evidence-based decision making, you're in it at an intermediate stage where you want to gather a lot of data. And this is very clear.

Karen Plum 29:14

It's important to say that this research had a small sample. So although the results are encouraging, it wouldn't be wise to draw massive conclusions from it. I also wanted to be sure that what Fiona and her colleagues saw in the think aloud activity wasn't simply people using more evidence-based language.

Fiona Chatteur 29:33

I would argue that if you're using evidence-based language when you're trying to solve a problem, you're using evidence-based thinking.

Karen Plum 29:41

And I think it's important to recognise that the think aloud activity is all about processing and thinking in the moment, to solve a problem, to address a scenario, not simply to talk about taking an evidence-based approach. For me, this research provides some quantifiable evidence to sit alongside what teachers may instinctively already be seeing in their students. They notice that students don't leap to solutions when presented with a problem or a scenario, which they may have done at the start of the course. They're more cautious, less certain, more open to gathering different sources of evidence before forming a view and making a decision. Courses that teach the evidence-based decision-making approach are fundamentally teaching students how to think, not what to think.

Eric Barends 30:50

We teach students, or we hope to teach students, how to think and not so much what to think. It's a sort of metacognitive skill. And I think it all boils down to what Denise always says. If

Eric Barends 30:50

I can choose between an evidence-based manager that knows everything about effect sizes, confidence intervals, biases, and outcome measurement, or the evidence-based manager that can ask really good critical questions, then I would always go for the one asking the critical questions.

Simple Ways Teachers Can Spot Change

Karen Plum 31:12

That said, if teachers want to do something more intentional to see whether students are changing their thinking styles after learning the evidence-based approach, Fiona has a suggestion.

Fiona Chatteur 31:23

Yeah, I mean, they might want to give them a scenario after and see whether they can notice. Now, maybe without the in-depth analysis that I've been through, but uh they might want to see, do you hear any of the phrases in the discussion? Maybe open up a class discussion with the scenario and see and see people bounce ideas around. And are you hearing more use of evidence-based, an evidence-based approach in the way that people attack the problem? Are you seeing the request for peer-reviewed research or white papers or statistics or whatever? Are we seeing that from the from the participants? Are we seeing them consider asking stakeholders and asking practitioners and gathering data that way? Are we seeing are we seeing that sort of evidence? Are they getting three sources of research? Are they appraising that research? Are they crystallizing that into a course of action? These are all the things that you would need to look out for.

Karen Plum 32:21

Eric also mentioned a course at Fontis University in Holland, which includes an in-person verbal exam, where students answer questions about what they would do faced with a given scenario, to demonstrate how they would approach the problem and reason their way through it. But of course, it all depends on the nature of the course, the time available, the cost implications, the number of students, etc. Every course is different. By way of a wrap-up, I return to the original teacher concern, that they never fully know what students carry into their practice. I often talk about planting seeds and never knowing when they'll grow and flourish. But if we look closely, perhaps students may reveal what's changing by whether they ask different questions, whether they exercise less certainty, whether they notice their own propensity to look for quick, neat answers rather than exercising more curiosity and critical thinking. Things they might have previously overlooked, as they focused more on performing than learning. And I think I'll give the last word to Eric to share his own journey.

Eric Barends 33:31

Well, let me tell you, my my struggle over the past 15 years is that when I started this and I came together with all these highly trained and knowledgeable academics, I remember that Denise, and all these people said, yeah, evidence-based practice, it's it's an attitude. And I thought, no, it's not an attitude, it's a skill set, something you do, and you can learn, you do this, you do that. Now we've gone full circle, and actually say, Yeah, you can teach them skills and steps, etc., and techniques, but at the end, it's indeed an attitude, it's an approach. It's it's about having an inquisitive mindset, asking, well, how exactly does this work? What do we know? Where are what are we uncertain about? What does this mean? Is this applicable? Where does this come from? Yeah.

Evidence-Based Practice As An Attitude

Karen Plum 34:22

And that's it for this episode. If you'd like more information, there are links in the show notes, and you can send us a message or a reaction to the episode if you'd like. My thanks to all the guests that contributed to this episode: Erin Czerwinski, Denise Rousseau, Fiona Chatteur, and Eric Barends. And thank you for listening. See you next time. Goodbye.