It’s Not About the Stats: Reflections on Teaching “Evidence-Based Medicine” 20+ Years into the Movement

Good afternoon everybody so good to see everybody popping up today today's session is is one that has really been on the minds of a lot of folks for a long time some of us know evidence-based medicine but don't necessarily know how to teach it take teach it some of us aren't as comfortable with evidence-based medicine in order to teach it but we're still excellent teachers today we're going to learn the importance of using evidence-based medicine and the importance of really creating an environment supportive for your learner's to use evidence-based medicine again I've been asked to do something along these lines for a long time because you'll see so on the ACGME survey each year all of our residents and fellows are asked if their faculty support an environment of inquiry for example this is one of the ways that we can really show that we support an environment of inquiry by getting into the evidence by get being curious and really seeing what's out there and not just maybe following the same standard protocol so today we are very fortunate to have dr. John Epling with us dr. Epling is a professor and medical director of research and the Department of Community Medicine at vtc and he's also a medical director of employee health and wellness for Karelian clinic and it's wonderful because he also he knows evidence-based medicine very well he also is very comfortable teaching it and teaching how to teach it now going through today's session if you have any questions or comments along the way feel free to use that chat function at the bottom of your screen if you float over the bottom of your screen you should see a little chat bubble click on that and type your questions and then there will be a point at which doctor Ebeling will ask for questions and then we can open it up for horrible questions so I'm just gonna go ahead and turn the mic on over to John thanks all right I think I'm sherry can you hear me okay okay good so thanks for coming this is a never quite sure how it talks about its based medicine I gotta go but we'll see how this goes I'm gonna I'm gonna disappoint sherry I'm not actually that comfortable teaching evidence-based medicine I never have been and I've always been a little bit discontented about how we've been able to teach evidence-based medicine as we go because I and as you'll see in the presentation dear I'm not sure we're always clear about what we want so this is going to be a little bit a presentation this can be intentionally provocative we'll talk about that a little bit and hopefully getting you to think I want you to please as much as you can if you if you have reactions or something use the chat we'll talk about how we're going to use the chat a little bit so I want to have us talk about the meaning and goals and what whatever is based practice is and how we're going to teach to that we'll talk about ideas for creating an environment supportive learning evidence-based medicine and we'll talk about this concept of information mastery those are the have been around Virginia and UVA for a while this concept was originally by one of the family medicine faculty that practiced at UVA for a while and so you may have heard of it we'll talk about what we mean what I mean by that and the difference between that and thinking about an evidence-based medicine curriculum why do you care what I have to say I don't know I have been teaching this awhile back in back in the sort of the mid 90s is when I got interested in this stuff had been teaching in one form or another that whole time a lot of my career has been focused around evidence-based medicine sort of learning research how we research how we teach etc and in this presentation the other point on this slide is to be aware of strong and I'm going to make some completely unfounded sweeping generalizations about how evidence-based medicine is taught now these days so don't take that bait in the chat those meant to be sweeping provocative generalizations instead if something when you're in the chat and talking about things talked about your your what you've done what your experience is those type of things don't don't don't pick me apart the comments for for the sweeping generalizations so let's talk a little bit about meaning in goals this is the the traditional and original version of the definition of evidence-based medicine it's the conscientious judicious and explicit use of current best evidence and making decision about the care of individual patients I don't think we come back to this definition enough and we'll talk about how evidence-based medicine is is perceived around there but I wanted to I want to get these definitions out there a little bit just to make sure we're all starting from the same thing there's nothing in here about learning statistics necessarily I will grant you that you have to be a bit conscientious in order to be conscientious you have to know a little bit about statistics and critical appraisal but there's nothing specifically in here it's use of current best evidence of making decisions and in this same article that I that I quote here on the bottom that define doubt as based medicine there's the define definition of evidence-based practice um practice is something I'll try to use mostly I often just sort of revert to evidence-based medicine I don't mean to be doctor centric about that I think the principles apply throughout evidence-based practice as these guys met it was there was evidence based medicine demonstrates practice this practice is the integration of individual clinical expertise need to find that pretty carefully down below and the best available clinical evidence from systematic research so this integration is the important thing it's this integration to preview some of the talk is where I think we haven't given enough attention when we teach evidence-based medicine that were evidence-based practice by expertise they mean the proficiency and judgment that clinicians acquire through clinical experience in practice a lot of the early definitions tried to distinguish expertise and experience a lot of clinicians when they were initially confronted with this would say well in my clinical experience works I am certain from that the point was made that unless you are systematically collecting that experience and documenting it and studying it you don't there's a lot of things that go into your impression of your own experience that aren't necessarily evidence based so they meant to contrast those things a little bit and when they talk about clinical evidence they mean clinically relevant research especially from patient centered clinical research but also from basic science research so as to so as to be inclusive of all the folks that do medical education they also talk about what evidence-based as medicine is not it's not old hat it's not something that everyone is already doing and then when this came out in the mid 90s that was a lot of the reaction we already do all that stuff we already pay attention to science I think I hope that since then we've realized we don't really pay that much attention to science in clinical medicine there is some some foundation of science it's not always the best current science but but hopefully the rigorous practice of evidence-based medicine is not actually at least the way these guys define what we do they argued in this article that it was not impossible to practice and that it's not confined only to ivory towers and armchairs that you could in fact do what these guys described as evidence-based medicine as in active practicing clinician however it's my view of what evidence-based practice has become and here are some sweeping generalizations just a warning I think it's been a bit of a bludgeon used to shout down people in arguments we used to in an authoritarian manner to say well this is evidence-based and everything else is not evidence-based so we definitely have to do this all the time I think it's become a marketing label I know it's become a marketing label you can you can see evidence-based popped up on you go to any of the publishers bookshelves at publishers book stands at conferences you see evidence-based everywhere it's not always as evidence-based as we'd hoped I think it's become a synonym for guidelines sometimes for randomized control trials that we just sort of assume that if you're following a guideline therefore you're evidence-based without any it additional thought about what that how that guideline was produced increasingly in this land of population health I worry that it's become a focus on some narrow scope quality metrics that if you knew those quality metrics you are a good evidence-based position again without attention to what goes behind those quality metrics and I do think it's become hard to practice we'll get into this in a second and and you know I think as originally maybe described and defined whether it's been relegated to the ivory towers and armchairs it's not so much the issue as to whether those are the folks that have the opportunity to practice full-scope evidence-based medicine so just continuing to be provocative and here's some of the issues you know as when you talked about the biostatistical component of educating evidence-based medicine you talked about things that were relatively easy to understand p-values odds ratios they were at least somewhat straightforward to teach some of those concepts for fine but then we've evolved a little bit and so we've we've had to learn area under the receiver operating characteristic curve and I squared test for a heterogeneity you know these things are the statistics that are used in routinely in evidence-based medicine are becoming more challenging and I will tell you that that the learners even today still struggle likelihood ratios were developed as part of the early part of this movement as a way to interpret sensitivity specificity I am looked at as if I have two heads when I say the words likelihood ratio is still two learners just out of medical school and just out of residency sometimes so I don't think we've kept up with the evolution of some of the statistics that are used quite commonly now same with study design if a case control design is something that's been used if you can't think of any other study design to to say as part of an answer to an exam question it is frequently misinterpreted but we've gone from relatively straightforward RCTs case controls and court studies - now cluster randomized control trials with spouts of a stratified blocker position non-inferiority studies a very popular one in pharma research Network meta analyses again lots of evolution of these concepts with when when learners I think still struggle with the basic concepts and then there's a whole new realm of synthetic research so from systematic review and meta-analysis decision analysis and statistical modeling all of these things are becoming more and more part of what backs up guidelines and backs up other this other synthetic research that I think we can use if we're trying to be evidence-based practitioners but it requires a lot to understand and to keep up with so what is the goal so we've just we talked about the meaning of evidence-based practice and what we're trying to get to maybe I would argue that there's still some unclarity about that but the goal is is really figuring out my I've done two previous teach dogs and they both talked about using objectives to figure out what you're where you're going and I think this is really important I think as Stephen Covey would say I'm gonna and others would say you got to begin with the end in mind that that you really have to understand what you're producing in an educational environment in order to teach it the right way to teach the right concepts so the question is what is our goal in teaching evidence-based medicine nowadays forget what they meant in the mid-90s what is our goal nowadays so we're gonna try this polling issue here let's do this so you should be able to see the poll here so for your learners in your setting what is the most important objective of instruction in evidence-based medicine is it critical appraisal aversions research articles design and implementation of research study use of the most authoritative guidelines of clinical practice Qi in the setting or enabling the learner to endless to argue endless arcane by statistical points I'm hoping there we go there are some responses there go ahead and fill them in there are some we're getting some early responses for enabling the learner that's good there's some quality improvement there you know the results of these aren't so much important but I think it's it gets us to think about this obviously you're gonna have different different goals in your setting and and there may be more than one goal here but I'm just curious about what people would say so we have guidelines use of the most authoritative guideline now I said authoritative for for a particular reason we'll talk about the critical appraisal guidelines but critical appraisal immersion research articles is important and some element of quality improvement that's good I think what it gets conflated as frequently evidence-based medicine is maybe design and implementation of a research study a little bit a little bit too much and then whether or not it's intended I think the the arguments about arcane biostatistical points are often an unintended consequence of this so we'll get back to that those polls in a second and so in the chat I want you to think about what are if we didn't mention any what are your real goals and objectives for teaching your learner's what do you what do you want to get out of teaching evidence-based medicine think about that for your specialty I imagine it differs in fact I know it differs specialty specialty I imagine it differs teacher to teacher so the the folks that teach basic science may have different goals and objectives for this than the folks that teach clinical science and I think it would be useful to talk about that so if you get a chance to participate in the chat and then this is one of those questions for that so what challenges that we faced in teaching this traditional evidence-based medicine stuff I think in medical school and these are again provocative sweeping generalizations based on some of my experience here I think iBM has traditionally been de-emphasized relative to other courses whether that's by the amount of time allocated to it the placement in the curriculum often bundled with a bunch of other courses that I referred a couple of bullets to as wastebasket courses stuff that you can't quite figure out where else to put the curriculum you sort of put it another in one of those those garbage basket courses grading often times grading is sort of iBM is often graded on a pass/fail rather than on a normative system if you have those in your in your school so it's it's the emphasize by virtue of that an interesting phenomenon is it's often it's often substituted for bio stats and epidemiology you can think about traditional courses in both of those things there's a lot of detail that doesn't get covered in evidence-based medicine course that would get covered in bio stats and epi and vice-versa and so I think it's not a clear substitute all the time for traditional bias that's an epi courses and to the extent that your curriculum needs those to be taught it's not always a clear substitute the and then the integration is challenging so it's fun to be able to teach it well in a course and have it valued appropriately but unless it's reinforced throughout the curriculum by integrating that the concepts and practices it won't get used that's part of what we know about how education works and so faculty development is really key in fact the development is its own barrier that creates the sort of feedback loop if we're not if faculty aren't interested don't want to pursue evidence-based medicine instruction then it's going to be hard to integrate we'll go over that a bit in residency some of the same things are true in residency particularly the learner baselines are incredibly variable there are folks that that had zero stood by statistical training or evidence-based based medicine training in medical school to those that have done n PHS and arrived in residency after that and so teaching to that diverse cadre is a little bit difficult it's often lumped in with research scholarly activity and quality improvement so it gets sort of lost in all these things so maybe the the the real conception of what evidence-based medicine is as opposed to some of these other issues gets a little lost and you have the same continuing challenge with faculty development continuing medication admitted my first comment too that I retained which is that I you know it's really hard to teach practicing physicians evidence-based medicine that was not for a lack of trying in the early 2000s and and onward still I think it's what what we've really focused more on in the continuing medical education community is shifting information seeking behavior and and finding shifting reference use and those type of things that's been the the one place that I've really focused more in the CME realm for that I just think it's hard to teach folks that are going 100 miles an hour and clinical practice to adopt a new way of looking at information in 2004 if we talk about a conceptual framework for evidence-based medicine in 2004 the BMJ published a an issue devoted to evidence-based medicine they had on the cover an ivory tower just hammer on the point and Sharon Straus was the lead author on this one she talked about how she envisioned the different roles in how you would practice evidence-based medicine so for a for an individual position and she talked about this more in terms of an individual position switching roles here so they might be in the dual role where for frequently encountered conditions stuff where you really needed to be make sure you were practicing with the highest evidence you would do full scope evidence-based medicine you would ask a question you would acquire the evidence you would critically appraise the evidence you would apply the evidence and then assess how you're doing and that's that's and do the doing that I'm in primary care there are a lot of frequently encountered conditions I can't imagine and in each specialty there's a you're gonna have a lot of work that's associated with that that seems hard and seemed hard at the time she defined a user role whereby a user would use pre appraise sources and other synthetic sort of research to put together things and then a replicator replicator not a replicant that's a different movie a replicator role would be to follow sort of respect to DBM leader so some model your practice after those who you think practice evidence-based medicine and so that was an interesting way to think about it I think it took some pressure off because the initial conceptualizations of evidence medicine work that everybody would be in Dewar statuses and that seemed a bit much so I think her introducing these various roles was helpful I think when it's evolved into and what I've been teaching a little bit more these days is the doers a relatively few clinicians they might be more academic clinicians folks that have a little bit more time and generally that that are involved in generating evidence based products if it's anything from from Emma's base newsletters or critically appraise topics for a residency program to digest something like that or for pot Frank for publications or anything those are the doers the folks that really have the process down keep up with these statistical literature know what they're doing in terms of critically appraising articles and that users ideally would be most the clinicians that they've that practicing clinicians would find a set of good quality pre appraised and synthetic materials synthesized materials to to practice based on if they knew they were evidence-based and really rely on those most of the time I think unfortunately most of it and here's more provocation most clinicians most of the time are really Replicators I think if they find somebody either either a specialist that does a particular thing or a colleague and practice a lot of clinicians end up being replicators ideally these folks would would you would have the ability to assess these folks in terms of their evidence-based miss but I think a lot of times through so there's something I'll come back to over and over again through social learning principles we learn a lot from how others practice I think we default to either the loudest folks in our practice the most authoritarian specialists would come across or you know whatever whatever that you're your source is for that or the just the easiest to talk to when I've taught this framework to students I've talked about a given an example of a cardiologist I knew in Syracuse who who would quite regularly quote chapter and verse of whatever the recent studies are that he was basing his practice on and so I found really good replicating a lot of my cardiology practice based on what I learned from him and that was a I think the the ideal replicator relationship in the terms of that and so but I think that ideal state is is a little harder to find so in 2004 my colleagues and I embarked on we were we were at a Center for evidence-based practice we called it up in Syracuse and and we're trying to decide well what does it mean to be evidence-based how do we know if somebody's Evans based and we went well what if they what if they do the evidence based thing roughly 80% of time which is just a question like that it's kind of just begets more questions and so we wanted to figure out what what other clinicians that that viewed themselves as evidence-based thoughts so we did a what's a Delphi study the telephony is a technique developed at the RAND Corporation for devising structured consensus in a field and so it has very specific methodology and we did what's called snowball sampling which is a we we the the center the folks that did some evidence-based practice stuff contacted everyone we knew that seemed very evidence-based asked them to refer people that they knew that seemed evidence-based and so on so on so on and we asked them questions like what is evidence-based practice what characteristics make you a physician evidence-based and how would you evaluate the evidence-based nests of a physician there's lots to this study however it remains unpublished the turtles did not like its small size so that's fine I think we learned some stuff from it I think and I highlighted a couple of things so we the characteristics that make physicians evidence-based there was there was the traditional stuff about the evidence-based medicine stuff so integrated clinical experience with the best evidence and realizing the heirarchy evidence those things but there's some specific things that are highlighted here that kind of surprises admitting knowledge deficits understanding the limits of evidence-based medicine clear communication willingness to be questioned healthy skepticism when we talked about evidence-based Ness understanding your limitations examining your decisions curious skeptical so all sudden we ended up with a list of one the expected things but to a bunch of personality characteristics that described what went into evidence based practice and it really gave us gave us some pause because I don't think we don't have a great way to teach that sort of thing and it made us really consider what we were trying to do in the center and I think it's as we talked about here I think study designs that's a critical appraisal those you can formulate well and easily into a curriculum whether we do it well it's another whole question but at least conceptually it's pretty pretty straightforward to teach and evaluate personality characteristics relationship orientation some of these other things are these more taught to learners or they selected foreign learners we don't particularly do either well but when we thought about how we would we've changed that there is a real issue about teaching those things versus selecting your learners for those things and we had rather little concept also about how to evaluate those so my my additional provocative statement here is that our current system of vbn teaching is based on what's easiest for us to conceptualize teaching and that's where we've lived I think for 20 years we can get our heads around teaching statistics and critical appraisal and evaluating that it's harder to do the other stuff so let's talk a little bit about environment and culture because I think that's that's where we need to really think about going here so we'll do another poll question just for giggles here pull up pull question number two and go from there which is most characteristic of your experience with teaching and learning evidence-based medicine long confusing learning presentations of the critical appraisal of articles they don't really understand so this is what you've experienced in teaching this borings different there's one vote for that Mori statistical presentations by the research guy in the department who doesn't see patients invigorating discussions about this strength of evidence hopefully folks would see that and then faculty arguments that result in fights about clinical guidelines which have you seen in your practice which is the the most the thing that you carry around in your head about what ABM teaching is about and it wouldn't surprise me about the faculty arguments I mean we've certainly seen that a lot of times in my experience in that is that and and learners Boyd they've developed the survival mechanisms for that they check right out of the process as the faculty just sit there and argue about what are the other things so folks I'm glad that most people have had the invigorated discussions I think that ideally is the goal here but I think it's pretty telling that there's enough of the other stuff going on that we're we're concerned that we don't know that we're not entirely comfortable with us how we're teaching evidence-based medicine currently all right that move on so again discussion trust question for the chat what and I will look at the chat when we're done here it's over to this side so I want to be facing you guys what does the ABM teaching culture look like in your settings so what are you what are you doing in your setting how is it how is it supported one of the other faculty think if you're the if you're the ABM teaching person how do you feel supported amongst your co faculty if you are not the iam teaching person do you like it do you show up to any of the things do you participate in any discussions I want to hear from you so we know that the role of teachers is ideally changing in medical education and in all education I can speak mostly about medical education that that the concept of teachers is supposed to be evolving from teaching facts providing explanations instead to creating a learning environment and helping learners learn within that learning environment there is a great quote somewhere about that that for the life of me I could not find if I find it also did on but but it's a pretty common concept certainly the the calls for the integration of medical education that happened around the hundred the 100th anniversary of the Flexner report and subsequent medical education guidance documents have talked about this that we really need to move that that the teaching along to the concept of a learning environment a safe challenging and richly supported learning environment and helping learners learn in that environment it's obviously a focus of ACGME and LCME focus ion learning environment we have a whole clear process that focused on learning environment that's the big topic in that is mistreatment but it's may also meant to look at wellness safety and and this concept of a hidden curriculum and I think the hidden curriculum is an important concept in evidence-based medicine if you have the research guy or the EBM guy that's the lone person in your group teaching evidence-based medicine and it's not reinforced by other faculty that the hidden curriculum is is if you want to be like dr. Epling for for instance then then that's great but you know the real doctors do this and I think that's a really dangerous dangerous hidden curriculum that we know we fall into a lot and then you know I have a particular predilection for looking at learning theory and I think we're pretty comfortable in using cognitive and behavioral learning theories cognitive learning theories where we're reinforcing cognitive associations and teaching based on that and by behaviorism where we're we're we're reinforcing and using feedback and those other behavioral learning techniques what we what we should be better at but I don't think we bring into consciousness very much is this because we're derived from an apprenticeship model there's a strong tradition of social learning but we don't activate that as much as we might we sort of let it happen by default and I think that social learning theory and this idea of constructivism which is another sort of half philosophy have learning theory that emphasizes these rich learning environments and authentic ways of educating learners those two are real important that we need to really attend to those ends of the learning theory in order to improve how we teach evidence-based medicine they're really important to get to learn as milieu right so in a couple years back colleagues of mine in our national teaching organization and family medicine said to a survey and we wanted to look at EBM teaching culture we we felt like well at least we know where we're going in terms of teaching the concepts of Emmons based medicine critical appraisal and statistics and those things but how about all this other stuff that we're thinking about in terms of culture so we we surveyed and this was part of them you know if we had it to do de novo we might not have done it this way but this is part of a larger omnibus survey that goes out to all program directors and family medicine and individual researchers are you're allowed to submit a set of questions for it so we did a program director survey asking about you know the details of their programs but we adapted a survey that had already been developed by a researcher in California about learner's perceptions of the environment for EBM teaching so she had done it from the learners perspectives dr me who is quoted who we brought into our little research group here she developed this for learners validated the survey did all the right stuff for the survey we adapted it for program directors what program directors would be able to answer about their programs got a good response rate and there's lots of stuff on here I don't need you to concentrate on all of it and and when you're analyzing data about the program directors impressions of their own program there's lots of limitation there so you focus on extremes I think and the extremes that I focused on we're at the bottom here so it's clear that program directors is a bit of step off we're worried about the bottom three issues here and that we protect resident time for EBM training there's a high level of faculty involvement in teaching EBM at our residency training site and our faculty members provide residents with clear feedback on their EDM practice the fact that those sat at the bottom worried us a little bit of EBM culture that we hadn't quite gotten this then we were still probably clearly dealing with the one young person in the in the residency and that we hadn't it hadn't diffused well enough we don't protect time the learning environment needs some help in that sense so those that's the conclusion we drew from that primarily there are other studies that reinforce that these are just a couple of systematic reviews to look at it when they ask doctors about the perceptions and use of bbm they were worried this is a a systematic review of qualitative studies which I have to dig into a little bit I just I just found this in preparation preparing for this talk there's a whole methodology about how to quantitatively or at least systematically evaluate qualitative studies they were worried most about group norms for safe communication that was primarily about disclosing that they didn't know something or just closing learning that that they didn't quite feel it was safe to do that across across the different career stages and that we ought to share the learning across career stages that that the expert authoritarian manner was not a particularly good way to do things that we need to understand that we were learning all throughout our career and to share the experiences with doing that through those stages might be helpful and then there was a systematic review of teaching studies in ABM these authors concluded that in general there were poor knowledge and skills of the folks that were that were part of these teaching studies that we weren't real good at doing those and different different studies have have found different findings with that specific component but in general that was a positive attitude toward ABM so that attitude didn't seem to big be a big deal but that predictably workload as a major barrier to practicing evidence-based medicine and the availability of pre appraised sources or key overall I think this gets to the fact that that our teaching a TBM is limited by our environment that we that we need to work on the environment that supports both evidence-based medicine practice as well as learning so on the ground as I said I think evidence-based medicine is still relegated to the EBM person I see that the quote here about having met the MA is us I'm the ABM guy in our residency and this happens what am I one of the residents told me that they were instructed by by a faculty member to to Epling guys the study which apparently met critical appraisal critically appraise it or something like that I have no idea but the point is if we're saying that sort of thing we've lost the game because then you're saying act like this person for this defined period of time don't incorporate and learn using the literature and I and I worry about that and is it really effectively role modeling do we need to train a bunch of people like me who my wife will tell you who loves to read statistical books in bed at night you know that's not what we want out of most of our medical graduates what we want are people that use evidence and practice with evidence so I'm not sure we are embracing the role modeling as much as we should there's rare formal socialization of the information seeking process what do I mean by that I mean that faculty are often either have a self a self goal of looking like an expert or are perceived as experts and this idea that faculty don't know something as a grand provocative statement is is an important sort of hidden curriculum to this that unless residents concede UM's see faculty looking up things going to evidence-based sources and looking for information in that evidence-based manner they won't do it themselves faculty development is if we focus that on critical appraisal we're going to turn off most of our my co faculty I'll tell you that right now my my faculty in family medicine are amazing set of individuals that are really good in a bunch of different things and are very good at clinical care and and frankly they don't have the time to go back and relearn statistics and all that so we if we end up focusing on that stuff a little bit as necessary a little bit is necessary to just enter the conversation but if we end up if that's all it is and we end up focusing on that it's going to be a big turn-off if we fail to develop the user skills if we fail to Train faculty as model users of evidence I think we've missed the boat and then we have to figure out how we evaluate all this and that is a that is a an open question at this point I don't have the answer for that but we need to work on those skills and out evaluating what we want our learners to be in evidence-based medicine I will say that I think the the goal here is not to focus on the outcome its to focus on the process don't focus on getting the right evidence-based answer pulling it from wherever it lives in your brain but instead we focus on evaluating the learners process at getting that answer and it can be as simple as where did you get that answer what's your source for that just incorporating those sorts of questions into the conversation we'll get to that a bit so when I talk about this evidence-based medicine versus information mastery thing here's where I'm getting at so this is a systematic review done recently 28:18 and involving some of the some of the names and ears Sharon Strauss is in here but our quality is the primary author this was basically what I did back in 2004 but done way better than I did so this was a another Delphi study using largely the same techniques that we did back in 2004 asking asking they did a first of all they did a systematic review which we didn't have access to systematic review of existing curricula an existing competency statements at Emma's based medicine and then did a Delphi study of folks with significant experience in teaching and practicing to figure out what were the most important things and developed a list of EBM competencies incredibly potentially useful document will talk about potentially and certainly comprehensive certainly done with the appropriate rigor when they talk about the ratings of these things just so you when I've put an M behind things that means it was it should be mentioned these were the the usefulness ratings by the Delphi proponents or Delphi sources or mmm means mentioned e means explained and P means practiced with examples so this is a come this is eight or ten slides I'm not going to go through all of them I wanted to highlight a few things that were interesting so here are the ease and the ends just for your review if you want this PowerPoint afterwards we could get this we can get this to you I believe sandy and sherry will make it available so you can look it over or you can just look at that reference and go find it yourself it's a good article to look at so in the introductory competencies here they definitely talk about the five steps of EBM and the highlight down there but at least they talk about this the station between clinical decision making evidence-based medicine versus research so i'm glad they got to that and the differentiation between EBM and research they do focus on clinical questions and and emphasize a lot of practice with that i think that's good they do talk when they talk about a quarter so this is organized by sorry organized by asking you can see up in the top left and then the next step is acquiring so this is the search for evidence they do talk about using databases of filtered and pre appraised evidence to resources that's good but you what you'll see here is that the next three or four slides are all about appraising and interpreting yes that's important here but there's a lot of focus on this there's there's all sorts of stuff about random error and systematic error bias uncertainty lots of lots of stuff here lots of stuff that ends up overwhelming pretty much the rest of the curriculum here systematic reviews treatment studies diagnostic accuracies when i focused on critical appraisal information it's these three study types that i'm mostly focused on but this goes on to talk about prognostic studies harmony ideology studies even talks about qualitative research qualitative research is really hard to explain to somebody that's really struggling with the concepts of qualitative quantitative research and so i think just sort of explaining and mentioning qualitative research i think is potentially problematic here they do talk about differentiating evidence-based from opinion based clinical practice guidelines that's getting more toward the synthetic stuff that we were talking about before and I think a useful thing to talk about there's a tiny little thing here about understanding and practicing shared decision-making any of you that have looked into this literature knows that it's a huge literature about how to do that how to do it correctly per taking in a review that looked for competencies associated with shared decision-making there at least 10 really foundational documents that go over competencies for this it's really hard to comprehensively review in an evidence-based medicine course along with all these other things and then different strategies to manage uncertainty that's really important for learners and and there's a lot of entails a lot of social learning and and and discussion and then the role of for effective clinical practice which arguably should be front and center here is sort of buried in the bottom I do think this evaluate part is really important so both in terms of thinking about how you're doing the reflection as well as the personal clinical audit figuring out how much how well you do on some of the some of the metrics that you consider evidence-based is report so that's a grand tour of this and I think this is a useful document I didn't want to focus on it solely here just to point out some of the some of the the divided mind we have about what we're actually trying to teach even still in 2018 so where do we go from here I think in general directions I think if we focus meaning and goal wise on the process of evidence-based practice rather than the outcome of getting the evidence-based answer I think that will help us as a general direction I think applying critical appraisal not necessarily learning the specifics of critical appraisal for every different information source but applying the general idea of critical appraisal the informations all information sources I'll talk a little bit more about that a couple of slides I think culture wise we need to orient more toward the the power of social learning and constructivist learning theory of the establishing the rich learning environments I think using faculty development to increase user skills is really important and we have to focus on how we evaluate our learners developing the skills that we're talking about in evidence-based practice they're their appraisal of where they find information sources and and how they look at the world and new learning and then we need to promote integration we got to teach how I learn and we got to talk about this evidence-based shared decision-making stuff rather than just evidence-based outcomes all that I think will help lead to information mastery all this is basically about critical thinking critical thinking is a big sort of hairy mess of a of a a tional concept but I think it's it's really important here and it's interesting that we we're very good at medicine it focusing on critical thinking in terms of clinical reasoning my question is why don't we do it for information seeking we know we assume that people have just sort of filled up in medical school with ideas why don't why aren't we talking a little bit more critically thinking wise about where you got the information not what information you're putting together to reason from and so I would advocate for adding that and you know in clinical precept in conversation just adding that sentence where did you or will will you get that information can be important reinforcing what's your specific question if you if they're going into the literature having them narrow down the question is important and then considering teaching about cognitive bias and other impairments to decision making we know that our ability to do some of this critical thinking is impaired by these biases and when we're tired and when we haven't had enough sleep and when we're thinking about other things so a little bit more attention about the thinking process that's a whole other topic obviously but I think that's the stuff that we need to start thinking about in terms of teaching information mastery I know we need a foundation of statistics and study design and critical critical appraisal I consider it a basic science of medicine sort of that we that we teach the ability to do this stuff and but we sort of leave it there and consider ABM covered and as as we're getting away from this and all other medical sciences we're trying to help the learners integrate you know the anatomy and physiology by by creating these courses that promote that integration we don't really do that for evidence-based medicine as much as a sweeping generalization and and we're just hoping that it happens and I think unless we're able to to teach how critical appraisal and statistics and that sort of thinking moves into clinical practice in a more seamless manner we'll lose the opportunity critically appraising all information sources can be as much as this usefulness equation this was developed by the guys that talked about information mastery the usefulness of any information source is its relevancy relevance times its validity divided by its work so anything that that requires a lot of work we already know won't happen in a busy clinical practitioner so we needed to be good quality stuff that's relevant to the patient but has to be easy to get to and I think thinking about all your information sources that way from a consultant to a review article that you have in front of you to an original research article any of those things can be appraised in that manner and I think adopting that strategy will help and that will lead to information consumerism recognizing what high quality pre appraise evidence-based information is making sure that you're being aware of labeling and and really figuring out if it's evidence-based if they if their process is a good evidence-based process and then focusing on appraisal of these knowledge synthesis products and pre appraise sources is where I think we need to divert our energy there's another challenge here and that's lifelong learning and this concept that gets even a little bit fuzzier I won't spend a lot of time on this this concept of mind lines these guys Gavin LeMay observed different practices of the UK in the US and found that that adoption of evidence-based practice happens as part of the socialization getting back to the social learning theory and this is for for practicing clinicians now but a lot of reinterpretation happens in the context of practice so they talk about knowledge and practice in context so so it's it's and we know this from change literature that that people need to be able to bounce it off need to be able to bounce an innovation off other people and need to interpret it for their own specific circumstances before they will implement it so my question how can we improve this process as a way to reinforce this at the CME level how can we improve the knowledge inputs to to this process how do we get the right sort of knowledge to the folks to allow them to socialize and interpret it and how can we make that process more explicit intentional that's sort of where I'm I'm mostly interested right now so in our residency we do a little bit of this stuff I am I would be remiss if I led you to believe that I had solved all this this is this is where I'm at in in the family medicine residency curriculum that I'm responsible for and again I am the EBM research quality improvement guy so that is that is the first caveat to this but in journal club we have a journal clubs though we do a review article the first years do a review article the second years do an original research article the third years however we want to make it development or the third years do these synthesis things a systematic review or a guideline and they critically appraise those and prevent and present those there is both peer and faculty evaluations so trying to bring out the discussion about how folks are do at at appraising and looking at evidence we limit the didactics nothing is is worse than hearing people are drone on about statistics forever so so we limited and sort of do ad-hoc didactics and we have a website a website I created that tries to provide the rich support for for learners learning on their own when they are tasked with presenting these in the community medicine rotation we ask the residents to answer a community medicine question with pre appraise sources and present a report of that and we ask them to answer a clinical research regional research sorry answer a question with original research using those six steps of evidence-based medicine and they have time to work on and get feedback from me about that in clinical precepting we're trying to work on these ideas of making more apparent cognitive bias and decision-making that way we try to ask about sources of answers try to get that discussion going educational prescriptions come in really handy I'm I'm experimenting with that a little bit writing that down that's seems to be a concept that comes back every few years where we'll so giving the residents homework where will you find the answer to this question and and making that process explicit and in their quality improvement curriculum we try to base the quality improvement ideas and decisions and and goals on evidence sometimes it's alright so we talked a lot about that I want to be able to get to the chat inevitably I tend to run over let's see what's going on in the chat here so there's the goal to achieve the provision of best quality care likely to yield how the outcomes that's an incredibly ambitious and good goal nobody could argue with the provision of best quality care I will argue that just teaching evidence-based medicine is necessary but not sufficient to achieve that goal there's lots that goes into that to answer clinical questions for a specific patient encounters I think that's the the most that that's the most sort of entry criteria to what we're teaching we need to train our learners how to get answers to questions in clinical practice and yet we know that still most questions go unanswered the promise of evidence-based medicine was to be able to answer all these questions that a clinician would be able to answer them quickly hasn't yet panned out that way to see guns from best evidence medical education designing or curriculum content yes so so applying all these theories to designing our education would be great we just need the efforts there's other things are there any Studies on the misuse of EBM by insurance companies not that I'm aware of and misuse of BBM is an interesting Act so you're talking about okay you're talking about two insurance companies misuse EBM and other studies of that yeah misusing EDM I think I see it most in I know pharmaceutical companies have have leveraged the discussion about evidence-based evidence-based medicine I won't say they misuse it but I think that you know the creation of things like non-inferiority studies think it's one of the ways that pharma has just money in the waters of this so - you know the goal then is to prove that this medicine is no worse than the other medicine and the statistics and and just teaching the statistics around that is so incredibly confusing that I think it's hard because you're you're using a null hypothesis - - to give the counter of a null hypothesis to prove that there's no difference to begin with it's all very confusing I think there's there's I don't know if there are studies about misuse but I certainly know that all of us have felt a bit of misuse here I will say that as I'm in part of my other life I've talked to folks from the National Quality forum they who are folks that develop some of these metrics and I think in general the the principle for the development of these metrics is to be based on high quality information often evidence-based recommendations from guideline groups that use the right processes for developing evidence-based recommendations it works a lot of the time I think the the choices about how we measure people relative those metrics are important and so I think that's a that's a big unanswered question let me open the open it up for other sort of verbal questions or comments or ideas that you guys have let me know if I didn't make anything unclear if you have particular questions John have you been able to see much response from your own faculty as replicators given the steps that you've taken within family and community medicine to answer that would would sort of self-aggrandizing to a level that are comfortable I think you know I think we have a lot of good users in our faculty in terms of using evidence-based practice I've seen that and I think that's the that's what I want to do is is foment that and and I think the real challenge is getting folks to talk about that stuff so I suspect lots of faculty and this is a natural reaction you want to go home and you want to be the expert so you study up and make sure you're ready to perform in the clinical setting when it would arguably much be much more useful to have our learners search with us and with our guidance a little bit and so and and they've all heard this before we've talked about in faculty meetings that that showing a little bit behind the curtain showing what that search for evidence looks like and how you choose a particular guideline on what basis I think is the important bit of them okay great and I think that's where you do get a more positive response on some of those surveys when when folks don't residents and such don't necessarily resonate with the term environment of inquiry and but it's sort of like other things you've heard of when you when you see it you know it so if if you see your faculty really engaging in that and struggling with the literature and things like that with you I think that's when that's when it's clear that your faculty are engaging in it in supporting an environment of inquiry yeah great are little shortcuts like culture of inquiry don't really help to explain what we're getting at especially when they're used to help learners evaluate us so yeah there's a whole issue with that there was a question that popped up have you seen any organized grading system for evidence-based medicine oh boy have I so they I haven't seen it I haven't seen it updated for a while about five to six years ago hrq did a systematic review of grading systems for evidence-based medicine and found that there were 140 at least published grading system so you have the you had the US Preventive Services Task Forces a B and C system you had the American cardiac College of Cardiology 1 2 & 3 system you had the 8 cogs 1 2 & 3 system with subheadings there and then you had the grade system which is a international collaboration of a bunch that's that actually has been been since that since those that study was published the grade system is has become pretty much the dominant system at least for using guidelines yeah it's it's an incredibly important thing to think about because grading evidence is a shortcut method that Priya Priya sources and guidelines can use to communicate the evidence-based nough subtending so it's an incredibly important idea what I see know is is in the implementation of grade which is very specific about how it should be implemented I've been on guideline committees that say well we modified the grade process because the evidence in this realm generally is terrible and so we we modified it to be able to include some of that evidence you know and you get that you get to issues like that you get to issues of how people interpret guidelines and you give a weak recommendation for something people and yet the guideline committee wants people to adopt that practice even though there's weak evidence about it I see a lot of that stuff and so it it ends up being a little bit harder to to interpret when people take what's supposed to be a standard grading exercise and modifying but it's one of the things we have to be able to teach we have to teach people to look for a grading system to understand the grading system and in order to use a shortcut we got to know that the people that created the shortcut didn't use shortcuts so it's important to know what goes behind all right thank you so much dr. Epling is an excellent resource if ever I have questions or others have questions about evidence-based medicine or how to teach it and I should have said at the beginning maybe he's not necessarily comfortable does anybody have a comfortable teaching evidence-based medicine rather than she's more comfortable than the rest of us most of us I think at times it I don't think there would be the term Epling eyes being used so frequently but shouldn't be the term that we got there but there is before reason but I think your talk really emphasizes the importance of being intentional about integrating a VM into your practice into your teaching and into demonstrating and really role modeling and if we know that most of us are replicators I think we need to one see good IBM role models as we're learning and then for ourselves to in turn develop as grow into being good IBM role models for others to replicate down the line I think that's how we can permeate the system and sort of really spread the good word of how to practice and how to teach this effectively so if you all do I'm I know that dr. Epling is open to questions later on if you want to contact him he's in the system just email him and he is a wonderful resource thank you all for joining us and we look forward to seeing you next time have a great day everybody.