Pitfalls and Opportunities for Educational Research

Thank you Mariah, um I appreciate it I appreciate the opportunity to be here and welcome to everybody that's here um so you know after listening to my titles and all that stuff and talking about educational research actually reinforces kind of one of the points I'm trying to make which is that educational research is just research and in that um we have a lot of opportunities to think about it a little bit differently and that's what this is going to be about um this is a little bit from my from my history of reviewing educational research several different roles uh as a an editor for a journal for um conferences and and uh different different other uh activities um I've come across a number of things and this talk is about a few things that can help um when you're thinking about how to how to get your head around some educational research we'll talk about um kind of what that means in a second here so let's see if I can advance slides go it's lagging a little bit there we go and when when we talk about educational research I I actually want to I I'd love to be able to sort of line through the term research and say scholarship all the time I'm not going to because I sort of default to research uh but remember that this this concept of scholarship is really important when you're thinking about educational research I know and have have seen and worked with a bunch of teachers clinicians Scholars whatever you want to call them uh clinician teachers that that really are allergic to the word Arch uh it seems and and are at least either very reticent to dive into research or um don't think they can do research there are lots of different reasons for that sort of fear and so one of the one of the issues is to get away from calling it research so it doesn't seem like something you can't do but we can't always do that and so I want to um dissolve a little bit of the uh the concern around that and provide uh different ways that you can get into this uh field of educational research and scholarship that that might help you feel better about it so we'll talk about choosing we the nitty-gritty of this we'll talk about choosing study designs that that can help with research and they don't all and often can be randomized control designs and those type of things so we'll talk about that we'll talk about uh ways to avoid bias when you're doing research not in detail and I'm going to present things that overly this so we're not it's not going to be a detailed methods talk um but just some things to think about when you're doing that so ways to avoid bias and enhance applicability for both qualitative and quantitative research um and then designing and develop a curriculum this is the the key if you design develop curriculum in a way that has you thinking systematically and uh programmatically you can um easily find ways to do the research and I will uh excuse myself I've been home with a nasty little virus um that I contracted over the weekend so I will try not to cough directly into the microphone but I apologize if I do okay my first hot take educational scholarship is a faculty Duty and I I do truly believe this I think that the scholarship the idea of getting our ideas out to the rest of the world is really really important even if you're quote just a clinical teacher um if you're in a Residency program teaching residents if you're just teaching students we have a duty to not only um do the teaching but also improve how we do the teaching and that takes a takes a line from quality improvement we have a duty to not only do medicine uh deliver care in in a good way we have a duty to improve how we deliver care and so that same thing can be said of of educational research and scholarship if you're familiar with Boyer's model of scholarship many are um this a scholarship of discovery which is traditionally the Big R research the the you know method heavy research that we all know there's integration which is synthesis research so systematic reviews and other other uh systematic way ways to integrate the research that's been done the research the scholarship of application how how you get um proven things into practice so in the in the case of healthare getting uh evidence-based medicine into practice is a form of scholarship so quality improvement and implementation science could be parts of that um and then teaching and teaching is a funny one because it's not only teaching the um the Next Generation but it's improving How We Do teaching so integrating each aspect of Discovery integration application into our teaching work I think is uh very important and so you can um so there are ways that that this all fits together and uh some of the some of the things we'll talk about some of the opportunities we'll have is to see how um overlaying application and discovery on teaching and that a more systematic way of designing your teaching can help facilitate some of the scholarship all right but first let's take a few um stabs at some some things that we might be able to do better in research these are some examples and we're going to go over these examples and come back to them at the end so consider this scenario um a faculty member is on some under some pressure to do some scholarship um I think we've all heard that that they you know if we're trying to get promoted or or the chair wants us to do some scholarship we've heard about those sort of things they read about a gamifying technique could be Jeopardy could be some other game uh game that you can get your learners to play for teaching and they apply it to their assigned lecture topic and they study the following how the residents rate the session they rate the they study the knowledge before and immediately after the session and they study the self- assessed confidence in managing the condition okay so that's one scenario for a a and I've I've seen multiple examples of these in my career um this one example of how a lot of educational research gets thought of second example faculty member sees a problem with inappropriate referrals to their particular specialty and pardon me they create a survey or intervention for referring clinicians now that's kind of interesting the survey assesses knowledge about the referral guidelines for this particular condition and provides a link to PowerPoint education for the respondents of the survey and then a follow-up survey sent a month later assessing whether they learned anything from that PowerPoint education module so I've seen this one go around a couple of times actually um it's an interesting situation and we'll go over the the pluses and minus of this of this a little bit later and then finally a researcher this what I've seen more recently research described a qualitative study of 13 clinicians across three states about their awareness of certain guidelines found out that they weren't particularly aware in these in these interviews um of of some of these guidelines and in their conclusion they recommend a national call for Action to improve knowledge of these guidelines so think about the think about the scope of that those two things the study and their their conclusions for a second all right on to the opportunities so keep those in mind as we go one of the opportunities is to structure the educational development a little bit we do this um through a uh set of instructional design models or methods and these are um I've talked about these before in previous teach lectures before I I I come back to them one my I have a degree in instructional design so I'm likely to think this way but two I think these are very accessible uh ways to structure how to develop curriculum that causes it to lend itself really well to educational research so if you if you go in with this structure about how you're designing curriculum you will be more likely and more able to develop that curriculum and research it so this is the old tried and true one um uh certainly the one that that I have the most experience with the Addie instructional design model um what is this so it's just a series of steps to think about as you're designing curriculum they can be linear they're often designed in different ways where they where learnings from one would flow into the other it's not really terribly important but just let's let's use them as things to think about so the analysis part is when you understand what the what the need for the education is to begin with so it could be national standards or calls for curriculum um on a particular topic it could be um you found a problem with your learners and need to address that um so the analysis can take a number of different forms design is where you you write the objectives for the education and you design some that you choose the educational strategies and you figure out how you're going to actually conduct the education development is in preparing to conduct the the education or training whereas a development of materials development of Assessments um uh faculty development to be able to uh introduce the curriculum all these things go in the development bucket implementation is when you actually do it and you you study how you're able to do it and how well it goes and then evaluation which is really baked into the process the whole way down is is both summative and formative evaluation how did each step go and then overall is the instruction meeting its ultimate goals which you've designed back in the in the design phase very simple model the reason this model is important is as you can see as I talked about each of these steps there are lots and lots of potential research questions in there and it's really um it's it's really a sort of a target-rich environment for conducting educational research and and this is an important point I'm going to come back to a lot of times if we go into to educational research without a a framework like this or without a without a plan about how the education was conducted we find ourselves trying to literally boil the ocean do way too much in our educational research when we don't think about it like the the fbri researchers around here who are studying you know a a particular gene or a particular protein that have a very specific place in a long Cascade of chemical reactions that ultimately results in a health outcome way down the road and while I don't generally I'm not generally a reductionist in terms of my focus in research um I think that that model is important to realize that that by breaking this down into different research questions using an instructional design model or some other model like that it it it reduces the the pressure you feel to prove that educating somebody can you know make a patient feel better down the road you don't have to do that whole Cascade you can by appropriately constructing a study you can just study one notch in that chain and then move on and have actually a nice research program a research career researching different aspects of that particular uh uh instruction so that's that's the fundamental basis here for why we talking about um instructional design Frameworks um those of you I think um sharing Mariah you can tell me uh we used to use KS uh six-step ID model instructional design model as one of the the instructional design models that was taught in a lot of the faculty development that that um teach has done and so this is a very similar model um and it frankly doesn't matter which one you use as long as you get one that's been sort of been used around different places so for Kars has the problem identification and general needs assessment here and a targeted needs assessment so this just separates the needs assessment into two categories one is the the overlying need the the national calls for Education the program requirements the RC requirements lcma requirements whatever it is Target needs assessment would be your individual Learners what do they need what deficits have you found that you need to address um goals and objectives and educational strategies crossover between uh design and development in the atti model so you can see similar things there implementation is implementation you're studying how it went and then evaluation of feedback last again perfectly reasonable to choose this model over Addie just what you prefer trying to click what's going on there it is so um now this is a different looking one isn't it um so this is the four component instructional design model by a Dutch uh researcher called van van Marian born um he has looked specifically his he and his group um he's passed away now but his group looks specifically at instruction on complex activities like oh medical education for instance um and divides them into different groups so where you're learning specific tasks and that are well um integrated into these authentic whole task experiences so a lot like what medical education would be when you're seeing patients there's part task practice here where you're looking at things and I'm not going over this that you need to know this um you know need to know the details of this particular model but look at the questions that it would come up you know how how do you make a more authentic um whole task experience for Learners that would lend itself to this model um down here is the supportive information so this is a uh sort of what they call a constructivist area if you surround the Learners with information and uh knowledge to be able to improve their practice they will consult that knowledge and use that pardon me when they're trying to perform so uh and then procedural information how do you get how do you uh teach Learners how to do specific procedures and and uh do them over time lots of potential research questions in here that take you away from the traditional Addie and Kerns model but into more uh specific areas of of learning repetition maybe some simulation um and and sort of uh you can break learning down into these various categories and study them particularly just an example complex model that looked a little bit different than Addie all right so what's the big wh with all these first of all the use of Frameworks as in all of research the use of research Frameworks are valuable because um they they they make sure that you understand the context of your research question it backs you up a little bit to make sure you know what's gone on before you want to be able to use previous Educators uh educational researchers work in developing your own research I think in in medicine I've seen we have a little bit of arrogance every once in a while when we think of some interesting thing and decide to teach it and or decide to do it and some educational intervention we've heard without really understanding what's behind it what's the learning theory behind it is it more cognitive behavioral whatever we'll get into a little bit of that later um has this been done before and in what in what settings is this an appropriate um uh kind of um what what which Frameworks guide you to the most appropriate questions um this that's the important important of a model and framework and some of that's about just a good literature search before you start but some of it's also to understand what's what's been done what's been worked on and how can you advance the science beyond that uh research uh Frameworks help you design better interventions more appropriate interventions and study better outcomes um you don't waste your time and money in repeating research I see this a lot you know we don't need a study about you know jeopardy for Jeopardy a Jeopardy game for teaching a concept in medical education anymore it's been done multiple times before um so we need to kind of avoid those and do something a little bit more useful um it keeps you from biting off more than you could chew again you don't have to do the whole prove the Education Works to create better doctor thing you can work up uh step by step along those educational Frameworks and and do research that way uh if you conceive of it as a multiphase process and you're continuing to work on things as you go any question about uh Frameworks right now nothing's popping up in the chat okay yeah I got the chat up okay good y all right we'll move on to study designs um always my favorite category but remember the the underlying principle of this is that Ed educational research should be good quality research we do a number of things unintentionally I think to say that uh we often think that educational research doesn't need IRB uh review and that's mostly wrong um that that educational research is always QA and Qi that's also mostly wrong um it should be good quality research it doesn't have to be particularly complicated don't they don't have to be a lot of barriers there um but we got to choose the right study designs and do the research in the right way in order to get the good results that Advance the field a little bit now there are all sorts of barriers to studying our Learners not that they not it's not their fault um but consent is one of them consent is a big issue um we we don't get to just experiment on our students just because they're our students so we have to make sure that we're Towing the line with with IRB regulations and all that to to um make sure that our Learners are consenting to being in a research study um there are furpa regulations fura for those of you who don't know is the sort of the educational equivalent of HIPPA it's a privacy act so this consent and furpa are very related just like consent and Hippa on the on the clinical side um so you can't just use test scores without sort of going through a certain process to makeing sure that that um uh privacy is being respected and all that stuff so you have to have to make sure you attend to those um the big thing in education is that you can't create differentiable educational opportunity and what I mean by that is you can't randomize somebody to placeo learning that ain't cool um everybody that pays for their education or is assigned to Ed or is is being paid to get their education like residents needs to have the similar education uh as much as possible so you can't create a no education versus education thing you could create comparisons in other ways uh different methods different modes Etc uh but you want to be careful with that that it's not a completely different experience and then the final one is that that the choice of the outcomes that you study need to be consistent with the stage of learning that the Learners are in I ran into into this all all the time still run into it when talking about uh educational research involving evidence-based medicine so um evidencebased medicine often gets sort of put in the first two years where there's no clinical context or less clinical context there's more now than there used to be um and so evidencebased medicine instruction was always meant to be taking real clinical questions from patients and then getting the evidence-based answer and applying it back to the patient you don't necessarily learn that when you are in the first and second year you don't have that application process so I routinely struggle with that teaching evidence-based medicine um when relegated to the first and second years and struggling to get that of medical school and struggling to get that into the third and fourth year the clinical aspects and getting that education reinforced and studying the right outcomes during the the clinical area so these are the the issues we have studying our Learners it's just what we have to deal with with educational research Don't Panic you're not meant to read this um this is a study design graph and I actually think this is a really really useful handout it's from the um Cochran collaboration the um uh effective practices and organization of Care Group so this was a group that focused on kind of Health Services Research a little bit studying how care was delivered they no longer exist as a group but their materials are still out still out there um and there's a lot of par between Health Services Research and um and educational research in that we're often doing things in in the delivery of healthare where we don't want to randomize somebody to Placebo versus the the um versus the intervention of choice we want to really sort of work on the work on the airplane as it's flying a little bit we want to make these uh interventions and then study their effectiveness down the road there's certain ways that you can do that in a valid manner um and the those are high Ed in the blue again I'll list them in a second so you can see them um all the rest of these little squares that are dead ends are less valid they don't necessarily control bias as much as you'd want so there are better and worse ways to to do both educational and Health Services Research and they line up pretty closely so what am I talking about over on the right hand column th these the these three I've seen most often in educational research uh especially the one with a red circle on it my the uba my my Nemesis if you will the uncontrolled before and after analysis so this is I measure something beforehand I measure their knowledge of a certain thing beforehand I put them in a session and teach them something and then study them either right after or even a month after and their knowledge goes up or down or whatever the issue with an uncontrolled before and after analysis is you have no idea if your intervention actually did anything different right you have no idea if the Delta even if their knowledge went up because you're not you're not controlling the um the intervention you don't know if your intervention is responsible for that change in their outcome and yet I see over and over again and I don't just see it here I see it in Health Services Research over and over again uh we we measure an average beforehand we do some intervention educational intervention and we measure some average afterward and claim success if it's if it's higher that just doesn't wash for uh educational research it's too prone to bias and other secular uh what do they call secular effects and everything else the other all the whole range of other things that could be going on that cause that change in outcome that you're seeing that isn't necessarily du to the instruction there are other ones I see um case series um case series are just studying you know doing doing something to one person at a time um this is great I guess if you if you're worried that your education will kill your students that's I'm ping of course but that's kind of what you're doing is you're making sure you don't put them to sleep or something like that but it doesn't tell you whether your education is really effective uh it's more of a more of a phase one study for your education um so I would avoid case series it doesn't really get you anywhere um it might be it might be tempting if you ran a rotation like a specialty rotation in residency and you have only one or two St one or two Learners coming through at a time and you're test some intervention that might that might feel like it's what you can do but is is not a great way to to do research for that um cohort studies where you're where it end up just being a um observational study um are are not as good there are other sort of there are other methods where you're not actively randomizing people or where you're where you're not just observing them might be better um but cohort studies generally are subject to bias again because of everything that that goes on and this is what retrospective cohort or prospective cohort studies um where you have two groups and you're watching them over time um and one group is doing something else doing something and the other group is not doing something you're comparing them that doesn't really work because there are all sorts of factors that might lead the one group into doing something at Baseline that that would change the outcome um so pardon pardon my phone went off um there are ways to interrupt to minimize bias however and so these are the the strategies the the study designs that can minimize bias here we talked about the idea of of fixing the plane while it's flying and that interrupted time series is one of the ways you can do that so if you are measuring uh an outcome multiple times like once a month once a week whatever and you have a set of data points and you do an intervention you you do a educational intervention and you watch for change in those data points over time and you're you're basically reg running running what are called regression analyses on you know before and after the um the study that's a valid way to assess for change that takes into account what are called secular Trends so it it can be a a better way to prove that something's worked without getting a control group involved you could get a control group involved you do a control before and after study where you're measuring the change from Baseline to end and you have a control group and a non-control group this is hard to imagine in educational research um I don't see that much but um but you that might also link to the non-randomized trial here where you uh down at the bottom where you're you're you're not randomizing but you're implementing something in one group and maybe you have a friend in a different University and they're they're not implementing something and you can but you can assess students on both ends that's the control before and after uh part of it you might be able to do that randomized control trial and cluster randomized control trial or sort of the the gold standard but as we've said it's often uh difficult to assign somebody to no education and so that might be hard I've seen in education larger cluster cluster randomized trials so medical schools innovating about something comparing themselves to other medical schools that have not done that yet as long as you're within the medical school or the The Residency and doing everything the same way that's less of a less of a um concern to accreditation agencies than if you're varying the education within a a single institution um so I have seen the cluster randomized trials on that scale repeated measure study is similar to interrupted time series you just measure multiple points and you're able to to assess differences that way point of all this being I'm not going to give you I I don't claim to be able to teach you all of study design in a single slide here um the point is that there are better quality study designs to do educational research um than others and please try not to default to this uncontrollable before and after analysis I see it far too often qualitative research also has rigor associated with it it's hard for people who are quantitative research to get their minds around what that rigor is sometimes uh they need the blinding they need the control they need all that sort of stuff qualitative research is different research but no less rigorous and so the method or framework you use to base your qualitative research whether it be grounded Theory or um uh crystalization those type of methods they exist there are lots of different ones in qualitative research you should use one of those um think about your sampling who you're getting for qualitative research who are you asking who are you bringing into focus groups who are you interviewing how are you doing that how do you get your text for analysis um is is important is it just open-ended comments or you soliciting information um and then the analysis is really important um rather than getting rid of bias you just have to acknowledge your bias um in qualitative research analysis that that you have certain biases as an investigator you go through a process of what's called reflexivity where you talk about your biases well Dr uping he does heaven based medicine stuff and really likes numbers and statistics and boring stuff so he's going to have a certain point of view when he goes in and asks about this um all that's really important as part of um designing qualitative research you want multiple perspectives so you don't just want boring old Dr eping analyzing the data you want somebody that doesn't have those sorts of biases and you get people with multiple perspectives uh looking at your qualitative data and that's really helpful and then member checking is a third way to to validate the analysis you you go back to the once you've themed out your um the qualitative comments and and developed your themes out of that through your with all the different perspectives and stuff you show those back to the people that participated in your study and you say did this capture what you were talking about before do was there anything that we missed here and that's member checking and then being careful about generalizability the purpose of qualitative research is not to generalize to a population the the conclusions of qualitative research are meant to develop look for and develop ideas and meaning and potential significance um and that they're really good for distilling down lots of thoughts into themes that sort of hang together so that you can um if you need to do qu quantitative research on them later um might be a good idea but they're they're not meant to generalize to a population because you haven't done the things in quantitative research that that enhance that generalizability so be careful when you're when you're doing your conclusions of qualitative research all right so why don't we worry about study designs um the uh one so rcts aren't required but ubas are unacceptable so somewhere in the middle there there's a good study design and um the the I'm certainly not saying that everybody needs to do randomized control trials because they're not feasible often and they're not really wanted in education um poor quality studies whether it's the um the choice of intervention or the outcomes you're studying or something like that they take time and resources they you have to consent people to be in them poor quality studies are unethical a lot of irbs we we don't have this in Korean Clinic IRB for a number of reasons that aren't necessarily bad but um a lot of irbs even for the lowrisk studies do not have um researchers looking at the quality of the study for rigger um in a several previous places I've been um the they would send even the low-risk studies out for researcher review to determine if the study is likely to show an effect if it's based on has a good framework behind it um you're studying the right outcomes all that sort of stuff stuff um we don't Coran Clinic IRB doesn't have doesn't do that and so we often get the mistaken impression that sort of any old study we can put together as long as it sort of conforms to the IRB is okay but but i' I'd want you to really focus on quality and make them the best studies you can using some of these principles and others um because otherwise we're just wasting people's time and effort um and the better you design your study the better chance you have of getting it accepted somewhere for dissemination so in a publication or presentation um the higher quality research um gets you gets you published better um about the IRB they are not should not be an object of fear the our IRB is really really helpful here actually um I I I work with both Brooke and Meredith um mostly because I end up doing the lowrisk studies mostly um and they're really wonderful they they have to take their stance about things and you understand that but that's if you consider that consulta to keep you out of trouble that's a really good service and so they're really good at at finding ways to keep everybody on the on the straight and narrow regarding research um my opinion is that most useful educational research is not QA or Qi it might be exempt um because it's done in the educational setting but it's really not QA or Qi QA Qi you're you're implementing evidence-based practices um and and maybe studying how they how they were implemented um that's different than trying a new instructional technique and seeing if it works and so you have to keep your your research questions really carefully delineated there all right so the very bottom line no unsystematic biased experimentation on Learners that's not okay um but if you find but getting a good study design and focusing your your research um you can use that creativity and passion Maybe over several different studies and that's the better way to conduct it all right last little bit here sorry my pushing and it won't go outcome Frameworks so this I have presented before so I apologize if you've seen me talk about this before um the old Franklin cvy or Steven cvy quote begin with the end in mind is very important here and it comes down to this question how do you know that somebody learned how do we know that that and this is as as applicable in our education as it is in our educational research how will you be able to show that somebody learned something and I've sort of given it away right there well we talk about understanding we talk about knowledge and all that stuff but in the end to be able to assess knowledge to be able assess understanding we have to have some sort of demonstration of that knowledge and understanding that's why I think people get a little a little off base with learning objectives and then are sort of hopelessly see when they when it comes time to actually um research those learning objectives so what am I talking about the answer to this is good old Bloom's taxonomy it's not the only answer but it is an answer and it it what it does is it merges that cognitive stuff that basically is the big black box and inside our brains about how we understand and remember and analyze and do all those things and it translate those into behavioral things that we can observe and both for educational outcomes as well as for research outcomes okay so the the bottom line here is when you're when you're this is the link between educational objectives and research when you're doing this sort of thing use the behavioral objectives as observable things to be able to assess for your learners to understand whether your your educational intervention has helped so when you ask understand is the big one we all want to create an objective that says the objective is that they will understand this well no I mean we can't go into their brains and figure out if all the connections are there for them to understand it so we have to ask them to do something they could explain it they could describe it they could predict something they could you got to get to the verbs and that's the mo most important thing about Bloom's taxonomy if you like this this I have a bigger taxonomy for you with more verbs um and you can search online there are a million and six um images that that have this basic format um what I like about this is W gives you all sorts of different verbs to consider and gets you thinking about um the type of not only the type of instruction and outcome that you want to write into your educational objectives but also about the sort of research that you end up doing and if you look at this the example learning objective over here learning objectives are written in a fairly researchable way um if you're uh a common research question especially in evidence-based medicines the Pico question the patient or patients the intervention the comparison the outcomes here we have the the patients in this case are the students we have those outlined in each of the questions whoops sorry about that um we have the students we have the Intervention which is um whatever your learning is that's what whatever the instruction is will will end up resulting in the learning objective here uh the comparison is sort of understood the outcome is written right here they can design an original homework problem dealing with the principle of conservation of energy um you'll probably want to choose a medical out but anyway um so understanding that if you write the learning objectives well be able to translate those directly into observable phenoma for research that's the link there also one of my favorite things that anybody knows knows me will will um attest to is this K Kirkpatrick's training evaluation scheme it's not the final answer but it is a is a um framework that has done me well both in terms of um writing objectives but also in terms of research uh I've used this in Health Services Research as as recently as a couple years ago to un because if you think about sort of how we affect practice change and how delivery how the delivery of healthcare is changed it's often through either education or training and sometimes we can um or at least that's a component of it and we can use this sort of evaluation scheme to understand how well that works to actually change practice it's a part of it we'll talk about that a little bit later but um this is a a useful framework so just to go over again for those of you that haven't seen it the first level is reaction that's you know how tasty were the donuts how warm was the coffee were the seats comfortable was the was the um uh instructor entertaining and I think a lot of our educational interventions uh are are more about edutainment than they are about actual uh educational outcomes but that's hot take that's an opinion um level two learning knowledge so this is the cognitive change again just that if we're increasing knowledge we have to be able to demonstrate that somehow so you're still having to look at these level two outcomes as demonstration of that knowledge demonstration of those skills so think again about uh behavioral verbs to get to that three is process change so Kurt Patrick was was an instructional designer in Industry so they looked for process change you know the number of number of widg widgets made you know you got or actually that would be results so uh how how did you do things differently as part of your um as part of your work would result in the change in results so um for this we can we can say in the health care setting um knowing how to listen to the heart is one thing but process change is being observed actually listening to the heart differently a result might be diagnosis of a of a heart condition under certain circumstances so the results are the the important um outcomes for uh for the patients or the Learners whatever you're whatever you're looking at so all sorts of different outcomes I think in educational research we tend to stop at level one and level two um often at level one this this whole idea of um the Learners feel like they could do this thing well in clinical circumstances I've seen that as an outcome assessed by survey I'm not saying there's nothing to the idea of self-efficacy however it does not mean that they can actually do it in a clinical setting so just just their their confidence their self-rated confidence or self-rated um Effectiveness in being able to do something down the road is more a level one outcome than it is I think a level three outcome so just um be aware when you're using this to not overstate the the game a little bit um Kirkpatrick eventually decided that return on investment was a level five outcome but I don't think anybody ever uses that it's a difficult one for education to use but I think this is a if you combine Bloom's taxonomy with its focus on behavioral observable out comes and Kirkpatrick's evaluation scheme to to to with each step in your research Cascade get closer and closer to the level four results that we all want um I think that's that's the way to handle choice of outcomes and uh designing your research in a way that will actually um Advance the field a little bit clicking there we are um one of my colleagues this is the journal I'm associ editor for the uh the editor-in chief is a friend of mine and wrote an article about moving on from self assessment so this is um basically we we struggled the primer is a journal for the society teachers of Family Medicine which is our academic society and we pride ourselves in being a developmental Journal so it's we publish learner research as well as sort of Junior faculty research we try to get research out there that's that's um you know maybe small ends or uh just part of the the Cascade that we're talking about where you're studying little pieces and putting them together so we try to get we we have a development Focus we try not to reject uh studies that have some potential and work with the authors to get them to get them published that said um at some point we got we were getting lots and lots of just these self assessment outcomes they were basically um ratings ratings at the you know I enjoyed the course I thought it was an interesting thing I liked how the professor did it and I feel like I could um do this this procedure or do this I felt like I could take care of patients more more uh I feel like I could take care of patients better having completed this course all self assessment stuff really doesn't matter much it's it's helpful to know but it's just it doesn't really help us much anymore so that was his his editorial for that all right so let's go we're at the end here let's go back and practice so we have our pitfalls from before sorry the slides are really laggy all right faculty member under some pressure they do the gamifying thing they assess short-term knowledge resident rating self assessed confidence problems with this well what's the underlying Theory they used they they talked about gamifying but did they did they go back and do the background search to see if gamifying has already sort of been proven to help or at least at least led to some positive cognitive outcomes um is this particularly novel or has somebody done this over and over again we find that a lot um look at the choice of outcomes here they're largely self assessment level one kpatrick outcomes level two kpatrick outcomes knowledge before and immediately after um at least if you can extend that cognitive outcome to six months down the road then you have a sense that somebody may be able to to do something that your education has helped um self assessed confidence again might be helpful when you think about self-efficacy but it's not the whole picture and and can't be used to say Yes mine works this instruction Works in in improving how we manage the condition so be careful that now there may be other things I'm interested to hear from you but we'll move on to the second one so the inappropriate referral problem again I've seen this happen so there are lots of problems with this one one apart from it being a little bit offensive to the referring clinicians that that somebody else would you know sort of steal their watch and tell them what time it is um there's lots of assumptions about the problem one that the problem is knowledge and so the the the survey has been worked into assessing knowledge and then immediately providing an education thinking that the whole problem here is lack of knowledge there may be all sorts of reasons for referrals or some complex process like that that has nothing to do about knowledge nothing to do with knowledge of the referral guidelines um so this is where if there's a lack of a guiding framework you haven't done your your background work sufficiently you can get a little lost um does education change practice now I've told you about at least one study I've done that that thought it was something that could help but in general education doesn't change practice education as a sort of a an hourong lecture forance um or or some some generic CME program in general they don't change practice really well so if you do education and and the the study I talked about where um we did some some education and hope to see a change in things we we helped that education we came back to it and refreshed the education we uh gave job AIDS and other supports for the education um to to see if we can change practice that way and so that's the sort of thing that that if education is your is a component of this that's great but you often have to um use other methods to make sure you're able to create a robust enough intervention that it actually change his practice and then finally in the spirit of nothing about us without us um if unless you're collaborating with a referral referring clinicians to do the research it all just seems a little heavy-handed um so um so be careful when you're when you're doing stuff that looks like educational research that is in fact just bad practice change research and finally um the this this was such a a sad thing um the researcher that did that uh sent a qualitative study out a solicitation to 2500 clinicians across three states got 13 interviews and quite literally uh called for a national call for Action to improve knowledge of these guidelines um first of all there was very little in way that the way of background of this particular study um there wasn't a lot of uh I don't think they knew their topic quite as well as they should um it's not a they didn't have a particular Focus research question this lack of awareness of guidelines was kind of a different thing that they found and wasn't really looking at uh they they hadn't planned to look at that a little bit um they're trying to generalize a problem after interviewing 13 clinicians are trying to generalize the problem across the country and call for National call to action to improve knowledge um just the wrong it's the wrong way to go about qualitative research um there can be themes there can be ideas for you to pursue um but you need to if you're if you're truly looking at generalizability from here um you've got to work on on doing a quantitative uh analysis of the problem before you have some sense of generaliz ability and then there's a whole field of quantitative qualitative rigor that wasn't addressed in this particular study I shortcut it here but um how did they how did they find those clinicians uh were they was it a purpose of sample or was it just sort of a convenient sample um how did they go about doing it what what method of qualitative research did they base their study on all sorts of qualitative rigor that was not well addressed there all right you've done well if you've stuck with me to now um so educational research should be high quality research shouldn't be a question um that we need to not compromise the quality of the research um just because we're dealing with Learners and not a clinical topic pick a passion find something that you're interested in researching and then do small um small approaches to the topic use the education the instructional design Frameworks to pick small questions and research them one at a time and that passion will help keep you going from study to study um but that prevents you from sort of feeling like you have to bow the ocean for doing premature research that you don't have really have a hope of making terribly valid because often we're dealing with limited resources and and other things so find something and then then bite off things that that seem doable um and study those use your educational Frameworks and models to generate research questions to get ideas um it's often simpler than you think it will be to to chew off those those small questions and and study those rcts are not required or even wanted but there are definite Alternatives that minimize bias um avoid the uba and then consider levels of the educational outcome um both in terms of which which verbs which Bloom taxonomy sort of level of understanding cognitive development do you want to look at and then um when you're looking at the effect of Education which level on the kpatrick scale um could you look at and one that's appropriate to both your subject and to the constraints looking for level three and four outcomes when you're teaching um in the first couple years of medical school may not be appropriate yet so you have different levels of outcomes uh for the stage of of training that your learners are in all right that was enough of me Amron I'd love to hear your comments that would help me to know oh there are 32 people on Wow hey John David music really appreciate your presentation very well done very help thank you I'm really interested in this idea of the impact of education on clinical practice and the the whole approach to CME basically seems to be based on the notion that education would have an impact on practice I know there have been some studies for example that have looked at things like you probably are aware of this where um future Physicians practice patterns are greatly influenced or impacted by where they do their residency training yep um so I don't not only that not only not only that but when they did their residency training so there was a study that looked at the their treatment of hypertension the the sole explanatory variable was the the amount of time since their residency train as to how they treated hypertension um but you're right I mean education has got to be part of it right and we we have this I think part of it is um the the the the lectures the the CME lectures that are are the lectures that if you're just sitting in a dark room and listening to slides that's the stuff that probably doesn't change practice terribly well um part of it is we need to understand the context of education so we're sort of filling up the empty vessel for the first two years of medical school or or something like that we're pouring a lot of knowledge in and that needs to be continually reinforced and that's the whole kind of design of medical school and then the question is how do you update from there and I think that that's where the the individual lectures fall short but we also know that that um incorporating other other sorts of in instructional strategies um looping back to education in a sort of a helix design or double helix design as many schools have adopted is important um the uh different learning theories probably um contribute to this so the ideas of social learning um where you're learning from peers and from mentors is really important in medical education um so and then uh constructivist theory is probably pretty important where you you know we select our students to be really high achieving and smart people and if we just let them in a room with resources all the resources they need they probably just learn themselves and we keep proving that to ourselves but we're we're we're sort of we feel it's hard to let go of the teaching aspect of it a little bit I think all those things contribute um and I think my my issue is that when we when we fill out our CME evaluation forms this is and sorry Deb I'm gonna like trample on this if you're if you're filling out a form that says I feel like I will be able to do this better I mean that's just falling into that whole trap of self assessment again um and and I know the CME offices tend to use that question now um but because they they want it to to mean something a little bit better than just the reaction level advice but I I I I think that's limited a little bit so great stuff thank you um couple of comments or questions Austin Austin Drake when is it better when is better the enemy of good enough the studies you've referenced have been published sometimes SE we don't Capac starting line because we're crippled by the pursuit of perfection oh that's fantastic so Perfection so Perfection come across it a couple ways and I think it's my point about using these Frameworks gets to that idea that Perfection is often considered my education will improve patient outcomes and that's insane that's really hard to prove that link I think um if Perfection is I want to find a way to measure um measure higher up on the bloom taxonomy for this little piece of instruction that I'm giving um I think that's a more reachable outcome and so I think you can find you can find ways your drive for Perfection needs to be tailored to your constraints and circumstances a little bit um and uh and if you can base it specifically on Theory and show that you are advancing the field by I know this framework and this Theory and I'm going to but there here's a gap we haven't studied it in this context or in this particular setting that's what I'm going to do you find it an evaluation then you've Advanced the field a little bit I think that's what makes you publishable um just you know doing the 60th study on whether Jeopardy makes makes resident lectures more fun is not all that helpful um dearra always searching for validated tools to increase reliability and rigor are there any good ones in educational research uh validated tools your your you're there on online what uh what sort of validated tools are you looking for just to be clear I I I'm stuck on I'm stuck on measuring those outcomes um yeah you know how do we how do we we capture that and I I think a lot of times I mean again with those pre- and post things and you're looking at your confidence level there's really no way to get a measurement that's going to be accurate or or what have you um I I have the opportunity to train a bunch of people this summer and and they're from different um majors and backgrounds and and I'm I'm wanting to know if they're going to gain knowledge from I'll be doing a lot of work with them on soft skills that are are are potentially observable um but I'm I'm trying to figure out what what I can do to show that that this program is making a difference and are there things I guess it's just tests that measure their knowledge but then going in and doing you know observations of their behavior um on the unit with people yeah I don't know yeah those are and those are exactly the questions and I think um you're constrained by your circumstances because you have a diversity of Learners coming all into the same program um and so one you can study that you can study maybe the the um in in A needs assessment framework you can look and go well where are the where are the deficits here I don't know what you're studying communication or what let's just pretend it's communication so um do an assessment an oosy or some sort of maybe live assessment or a peer-to-peer assessment with ratings and and then look at communication deficits across different Specialties or something like that uh professions and then maybe do that at the end and and you you you're trying to bite this into into ler sections that are within your control and you have some ability to do because it's not overwhelmingly complex um and you're you're dealing with the important issues which is how do you get heterogeneous Learners all swimming in the same direction with your content or something like that and that's the bit that I think is is really hard for us to step back and go okay I can't I can't prove that this will create are doctors or whatever it is that you're trying to do but where's the Gap here what what other programs have tried to do this and where have they failed or where have they not studied and what can I do to um to to take one step further not 10 steps further and I think that's the the concentration thank you um are there sorry to answer your actual well to not to address your actual question about validated tools yes but it depends on um um there is a it depends on on what you're looking at there is a um systematic review organization called the Campbell collaboration it's a a sister of the Cochran collaboration the Campbell collaboration and they did they do um systematic reviews across a bunch of different fields other than medicine and one of them is education you might look there you might look at the uh best evidence medical education Bey collaborative that um is a another systematic review group that looks at um those you might look at meded portal for um especially especially place like meded portal because people often upload their um tools and resources if they've that they've created for other things you might be able to adapt or uh coers one of those to to see if you can find that and then just centering what you're trying to do in a framework interdisciplinary education looking at those Frameworks you'll often find links to certain tools to evaluate those I think that might know uh Francis hi uh minute discussing dissemination best practices uh dis dissemination of so getting your your research published is that kind of what you're yes yeah um yeah so couple things um if if we take a step back and go the purpose is sharing the the purpose of scholarship is sharing then I think I think you got to look at conferences as a perfectly reasonable way to disseminate so specialty conferences and they don't they don't feel like you're writing something excessively scholarly if you're not trying to get published I think publication is hard these days because a lot of the educational places require fees um and we just don't often have that much money it's like 3,000 bucks to publish in an Open Access Journal there are medical education journals but in order to get into one of them you have to have these um you have to have good robust designs you have to have the Frameworks you have to have all the things that we talked about um if you're just if you're just doing a pre-post sort of um a uba study as I like to call them um they're unlikely to get published in in a reasonable um educational Journal um so be aware of that I think um so doing the doing the right sort of study designs um upping the the rigor of your research is the first key but then the second key when we get to scholarship is looking at your specialty societies looking at educational special societies as AMC um or the the IMC one of those areas to present research are the best ways to um are the best ways to disseminate things I also like to think about um going um a little bit local Regional National some and then maybe a a poster work B new script and then like locally just think about every year we have a a a teach education day um where we can come and submit an abstract and it's really a low stakes opportunity to talk about your educational interent Innovations and then you've got a captive audience of people coming around and saying well have you thought of this or you know you get that local feedback and then a little bit bigger on Regional and then National so I think those opportunities can be outstanding excellent point all right we're at time I think we are thank you so much this has been fabulous lots of great tips um Dr uping has always been fabulous about if you have conversations that you want to engage in or you want to ask him questions outside of these sessions he's always been super open to doing that so please do thank you um and if you all want to talk about uh education scholarship our the teach office um office of continue professional development and faculty development we're always here for you to engage in those conversations and getting started with within scholarship and really um preparing for maybe a local presentation so John thank you so much for your presentation and uh thank the rest of you for being here hope you all have a wonderful rest of the day thank you thank you so much everyone have a great day.