2015 TEACH Education Day

Think about it feedback is everywhere in our lives in our professional lives of course that's why we're here right now but it's also there in our personal lives and no matter where we are personal or professional we're always giving and we're always receiving feedback so today one of my goals for you is to recognize that we are both always giving and receiving feedback and that feedback is a two-way street this morning at 8 I met with many of your trainees and that was the message that they got so the message that you're going to receive today is somewhat parallel what I was saying is that feedback is a two-way street we're constantly giving and receiving feedback in our personal and professional lives and I met with some of the trainees some of your trainees this morning and gave them that very message and in fact the message that I will will give you today is also very similar with a few added details the learning objectives for today's session include distinguishing feedback from evaluation then I'm going to give you some strategies for not only giving effective feedback but also for receiving effective feedback and perhaps as educators in this faculty member we may think more about giving the feedback and less about receiving the feedback but I would argue that paying attention to reception of feedback is very important not only for us as feedback receivers but also for the people we're sending the feedback to and we will review some feedback delivery methods so I'm sure you all know what I'm talking about here on this slide we've got ACGME LCME a ACN maintenance of con certification all of this all of these bodies basically have competencies that have been described be it as an EPA as a milestone as something as a part of the maintenance of certification and today's learners are learning in the setting of competency-based education models so what is that well if we think of a competency-based education model and compare that to a traditional model and the traditional model we really focus on what is being taught and how we're teaching it I notice when I said the what and the how I didn't talk about the learner that's kind of important that learner is key and integral to the learning process the cop sees competency-based education model focuses on the learner and puts the learner in the center and this is accomplished by essentially elaborating outcomes and using assessment to do that so within the competency-based model our learners learn to learn using the self regulated learning model and you can see that this can be broken down into four different acts there's planning learning assessment and then adjustment and for this session I'm really going to focus on assessment only if in the next session if you stay for the next session beginning at 1:00 we'll focus on both assessment and adjustment in more detail so let's look more and consider more about assessment when we think about assessment they're really two ways of assessment in the self regulated learning model so we have internal assessment which is self assessment and we have external assessment which is essentially feedback and that's why we're here today for feedback to learn about feedback what do we know about physicians health professions educators and we can even extrapolate that to adults in general about our ability to self assess well it's summarized on this slide here and you can see that physicians have limited ability to accurately self assess so this is this is definitely problematic for an educator to see and hear and if you look at the the bottom point here which is the most confident are the weakest self Assessors so the people the people who are performing at the lowest in their class are often the most confident and so that gap between their their confidence and their ability is largest in that group and those are the learners that we have to find we have to identify because those are the ones who are potentially a danger to our patients those are the scary learners we have to find them so self assessment is problematic enter external assessment ie feedback I like to think of feedback kind of as the check and balance of the whole self-regulated learning cycle so this is the way that we can provide information to our learners and hopefully help them to diminish that gap for those of them for those of our learners who have a large gap we would like to bring their perception with reality closer together so in 1983 Jack nd published this landmark article in JAMA and I consider this a really high yield paper so if you walk out of here wanting more and wanting to read something that's high yield easy to digest and something that you can repurpose very quickly this is the paper so go to Google type in NDE de and JAMA and you will find the article this article has been cited well over 800 times in the medical literature again it was published in 83 prior to 1983 this is how much was in the medical literature on feedback 0/0 so dr. nd realized he's an internist and he realized that he was giving feedback to residents and and he wanted to learn more about giving feedback want to further develop this skill so he of course went to the literature first and found the goose egg and then decided okay oh I've got to turn somewhere else so he turned to other areas and then essentially wrote this landmark article so the importance of feedback he states this is a direct quote without feedback mistakes go uncorrect 'add good performance is not reinforced and clinical confidence is achieved empirically or not at all that is very true and it's very scary and I urge you to remember this and remember this visa fee what I said about the kitten sing the lion we cannot self assess okay so feedback is a must in medical education let's now think about feedback versus evaluation what are the differences between the two feedback is formative I like to think of feedback as tiny grains of sand that represent little bits of information that accumulate in a pile and as the pile accumulates we get a better picture of a person's performance so these tiny grains of sand or these tiny pieces of information nuggets of information should be given in a timely fashion so very either during the learning moment or very proximate to the learning moment we shouldn't wait you know weeks afterwards to say hey do you remember when we were seeing that patient and you said this well I've been thinking about that now for two weeks and I want to tell you about that because chances are the learner has no idea what you're talking about we should be descriptive when we're giving feedback so what I like to do is imagine as if when I'm giving feedback imagine as if I am watching a movie and I'm describing merely what I see in this movie and if you do that it kind of divorces the the the emotions to some extent and allows you to really just focus and hone on your observing skills so it's non-judgmental we're not going to say great job good work keep it up outstanding that was awful terrible we need to talk because that kind of language is evaluative the neat thing about feedback is that it fosters learning so we're giving these nuggets of information these grains of sand are happening just in time in a learning moment so that if I'm seeing a series of several patients in a clinic in one given day I can give my resident or my fellows a specific piece of feedback on patient one and saying now when we see the next patient I want you to try this and then we can observe that and then we can give some more specific feedback so that we're fostering learning around a specific piece of information and in doing so provide opportunities for improvement most importantly feedback is informal and I think that this is something that we all have to remembers this is a very informal process so these are just little tweaks along the way evaluation evaluation is essentially the opposite of what I just described and you can read on the slide its summative so evaluation is a snapshot of a learner's performance during some period of time very different from feedback we're making a judgment we're providing a final assessment and this is high-stakes frequently becoming part of our learners permanent records feedback versus evaluation so today I'm focusing on feedback now if we are providing feedback as we should and we're giving our learners all of those grains of sand and giving them opportunity to see snapshots of themselves by the time they come to their evaluation they will not feel like this poor cat who is potentially about to meet its demise so we we have to provide that information so that when it's time for the evaluation our learners understand where we're coming from from a snapshot point of view system they're having a hard time everybody wants to hear you there you go is that better hello better back there is that better I hear it's not I see a thumbs up okay I'll look back there if we're still having problems give me a signal okay thank you so what is feedback if you look to the medical literature there are all sorts of different definitions of feedback in the medical literature I have selected two the first one is one that was written by dr. Indy who is pictured here and he's of course the one who authored the landmark paper I described earlier and he says that feedback is information that describes the students performance and a given activity that is intended to guide future performance information that we give to a learner another definition is one offered by valerie shoots and she says that feedback is information communicated to the learner intended to modify thinking or behavior for the purposes of learning again information that we give to a learner think about that that is very you need directional it's us saying blah blah blah blah blah to them now think about your interpersonal reactions both professionally and on a personal level that seems counterintuitive that we're just going to have this uni-directional exchange and expect this to be effective so I'm a pathologist you all heard that I can't I can never forget about my microscope so when I think about this I of course have to think about it from a microscopic point of view so my Forex point of view is here that's low power so we think about the learner the educator and the environment specifically the interpersonal environment that we are establishing and this is a different way of thinking about feedback and in fact we can now think about feedback being a two-way street and the first mention of this concept in the medical literature was in 2009 by being you and then in 2015 early 2015 t leo at all described the concept of an educational alliance so what is this for the psychologists and the psychiatrist's in the room this may sound very familiar because what they did is they borrowed the concept of a therapeutic alliance and repurposed it for the purposes of medical education in Health Professions education before we look more into this educational alliance that you're going to be so excited to go start forging once you leave here let's think about our learners I can't tell you how many times I've gotten involved in discussions about Millennials and oh the Millennials that oh the Millennials this oh my goodness we're they're so different we're so different so I thought it might be useful for us to think about the Millennials a little bit who are they who are they visa vie us in our own generations all of this information is taken from the the pew social trends so if you're interested you can go and look at all of that information online so this time line delineates the various generations and interestingly people who are considered to be within the Millennial Generation were born in 1981 to 1996 and they are in general our learners that we're working with I saw this and I thought that this was important for us to see today together in this survey respondents were asked generally speaking would you say that most people can be trusted or that you can't be too careful in dealing with people so this is the percent of people who say that most people can be trusted and you can see that the millennial generation feels very differently than the other generations so they're they're less trusting of others so something to keep in mind when you're dealing with a millennial learner Facebook and number of friends how many people you know how many of us are in Facebook self-disclosure I'll tell you I'm not so therefore I have zero friends for those people who do have facebook what you can see is that as people become younger and younger they have more and more Facebook friends so our millennial generation is very involved in social media and friending I like this one too when is it okay to use a cell phone and you'll see here during class or lecture our millennial colleagues say 22% say yeah fine go ahead go ahead sure why not I would say no but that's just me so now Millennials and feedback you know as I was preparing to come here for my visit I was speaking with dr. wicker and she and I were saying well it would be so great to find some information on Millennials and feedback and how much feedback they want because there's this perception that they wants a lot of feedback and part of that perception I think is because they were raised with internet so there used to this instant gratification and there used to this instant feedback if you will that the Internet has given them when I look to the medical literature to see if I can find this in some kind of table or you know numeric form it's not there no one has done this study yet that at least I could find so for right now I had to basically resort to dr. Google and and still I couldn't find that data so there is no data there there's a as I said a perception that Millennials want increased feedback but increase relative to what do they want more feedback than I want as a gen Xer do they want more feedback than say a baby boomer does I don't know I think that would be something interesting to consider though and then finally this is a fun thing and if you want to get your phone out and do this I won't be offended so there is a quiz and you can go to google and google how millennial are you it's a 15 question quiz super easy and what it will do is give you a score you can see my score there I scored 79 so I feel really cool but I will tell you that this this is not a validated educational tool it's all about fun so now back to the educational alliance so as I before I digressed into the the Millennial learner discussion you know I had introduced the concept of an educational alliance so an educational alliance is similar the concept is similar to that of a therapeutic alliance and what we're what teleo at all-state is that as educators we should strive to establish relationships with our learners that is built on negotiation and dialogue and that this is an educational relationship and so I think just by definition the word relationship suggests that there's some kind of law javadi or ongoing nature to this and within this relationship we are seeking a shared understanding we're negotiating and we're working together and as a result of that dialogue negotiation creating these shared goals and working together we're creating opportunities for feedback in order to do this there has to be a reasonable interpersonal environment and being you looked at this you want to know what are the features of an interpersonal environment that helped land feedback if you will and these are the three things that were identified number one source credibility so that means the learners perception of your ability to give feedback based on your years of experience your expertise your knowledge of the specific subject area so for example if you are in your first year out as faculty perhaps the feedback that you give to your fellow may not be as well received as say someone like me who's been out for ten years they might believe me a little more or maybe want to discount the information more readily from the version of me that was from only one year out so source credibility is exquisitely important next is beneficence so we need to demonstrate to our learners that the reason why we are engaging in this feedback conversation is because we care and we care about them and we care about seeing them move along the continuum from incompetent to competent and we care about our patients they need to know that they need to feel that if they don't feel it and if it's not there your feedback will not land and then finally a perceived alignment so show them that you want to work with them that you want to help establish learning goals and help them reach their learning goals now as I said not all feedback lands and there was a paper that was written by velocity at all published in medical teacher and it's actually the series of best evidence in medical education sir this is the aka be me this is be me guide seven and what they found in this literature review is that of all the studies that they included in their review 74% of the studies were able to demonstrate a positive impact based on feedback that's great but that also means that almost a quarter of them more than a quarter of them were not able to demonstrate that and that the feedback didn't land if we look at characteristics associated with a positive impact source so again going back to those three things I just mentioned and duration so that's why I'm trying to make you understand these educational alliances these ongoing discussions that we have with our learners is key now that can certainly be a challenge when we think of about this as medical educators in our curriculum structure because sometimes our residents are only with us for a couple days or a couple weeks at a time and then they disappear for months at a time depending upon the structure and what kind of program you're in so that's certainly a challenge this is something that I really hit home with our learners this morning the concept of humility and this is this is a paper that was written by dr. Grupp in who's out of University of Michigan and this is I think really quite elegant and what he says is this that as educators and in Health Professions education we need to remain humble and when when we are remaining humble we need to be open to the idea that we as educators we as learners are fallible that we're not perfect that we do make mistakes that we have to ask for help that we all need a feedback and that we can all possibly benefit from changes I think this is really quite beautiful and I bring this to you today as educators because if we're expecting our learners to embody this then we too have to embody this we too have to come with this attitude to feedback into medicine one example I'll just tell you a personal anecdote that I do is as a pathologist we do correlations between different specimens and anatomic pathology cytology and histology and many of the faculty members will review these cytology histology cases in their offices alone after the residents have gone home or after the fellow has gone home not me I review it with my trainees and we talk about the cases and we talk about why is there this discrepancy is this discrepancy really here is this a sampling issue is this a diagnostic error how can we learn from this and I think if you are open to the idea of talking about the fact that we're not perfect then it shows them that they can be - back to my microscope so the 4x low power view learner educator and environment these are the basic components now if we go on higher magnification you can see that that this is far more complex than what I'm letting letting leading you on to believe so another piece of the environment that we have to consider in feedback is the physical environment so think about where you are literally and what your message is going to be if you're in an elevator or in a hallway or in a patient's room the information that you describe to your trainee is going to be far different than if you are in your office with the door closed in a box of tissues right that's kind of important and sometimes we are so rushed because we have patients stacked up in clinic or we have we know we have to do three procedures today and I don't have the time and so we don't think about this we fail to think about this and this is a very critical component of the environment and if you don't consider this then your feedback isn't going to land it's going to hit the floor learning goals within the feedback relationship you want to work with your learner to establish learning goals you're going to use the feedback guidelines well that's great what are the guidelines so today I'm going to give you strategies for not only giving feedback but also for receiving feedback so the guidelines will be used twice and then also you're going to help your learners make a learning plan one of the problems that we have is educators and this is well documented within the literature is that there is a disconnect between the amount of feedback that we as educators say we are giving right I see heads nodding and the amount of feedback that our learners say they are receiving why is this well I don't know that we know that there is one universal answer but potentially one solution is that we're not labeling our feedback as such and this is an easy fix so if you are going to give feedback when you're going to give feedback just look at your learner and say I'd like to give you some feedback now is that okay is this a good time let's do some feedback let's do it it's feedback Friday you know whatever you want to do one of my residents told me about at their prior institution they had feedback Friday so so I mean you can you can create whatever culture you want here but the idea is to work to create a culture of feedback and label it as such so that we're all aware when feedback is happening and that's why I started my talk with the whole think about it feedback happening all the time it really is and it's only when you become open and mindful to the idea that you are giving and receiving feedback that you begin to see it more and more in life as I said teacher and learner need to be allies with common goals so we need to work to establish these educational alliances and within those alliances established learning goals I always provide feedback in a timely and expected way so if our trainees don't seem like they are ready to have a serious discussion that's okay that's okay we all know that we're not always ready to have serious discussions all the time and perhaps the best thing to do is to pause take a moment to collect yourself or allow the learner to collect him or herself and then reassess to see if it's a good time and then always base feedback on direct observation feedback should be focused on specific behaviors not the person so as I said think about what you're seeing as a movie and then describe it as such essentially don't criticize the person don't say you you you that that's not going to work very well feedback should always be descriptive constructive and not judgmental and here's a here's one that I really like include positive behaviors and I put this here because some some people have a tendency to include only the negatives and some people have the opposite problem which is to only include the positives and that's I think that's probably what happens more frequently is when we're giving feedback we just say oh you did this well you did this well I saw this blah blah and all the feedback ends up being positive so why would we do that now I heard two great answers because it's easy number one yes that's very easy you don't have to worry about the tissues coming out you don't have to worry about your wristwatch and the time ticking away and you want to be liked you want to be liked so what you have to realize is that sometimes it's okay if you're not liked and that's a hard role as an educator for us to take on because we all do want to be liked right we all want to be liked as humans as educators but we're not doing our learner's favors by doing this so we have to sometimes have these hard discussions regulate the quantity we can only hear so much until we just kind of shut down and we say enough it's kind of like you know when a patient receives bad news you all are familiar with those concepts you know they can only take so much information and in bite-size pieces of information so same idea I think applies here limit feedback to behaviors that are amenable to change start with standard language so here's a great thing that I do and what I do is I incur my feedback I start by anchoring my feedback language in milestones and this is one that I use a lot because I have a busy fna service and I want my trainees to be able to anticipate the needs of the team and an assist without being prompted to do so and you can see that that's a level three so I get my trainees to a level three pretty quickly so what you do is start with the standard language and then paint in the details afterwards I find this to be very useful here's a feedback tool box that I have loaded for you all today and the ones that the methods that I selected are I think pretty easy pretty quick to learn you can walk out of here remembering what I said about them and start practicing them I'm not saying you're going to be an expert because you know I think giving feedback is always it's an art and I think we're always practicing to be honest so in the toolbox we have the sandwich Pendleton one-minute preceptor and soap how many of you know are familiar with feedback sandwich yeah okay so next slide this is pretty obvious so positive constructive positive one of the issues with the feedback sandwich is this this is one of the problems and so there are two ways that we could look at this excuse me depending upon your learner okay some learners are overconfident so guess what they're going to hear the buns I'm sorry the buns they're gonna hear the buns so they're gonna hear message received I'm awesome now a different kind of learner who's self-critical is going to walk away and hear the meat and they're gonna walk out thinking this which is suboptimal so that's the danger in this I think you know it's it's a great way to start and you can obviously gauge your interaction because you're doing this and your educational alliance now and so you can use the method and change the method as you see fit based on how the feedback is landing Pendleton's method Pendleton's method is also fairly structured and it elicits positives and negatives just like the sandwich but instead of us providing the detail what we do is we ask the learner to tell us what he or she thinks she did well and then we have a discussion about that and then move forward and repeat this whole process with areas for improvement the one-minute preceptor is something I am very willing to bet that many of us in this room are doing intuitively without really realizing it and that is you get a commitment so here's a real-life example I do this at the microscope with my fellows and residents and medical students all the time we look at a cervix biopsy and I say what's your diagnosis and my resident or fellow may say well dr. Beane I think this is invasive squamous cell carcinoma and I say wow really this person's gonna have a rad hiss now are you sure about that so I'm probing for supportive evidence and then I listen to what they say and based on the information in the knowledge and where they're coming from I then teach the general principles that can help them arrive at what I perceive to be the accurate diagnosis and then eventually I will do the great reveal and say yes I agree this is invasive squamous cell carcinoma this is not H SIL this is not else'll this is not any no carcinoma it's not a is and we talk about the reasons why how many of you have heard of this before this is great yeah so so if you haven't heard of it Mike as I said my my guess is is that most everyone is doing this intuitively it's nice just to see it on a road map soap this method is a method that was described by Lisa Halle out of the Carolinas Hospital in Charlotte and I like this method because it is useful for delivering very serious feedback and it has a built in method for making sure that the feedback is landing which is the asking for understanding so if you're having a serious feedback session and you need to make sure that they understand what you're telling them this is the method to use so you start with self reflection ask a learner about why you're here or what they think about subject X provide them objective feedback around it and then this is the critical step the learner to summarize what they've heard so far and in that one moment when they're telling you and parroting back the conversation you will know whether or not they understand why you're here and what you're talking about and then this also has a plan for follow-up which I think in these situations in these settings is very critical so here's your tool box that I have armed you with the sandwich Pendleton soap and the 1-minute preceptor there are all sorts of different methods out there that have been described and you know we can go on and on at nauseam about all sorts of different methods but I think if you use the guidelines and do so in the confines of the educational alliance you'll be well served so another thing to consider is that we don't always we we feel like we don't have time to give feedback but I would tell you that we actually do have time it branch at all published a paper in 2002 and in it they described three different types of feedback brief formal and major and most of the time we are giving this brief type of feedback so these are just little as I said grains of sand pieces of information that we're using to change very subtly sometimes even a learner's behavior we can also give formal feedback which happens following observations it can be five to twenty minutes long and then finally major feedback and major feedback is something that happens kind of at the rotation midpoint so this is a technique or a type of feedback that you can use when you want to give someone kind of a middle-of-the-road evaluation not evaluation but let them know where they are and essentially give them information on where they need to be for their evaluation they're in point evaluation so we've looked at the 40x view of component components of effective feedback and I've given you the guidelines for giving feedback now let's look at receiving feedback so how do you receive feedback I don't know it depends upon the context depends upon who you're talking to right sometimes you're happy sometimes indifferent sometimes no way so I remind you that feedback really is a gift and the more you are able to think of feedback as a gift and frame the conversation in light of feedback being a gift and being a learning opportunity not only for yourself when you're receiving feedback but also for your learner when they're receiving feedback the better the feedback will land and stick and be useful I put this picture here because this is the same picture I used on the humility slide and I think that it bears repeating we need to come at this with humility we need to be humble and so I remind you the words of dr. Griffin he said that we have to remember that we're not perfect that we all make mistakes that we all need help we all need feedback and we all can improve something and I think that if we come to this with that in mind that makes us better it makes us better for ourselves as individual professionals it makes us better for our learners and more importantly it makes us better for our patients so if you were in a situation where you're receiving feedback and the person giving you feedback isn't giving you much structure this is what you can do you can help them provide structure to your feedback and make it more helpful by telling them what your goals are so tell them what your goals are share them with the person giving you feedback and then with that person develop a learning plan so you're essentially forging the educational alliance for yourself and I think that's okay and I told your learners that's okay too yeah so the question is what's an example of sharing some learning goals that you have for yourself I'll tell the same story that I told this morning which is a really good one in pathology we rotate with our residents essentially on a weekly basis and oh I don't know over a year ago I gave some version of this talk to my residence fellows faculty staff educators etc which was a Friday and that Sunday night I checked my email my Duke email and there was an email from my first-year resident who was going to sign out GYN cases with me that means look at GYN cases with me on Monday dear dr. Beane I am so excited to be signing out GYN with you I have been on GYN three times before and this is what I would like to work on this week and he outlined in bullet point form specific things that he wanted me to help him learn so it doesn't get better than that and I would say that that is where we want our learners to be I was so impressed I printed the email and ran to the program director and said look at this look at this this is terrific so what he and I did during that week was we started with that email that I printed and I assessed his ability based on his learning goals and I said okay you know I think this one is terrific you don't need to work on this one let's consider adding this as an additional learning goal what do you think about that and so what we did was we were able to move him along and in doing so develop a learning plan by starting with those goals when you're in the moment and when you're receiving feedback there are a few things you can do the first is listen so just listen to understand turn off your inner voice your inner critic and just listen and I know that's really difficult it can be very difficult depending upon the context but the more you can do that the better you will be as you're listening be actively involved don't be afraid to contribute to the conversation if you're receiving this feedback if you don't understand what you're being told that's okay you can respectfully say wow I'm having trouble understanding exactly what you mean here can you give me some more information or provide a specific example to illuminate this idea seek clarification when necessary easier said than done try to control your defensive behavior we all have those defensive behaviors where the walls come up and we really just don't want to hear what people have to say so the more that you can be mindful of how you react in a feedback receiving situation the better you yourself will be at receiving feedback I'll give you a personal anecdote I'm a fitness I'm really into fitness that's one of my hobbies I love to go to the gym and I've been doing this with a trainer now for a full year and when I first started the trainer would say no do this do that this is wrong you need to fix this fix that and at first I thought really I thought I was pretty good you know and then at some point along the way I realized no she's telling me these things because she doesn't want me to be injured I'm taking this personally for fruit for a silly reason there's no reason for me to take this personally so then I began to realize she's giving me feedback I'm hearing feedback this is all feedback so I had this Eureka moment and then once I made that realization and began to be able to recognize feedback in my personal life it filtered over to the professional and actually made these things a lot easier so I would challenge you to start recognizing feedback in your personal and professional life as I said it's happening all the time whether you recognize it or not and it will continue to happen like if you're driving down the road and someone beeps at you guess what that's feedback that's feedback right okay so if you look in the medical literature there is essentially not much at all in the medical literature on receiving feedback there's one paper that I found that's really good on receiving feedback most all of it is written from the point of view of the educator giving feedback but lucky for us there's this great book that is written by stone and Hien and they are two faculty from the Harvard Business School and in this book there's a they also actually have a paper which is essentially a distillation of some of the concepts from this book so you can google that HBR stone in heme and find it pretty easily they have identified three triggers that block the message of feedback the first is a truth trigger and this really refers to the message itself so there's something wrong with the message has nothing to do with who is sending it it's all about what you're hearing and there's no way it's true and there's no way it applies to you the next trigger is a relationship trigger and the relationship trigger has everything to do with who is giving you the message everything to do with it so how dare this student of mine say that my lecture needs improvement that's just you know how would a student be able to judge my lecture that would be a relationship trigger okay and we all know we can all improve right all right identity trigger identity triggers are triggers that where we feel like our very core our values are under attack and the one I like to use about myself is professional attire so if someone were to say to me dr. bean you really don't look that professional today I would say really I would feel really tiny so how do we overcome these feedback triggers well the first thing as I said is to begin to be aware of when you're receiving feedback and how you respond so become mindful of the whole feedback process and recognize that when it's happening not only in your personal life but also in your professional life disentangle the what from the who so when you're getting feedback when you yourself are receiving feedback think about what is being said and who's saying it and then focus on the what if you are someone who has a problem especially with relationship type triggers invariably feedback and evaluation become commingled and when you hear a valued of language kind of bundled with feedback well there's not much you can do about an evaluation because the judgment has been made but what you can do is take that feedback information and use it to your benefit so sort the feedback from the evaluation clarify as necessary and this is one of my favorite things that I do the first time I did it I really through the people off and this idea comes directly from stone and mean the idea is ask for just one thing so how many of you have gone to your annual faculty review performance review and they say great excellent keep it up you're on your own target maybe two years three years from now we're going to have you assemble your dossier put you up for promotion excellent is that helpful no no it's not helpful we want to know what can we do better are there Mary major areas of deficiency that we need to work on let's hope not but if so we especially need to know about that and if there are minor areas that we need to work on we definitely need to know about that because we want to be the best for our patients our learners and ourselves and the only way we can do that is by hearing what we need to work on so if you're in that situation and they say great job keep up the good work excellent outstanding you're the best my rock star look at them and say can you tell me one thing I need to work on please tell me just one thing and in doing that you give them permission to tell you that there's something that you need to improve and remember we were talking about one of the reasons why we always only hear positives is because we want to be liked well our our supervisors also have that they also come to that relationship with that with the same lens the same human lens and then you can engage in small experiments so based on your observations of the information that you're receiving make small changes and then observe what happens as a result of your small changes if it's working great keep it if it's not working that's okay take two you can try something different the next time in the end feedback is a gift and as a receiver of feedback we have the power to receive it keep it and use it receive it keep it not use it or not use it at all and so that's one of the powerful things about this relationship is that we don't have to use the information now I would encourage us as educators and as receivers to always try and keep and hear and listen and make good informed decisions if you're going to reject feedback so the components of the effective feedback from the high power view again include environment and that's the physical and the psychosocial environment learning goals we're using our two types of feedback guidelines guidelines for giving guidelines for receiving and we're making a learning plan when we come down to low-power you'll remember that our feedback is best best occurs within the context of an educational alliance that we're forging as learner and educator together so in conclusion I will remind you that feedback is integral to learning so we cannot self assess accurately we all think we're terrific drivers right we're not we're not all great drivers establish educational alliances and as I said I know this can be a challenge depending upon the curriculum structure that you're teaching and learning in but insofar as much as it's possible when you are having that connection with your learners especially forge these alliances and build on them and in doing that model humility let them know that it's okay to be wrong and that you yourself have been wrong share your mistakes I think it will make for a more open and honest relationship and then finally label feedback when you're giving feedback use the feedback word so at this point I'd like to congratulate you all on your inaugural education day for showing up for teach Academy this is really incredible and I'm so happy to be a part of this thank you for having me your campus is very beautiful everyone's been so warm welcoming and I'd love to take your questions comments at this point thank you [Applause].

How many of you are familiar with plan do study act the plan do study act framework ok this is kind of what I expected a good number of you and that's good note good news for me and good news for you because the concepts of plan do study act essentially parallel what we will be talking about today in this next session we're going to look at self regulated learning and then I'm going to focus my microscope in on a few details of self regulated learning and these are ones that specifically interest me so that's why I kind of focused on these areas these were areas that I want to know more about we're going to discuss the challenges of self assessment identify barriers to direct observation and then understand the importance of self reflection today so I started with asking you to tell me show me if you know what plan do study act is and the reason why I did this is because it occurred to me that plan do study act is very similar to the self regulated learning cycle where we have planning learning assessment and adjustment so plan do study act this is the plan do study act of learning great so now you can walk out and say I've got it right I touched on this in the first talk we're thinking about competency-based education and for those of us in the room we're all educators and you know now that our trainees are being held to compensate standards and these are put forth by the various groups that I have listed on this slide so we've got milestones EPA's competencies and we're judging our learners against these specific standards so the competency-based education model is different than the traditional model and it's different in that we put the learner at the center of the competency-based education model so we're focusing on the learner and we're focus focus on specific outcomes based on these benchmarked criteria that we have from the various societies so today what I'm going to do is take a stroll around the self-regulated learning cycle and I admittedly will be focusing more on assessment and adjustment and much less on planning and learning before we start into the specific quadrants of self-regulated learning I think it's important for us to consider kind of the things that govern self-regulated learning so the first is metacognition and metacognition is thinking about thinking so you can see Einstein's quote here education is not the learning of facts but the training of the mind to think so this is essentially metacognition thinking about thinking and then as part of the self regulated learning cycle we have to think about strategic action our learners are making strategic actions as they move through this cycle another thing to consider is motivation to learn now if you look at this slide you probably can't see it in the back I receive some feedback after my feedback talk that was kind of fun so the feedback I said I heard was that they can't read that you can't see everything in the back so I had the most relaxing morning this young woman says I took a run watched the Sun come up went to loge yoga class and meditated and then she sits down I rescued an endangered wetland never sit next to an overachiever before noon so we all know that that we that there are learner differences out there and some we have some / performers and some underperformers and so what are what are there differences between our underperformers and our high performers well in this study by RT know at all they identified some specific differences and what they found is that low performers compared to high performers have decreased task value so what does that mean if you give a low-performing learner an individual learning task to complete that they will place left less emphasis on that less value on that specific learning task than say a high performer the other thing is that they have decreased self efficacy beliefs so and here comes my mic so self-efficacy beliefs this is the belief that individual learners bring to a given task and it reflects their amount of confidence that they bring to a task so how how they believe they can or cannot successfully complete a learning task so our low performers have decreased self-efficacy beliefs compared to our high performers also interestingly low performers have increased course-related anxiety they are increasingly frustrated compared to the high performers and they also express boredom and learning and I thought that this is useful for us to understand because the self regulated learning cycle is essentially all learner driven and as I said in the prior presentation the only check-in balance there at least the only obvious check and balance there is the external feedback that we give so we have to I think remember these things when we are providing feedback and be aware about this so let's look at planning in the planning phase our learners are setting goals have you all heard of SMART goals before yes good this is not a new concept so I think this is a great idea to use of course I use these personally and professionally and I use this this concept with my learners as so help your learner's to establish SMART goals goals that are specific measurable attainable relevant and time-based and know that the goals that our learners set are to some extent going to be nuanced by the self-efficacy beliefs that they bring to the learning moment and that this is all contextual so it all has to do with a specific learning event so if you know if you were to ask one learner to go in and perform a focused physical cardiac exam they may go in and do very well but if you ask that same learner to go in to see a patient and do an examination of the hip that person may not do well and they they will bring different beliefs based on their ability and perception to do that another thing that we have to consider during the planning phase is motivation and self-determination theory and this is a theory that is from the psychology literature written in 2000 by Ryan and DC and essentially what they describe is a continuum of motivation with respect to self-determination so on this side of the continuum you see someone who is completely a motivated so there's no motivation there's no external or intrinsic motivation that they're considering and as we move along the continuum toward fully self determined and intrinsic motivation you can see that the learners begin to contemplate extrinsic motivation and then internalize that motivation to varying degrees as they move along and become more autonomous self regulators and now for the learning so this is the the do okay if you're thinking of plan do study act this is the do the learning I think this could be a whole you know series of lectures and you know I'm I do not have a degree in education and I do not pretend to so what I will say to you is that learning the things that have to be considered in learning are those things listed here on this slide Epis Tamala G which is individual beliefs about knowledge and knowing learning styles how does a specific person learn I know from you know for myself I'm a very visual person I am an anatomic pathologist so I learn by seeing I also learned by doing kinesthetics too so we have to help our learners to understand how they learn and then they're going to use learning strategies and principles and methods to execute the do of the plan do study act of Education and now on to assessment in the first session I really focused a lot on the assessment piece and in this session I'm going to focus a lot also on the assessment fees piece but instead of focusing on external feedback we're going to think about self assessment and also direct observation as you saw in the prior session assessment is really composed of internal assessment and external assessment so by internal assessment we mean self assessment and external assessment we essentially mean feedback so I ask you what is self assessment what does this mean to you you know I guess we all do it all that reflecting on yourself finding out what you know we do it all the time yes me and also the self assessment what so he's he's giving us a very complex explanation of and not I'm not judging I'm just saying I'm doing this for the purpose of showing us that there is no one definition so he's talking about the who in the what of self assessment who are we and then also the what what am i doing how am i doing it what else what other definitions or things come to mind when I say self assessment to you today strengths good so I hear strengths and weaknesses needs improvement what else emotional intelligence that's right what is it that you're setting what assessing what part of yourself are you assessing anything else any other comments what is your measuring stick yes that's what this this is great this is why self-assessment is so difficult because self-assessment can mean any of these things and all of these things I went to Google and I asked Google what is self-assessment and this is what Google gave me and I kind of liked it so it's assessment or evaluation of oneself or one actions and attitudes in particular performance at a job or learning tasks in relation to an objective standard I really like that it's precise and I think in education this is perhaps a useful definition that we could use so if you think about stuff the self-assessment as coal-tar at all did in their review article in 2008 they did a meta-analysis of self-assessment literature and when they set out to do this study they want to know if self-assessment could do the following things can can it identify learners needs so can self-assessment be used almost as a needs assessment tool for our learners can self-assessment influence learning activity can it change clinical practice improve patient outcomes be accurate sounds like utopia so what do you all think the answer is no you knew at least one one answer right you knew that it's not accurate so the answer is no they found no evidence in the literature that it could do any of these things and I think part of the problem has to do with the issues that we just discussed in the definition and what are we measuring why are we measuring it so what do we know about self-assessment self-assessment the the classic study of self-assessment is written and published by Kruger and Dunning and I'm going to show you two pieces of their study that they published in 1999 what they did is they asked a group of Cornell undergrads to read ten jokes and rate the jokes that they read on a funniness scale how funny is it is it funny and then after they did that they asked the the students the participants to rate their perceived ability to identify comedy and whether or not something is funny and then they plotted their data and as you can see this line here is the actual test score and it's sorted out in quartiles and the actual test score is plotted against the perceived ability there are few observations you can make and use I'm sure I've probably already made those observations by now first thing is that from here on over these group this group more than 75% of the people overestimate their ability to perceive comedy this tiny group here are high performers actually underestimate their ability and the group with the largest Delta is the bottom quartile and this is what I was getting at in the first session this is the group we have to worry about because these are the people who don't understand their actual performance so Kruger and Dunning did another study they said okay this this comedy study is great but can we reproduce this information in a cognitive domain and so what they did what they compiled a series of questions that were excerpted from the LSAT exam the standardized LSAT exam on logical reasoning and they did a similar study so they gave the Cornell undergrads this logical reasoning test and then they asked them two questions this time perceived ability and logical reasoning and perceived test score and plotted them so in case you can't see in the back this is the test score line perceived ability is here and perceived the perceived test score is here and again the same three observations can be made you all saw this slide already right so this is what we know about medicine so the question becomes then okay so Kruger and Dunning studies are very important can we show the same thing in medicine and in fact we have shown that these results have been duplicated numerous times including the concept that our most confident self Assessors our most confident students are the weakest self Assessors so what are some things that we have to consider in this whole self assessment picture what what are things that can cloud the information or make it difficult to interpret the one thing is motivation so how motivated was the self Assessor to accurately self assess that's not an easy question it really depends upon what the tool is what you're assessing why you're assessing it how you're assessing it how long does it take you know is this person in clinic you know all of these things factored in to the mix here next is metacognition and in fact some people claim that the people who are the lower the low performers but overestimate their ability do so because they have deficiencies in metacognition there can also be selective recall of past behavior so depending upon the timeline of an actual event and the self assessment measurement then that can also confound the information another piece to this is that we tend to ignore proficiency of others and one of my favorite things to do is ask everyone to raise his or her hand if you're a good driver we're all good drivers right we're all good drivers I think I made my point right we're all really good drivers and so that's that's the concept here basically is that where we're not taking into consideration the performance of others when we self assess and then another thing which i think is a really useful piece to keep in mind is how is this self assessment task going to be used based on the information that is provided it may be used in different ways so if we're really just using this in the self regulated learning sauce cycle in the context of feedback low stakes then you may get one answer but if you tell a learner well you're going to turn this in to the CCC the clinical competency committee and we're going to look at your self-assessment and then we're also going to assign milestones which I do I mean you're going to get a different answer in theory and that I will tell you has been a very interesting process that I have learned a lot about my trainees using that self-assessment so as adults we're all good drivers we're all really smart we can detect comedy so if we were left to our own devices we would just kind of lead ourselves blindly off the cliff like this poor student so can we improve self-assessment well my recommendations to you I'm trying to make this practical so that you can walk out of here and have you know if you're inspired about a specific topic so that you can have some action items or at least somewhere to go look and do something so improving self assessment here are my recommendations for this provide explicit assessment criteria so be specific and let them know let them know what you're looking for benchmarking no benchmarking is is a really interesting concept and what you do in benchmarking is essentially show them differing examples of high mid and low performers and then have them self assess you have them self assess before I mean if you're doing a study you would have have a self assess before show them the benchmarks and then re self assess and you can see how that self assessment changes and it does change somewhat after you benchmark and then of course external feedback be it verbal or video so let's think about about the self assessment and feedback as I said feedback is the checks and balances - self assessment into the whole self regulated learning cycle and you know the importance of feedback by now you're drinking the kool-aid so this slide looks very familiar to you because I use this slide in the prior presentation and you'll see then the third item is underlined so when we're thinking about giving feedback feedback always has to be based on direct observation so I want to delve a little more deeply into direct observation so what happens if has anyone ever tried to give feedback to someone about something that you didn't personally witness mm-hmm mm-hmm and it was really great wasn't it yeah it doesn't take very long for you to realize that you really should have seen it and for the learner to realize that gee you really should have seen it so and if you haven't done it maybe you can like watch someone else do it or something you don't want to do it yourself it's unpleasant so it's like the blind leading the blind so direct observation is really important because in this study by Mazur at all what they did is they asked a bunch of USMLE registrants about their perceptions of the knowledge of a learner's performance and feedback and direct observation and what they found is that direct observation is strongly associated with the perception of accurate performance and I think if any of you have had that you know experience of trying to lead the blind while you yourself were blind you understand where I'm coming from here the other thing they found is that the perception of accurate knowledge is associated with increased feedback so the more we observe the more accurate the perception is of the learners performance and then the learner receives more feedback so why is feedback important well if you consider Miller's pyramid feedback is the penultimate it's the best this is the best way to assess because our learner is doing what he or she knows so are there examples of direct observation that are going on in your training programs that you can share watch the pre patient practical exam skill check off watching them treat a patient I'm sorry I didn't hear you okay he said we worked with the trainee on each and every case that's aspirational anything else so this this gentleman in the front said that he's working on helping a learner to talk about communication style with patients so I think that you know the very basic form of direct observation is us just going into a patient's room and being there and being present and watching and not being involved just merely observing that's the basic example and it seems so easy to do but it's not we're busy really busy more aspirational things that are probably happening here here's an auskey obstructive clinical objective structured clinical examination and this is an auskey that is depicted in a nursing school and then here's a mini cex tool this is a tool that was published and is on the ABI M's website so you can find out lots of information and potentially use that tool for direct observation also I would say that this JAMA article in in 2009 by Cogan at all is a compilation of all sorts of self-assessment tools and if you need other assessment tools that are validated I would point you toward the double-a MC they have a meded portal and they also have a dream which I can't remember all the acronym but it's something like repository and education and assessment tools anyway there are all these validated tools that you can use for free all you have to do is just sign up and get a login yes and you all see it all the time when your faculty come into your office as a program director I'm telling you the students that I worked with was horrible and that they tell you exactly what they should be telling the student but then they are expecting you to communicate that to the student so this can be really helpful so dr. wicker made the comment basically that sometimes as educators we receive feedback verbal feedback or informal conversations from faculty members and they're telling us that our learners are underperforming and they essentially want to hand it off to us and want us to fix it but really what we need to do is say okay here are the tools that you can do this is really something that you need to handle if I try and fix this at this point it's not going to be helpful and so you can provide them with these direct observation tools so that you can begin the conversation of feedback so does direct observation happen there is an nvme study that was published online from the early 2000s and they wanted simply to know how many times are medical students observed and who is observing them and you'll see that medical students more often were observed by residents rather than faculty members and 20% of them reported that they had not been observed or only observe zero to two times that's kind of sad here's another study by Hallie at all and this is a study of medical students third-year medical students at UVA and what she did was she asked the medical students whether or not they had been observed by a faculty member doing these three things history-taking doing a focused physical or a complete physical and the percentages you see here are the percentages of students who report having never been directly observed doing these things so I highlight this I mean I think these numbers are glaring you know they range the lowest number is 25 the highest number is 91 so there is a range and this is one local institution but I think that you know it's certainly probably not unique we're all facing similar pressures so what are some of the barriers to doing direct observation that you all are facing in your daily practice time time time and more time just yes that's an excellent comment so she says when you're going to observe someone it's very difficult not to step in and not to engage and not just to take over so avoiding that urge to just do it because you know you can do it quickly and move on to the next step or whatever anything else any other barriers you've encountered so far time mostly time okay so here are my recommendations for direct observation if you want to bring direct observation into your learning programs and into your curriculum these are the high level recommendations that I have for you and these come from Eric home Bo first is develop policies that encourage and reward direct observation I think that's pretty obvious we adults respond to rewards and carrots so if you want it you have to build it in and create a culture for it and so developing those policies will be huge next he he says that we should create educational champions so in building these policies what you want to do is identify people who are going to go out and serve as your educational champions and these are the people who are then going to go out and cross pollinate the concept of direct observation so they're going to work with other faculty and help them with faculty development and they are also going to serve as a supportive role and a mentoring role your educational champions can do that for you and then also he says well if you're going to put all this time and effort into it and money you should evaluate the process and see if it's actually working do you have more increased direct observation and is it helpful now here is the here's the more granular look at my recommendations and these essentially come from Howard at all and how many of you are familiar with the 12 tips journal or the 12 tips article series and medical teacher yeah couple so the 12 tips I think are really high yield and this comes from this information is coming from one of the 12 tips so these are essentially there are articles that are usually just a few pages long and they outline 12 really high yield tips on great educational topics there's one on feedback there's one on self assessment there's one on direct observation you name it is probably out there so you can Google 12 tips med medical teacher and find them so if you're doing direct observation define the objectives to guide the direct observation so you know you want to give it a framework you also want to decide is this formative or summative okay you know it could be both right you know if depending upon how you want it to function in your curriculum next always try and use an existing validated tool you do not need to recreate the wheel it is so hard to create an assessment tool and to validate it it takes so much time and energy so if there's something out there that's already works well and and has been peer reviewed then use it absolutely direct observation is probably going to require a culture change to some extent and I think culture change and change transformation is a whole session unto itself obviously we're going to need to have faculty development and if we're doing direct observation the reason why we're doing the direct observation is to give feedback so you have to remember to actually follow through and give the feedback make a learner action plan another thing you have to remember to do is orient the learners so if you're going to have them participate in these direct less you observation exercises you need to let them know what you're going to observe why you're going to observe it how you're going to use this information what are the stakes for them what does it mean them why is it useful for them get some buy-in from the learners and give them an opportunity to understand and to ask questions and then I would say use the tool multiple times per trainee so the idea is you know if you're if you're going to to do this so so that you get a whole picture remember my analogy to feedback being like grains of sand that accumulate into a pile that's what we want this that's a similar concept here so accumulate the grains of sand so that you get a more accurate picture of your learners performance we all know that an N of one is just an N of one right and then finally measure the outcomes now for the adjustment phase there's self reflection in attribution when I look to the medical literature I will tell you that most of the medical literature out there is all theoretical on this this is all theory and I think I think this is a really interesting piece of the self regulated learning cycle and something that I personally became more interested in because I've been participating in an educational leadership fellowship for the past two years and part of that fellowship has required self reflection I'll tell you I'm not much of a self reflective kind of person and so when my mentor was telling me ok I need you to just write for 15 minutes after this learning exercise just write whatever comes to your mind I was like okay Robert here goes nothing and I literally set a timer I mean it was that bad and then I would send my mentor my comments and you know what he did he would highlight things at his own comments and then kind of reiterate learning points and bring in other things and I discovered oh my gosh I think there's something to this I should wake up so self reflection that's that's my personal story about why I want to know more about it so what is self reflection in the medical literature Aronson offers the definition on the top which is analyzed questioning and reframing an experience for the purposes of learning and then in the psychology literature I believe booed it all describes it as recapturing and experience thinking about it mulling over it and evaluating it I like there's simplistic definitions I mean we can get really theoretical here but let's keep it simple so what are some examples of self reflection well one of the most obvious examples is just simply talking so after a specific learning experience you might look at your learner's or group of learners and say how did that go how did it feel and then have a discussion about what comes up another example would be root cause analysis and of course that's more of a systems wide systems based self-reflection but it is a self-reflection exercise nonetheless I already mentioned you know personally what I do journal writing other people might just write or they might draw or they might draw diagrams Venn diagrams bubble charts connect ideas some people take notes like that even so that may be a form of self reflection also storytelling it could be a very different way you can give very prescribed self-reflection exercises and then you can also be creative I went to YouTube to prepare and to learn about self reflection and there are all sorts of creative ideas out there on YouTube and some of the more creative ones I found were art Twitter and even consideration of dreams with respect to self reflection so why is self reflection important well it's important because it's part of the exponential learning cycle Kolb's learning cycle is the most simplistic version of this and there there's all sorts of theoretical debate out there on exponential learning and oh it's not cyclical and it's not so linear as Kolb has described I'm not going to really talk to that the concept is though that reflection self-reflection occurs for adult learners in the context of a concrete experience and abstract conceptualization and experimentation and the real problem one of the problems they say is well adults may be engaged in more than one of these things at any given moment so it's important just to know that self reflection though helps us learn and this is how we learn through experience as adults now in the medical literature we know that self reflection adds meaning to complex situations and it enables exponential learning this is not based on data or a study by the way this is all theory and I think that this is useful for us to consider because as Health Professions educators we are dealing with complex situations constantly when was the last time you dealt with something simple doesn't happen that frequently so self reflection is perfectly suited for what we do so here are my recommendations your take-home recommendations for self recommendation for self reflection first thing is role model the behavior so just like I spoke about humility and the noon session you know you have to role model the behavior if you if you want someone to dress professionally than you dress professionally if you want someone to be humble then you need to be humble if you want someone to self reflect the new self reflect and you talk about your mistakes you talk about your errors you talk about your feelings about this exchange that you just had as a group between the patient establish an educational alliance with the learner so take take that information that I gave you last hour and then use it to your benefit because within that that educational alliance you'll have trust and you'll have a more open and honest and perhaps be able to get into the self-reflection and then finally facilitate reflection so make create opportunities for it to happen so setting the stage self-reflection links the past present and future and we use it to integrate the cognitive and emotional another great thing about self-reflection is that we can purposely and mindfully think about multiple different perspectives so we might consider alright well this is how I felt about this situation how did you the fellow feel how did the medical student feel who wasn't allowed to participate because this was you know an FAA ultrasound-guided FAA how did how did that student feel how did the patient feel how did the patient's sister feel who was watching all of this so when we consider those multiple perspectives we get we begin to see and reframe the situation and learn from those multiple perspectives and then you should always describe the lessons learned and that was one thing I had discovered that I was inherently doing in my own self-reflection is that at the end of the self-reflection exercise I would list three or four things that I had learned like life learning lessons and it's it's a really useful thing actually so you can use self-reflection also to plan for future learning so if you are really inspired to bring self reflection exercises and to give it a spotlight in your curriculums I point you to this paper and this is an actual snapshot of what it looks like so and this is of course from the 12 tips article series that I mentioned earlier so this is this is aspirational aaronsands paper is aspirational and what she does is in 12 tips outlines on how to guide for bringing self reflection into your work so we have taken a trip around the self regulated learning cycle or the plan do study act of teaching and learning in medicine if you will in conclusion it's important for you to know that self regulated learning is the theoretical foundation of competency-based education so we're expecting our learners to learn how to learn and the onus is really on them in this competency model we do not accurately self assess especially our weakest performers do not accurately self assess so we really need to be aware of those learners and especially give them feedback because our check and balance in this whole self regulated learning cycle is the external feedback which of course is obviously paired with direct observation because we will never ever do one of these again right once you do it once you'll never do it again self reflection enables exponential learning and so self reflection is the last piece of the the self regulated learning cycle and I think that this is a very important and integral piece to bringing everything together for the learner and at the end of the day if we've done what we're supposed to do we've brought our learners through the self regulated learning cycle we have watched them march along the continuum from incompetent to competent and then they're standing downstairs on these steps in this building this building next building this building yes throwing their caps in the air thank you very much you.