Competency-Based Medical Education

Thanks so much Mariah as as well as um Sherry and JT and Deb appreciate this and excited to be to sh be sharing the space with Dr Padet I'm gonna kick it off with you and me and then then we'll hand it over to Dr padet to to move to the GME space Uh let me get this moving So before I get started I do want to acknowledge that several of these slides were are either stolen from or or developed with um Dr Gonzalo our CC senior associate dean for medical education And just to refer back to the the teach the teach website which has a really comprehensive compendium of all of our presentations Jed did a presentation on competency based medical education back in last January and so that's available and you'll see several of these slides in that one as well Uh I also many of these conversations have been had with Dr Brock Mutesen So much of this is as with with all of it is work that that we do in the space all the time with a number of different people So Mariah already went through the objectives but just as a refresher we're going to talk about competence versus timebased curricula and I'll get to that in a in a few slides But then thinking about what this looks like in undergraduate versus graduate medical education and then how do we actually assess competencies what are the similarities and differences in which is what I'll be saying quite a bit and GME so our outcomes in undergraduate medical education are what we are ultimately striving for when we're developing our curricula our learning experiences our assessments We're thinking about ultimately what do we want and this was a slide um we went as many of you are probably intimately familiar with we recently restructured our phase 1 curriculum and that will be flowing up into our phase 2 and three curricula here in the next couple of months But when we were planning for our new phase 1 curriculum this was a slide that we often used to sort of anchor ourselves in in the development of of everything we were doing So when we think about phase one a lot of times students and and some educators think well our top priority is to make sure that they can pass step one but you can see that I I wrote that in much smaller font because that really is not the ultimate goal We certainly want that but that's the bare minimum Our next threshold of outcomes was thinking about we want to make sure our learners are ready for clerkships And that really helped us think about what we wanted to include or not include as we as we consolidated some of our our learning in phase one And then finally what we wanted to really keep in mind was our VTCSOM values which are represented in the graphic on the right of that slide And at the center of that is is the patient centered physician And then surrounding it are the identities that have really been at the heart of VTCSOM since its foundation and now encapsulated in in this graphic with master adaptive learner scientist physician and and systems citizen So these are identities that we want to be thinking about as we're even as we're developing a curriculum for our for our brand new firstear medical students And that's really the heart of of competency based medical education is focusing on the professional identity that we're striving for That's really our our north star But in order to get there we need we need the competencies underneath to to build up towards that identity So this is on the right is Miller's pyramid with an extra box at the top that was added in this um Cruis and Cruis model pyramid for competencies and professional identity development So again what we're striving for at the end of the day is having a graduate who can consistently demonstrate attitudes values and behaviors expected of one who has come to think act and feel like a patient centered physician But underneath that we have to describe what makes that up So those are the competencies and you can see sort of the foundation at the bottom with knowledge and then moving up that pyramid to knowing how and showing how and doing And competency based medical education is really thinking deeply about what are these competencies how do we describe and define them and then how do we measure them on this path towards identity formation so medical education is filled with all sorts of acronyms at the and and this is a slide that that sort of summarizes those but at the top of this list is entrustable professional activities and Dr Padet's going to talk quite a bit about those So that's in that red over on the left side of this diagram But built into those are domains of competence sort of categories of competencies that capture uh a specific dimension Underneath that are competencies or actions that someone performs or demonstrates within the practice environment And then to define each of those competencies we need specific observable and measurable achievements or milestones that show the development of that learner's competence On on the far right of this figure you can see our narrative descriptions As as faculty are assessing learners they are providing feedback and often that feedback touches on a number of different competencies And so teasing apart what is relevant to a given competency from that narrative description is an important part of of assessing and providing feedback for a learner So when we think about this this continuum of ultimately a a patient- centered physician who's an expert we want to think about medical education as as something that begins in medical school and then progresses throughout one's career So over on the left we see medical school really where learners enter and they're not allowed to practice They're noviceses but by the time they finish medical school they've become advanced and they can practice with full supervision And that's the handoff point into the GME space And during those residency and fellowship programs those learners those trainees are becoming competent and able to first practice with supervision on call and then ultimately without supervision And then there's this phase of learning where where a professional becomes an expert And so ideally this is a continuum and there's these handoffs that happen from one stage of learning to another and it and there's not sort of a disjunction and there's continuity and so that's what we're always working to to get towards and to do this I want to start with what you see on the left here these ACGME uh core competencies so grad graduate medical education about 20 years ago or a little over that establish these competency domains that you can see in blue around that that hexagon and they describe different areas of of development and in undergraduate medical education we haven't had that sort of a solidified framework uh to build from but using the GME core competencies last year or two years ago when BTCS Soom underwent our medical educ education strategic planning process We built our educational program objectives intentionally aligned with the ACGME core competencies to provide this continuity of learning There's been work since then not since then but there's been ongoing work in the undergraduate medical education space to come up with similar a similar framework for GME in the space You can see this report shown here The foundational competencies for undergraduate medical education It was work done at by organized by AMC with ACGME input and ACOM input to to come up with a similar framework because we need if this is going to be a continuum there needs to be similarity from to GME Fortunately our EPOs are very much in line with what what is recommended These aren't mandatory for for medical schools yet but more like guidelines to help shape curricula but they do align with the ACG MU core competencies So as we think about competencies in medical school and undergraduate medical education it is important that they be vertically al integrated so that what we're doing on day one of medical school is providing a platform to build on um by the time a learner graduates And so how do we do that here is a a really busy graphic showing all of our phases So in yellow you can see phase one and then moving into phase two clerkships in the burgundy and then finally phase three career exploration Ideally what we're what we're doing is building we're scaffolding in phase one the medical knowledge and and skills and other competencies that allow handoff into phase two and then ultimately into phase three That's what we mean when we say vertical integration And I think that it's it's we often think of phase one as really being that medical knowledge foundation base And to build on that in phase two and three rather than leaving that behind what we're doing with our class of 2028 as they progress into phase two we will have these landing spots and interessions shown here in those gold spots for some of that medical knowledge to be vertically integrated up into the clerkships There's a lot of other plans for those interessions as some of you I think on the call know who've been helping plan for those and this is something that working closely with Aubrey Knight as phase 2 director and Christy Stewart as phase three director Jed is helping us ensure that we've got this vertically integrated space within these inter sessions But the higher order competencies and we'll talk about harmonized milestones in a minute are also important to begin vertically integrating and not just waiting until the clerkship phase or phase three to be thinking about some of these more less knowledgebased competencies So this is a quick quick segue um or quick stop about thinking about competency based medical education versus time So when we think about the ideal version of what competency based medical education would be it it really would be thinking about when a learner has achieved competence in a specific area then they're ready to progress And so if they met all of their their milestones then we would move them straight into residency But but we all know right now medical education is either three in some cases but usually four years and so it's fairly timed driven So how do we reconcile the idea of a a very timebounded curriculum with the philosophy of of competence development And some of the ways we're doing that in undergraduate medical education is thinking about an individual learner and helping them identify where they need to focus to develop competence So one of the things we've baked into our new curriculum is frequent low stakes assessments rather than a single highstakes assessment that that doesn't allow time for those micro adjustments that we would want on the on the path towards competency development developing personalized learning plans So right now in semester two of phase one Dr Carvio and Dr Schmid Dalton are busy developing learning plans When when students have gaps in their medical knowledge or their clinical skill sets developing a plan that helps a specific learner identify what they need to be working on and and some additional work that can help support that We have a coaching program that supports our learners to identify their own gaps in knowledge This is to really provide that that sca scaffolding so that as a a student becomes comfortable with identifying their gaps and how to fill in those gaps when they move into phase 2 three or residency or even into professional practice They can be focused on developing their own competence rather than relying on external uh assessments to guide them So these are some of the ways we bring the the principles of CBME into even though we have a four-year hard hard time stop for for how long it takes And then one last way we've been thinking about doing this is by we've shortened phase one and really expanded phase three that post clerkship phase to allow for more personalized learning So that instead of entering um ending clerkships at the end of right before fourth year begins now students in the class of 2028 and beyond will be beginning that phase three curriculum in February And that allows for some personalization of learning throughout the the last several months of medical school which is again another hallmark of competency based medical education is this idea of tailoring one's path towards what is most needed So in order to assess competence we need milestones And I think it's often easiest to think about this in something a little more concrete like providing an oral presentation of a clinical encounter And so providing those descriptions that allow us to determine where a learner is along this developmental phase can look something like this where a novice is is someone who understands the core components of a case presentation and then an advanced beginner delivers a presentation It might not be concise It might wander a little bit A competent learner along this this competency domain or competency might be delivering a presentation that's organized around a chief concern And then proficient might be integrating pertinent positives and negatives And finally the expert is able to filter synthesize prioritize information into a concise and wellorganized presentation So you can see each of these descriptors allows us to provide feedback for a learner about where they are But that dotted line shows that this is not just a a straight arrow Once a learner hits novice they you know they they are at that time point You can see if we if we just did one assessment at at this point in this dip that this learner experiences here we might not be giving them adequate feedback about where they really are And this speaks to the need for those sort of low stakes frequent assessments because in order to to move towards competence a learner needs to be observed and then assessed and then coached And it's this cycle And so this frequent feedback um moving a learner towards proficiency is an important part of competency based medical education And a big difference between UME and GME is that this assessment phase is not overseen specifically by what we see in GME as a clinical competency committee Sometimes in the clerkships there's something that looks a little bit like that But this is really in the UME space up to each individual faculty member to be providing feedback so that a learner can can move through this continuum And that's why frequent feedback making sure we're all on the same page about what a milestone means it requires a lot of teamwork so that we can be giving what what a learner needs in order to to be working towards competence So where this can get a little more challenging is in some of the m in some of the competencies that are not either medical knowledgebased or or patient care skills-based So that brings us to this idea of harmonized milestones which is a phrase that is used most frequently in the GME space So if we go back to those ACGME core competencies you can see I mentioned patient care and medical knowledge are are some of those areas that are a little more readily or regularly assessed But these other four competency domains systemsbased practice professionalism interpersonal communication skills practice-based learning and improvement those are all a little less readily assessed if you will and yet they are common across all specialties So in ACGME these patient care and medical knowledge competencies you'll hear Dr had to talk specifically about what this looks like for surgery but these competencies these other four competencies make up what are called the harmonized milestones They're common across all residency programs And so we have those of course baked into our EPOS our educational program objectives and subcompetencies and bringing assessment of those into phase one has really been something that we've focused on for the past couple of years with all of our phase one educators So in those are these milestones that span all learning settings whether you're in research and inquiry or integrated foundational sciences These are these are skills and behaviors that you can assess and and exhibit no matter where you are and they're really critical for preparing a learner for clerkship and for that identity formation that I talked about at the beginning And so the way we've done this um preliminarily at least is thinking about things like interprofessional and team communication in in the pre-clerkship phase commitment to personal growth and reflective practice accountability and conscientious behavior which falls into the practice-based learning and improvement competency domain And the way we're doing this is breaking it down into those milestones that we can actually give feedback on So attendance and timely completion of assignments and attention to course details if we're looking at this accountability and conscientious behavior uh competency And so what does that look like well we have we have an assessment form that all of our course directors are responsible for filling out and and they're able to look at this milestone and say "Okay if I'm asking whether a student is demonstrating conscientious behavior maybe they're arriving late and they're not attending all the required events but they are demonstrating some conscientious behavior by reaching out and communicating that they weren't able to make it and they're being proactive about it and they're finding out what they need to do to make make up whatever they missed So we're capturing these sorts of assessments These are formative cumulative assessments They're released with semester grades They they allow us to illustrate some growth across the competencies So maybe in the first semester a learner is struggling with that that proactive communication but over time they're getting better about doing it It also allows us to capture comments from faculty on these milestones and not just the negative ones we're able to to let a learner know that we see that they're being really good about paying attention to course expectations and that that they're the ones who come in prepared and ready to jump into a learning experience This also helps us inform learning plans and coaching discussions and remediation because if a learner isn't doing well in the medical knowledge competency domain sometimes that's intimately linked with how they're doing in the conscientious behavior area So if they're not coming to class or they're not making up the the work that they need to make up it's good to have this documented to be able to have a conversation about how that is influencing their ability to achieve competence in the medical knowledge area I I can't talk about our harmonized milestones without at least showing one of these spider plots So um Brock would be hopefully very excited to see this up here But what we would what we hope to see over the course of several semesters and and throughout phase two and three is that this blue pattern fills out towards the perimeter as a learner approaches competency across all of these different areas We can visually see that they're marching through the milestones So along this accountability and conscientious behavior path we would see from semester 1 to two to three this sort of filling out So this is I'm going to hand it over to Dr Padet in in just a minute because what we want to be thinking about is developing competence across the continuum to GME and and ultimately to professional practice and I think aligning our our harmonized milestone assessments with assessments that are happening in the clinical learning environment in clerkships begin to build a foundation for for what is going on in GME Another point I wanted to to bring us back to is that when we think about Miller's pyramid that knows knows how shows how and does those higher order competencies are harder to assess but they're really essential And that brings us really back to our identity formation of the patient centered physician And then finally one of the competency that I think is a real strength at VTCS Soom that that we're seeing um a lot of interest in is this idea of systems thinking which is in our in the third pillar of medical education health system science It's really this linking domain and we're beginning to help our learners with that from day one Even though it's a challenging and and complicated set of skills it's really critical to our system citizen identity and and the patient centered position So in a couple I think next week we have a workshop that that looks a lot like what we do with our students and hope any of you who are interested uh can come to that because that helps make more concrete how we can assess some competencies that that can be a little more difficult So that I'm going to stop sharing and hand it over to Dr Padet Thank you Dr Gant Um and actually this makes I was I'm really happy with yours because it makes mine easier to present now and there's a couple slides I can probably skip Hey Um so so first of all two acknowledgements to American Board of Surgery who's been doing a lot of the heavy lifting in the last couple years and a lot of my slides are coming from and probably more importantly the society for improving medical professional learning which is awkward because they're trying to get it to simple um uh which has been doing the work behind this uh the theoretical work behind it and the actual practical work to bring us to something we can use Uh as a disclosure I am a part of the advisory board for the simple uh learning uh um which is a nonprofit which I do not get paid for but is basically a labor of love to try to figure out how we measure um first initially surgery residents It's actually brought branch out to a lot of other specialties Um so real quick Dr Kadet um you're sharing your notes Uh okay Let me try that again and stop sharing Any How's that perfect All right Thank you Thank you JG Um so here's a couple major themes that I'm going to try to uh uh discuss Our present methods are bad and have been bad and are not gotten better in a while They are in general retrospective uh which tends to have bias introduced they're reductionists and and in the end they often just go back to expert opinion um rather than having something more uh robust Um the next part is what are EPAs and how are we using them to evaluate surgery residents uh what are workbased assessments and that is essentially what we are using and we're using it both to define the uh the spectrum of general surgery and competence um and how we can use a group of EPAs to assess a resident's competence for independent practice So uh up till now and still uh we are a timelmited program We are at minimum five clinical years During that time there's some other notes you have to accomplish 850 cases 250 have to be the last year There is both a what they call a qualifying or written in exam and a oral certifying exam And most importantly this is goes back to it's essentially your endorsement by your program director that you're competent That is the actual most important thing within there Again this is anecdotal and back to expert opinion and probably doesn't um correlate across uh programs as well as we we would like The big change and why I'm talking about this and not one of the other program directors is surgery's been the first one that decided to go allin on entrustable uh professional activities such as the entering class last year um have all been measured by EPA and this is a huge project and still in its infancy and I would not say that it's even close to where we need to be Uh the current system that Dr Latan has spoken about is essentially time dates The middle line is what we expect everyone to do to just kind of go at the same rate and everybody will get there Um and that's just not true If anybody has uh worked with residents students not everyone follows the same path Uh what we are looking based one a fixed outcome so that we've defined a minimum for what a practicing general surgeon can do Milestones have been around for the last decade Um they are helpful but hard to do They suffer from the same problems How we assign these are often uh retrospective uh based on what our mo what our memory was and not necessarily perspectively I I've I've showed one here that I think is probably the core of being a surgeon is can you operate and how do we measure technical skills There's a second uh patient care one that um talks about steps of an operation But this is probably the the uh core of of the process that uh a uh very novice surgeon goes through to being uh to be able to uh functionally practice independently I'm not going to go through all the different ones Notice that we do have very good descriptors So uh what is the competence of a surgeon is it the technical ability to perform an operation is the decision making the pre probably is all of these um and also how to fail safely in integration of knowledge and experience and this is from tenant um also by the way this is in 2016 I think Dr Lant had one from Tenante in 2012 And this also goes along with about the time the surgeons were writing were doubting their own graduates um competence I think the my favorite one was the title was a uh invited lecture called why can't Johnny operate uh in 2009 and there have been a number of ones since they're questioning whether we actually are producing surgeons in this day and age that actually can do this So this is the problem that we're facing We don't even believe that we are actually uh accomplishing what we want Uh the competency model it tends to be a top-down one and it doesn't always give us what we want and it if it's done way poorly can decrease acceptable standards and really doesn't do much for us Here's some uh theory versus reality Um you don't need to read all of that This is both quotes both for people But probably the most important thing is there are 88 what we the general surgery defines as essential common operation and 70 essential uncommon operations about 150 that we're supposed to be training our graduates to do on um during their five years Well let's look at something Uh these are the 20 most common Don't read all of those but this is the take-home message Only nine operations do they do more than 20 times and only 20 operations do they do more than 10 times And that could be spread from their very beginning time to the later part So it's not like they're doing anything other than a few operations a lot they're doing a lot of operations and then sometimes they're just doing it once or twice or zero times during their um um expect during their uh five-year experience and we expect them to be able to do this independently without supervision on their own So that's the that's the hard climb Um going back looking at the technical skills again I want you to this is I've just put it up again to uh highlight what the words are but this is what on average we look like in terms of progression through the various uh milestones within this patient care uh one and I will refer back to this again because this is going to look very much like another uh curve So it takes usually by the end of the fourth to fifth year to get into that uh truly that they are uh firing on all cylinders and from a basic technique a milestone Okay how do I measure those right now we get in a group we say who has worked with this resident in the last six months Okay where do you think they are now look at all the recall bias Look at all those things This is how we are doing it now This is how everyone is doing now because they don't have a better way Switching to simple uh a workbased assessment that came up with how do you evaluate a resident after um their operation So I will show go this is the tool right now that we are using I pick out the procedure I describe whether it's an easy intermediate or difficult And now the two important ones that I do for a resident is I say are they inex very few of them are unprepared Very few of them are exception I would say I hardly ever use that But they all fall into most of them fall into inexperienced intermediate or practice rate So this is one for a hernia repair skip and I can put down how they did I'm going to say this was straightforward straightforward um limited participation skip and finish That was an observation based on it That's how long it takes to document it We also in addition to looking at their performance we look at autonomy This is very important to surgery It's one thing for me to show them how to do it and they do it very well while uh they are being actively helped That means go here dissect that grab that fall harder No that's not a good stitch That's very active Passive is when I'm there but I'm kind of just letting them do this and every once in a while maybe correcting them And that moves on to when it's supervision only which is what we're really trying to produce Okay So this is the tool we've had for about a decade and this is what if you look at a typical uh performance over time on a given resident you can see over months they gradually get to that practice ready one on most of their u activities but also we have to measure autonomy at the same time and this you can see show and tell means they're watching me do it which is not ideal for teaching but it is the lowest level That's what we're trying to get to is autonomy so that they are requiring super supervision only So um were there anything before simple yes they were and they were laborious Um the O scale degful OSAT um look something like that Um imagine trying to get some an observer to fill that out on all of the operations Um well one of the magic of simple is they found that that a single value how did they perform worked as well as eight values and it was almost identical to the uh gold standards behind us Um and how many observations do I need now let's think about this over five years they progress a lot What they look like at the beginning of their third year is nothing look like they look like at the end of their fourth year So how do I how is my ability to assess them to see if they're making progress and to assure myself that they are actually progressing enough that I can sign their certificate at the end and put them out into the general public So for our most common procedure for laparoscopic with policyctomy 23 ratings are not needed to get a auton reproducible autonomy rating Um if I look upon all operative cases it's actually less and they've concentrated on how often do we need to do this So this is a very doable sort of amount of ratings Um if we want to look for a group of undifferiated mix of procedures 60 ratings and and this is for the uh reproducible for the performance rating So it becomes a doable number Uh if you look back if you remember we're talking somewhere about 850 to a,000 cases they're probably doing 850 is a minimum If you think about that over five years I should be able to get 40 ratings And this is um how they progressed This is uh based on a Jeff probably Brian George who's done majority of this work has shown that the you start to the only time you start to get not nearly as much separation on these is when you get to the last couple of years And that becomes subtle because many times the fourth years are uh equivalent to the fifth years Not all of them are um but that's what that looks like So for the most part we get pretty good differentiation as they're they're doing So back to EPAs there are units of work a physician does uh that I do every day and we can evaluate whether how they engage with the patients how they address the medical knowledge basis Uh competency or much more broad milestones are capable of distressing progress But what we really need is a suite of EPAs that define a specialty and my specialty While you could probably put ENT surgeon here and it would be fairly similar this would be very different from an emergency room physician all all specialties can be described by a suite of EPAs Uh this is a looks a lot like something that we saw earlier so I'll skip it Um so for the EPAs for general surgery uh this we have they have created what they call what they think is core skills of a general surgeon This person could be employed This person could be asked to do these on any day Uh they try to use common conditions They also describe the essential non-technical skills such as professionalism and communication Um and they talked about management of the entirety of the disease process So they have a preop an operative and and a post-operative case on all of these issues And these are the 18 of them Um I talked to the people that developed them I don't think that this is going to come back as the final number Some of them may be dropped some of them may be added but this is our first approximation they have to always think about this as we move through different ideas and different measurements Uh these are some of the uh before they um this is sort of the in the test centers where they showed um overall change and improvement using these in all over five years So in theory this should be able to give us some idea This is what my EPA for the most common operation that we do it looks like for uh colisstectomy Um notice don't bother reading all of those By the way this looks overwhelming Actually after you've read this and done this five times it's easy because I know what um each of those look like in a learner Um so I don't have to refer back to them each time But in the beginning they take a bit of time The reason I could do that one so fast is because honestly these are very they residents fall into different groups and different times where they're doing Um notice they have like MKL2 and L2 Those are all milestones these things track to with the hope that eventually if we have enough data on the um EPAs we will be able to prepopulate and I think right now 17 of the 18 uh milestones we have will be prepopulated from this and would h actually be much more reproducible across this and so they are been they are looking to how these track you'd find that some of the ones of um the later ones and sort of the uh post-operative phase would track differently the pre-operative phase would differently but the idea is they do go back to what we originally were measuring progress the milestones So my take-home messages from this is how do we move beyond the stal sort of way of uh evaluating our uh trainees and u moving ahead Um I imagine somebody will still need to sign their certificate but it takes a lot off the program director that it's not just me that's uh believes that this person is competent that uh we have data to show that they are um the demands of a practicing physician are wide ranging have both frequent and infrequent tests This is a way we can measure both of them Um finally I do believe competency can be measured Um and I think the simpler the data collection the bet more uh buyin from faculty Uh if you can uh get people to buy into it takes 20 seconds to do an evaluation By the way the valuations are also uh within this app-based phone-based app also can be done a dictation can be done at the end so that the you can get specific feedback that's outside the ramifications of the the evaluation you've done and that's also a very buyin from learners getting frequent feedback The key to this is essentially what Dr Lante mentions as low stakes evaluations and lots of them You can't get by with just doing this once or twice uh per year for all of your residents You need multiple ones in the everyday existence in real time to get that Uh by the way the way the original Simple app if you didn't fill it out within 72 hours they made the point that you were not evaluating a resident at a given time you were evaluating what your overall opinion of the resident was much less accurate than actually doing it in real time And I think the key thing that we're still going to need to work on is what do we need to measure in these uh the other things will come along as they do So that's all I have for today I'd be happy to take any questions Thanks so much Dr Padet Any questions what type of questions do you all have for our wonderful experts here got lots of experience and work done in these spaces both in the GME world Maybe maybe I can uh ask a question or at least make an observation Chuck I I will speak mostly in the GME world and uh talking from a medical subsp specialty and evaluating medical residents and medical fellows I I really think your your push to make this simple and to and able to be done in real time is extremely important because one of the last thing you things you said I think is exactly true Unless you're doing this pretty much in real time on the GME side it becomes very difficult Um and to do it in real time is a challenge The administrators don't always push out the MedHub evaluations in real time They'll come a week or two or even three later The questions that are listed in the various categories systems based practice based knowledge and so forth Ask for details and levels You end up checking I don't know 50 boxes trying to get the level correct it it becomes very frustrating for busy physicians to do that So I I know exactly how we got to that point So I guess my question is both for Leslie on the side and Chuck on the GME side how do we make this simpler and undo some of the complexity that we've taken years and years to build i know that's a big question but I do think it's an important one to get accurate evaluations I mean I think from a from a standpoint I know what Dr Padet was mentioning with sort of real time very simple workplace-based assessments that is what we're I know I saw Dr Atkins on here I don't think he's still on here but developing those for our clerkships directors and our clerkship evaluations as well where it's not sort of a gestalt but it is immediate appbased quick um and frequent is something that I think everybody recognizes is important So the boring part of my talk is always all that data about how many observations does it take and this and that I always put that in there because I think that that was really the work behind it The valid the validity of lots of little and just asking them three questions about how people did gives you such a broad understanding of things So I think it it first of all we needed to move away from the way we do it now end of evalu end of rotation evaluations um are they're just um littered with bias and and incorrect evaluation So you have to just kind of um and why I became part of this cooperative is they seem to have an idea about it that was different I think somebody asked uh it's actually been applied to a number of different specialties Um TEDS has pediatric has looked at it Uh there's a simple bedside that they've come up with So it can be and um and essentially you just have to write the things that the and your um questions and put in the the what what constitutes this So it could be done at a lot of different levels but it it moves away from um the ideal time to do it for us is right after we scrub out and uh walking away or in the next 15 minutes And and that's the take-home message And they've they've gotten lots of data to show that if you don't do it that way it becomes a poor evaluation But I think it's we have to we we thought more questions were better My favorite one was they found eight questions on the OSATs could be replaced by one got the same information Um but rather than thinking about why how do I get more detail for a given student learner you just have to ask the question over and over again with lots of different um um observers So one of the interesting thing I found somewhere along the way someone decided to try to evaluate how good a surgeon was operatively and they found actually you could take um videos of them from the operating room show them to the public and the public can accurately discern them but what you need is lots of observations not just one and I think we tend to base our opinions on students residents fellows on very limited interactions maybe a few and what you're trying to capture is multiple small ones And again they become low stakes because it's not you know you may not do well today but in general you're going to perform at some level and that's probably going to be pretty accurate So I think multiple observations is the strength of this Well I I applaud your effort to simplify and the tough part is validating So the the there to just give an analogy there there's an instrument called the SF36 a quality of life instrument It's been validated in multiple languages over years and years and yet it's very difficult to administer outside of a research setting I had a colleague who said 'Y yeah I have an SF1 It's called how you doing but of course that's not validated So you're the struggle is going to be where educational theory and research meets practicality but I but I really think it's critical So thank you I mean I think another piece of that is is the the coaching aspect where we begin to invest some of the responsibility for learners development within the learner themselves so that they are aggregating I you know data for themselves and then talking through that information with a coach who's got the designated time to to do that So those frequent assessments bringing those together helping the learner begin to develop those self-reflective practice habits that allow them to to grow through the continuum I see Dr Schlupner talking about obtaining real-time assessments very difficult Absolutely And indeed And so short frequent and then then aggregating that information for sort of longer term processing is important We did have a comment in the chat but Dr Lante you just addressed it from Dr Schlubner And then um there was a question about the simple app but Dr Padet but I think you've already also covered that um regarding you know could it be applied to non-surgical specialties and you mentioned pediatrics um potentially has uh has has used it before So um anything else about the simple app and other specialties that might be using it i mean from inside the simple organization suddenly they went from somebody with three employees to 40 and they're trying to keep up and and these are people that are doing this out of love of education Um but it's it's an interesting um to watch in the last couple of years until we've got this because the the volume has so uh in incredibly increased However they are um um creating a simple bedside one which would be a lot of the same skills that we look at um as as medical students and interns in all phases of um work Uh so um that's one of the things they're working on Um the other thing uh I I realize that I didn't mention is um we so I talked a lot about why we don't want timebased learning and people say okay great the idea of time based learning some people need a little bit more some people need the average what about the people that uh excel what do you do with them and and that's the other question um and one of the things within surgery is if you are um signed off as practice ready on say apppendecttomy and you're a fourth or fifth year resident uh they talk about practicing in place that thing that you would be credentialed to do in your institution once you reach that whereas the other you may not have achieved that in all 18 of the categories but it actually makes some you know how do you actually grant autonomy and say that this person is actually ready for practice That is one of the things that's been brought up for people once they get to that level Um I won't bother telling you of all the uh things that will be in the way of that including um how we credential physicians at a hospital level and being paid for by insurance But that is the idea is that there is you're trying to quit making one sizefit all and adapt it to the learner as they progress to actually being a practicing physician.