Competency-Based Medical Education & Assessment

Good afternoon everybody what a great group group so good to see you all thank  you for joining us today so today we have the pleasure of  having Dr Jed Gonzalo with us um and most of you know him I would hope by now  uh Dr did I say Mr Jed Gonzalo Dr Jed Gonzalo um senior associate Dean for  medical education um and professor of internal medicine and health systems and  implementation science for Virginia Tech kilan School of Medicine um Dr Gonzalo is here and he oversees  all of our um Medical School curriculum so I think probably most of you have  heard the term um Competency Based medical education um I don't know uh if most of  us know exactly what that means Dr Gonzalo is really going to help us to think through that um also because he  does oversee um the medical school curriculum he's also going help us to understand  more clearly what that means for us here um so uh Dr Gonzalo strategically has  placed um lots of points for questions and discussion throughout um his presentation today and he strongly  encourages um interaction if he's asking you questions please feel free to unmute yourself put your video on if you'd like  um and go ahead and engage in some conversation it's going to make us make it more um exciting for us and we'll all  learn a little bit more um uh I don't think I have anything more  to say go ahead and take it away Dr Galo thank you Sherry hi everybody it's so  good to see every several of you on the uh gallery view here and let me share my  screen let see the slides okay everybody all right so thanks to Shar  and team for asking me to lead this session on Competency Based medical education an assessment so as Sher  already outlined I've got several stop points here throughout the talk so I am less wed it to getting getting through  all the slides I think having dialogue and discussion and discussion box would be fantastic there is an attachment um  which Sarah's going to post in the box when she gets a moment to do that so please feel free to download  that so so I'll start with some disclosures and acknowledgements I have no conflicts of interest to  disclose and it gave me a chance here as I Was preparing these slides to acknowledge like how did I actually  learn about Competency Based medical education so a lot of the learning came from mentors and colleagues of mine over  the years from my former roles Eric combo who's at the acgme and several  other places has has taught me a whole bunch several of the slides you'll see here have been modified from Eric's  multiple talks that he was willing to share with me and of course the BTC uh  Medical Education team we've got a a wonderful team here who had been working on competencies and cbme well before I  came um but certainly the past year and a half I would say a Common Thread in what we're doing is Competency Based  medical education so I just want to make some  connections with with where this sits with the teach offerings and I think there are two direct connections with  what's coming up so uh Mariah Rudd and I think is leading the journal Club on  Wednesday January 24th um in its direct relationship to this session but evaluating a competency based Blended  Health Professions program a programmatic approach which will be a nice node connecting from this session  right to this one and our vice de Dr Rebecca P will be leading one of the Health Professions educator series  related to teaching professionalism so there's a relationship between professionalism and identity formation  with what we do in cbme so I think it's an exciting month of sessions coming up over the next four to six weeks um from  what teach has planned  out so here's the objectives uh for us in the next several  minutes we'll Define curriculum and Competency Based medical education and articulate the differences between both  we'll walk through the five-stage model of skill acquisition as described by drus and dyus which really comes outside  of medical education then we'll highlight the essential components of cbme and clinical learning environment using the  van Melly framework this framework is one of my favorite Frameworks that exists I think it's so applicable to all  of our work and seeing all the individuals on the phone on this Zoom  call um I think it's applicable to all of our work no matter which type of role you play in medical education we'll  discuss a little bit of the pyramids of the educational objectives but M Miller's frame work for um behaviors in  the clinical learning environment as it relates to Identity and then threadit throughout will be some examples from  vtcsom our very work going on in the medical school currently but also in future here's my disclaimer to make me  feel better because cvme is massively big um and I'm just trying to I feel  like I'm I want to deliver for everybody but we cannot do it all this is modules that takes hours courses what I'm going  to do in the next several minutes is hopefully facilitate some discussion and share some key elements of cbme but  there's so many areas that Merit a whole bunch of work for example coaching or programmatic assessment along the way or  dashboards or position education we're not going to be able to do deep Dives so we're really after this systems thinking  the bread uh and scope rather than the depth of every particular  area so I'm going to start with the patient case and just as a as a primer this this is a um your first reflection  so you're going to be asked to reflect on this case Larry's a 79-year-old currently working CPA who experiences a  large bowel infarction undergos two operations and stays in the Intensive Care Unit with complications of delirium  and fever he's transferred to the internal medicine unit and care for by hospitals nurses therapists and several  Consulting Services unfortunately several complications ensue including a delayed  diagnosis of Hospital acquired pneumonia the stage three sacral de cubitus ulcer  perceived subpar coordination between Cardiology infectious disease Internal Medicine including several handoff  errors both between the services but also between day and night teams despite these events Larry's  discharged to a rehab facility where he receives unnecessary salt tablets that result in anasura this salt um salt tab  order transferred over to the rehab he under goes a trip to the emergency department for a diagnostic error he  exper is repeat pneumonia due to aspiration due to a feeding error that occurred at the bedside and and patient  positioning and he misses several physical therapy sessions Larry has discharged the home with hospice care he  describes his experience at the rehab as being treated like a thing he dies eight days following  discharge so what may come across as a little pointed this is a an example that  um I think resonates um very few times in in healthcare can we see the whole continum we're seeing small Snippets but  this could be a fairly common um example of reality so here's your reflection have you ever experienced on  a personal level or as a health care professional poor unsafe care but most importantly what is the role of the  medical education system the work of what we do in improving care for patients Like  Larry so ple please feel free to unmute or type it in the  Box hi my name is Karen bouf and I'm in the nursing department um at the College  of Bradford and I have experienced working in both um long-term care and  the emergency room um experience that care transition um but I do feel that we  both in hospital and in the emergency room first of all we don't listen to our patients to see what's going on and the  second thing is we don't look at the long-term ramification of what might happen if they don't have certain  resources or the ability to care for themselves at home um including in the  rehab uh you know I don't know that rehabs sometimes do what they're  supposed to be doing and we need to follow through with that if that makes sense yeah Karen I fully appreciate that  and yes it does make sense this idea of listening to patients one understanding their context and they're bringing to  the table and then looking at the long-term ramifications what we do systems thinking comes to mind how do we  see this holistic picture of what the patients need so I think Karen that that's spot on and I really appreciate  you sh being the first to share other thoughts or  comments D I think um all too often that it's the patients who are the only ones  who see that full perspective right because they're seeing the patients and or family they're seeing their family  member bounced around um from specialty to Specialty in the bigger picture um in  the the various error errors throughout um while each specialty doesn't have  isn't privy to that Fuller picture that you described Sher it's a great point I  mean as Healthcare becomes p perceiv more complex and specialized there's  different units different teams and it's hard to see and be part of all those different teams but the patient is the  common thread the patient is the individual with their social network with their family experiencing all of  these different areas so it makes it although good for have the complexity it makes it hard to see the holistic  picture as the patient is experiencing any other comments I see David you're you're coming in here not  being a clinician yes but issues pertaining to handoffs um physicians may be resistant to standardized training on  handoffs and that might be true David do you want to add to  that um not sure I can add much I've I've experienced it as regards to  Patient family members just observing at the time of of handoffs when family members have been in the hospital and  noticed that um information is either missing or sometimes it seems like the  patient or the family is in the position of having to constantly repeat the same information to different people as they  kind of rotate through um and then I was involved a few years ago in an effort to try to  standardize patient handoffs with incoming residents um and I forget the model it's  a pretty prominent handoff model that's used a lot in Pediatrics for example but  didn't get too far with that it just seemed like it ran out of esteem in terms of being an urgent  issue thank you David any other  comments so I hope what we see in the in the comments everybody's sharing  um that it resonates with all of us about what patients need and I think that's the end goal of what we're  talking about with Competency Based medical education so I do hope that's one of the threads that you see in the  next several minutes that we're together talking about this concept I'm going to start with the competency let's define  some terms here competency an observable ability of a health professional integrating multip multiple components  such as knowledge skills values and attitudes just have a little diagram here on how I kind of keep it in check  for myself KAS knowledge attitude skill some place a B on there knowledge  attitude skill Behavior that's what a competency is but a key point being what I underlined an observable ability that  would be expected for practice of that individual I did want to start with our  nordstar medical education and how professional identity and our mission and vision as a medical school or us  medical education RIT large ties the competencies competencies one thing we don't want to  view it as is a list of observable actions or behaviors that are divorced from our end goal of what we hope for in  our graduates in a medical school or graduates of residency programs Etc so I wanted to start with our identities  these are our VTC professional identities um that we have had since the beginning of the medical school that we  have recently codified in our strategic planning process certainly at the center is the patient center physician The Good  diagnostician the person with skills for therapeutic skills to meet the needs of patients but there's these three  synergistic professional identities the lifelong Master adaptive learner someone who who can identify their gaps in  knowledge or skill or expertise use reflective practice and iterate to hone  their skills they're on this lifelong journey the scientist physician the or  perhaps another phrases inquiry driven scholar someone who has the ability to ask good questions use that data iterate  and employ that into into best practice and a system citizen um we're  all citizens of a Healthcare country nurses Physicians therapists and it's  all of our responsibilities to be thinking about patient safety and quality improvement all the social  determinant of Health that Karen alluded to in our opening comment these are our identities as at btcs so and I wanted to  ju suppose that to Miller's pyramid we had mentioned Miller's pyramid in the beginning around what an individual or  Health Care trainee knows knows how shows and does and Miller's Pyramid has  been around for several decades and really looked at the bottom four T tiers of this pyramid Cruis and Cruis modified  this pyramid to place identity at the top it's the identity of the  professional and it's all related because the knows and shows how and does is related in the same block the same  pyramid and I just want to connect our own identities our identities as a medical school is at that Apex of that  pyramid but everything in that same pyramid that is below the identity are  competencies it's the knowledge attitude skills and behavior and there's a direct relationship between those skills with  the identity it's our North Star so I hope what you take away from this slide is that there's a relationship between  both the competencies don't live out in some document collecting dust it's actually the means by which we achieve  our goals in the mission and vision of a medical school so here's your next prompt so I'm  going to walk through three definitions but I'm going to ask all of you to reflect and share in the comment box hopefully you'll unmute and share what  you see as the common threads so I pulled the three of the most cited least  from my perspective three are the most cited definitions of Competency Based medical education so I'll walk through  all three but I'm going to ask you what are the common threads that you see in these definitions and I have them working backwards I have all the  citations at the bottom one is from 2010 the Frank Ed all article Ron Harden who  was just here um for medical education day and mahe at all from the World  Health Organization so here's definition number one an approach to Preparing Physicians  for practice that is fundamentally oriented to graduate outcome abilities and organized around competencies  derived from an analysis of societal and patient needs it deemphasizes time and  Promises accountability flexibility and learner centeredness the second definition is by  Ron Harden an approach to education in which decisions about the curriculum are driven by outcomes  that the learner should display at the end of their experience in outcome based education product defines the process  the educational outcomes are clearly specified and decisions about the content and how it is organized the  strategies the methods the assessment procedures are made in the context of those learning outcomes and then lastly  the the oldest one from the wh the intended output of Competency Based  programs and health professionals is at a defined level of proficiency in with local conditions to  meet local needs so three prominent definitions of Competency Based medical education I'm asking everybody what are  your thoughts what are the key features or elements that span or Crosscut all of  these  definitions thanks Hillary keep them coming  everybody  great comments in the box so far outcomes there's a goal at the  end local Learners thanks Anita local Learners individualize learning Kim thanks for that outcomes  are tied to societal and local needs it's a key point that I think is pretty prominent that we might not expect if we  were to ask what what's the definition that comes out of cbme and David's comment about  deemphasizing fixed time to achieve outcomes thanks Rebecca flexibility and  adaptability these are great any comments please unmute time's yours please share what your thoughts  are have one thought um instead of typing it in I'll just put I like um when it says  that deemphasizes time-based training and Promises accountability flexibility  and learner centeredness and I don't think that we always do that thank you Karen I think this has  been a a poignant point of discussion in medical education is time variable  education if we were to shift fully to cbme and our model set up to not be time  flexible um so there is this time component that comes into cbme other  comments I am curious what they mean by local like do they mean geographically  local or local to the specialty or how they Define do you have any insight into  that yeah I do Does anybody else have Reflections thank you Sherry um I'm happy to share my thoughts anybody else  have any thoughts or Reflections on the idea of  local I took that to mean the person who's learning local to that environment  not necessarily geographical but I get that that it could be  both I also took it as local conditions B maybe um another word for it would be  individualized conditions or meet individualized needs um that's how I look at it but I could be wrong yeah see  I didn't even need to give my perspective I think both of those comments um align with what the end goal  is for cbme um if if you're building an education program and you're in there's  a a program called Compadre it comes out of Oregon and California on the west coast but it's really meant to meet the  needs of rural patients patients who are living in rural settings your learning goals in that program might be a little  bit different than if you're at Mount Si or or Harvard medical school in Boston  and it's not to say that the competencies are totally disparate but the emphasis or prioritization might  differ a little bit based upon local needs which we could in theory think about our local needs in Southwest  Virginia um for Virginia Tech corillian and klian clinic so local I think could  be local at the level of individual with faculty member or local in the context of our neighborhood or our  environment Thanks Elena yes each Community might have specific needs um atena does work internationally so  what are the needs when the healthare delivery system looks different in Brazil compared to the US I think it's  another context or lens to look at local all right so I'm going to unpack  these definitions a little bit and I hope these are some of the key takeaways that you have when you think about cbme cbme and the competencies are our  outcomes in medad they are what our end goal is what we're trying to do with every every  ounce of of what we're doing in curriculum and clerkships in in the um to gme transition there are outcomes  when somebody says oh we show us how medical education improves patient Health outcomes I always pause a little  bit when I hear that because ideally we would the academic  presence would be improving patient Health outcomes or population Health outcomes or the quadruple aim and I  think it's all related but the intermediary step is the learner skills and ability ities in the cbme Paradigm  that are aligned with improving those patient outcomes cbme starts with an analysis of  both society and patient needs um as Kim highlighted in the Box um I I think of the world uh work of Charles Bolan and  The Who and the social accountability missions of medical schools medical education is not designed to be a static  process it's designed to be dynamic to meet the needs of communities and people  that evolves over time so if you think about prior to 2000 well all the patient safety events and diagnostic errors as  evidence sprang up we probably focused on patient safety quality improvement a lot of the health systems principles  less prior to 1995 and 2000 but as we learn more and the society needs are  identified more than cbme aligns with where those needs are for patients cbme is an approach it is not a  list of competencies that live on a piece of paper it really defines a lot of work processes holistic  informing all aspects of medical education from curricular design learning objectives in a session  assessments programmatic evaluation and it requires a high quality learning environment if we're  talking about knowledge attitudes and skills that are the those outcome abilities of Learners or residents or in  our context physicians in the learning environment then the the clinical learning environment is where it's  practice learned and taught so the quality of that learning environment matters as much as any other aspect  because that's the social cultural interactions the organizational structures that allow for the practice  learning and teaching of those skills and cbme is tailoring of Education to Learners needs just like  this discussion around local it requires this faculty learner interaction for  growth it's very iterative um along a Continuum so I hope this slide this last  couple minutes these are the Salient points you take away from C BME it's on a list it's holistic it's synthetic and  it informs an approach to what we do in medical education RIT  large it's not to say that everybody's in full agreement with cbme so I just want to acknowledge some of the debate  perhaps that exists out there um the idea of te steeping um Brian hodgers wrote a great article um 10 15 years ago  about te steeping um and the OC model for medical education but there are some  who feel that cbme may be too scripted they're saying you know medical  education could be like AAR you place a Learner in a learning environment and if they're in a good learning environment  that with time they will develop those skills and competencies on their own you don't need to think proactively about it  but it's like a tea bag you're placing a tea bag in the hot water and with time it Steeps so you place a learner into  that environment if it's a good one with time you'll have tea at the end and you can less prescriptive and I think  there's been a debate and there's probably some legitimacy to that not going too far not being too analytic in  what we do in cbme but I just want to acknowledge um on how some View cbme and  going too far with the analytic process all right so I'm going to shift  here into curriculum here's your next stop point to I just want to make the connection between we talked about  identities into the into the competencies and now we're working our way back competencies in the curriculum  so please share your thoughts here what are the core comp when you word curriculum what are the core components that you think  of it's a safe environment everybody whatever whatever you got just shout it  out thanks Elena  great comments coming through the box here and there's diversity in these comments too which which is great there's multiple components of a  curriculum but Hillary is referring to an approach Sarah might be referring to  the experience itself meeting the student needs the methods the assessments the competence  Leslie's assessments that reflect our desired outcomes that we talked about meeting societal  needs Kim's talking about developmental skills that it develop over  time yeah thank you Kristen so identifying those most critical information and skills this home process  and prioritization of what we really want to be learning and I I want to hold on to Kristen's put a pin in Kristen's  comment because it's going to thread through in the next couple slides thank you Ki yep formal mechanism  for teaching so let's generally curriculum is a planned  educational experience so it must have four components this is generally speaking you must have the goals and  look at all these comments in the Box several of them are are highlighting to expectations or benchmarks for teaching  you need to have have some kind of methods instructional methods whether whether it be a rotation in a operating  room or a an integrated foundational science course for the medical school  the ex the methods need to be there to provide the opportunity for the development of those knowledge or skills  need materials and then you need that assessment and there's a direct linkage between one and four the goals and the  assessment that you have in place so the curriculum here is used broadly because it's not a we might typically think of a  course it's a curriculum could be a clerkship it could be larger experiences so I'm just showing three of  the curriculum design Frameworks that I have used in the past um we don't need to get into the details of these but  Theory based design col's learning theory experience to Observation to conceptualization the experimentation  and you repeat that cycle you can use that to design a curriculum understanding by Design Hillary referred  to this in the Box this backwards design identify those end results and you work your way back into the learning  experience experiences and then current curriculum development framework which comes out of medical education  specifically probably the most cited that I have seen it's an iterative cycle but in there cross threading all of  these is this Common Thread that there is learning it has occurred or will occur and then there an is an assessment  of that learning so we're connecting curriculum and these curriculum experiences to  those competencies and this is one of the most referred to um figures if you will in medical education that compares  and contrasts the traditional model at the top structure and process approach to the cbme Competency Based medical  education approach they're both at ends of a spectrum in reality they kind of play  out in this gray Zone but I think for demonstration purposes here on the next slide I'm just showing the polls of this  spectrum just to make some salum points but in the traditional model is you have a structure and process and you create  the curriculum based upon the learning objectives or the content that you think the learner should know you build your  objectives you create the assessment and the learner experiences that curriculum  through a unit of time cbme is a little bit different it's it's in many ways it's reversed you start with the health  needs or the health system needs patient needs for example you build those competencies and the assessments that go  with them and then you build the curriculum based upon those outcomes so think about the vignette with Larry all  those opening com that all of you shared you're essentially highlighting the competencies that are needed for that  vignette to have been a more positive one you start with the health system needs you identify what are those  competencies that are needed and then you build the curriculum it's almost in Reverse to the structure and process  approach and that's what I'm going to walk through here in this table this is the heaviest slide I've got um but I  think if we grapple a little bit with the structure process and the cbme approach I think you can learn the poles  now granted it plays out a little bit in the middle but what's the driving force for the curriculum in these two approaches  content knowledge acquisition is your focus and a structured based approach but outcome knowledge application is  cbme the driving force for the process if a teacher or or educator is coming up  with the learning objectives the driving force comes from them in a cbme approach it's the learner  and teacher in a more co-produced manner because the learner is iteratively  acting on or assessing their Gap apps the path of learning is hierarchical versus non-h hierarchical  it's not to say Educators and teachers do not play a role it's critical but the Learners have some onus of  responsibility and accountability in that process the goal of the educational  encounter sometimes is knowledge acquisition in the structure and process approach but cbme is all about  application again a competency is an observable behavior in that clinical workplace and it's about application  and then the assessment tools usually they're proxy or removed while in competency base it's authentic it's in  the trenches is with the the core faculty in the internal medicine clerkship or Pediatrics in the  clinic the evaluations are different too in these two approaches so Norm reference versus Criterion reference  Norm reference in a structur process approach is hey we've got 49 medical students and seven students are going to  get an honors and 15% are going to get a high pass it's predetermined criteria that is  reference across your population of Learners versus Criterion reference if all the Learners meet those Criterion on  these milestones and competencies then we're okay with that we're okay if all 49 are able to meet those outcomes  because that's what we want in practice all Physicians who were able to meet the the needs of those of the patients  through those competencies and the timing of assessment the emphasis in a structured  approach is on summative while in a cbme approach is on a iterative formative and  low Stak summative process kind of like the analogy of food in a restaurant the summi of approach  might be you have this food making process makx it to the customer in in  that restaurant and they taste it that's summi of it comes at the end of that process but a formative process is more  where the chef and the team is tasting hey I need a little bit more salt hey I need a little bit little bit more pepper  a little more spice and you're iterating until you know that you've got a good taste and then it makes it to the  customer it's a very iterative proactive process and the program of completion is  always at fixed time in the structure process approach but cbme may have variable  time heavy table um but I'm hoping that we're we're coloring a picture between  polls of a traditional structure-based approach with cbme and cbme really is a  paradigm shift that is more balancing between this non-h hierarchical application spreading into assessments  and potentially opening up the space for a variable time  okay so I want to comment on sense making a little bit in cbme are there any comments to this point let me just  do a stop and a check-in anything I can clarify or any Reflections that you have Jad I've got a question this is  Joanne so how like looking at our new curriculum I  mean are we going to be flexible enough to really do a variable time program completion yeah great question Joanne I  I think back to my prior point a couple minutes ago I think variable time has been a key point in discussion in  medical education although I'm I'm just speaking personally this hasn't been you know  discussed and approved by everybody but I think the changes that we're making in the curriculum particularly when it comes to shortening of phase one it  opens up spaces for an accelerated program to occur we have the accelerated um pathway task force 20% of medical  schools have a shortened curriculum it's typically four 20% have a program where you can do it in three years I think we  could do variable time in that context but it is hard given all the structural  con constraints that exist to do to if you employed cbme to have variable time  I a learner graduating three and a half years they met all their milestones and competencies what do they do with the  last 6 months or eight months of their training those kind of logistics are are really hard there's Pilots out there um  particularly in Pediatrics that have piloted um time variable but this is a bigger Beast  Joan Jed I have a question for you Paul skolik here um great  table the goal of the educational encounter row you have knowledge  acquisition position and knowledge application talk a little bit more about  that because to apply knowledge one has to First acquire it and it strikes me  that a lot of our discussions  um or at least some of them Senter around the best ways to acquire knowledge yeah Paul thanks so much for  that comment I'm going to comment on on the next slide too because I think Paul Paul's point is spot on like you can't  just jump that's why I said there's two poles and there's there's some space in between you can't go apply without  knowledge acquisition I think to this isn't my table by the way this exists out there in the literature  um to flip it you have to have that knowledge base  to apply the structure approach might if it just stops at knowledge acquisition  and it's assumed that it could then be applied is where it breaks down and I I think that's where CB comes into the  picture it says hey we can't just stop at knowledge acquisition we got to translate that into the clinical  space does that help Paul or no yes thank  you all right so I know some of the terms are challenging so I got um two slides here on sensemaking and I love  history so I am going to try to provide some context on History I'm just going to hit some the  Salient mile poost just to Nestle where we are for cbme and I I take it back to  the flexing report that was funded by the Carnegie Foundation back in 1913 this is to Paul's point that he  just made um this this codified medical education in many ways exists to this day the two plus two botle learn in a  classroom take it in the clinical space and apply it was this transition from practice Bas knowledge to  knowledge-based practice that's what the 2+2 model was seeking to do the early Forerunner of  cbme in the early 60s the Office of Education developed National Center for Education grants this is not a medical  education this is in higher education really focused on outcomes based education hey we need to transition away  from this structure process approach that's fixed in time and think about what are those outcomes and then in 1978  the Mahi work the who work said we need to promote CBM cbme in medical education  so you see we went 60 70 years it not without cbme but it wasn't a cbme  approach until the the 70s and 80s but it's still wasn't adopted in US medical  education there were another number of events between the 80s all the way up to 2001 and I'm just going to highlight  three of them the Libby Zion case 18-year-old woman admitted to a New York University Hospital dies within 12 hours  of AD Mission um pres presumably from a drug to drug interaction identified a whole bunch of systems failures and and  medical education gaps that has informed a lot of what we do in medical education today the  Harvard Medical Practice study um a study that's looking at Adverse Events and medical errors in hospitalized  patients which was about 4% of hospitalized patients experienced in adverse event or medical error and the  reports which articulated a lot of the the patient safety gaps the the crossing the quality Chasm for example the  diagnostic errors this laid the platform for Competency Based Medical educ in us  medad which allowed for the acgme to approve the core competencies at the gme  level and then at the Medical Specialties level and they were formally adopted in 2001 so we're about 23 years  into this cbme Paradigm and we have a long way to go I  mean we're we're still sprinting on it um even though it's two decades in Carnegie Foundation again um put out a  report in 2010 and said hey we need to keep moving with assessments and the clinical space 2013 the work by  Englander and Kario identified the gme CME competencies we had a framework  although the competency started in gme has really struggled with a common framework to the today we do not have a  common framework for competencies that exist across medical schools this article  helped the acgme NEX accreditation system said hey we we're in this process for 10 or 13 years we got to add some  granularity to the competencies so the Milestones were adopted Ed the epas the  entrustable professional activities were adopted in the not adopted they were recommended in the lens in several  medical schools Incorporated those in as did we as a medical school and the  harmonized Milestones of the GM level help provide granularity to the competencies so the past 20 years has  been a honing of the cbme journey that we're in so all the work we're doing  locally right now comes in the context of this historical  past when I have struggled with the language in in Competency Based medad I  think of this figure it's why I'm sharing it this is one that integrates um EPA docc is domain of competence C is  competencies and M's are milestones and they're all related they're all part of the cbme  Paradigm so the entrustable professional activities are units of professional practice that may be entrusted to a  learner to execute execute in an unsupervised manner once they have demonstrated competence so in many ways  the epas requires the building blocks that exist to the right of the EPA in this figure let me see if I can annotate  so here's your EPA but all of these are building blocks that build up to the EPA  itself so this is all a workplace descriptor so epas if you think about  epas it's everything that's happening in that workplace the competencies are  mainly person descriptors they could happen in the workplace they could happen in a nski lab they they could  happen in other venues but it's mainly about the individual and the Milestones are the building blocks for the  competency so you become more granular As you move right in the figure the EPA is this overarching unit of work that we  can observe in the in the clinical space what can I clarify here what's confusing  here or share any thoughts that you have in this spot  Jed I think one thing that comes to my mind when I look at this slide is that it would be easy to look at this and  feel like there is universal agreement on each of these categories but in fact  there isn't right so it what what one group says is a competency in say for  example systems based practice some other group or some other specialty may take that same thing and say no this is  is a competency in a different category and the same thing with EPA so it's I'm  glad you said we're only 23 years and we got a lot of work to do because a lot of the work we need to do is figuring out how to talk about this so that we're all  speaking the same language and having the same things of mind when we use these terms because it's we're far from  that in my judgment am I right about that yeah David I think you're spot on I think you bring up a couple points but  one of them is the Continuum in the I'm I'm seeing Karen here so I'm not excluding the nurses I'm just thinking  in the context of the glasses of a position the Yi to G to se me transition  that exists um we are all working on different models of competencies or cbme  in some variation there is commonalities but it's hard to think about the transition when you and me we're trying  to prepare for the transition to residency so it's hard to facilitate along that that Continuum I I do think  there's we're further along than we were five or 10 years ago on what lives in  system-based practice or practice based learning and Improvement I think we're we're we're making there's increasing  clarity about what goes into those domains of competence but thank you  David any other  comments All right so this is the vanelly framework  this is one of my my favorite Frameworks if if anybody's interested in about your local work I I  would just read this article I'm going to walk through the the five tenants but I'm going to apply it to a btcs so  example um for each of them but I'm also going to ask you to reflect are there any examples that you have using this  framework so one there needs to be an outcome based competency framework that exists these are the desired outcomes of  training identified again based on societal needs and the outcomes are Paramount for functioning in practice  it's that knowledge based application so a VTC example could be the work of the scientist physician or our research  domain that we have across across the years there's 159 MD schools there's 41  do schools that exist in the US what what is our brand what is our differentiator one of the key areas is  the scientist physician or inquiry-driven scholar we believe that in this outcome  based competency framework those skills those deep skills of being the scientist phys will be part of the workforce  leading clinics leading initiatives in that clinical space in Health Systems so  much that it's part of our educational framework that meets the needs of  Society number two to have a full cbme program you need to have Progressive sequencing of competencies there must be  building blocks that work along that Continuum from time point A to B to C  and this is vertical integration how do you integrate across from the learner's experience from phase one for example to phase two and  phase three the example I have for this is uh clinical science osis and and  standardized patient so we do amazing work let's just take the example of history history taking or history  construction we do great work in Phase One and the Learners are learning the core components and they're practicing  and they're demonstrating and we're assessing and then when they transition in the phase two they should be picking  up where they left off from phase one we shouldn't be repeating that cycle through the ms3 orientation through the  clerkships and into phase three that it should be Developmental and it should progressively sequence along the  learner's growth this easier said than done medical schools really struggle at vertical integration between those  phases and we're just in the throws right now of thinking um developmentally along the sequencing of the  competencies The Learning Experience should be tailored to competencies in in cbme so the experience itself should  provide the venue or the millu for the competencies to be developed the example  I have for this is the work by uh Dr alak Tong in surgery and Dr Natalie karp and and Obi G as also the HSS Champion  lead the work that they're developing now is an elective hey we have we have gaps in health system leadership patient  safety Qi so they're building an elective um that will be piloted for the first cycle  to help Learners develop those skills in real world examples of surgery um with  the idea that it's going to be developed further into the Capstone but the experience itself of the elective or  Capstone is a vehicle for the competencies that we want these competencies are clearly listed in our  educational program objectives rather than the alternative would be hey well we have this idea for  an elective let's let's build the build the elective and then think about the competencies afterwards we're trying to  back engineer with where the needs are with competencies first and then the learning experiences follow that the  teaching in those experiences then should be tailored to the competencies and this is where it comes down to Learners and and teachers or Educators  co-producing their learning it's the feedback it's the iterative growth imagine if you're a hospit on internal  medicine and you're providing feedback and and engaging in that education process with the learner the guide  should be the competencies it shouldn't only be about the knowledge piece of  sadh or elevated TSH yes it should be that and it's also the competencies of  growth through teaming and communication and and professionalism and systems based practice so the integrated  foundational science course is a great example this is one of the most active areas of our curriculum right now as we  try to build a horizontally integrated integrated foundational science course take for example anatomy and clinical  science how do they come together in the integrated in the in the mind of the learner to think about functional  anatomy and the clinical science skills on a knee exam the teaching is tailored to what  the competency is for the individual in practice in our context a student on a clerkship or a student entering as an  intern and number five is programmatic assessment this is more of a synthetic view um a lot of it comes with Dr  mutcherson shop but multiple data collection points it's an emphasis on workplace based assessment progression  based on ass uh entrustment many medical schools have  transitioned to coaching programs we're we're hoping to Pilot a coaching program coming up um in the new ms1 class the  building of dashboards that ties into the Precision education Precision education is using the data to inform  individual learner development and growth along their Continuum from entry  to graduation very few Med schools have a clear understanding across the spectrum of  these Milestone and competency development about where the learner is and how do you coach coaching relies upon that data that is iterative so this  is very aspirational for us and many medical schools many medical schools already there Vanderbilt Oregon NYU  they've got robust dashboards that really Drive their Precision education this is the vanell framework it gives  you the systems thinking view of a cbme program across these domains so I'm going to ask you to  reflect here or what examples do you  have  everybody's good with this concept these  Concepts okay all right I'll close with learning  Curves in cbme very briefly um the EPO document we sent around that's there for your reference  that you can look at um but they're the epos that we built for the school of medicine this is my daughter's first  basketball game this weekend this is Caroline right here um she is not getting the concept she's five she's not  yet getting the concept that she needs to look at the ball and we're trying to put the basketball in the hoop so I  guess I have failed as a dad and this is our great coach one of her great coaches showing her hey Caroline you got to look at the ball um and I may or may not have  been dreaming about daydreaming about cbme when I was watching her on the court um but then I was reminded this is  a learning curve for her this is her first basketball game um in many ways it  ties directly to cbme and the dryish drish model that we have been alluding to here in this talk the whole time but  here it is the dyus dyus model these layers of development or growth from on  the x is time spent and effortful training or practice and the Y is the performance of the individual those  skills and competencies so D and driers outside of medical education built these novice  Advanced competent proficient expert and with time and deliberate practice you  make your way along that Journey um to become more deliberate practice and and  expert in many ways when you in the physician lens you put medical school is  generally at the more novice or Advanced phase graduate medical education is further along and then you make your way  up along that Continuum that's why the Continuum is so important so what's required though for  that learning curve there's a couple key Concepts here that um I think is really exciting to think about when we think  about medical students in particular um certainly legitimate participatory roles  you can do a whole several talks on this concept alone it's the idea that to actually develop your skills you have to  be able to demonstrate or practice those skills you need legitimate participatory  roles in a community of practice if we're sending Learners out to observe  then that implies that what they're doing is they're just observing the knowledge piece but they have to be  given opportunities to practice if we're teaching them the core areas of History construction for example then they have  to go out and be able to practice through significant roles in that process easier said than done they need  reflective practice they need to be able to reflect on what those assessments are in their Milestones get feedback through  that coaching process and then act on it do this deliberative practice is the work of of Ericson at all to say if you  don't practice iteratively then you're not going to be able to develop that skill and that comes through coaching which is one of the it's not a fat and  medad it's it's really one of the foundational bedrocks of cbme and all of  this requires a growth mindset and systems thinking to actually make that happen you have to have a growth mindset to actually engage in this growth  development along these curves so this is just an example and I'm going to close with this  one so this is the dfus dfus model across these spectrums I'm just taking one example of providing an oral case  presentation of a clinical encounter this is one of the untrustable professional activities so these are hypothetical the  novice might be does the novice understand the core components of a case presentation so a first year medical  student who's one month in they might many of them have been scribed so they understand that but but for everybody  novice level is understanding the core components the advanced beginner can deliver a presentation that may not be  concise or maybe it wanders that when I'm working with a a third-year clerkship student and it's their first  rotation many times this is the case they they know the components of it but  it's it's there there it's not hone to the degree that we want which is okay  the competent student can deliver a presentation that's organized around a cheap concern the proficient student can  integrate pertinent positives and negatives to support a hypothesis then the expert can do it at the at the full  grown filters synthesizes prioritizes information in a concise well organized presentation this is an example of what  we're trying to do when we think about Precision education or dashboards or our Assessments in cbme we have to be able  to create those Milestones which we're doing where do we assess each one of these Milestones there's a lot of back  workor that needs to happen well where do we ensure sure that we can assess that they're at the novice level that they can deliver a presentation that may  not be concise or wonders we have to intentionally design those assessments into our  process so I'm going to end with this I had some things about the competency Frameworks but the epos is what we're  building that's what the attachment is you can keep that as your reference this has been approved by our um Core  Curriculum committees and it's really our journey our ongoing Journey towards  cbme so I'm hoping we're able to accomplish some of these uh these learning  objectives we Define curriculum and Competency Based medad hoping you understand the dyus dyus model at least  the core principles the velli framework you can refer to as you're thinking about the systems thinking view  holistically of a program Miller's pyramid and the relationship to identities professional identities and  then some of the work that we're doing in the school of medicine that's really on a journey for cbme so thank you to  sharing thanks for the the team for having me and happy to talk comments  questions thank you so much Jed this was super helpful big Concepts big  Concepts um lots of uh wonderful models to help us to connect these big Concepts  to reality um what types of questions or comments do folks  have either in general or practically speaking  implementation I have just a clarification Jad I'm still struggling this is Joanne with the con with the  difference between a competency and an  EPO let me share  again come back to this one I know you said person and work  centered but I didn't quite understand that yeah so for everybody I  who may not know every medical school from lcme our accredit our accrediting  body is required to have a compiled list of educational program  objectives these are essentially the outcomes of the medical school so our  EPO document which I had in the last segment is really a compilation from three or four different Frameworks so  our EPO document Blends competencies and and epas into one un unified framework  of epos so it is a it's a it's a little bit of a mishmash of competencies and epas  but there's a reason for it it's because some of the epas that the goals that we  would want for the long term we don't think that Learners can do by the end of for a medical school that it actually  happens in the R2 or R3 of residency which means our apos are components of  that which are the competencies but there's things that we believe that they can do by the in a medical school that might be  epas okay so that was yeah I think I might have said EPO when I meant EPA the difference in a competency and an  EPA was but that last thing actually was helpful too yeah sorry Joan I didn't so let me  just I'm just going to show everybody the I think I saved it here here's the epas these are the  educational I'm sorry in trustable professional activities that have been codified in  2014 on that uh history timeline by the double AMC so you can see these are like  just take one um I don't know collaborate as a team member of an interprofessional care team EPA 9 which  um we're required to do by accreditation for interprofessional collaboration there's a lot that goes in the EPA 9 if  we were to unpack EPA 9 there would be several competencies and Milestones that would go with the EPA that we're doing  already in the curriculum we don't just say hey we're going to do EPA 9 assess it one time and done we're doing the building block and ideally by the end of  medical school maybe they're able maybe they're able to do EPA  9 okay thanks that's helpful yeah David has a question uh I  sometimes hear clinical faculty bemoan the fact that medical students are not allowed to practice their skills as much  as in the past for a variety of reasons are you concerned about  this and we all we recognize that it is one o'clock so if anybody has to leave please feel feel free to leave whoever  can stay stay for a couple more minutes for uh answer to this and other questions thank you yep thanks everybody  David I I I really great question I mean I'll give you my own my own opinion um  and we've written a couple papers about this and the textbook that we wrote that's entitled value added medical  education in many ways was the the I want to call it the burning platform but  the rationale for the book itself was based upon the fact fact that Learners  are becoming more marginalized in clinical settings for a number of  regulatory cultural hidden curricular reasons and they're not able to perform  legitimate peripheral participatory roles they're not able to contribute in some way we wouldn't expect the  clerkship student to decide on the dose of the diuretic and prescribe it unsupervised that's not reasonable but  it is reasonable for a learner to be pre-rounding on patients constructing history presenting placing orders that  can then be co-signed these are active participatory roles that I think in some  ways has got us far away from the the era of World War II and we were relying  upon acting inters to actually run the hospital to actually do the work and contribute in the gap of a Workforce  need from a physician lens and some of it's legit some of it it probably is  good maybe we had them doing students doing too much but there is a balance somewhere in the  middle Jed Paul schik again great talk thanks for putting that all  together um you know it strikes me that we we've been churning out  Physicians for as you pointed out decades and they seem mostly to get a  good to a good place this is a new methodology so I have I have two questions well one more comment a lot of  what's Happening Now strikes me as writing down in detail expectations  there were expectations all along but they were never made explicit or as  explicit as they are Nam so my question  is is there a way to compare outcomes if you will in terms of quality of  Physicians what whatever parameters you want to decide or prove past you know  testing board examinations it would take a long time and I can't think of a design to do that  comparison can is that you know because what's just like with the flexer report  compared to what is cbme better compared to what  I it's hard for me to Envision how we can know that yeah thanks Paul I mean as  usual Paul's got some the Deep conceptual questions which are really good um your question reminded me of the  work by Ken lmer if anybody's ever interest in the history of metad I mean these books that Ken has written are  they're fascinating um let me heal time to heal the let me heel book is around  the the Genesis and iteration of graduate medical education in the US goes all the way back to the 188s and  the launch at Hopkins and other schools and progresses all the way through I think to the turn of the  Century one of the premises in that book is when the resident Physicians used to  live used to live in the hospital they had that's what that's what their home was their hours were not counted they  were they were there they were fully immersed so take for a surgical residence for example I think one of the  example he uses is the number of appendectomies that a a surgery resident would have participated in or LED could  have been X number in certain number but it really high and then you fast forward  to today with with Duty hour restrictions um the the the teaming  changes in regulatory of who's actually doing the surgeries there is data to show that the number that a a a surgery  resident would do has come down so that Delta then creates the question how do  we know that the instead of a 100 appendectomies that the 20 they did  allowed them to develop the skills to go out and perform the app indomy without the supervision of a surgeon and I think  that was one of the spawning of cbme to say times have changed contextual  factors have changed so how do we ensure that the Lear the outcome is learning  it's not a process of you immerse you can teas steep in the old model of the 20s 30s and 40s because maybe because we  didn't tabulate outcomes we didn't look long-term at the outcomes for some of it but it was also because there was many  more hours doing appendectomies in that example maybe not so much now so I know  that clearly doesn't answer the complexity of your question Paul I think I'm trying to highlight the contextual  factors that exist on why cbme is so critically important which I know you'd agree with as compared to the timebase  model in today's context thank you  J thank you all for being here um than thank you Dr Gonzalo for your fabulous  presentation as you all know he is right in our uh medical education building and  throughout um the hospital when he's caring for patients so please um don't  hesitate to email him stop by his office see him um and he's happy to he loves  talking about this kind of stuff so happy to answer any questions further um if there are no more questions or  comments uh I will bid you all Ado and have a wonderful day and stay nice and  warm see you all.