Why Does Everything Need to Align? Assessment, Objectives, Outcomes

Good afternoon everybody welcome thank you all for coming today and we have what I consider a very foundational very important discussion I think outcomes assessment evaluation objectives are all and feedback they all feed into this very framework that we all need to implement as a part of our teaching because if we don't have we've very important elements something has lost in the mix where we're evaluating learners on something that we don't make clear to them as an expectation but today we have our chair of basic science education here with us dr. Rene LeClair was very familiar with these concepts she's going to speak very generally and hopefully they will all apply to your various situations please feel free to engage her in discussion and making the helping to make if concepts even more practical for you in your own teaching environment so please join me in welcoming dr. look there thank you everyone um I really want to make this more conversational because we're all coming at this topic from very different perspectives as undergraduate medical educator third and fourth year of clinical educators different aspects and see there's some pharmacy so-so rotations there's been all kinds of different ways in which we use this language and think about it in our context but what I was putting these sort of slides together I realized I think about this day to day at a very small level and so it was really nice to kind of step back and to see how I can think about assessment and objectives more broadly and how that may change things that that we - with our student encounters and so I have some very basic objectives and I wrote these thinking I wrote these any number of months ago thinking again in a very narrow viewpoint of how I personally think about objectives and so on you know we are gonna talk about the importance of assessment and objectives and outcomes and looking at you know diverse learning activities and these are all well and good but when I started again putting these together I finally thinking devise a plan to review educational elements and improve outcomes we really need to think about this beyond our own small microcosm and the way in which we interact with the students and it becomes a more interesting and a more powerful standpoint when we start thinking about hey what I do is maybe a very small interaction with the students but it builds on to something very much larger as a Maddi entity as well as an educational continuum for these students so I only interact with the students or predominately interact with students in their first and second years but I want to make sure that what I do in that interaction is something that they can build on it's not something and I'm a biochemist 400 others know in the audience I really want to meet service that's foundational knowledge that I heard something that they've met please remember recall or use in a setting that is that is most useful for them so I just want to start off with some language when we talk about alignment and talk about alignment in the context of assessment and objective what are we really talking about there what does that mean so I've aligned my objectives we tell our students that ESS man is lined to the objectives so yeah it should be a lunch the rest of our curriculum in the rest of the school at 3:00 so [Music] it's the degree to which assessments yield get results that provide accurate information about student performance so this is a very long definition and but it is that that's really what we're talking about so it is our learning activity going to provide results for the students that day new men grow and build on and the assessment must adequately cover the content standards with a program that's independent life [Music] - they have the best shelving man so are we aligning what we're doing with all of those things and this last part is a piece I don't always think about but in monthly reported in a manner so that other people can use that information right so that gets to you know how we assess do PR TV in a vacuum we have to have a global understanding that at some point in time that information has to be transmitted to another program to another residency director that conveys student proficiency right so we have to rate right miss fees for each student and it needs to be some sort of universal language of assessment you know it would be very difficult for a medical program say well we have as our students do who needs to assess their you know agility well nobody else is doing that all attachments there's anything and this taken from a middle school middle school educational textbook so these are pervasive definition of assessment if I look at key through 12 literature which is where most of the discussion started these are these are pervasive so what is your viewpoint when you came here today or you read my title what was what were you thinking about immediately it was a first snapshot you thought of with assessment objectives and alignment sir I just all these things usually never have any surprises so no surprises on either end that's a great point I'm not a fan of surprises I'm Addie on the disc tail so my surprises are not in my lexicon and hunter lives with me knows that all too well but I thought about this assessment and objectives I was thinking this is me right I defined objective in a classroom setting in some educational format in an elective in a clerkship and I defined the assessment and that in a very narrow viewpoint anybody else share that with me were you all thinking much bigger right I was thinking what do I do for that and so for most of this talk I really want to think about this from this standpoint so we all can't interact very at some levels with the students and these are different sessions right I have a session it for biochemistry dr. Elson has a session in radiology dr. Bing says physiology and we all have our own sessions and if we're thinking about those individually for a student that becomes very taxing now they have to align to the way we write objectives and the way rewrite assessment and that's a very unique but if we look at all of these more strategically and how these feet up into courses what does that course look like what is that whole construct of information what are we trying to convey collectively and then think of that each course has its own objectives and each divorce may have its own assessments to address proficiency the concept of great each horse typically has it great right and then courses compiled together into a line of study so it's really important that if we start thinking about this more programmatically what we do here at this level is really important to making sure that this line of study is cohesive that there's consistency that there's reduce redundancy that we're using that all very efficiently flow in students you have to assessment they feel very much or cared and not overwhelmed right and all this feeds up into our program and programmatically we're here to develop position thought leaders that's a School of Medicine sort of overarching goal so that's a huge trajectory to go from I teach the Krebs cycle here and then think about how that leads to a physician Collier but when you start thinking about it more programmatically it kind of helps contextualize the impacts that each of your sessions may have there's anybody only interact with students maybe once or twice over their academic career we help big pieces in which they contribute to try to go in like how big is big so I made our four major elements that we can use to say let's unpack this and how do we go from that session level that one section of you interact with somebody to a meaningful contribution so we've got to of course define the learning outcome but for that we are learning outcome is defined at the School of Medicine we're going to produce physician thought leaders and so however we interact with them in the session we need to think about that we need to think about how what we're doing at any level it's contributing to that goal this one is a little more more difficult choose learning activities that support that how do I at the basic scientist she's eight-and that a learning activity in bio comes through your cell biology that produces a physician thought leader that's a bit more taxing than developing a career clerkship elective that produces a physician Sawyer but it can be done with a little creativity and then assessing students so how well are these aligned how well are these matched are we all pairing things together or are we all only assessing knowledge or all only assessing very basic skills and then final grade and final grade sort of rolls up into this and for us that's not a decision that we made right a final grade of the School of Medicine for our students is satisfactory unsatisfactory but but we can kind of unpack that a little bit so define the learning outcome and as I mentioned we've got physician thought leaders that's our outcome right anybody have any different programs they work with where there's a different overarching outcome or gold so you have residency programs residency students Dietetic intern and Dietetic internship good so those may have different goals but I'm guessing it's something similar to have a career within that field right that's the success there so we've all seen Bloom's taxonomy and this my only slide with it which is pretty amazing with a whole presentation on assessment and the line meant to only have blooms on here once but really because right our objectives we need to determine what level you want the learner to come to come prepared to class and then we have a classroom activity that will elevate the learning and so so we may start with lower level objectives and then they have an assessment that is assessing higher up on the the level and this in-class activity or this learning activity is what's going to leverage this and this is all very good of them again if we're thinking very narrowly back to our session but if we think about courses now we could think about this is how does my course with basic science education maybe only needs to achieve knowledge right and maybe the clerkship electives maybe need to push the envelope so that the students are then raising the knowledge so if we think of that list of courses is more of a hierarchy as opposed to saying every session we need to push the students to create on Bloom's taxonomy at every single 50-minute interval which is really not realistic we can start compounding on that learning a little bit better and so as much as we all want to say my students reach the highest level of Bloom's that might not really be necessary on any individual session or any individual course and you see this in undergraduate education with the 100 level courses that maybe more knowledge Milkin see and then 200 and 300 level courses and at some point I think in medical education and feel free to chime in I think we all really became very conscious that there's not a lot of time and so we really tried to cram everything in and get the students to be applying at every level at every session which is the rayon in altruism but I think we need to step back and recognize that at some point there's knowledge there's knowledge that they need to achieve in order to apply sherry I think it's really one of the biggest challenges I all the engine people different programs from different levels so if I Parker trips do now early resident if I have senior resident how do what they call this in you may have your objective set for to understand senior resident play over the top or family about some of the shirt there for them thank you thank you so again if we think about this more in the context of a program we can think how does our session feed up into a course what is the expectation of your course or your section that you're delivering what is the expectation of that clerkship are you really asking them to evaluate something or to apply or to create something or is it really only necessary that you meet somewhere halfway up that and do your assessments align and I think for us if we sort of step back from our physicians thought leader goal um we are largely mapped to the ACGME core competencies and so you can then feed these sessions and start to link them and link your courses to your core competencies and that you know for me is mostly medical knowledge but there may be some team building in there etc whereas for the the clinical courses right we're definitely looking more at practice based learning and and other concepts of patient care but the medical knowledge is still essential to that activity within that that session so so that's sort of the first step what are we defining what is our learning objective and how are we thinking about that collectively and then if we look at learning activities right and so user and I have given any number of sessions on different learning activities and concept mapping and different things but it's really important is to then think what learning activity is gonna best serve this whole paradigm and so right we are an educator and we think of this is that we have objectives and they have a teaching activity and we have an assessment that's sort of our linear flow and you really should be thinking about these simultaneously as part of planning but that's what we're gonna that is how we sort of longitudinally think about this and the assessment is a really a reflection of those teaching activities from a stupid perspective they think about this a little differently students want to know what do you want me to do what are you going to text me or any of that that's their initial curiosity they're not really thinking about learning objectives in the same way I are they will use those learning objectives to determine their outcomes but the this is really predominant for them as opposed to the objectives so how are you going to to determine the effectiveness of that learning activity and this is very simplistic but when I was looking at it like you're right the students want to know late and I had a student class the other day we were going through some material with one of the clinicians we said well on an nvme question what would this look like and I thought you're right that's what they want to know how am I going to be thank you how am I going to be able to differentiate between a cluster a and a cluster B personality disorder using the stem of an nvme questions like how very specific so that's how they're thinking about this so I think it's important you know when we're talking about this is the students all have an understanding of how we use these words and so uh learning objectives is really what what do I want you to know when you leave the course or the session right what do I want you to know and then there's a Sussman and these are tasks really when we say assessment we think of tasks when we say assessment to a student they think test right and they think they typically although this paradigm is really shifting they typically think high-stakes tests non there they don't feel that that assessment word has a lot of forgiveness in it but I think that might be changing that culture or the students and then instructional strategies right and so that's your learning activity so what we are going to do - before we have an assessment so and your assessment may be part of your instructional strategy right there may be an assessment before the instructional strategy just to determine where everybody is at that that may be a piece of your instructional strategy so I think they need challenged by instructional strategy like teaching SEP 4 and there was came through our misrule ostensibly become here that became a secondary issue and so the contention is you know particularly if you're going to go into a class and lecture to the half-dozen kids would show up dozen that's a good day essentially are these the kids that are need remediation and support and want to be spoon-fed or are they there to really become father's very much so do you want it yeah so that the class attendees tend to be right now the highest end of the class they are the students that think they've got time to go to class the ones that should be coming to class because there are cheap that mastery of the content staying home and it's for them its efficiency under the second betina's trucks do they trust in fact because that's it gorgey question though is a comment because they have taught me but it is it's the expectation that the knowledge that those low level of blooms is something the students do on their own like you can do knowledge on your own and we expect that we say that you know they can do knowledge on their own students don't come to class and they need us for the apply and it's really those six students who recognize like if I want to apply and I want to create or I really want to conceptualize this knowledge I have to engage with this information at a different level and they begin to understand that early on I don't know what that looks like for you guys in the clerkships because they have to be there but I don't know do you you must see a difference in do you see a difference in students on clerkships with those that are there versus those that you feel like you really want to be here but how does that translate to you I mean it's not about tendons is it just purse tell not that you know what kind of questions do they ask you know what kind of attitude do they have how enthusiastic there's a whole range of experience right there on the Percheron right and those are thank you for bringing up because those are the things you that we use collectively to then translate to grade so you may have students that have the same exact score on a shelf exam but what their clerkship ultimately looks like in that evaluation includes those things as to how they show up and that's sort of the difference with what is the assessment versus what is the grade and that's how how we talk about it here so thank you there's a national discussion right now places my yeah and I think it's fascinating I I personally predict in ten years but you were just talking about failing the students are missing out on some of the best thing especially in the first they open once the caveat on the other side of the clinical situation is you never have an ophthalmology resident taking radiology workship never because all I want to do is go eat breakfast so it's very simple it anyway the step one is are really challenging for us as instructors because the students right away and within the first month as we know figure out that's what counts right and we can talk we could talk for an entire afternoon about why that counts so much or who who who values that much but fundamentally at some point that is the assessment right that we have come to value as medical education for better for worse but you're right supporter but several years ago we had we are all interested in the urban area you might get past the finite game [Music] running for you can say if this were a step exams the biggest so it is instructional strategies or something like well I know how and the about different essential strategies in flipped classroom and we don't use that word anymore inside that word like it or whatever model we want to use but this is a resource that I we use with our faculty for for mapping and I find it to be very helpful and so if this is fairly fairly old now but the double AMC has thirty different types of instructional methods that they have defined and so if you go to this website and you say I want to do this I don't know really what it looks like but I also am not sure if that's the right instructional strategy for my assessment I want the students to be able to apply something at the end is this an interruption 'el strategy that will help me get to that level this is a really great a great useful tool and so if you if you go to the link each one of these is linked out to a very short definition to say this is what this type of teaching activity would look like and these are the purposes that it would serve for you so this is really helpful so we a math all of our sessions so I know most of my BA chemistry is delivered through case based delivery as well as some self-directed learning so so I can say that these are tools and strategies which I employ and I also know what my assessment outcomes are and how those are linked right so if you find that you have a session that you're like I feel like the material is presented well but the students continue to struggle with maybe you should just step back and say am idle it is my instructional strategy aligned with what I need them to do when should I should I be refocusing that a little so some of these definitions get very rigid and you know not necessarily pedantic about it but it does sort of help you to say am I using the right words and the right type of instructional design instructional teaching has anyone use this or seen this inventory it's really helpful so then following up how do I know if my instructional strategy is effective does anybody so what you should not say is my assessment well you should not say if my instructional strategy as a set is effective when the students perform very well that can be true that can also not be true it depends on whether your objectives are aligned with your assessment right we could all sit here and the end I could give you an assessment of what the sky you can all say blue and I say well I was really affected my instructional method was wonderful and that would be ill aligned right so it would be more effective if you use something that was potentially non assessment and these are just some ideas you know like we use quick quick classroom polls so pull everywhere direct paraphrasing which is probably more what you would do the function setting or a teaching setting could you tell me what what you think the diagnostic diagnosis is did you tell me what you think the next plan of action is that's sort of feedback if you notice the student is always giving you the wrong answer or always not connecting some dogs that gives you some excellent feedback to teach it to change in that moment a documented problem solving which may be in the form of a pen note for the clinical aspect for us it may be some of us just say okay here's a clinical question take two minutes write three sentences on how you would you know what's the physiology behind that you read them when you ask students to commit to an answer often that answer is really inventive sometimes especially him we even it's open they're very good at multiple-choice questions they're very good at that that's what we've trained them to do but when you ask them to articulate something or to write something that's when you really begin to understand whoa I missed the mark or they missed the mark we missed the mark collectively we need so that's helpful third stop SS right to be good our objectives we've got some learning strategy and now we need to assess the students in some form or fashion and we need to understand that learning takes place right in our heads so what you're assessing is is a doing activity you're asking them to do something that implies they learned something right so we're asking them to do something with their learning and recognize that this can involve any range of assessments and I think this is something dr. ma Cheston isn't here but um it we do a really great job of here at VGC som programmatically with diversity of assessment we have all kinds and so just books generally we have formative and summative assessments we have all kinds of them and I think we have a lot to say that we don't even think of them necessarily as formative assessment but anytime you're giving feedback so if you call me on this time you say hey that was a really good time that's formative feedback right that's sort of formative feedback the students don't always recognize this so a lot of times we're told to go to the students say I'm gonna give you some some feedback and then they're like differently right Foreman in feat formative assessments can also carry a grade if you choose them - they don't have to be nominated and then summative of course it is summative so again this med bifidus also has a range of assessment types which is really helpful say how are we assessing what in our diversity of assessment but also are there ways in which other programs are assessing things that may be helpful to us here and I think we cover across the four years a lot of these I would dare say we cover all of them and then probably a few others as well anybody see one that they're like I'm not sure we do that here I think we hit on most of these so just keeping in mind you know going back a little bit to blooms is that hierarchy of assessment linked to our objectives so what types of assessments should we be targeting depending on what our outcome is right if our outcome is up here theorize hypothesize reflect our assessment is probably a bit abstract so it's gonna be something that's going to be written or organic it's not going to be multiple choice it's gonna be something that there's gonna be a doing task so that may include something like a standardized patient encounter or a or something of that nature where there's reflection on both sides of that fence versus these um lower level quantitative assessments which are mostly our multiple-choice standardized assessments so if our defined learning outcome right like our goal position develops physician thought leaders is never going to be something that we kept going to us that with a lower level assessment right we're never just it's never going to be binary did we do it yes or no right that's never going to be enough for the LCME and that's what we're paying area we're going to have to provide relative evidence as to why we perceive that that goal has been does every course need to include all of these rid this range of assessments no it doesn't and I think that's where we often get hung up for us for our early for our m1 m2 courses we do include a variety of assessments we just not which is a good thing that's a good thing but not every course needs to achieve this theorized level of learning all the time it's not always necessary sometimes it is but it really depends on what your target is what do you really want to make sure that they walk out of the classroom knowing or finish that course completing so here these are just some some basic levels of assessment that you would link to those words so if I have described adjectives as my defining learning outcomes how would I assess that what would be the best way to do that and there's a whole range of them so for us for the first years we have standardized nvme exams for summative assessment we have a written ice exam they have a spiral evaluation which is a piece of facilitator evaluation they have off skis they may have ultrasound practical so they're able to demonstrate skill on a whole range of levels what's the value of doing that reinforcements yeah it's great for reinforcements it's also really important early on to understand that if I ask a student a multiple-choice question that's like what kind of probe should you use on the ultrasound to look at the knee and they know that but then they do a skills assessment and they can't proficiently find the gallbladder you know if we only assess in one of those ways we wouldn't know where the downfall of that learning was on that spectrum so that's a real advantage we have programmatically here programmatically we focus on skills that's what we do a lot of here we focus a lot on skills we just mentioned it it's not necessarily the best for our teaching activities to focus on knowledge we have really high performing students so if we consistently focus on knowledge as we've talked about earlier we've alluded to students don't tend to come to class because the knowledge is the easy part they can do that on their own so I'd encourage you all to think about your sessions and say how could I maybe transition that a little bit because our assessment is generally a skills assessment we do do a knowledge assessment and the step exam is a knowledge assessment and so I think that'll always be sort of our challenge to say until those assessments become more skill driven the knowledge piece is always going to be the higher of the importance so they don't get to the skills assessment until step two so they've got to be better I don't know what the answer is and on going stablished with her maybe you know as an intern a over two they're personal doesn't even have an idea of which is easier to manage in a student like if I is it easier to deal with a deficiency in knowledge or sufficiency and skills I was trying to think about the other day and I wasn't really sure as a new student every student is the other day we might have a student that's always were quite we just great might not and I'll send the student today to talk and empowerment and you know like have the last rivet nuts it's really hard a lot inside the skills and they have different they are more willing to work on that area and if they think don't know what you're talking about brain and they're gonna be resistant to try to impress them yeah their self-perception sure yeah this might help of the knowledge to great because they might they might not know something about compatibility and then but you show them on the ultrasound they had the skill to find the gallbladder and then explaining about what see there my elbow knowledge I'm weighing a circle yeah good my point I think for a fur for myself specifically in the first year it helps for the help students to understand the skill or the application of the knowledge otherwise the knowledge is just really boring right knowledge is just like oh wow this is but understanding the application of that or where I might see it or use it is also kind of a helpful piece so we went through types of assessments and this was really just a rapid list of looking at the diversity of assessment that we have programmatically here and it is all-encompassing but we really do an excellent job in both formative and summative assessments across the range of the four years here to really address is it are we dealing with a skills deficiency or knowledge deficiency or where where are we excelling or where do we have educators need to focus more of our time and we really help kind of parse that out so doctor budgets and scheme does a really great job of that here which is very helpful and finally step four which is grading so grading is sort of a mundane term and we have the luxury of not necessarily having to deal with that too much directly but assessment does not equal grading anybody feel differently about that so so assessment right is what is an application of the learning right it's what students can do with their learning and then the great how does the grave differ isn't it also like what they do with their money [Music] there's there's an aspect of us the need to evaluate somebody in comparison to something else so there's either mastery learning approach for you they need to know what they can do with their learning at a certain commercial or there's a normative approach which how do you compare with us a little bit of information yeah sure they're definitely interrelated they're not they're not independent they dance all-star this has no correlation in my opinion how are you [Music] that's exactly right so so we're not girls into these you your performance your URIs your performance on a single assessment may be excellent but overall and that may be challenging and so it's this includes what students can do but it also includes that how they show up so if you look at grading schemas and just I just sort of panned a whole bunch of them they often include elements that we don't actually assess right we sometimes it's just like did they show up participation like we didn't that's just a binary we didn't assess whether they showed up and they were engaged or whether they were shopping it's just like were you here or were you not here it can be very subjective it might include something such as a peer perception but it's not necessarily something they can include lots of things so there is a little bit of subjectivity that can leak into there but it is essentially it will include multiple assessment so it would include all of the skills verse and the knowledge assessments and really define itself as a great so I just wanted to grading so we grade here at satisfactory unsatisfactory here and I pulled some slides from a previous presentation that was actually led by dr. banks and so in 2012 the am the let's go to pass/fail that was sort of this wave across the nation 2012 and the AMA followed the student the student urged to say pass/fail and even now will sit and we'll have meetings and and a lot of times we'll hear perspective as to why are we doing that or should we go back to an old school grading method and and those sorts of things but I just want to reiterate that we're in this paradigm in this model for some some very positive reasons so if we look across them the Maisie schools and PTC som sets it's right here as satisfactory unsatisfactory with a binary model some of the other schools do have diversity of models with Texas Tech having this five system scale which in essence is a through F so it's just different but the same and then these intermediary scales of you know pass/fail honours etc and a lot of these scales have been very much debated but when you start preening through the literature and you look to see well who is a pass/fail school in this data is from 2015 the majority of medical schools are binary pass/fail and then everybody sort of shakes out into some sort of other grading scale so when we we have that conversation about grades or how do we discern between students being successful or unsuccessful with this binary scale I'm not a residency director so I don't know but we must have a good way of doing it because the majority of the applications according to this are coming in you know with a pass or a fail on it and so there was all of these concerns like we're gonna change academic performance students aren't gonna perform as well in the in this binary mode they're not going to be as competitive at cetera and what we saw in looking through literature is that there's there's no change there's no change of student performance it can be better or it can be worse and and we're not seeing any change in step scores based on that big massive shift to a pass/fail system so um as much as we you know there was a lots of reservations about this there was no real change there but there was a change in student well-being generally a change in student well-being and we make essentially beyond the back swing of that where where students are now accustomed to this pass/fail system and the anxiety for all kinds of other reasons may be up sweeping and I don't know that's just speculation but initially after this we did see this trend of student well-being student positivity so student satisfaction for personal life etc and so on so those were positives and this 3-part scale where was this pass/fail honors tended to accrue let allow some of that anxiety to creep back in to say well I have this other thing I could strive for so I probably should strive for it and they felt a little more challenged by that so that was a drawback to these these three-part scales so this whole transition from this A through F scale really fueled a lot of stress and competition and really was it was negative toes group cohesion so that's sort of how we ended up in this medical education paradigm of pass/fail right now and we can you know that can be challenged by all kinds of things sure [Music] anything right cats low-caste as high test move straight yeah so if you look at the data and when you start adding back in those paths high paths the stress creeps in the stress just it's like well then that's just really an ABCD system and so we actually put this presentation together before we when we were both at the University of South Carolina Greenville before we transitioned up here and they had a really archaic scale that was like a a minus B B plus C and and so they we had a large cadre of faculty that really were staunch on that and so they initially got rid of the minuses so it was like you could be a B or a B+ or a C or a C+ but there were no minuses and people were like having anxiety about the loss of the minus the loss of the c-minus and our students really didn't like it because I said like this is just so many letters and pluses and no one has this so we put the presentation together in the school is now subsequently a pass/fail school but it didn't make any sense sure just praise me distinction in there are not adding degree yeah so so that is a bit the letters of distinction does add a piece of competition to our students here yes that's a very very very good point it does it does add a level of stressor that was accompanied my question too with students graduate those how does that work the letters of distinction would go onto the record a part of the the Dean's letter for the students for residency so so we have a chart that sort of illustrates the overall distribution of Bob you can chime in you've been in these means that the overall distribution of Dean's of of letters and then the number that the student has achieved four chairs letters they frequently become critical they you know setting percent or whatever and unfortunately in your highly competitive residency the student didn't get any of our letters of the station kisses that yeah so how do you differentiate between somebody who's okay like the commercial this is important in the world we're gonna live you know be paid by performance graduating just okay people is not probably like describing before right and for good or for bad pass/fail system creates a single line where you can't create a sense alter desired for further achievement to prove your quality process is all I got a new EZ Pass yeah so that's a great all I have to do is pass so that's a great that's a great point and so what there's actually a whole body of literature and the whole grading system called the specifications gratings that has Illustrated that that binary it really depending on the cohort they upped the ante themselves they strive for something better so that that pass is really not seventy it really becomes a B or a B+ so the past we have to think rethink a little differently it doesn't necessarily remove that sort of grading that the learning becomes more intrinsic motivation because you've removed that sort of extrinsic numeric value so intrinsically the students become motivated to learn and to achieve and so I agree it's difficult to say well is your past the same as someone else's past we have really highly motivated students but it's difficult for someone to know that our passes is really a high level and I think that is how the students and how these letters are really gonna have to speak to say you know what does that student look like how does our path look different I don't know does any any anyone doing residency application review you want to say how do you look at this and say that's why there's a debate about the assimilate recce because a pass/fail all the letters look away I mean everything looks alike and so that's just hoping for the best right so you have to do they pass the USMLE zone the first test I think you guys kind of have a stratification so we looked at the you know that one of the big arguments where we were was that how will residency directors be able to distinguish our students without this number and I think I have a slide at the end am just better to go through it but if you look at sort of a list of things that residency directors look at the actual grade in in its so low it was like after whether or not the student can speak another language it was actually that low because the step score was so high on this list of value systems to say and the reason for that is medical education looks different right so if if I have a class like - foundations of tissues and cells and the students get an A and somebody else has a course that's the underlying pathophysiology of the heart lung and kidney what does that mean that they got a B or a C in that I mean you can have speculative guesses but unless you're really gonna dig in and look at the course descriptions how do you how do you judge those across the range I don't know so that's great point so just to summarize I would encourage us all to sort of go back and look at what we do with these very teeny levels and I started I'm teaching a class on Wednesday digestion absorption if anyone's really interested for them ones there'll be a lot of fun and I really thought well ha how does how am I going to make that transition to position thought leaders and what do they really need to know about that is it really just knowledge or do they need to be able to apply this right they're not doing a differential on someone coming in with gi pain or diarrhea like I just need to make sure that they leave there with some fundamental information so is it within how your course feeds up here and what we're all doing to contribute to this physician thought leader concept so thank you all for your interaction and engagement I'm happy to take any questions.