Testing is Teaching

Good afternoon so good to see all of you on this very cold day so good to see you hence my my snowy background I thought that um that's a little bit more appropriate than my fall background at this point so we have been talking for many many years at this point about um about moving from a a more teacher-centric uh to a much more learner-centric uh teaching learning environment right so so both Learners and teachers this is confusing for both Learners and teachers a lot of it you know what does that mean does does that mean that you know as a teacher I just stand back and they get to do a lot of the learning them by themselves and I'm just sort of a reference point and for uh you know the Learners what what does that mean for them so so one step in the right direction is really helping our Learners to be more comfortable with metacognitive uh techniques and so I think that's going to be really uh the focus of what we're talking about today we have two outstanding presenters today um Dr Andrew Binks who is associate professor professor for the Department of basic science education at our medical school and Dr Emily hope Forest who is assistant for student affairs and director of academic counseling and Enrichment Services and assistant professor of the Department of basic science education at Virginia Tech Carilion school of medicine so thank you both for joining us and we look forward to hearing and engaging in your presentation if anybody has any questions or comments please feel free to either unmute yourself at the appropriate time or just go ahead and type them into the chat if you feel more comfortable to do so and we'll try to monitor the chat as best as possible and interject appropriately go ahead Andrew and Emily take it away thank you Dr wicker okay testing is teaching Andrew and I both feel strongly about this and um as we navigate into this space I want to use this quote give a man a fish feed him for a day teach a man to fish feed him for life I think metaphorically speaking when we do this work when we teach students how to practice metacognitive habits we teach them how to fish there's a variety of factors that contribute to a learner being successful in medical school and metacognition is one of those features and it's among a variety of features but it also tends to undergird some of the other elements and so in terms of cognitive processing habits and thoughtful schema development and kind of reappraisal in terms of the learning setting shedding short-term goals and long-term goals it can play a role in a lot of these other elements that contribute to medical student or residence success so let's move forward and talk a little bit more about it today's learning objectives are going to include the positive impact metacognition can have on learning behaviors some barriers that do get in the way of students practicing metacognitive habits how you can determine how barriers of development self-reflective processes can be overcome and then incorporating it into the curriculum and so Andrew will talk about that last piece quite a bit so who is a self-regulated learner and so before we get into metacognition specifically I want to pull back a little Google Earth view of things and look at a self-regulated learner it's an individual who takes a proactive approach to their learning experience and it's the type of learner who exhibits the traits that many medical of those of us who are seeking to support medical education want to see in our Learners they're self-aware they're self-motivated they set proximal goals and learning they have the ability to identify appropriate learning strategies for a given goal they monitor their learning they reflect on their performance they're willing to adjust when they identify a space and time when they need to be adjusting for better performance in the future thanks self-evaluate and they adapt as needed and perhaps most crucial they manage their time appropriately to make sure that all of this is possible metacognition undergirds many of these features of a self-regulated learner it's step one metacognition how one moves from being a cognitively passive learner to a cognitively active learner so to be metacognitive to think about one's learning there are a variety of different characteristics or building blocks that are necessary in this space task awareness strategy awareness and performance awareness so let's break those up and do in their individual Parts task awareness the ability for a learner to say okay what is the goal here what are my goals in learning and not just to say to pass my next board exam it's setting those more proximal goals those intermediate goals that allow them to get to that longer term goal and be thoughtful about that and what those goals look like so today I need to learn about metabolism so that I can build upon that and so then it gets to what do I know about this topic already am I a novice when it comes to the concept of metabolism do I Define my learning status as intermediate or am I an expert in this and I therefore don't need to spend a whole lot of time on it identifying where one falls in the Continuum of knowledge relative to a given topic really can help them identify what and how they're going to do in that next step that we'll talk about in a moment they also need to really understand is this a topic that I struggle with can I pinpoint where within this topic that I struggle if a learner does find something challenging doesn't like it or both that's going to play a role in how they uh strategize that task it really does matter how you feel about content your brain is listening to you so you need to identify strategies and therefore you want to ask yourself strategy awareness questions what strategies have worked for my brain when learning new data in this this type of data in the past what resources align best with these strategies so if I know that it really helps me learn metabolism by watching a video about it and having a textbook open at the same time so that I can compare and contrast pausing the video and looking at the text at the same time or maybe annotate the text with something that's happening on the video then I want to employ that strategy in this moment if I've watched a video and it has been unsuccessful for me in the past I want to ask myself why perhaps is that the case was I watching it in double speed and not really learning anything did I posit it logical intervals and so allow yourself to really align a resource with a strategy and then ask yourself questions about how that strategy is working do I organize my material or do I just dive in It's always important to really get a plan of attack with a given topic area and so it's important for Learners to really reflect on whether or not they're identifying a plan of attack for their content or if they're just opening a text or starting a video page one which strategies do I want to apply to different types of information not all information is the same information is manifests in a variety of ways some of it's semantics some of it applies that semantic detail some of it requires us to integrate information in a variety of different ways and so the strategy that we employ for different types of information matters you'll hear Learners in the med school level talk about using on key on a regular basis Anki is a flash card type resource that has a time and a place but it isn't the end-all be-all questions are far superior to Anki overall unless of course the student is getting hamstrung by some sort of semantic detail that a repetitive process that a flash card can offer might help ameliorate performance awareness questions can I successfully recall or apply material to practice questions so I've identified my task I've identified a strategy to attack that task and now I need to really look and see if my plan is coming together as I intended it to and the way I can really self-assess that Midstream is to do practice questions though those questions aren't just a check box experience they are a learning opportunity in the formative stages and summative stages of learning and when I struggle with a topic do I modify how I learn or do I flee the scene well I'll get to it I'll learn it eventually it'll come along I just need more time that's fleeing the scene a learner needs to pause in that moment no I don't understand why don't I why don't I understand what has been my strategy toward this task and what do what perhaps could I do differently in terms of this um the approach that I'm taking to learn this content do I strive to learn from my knowledge gaps informative moments so formative moments happen all along the Continuum of learning so if I'm watching a video and I pause that video and I say all right what have they talked about so far what are the key takeaways from this first 15 minutes of this video If I Were a professor and I were writing a question over the first 15 minutes of that of this video what would that question look like if I can't self-assess in that moment then I need to rewind the video and watch that 15 minutes over that's one example another example would be I get to the end of the video and I do some practice questions and I can't answer any of them correctly I have not learned from that moment in time so I need to assess what was going on in that space and how can I correct not just push forward and hope that it comes along later what types of mistakes did I make on The Format exams why and do I need to Tinker with my current strategy for a given topic now formative exams is very much meant student language but you could easily change that and say at the bedside when I'm being asked questions by my attending or you can add any sort of formative learning opportunity that is presented to a learner along the Continuum of learning and Andrew is going to talk more about this particular piece in a moment so potential positive impact that metacognition can have on learning one study that I want to raise in this moment is was done on medical students with an nf55 it was a five-year qualitative study that interviewed students who had remediated and these researchers identified that the students that they interviewed their um they found they had a cycle of failure four maladaptive themes that led to this cycle of failure and those four maladaptive themes included problematic goals and strategies which metacognition very much undergirds and so really reinforces in my opinion the importance of metacognition a second maladaptive tendency was not seeking or accepting support and so in this one if we can get our Learners to allow us to help them in the moments that they need help we can help ameliorate that first one by helping them identify goals and strategies to help be successful if we don't get them to seek and accept support then they fall through the cracks normalizing failure or attributing it to external causes sometimes our struggles are related to external causes that that is very much a factor in some learning settings but it's important to really get into the particular Learners struggles and identify if in fact that is the cause it can't be I would argue that it can't be the cause all the time and then protecting self-worth it's not me it's not my fault I didn't do it again it's important for us to protect our self-worth sometimes when we're feeling moments of struggle but it's also important for us to own where we have growth and where we have opportunity to do better and if we help establish a very safe learning environment we can help our Learners both protect their self-worth and own when they when they have growth to do some additional barriers to implementation would be time sometimes Learners just don't feel like they have the time to do this type of work but I would argue otherwise and I'll get to that in the next slide poor understanding of its value they don't see that metacognition is something that they need I'm not the type of learner that benefits from that stuff I've gotten this far I'm I'm in medical school I'm a resident I don't I don't need to do these these exercises and mindset an individual with fixed mindset often thinks that mistakes failure are um that's something to really investigate get into because it again that notion of protecting self-worth it's important for us to really help Learners achieve a more growth oriented mindset in that they lean into those mistakes they allow themselves to really dive into that space and how we do that is helping the Learners understand that time spent on the front end is overall a time saving measure if you can organize and strategize give yourself clear paths to your goals and objectives at the front end and then activate that plan self-assessing it along the way you really in the end save yourself a whole lot of time because you're not trying to do it at the back end and and remediate and Repair on a regular basis incorporate many cognitive questions into learning moments that help students experience the benefit and develop the Habit any time that we as Educators can incorporate metacognitive opportunities for our Learners to help just have it be part of the routine and that part of their daily practice they will then themselves develop these habits I can't tell you the number of students I've met with one-on-one who have said to me it while they're studying for step one for example that they are really thankful that they that I have talked to them about some of these habits so the day that while they were studying for stat were employing them conversely the students who regretted not employing these habits um and because they didn't have the skills to apply during their step study so really encouraging students to do it early and often is the way to go developing again developing that growth mindset and Learners to promote the importance of learning in the interest of avoiding sub-optimal learning outcomes and so really all right now I'm going to stop sharing for a moment so that Andrew can we can shift to Andrew's sharing screen this would be a great time for anyone who has questions to just jump in and ask and I can happily answer well Andrew is pulling his slides up okay um so with those barriers in mind and and trying to overcome those barriers we've started a project at the medical school to try and have that regular uh integration of uh metacognitive processing uh built into the curriculum and the idea is that what we want to do is when a student goes through an assessment they'll immediately look at what they got wrong or perhaps not why they got wrong and one of the barriers that we found was most of our assessment there's multiple choice a multiple choice gives you a binary input we either get their question correct or we get it Incorrect and that helps a student focus on what they got wrong but perhaps not why but within those two binary outputs of the multiple choice question uh answers whether it's correct or incorrect there are hidden uh there's hidden information the students if they get the correct answer we're hoping it's because they're competent and they actually knew the material alternatively they are consummate test takers and they're less likely but plausible uh reason for getting something correct is that they actually guess correctly by um looking at the question and answering hundreds and hundreds of questions prior to taking the exam more importantly in the incorrect uh category we may have it's just a simple cognitive error they didn't know the material but there's a number of different reasons for even that they may not have studied the material or they may have purposefully not studied the material prior to the exam and triaged it they may have learned it incorrectly or they may have thought that they'd learned it but actually uh hadn't learned to we've labeled that a learning illusion on the other hand it could have been a simple test taking error and these are well described they could have misread the question misunderstood the question simply run out of time or that classic of changing the answer to the wrong option after they've actually selected the correct one So within all that binary uh output of the multiple choice response there is a direction for the student to ask just what they got wrong but we tried to integrate a mechanism to get them to encourage them to ask why did I get that question wrong so that they can correct uh correct that learning Behavior so here's our rationale we have regular multiple choice questions in our formative and summative exams but our formative exams also have a review session where we can go through this go through the material and these have traditionally been very content orientated where we just assess why what the students got wrong and they're interested in then correcting their understanding we wanted to take this as a regular opportunity embedded in the curriculum for them to look at why they got something wrong and go through US error self-assessment the rationale for this is we want them to regularly self-reflect and make that transition of what did I get wrong to why did I get this wrong but there are other opportunities from that from that information as well if we go through a formative exam and see why they're getting questions wrong this will help uh Dr Holt and her work and student Support Services to try and improve their learning behaviors if they can come in knowing their behavior or knowing why they're getting questions wrong we can focus our support similarly we can actually use that information to actually develop our content delivery from large number of students are misunderstanding a material we can develop our curriculum to cope with that and we can improve our assessment strategies as well if uh errors are showing um some some consistency over uh certain questions so we've adopted what we've referred to the uh error reflection method or the um and um it's based on previously published uh data from a couple of sources including Dr Nolan here at the school and at the end of a formative exam the students go through their review and we ask them for any question that they got wrong to identify from this list what type of error did they make and the first four errors that we list are those test taking errors that I've misread or misunderstood the question ran out of time or uh skipped the question or change my answer to the wrong option type errors five through nine are all what we've labeled as cognitive errors that I never saw this material before I've learned this incorrectly the content looked familiar but I couldn't determine the correct answer I triage studying this content which is similar to our type 5 error and I was convinced my answer was correct there's our learning illusion and just in case there was something else that we give them an option to take other and here's what we started to see so I'll explain this graph a little bit more uh thoroughly because we asked the students to assess what type of Errors they made um during uh the formative exam review and then we look at their performance in the summative exam and what we saw was as they increase the number of errors that they report they gain more performance in the summative exam so this is a number of errors that they're reporting and they are in the formative review and this is their performance uh in the uh the following summative exam so what we see is with the cognitive errors shown with the filled circles there was an association between reporting more errors and then doing better in the summative exam but we did not see that relationship with test taking errors that seemed to be independent so we made the tentative conclusion here that as we increase their um self-reflection they actually improved their performance what type of Errors do medical students actually make well here's our test taking errors here and this is the percentage of their occurrence and here's our cognitive errors as well fortunately most of our our responses types one through nine covered um the the type of errors that students make with very few needing to tick their other option always the first thing that we saw was that cognitive errors are much more common uh than test-taking errors and that wasn't really a surprise to us what was a surprise was actually the frequency of test taking Errors By what we've always assumed what consummate test takers one in four of our errors were actually test taking errors and this is inconsistent with our learning learner's perception because when they come into uh Dr Holt's office the most commonly commonly reported reasons self-reported so why they get a question wrong was they actually label them as the test taking errors are being the most commonly reported well we see from our data that they're not the most common error and if we ask them to formally assess what type of Errors they make it's actually the cognitive errors what they're coming into Dr Holt's office saying that I'm I'm it was my fault on the exam when really there's a more significant problem with cognitive errors so looking a little bit more closely at those cognitive errors I'm coming off there at around about 10 to look at those um an arbitrary cut off right really to see one of our most common errors and then number five number seven number eight and number nine looking at those a little bit more closely number five is I triage the material or I um I left this I never saw this material before and so this is a formative exam so one of the possibilities is that they've not studied this material for the formative with the plan to then study it for the summative exam that was to follow so this was a fairly common error and this is also seen in uh number eight which is I triage this uh that's studying this material but by far the most common error was number seven the content looked familiar but I couldn't determine the correct answer and we've attributed this somewhat tentatively to shallow learning behavior that I read the material I looked uh I looked at the content I thought I understood it but I didn't have a deep enough understanding to answer correctly so this was the error that I actually made and then this number eight here was I actually triage studying the materials so they're fetting up that they didn't study before the exam and they're going to do this for the summative Maybe number nine was our learning illusion and that comes close to our arbitrary cutoff of 10 10 and we've attributed that to learning illusion again I thought I knew this I was convinced my answer was correct but I actually hadn't studied this uh sufficiently to get a full understanding to get this uh question correct we then in the theme of sort of trying to improve our our uh our curriculum we looked at the errors and how they were distributed by discipline so here's the same graph with all of our errors but this time we've split those arrows up into the frequency of occurrence for these five different disciplines embryology pharmacology biochemistry physiology and histology and generally there was no real effect of discipline they make the same errors and the same for each of the disciplines apart from number eight and number five where there was a there was an effect of discipline and there's a significant difference in the distribution of errors and so we drilled down a little deeper on that and number eight of course this one here being eat I triage studying this content so to look at number eight a little bit more closely this is what we saw most of our biochemistry histology and Physiology were about the same the same number of Errors per question uh occurring for each of those disciplines but we saw this jump up for embryology and pharmacology and should we be concerned about this uh they met them making this uh this error this type of error but no we didn't have to because when we looked at the summative exam results and drilled down into discipline they performed really well on farm and embryology so while this may have been an alert to us we knew that we didn't have to change anything it was a behavior of the students that were they were just leaving studying embryology and pharmacology until later and closer to this at the summative exam and there was no need for us to make any intervention so with those uses of the uh of the ERM what we've tried to do is to regularly increase the Student Self of reflection on what type of errors that they're making uh and so they understand why they need to change their learning behaviors and to make this a normal activity and have it embedded regularly throughout the curriculum we've also been able to use this to help Foster that self-regulation in conjunction with Dr Holt's office and support that performance and strategy awareness much more because now uh M has that information as the students walk in they can say this is the type of error that I'm making and so she can focus that support and we've also been able to focus curricular responses or not have a response and just be able to interpret the students learning behaviors much more readily as well as help them address those uh those learning behaviors with the student Support Services just to finish up um this is what the error reflection method actually looks like there are the errors uh that we that we've listed and so for every question that they get wrong the student then just fills in the question number and labels it with an error type and that allows us to map back to discipline um by using the uh the mapping that we have for our assessment and we've published this method uh in um and uh so so in medical education sorry if you want to have a look at that paper the QR code there will should that directly lead you to the uh to the PDF if there's any questions we can uh we can more than happily answer them there we do have a few questions in the chat um Andrew and Emily if you just want to go through I think it just starts right with David music is right where the questions start if you want to take a peek do you see them or do you want me to I've just opened it up so David is it possible to assess our students fixed versus growth mindsets upon their arrivals so that we have a better idea of which students may need additional support or is this already being done in some fashion um do you want to address we currently don't assess their their mindset but we we certainly could look into uh how we might do that there's a couple of mechanisms that come to mind something that I can definitely investigate good question h so the next one is uh from Charlie she uh don't we actually emphasize the what rather than the Why by our emphasis on boards part one and two absolutely uh that's that's a very good point and it's uh it's a it's a one of those barriers is that the students learning is very grade focused rather than learning focused and this is an attempt to make that help them transition from that great focused learning to learning focused learning uh by making them just um look at the why rather than the what uh David I'm surprised they didn't report more number five never saw this material uh before this is a frequent complaint by students something shows up on the exam that was not covered in class or other instructional settings uh when you formally asked we did get those surprises and this is one of the surprises that we did get it and the the frequency of complaints by students um was similar for test taking errors as I explained when um what um Emily hears in her office and what we hear is actually different to when we asked asked them to be much more self-reflective and when they are more self-reflective we get different responses uh Arthur uh this approaches will help develop the skills that are used in uh practice based learning Milestones called self-reflective reflective practice and commitment to personal growth this this is really what we were trying to do I think many medical curricula have elements that are really focused on this is how to be self-reflective and it may be a workshop and it may occur for two hours in four years and what we've tried to do is to really embed this that there is a focus on generating self-reflection but it's that persistent behavior that we've really tried to um establish uh with using this method and using um multiple choice assessment there's given us that Forum because it was not a well-tapped resource because of its binary output and drilling down into that binary output we're hoping that we're increasing that self-reflection uh Rebecca pawley uh going back to David's question on mindset could that be assessed during medical school interview days um I think so um I but I think that one of our issues is um the students that are not self-reflective are tend to avoid self-reflection and do you want to so those that really need this um mindset is also such a tricky um tricky topic because one's mindset isn't um it isn't stationary and it isn't the same among all things and so I could have a very growth mindset when it comes to snow skiing and I could fall down 12 times and not have any issues with it get right back up and keep going but I could concurrently have an incredibly fixed mindset when it comes to math and just I can't do it it's not for me I'm I'm out and so and and both of those things can change over time over time I could decide that nope I can't fall anymore the skiing is not for me I just don't do it and concurrently I could have a math teacher who just inspires me in a way that and allows me to fail in a safe way and such that I'm like oh maybe I can do math and and so I I changed to a more uh growth mindset and I I lean into those mistakes I lean into those Falls so to speak and so I think that it's it can be very tricky to try to assess because I think if in um in an admissions capacity we could set up a question that really kind of puts in there are you growth or fixed mindset but it's going to be very specific to that student and that the format of the question in that moment in time so um it can be tricky to try to do the next setting and with that said I'll still investigate and see what's out there I I worry for the same reasons Emily about that um especially in in the vulnerable place they are when they're applying for med school um but it is so context specific it looks like John upling has a question as well yeah pardon me I'm driving so sorry if this if there's no um Andrew and Emily I love this presentation can you do you have any comments on how this would translate to the clinical setting where the student is might be asked to assess themselves uh their their performance is assessed on a little bit more subjective criteria um and rarely the the hard yes you got the question right no you didn't get the question right um any thoughts about how this would translate to a to uh to allow us to objective setting like a clinical rounds or clinical precepting the ERM that we are using in the formative setting for the medical school I think there's still an opportunity to reflect it with a Learner in that in a safe way um post encounter to say all right let's let's reflect on that moment in time and then where were the strengths and where were the weaknesses in this moment in terms of knowledge gaps uh is it your perception that this is something that can be ameliorated through additional reading or is there something more going on here that we perhaps would be best addressing in the Sim lab for example and and really breaking the whole experience down and do its individual component parts and identifying where within that process things are going astray so that we can address them in their individual component parts um I I do think that while it would look slightly different it can be still applied to the clinical setting as well and I would think it would depend somewhat on the mega cognitive abilities of the of the preceptor to understand where the where the process might be falling down absolutely I think providing that scaffolding as an educator can go a very long way to helping a learner develop it themselves I also wonder if if continually reinforced if as a faculty a group of Faculty we work together to continually reinforce these metacognitive processes even though it's context specific it becomes more um more natural for a learner to think in that way so they more naturally adapt regardless of the context to that more reflective process because they're more comfortable with it their self-efficacy increases because they've been successful with it in some contexts and so they feel they're able to are better able to apply it to more challenging areas Dean lehrmann mentioned the importance of mentoring in his progress notes today and I think that that fits in with this as well so if an attending or a resident that's further along in their process really connects with a student to try to help them in this space this is what I do in these moments in time this is what I've tried during these moments in time this is what's helped me they really lean into that opportunity to also Mentor along the way that can go a very long way for a learner not just in in an individual learning moment but long term in in terms of fostering their caring their sense of belonging and their overall learning process long term oh looks like we're we've got some things coming up in the chat as well let's see I think it's starting back with Arthur some schools use situational judgment tests to help discern fundamental skills and or values but what uh but are limited in the way that Emily just described right um Charlie mentioned unfortunately the Prelude to med school I.E undergraduate also emphasizes the what exactly graduate school is quite different the why is more of an emphasis I was thinking the same thing Charlie as we're discussing this um it would be so nice if they came to us so much more metacognitively aware and I wonder if that's a little bit of an equity issue too there are some schools that really reinforce these processes throughout uh do do you tend to see that Emily where where a lot of Learners in their undergraduate education or even in high school have been able to learn some of these Concepts and apply them so they're better ready when they hit us I think it depends on on the setting truly I in K through 12 and they are definitely leaning into metacognition and and different strategies to help Learners develop skills over time but that is happening concurrently with this grade focused learn learning we are grade focused people and so it's they don't always coexist well right that was a great point about uh how graduate school is so different um and it is so so funny that they use um you know multiple choice tests uh to deem qualifications for uh great uh graduate school um Jerry I I feel so fortunate that I went to grad school before med school to be candid absolutely as I approached medicine it really was great I think one of the hopes was with the transition of step one to pass fail this would reduce the great Focus because there wasn't that the heavy weight of that number of a graded score anymore uh I think if we were naive um back then back when it changed I don't think we'll see that that really helping but there is still that great Focus now it's just a pass not a number and I think we've still work to do hopefully in different numbers okay and David says I think that Carol dweck is one of the primary principal authors of the concept of fix versus growth mindset true uh she has an assessment tool available online absolutely and then sure mindset is variable that in itself is not a sufficient reason to avoid testing it um agreed uh Charlie this was a great discussion verbalizes my experiences between grad school and med school a new insight changing attitudes however is the next thank you very much Arthur says medicine tried a similar approach 25 years ago when they looked to find students from non-science majors yeah so thank you this was um a particularly um appreciated the breakdown of the data with basic science this was a tool that Emily introduced to me I guess at the end of this past academic year so we started for the multiple choice portion of clinical science started incorporating that for this Academic Year so we'll be interested in um similarly breaking down our our data for clinical science we've been doing more of a like video reflection uh more qualitative with regard to our standardized patient encounters and having faculty facilitated and peer peer contributing small groups for reflection opportunities but um just trying to think of more and more ways to incorporate pauses particularly within some of the the clinical science activities for self-reflection so thank you both it was an excellent presentation thank you Arthur also made a comment grade Focus will worsen as there are fewer residency positions than um graduates hoping to fill them any other comments questions areas of concern S period just a very helpful discussion thank you all so much and as you anybody who knows Emily and Andrew knows that they are always available for further discussion um very involved in our faculty development opportunities and ready to help our faculty in any way shape or form um to improve the learning experience for all of our varied learners so thank you if you want to contact them at any point um Beyond this discussion please feel free to do so thank you and I hope you all have a wonderful day thank you thank you.