2021 TEACH Education Day

Welcome everybody good afternoon welcome to the second uh repeated keynote session for our 2021 education day um the first one was fabulous so you all are in for a treat and this is the last session the last keynote session but as a reminder this evening at five o'clock we will have our um awards and recognition poster recognition and awards ceremony all virtual of course um and so you can log into that if you have colleagues that you'd like to encourage to attend that who haven't yet registered all of the links will be on our website so feel free to leave them there and they can attend that way um it just is a little reminder there's no eating or drinking in here so you can stare at your lunches today for a little bit and let your stomachs grow a little bit but um just if if you just grab them on your way out that would be wonderful um so uh moving on the for the next hour we really get to focus on retrieval practice and spacing this may or may not sound exciting to you but it is really exciting to me and uh mainly because the vast majority of concerns that i hear from folks about teaching and learning in the health professions is centered around how do i make it sick how do i get folks to learn and then be able to effectively recall that information at the right time so dr megan sumraki is here to talk with us about two really good approaches really effective approaches for um helping us to do so and if you know me at all you know i really like theory i really really like practical things and i love when we can put those two together when we can really look how theory impacts practice and how we can it's a very circular thing so um that's what this session is all about today so this impacts our learners all throughout the health professions and all throughout the continuum of medical education and of course all of the teachers who teach those learners so that's all of us um so uh dr megan sumraki is an associate professor at rhode island college she received her master's in experimental psychology at washington university in st louis and her phd in cognitive psychology from purdue university megan studies human learning and memory specifically applying the science of learning and educational context her research focuses on retrieval based learning strategies and the way retrieval can improve meaningful improve meaningful learning megan has delivered talks and workshops in both the united states and abroad at scientific conferences and with educators in primary and secondary schools institutions of higher education medical schools and teaching hospitals and also with the united states state department megan is passionate about bridging the gap between research and practice and education please join me in welcoming dr megan thank you thank you can everybody hear me okay i think that yes mike's good okay so uh before i jump into the talk i really just want to acknowledge um the team that i work with um the learning scientist team we're an international group of cognitive psychological scientists who are really passionate about uh bringing the research and practice together and making sure that we're not just writing in our in our peer-reviewed journals which is important but also um you know getting out there and talking with educators of all kinds and so we've got dr cynthia neeble who's at vanderbilt dr carolina cooper tessel is at the university of glasgow in scotland and dr alfiani kaminsky is at st adventure university in western new york and so this is my talk but really it's all of our work so to give a little bit of an overview for our talk today i'm first going to explain how we apply cognitive psychology to educational contexts just to explain that process a little bit where the evidence comes from and then we will talk about two strategies based practice and retrieval practice i'll explain what the strategy is provide an example of evidence to support its effectiveness and then we'll go into some of some practical applications that can be used in medical education from the med students all the way up to continuing professional development uh we'll talk a little bit about putting them together i'll give you some time to think a little bit about how you might utilize the strategies and then i will provide some free resources because i know from my work that talking to you for an hour does not mean that you only have everything memorized and become experts and be able to use it forever so um [Music] okay so how is it that a cognitive psychologist goes about figuring out how their work can apply to educational contexts whether that be a classroom a practical teaching hospital whatever the scenario what we do as cognitive psychologists is we really study at our core basic human processes the way we process information so we are learning about the building blocks of how of how we think how we learn how we do so things like memory attention perception not just you know how the eye biologically helps us um get those sensations in but also sort of how we make meaning how we interpret okay and so from this these building blocks we figure out okay how should learning work in this context and we build learning strategies and instruction strategies on top of those building blocks so we're not just starting from scratch and making things up we're basing it on how humans process and figuring out okay theoretically how should this work but we can't just go based on theory we need to test it then in an empirical environment to make sure that it actually works and to learn what the boundary conditions might be when will it work how flexible do we need to be so so we then test these strategies out and we use something that we often call the lab to classroom model but using the term classroom very loosely k through 12 classrooms yes but higher education classrooms medical classrooms the hospital settings really anywhere that people are learning information and actually most of this research is based on adult learning in undergraduate graduate school medical school residents continuing professional development it's actually a lot harder to get into k through 12 classrooms and so much of this work is based on adult learning and it's a harder self actually when we're in the age of 12 class term than it is here um so classroom really most of the time means adult learning in this context but we we start with basic laboratory work cognitive psychology has a bit of a bad reputation for doing too much basic science and to some extent that's true we start in very controlled artificial situations where we have typically it's an undergraduate student in the lab learning either lists of words or even nonsense syllables why would we do that because we really want to make sure that we have a lot of control over the experiment to determine those cause and effect relationships we have you know the random assignment we've controlled a lot of aspects so that we can isolate what is affecting learning how where can we be flexible where can't we be flexible okay so we've got there's a whole band of undergraduate students who have learned swahili english word pairs in these playoffs across across the world because a lot of our students don't know squeaky we don't want them to already know what we're trying to teach them so that we can isolate these events but we can't just you know teach students some nonsense syllables say hey that works so let's let's put it in the classroom let's throw it in the hospital we can't make that jump it's too big this is important but it's just a start we then move into the applied laboratory context this is where we're still in the lab but we're using relevant materials like portions of a textbook video lectures something along those lines it's in the lab but what the lab looks like is basically a computer so it does look like how the learner might independently study for some sort of upcoming exam but we just control it a little bit we probably have less control than we might like you can tell them to put [Music] and we make sure that the research still holds when we move into more complex learning materials and then from there we actually go into the applied context so here we're in the real learning environment with their relevant materials being instructors altering their instruction whether that be a team member working with a group of residents in a hospital whether that be you know manipulating something about the grand round sessions that residents or continuing professional development when you get your cme credits there's research showing you know manipulating in that context or you know an undergraduate or medical school classroom those types of things and we make sure that it works in those live settings we don't move linearly um we can move back at any one point you might have research going on in a lot of these different levels but this is essentially how we test the strategies to make sure that you know what we're pretty confident that these are going to help students learn and the strategies i'll be talking about today have been tested pretty rigorously through this model to really determine the cause and effect what is the thing that's causing learning and also where does it work how can we flexibly apply it in a lot of different situations and the flexibility comes from being able to determine what is actually causing learning and what is what is something that you can change based on your context okay so we're going to talk about the two strategies and um the color contrast on this screen is a little bit different than that computer screen so if you're having trouble distinguishing the colors there i will make sure that these lights are available i know this is being recorded i'll talk you through it i promise um so don't don't stress you can't read you're not necessarily supposed to read um listening to me is better but we're going to talk about two strategies based practice and retrieval practice and these two are the ones that have the most evidence to support their effectiveness these are not new they've been studied for a very long time ebbinghaus discovered space practice in the late 1800s and the earliest paper i've been able to find on retrieval practices back to 1909 so we're talking about century long work showing that these work in a lot of contexts they have the largest effects they're also extremely applicable in a wide variety of situations you're probably already using them some and so that's why i'm going to be talking about leveraging these um today so let's start with space practice space practice is all about kind of the when when to study when to go over material and when we get to retrieval we'll be talking more about the what we'll start with when so space practice is exactly what it sounds like okay taking a session that you might use for learning maybe that if you know students might cram for an exam right before you know the night before boards or they might um you know residents who are you know trying to cram all of their their cardiology information in during their their car their cardio unit or whatever right um taking that and we're not talking about more time we're talking about taking the time and splitting it up so that they are revisiting in shorter intervals spread out over time and ideally there is sleep in between these intervals but if you're dealing with a situation where you have a day of professional learning like a continuing professional development day you don't have sleep in between if it's just one long marathon day and so you know at least breaking up the session and you know kind of spreading things out switching up what you're talking about and then revisiting with some space would be better than would be better than nothing and now with a lot of virtual i think we're learning how to spread things out over multiple days even when there's distance involved when there used to have been travel um anyway so spreading it out that's the idea okay so why does spacing work i have a demonstration that i think will help illustrate uh will help illustrate this and those of you who praise a couple of you that were at the morning session you probably i don't know if you remember the answers or if you're gonna have to solve again but you guys will get kind of an extra little benefit from this so i want you to solve i'm going to give you two problems to solve in your head i'm not going to ask you for the answer but please play along play my game with me okay solve these in your head here's the first problem 37 plus 15 plus 12. just kind of nod when you're there okay so we're getting there so when you solve this problem if you're anything like me 37 47 plus five okay 52 62 6264 you go through this process of solving the problem and that processing is analogous to the kinds of processing that improves learning you go through the whole process okay so now next problem tell me when you know the answer did anyone solve the problem again no you can't even really do it because you know the answer 64. it's because we masked them we crammed them and so the second time through it feels really easy you might think wow i'm really good at adding numbers together i know the answer i'm getting good at this right you don't go through the same processing that you did this is what happens when we cram you don't go through the same good learning processes that actually make things stick that make it durable you just sort of remember the answer it's counter-intuitive but you actually need to forget a little bit in order to really remember in the long run to make it durable to make it last over time obviously you don't want to forget too much a year is probably too long right and we get that in these contexts when you're you know you're an anatomy one and then you go back to you know anatomy two or whatever you're connecting especially with pre-med when they're taking these different courses and their space in between if they don't revisit they've forgotten too much so what's the optimal spacing you kind of have to play with it but you want to forget a little bit but not as if you have no clue if in a week i asked you to solve these problems you probably wouldn't just remember the answer you would just solve it those of you that had it this morning did you remember the answer or did you just solve it some of you are kind of like i think i remembered it some of you may be solved and that's just a few hours right so sleep would be better a little bit you know a day a couple days a week you gotta play with it okay spacing really is an all-arounder it's been tested in so many different domains it works across the board so it does work for vocabulary learning you can learn swahili this way but also you know anatomy terms other things that you might just need to memorize there is some stuff that we do need to memorize fact learning again when my sister is a medical resident she's a third year um in obgyn there's just stuff she needs to know that's that's true even though i mean sometimes there's a kind of a negative thing about learning facts but some we do need to learn facts in order to be able to apply them but it doesn't it's not just for that it works for learning content within text passages even making inferences it works for problem solving it works for musical instrument learning motor skills like dancing or playing tennis specifically it's been shown to work for surgicals learning surgical skills and in medical and health cpd settings so that paper there then hoof at all connecticut in their in their medical division um i'm the cognitive psychologist on the papers and there's a neuroscientist as well christopher madden and we've collaborated um to basically go in and pull studies looking at space practice in cpd there's a fair amount of it i'll show you the reference of the um so it works in a lot of domains okay i promised you some evidence so here's an example of an experiment that was conducted um not by myself but by rossen and kinch in this applied lab setting but it's a really nice demonstration so in this experiment the adult learners are studying a lengthy science text okay probably about how lightning storms develop or something like that okay and in the single condition we're basically just getting a baseline there i just when i say the end in the field i it's not my study but i'm just going to keep saying we so i just want to acknowledge that so they what they do is they have them read maybe i don't know five ten minutes depending on how long this test is i think i think it's probably five minutes in this context and then in the masked condition they really just have them read and then reread right in a row so instead of reading for five minutes they read for ten minutes so they're just cramming they're repeatedly reading this is what learners often report doing in order to study they reread their notes they reread the book sometimes they highlight or they copy things out of a book and then they reread that if they have some sort of study guide or set of questions they often look for the answers in the book write them down and then they re-read the guide this is what they love to do and this kind of gives a hint as to why they might do that in the space condition we take that initial read and just back it up by a week so now they're getting us a weak space in between they read for five minutes they leave they come back a week later and read for five more minutes and then at that the reread for the single it's just the first read from that point they are then tested in one of two different conditions some get an immediate test in a five minute delay others have two days and then they take an assessment test to see how much they've learned from all of this reading and rereading okay especially six different conditions of all different groups of adult learners it's a between subjects design okay here's what we find immediately the massing group wins in the very very very short term cramming actually does work and this is probably why students do it all the time because in the moment they feel like they're learning a lot and if they're getting the test right then they remember it but then as soon as they are done with that test basically they forget it all and that is pretty much what happens when we cram or mass like this even just two days later the masked group has now dropped down to the same level as the baseline group okay and they it's like that extra time was wasted whereas the spacing group they have slowed their rate of forgetting essentially over time okay there's some air there's these are between subjects design so a lot of people will point out like does the space group get a little bit better not a significant difference basically what spacing does is slows slows the forgetting curve you forget a little bit but then you remind yourself and that helps you kind of maintain it over time especially when there's sleep involved when we're talking about more more than 24 hours time so spacing works in the long run this is what makes it a little bit of a tough sell i think for learners because cramming does work and probably when many of the individuals entering the medical profession when they're pre-med in college they're they're probably gifted students they are they might be a big fish in a small pond or maybe even a big fish in a big pond it's not as difficult and so they can cram and they can do well on their tests and they have enough time to relearn it all before the next test they can get away with it as you know medical education it's just a vast vast vast amount of information that you need to know and at a very detailed level and so it is very difficult to continue with this cramming and if you then forget it all and you have to kind of relearn it much later you just can't get away with it anymore but for many students they've experienced cramming massing like this as working and so this is i think what makes it a tough sell but of course they care about learning in the long run they don't just want to pass the test right now they need to then pass the next one and the next one and then the board and then do well when they're actually seeing patients right um who are doing research or whatever it is that they're doing and so it's really important that they maintain this over time and that's why spacing is really essential in terms of applications is a little tricky to see but in terms of application essentially what this is showing is just that spreading things out over time when you are giving instruction wherever that might be is ideal and in theory you should be trying to get the learners to remember things from yesterday from a week ago from a month ago and so on i would not worry about the exact amount of space because what we find with the research is that spacing at all has a large effect on learning but when we start getting into the minute details was one day better than two days is one week better than two weeks or what you know where's the perfect timing it turns out that it's really really context dependent and it's even different for different individuals and when we you know even if you optimize it we're talking about small differences when we're talking about spacing one week versus two weeks really i would just say do some spacing go with whatever practically works given your context if a week works really nicely you see this group of residents you know for two weeks total and we can kind of go over some of the stuff from last week at the end of the second week fine that is going to be great helping them to just revisit that information and don't stress too much about exactly how much time if you can get sleep in there in between great if you can't some spacing is still going to be better than not okay and then for learners this is where you can try to get them to commit to just a certain number of minutes every day even if it's just 20 minutes or 10 minutes just to quick review something and to kind of mix up the topics that they're going over so not just focusing on you know the one rotation that they're in right now or just focusing on the specific course that they have an exam coming in if you can get them to keep cycling old stuff in it does take some time but it's more efficient in the long run so they can commit to short amounts of time and make sure that they're thinking about things of these different lags and also across topics so that they're forgetting everything just a little bit and then refreshing it and using it and that's actually going to help them make connections between the different ideas as well okay okay so you just need to keep different okay and then if you are in charge of curriculum in some way if you if you have sort of a calendar and you're going to be with a group across a number of sessions so s1 s2 s3 just sort of think of those as sessions whether that's you know a given number of days um in a couple weeks span or if that's you know maybe you're seeing them once a week for a number of months whatever it is you can teach a principal or teach a concept and then just take the practice you would normally have them do whether it's you know working in a skills lab or practicing under my sister has one of those like practice at home skill labs where she practices the laparoscopic stuff on the tv screen um you know they're doing that sort of thing at home or whatever practicing exam questions for an upcoming board whatever it is whatever practice just delay it a little bit so you don't have to reinvent your curriculum just take what you would have had them do right away and delay it by the week oftentimes learners are confused by this so when i see undergraduate students mostly i do master students but i teach undergrads and when i give them an assignment with questions from old stuff they're like oh you must you must have messed it up dr samarachi we talked about that last week it's still important this week i know i'm hoping you remember it from last week here's the assignment did you get confused because they're very used to learning the thing doing the practice for the thing and then moving on all you have to do is just put a little bit of a delay you don't have to reinvent the wheel some of the things that are taught early on especially if you are doing like once a week across a number of weeks or months some of the stuff that they learn in the beginning will kind of fade away as time goes on and that's where inserting maybe a space to learning session that's what sl1 sl2 is just revisiting if this if these ten sessions are in the span of two weeks you might not need to do that but if it's across a number of weeks or months maybe revisiting some of those things in the medical context though i really think that often they're using the information and they're they're they're using it on the fly and if you really think about it life comes at you in a spaced way so it's not like you can say and you know if you work in the er it's not as though you can say okay today i'm only going to see patients and diagnose this one particular heart problem and then we'll do you know we'll do a different kind of problem next week that's not how it works it's all intermixed and interleaved which produces spacing and actually interleaving is another strategy we don't have time to go into it much today but this idea of jumbling all of it up and alternating between examples also really helps which spacing sort of produces anyway so this is one way of thinking about it if you are designing or kind of spacing out a specific curriculum where you're teaching and then they're all practicing so that's space practice that's the when so when you're giving them practice to do what should they be doing what are the types of things that you can do when you're trying to space it out that's particularly helpful it's not just reading retrieval practice is extremely effective and retrieval practice really is just thinking back and bringing information to mind technically from your long-term memory but we actually think about short-term and long-term memory incorrectly in sort of everyday society if you think back and try to remember where um where sherry said that i work what university or what institution that was uh 29 minutes ago 20 28 27 minutes ago that's well into your long-term memory unless you've been sitting there the whole time thinking it in case i might ask you but you i hope you haven't because it's not useful information but if i ask you to think back to you know where do i work if you can bring it to mind you've retrieved it if you can't remember you've technically quote unquote failed to retreat but i don't like using the term fail because actually by trying to retrieve it and not appreciating it when i say it now you're actually much more likely to remember it in the future even though it's basically useless information for you but i i teach at rhode island college so if you remembered that you retrieved it and you're now more likely to remember where i work i am useless um and if you didn't now you're like oh yeah and you're also a little more i remember where i work don't worry about remembering it but it's just a good example okay so that's thinking back and retrieval practice works in a few ways some of them a little more obvious than others we're going to go through a few so retrieval practice has some indirect effects on learning what that means is that retrieval produces something and then that something produces more learning okay so you might hear this as a mediated effect okay so for example retrieval practice gives you feedback on what you know and what you don't know and that can allow you to allocate study time if you're a learner more effectively or efficiently you can continue to revisit the things you know well but really focus on the things that you don't know as well and retrieval practice is really good at identifying what you know and what you don't know it's difficult to do that when the information is right in front of you because it all feels really fluent and familiar system working the formative assessment context so if you are the instructor you can identify what the learners are doing well with what they need more practice on and adjust based on their performance all of this can lead to just more efficiency and more efficiency might mean more learning is happening overall because we all only have 24 hours in a day and we are supposed to sleep and eat and stuff right so um more efficiency could be more learning this is all great indirect effects are wonderful in addition there actually is a direct effect on learning the act of bringing information to mind helps you remember it better and it's not just about memorization it makes you more able to apply it in new context you are more flexible with it it's more durable and you don't as an applied person have to pick you don't have to say like okay well i'll do retrieval practice for the direct effect today and i guess we'll do it for the indirect effect tomorrow it all piles on if you are giving feedback and doing you know formative assessments seeing how how the learners are doing or if the learners are getting their own feedback they get the indirect effect benefits along with the direct in the lab we can use it apart because you want to know you know what are the different effects we want to control all of the different aspects but but in practical settings it all just piles on to make this an extremely effective learning strategy so i'm going to show you an example of evidence this is a direct effect example so probably these learners would have learned more if there was also feedback but this will just demonstrate that it does have a direct effect and then in addition there's another reason i want to show this one to you so we'll get to that in a second so in this experiment uh this is 2006 it's one of the kind of classic papers now on retrieval that kind of revived the work in this area um and so what they did these were actually both my advisors one of them was my master's advisor the other was my phd right there so what they did is they had learners these were undergraduate students at washington university so they probably all thought they were going to be a doctor lawyer or an entrepreneur of some sort because that's that's what she said and so they read just a text of information they probably read it for about five minutes and then one group just kept reading and rereading and rereading okay just repeatedly read this thing what they say that they do all the time to study another group where they were given a blank sheet of paper and they had to try to retreat write down what you can remember and then they take the blank sheet of paper away give them a new blank sheet of paper and say do it again and they're kind of like one why write it yep rewrite it repeatedly retrieve it they do that three times then they all make predictions about how well they think they'll do on a test in one week and this is what learners are often doing when they're studying or reviewing how well am i going to remember this when i need it we're trying to make a judge a postdictionary or i'm sorry a prediction um of what's going to happen later on okay based on kind of their post dictions of how they've been doing so far now the re-read group they're pretty sick of the text the group that had to retrieve they've been um they've been shown some harsh realities about what they can and cannot remember when that thing has been taken away from them really they only are retrieving in that retrieval condition about 55-ish percent i don't remember the exact number but it's in there so they're actually going to disadvantage a little bit because they're not getting to re-review everything they would have learned even more if they inserted through study but that would be an indirect effect they make the predictions that they actually do come back in one week and take a learning assessment test just to see how much they actually learned and remember a week later okay so let's look at the data predictive performance is on the left and what we see is the repeated reading group thinks they're going to do better than the group that read and practice retrieval probably because the practicing retrieval group knows they could only remember so much the repeated reading group is like i'm so sick of this thing you've been having me read it and probably 20 30 minutes at that point they've reread it four times they think they're never gonna forget it it's very familiar and fluid this is probably why students repeatedly read it feels really good in the moment and you think you'll do better than you actually end up doing retrieval practice group thinks they're not going to do so well and then when we look at performance a week later of course the opposite is true this let's highlight some things if we were judging what works best based on students sort of own intuitions and assessments of their own learning we'd be wrong there are plenty of students at least that come into my office and say i don't understand why i failed the test i studied for hours i reread everything you gave to me i usually say no because i gave you a i gave you a chapter on youtube did you read that anyway um they are the ones that feel like they should have done better and they don't understand okay and maybe they've been rereading and cramming and they just hit a point where there's too much information to try to remember they can't reread everything and it's not sticking right the retrieval practice group actually does a little bit better than they think they're going to i've seen that too where they come in and they're nervous and they they they're the ones that were like i don't know you told me to do this retrieval thing and i'm doing it because i believe you but it doesn't feel well i don't know and then they do much better than they thought they did retrieval practice is like spacing in that it works really well in the long term and long it doesn't even have to be that long a one-day delay is enough to do it and the effects maintain over time spacing and retrieval both really just make you flexible and it makes the learning more durable then as you space and you leave you can make connections it's just it's a really good it's a golden ticket putting these two together but it is again a hard shell because you have to convince students that their predictions are wrong and that this thing that feels more difficult is actually very good with that for them and i i do think that many of you are probably using spacing and retrieval in your instruction whatever that environment might look like already the key then this is good news right i didn't come here to tell you you just change everything that you've been doing i don't i don't need you to overhaul medical education entirely the key really is just to take these things that you're probably already doing and become a little bit more aware of them and make the learners a little bit more aware of them help them to be to learn that hey this is a really this is a really good thing and it feels difficult but ignore your intuitions follow the science and and and go [Music] and does that have any impact it's one thing to read a couple paragraphs or a page nothing to read like a chapter or something like that yes you're actually asking about the length of the text and in this particular study it was probably like 500 800 words so you're writing not a whole lot um however it has been done with longer chapters it's also been done like literally in medical education to look to see how they're learning the content that they need to learn in their in their textbooks yeah and it's been it's been done i know in continuing professional development as well where cpe planners will actually have manipulated you know retrieval within that one idea sort of send text messages to get the physicians over time to kind of keep retrieving with the spacing um so it works in a lot of context um that's a it's a good question and there is a limit to how much we can have them try to learn within an experimental session um but we know that you know they can help them learn a lot of information it does take more time to learn more information but it's more efficient to do it this way and so there's probably some tipping point i'm sure yeah yeah thank you that's great i don't mind answering questions on the fly not at all this is really fun it's my first in-person talk in a very long time and i love it okay so some applications some things to keep in mind uh one thing is that retrieval really can be verbal or visual so this used to be called the testing effect we don't use that term much anymore because number one retrieval doesn't just happen during tests and it and it does it makes people think oh well i have to give a test or quiz you don't have to give a test a quiz you can have them write on a blank sheet of paper you can have them answer questions sure you can draw you can have them create a concept now but it's got to be from their memory not you know the book is open and making a map they can talk to one another when you have them think pair share that is probably retrieval practice if they're describing and explaining to one another when you just ask them informal questions on the fly during rounds that's retrieval practice anything where you're bringing information to mind so it can be verbal it can be visual it can be verbal in terms of speaking um so the testing effect really is just a very narrow example of what this is we also have to get away from the word test because i mean i could give a whole talk about how we feel about tests in our our country and this is not just our country society kind of a school kind of why that might be some problems with that but i mean anyway that's a whole separate discussion i know that when i'm talking to medical educators whether or not you agree with it you just know that there's going to be tests so um right so um but but it can be all kinds of things it doesn't just have to be a test and if in your context it would be kind of silly to give a quiz and then give a grade like what what does that even mean you don't have to actually low stakes learning opportunities where you're not grading or giving points or even scoring it necessarily you're just letting them retrieve and then giving them some feedback that actually might work better than where there's some stakes attached to it and they're earning points or they might be embarrassed if they don't do as well the low stakes opportunities actually help kind of relieve some of the test anxiety and helps them feel more prepared so it does not have to be a quiz or a test they can use flash cards um the one thing you want to be careful of is to make sure the learners don't just flip the card so at least what i see my students do a lot with flash cards is they look at the terms it's always terms and definitions which is another issue sure yes learn the terms and definitions but you also need to know what it means but they'll look at and we'll be like yep and then they move on that's just repeated reading with like a card right they need to actually retrieve and programs that a lot of medical students and residents use like anki um and others like that they do force them to retrieve a little bit they also implement spacing into those programs a lot and so you can encourage them the learners to use that just remember it's not just about the terms and definitions right there certainly are facts that we need to learn this can be great for memorizing but they could also just have a concept on the card and then maybe they have to come up with an example of it or tell them grab two cards at random and come up with ways that those two cards are similar and different sometimes the things are very different and coming up with a similarity is difficult sometimes the two things are very similar and it's the subtle discrimination that's important so randomly doing that can be a really effective way did i just go out okay that's all right doing that can be a really effective way of of utilizing flash cards or these flash card apps okay um also remember that it can be used with complex materials so if it's fine to remember facts better to be able to use those in new situations and think of more complex information retrieval works very well in those contexts and then just remember that this is a difficult thing to do retrieval practice is very hard and that's probably why it doesn't feel like we're learning as much that difficulty is really good so again it can be a tough sell but showing the students the the learners the evidence and and making sure that they understand why we're doing these things and trying to convince them to do it can be can be really helpful especially when you have a group of learners that believes in science which is a good thing i hope most of the the medical um the medical people are sort of done with the evidence a lot of the time lots of different formats you can get really creative right so having them create concept maps from their memory is a way of trying to get them to understand relationships and dissimilarities among ideas even just having them talks of journal clubs where they might read something and then come back and have a person try to explain they ask each other questions i know my sister in residency does does that occasionally where they they read and then they get together talk about a specific topic right those can all be good ways of implementing retrieval and making sure that they know why we're doing this and helping them to see kind of the importance of it i think is is a big deal because it it really it helps them understand what they should be doing on their own and kind of justifies why we're doing the things that we're doing right okay putting spacing and retrieval together is sort of the golden ticket and one very easy way to do this just sort of off the fly really is to have them answer questions whether this is a quiz that you just give to them or it's just you make up questions and it doesn't have to be the same questions for everybody necessarily right it doesn't have to be a quiz doesn't have to mean that it's standardized so you know if you saw a certain type set of patients um when you were on rounds with the residents in one week maybe the next week you asked them questions about some of the things that they saw or the things that they didn't happen to see but might have been good to have seen it's just that's not what was in the hospital that week that they were there right so um if you if you want to kind of formalize it a little bit you could ask a couple of questions about things that you saw from the last session whether that's yesterday a few days ago or whatever something from last week something from last month and then making a link but it doesn't even have to be this formal and i like this i don't always come up with all of my questions in advance and and give my students you know specific multiple choice items i sometimes just make it up and say hey tell me about x get them to describe and explain it to even to each other this can be done very informally it's it's it's effect it's quite effective and giving them practice retrieving in a low stakes environment is important um and and i just want i i mentioned that my sister is a resident now she used space retrieval a lot when she was in medical school um and so this is her um this is her in her room she she lived in our parents basement during medical school because she went to roslyn franklin and we grew up in that area and so the way she did some of this is really just to try to save time she took dry erase markers and drew those curves i learned it has to do with lung volume in the first session but she drew those curves on her mirror and then i think there was a study schedule and some prompts on that paper there that you see and she would just while she was getting ready in the morning to go to school or practical or whatever at night would just sort of go over those things and then sometimes she'd on a different part of the mirror or you know a different day she would erase and just have props to try to just retrieve what she could remember and this is how she fit in some extra spaced retrieval just on her own so the idea of committing to x number of minutes per day this is how she fit it in with very little time to spare and residents can do this too i know she talks about spending a little bit of time answering maybe three questions from like a pro law i think she calls it a prologue book maybe that's just what she has um in ob gyn but you know the questions that they are practicing with to prepare for any sort of board or certification exam i know that they have practice books and things answer just a couple questions a day they don't have a lot of time but if they if they do that or they even have you know a spare minute to talk to one another about these concepts that can be really helpful and ways to kind of sneak it in time is always an issue but this can be quite hopeful okay so what i want to do is i do want to give you guys a few minutes to think um on your own i know that you're not going to redesign everything i don't want you to redesign everything anyway but you're not going to have time to do that um but just try to think and maybe jot down if you have paper or if not you can you can think about it current examples of spacing and retrieval practice that you're already using within your instruction within your program the learners that you work with or for yourself what are some ways that you're already using this and i'll give you just a minute or two perfect this is rb here i'm one of the attendings in r3 rehab building one technique that i have used is um i don't know if it fits the retrieval program but i teach a part of it and then ask them to look or go into the details and then come back and say or discuss more about it in a week or so yeah perfect yeah so that let's let's share some ideas that you might have um and the the goal people can share in the chat they can share the microphone they can share by by speaking all the different ways are fine um but you know thinking about short-term and long-term implementation so what you're already doing can you do a little more of it can you do a little more of it if you have more time are there challenges are there resources you might need these are the types of things to think about so there go ahead sherry so robert how do you best use these tools when you have minimal time allotted with learners for example in sin and you answered this question during the first session yeah so the tough time always comes up there's just never enough time especially in in medical education at all levels there's never enough time i think the key thing to remember is that it does not have you don't have to insert hours of this every week the little bits that you can do will have a large impact so i'm all about small changes that can have a large impact if you can insert even just a little bit of it especially early as early as possible it will make it everything more efficient in the long run so if you can even just ask them a couple of questions if you're teaching a class in a lecture context or a classroom or in in the sims lab you might have them coming and and working with you on a schedule if you can even just ask them a couple of questions like hey get together with a partner talk about what we talked about last week just for like a minute then they're more likely to remember that and you won't have to go oh they won't have to spend as much time studying it later they won't have to you won't have to go over it again in a year when when they um you know they inevitably forget it right it keeps things fresh for longer and keeps them more flexible it should help them in the long run but these don't have to be very long especially if you're in a context where having them sit down and write everything out isn't going to work you can encourage them to do that at home too and then you can also infuse just very quick it's much faster just to verbally say hey talk about this for two minutes than it is to formally write questions and then have them formally answer the questions and then you have to score it or you have to have them score it i mean you don't have to do all of those things some level of feedback is helpful but that could also just be hey here's how i would have answered this question so i actually and again i know that this is under for me it's undergraduate students but i think this would work in a lot of different contexts i have my students now that we're doing more online learning i have them do they watch lecture videos um that i've pre-recorded and then during our synchronous time together i put them in small groups and give them prompts usually it's like three prompts and they in their groups talk in breakout rooms they talk and they go over those prompts together i tell them to put their stuff away i have sort of a captive audience whether they listen to me or not is sort of a separate issue but they're adults they should be listening um to what i'm instructing and and for medical students and residents that they want to succeed so i think if you say this is going to be really good for you here's why put it away that they will listen to you um and if they don't they're they're adults um so you know talking to one another they do that for a pretty short period of time and then when we come back i try to get them to share because we have an hour together but if we're short on time i say here's how i would answer this prompt and i say what i say is not word for word what you should have i'm just talking through it for you here's how i would have answered it and i spent maybe two minutes just sort of describing and i say if you had all of that great if you didn't have all of that that's fine and that will help them a lot because they've tried to retrieve they're not going to retrieve everything but we know that's okay the things that they didn't remember they're going to learn it better for me saying it now because they just tried to retrieve so it's called test potentiated learning essentially when you try to retrieve and then you re-read or re-hear it you learn more from that re-exposure than you would if you just had re-exposure without retrieval prior so it really does kind of even make the feedback and discussion more effective so i know time is always the thing that comes up it doesn't matter who i'm talking to if it's the state department medical education um you know kindergarten teachers it's always time always and so um do the best you can do the best you can and eventually it will become more efficient but they i mean unless someone figures out how to give us more hours in the day time is just always going to be an issue but the more durable all of this is the better off they'll be especially in the long run good chair somebody else says in my neuroscience didactic session the series with the trainees we watch a video practice drawing the circuit together and then in session two we start by reviewing and drawing the circuit together before progressing the session two is spaced out from session one exactly yep so that's great a great example of spacing and if they're drawing some of that stuff from memory that's actually retrieval practice as well this is being used a lot some some of you may have even maybe you read make it stick um or i wrote a book and had it illustrated with a colleague called understanding how we learn and so that that book is all about kind of these principles and putting them together so maybe you already knew about some of this um i really think that it's helpful to try to teach the learners about it as well so that they can be more intentional and to help them recognize that when they are struggling that that's okay um i've done some research it was actually with fourth graders but it's not like fourth graders versus adults they're not a different species even though sometimes they it might seem like it um but essentially fourth graders are really struggling when they're making that transition to from learning to read to reading to learn and so they really do struggle with the blank sheet of paper and honestly when you've got really long like whole chapters a blank sheet of paper is you cannot recall the book on a blank sheet of paper it's not going to happen you could try to break it up into chunks and try to recall as much as you can remember but maybe that's not the most effective way to go instead if it's too much and too difficult and you're ending up just staring at the paper or only remembering a very small amount scaffolding the retrieval opportunities can help a lot what we did with the fourth graders was gave them little prompts to remember portions of the text and then they did much better and we were able to work them up to being able to retrieve the most important concepts as opposed to like early on when the kids were like the text was about stars and that was it i was like yes anything else and they're like nope um nope can i go to the bathroom so um you know working them up in a larger like a medical context it might even be telling them to look at the headings of the text but cover up the actual text and try to just produce what they remember from that section what were the most important concepts can they explain it in their own words or having them generate a list of concepts or maybe you have them you generate a list and so here are the things i i know that can start to look like a study guide and then study guides get weird because then it's like well i'm am i telling you what's on the test and when we're talking about board exams you can't actually tell them because you don't know everything that they might or you know but it's you're basically like everything everything will be on the test right but if you give them a list of concepts that are particularly important and can they describe and explain the similarities and differences those types of things that can be helpful for them as well if it's a lot more information any more questions either online i'm not looking at the chat online now so you don't have to verbalize it um either online or in in the room we have about one minute uh dr polly um so we do modules outpatient topics and i put the learning objectives up at the beginning and then we do our case base and then i put those up again at the end and say this is what i was hoping we learned and then i asked them what did you learn and we go around the room and i think that would be retrievable yeah and then they were able to learn from each other what one may say that the other didn't revise and also helps me to think about which points might have been emphasized enough right and then if i had something in mind that i was hoping they learned and no one brings it up then i can revise that for the next time yes do it in the clinical setting too after rounds what did we learn today and i find that absolutely i mean we do that with our kids right okay what did you learn in school today that's actually retrieval practice right if they they describe it you don't have to know what they're saying if you don't understand physics that's fine they can explain it to you right so yeah that's really good the one thing i might add when you're in the group setting it's great if they all retreat it's certainly better than nothing if it's possible to have them jot it down first before having them say it the only reason is oftentimes when we retrieve in a group context it disrupts essentially it disrupts our retrieval structure so if i am trying to think of something in sherry let's say i'm thinking about point d and sherry jumps in and starts talking about a it kind of cuts me off from thinking about point d and actually maybe some of those things that you thought weren't emphasized enough this they just didn't really think about it right um i can think of examples where the first person who shares an example kind of drives the theme of all the other examples and so if they're able to jot down first and then start sharing that will help kind of diversify the types of things that are said and will help them kind of use their own structure but if there's not time for that or maybe there's not paper for that you know if you're like running through the hospital or i mean no you're not gonna have them stop and write down like everyone grab a clipboard in the middle of the it's not going to work don't worry about it that in that context but if it's possible having them jot it down can be can be really helpful but not necessary thank you um if you want to provide a little bit of spacing and then revisit this this session has been recorded it will be up on the teach website in a few days so please do revisit it encourage your colleagues to revisit it as a reminder 5 o'clock this evening online we will have our poster recognition and awards ceremony please attend that we will have lots of great information scholarship information from our colleagues our educator colleagues and yourselves um this evening and the awards will be presented as well so please if if somebody didn't register and you know that they didn't register just go right on the teach website and um there will be links there to attend online so looking forward to seeing you all there you can all feel free now to grab your lunches those of you who are in person grab your lunches i did also promise i'd give you some resources at our website no it's okay so at learning scientists within us dot org there's a lot of free resources there and then these are three in particular for medical so we wrote a chapter it's chapter one coming out in the new edition of an introduction to medical teaching we just looked at the proofs earlier this month so i'm guessing that's going to be available soon and it talks about these strategies in medical context and then two papers first author is then both published 2021 in uh the journal of continuing education in the health professions one talks about spacing the other retrieval practice in that cpd conte um domain basically so thank you very much for your for your extra time i know i went a couple of minutes over