When Delayed Feedback Is Superior To Immediate Feedback

Craig Roberts, writing in EdSurge:

Beginning in the 1960s psychologists began to find that delaying feedback could improve learning. An early lab experiment involved 3rd graders performing a task we can all remember doing: memorizing state capitols. The students were shown a state, and two possible capitols. One group was given feedback immediately after answering; the other group after a 10 second delay. When all students were tested a week later, those who received delayed feedback had the highest scores.

Will Thalheimer has a useful review of the literature, beginning on page 14. One might object that whether immediate or delayed feedback is more effective turns on the goals of the study and the design of the experiment.

To which I’d respond, yes, exactly!

Feedback is complicated, but to hear 99% of edtech companies talk, it’s simple. To them, the virtues of immediate feedback are received wisdom. The more immediate the better! Make the feedback immediater!

Dan’s Corollary to Begle’s Second Law applies. If someone says it’s simple, they’re selling you something.

About 
I'm Dan and this is my blog. I'm a former high school math teacher and current head of teaching at Desmos. More here.

16 Comments

  1. First, I have to say I get quite a kick out of how such an anti-edtech blog produces such a useful dialogue for edtech. Sincere kudos!

    As for this post, wait, we are assessing feedback on math problem-solving by looking at the memorization of state capitols? Does that work?

    Even if it does, Dr. Roberts soon adds: “Most early attempts to replicate this delay-of-feedback benefit in schools had flawed designs and largely failed. ” Oh. OK.

    And wait again: delayed feedback means ten seconds? If you want ten seconds you need edtech. It is human scoring that takes an hour or a day.

    The investigator who used our app for her study requested we implement a delayed mode, but the delay was to hold back on step-wise feedback and just offer feedback when the student indicated they had reached the answer. ie, even if research did suggest these kinds of delays were helpful it would be easy for edtech to execute (and you would need edtech to even identify the suitable timing).

    Speaking of timing, let’s look at Thalheimer’s summary: “When should feedback be given, immediately or after a delay? A tremendous amount of ink has been spilled in attempts to answer this question, but a lot of what has been
    communicated is smeared and blotched in confusion, and remains so, as we will see. ”

    That is a great paper, btw, thanks. I like Guthrie(71) finding 474% improvement when feedback is given for incorrect answers, and our target is strugglers. Feel free to cherry-pick a different stat.

    Now it gets downright silly. Thalheimer mentions a study in which “feedback” did not work, but their idea of feedback is to give those students the option of seeing the answer before responding! And they all did. Uh, yeah, not a lot of productive struggle there.

    The big take-away for me is that there is little hope for educational research, and even less for its being applied well.

    Except for that 474% benefit of feedback, of course.

  2. Dan, with all my best intentions (not hurt you): you are in a tech company which, I think, gives immediate feedback. Why do you promote the non-immediate feedback? Is it coherent? Will you incorporate the delayed feedback in the Desmos framework?

    On the other hand, how would you implement this into classroom? How could we offer delayed-feedback? With homework?

    Can you clarify this points please. I’m lost here.

    And please, do not get it as a critique.

  3. 10 seconds delay. That’s the delay used in Face-Off and other “competition” programs between “Now, who is going home today?” (failure), and the giving of the name of the unlucky one.
    Very popular!!!!!
    Seriously though, the idea that it is even possible to give useful feedback by computer in a “working alone” situation is a bit of a pipe dream, especially with all the optimistic chatter about “adaptive”. I am a pessimist on this one.

    Scenario of the future:
    Student types something
    Machine gives feedback (immediately, or after 10 seconds.
    Student does nothing.
    Machine analyses student face and sees puzzled look.
    Machine offers alternative feedback.
    Student still does nothing.
    Machine gets fed up and writes “Dumb-ass, go on to next problem”
    Student hits computer.

  4. Dan,
    Thank you for posting about this article. When I first read it, I had the same reaction. Not only is a complicated matter over-simplified, but also it takes the students’ efforts and agency out of the equation:
    “She scored 94 percent, her first A on an exam! By simply delaying the return of Emma’s homework by a week, I’ve employed the learning sciences to increase Emma’s chances of success.”
    The teacher takes the credit for the student’s personal achievement.

  5. There are a lot of interesting other assumptions like you’ve done it 4 times today, so that you will remember it next week and forever. Yet, students oftentimes forget skills and concepts by later on in the school year and need them to learn other skills and concepts.

  6. Xavier:

    You are in a tech company which, I think, gives immediate feedback. Why do you promote the non-immediate feedback? Is it coherent? Will you incorporate the delayed feedback in the Desmos framework?

    At this point the only immediate feedback we offer is when we re-draw your function immediately after you type it, or change a slider value. That isn’t to say we won’t ever offer immediate feedback in our Activity Builder or that I’d be a hypocrite if we did. This post is simply to say the prevailing wisdom in math edtech is simpleminded about a complicated matter. And, unsurprisingly, the people who have rushed to inform me that the matter is that simple, are selling something.

  7. In general I’m confused over how a 10 second delay is the difference between immediate and delay.
    When speaking about education, to me the difference that should be studied is immediate (within a few minutes) and delayed (the next day). I would consider waiting 10 seconds to learn of my answer’s correctness to be pretty immediate.
    If we are talking purely about how fast technology reveals answers, that’s a different conversation, and most likely won’t affect things in my classroom. (We have virtually no money for technology)

  8. This reminds me of your thoughts on handwriting recognition. In your ideal case, the student was able to finish solving their problem, and spend a moment with their own thoughts before they were given a correction.

    I see that happening in the classroom when students ask me to check something, and I pause for a moment to compose my comments. They take that time to evaluate their own work. Not so if I just blurt out an error.

    And I forget where I read this, but I saw an article about the negative effects of waiting multiple days to pass back assessments to students, and the effectiveness of returning things the following school day. This seems to be a similar thought process – students need to be given time to evaluate their own work, but not too long that they’ve forgotten their original thought process.

  9. Hey Michael Pershan, we’re talking about feedback over here! :)

    I’m willing to sign up for pretty much everything in Dan’s post: the complexity of feedback, delayed feedback is often better. I also find myself sympathetic to Kenneth’s concerns about making too much of such a small delay with non-math material. Seems to me that these sorts of studies are mostly valuable for us as prompts for theory — so what’s the theory as to why delayed feedback is more helpful for this sort of memorization task?

    As a more general comment, feedback is a mess. It’s an enormous construct, encompassing everything from yes/no to a lengthy one-on-one discussion. The knowledge is also widely distributed across fields, from edu psych to math edu. A lot of the conventional categories that we use to compare feedback (e.g. delay vs. immediate) seem silly to me once we look carefully at practice. [I once wrote a bunch of posts on this theme here.]

    I’m not exactly sure what would clarify how to use feedback in practice. Where I’m at right now is that we could improve our understanding of feedback by committing to a couple of shifts in how we analyze it:

    * I think it’s a mistake to study the *moment* of feedback. The grain-size that would be helpful to study is larger, feedback routines or something like that.
    * The appropriate feedback to give is intimately tied to the particular mathematical goals, and so we should study feedback as it connects to particular areas of content. Feedback for learning to divide fractions might not be the same as feedback for getting better at geometric proof.
    * We need a richer vocabulary of feedback. Comments on a paper or pop-up windows or a one-on-one conference all afford really different possible responses. They shouldn’t all be mushed together and compared together.
    * We should think of feedback as one of any number of possible instructional decisions facing a teacher. Often, when I’m staring at a pile of work, I decide against giving comments and instead talking to the whole-class, or running a different lesson, or whatever.

    I sometimes like reminding myself that “feedback” is a fairly new term in educational discourse, imported in from electrical engineering via psychology in the 20th century. We don’t have to think about feedback all at once, and I think it’s usually unnatural to do so.

  10. Megan:

    In general I’m confused over how a 10 second delay is the difference between immediate and delay. When speaking about education, to me the difference that should be studied is immediate (within a few minutes) and delayed (the next day). I would consider waiting 10 seconds to learn of my answer’s correctness to be pretty immediate.

    Perhaps big categorical variables like “delayed” and “immediate” aren’t helpful here. So let’s define a continuous variable t, the time after a student thinks she has an answer to a question before she gets some feedback on that answer.

    My question is “all other things being equal, what value of t is best for a student’s long-term retention and meta-cognition?”

    The difference between t = 0 and t = 10 seconds, no matter which category we’d want to assign those numbers, immediate or delayed, seems significant in some circumstances.

    If we are talking purely about how fast technology reveals answers, that’s a different conversation

    Agreed, and I think that’s the conversation we should have. I’m not making a distinction between feedback from humans v computers here. Just the timing.

  11. I do see a huge difference in the 0 seconds vs. 10 seconds verbal feedback. At 0 seconds, the focus (from the student’s perspective) is often on “Did you get it right or wrong?”, but 10 seconds may be the difference to prepare students to listen to anything in that gray area, anything outside of whether the answer is “correct”.

  12. These are some really useful descriptions of research on feedback. I winced at several of their descriptions of theory, but I suppose that’s somewhat to be expected as neither is a lab researcher.

  13. Thanks, Michael Pershan, for chiming in.

    Here’s something from Marzano (apparently on p. 5 of Classroom Assessment and Grading that Work, though I don’t own the book…this was cited in a presentation I went to). It shows the effects of different types of feedback on learning:

    a). Just right or wrong -3%
    b). Tell student the correct answer: +8.5%
    c). Make criteria for a corrent answer/response more clear to students: +16%
    d). Explain why student’s answer is right/wrong: +20%
    e). Repeat task until student gets it correct +20%

    I mention this only because I think lots of the online apps Dan is criticizing here essentially use version a) of feedback. Sure, they may attempt to explain why an answer is right/wrong, but if their explanations are too hard for students to understand, then from the student’s perspective they’re just giving right/wrong feedback.

    I’m a little suspicious about e). Repeating ad naseum could be good unless the student has no idea how to get it correct, in which case motivation suffers. I don’t know what research Marzano was summarizing here, so I can’t judge it too much.

    Mainly, I just wanted to point out empirical results that show immediate right/wrong feedback can truly hurt rather than help.