Mathspace is a startup that offers both handwriting recognition and immediate feedback on math exercises. Their handwriting recognition is extremely impressive but their immediate feedback just scares me.
My fear isn’t restricted to Mathspace, of course, which is only one website offering immediate feedback out of many. But Mathspace hosts a demo video on their homepage and I think you should watch it. Then you can come back and tell me my fears are unfounded or tell me how we’re going to fix this.
Here’s the problem in three frames.
First, the student solves the equation and finds x = -48. Mathspace gives the student immediate feedback that her answer is wrong.
The student then changes the sign with Mathspace’s scribble move.
Mathspace then gives the student immediate feedback that her answer is now right.
The student thinks she knows how to solve equations. The teacher’s dashboard says the student knows how to solve equations. But quiz the student just a little bit â€“ as Erlwanger did a student named Benny under similar circumstances forty years ago â€“ and you see just how superficial her knowledge of solving equations really is. She might just be swapping signs because that’s why her answers have been wrong in the past.
Everyone walks away feeling like a winner but everyone is losing and no one knows it. That’s the scary side of immediate feedback.
One possible solution.
When a student pulls a scribble move like that, throw a quick text input that asks, “Why did you change your answer?” The student who is just guessing will say something like, “Because it told me I was right.” Send that text along to the teacher to review. The solution is data that can’t be autograded, data that can’t receive immediate feedback, but better data just the same.
Related Awesome Quote
If you can both listen to children and accept their answers not as things to just be judged right or wrong but as pieces of information which may reveal what the child is thinking you will have taken a giant step towards becoming a master teacher rather than merely a disseminator of information.
I would want to emphasize that the issue is that Mathspace (and tech folks generally) tries to give immediate, “personalized” feedback in a fast, slick, cheap, low/no-labor kind of way. And, not surprising, ends up giving crappy feedback.
Daniel Tu-Hoa, a senior vice president at Mathspace responds:
[T]eachers can see every step a student writes, so they can, as you suggest, then go and ask the student: â€œwhy did you change your answer here?â€ For us, technology isnâ€™t intended to replace the teacher, but to empower teachers by giving them access to better information to inform their teaching.
2014 Sep 4. I’ve illustrated here a false positive â€“ the adaptive system incorrectly thinks the student understands mathematics. Fawn Nguyen illustrates another side of bad feedback: false negatives.