Fantastic ideas here. Looking forward to your further examples. One thing occurs to me: you didn’t show the SMB feedback on the math software, a “bummer sound effect” and being thrust back to the beginning of the problem, forced to run through perhaps dozens of excruciatingly routine steps to get simply get back to the point of error. Curious to see how these two conceptual frameworks actually function when the design elements match up with the contexts. I get it that the idea here is to think of what the interpretive feedback would be in the math software, but in this day and age video games are massively complex, and I’m guessing there are scores of games that have pretty serious evaluative feedback after a FAIL. And, FB that players appreciate, devour, learn from, and completely value. Still, cool to differentiate between the two as possible axes out there in the ecology of learning, playful or otherwise.

I like the evaluative and interpretive labels. Dylan Wiliam quoted others in a succinct paragraph about how this is a problem even separate from software. “Teachers who listen evaluatively to their students’ answers learn only whether their students know what they want them to know. If the students cannot answer correctly, then the teachers learn only that the students didn’t get it and that they need to teach the material again, only, presumably, better ” I wrote more on this quote a couple years ago http://scottfarrar.com/blog/evaluative-listening-and-khan-academy/

Some background on “try again”: this was put in when KA moved away from needing streaks of getting questions right (N in a Row). So instead of the feedback saying, “wrong”, this was a new more forgiving feature to say “try again”.

Insider’s perspective!

At it’s core, the math practice software isn’t well set up to listen interpretively because (1) it doesn’t do (simulate) the math at hand, but (2) give the first problem, it asks narrowly answered questions. In contrast, the Mario engine simulates jumps and Mario can do many different kinds of jumps, as it interprets the various combinations of buttons pushed, and the surrounding obstacles onscreen. Or: a well crafted Desmos/Geogebra graph will simulate some scenario and any *valid* student input can be interpreted within that scenario. (Like Kevin noted in Function Carnival’s Cannon Man)

But I think math practice software’s constraints and behaviors ties into one of Dan’s “Classic Hits”: Clever Hans. The horse that learned how to “do math” by carefully interpreting the evaluative feedback of it’s trainer. (stomp out 2+3. stomp, stomp, stomp, stomp, *oooh* stomp *ahhh* “see he stopped at 5”)

Students “Clever Hans” their human teachers in this way all the time: play the teacher’s unwitting feedback to the student’s advantage to pursue the goal of answering teacher questions correctly. Acquiring understanding in this way is beside the point.

And I believe students practicing on software can Clever Hans the evaluative feedback to similarly sidestep learning in favor of rewards. They learn the feedback-space of the math practice software, and pick up patterns of question authoring, and can efficiently move through the work to achieve their goals: complete the assignment. I explored a little more with Khan Academy answer data in a more formal way recently and I developed another hypothesis that much of the wrong answers given on Khan Academy could be from basic heuristic that is “input the first thing that comes into your head”. (the paper is here http://people.ischool.berkeley.edu/~zp/papers/ICLS_distributed_misconceptions.pdf)

The student enters “9c” and the software can’t interpret it, so it gives feedback of “wrong” or “try again”. But the student now must try to use that information. They move onto their second guess without much additional thought. It’s a very thin “conversation” between learner and software because neither one is really listening to each other. The student is doing some “success finding” heuristic and is not necessarily thinking about the math at hand– *because* the system cannot contribute anything about the math at hand.

Dan wrote in another comment “interpretative feedback version of the math software (in my head) is a sentence that says, ‘Oh you’ve written the product of 9 and c, not the quotient. Try again.'” Regardless of if the system should give that full sentence — the fact that a system could do so is the key there. The system must have the capability to interpet the user’s input in order to hold a meaningful conversation.

Fantastic demonstration of Mario seizing up in midair. A physical reaction grips me when I’m denied seeing the natural consequence of the action.

Featured Comment

Backgammon was an example of the latter sort for me: because so much luck is involved, and winning is about knowing when you have a 55% chance of success and pressing your advantage, you often get punished for good decisions and rewarded for bad decisions, which confounds the learning process unless you have the right lens.

I’m not a big fan of these practice kinds of programs, but if I wanted to improve this one specifically, I might include a couple of “checks,” of the type I might do when thinking about translating some description into algebra. For example:

“The quotient of 9 and c”
Input: 9c

Let’s check: if c =2, the quotient of 9 and c would be 4.5.
Your answer: 9c = 18 when c = 2

If c = 27, the quotient of 9 and c would be 1/3.
Your answer: 9c = 243 when c = 27.

Something weird is going on…

My first thought was to shape the feedback more positively so it didn’t simply say ‘you’re wrong’ & follow up by showing ‘here’s why’. Trying to avoid the verbal equivalent of a big red X, I came up with: 9c would be the answer if the question was asking for a way to show multiply. Ttry again. That seemed to address your big idea of being interpretive rather than evaluative. After some reflection, however, I now see 2 big problems with it.

First, the phrase ‘try again’ can be taken by some learners as an invitation to game the system. They will devote their efforts to figuring out how outsmart the program so they can get through the activity as quickly as possible. <> Beating the system can seem a lot more rewarding than doing the math.

More importantly, though, I realized I’d fallen into the assumption trap. Without first asking why a learner might choose the raised dot, I’d just assumed that the underlying concept being tested was which operation was correct. What if a student simply didn’t remember the meaning of the word ‘quotient’, or didn’t recognize that a/b is a way to represent division, or was on a smartphone and had touched the 3rd button instead of the 4th, or, or….? My feedback would only be meaninful and useful if the learner’s difficulty matched my assumption.

We need a lot more information from students about what underlies the errors we observe. Only then can we address learners’ issues as they see them and stop ourselves from jumping to solutions based on our (often unconscious) assumptions.

So here’s my second try. After the arrows are suggestions for corrective follow-up. The original question would have to be completed correctly in order for the student to move on.

9c would be the answer if the question was asking for a way to show multiply. Where do you think you might have gone wrong?
(a) I really wanted a different button. –> “OK, you have 1 more try.”
(b) I’m not sure what ‘quotient’ means. –> activate prior learning by reminding when/in what context this was previously learned; if want to just give definition, interleaf similar questions in future activities to reinforce learning.
(c) None of the answers looked right, so I guessed. –> provide micro-lesson (5 min or less) on ways to show multiply & divide.
(d) Other: _________________________________________ –> “That’s interesting. Please share it with your teacher.”
(e) This is my 2nd try, & it’s still not right. What now? –> “You have 2 options. See the teacher now & then finish the activity, OR do all the questions you can & get help with the rest at the end.”

The best of these mathematical games that I’ve seen is Project Euler (projecteuler.net). When you answer a question wrong, it just gives you a big red x. When you answer a question right, a nice green check. There’s no prompt to get help or skip the question.

Always woven into the question is a simpler particular case of the question with an answer that allows you to check your own reasoning as you work. It’s marvelously effective at allowing a mathematician to work towards an answer from a place of discovery.

The reward for answering correctly (!) is access to a discussion forum where other people who have answered the question correctly can discuss their answers.

Socially it’s totally different from most online (and classroom) pedagogies, and marvelously effective.

I think there is a better way of explaining how video games interact with the player. Saying the game “interprets” player input is too simplistic. For your argument, I claim the better term would be “evolve.” Video games are a series of challenges that rise in difficulty as the games progress. When a player interacts with them, the challenges evolve based on what the player did.

The evolution of the challenge is critical to player learning the challenge. Because not only does it allow the player to interpret the results of their actions, but the player understands how the game changed because of their actions.

Here is an example.
Here is a gif of a Mario fan game granting Mario the use of a portal gun.
https://i.imgur.com/Htssq.gif
When presented with an obstacle (Hammer Bro), Mario uses the portal gun to have the hammers thrown at him come right back to the Hammer Bro. The Hammer Bro. is defeated, but the lethal hammers are still in play being thrown over and over again in a similar arc through the leftover portals. Bringing about a new challenge for the player to solve.
If the game were to just “interpret” the player’s actions in this situation, the Hammer Bro. is defeated. However, we think in terms of how the game “evolved,” the player has defeated the Hammer Bro. BUT, because of the specific action the player took in defeating the Hammer Bro., a new problem presented itself. Allowing the player to understand more of their own technique and problem-solving method.

There are not many math subjects that may allow for this type of “evolutionary interaction” in their current state. But developing a teaching style or game with this in mind will allow for students to better understand the subject matter.