I took machine-graded learning to task earlier this week for obscuring interesting student misconceptions. Kristen DiCerbo at Pearson’s Research and Innovation Network picked up my post and argued I was too pessimistic about machine-graded systems, posing this scenario:
Students in the class are sitting at individual computers working through a game that introduces basic algebra courses. Ms. Reynolds looks at the alert on her tablet and sees four students with the “letters misconception” sign. She taps “work sample” and the tablet brings up their work on a problem. She notes that all four seem to be thinking that there are rules for determining which number a letter stands for in an algebraic expression. She taps the four of them on the shoulder and brings them over to a small table while bringing up a discussion prompt. She proceeds to walk them through discussion of examples that lead them to conclude the value of the letters change across problems and are not determined by rules like “c = 3 because c is the third letter of the alphabet.”
My guess is we’re decades, not years, away from this kind of classroom. If it’s possible at all. Three items in this scenario seem implausible:
- That four students in a classroom might assume “c = 3 because c is the third letter of the alphabet.” I taught Algebra for six years and never saw this conception of variables. (Okay, this isn’t a big deal.)
- That a teacher has the mental bandwidth to manage a classroom of thirty students and keep an eye on her iPad’s Misconception Monitor. Not long ago I begged people on Twitter to tell me how they were using learning dashboards in the classroom. Everyone said they were too demanding. They used them at home for planning purposes. This isn’t because teachers are incapable but because the job demands too much attention.
- That the machine grading is that good. The system DiCerbo proposes is scanning and analyzing handwritten student work in real-time, weighing them against a database of misconceptions, and pairing those up with a scripted discussion. Like I said: decades, if ever.
This also means you have to anticipate all the misconceptions in advance, which is tough under the best of circumstances. Take Pennies. Even though I’ve taught it several times, I still couldn’t anticipate all the interesting misconceptions.
The Desmos crew and I had students using smaller circles full of pennies to predict how many pennies fit in a 22-inch circle.
But I can see now we messed that up. We sent students straight from filling circles with pennies to plotting them and fitting a graph. We closed off some very interesting right and wrong ways to think about those circles of pennies.
Some examples from reader Karlene Steelman via e-mail:
They tried finding a pattern with the smaller circles that were given, they added up the 1 inch circle 22 times, they combine the 6, 5, 4, 3, 2, 1, and 1 circles to equal 22 inches, they figured out the area of several circles and set up proportions between the area and the number of pennies, etc. It was wonderful for them to discuss the merits and drawbacks of the different methods.
Adding the 1-inch circle 22 times! I never saw that coming. Our system closed off that path before students had the chance even to express their preference for it.
So everyone has a different, difficult job to do here, with different criteria for success. The measure of the machine-graded system is whether it makes those student ideas invisible or visible. The measure of the teacher is whether she knows what to do with them or not. Only the teacher’s job is possible now.
Featured Comments
This doesn’t even touch the students who get questions RIGHT for the wrong reasons.
Dashboards of the traditional ‘spawn of Satan & Clippy the Excel assistant’ sort throw way too much extremely specific information straight to the surface for my liking (and brain). That information is almost always things that are easy for machines (read. programmers) to work out, and likely hard or time consuming yet dubiously useful for humans to do. I wonder how many teachers, when frozen in time mid-lesson and placed in the brain deli slicer would be thinking “Jimmy has 89% of this task correct and Sally has only highlighted four sentences on this page.”
28 Comments
Mike Caulfield (@holden)
October 3, 2013 - 2:47 pm -I’m more optimistic than you about future capabilities. But ultimately future capabilities don’t matter, because they are a ways away and there are students we must attend to now.
I think the most obvious question the Pearson response raises is “Who is Ms. Reynolds?” Why do salespeople and think-tankers always want to discuss these “visions” involving an array of fictional characters, and represented by stock art with names like “Female_teacher_helps_male_student_laptop.jpg”. If it’s truly extant practice, link me to the teacher’s blog who’s doing it now.
Kristen
October 3, 2013 - 3:09 pm -Hi Dan,
First let me say that I picked up on your post yesterday because I am a big fan of your work and subscribe to your blog in my reader.
Nice example here. Thank you for the reality check. I will continue to use your blog as one way I stay grounded.
Kevin Hall
October 3, 2013 - 3:26 pm -What do you think about doing the Super Mario 3-act (or an abbreviated version) right before Penny Circles, to discuss assumptions of linearity and nonlinearity?
Mike Caulfield
October 3, 2013 - 3:29 pm -Now, of course, I feel like a jerk. Never type a comment after reviewing session proposals. It’s not going to be your most subtle moment.
Jeremy Bell
October 3, 2013 - 6:47 pm -Dan, I’m a really big fan of yours, but I’m just always still baffled that your STILL against using machine grading to ANY degree in the classroom.
I think you dipped out of the classroom right before the whole machine grading movement started getting traction.
I, like you, was really skeptical at first. How in the world could this be anything but more dehumanizing. But life is full of ironies. What’s weird is that once I put my curriculum on a Moodle LMS, my classroom became a million times more humanized. It’s really incredible. And now that I have a strong foundation, I can place more level 4 stuff on top like your 3-act math.
The machine grading/blended learning/whatever could really be your greatest ally if you could see the potential inside of it.
MaryAnn Moore
October 3, 2013 - 7:02 pm -I’ve been following your pennies lesson. I really appreciate this follow-up post. That’s what I love most about teaching. My students always surprise me by coming up with strategies I had never considered. You can’t script that kind of stuff.
Brendan Murphy (@dendari)
October 3, 2013 - 7:10 pm -I think Person is missing the most important part of school. The students do not like to sit at a computer and work diligently at math problems. If they are playing a game they don’t want to be interrupted to listen to a teacher rant.
However, you want to describe it, the idyllic view of the classroom is the foundation of how they envision the program to work and that is wrong. Pretty much everything after that will also fail
Jonathan Claydon
October 3, 2013 - 7:38 pm -I had my students work through pennies last week. The day before I had them work through an analog component though. I put circles of various sizes on my tables (random non-whole numbers from 2in to 4in) and gave them pennies to fill them with. Then I put a picture of a 12″ circle on the board and had them come up with a guess as a table. The data they just collected was on the board for them to make a basis of comparison. It was at this point that they got to have discussion about scaling, adding up circles of different sizes, etc. Some started asking about the diameter of a single penny and trying to fiddle with area formulas. All those quirky strategies you mentioned were here. Then I tortured them by withholding the answer. The next day a few told me about things they tried at home to figure it out (one kid works at Dominos and emptied the register on top of a 12″ pizza grill).
After working through capital P Pennies and seeing what they could do with a 22″ circle, I asked if our 12″ circle could be found with what they made. While discussing this, I marked off a 12″ circle on a random table and had some kids physically fill it in to compare reality vs. model. In all but one case the predicted value (~200) was quite close to reality (195-220).
I have an overdue write up on the whole experience.
Tim Stirrup
October 4, 2013 - 12:31 am -picking up on the twitter conversation I had with you yesterday, now I am off the phone.
The assertion that machine marking of misconceptions is decades away is the one I am addressing.
Times do, as Jeremy says, move on. Your work with Desmos moves us on from handheld graphic calculators for example.
Mathspace, the company I work for, has just released a product that automatically grades work as students progress through solving a problem, giving appropriate feedback at that time, and recording all this for the teacher to see. In real time, whether at home or in the classroom, on what we hope is a simple to use and read dashboard.
We are very new, and so I do apologise for what may seem a bit of a sales pitch, but many teachers read your blog, and to say that such things are decades away misses out work being done by us (and probably others). Mathsoace exists, you can have a play, and it is being used in classrooms now. It is not a hypothetical situation.
Students enter their mathematics either through a simple web interface that makes sure the mathematics is written as mathematics looks. Latex it isn’t! For those on iPads (and soon android), then students use their own handwriting – either using their fingers or a stylus. Mathspace digitises this handwriting and then does the checking as normal.
The work is checked for mathematical accuracy. so it doesn’t matter what route students take to get to the end of a problem, if it is a mathematically correct step, then great.
And our developers are working on adding in the misconceptions commonly found. So Mathspace will recognise these and give feedback as appropriate to teacher and student. this admittedly is a couple of months off, but not years or decades.
Maybe the holy grail of marking completely open-ended problems automatically, with all misconceptions (even really off the wall ones) accounted for, may be some way off. But then does anyone use such problems ALL the time, EVERY day, and every lesson. I think not. The day that happens everywhere is indeed many decades away, if a all possible.
but through the use of machine marked problems that show working out, process and are easy to use, then maybe we CAN release teacher time to prepare and use problems like pennies and others more often?
Isn’t that a good thing?
Bruce Ferrington
October 4, 2013 - 3:17 am -Hi Dan – I gotta agree – you cannot anticipate all the creative ways that kids will solve (or not solve) a problem. That’s what makes teaching fun – the surprises you get every day. And the realisation that as a teacher, I’m still a learner.
SueHellman
October 4, 2013 - 3:19 am -This doesn’t even touch the students who get questions RIGHT for the wrong reasons.
J Thomas
October 4, 2013 - 5:40 am -Finding misconceptions with ordered multiple choice or multiple true/false is far more likely in the immediate future. Paige Keeley has written several books of probes and books on Formative Assessment techniques for just this purpose.
I liked to use sticky bars, to get a quick feel for which misconception my students had on a probe. So, if 6 kids put there sticky note on answer ‘b’, I knew that I needed to address that misconception. But the follow up question of why do you think that is far more important.
As for analyzing step by step hand written work that quickly with a school network…
Andrew
October 4, 2013 - 6:03 am -@Tim
I am interested in this quote from comment 9:
“a product that automatically grades work as students progress through solving a problem, giving appropriate feedback at that time, and recording all this for the teacher to see.”
I guess my question is why it is desirable to remove the task of examining work and giving feedback from the teacher. In my experience, that is where some of the most meaningful teacher-student interactions take place. It’s where the teacher gets to know the student (as more than a series of mathematical strengths and weaknesses) and it’s where the student develops trust in the teacher.
It seems like this incredibly important interaction, the one where a student makes an attempt and offers it to the instructor as a conversation starter about that student’s understanding, would be the very last thing that we would want to remove from the job of teaching.
George Bigham
October 4, 2013 - 8:30 am -The technology for reading handwriting, at least handwriting that is a human could also read, is surprisingly simple and the post office has been using it for years. There is an free app called MyScript Calculator and MyScript MathPad which are primitive, but demonstrate that the ability to translate math script to computer analyzable data is not so far off. Yes, the database for analyzing errors is tricky, it would certainly be worse than a good teacher, but its goal should be to save precious teacher time, because as you point out, the most precious resource is teacher time (which I’m sure every teacher will agree there is never enough of.) If these computers are just giving more data, they are actually using teacher time with questionable value.
Tim Stirrup
October 4, 2013 - 10:14 am -in danger of a ‘blogjacking’ – sorry. Last post from me on this.
Sue: right answers are recorded and the method as well as wrong. just like in a student book.
Bruce: the surprise element is definitely still there.
Andrew: removing the task from the teacher… ugh! no. Just making it easier for the teacher to do so, and letting students have that feedback when the teacher isn;t around – when they get stuck for example on a homework task.
George: Myscript is what we use – optimised of course for every single question to fit in with the context. so, not so far off… no, it’s here. More data without a reason – waste of time and many online products overdo the ‘data’ because showing lots of data hides the fact that the important working out and reasoning is missing. Nice graphs and percentages often don’t mean a thing.
and after saying all this, still really like the work that Dan is doing, the 3Acts are excellent, the pseudocontext Saturday surely underused (I reckon we have a couple of those to weed out…). and we shall keep on working on the misconceptions.
Dan Meyer
October 4, 2013 - 3:02 pm -@Kristen, thanks for the provocative post. Good food for thought.
Kevin Hall:
I struggle with the ordering, Kevin, but I think it’s a real defensible idea to expose students to non-linear patterns before they fall into an all-linears-all-the-time mode of operation.
Jeremy Bell:
I didn’t say that in this post.
Sue Hellman:
Big point.
J Thomas
Certainly true. Indeed, that’s already here. This post is more concerned with the evaluation of unstructured, handwritten work, though.
Maria
October 5, 2013 - 6:10 am -Maybe the only point a machine can help us is in knowing which students have understood the concept and are able to answer correctly all the questions.
The problem, however, is still in what we do after grading: even with best students, I don’t know if a machine could assign them more challenging activities on the basis of the knowledge they showed in their previous works, because it’s still a subjective point of view.
And, for me, the most interesting thing of doing exercises is the moment I’m discussing the different answers with my students, but we’re far from machines having a good conversation… ;)
heather
October 5, 2013 - 7:34 am -I think math is very difficult have graded by a machine. There are multiple ways of finding an answer, and multiple ways of expressing that answer. I like sites that use math to help kids learn like MatLab, early in their life because they will need to use these tools possibly in their career, so shouldn’t we expose them early? I wish math was more technology oriented, but not graded, it limits the artistry, and imagination of a student!
heather
October 5, 2013 - 7:47 am -@ Tim Stirrup and @ Andrew
I liked both of your comments. I was shocked to hear that there is software out there. That would fee up a lot of time. And Andrew I think you could use the software since it is recorded for the teacher and review each problem. Then you will understand the student and have time free to invest in imaginative lesson planning. I would love to use software such as Tim suggested.
Thanks,
Heather
heather
October 5, 2013 - 7:54 am -@ Maria
This time for Maria I am strongly FOR using a computer that interacts with students. It is so difficult to work in a classroom teaching algebra with 45 students, where some still don’t understand basic multiplication, and some students are ready for more challenging work. It presses the teacher to help the students needs be met. But if we could have a computer that interacts with the students and focuses on their developmental level, it would be wonderful for each student because for a limited amount of time they could get their needs met.
Jeff Layman
October 6, 2013 - 3:02 am -In a note unrelated to math instruction, you keep churning out great examples of ways to constructively disagree with others that a lot of blogosphere folks could really take to heart. I don’t teach math, but there’s a whole lot more I learn around these parts.
Gnostradamus
October 6, 2013 - 3:46 am -“It is so difficult to work in a classroom teaching algebra with 45 students, where some still don’t understand basic multiplication, and some students are ready for more challenging work.”
Well, maybe your district or school is either ignorant or apathetic to the fact that you CAN’T teach 45 kids in one class, even if they’re the best students in the world. Add to that skill levels that are too disparate, and you’ve got a certain-to-fail model of education that is forced on too many public schools.
You know what they’ll do when they put computers in your room, heather? They will cut even more teachers to buy the (and maintain) the equipment, and have more students.
Not that 45 kids in a lab work, either. I teach computer science also.
This, folks, is the tragic reality of American education. Our teachers and schools usually do the best we can. But until we fix all the surrounding problems with poverty, funding, etc, some students never get a fair chance.
Dave Major
October 6, 2013 - 4:01 am -Cognitive load is generally (at least to my experience) explored with reference to students, but I bet there is some interesting stuff going on with the teacher. Dashboards of the traditional ‘spawn of Satan & Clippy the Excel assistant’ sort throw way too much extremely specific information straight to the surface for my liking (and brain). That information is almost always things that are easy for machines (read. programmers) to work out, and likely hard or time consuming yet dubiously useful for humans to do. I wonder how many teachers, when frozen in time mid-lesson and placed in the brain deli slicer would be thinking “Jimmy has 89% of this task correct and Sally has only highlighted four sentences on this page.”
My current thinking is that machine grading is not yet useful unless you are pushing some sort of industrial-level mass teaching in a situation where meaningful student-teacher interaction is either impractical, impossible, or undesirable. Being a music teacher by trade gives me a good viewpoint of thinking “Would I let a computer judge this student performance? No, but it could record it for me to use later.”
If as Dan suggests, “only the teacher’s job is possible now”, perhaps we should look at making the machine more of an assistant: gathering the information a teacher does think about and presenting it in a way that makes their thought processes and decisions easier.
I don’t want marks & grades. I want my computer to saying “Jimmy’s previous work? sure – here you go buddy! Spot any problems we can tag and file with others? Oh hey, Sally has just asked for help – here is her screen and a timeline of what she has done. On the big screen? but of course.”
Dan Meyer
October 6, 2013 - 6:33 am -Dave Major:
Well, all the teachers in videos like this one.
Evan Weinberg
October 6, 2013 - 7:07 am -Machine grading might be able to identify mistakes and grading answers in handwritten work soon, if it isn’t able to do it now. I fear that machine grading has the potential to isolate students from each other in the same way watching videos does. I’m more interested in getting technology to leverage the benefit of having a bunch of students in the same room that are working through the same general concepts.
Dave Major
October 6, 2013 - 9:48 am -I always feel like an apologist for ‘big data’ because I can see ways it could be so amazing without major advances technological, but there are so many things that terrify me about that video:
– Not one educational intervention shown is a reaction to anything but a colour code. Not one.
– Reading time with my child?!? Quick, schedule some television.
– 3/4 for counting to 100. Cool – BUT WHERE DID THEY GO WRONG? If a kid is struggling with something, I want every teacher of every subject to know and to be feeding back their experience.
Data is cool, powerful and amazing, but it should part of a varifocal lens through which educators view their work.
Maria
October 6, 2013 - 9:57 am -@heather
I agree with you that it must be really hard to teach in a 45 students group….
When I said that I enjoy discussing with my students their answers to exercises, I was talking of a class with a maximum of 25-27 students. In my country it’s not possible to have 45 students in a classroom, by law. Of course that it’s impossible to have a normal conversation with 45 people, students or not…
Maybe in your case I would use software for dividing the classroom in two parts: while the half of the students are doing exercises in their laps, you can discuss and debate with the other half what are the best ways to solve an exercise, etc.
Christian
October 7, 2013 - 6:18 pm -I think added value is (and was) upon us with moments where students are NOT in the classroom. Getting some practice done between lessons, with lessons for explanation, group discussion, collab tasks, and the day before the next lesson a teacher glancing over an intelligent panel with scores (cos a teacher can’t check 30 students’ 10 tasks, maybe even over multiple classes). Then, notice some often made mistakes, use these on a projector to show and ask what has gone wrong (use a pedagogically sound way of course) but also what has gone right but not all that efficient. Yes, that can be done. But it needs the teacher for the final analysis (computer ok for rough) at the moment, and student-teacher domain knowledge of misconceptions and good/bad strategies. A GPL-ed MySCript type interface should appear soon too. Still a long way to go, but don’t let the ‘big names’ be typical for the developments.