Posts

Comments

Get Posts by E-mail

Stephanie H. Chang, one of Khan Academy’s software engineers:

I observed how some students made progress in exercises without necessarily demonstrating understanding of the underlying concepts. The practice of “pattern matching” is something that Ben Eater and Sal had mentioned on several occasions, but seeing some of it happening firsthand made a deeper impression on me.

The question of false positives looms large in any computer adaptive system. Can we trust that a student knows something when Khan Academy says the student knows that thing? (Pattern matching, after all, was one of Benny’s techniques for gaming Individually Prescribed Instruction, Khan Academy’s forerunner.)

It is encouraging that Khan Academy is aware of the issue, but machine-scorers remain susceptible to false positives in ways that skilled teachers are not. If we ask richer questions that require more than a selected response, teachers get better data, leading to better diagnoses. That’s not to say we shouldn’t put machines to work for us. We should. One premise of my work with Dave Major is that the machines should ask rich questions but not assess them, instead sending the responses quickly and neatly over to the teacher who can sequence, select, and assess them.

BTW. Also from Chang’s blog: a photo of Summit San Jose’s laptop lab, a lab which seems at least superficially similar to Rocketship’s Learning Lab. My understanding is that Summit’s laptop lab is staffed with credentialed teachers, not hourly-wage tutors as with Rocketship. Which is good, but I’m still uncomfortable with this kind of interaction between students and mathematics.

[via reader Kevin Hall]

Featured Comment

Stephanie H. Chang responds:

We think the work you’re doing with Dave Majors is really exciting and inspiring. Open-ended questions and peer- or coach-graded assignments are incredibly powerful learning tools and my colleagues at KA don’t disagree. We definitely have plans to incorporate them in the future.

Mg:

My old school last year relied on a teaching model where the students had to try and teach themselves a lot of math by utilizing classroom resources. A lot of the practice was through Khan Academy or by students completing practice problems with accessible answer keys. Ultimately what happened was that the students only looked for patterns and had no conceptual understanding of the math at all. Even worse was that students who had “mastered” the concept were encouraged to teach the other students how to solve problems but they could only do so in the most superficial manner posssible.

Bowen Kerins:

One way sites like Khan (and classroom teachers) can deal with this is by retesting — say, three months later, can a student solve the same problem they solved today? If not, they clearly only had a surface-level understanding or worse.

I’d like to see Khan or other sites force students to retest on topics that were marked as “completed”. But then again, I feel pretty much the same way about miniquiz-style Standards Based Grading.

jsb16:

Reminds me of the story about the tank-recognizing computer. I doubt we’ll have worthwhile computer scoring that isn’t susceptible to pattern-matching until we have genuine artificial intelligence.

And then the computers will want days off, just as teachers do.

Noam:

KA does force review of concepts after mastery is achieved, generally a few weeks after completion. Problem is, doesn’t take students long to do the pattern matching again.

We instituted a policy where students must make their own KA style videos explaining how to solve a set of problems that they struggled with. Best way we found to deal with the issue.

Zack Miller, comments on the laptop lab at Summit where he teachers math:

Our math model as described as concisely as possible: students spend two hours per day on math; one hour in breakout rooms and one hour in the big room (seen in your picture) where students are working independently. In the breakout rooms, students work on challenging tasks and projects (many of which we can thank you for) that develop the standards of math practice, often in groups and with varying amounts of teacher structure. Development of cognitive skills via frequent exposure to these types of tasks is paramount to our program. It is also in the breakout rooms where students’ independent work – which is mostly procedural practice – is framed and put in context. Students’ know that their work in the big room supports what they do in the seminar rooms and vice versa.

38 Responses to “Pattern Matching In Khan Academy”

  1. on 25 Jan 2013 at 7:01 pmMark

    I am generally a fan of IXL.com, but I see the same sorts of pattern-matching issues there as well. Anytime we ask repeated questions about one isolated skill, this is a serious concern IMO.

  2. on 25 Jan 2013 at 7:02 pmMatt E

    I have a hunch that something important is lost when we stop scribbling on paper. At this point I can’t be any more specific than that.

  3. on 25 Jan 2013 at 8:02 pmChris Robinson

    While I don’t agree with the pedagogical decisions of KA, we can at least take solace in the fact that students are recognizing and utilizing patterns for their benefit. Seems like an important skill to be successful with mathematics.

  4. on 25 Jan 2013 at 8:08 pmblink

    The pattern matching problem is real, and it is most obvious with computer-based instruction. Sadly, though, I fear that the assessments on which many (most?) human teachers rely are almost equally susceptible. The battle lines seem to be drawn around the medium, but that misses the larger picture and the problem with current practice.

  5. on 25 Jan 2013 at 8:12 pmjsb16

    Reminds me of the story about the tank-recognizing computer. I doubt we’ll have worthwhile computer scoring that isn’t susceptible to pattern-matching until we have genuine artificial intelligence.

    And then the computers will want days off, just as teachers do. :)

  6. on 25 Jan 2013 at 9:11 pmRebecca Phillips

    Isn’t that something we *want* them to learn? “Look for and express regularity in repeated reasoning”? {wink}

  7. on 26 Jan 2013 at 1:56 amwilliam

    Concur with Rebecca.

    What you call pattern matching could also be called access to long term memory. When we learn the multiplication tables, we’re learning to pattern match. It’s ok to _teach_ multiplication in a way that reveals meaning, but after that we just want students to _know_ it.

    How many times have you come across algebra students who can’t multiply and are doing poorly because of that missing pattern matching routine in their heads?

    If there’s any valid criticism it’s that students are learning the wrong patterns.

  8. on 26 Jan 2013 at 3:20 amDolores Gende

    Great point!
    Same thing happens in traditional physics classrooms that focus heavily on problem-solving.
    Teachers solve a problem on the board, homework exercises offer the same physical situation with different numbers, then quiz replicates the item with yet another set of numbers.
    Are students demonstrating understanding of physics concepts or are they just good at memorizing algorithms?

    These are more examples of @fnoschese pseudoteaching or even better: pseudolearning!

  9. on 26 Jan 2013 at 3:57 amR.G.

    That’s why open problems are so important. Problems, with not the slightest hint how to proceed. Maybe not even the question is clear. This way of teaching has been discussed in that blog before, I believe (Japanese versus American way).

    Note, however, that pattern matching is okay for drill stuff, and even necessarily so. Some drills are unavoidable in math too. We need that before we can do useful things.

  10. on 26 Jan 2013 at 8:36 amMichael

    “If we ask richer questions that require more than a selected response, teachers get better data, leading to better diagnoses. That’s not to say we shouldn’t put machines to work for us. We should.”

    IMHO, we should use Khan Academy and similar tools just exactly for this sort of work – routine skills, even if the skills involve pattern matching. That frees up my classroom time for the rich tasks, whether they are based on a computer or in a discussion. Definitely outsource the lower-level work (Bloom’s Knowledge) to the machines and give us humans the Synthesis and Analysis and Evaluation – the tasks that so often appear on this blog.

  11. on 26 Jan 2013 at 9:27 amCurmudgeon

    I find these kinds of errors showing up often on the gray, blurry line between teaching and computing/programming.

    Those who teach don’t necessarily understand programming so they tend to not be able to specify what works in a classroom and the processes or habits that the software is meant to address.

    Those who program have never taught so they don’t know how to write the code that solves the problem because they fundamentally don’t understand the problem.

    Unfortunately, those who have done both to a sufficient level aren’t writing the programs that most of us need.

    You see this in grading programs that ignore the ways teachers operate and get in the way instead of making life truly easier, IWB software that doesn’t work or fails in bizarre ways, or tools and software that include limitations and drawbacks unnecessarily (Looking at you, TI-SmartView).

    “I observed how some students made progress in exercises without necessarily demonstrating understanding of the underlying concepts.” – yeah.

    We’ve come a long way but not nearly far enough to allow the replacement of teachers; like 11 Micheal I see the best use of Khan as taking the rote memorization out of the hands of the teacher-in-class (though Flash Cards is the Number ONE “really cool idea” promoted by the parents I talk to, especially the home-schoolers).

    Let them practice outside of class and bring basic skills to automaticity so that we can do some real work together in our limited time together.

  12. on 26 Jan 2013 at 4:51 pmBowen Kerins

    While this happens a lot with Khan and IPI, the same thing can happen in all sorts of instruction where we care only about the outcome and not about the methods or underlying thinking involved. I see a lot of things taught in “test-prep” mode where a student can learn to get correct answers to simple questions with no clue what is happening.

    One way sites like Khan (and classroom teachers) can deal with this is by retesting — say, three months later, can a student solve the same problem they solved today? If not, they clearly only had a surface-level understanding or worse.

    I’d like to see Khan or other sites force students to retest on topics that were marked as “completed”. But then again, I feel pretty much the same way about miniquiz-style Standards Based Grading…

  13. on 27 Jan 2013 at 7:59 amMg

    My old school last year relied on a teaching model where the students had to try and teach themselves a lot of math by utilizing classroom resources. A lot of the practice was through Khan Academy or by students completing practice problems with accessible answer keys. Ultimately what happened was that the students only looked for patterns and had no conceptual understanding of the math at all. Even worse was that students who had “mastered” the concept were encouraged to teach the other students how to solve problems but they could only do so in the most superficial manner posssible.

  14. on 27 Jan 2013 at 8:44 amMr. K

    I’m surprised no one mentioned the #1 takeaway in the article: Work it out on paper, first.

    This is promoted by someone who is using technology to develop technology. The fact that they use paper to do their actual thinking probably says something about how much thinking we can get our kids to do if we ask them to use a laptop or tablet as their primary resource.

  15. [...] blog post by Dan Myer, Pattern Matching In Khan Academy, recently caught my attention. I’m a huge fan of the power of pattern matching, so I was [...]

  16. on 27 Jan 2013 at 12:12 pmStephanie H. Chang

    Hi Dan,

    Thanks for your thoughts.

    “One premise of my work with Dave Majors is that the machines should ask rich questions but not assess them, instead sending the responses quickly and neatly over to the teacher who can sequence, select, and assess them.”

    We think the work you’re doing with Dave Majors is really exciting and inspiring. Open-ended questions and peer- or coach-graded assignments are incredibly powerful learning tools and my colleagues at KA don’t disagree. We definitely have plans to incorporate them in the future.

    Stephanie

  17. on 27 Jan 2013 at 2:16 pmDaniel Schaben

    I am finding the same issue using another online program that will remain nameless. I have been teaching like I normally teach, then using the online assignments rather than book assignment. I then test and quiz with existing paper pencil tests. What I am finding is that the computer practice seems to be leading students to the answer without much thought on the part of the student. It is still the same concept and problems are very similar to what I would assign in the book, but there are only a handful of students that can transfer what they did on the computer to a paper pencil test. It has been very frustrating, and next year I am going to back way off of any online assignments like this until there are more PROVEN strategies that work.

  18. on 27 Jan 2013 at 3:40 pmChris Shore

    Just because I rock at Madden 2012, doesn’t mean anyone is going to draft me to play in the Super Bowl.

  19. on 27 Jan 2013 at 4:56 pmNoam

    KA does force review of concepts after mastery is achieved, generally a few weeks after completion. Problem is, doesn’t take students long to do the pattern matching again.
    We instituted a policy where students must make their own KA style videos explaining how to solve a set of problems that they struggled with… Best way we found to deal with the issue

  20. on 28 Jan 2013 at 3:19 amKay O

    Thank you for bringing Benny back!

    I’ve had a copy of Erlwanger’s article (http://www.wou.edu/~girodm/library/benny.pdf) on my desk since our district started touting flipped mastery. I’m so glad others are sharing this informative piece of research.

  21. on 28 Jan 2013 at 3:28 ammg

    Wow Kay, that’s exactly what I was talking about, thanks for sharing that research article, I had not seen it before.

    When we don’t give them good direction, and formative assessment, students start to develop their own algorithms which work in some, but not all cases. This can be quickly deciphered by asking them “why” they did something, but the longer they are utilizing their incorrect algorithm, the harder it is for them to unlearn it.

  22. on 28 Jan 2013 at 6:46 amBrendan Murphy

    In the past I’ve heard mathematics could be described in its essence as the search for patterns so this entire article is a surreal experience of ‘Students are gaming the system of learning math by doing the essential essence of math.’

    In any event is it not obvious that the use of adaptive learning software must require the close attention of a real teacher?

  23. on 28 Jan 2013 at 8:45 amPhysics Games « EDufiCATION

    [...] about relaitonships between quantities in the games. It’s too easy (and fun) to just use pattern recognition to play the game without caring about [...]

  24. on 28 Jan 2013 at 10:24 amDavid Petro

    @ChrisShore actually that is actually happening. For example professional football players actually Play Madden and are actually starting to alter their play because of it. Check out this article from Wired (http://www.wired.com/magazine/2010/01/ff_gamechanger/). Of course you still need the physical skills but there is no reason it can’t enhance what they are already doing.
    Now you can take that a step further with Lucas Ordonez who won an online gaming contest put on by Playstation and Grand Turismo. The prize? He got to become a real professional race car driver and has actually been on the podium in Le Mans http://www.youtube.com/watch?v=3lCn38ojx48

  25. on 28 Jan 2013 at 2:31 pmDan Meyer

    Brendan Murphy:

    In the past I’ve heard mathematics could be described in its essence as the search for patterns so this entire article is a surreal experience of ‘Students are gaming the system of learning math by doing the essential essence of math.’

    This has come up a number of times, whether from Rebecca Phillips’ nudge in #6 to Jared Cosulich on Twitter. The trouble is there are patterns students can attend to that have nothing to do with the mathematics. (ie. “Always choose C.” or “Always choose the answer that’s across from the even number.”) Attention to those kinds of patterns is counterproductive.

  26. on 28 Jan 2013 at 3:55 pmChristian

    Why do people keep persisting in an -in my opinion- artificial distinction between insight and flexible skills (with maybe patterns in between)? Both are closely linked. Skills come forth out of insight, insight can come from skills. I like what Arcavi has written about this. I clearly remember getting insight in epsilon-delta and limits by writing out some of them. Rote, but repetition did increase insight in structures and patterns. And yes, I think this is a major part of maths. Open problems can also become rote. Rote problems can be turned in interesting experiences with slight modifications.

    So I would say -just as in f2f teaching- that your lesson materials, be they ICT or pen-and-paper, must be designed carefully:
    - with scaffolding: with feedback at first, then gradually less, and then nothing so they ‘stand on their own to feet’
    - feedback that goes further then wrong/right
    - use an open,expressive environment where they can actually have different strategies.
    - include nonstandard tasks so ‘the algorithm doesn;t work’. Many people have written about this: cognitive conflicts, impasses, productive failure, crises

    Think Buchberger (1990) openbox/whitebox. Think Beeson’s glassbox. Two worlds can co-exist: not everything should be hour-long rich open-ended classroom discussions, not everything should be ICT (note that I think computer labs should go any way).
    Mix and match.

    Re writing on pen and paper: I wouldn’t mind formula recognition on a tablet, connected to good pedagogical software. Hopefully one of the PhD students is able to make this in the coming year(s).

  27. on 29 Jan 2013 at 6:52 amBrendan Murphy

    Dan

    Attention to those kinds of patterns is counterproductive.

    Is anyone asking the question, “Why are students attempting the much more difficult task of looking for patterns in test questions instead of grade level math?”

    I get the idea that students don’t understand math, there is a large measure of guesswork, and students have to fall back on something, but why do they feel more comfortable figuring out linguistic patterns on the fly instead of math procedure?

    I suppose there are some obvious answers, but perhaps not.

  28. on 29 Jan 2013 at 8:24 amMichael Pershan

    Is anyone asking the question, “Why are students attempting the much more difficult task of looking for patterns in test questions instead of grade level math?”

    Isn’t that evidence that, for these students, grade level math is more difficult?

  29. on 29 Jan 2013 at 1:48 pmcrazedmummy

    I’m not sure about difficulty being the factor. I suspect that figuring out the puzzle is what we are programmed to prefer, and students prefer to figure out puzzles, which is human work, than to repeat rote work that can be easily done with a calculator.

  30. on 30 Jan 2013 at 6:43 amBrendan

    Phil Daro discusses this “answer getting”
    http://vimeo.com/30924981

  31. on 30 Jan 2013 at 6:45 amBrendan

    Michael I don’t think it is more difficult, but I think for many they have very little faith in their math abilities.

  32. on 31 Jan 2013 at 5:38 amIan Rae

    Phil Daro’s point is that teachers are promoting “answer getting”. So this is much larger than Khan or IPI; it’s endemic in educational practice.

  33. on 31 Jan 2013 at 11:50 pmZack Miller

    Dan,

    I’ll stay out of this interesting discussion on pattern-matching for now, but I’d like to set the record straight on Summit SJ. Our math model as described as concisely as possible: students spend two hours per day on math; one hour in breakout rooms and one hour in the big room (seen in your picture) where students are working independently. In the breakout rooms, students work on challenging tasks and projects (many of which we can thank you for) that develop the standards of math practice, often in groups and with varying amounts of teacher structure. Development of cognitive skills via frequent exposure to these types of tasks is paramount to our program. It is also in the breakout rooms where students’ independent work – which is mostly procedural practice – is framed and put in context. Students’ know that their work in the big room supports what they do in the seminar rooms and vice versa.

    I have no qualms with resources like Khan Academy or textbooks with answer keys that allow students to work relatively independently (although the big room still has 3 teachers available) and most importantly frees up my time to work with students on math practice and cognitive skill development. Like you, we see huge potential in tech that helps students further develop in these areas, either by giving good, actionable feedback to students and/or by giving data to teachers who can then make appropriate instructional decisions. In other words, please keep up your work with Dave Major so we can stop using Google Forms and various other makeshift tools!

  34. on 01 Feb 2013 at 1:03 amDan Meyer

    Thanks for the background info on your program, Zack. I went ahead and added it to the post above. It’s good to hear you guys are posing the procedural development as a necessary component of larger, more interesting problems.

  35. on 04 Feb 2013 at 9:51 pmLisa Nussdorfer

    Plugging away with a student this morning through their comprehension check on the computer generated Algebra I lesson. Student picked answer “d.” I inquired how the student came to the answer. It was an interesting method that involved multiplying the coefficient of a term by its exponent. In other words, he was lucky.

  36. on 05 Feb 2013 at 11:38 amJason Dyer

    It was an interesting method that involved multiplying the coefficient of a term by its exponent. In other words, he was lucky.

    @Lisa: Did the method work just on the one problem (which can happen with low-tech teaching), or in general (which is a flat-out software bug and the sort of pattern matching that is bad)?

  37. [...] haven’t achieved mastery). People at the Khan Academy have recognized the opportunity for pattern recognition as a problem in online learning, and there will no doubt continue to be work to solve this problem. Cheating is a deeper concern [...]

  38. on 08 Apr 2013 at 4:58 pmAngie Sink

    I am a 45 year old college student, who recently passed college algebra course using khan academy. I could not have passed without his knowledgeable math videos. I would love to meet Sal in person to extend my gratitude.

    I have always struggled in math, but with his wonderfully detailed step by step videos I was able to make a ‘B’ in college algebra, and let me tell you, I had previously took this class twice and dropped both times because I could not understand, and the teachers didn’t have time to show me each step in detail.

    I truly believe that Khan academy is one of the best math websites out there for learning.