Bill Gates Just Put A Hit Out On John Golden And David Coffey

This strikes me as a really, really effective way to assess the pedagogical content knowledge of new teachers: critique the pedagogy of the Khan Academy video of your choice. You could write an essay and add timecodes for reference or you and a friend could sit in front of the screen MST3K-style and snark your way through Khan’s lecture like John Golden and David Coffey.

I’m really curious how the Church of Our Lady of Technology in Silicon Valley will react to this kind of critique. That church tends to write off most educators’ criticism of Khan Academy as some admixture of jealousy and entrenchment. They aren’t always wrong about that. But the criticism that “this is actually fairly poor lecturing that’ll leave students with shaky procedural understanding and even shakier conceptual understanding” is much harder to refute. It’s also a difficult criticism to illustrate for people who aren’t teachers. This is the best illustration of that critique I’ve seen.

BTW. The low-rent production values don’t do justice to the quality of their concept and critique, though. Thirty dollars on sound equipment would go a long way towards making this a series math supervisors around the US would make required viewing for their inservice teachers.

2012 Jun 20. Kent Haines has eagle eyes and points out that Khan Academy pulled their video within a couple hours of this post. Christopher Danielson asks the right question, I think. Are they pulling the video to correct the mathematical errors, the pedagogical errors, or both. It’s one thing to mistakenly refer to the transitive property when you mean the commutative property. It’s another to teach students that multiplying integers requires the memorization of a bunch of rules that look like magic but just memorize them because okay?

2011 Jun 21. I had high hopes for that comments thread but it wobbled off course pretty fast.

2011 Jun 22. A reader e-mailed asking what kind of audio setup I’d recommend. Here’s what I wrote back:

There are lots of configurations that’ll serve our needs here and probably several that are cheaper or less cumbersome than the one I use to record audio of myself in presentations and lectures. Lately, though, I record video using whatever I have on hand. Then for audio I use:

Then I sync the audio and video in post. Here’s a video explaining the setup.

2012 Jun 22. Khan Academy has re-uploaded the video and the difference is stark. The new version is oriented towards conceptual understanding whereas the last offered you the bare minimum necessary to pass a multiple-choice test or keep your teacher and parents off your back.

I'm Dan and this is my blog. I'm a former high school math teacher and current head of teaching at Desmos. He / him. More here.


  1. I watched this, enjoyed it, and agree that it is a good illustration of the problems with KA. However, I think a non-educator audience would have trouble appreciating the pedagogical nuances they are pointing out. I just can’t see a policymaker being significantly impacted by this.

    However, the idea for new teachers and grad students to dissect the videos for bad pedagogy is one that could really make a difference for first years who focus on improving their lectures and haven’t yet learned what type of teacher they would like to be.

  2. THANK YOU for sharing this! This is the conversation that needs to take place in a rational manner. There are many aspects of the Khan videos that reflect very poor mathematical teaching (and poor mathematics). Somehow discussion around the Khan videos has become linked to pro or con Bill Gates and corporate “reform” and not at all about the actual “instruction” seen in the videos. I am completely baffled as to how Khan’s “instruction” has garnered so much attention and so many accolades – other than the fact that he has done all this “for free.”

  3. My post on John Golden’s blog:
    “I love it. At 5:40, Kahn says that -4 x 3 = -12 because it’s essentially saying it’s -4 times itself 3 times. Sorry Kahn, the result of what you said would be -64. Sigh!
    Sorry Bill Gates, I disagree with your assessment of Kahn.
    I hope Kahn academy is a one-minute wonder. Let’s find real math rock stars. They exist!
    Dan Meyer. Frank Noschese. Shawn Cornally. Fawn Nguyen. Etc.”

  4. I have an idea. There are thousands of Khan Academy videos. Maybe this particular video is just a poor example, and all of the other videos are really good.

    Maybe we can crowd-source some research. Each of us takes 1 or 2 videos and points out both the good and bad pedagogical aspects of the videos, and then aggregates the results somehow.

    I think it would be fun to have an army of video critiques of each of Khan’s videos, but it would be much more useful from a research point of view to qualify these videos using some sort of standardized protocol (kind of boring, but this might be more useful for future critiques).

    We could take the next step and generalize this protocol so that we can critique any video that we find fits the format of our critique (it would probably be difficult to take a video that is asking questions of the viewer rather than listing information and make a useful comparison). Further, it is likely that such a protocol exists already, possibly as a result of the TIMMS video study (I checked – they have a protocol they used).

  5. Hilarious. Loved mystery science theater so this was just classic.

    I too am baffled by the power of Khan (loved the Star Trek Khan reference in the video, by the way), as if it is some miracle for education when in fact, it’s just lecture, and rather uninspired lecture at that. S. King makes a good point that it is probably tied to the ‘free’ factor, the Bill Gates support, and additionally, I think, the standardized testing frenzy and hysteria surrounding education. Gotta get those math scores up, so miracle cure is videos that teach memorization of facts! (As this video points out, often time incorrect facts). Everyone wants a quick fix rather than addressing the real issue, which is mathematics education needs to be taught differently – not replacing classroom lecture with video lecture. Replace the lecture!

  6. Loved MST3K as a kid growing up so had to watch this, oh the memories. I can’t believe he would leave something so terrible up for the public to view. Am I the only one that cringes every time I hear him says minus 4 instead of negative 4. Is he teaching subtraction or multiplication?

  7. The best thing KA has done is show us teachers that we need to be on the internet, making better resources than KA. There are already lots of “how to solve this type of problem” videos, and of course there are 101questions prompts, and Veritasium’s videos, to name only a few things teachers/educators have been doing. So, let’s keep going, and take our criticism and make corrections and edits, and make good, useful stuff that helps our kids learn meaningful things!

  8. I agree it wasn’t the best video, but two thoughts.

    1. Is snarky the best approach here? Would a less-snarky, more pedagogically-focused critique be more effective? Not sure, but my fear is that snarky turns people off.

    2. I think this begs another question, though. What if we did critique this video (or others like it), and continued to refine it until it was a video with reasonably sound pedagogy and better procedural understanding – would that video have a place in our arsenal?

  9. Kathy Sierra

    June 20, 2012 - 3:14 pm -

    When you have actually taught a topic to actual humans, you discover all the tiny glitches, brain bugs, “wait, what?” moments that deplete a learner’s scarce, finite cognitive resources — the resources we NEED if they are to focus and learn. I am beginning to think of KA as Death By A Thousand Tiny Cognitive Cuts, and that’s for the ones that are *technically* correct. It’s those million seemingly small things like inconsistent use of + sign when discussing positive and negative numbers, for example. Each moment a learner spends wondering if that inconsistency MEANS something is a moment that could/should have even spent on what is important. And boy, did those cognitive paper cuts pile up in this video.

    Of course this is no different from good product UX and UI design… every moment of a user’s precious cognitive resources (and time) should be spent doing the thing they want to do, not figuring out how to tell the interface how to do the thing they want to do. KA might read “Don’t Make Me Think” for inspiration and guidance, given that what the Web design classic means is “Don’t make me think about the wrong things”

  10. Notice how often he’s sloppy with his vocabulary. Transitive/communtative (sic)/commutative.

    I had an administrator tell me that Kahn is good because its “interactive and engaging”.

    Face palm.


  11. I agree with Karl’s first point – I think that the snarkiness is going over well here because readers of this blog are likely to have the same opinions to begin with, but I worry that the snarkiness of it could backfire with other audiences. If the snarkiness prevents non-likeminded people from actually listening to the insightful criticisms, then the video’s just preaching to the choir.

    I research math education as my job, but I’m not a teacher, and while I did find the insightful parts of the video to be good, I was also really turned off by the amount of snark. (I get that this video was riffing on Mystery Science Theater and that the amount of snark is probably meant to be tongue-in-cheek, but I know that the reference would have gone over the heads of a lot of my acquaintances, meaning that the snark would just come across as overwhelmingly mean-spirited.) For example, when Khan writes out an arrow that looks like “=D positive” and they immediately jump all over it and start guffawing about “what’s D POSITIVE?”, that sort of snark came across to me as irrelevant, unfunny, and even intellectually lazy – it wasn’t the best arrow, but anyone who tried to decipher it for more than a few seconds would likely come around to seeing it as an arrow, and the attitude of making fun of it before even trying to do that was just incredibly off-putting. Like I said above, I think that this type of video would go over well with teachers who are ambivalent or negative on KA, but I have a feeling that some non-educators (esp. ones who aren’t familiar with MST3K and who’ve been exposed to the TEACHER-UNIONS-BAD rhetoric) would just see the video as defensive and technophobic – in other words, potentially making things worse.

    Also, students obviously aren’t the intended audience here, it seems a terrible example to be setting for any students who did come across this — if I’m a student, and I see that a math teacher is making fun of and talking over another teacher’s lecture, I’m going to start thinking that this behavior is acceptable (especially because the Mystery Science Theater reference is almost definitely going to go over the heads of most of today’s students).

  12. Kathy Sierra

    June 20, 2012 - 3:44 pm -

    Rollie, when you say: “…anyone who tried to decipher it for more than a few seconds would likely…” you’re highlighting a crucial difference between teachers and non-teacher. One who’d spent time in front of students would know that in the context of struggling to learn a challenging new thing, those “few seconds” of deciphering ARE a big deal.

  13. It’s usually the well-trained and engaged teachers who are most vocal when criticizing Khan videos. Many of the things pointed out in this critique would be seen if the highly qualified sat in on classrooms of the less qualified. I see it in high school science classrooms all over my state. It’s not about video. I think that is what Dan is getting at when he suggests this be training for new teachers.

    The problem is severe in many states – STEM teachers are not required to have deep content knowledge and that is Sal’s biggest problem. He knows a lot but probably falls short of the content knowledge most of us well-prepared teachers expect of ourselves and our colleagues. He makes the mistakes one sees from incomplete understanding.

    Unfortunately, the general public doesn’t believe that deep content knowledge is needed to teach K-12. The problem is compounded by support from successful people like Gates and Google – people who have succeeded without complete and deep content knowledge gained from study.

    We need to make a better case that the best teachers have deep content knowledge. How is that done most effectively?

  14. @Kathy – right, but like I said, I’m concerned more with how the video would come across to non-teachers. My sense (from reading this and other teacher blogs) is that a lot of math teachers already know the flaws of KA, but are frustrated with how the public is holding it up as the be-all-end-all, so my point was that this sort of video isn’t going to help that problem at all. Even if non-teachers aren’t the intended audience, there’s nothing preventing them from seeing it anyway (like if someone who comes across it and doesn’t understand the context posts it to some news site with a hyperbolic title like “LUDDITE TEACHERS HATING ON KA”) and coming away with the thoughts I outlined above.

    I think this comes back to a point I also wanted to make, about how the MST3K allusion doesn’t work as cleanly as the authors might’ve intended it to. When MST3K snarks all over Manos: The Hands of Fate, we find the snark funny because everyone sees how awful Manos is, so we’re with the MST3K guys from the beginning. But not everyone agrees that KA isn’t all that great, so for people who do like KA, or are ambivalent about it, the snark doesn’t come across as funny, because snarkiness is generally ugly if you don’t agree with it.

  15. Kahn Academy is such a lightening rod.

    I guess I just don’t understand the vitriol, disgust and passion with which math teachers seem wiling to attack things that are different. Is it perceived as a threat? I understand that various administrators may get turned on by KA’s advantages and I’m sure that there are school districts that are investigating ways to formalize its usage; what I don’t see or hear is anyone clamoring to replace classroom teachers wholesale with computer labs. I know that the California university system has made some bold statements and has some interesting plans, but I don’t know of anything similar at the secondary school level. Where is the emotion coming from?

    My point, I suppose, is that the response of the education institution has been disheartening. KA represents something new, a potential paradigm shift in education for some students in some circumstances. I think that’s undeniable. KA doesn’t have to serve every student in every circumstance to demonstrate its value as a delivery mechanism. Is it equivalent to a daily interactions with a perceptive, deeply-skilled, passionate educator? Of course not. But it doesn’t have to be.

    But you can rewind it. You can do practice problems. You can track your progress. You can access a curriculum you might have not have had access to before. As an educator, why not embrace that and try to make the best of it for the students that are in your sphere of influence? I can’t believe that there’s zero (or even negative, as some seem to claim) utility to something like KA for almost every public school teacher in America, to say nothing of an international audience. I can’t disagree with Rick Fletcher’s call for teachers steeped in deep content knowledge; why does it feel like an either/or? KA doesn’t obviate that need.

    Technology adoption is increasing and won’t stop; the sooner we all come to grips with it and how it affects our praxis, the better. Harping on a new tool because it doesn’t exactly replicate classroom best practices and droning on about its drawbacks and limitations feels like a fast way to distance yourself from a huge component of the future of education.

    My girlfriend is asking me to add a personal plea to provide feedback directly to KA, perhaps in a less stylized fashion than the video in this post, to improve the tool and the educational outcomes of everyone who can use it.

  16. Kathy Sierra

    June 20, 2012 - 4:22 pm -

    Rollie, I agree with you about snark. In the context of MST3K it makes sense, but overall I agree. But their video immediately struck me as a specific example of a more general (and I think wonderful) notion of “annotated KA”. And annotated KA could be implemented in a zillion different ways (including snark-free).

    I disagree, though, that this is anuthing like “math teacher making fun of another teacher” (though I suppose it could look that way to one who had somehow never heard of KA). When “the teacher” is the Ted-heralded, Gates-funded, Future Of Education, a whole different standard applies.
    (apologies, Dan, for so many comments)

  17. So I was hopping you could clarify something for me about this post. Specifically the point that “this is actually fairly poor lecturing that’ll leave students with shaky procedural understanding and even shakier conceptual understanding”

    Is the argument here against the idea of video lectures as a tool for learning? Or that his video’s could be better made?

  18. Kathy, I definitely think annotated KA could be great, and hopefully this idea keeps getting revised and polished.

    And sorry, Dan, if this comment thread ends up spiraling into another KA-war, I just meant to comment on the video itself, not about the merits of KA.

  19. I have to agree with Karl and Rollie… the snarkiness only plays well for a niche audience and distracts from the overall point that John Golden and David Coffey are trying to make.

    Is Khan Academy good pedagogy or curriculum? No. But I haven’t heard many people claiming that it is. Does it have to be? It would be nice, but as long as it is “average” — which it is based on my observations — then it is better than the pedagogy and curriculum that many students are currently receiving. It also comes across (I imagine) as non-judgmental; this is a huge advantage when so many struggling students come into class feeling defeated because they feel like their teacher has something against them. Is it annoying that Khan Academy is getting so much attention? Yes, but it has gone mainstream exactly because it is what most people think good teaching looks like, which is highly procedural. It’s not like people hated standards-based math programs until they got a closer look at them. They hated them because they couldn’t relate to them.

    The main advantage of Khan Academy is the flipped classroom model and the fact that students can play the videos over and over again in a safe environment. I personally don’t think it (or any of the online education initiatives which are in favor right now) are a panacea because almost all of them are simply replicating the pedagogy and curriculum that exists in schools today. But at the same time, I think that they are difficult to criticize if you don’t have a better alternative to point to. As a newbie to the Math Edublogosphere, I’m curious to know if there is any kind of consensus developing as to what math education should look like, and where is it being practiced?

  20. I’m shocked they don’t know the difference between “Star Track” and Star Trek. Maybe they should turn that critical eye inward. Glass houses and all.

  21. To be fair, not every teacher knows every conceptual connection to procedures. In other videos (e.g. “Logarithmic Scale” and “Negative Exponent Intuition), Khan does a better job.

    Regarding Gates, it’s not accurate to peg him as an anti-constructivist technogeek. Afterall, he funded Marilyn Burn’s MRI project and a guided math curriculum called JUMP math. He also funded the MAPs project for the Shell Centre (feel free to google these). Clearly, a more complicated view is in order.

    Incidentally, Dan, Week 32 of your algebra curriculum has you presenting completing the square without a single square on your slides. Other slides and handouts show similar gaps in conceptual presentation. Be reasonable.

  22. While I agree that the videos do need more analysis so that the instruction is better, I do enjoy Khan Academy. My 9th-grader nephew has benefited from the site. I propose that more instructors be brought into the fold.

  23. @ Greg – nobody is clamoring to replace teachers with computer labs? Come to Idaho. State Superintendent pushed through laws last year requiring all students must take two online courses to graduate. He wanted 6 (of 48) compromised at 2. You need to get out more.

  24. I also agree with Karl’s and others’ points about the “snarkiness.” That is not helpful. However, I do not think the legitimate concerns about the pedagogy and incorrect use of terminology is something to be ignored. Why is it that if anyone points out those shortcomings they suddenly are pegged as being against any type of different approach? There are plenty of math teachers (Dan is an excellent example) who are doing phenomenal work to explore and use new and effective ways to teach mathematics. And yes, there are many teachers who are entrenched in their teaching approaches and seem unwilling to change; some even teach multiplication of integers as poorly as Khan does in this video. However, those teachers are not being touted as being a shining example of “better teaching” by Bill Gates. To me, an aspect of teaching that has been missing is the critique and discussion about pedagogy and effective instruction. Such a process would be far better than “accountability” based on student performance on high stakes assessments.

  25. You can flip a classroom with a textbook. You can rewind a textbook.

    I’m all for new technology. Video is cool.

    But be accurate people. Don’t ascribe to a new technology advantages that aren’t. It’s just a repeat of the grand educational promises for television when it first appeared.

  26. I’d like to see kids doing more thinking in math classes. I’d like to redefine what it means to learn mathematics, and to talk more about the difference Seymour Papert makes between school math and Mathematics. I’d like to do a host of other creative activities with my students, and of course I can.

    But when it comes to trying to help educators implement these practices at other schools, it feels like I’m swimming upstream against a gigantic current trying to pull me down, and unfortunately, I see the Khan Academy riding this current, and maybe even helping to propel it a bit. The Khan Academy is helping create some momentum behind a very different model of learning mathematics than what I’d like to see. So while I recognize that there are struggles in changing our current model, and perhaps very little progress will be made any time soon, if something is going to be huge and helpful in education, I’d certainly prefer it was using my definition of Mathematics as opposed to math.

    Why couldn’t a site that promoted exploration, inquiry, and thinking in mathematics become huge? Why did it have to be the didactic model? Is it not possible to think outside the box a bit more?

  27. Pieter Kuiper

    June 20, 2012 - 11:37 pm -

    The Khan video was made private, but the comments are still there: It is interesting to read teactions by students and teachers over the last year.

    Some of these comments deal with what is behind the rules, some of these try to give detailed explanations (which is a bit difficult without figures), someone links to a web page at that does a much better job than the Khan video.

    I am not a math teacher. And I confess that I liked the snarkiness.

  28. Doug, Gates being labeled as anti-constructivist probably comes from him being quoted in Wired:

    JUMP Math doesn’t build a case for Gates as embracer of constructivism. There is no way this program could be described as constructivist. For example, take a look at the Grade 3/4 introduction to fractions material:

    Adding and subtracting fractions is broken down into seven incremental steps to be practiced by students. The first three of these steps are:

    1. “Write the times signs beside the fractions” (I’m not making this up)
    2. “Switch the bottom numbers” (How’s that for developing understanding?)
    3. “Write the times signs and switch the numbers” (Now that you’ve mastered both)

    If I were attempting to create a satire of what math education looks like for far too many of our students, I’d be hard-pressed to top what can be found in JUMP. This is not respectful of the mathematical thinking that children are capable of. The approach here is similar to Khan’s video on multiplying and dividing integers: both promote the memorization of rules rather than conceptual understanding.

    As Christopher suggested, the video wasn’t pulled because of pedagogy. It was removed because of math errors. With respect to these errors, it might be a bit snarky to nit-pick at things like ‘minus’ instead of ‘negative’. Like many teachers, I catch myself saying things like this from time to time and have to correct myself in class. Thankfully for me, these moments aren’t broadcast on YouTube. Still, I don’t have a problem with snark if one is being held up as the lone stranger riding into town on horseback (or bicycle) to save the townsfolk.

    Let’s ignore the criticisms from the constructivists for a moment. Let’s also ignore the nit-pickers. For me, as examples of direct instruction, Khan’s videos just don’t hold up. As John and David pointed out, is there a worse example to start with than -2 times -2? This lack of basic preparation appeals to some who think that it adds charm to the Sal Khan story. To many math teachers like myself, it’s alarming, not charming. If I observed a teacher do the same thing, I’d say s/he should plan more and wing-it less.

  29. Jared beat me to the punch: you can rewind a textbook.

    To expand, you can find practice exercises in a textbook. You can track your progress in a textbook. A good textbook is non-judgemental (but God condemn the author who writes “it follows easily that”). Free textbooks are also available online (some better than others).

    Sure, Khan’s videos are all sorted and in nice “bite-sized” chunks. But they are mostly sorted by procedure — the organization of the site itself helps inhibit the forming of the mental organization that will help our students. This is reinforced when he doesn’t explain the ideas behind why the procedure works.

    I agree that video has its place. A few students have severe cognitive impairments that making reading the textbook completely out of the question. But when I poll my incoming first-year math majors, 98% or so of them have never read an entire section in a single math textbook! Never, and they are mostly planning to become actuaries, math teachers, and mathematicians! The simply CANNOT READ technical content. Sending students to Khan is easy, but does nothing to fix this problem.

    I’ll part with this video:

    I’m surprised at how many people think the guy just doesn’t know how to square and add. Of course he can! The problem is that neither he nor the audience can read.