What Students Do (And Don’t Do) In Khan Academy, Ctd.

My analysis of Khan Academy’s eighth-grade curriculum was viewed ~20,000 times over the last ten days. Several math practice web sites have asked me to perform a similar analysis on their own products. All of this gives me hope that my doctoral work may be interesting to people outside my small crowd at Stanford.

Two follow-up notes, including the simplest way Khan Academy can improve itself:

One. Several Khan Academy employees have commented on the analysis, both here and at Hacker News.

Justin Helps, a content specialist, confirmed one of my hypotheses about Khan Academy:

One contributor to the prevalence of numerical and multiple choice responses on KA is that those were the tools readily available to us when we began writing content. Our set of tools continues to grow, but it takes time for our relatively small content team to rewrite item sets to utilize those new tools.

But as another commenter pointed out, if the Smarter Balanced Assessment Consortium can make interesting computerized items, what’s stopping Khan Academy? Which team is the bottleneck: the software developers or the content specialists? (They’re hiring!)

Two. In my mind, Khan Academy could do one simple thing to improve itself several times over:

Ask questions that computers don’t grade.

A computer graded my responses to every single question in eighth grade.

That means I was never asked, “Why?” or “How do you know?” Those are seriously important questions but computers can’t grade them and Khan Academy didn’t ask them.

At one point, I was even asked how m and b (of y = mx + b fame) affected the slope and y-intercept of a graph. It’s a fine question, but there was no place for an answer because how would the computer know if I was right?

So if a Khan Academy student is linked to a coach, make a space for an answer. Send the student’s answer to the coach. Let the coach grade or ignore it. Don’t try to do any fancy natural language processing. Just send the response along. Let the human offer feedback where computers can’t. In fact, allow all the proficiency ratings to be overridden by human coaches.

Khan Academy does loads of A/B testing right? So A/B test this. See if teachers appreciate the clearer picture of what their students know or if they prefer the easier computerized assessment. I can see it going either way, though my own preference is clear.

About 

I’m Dan and this is my blog. I’m a former high school math teacher and current head of teaching at Desmos. More here.

17 Comments

  1. It’ll be interesting to me to see what happens with SBAC’s computer-AI scoring of students’ written work. Is that only for the ELA test, or also the math?

    It’s not ideal but perhaps Khan could offer that or something similar, such as sample responses at varying degrees of understanding.

  2. While I agree that this niche of math help sites must address the problem of asking only knowledge questions, I am more concerned with which group of students will benefit from such a system. I think most companies, districts and teachers are hoping to use online math practice sites to help those struggling in math, but I have yet to see a system that actually “hooks” the student to want to engage on Khan Academy, Mathspace or similar. If I had to guess, students who are already in the middle to upper levels of their class are on these sites rather than those struggling.

    So while I think improving the types of questioning is important, the most urgent area in need of improvement would be engagement – not simply adding badges, points, etc., but rather questions that are interesting to solve.

    Until then, I predict these sites will continue to help those who are already interested in math or at least their grade, and not necessarily show any sort of measureable improvements for those who we are aiming to help most.

  3. Khan Academy started has a placed to watch math lectures and listen to someone explain examples. Based on that origin, I have been biased against KA. My question, why do we need a lecture to learn math? Carefully scaffolded activities, with group roles and structure create an environment where lectures are rarely, if ever needed.

    That being said, I have seen that KA has improved their lessons since their birth, and offer more than just lectures. I would recommend it to a student who shows great skills in thinking about math concepts but struggles with calculations. Never for the reverse.

  4. Hey Dan,

    This is great stuff; really getting to the heart of what purposeful, meaningful, and useful learning is.

    On the idea of learners writing text explanations of their thinking and sending it to someone who’s never met them, I think this is an improvement on “answer only” tasks but still extremely limited. Not all learners are particularly adept and putting their thoughts into words unaided (If they could, they should be Math teachers themselves). That’s where peers and teachers can help but they need to “connect” with each other in real time to provide that support and assistance (online, to-and-fro text-based message exchanges are just too problematic). This idea of support and assistance, AKA “instructional scaffolding”, was a term first coined by Jerome Bruner but implicit in Vygotsky’s work.

  5. This approach to learning, of having a kid face a computer and do math, and the computer responds reminds me an awful lot of the Skinner Box.

    This iteration is just for children, not babies.

    It has been my experience, working in urban public schools in 3 different states since 1995, that children from working class backgrounds want a personal relationship with the adults who are teaching them or they will rebel against learning.
    (Kids from upper middle class and wealthy classes seem to benefit from personal relationships with their teachers, too.)

    Did you notice that, too, Dan, in your years?

    While written responses graded by a human coach are better than multiple choice graded by a computer, it won’t satisfy the children’s psychological need for interpersonal contact with caring adults.

  6. I agree with Kyle: the primary beneficiaries of this resource won’t be strugglers, but will be the middle and high achievers.
    My issues are many, but the big two are that a: Sal Khan states and the many, many videos hammer home his philosophy that the way you learn to “understand” math concepts is to do a whole lot of problems. The instruction is extremely focused on procedure, procedure, procedure.
    There are scarce visuals used and when they are, they’re used poorly. Basic videos assume more advanced knowledge, and then there are the places where you’re tested on something completely omitted in the instruction (the video on rectangle areas was followed with questions about the area of triangles). When he scrawls rectangles to show area, the one with the bigger area is a smaller rectangle. Hey, I’m not asking for scale, but how about some faint resemblance to the concrete? Oops, math is all about (in the Khan Academy world) manipulating symbols.

    My second one is his complete disdain for things like experience and prior knowledge with teaching students. The job descriptions ask for somebody with good “intuition” about ways to present math. Believe it or not, there really are some non-intuitive nifty ways to present things (like bridging from concrete to abstract to develop concepts and not requiring algebra with variables to figure out averages)!

  7. sjones:

    My second one is his complete disdain for things like experience and prior knowledge with teaching students. The job descriptions ask for somebody with good “intuition” about ways to present math. Believe it or not, there really are some non-intuitive nifty ways to present things (like bridging from concrete to abstract to develop concepts and not requiring algebra with variables to figure out averages)!

    Good point. I find that cult of amateurism to be one of the most irritating aspects of the ed tech industry. We’ve worked hard to figure out even a little bit of how this learning stuff works. Why throw that away? Signed, Person in a Doctoral Program.

  8. Computer based assessments are one thing. I don’t think KA is selling itself as an assessment. As as platform for teaching and learning it fails. You cannot ignore the fact that there is a social aspect to learning. You learn better when you can discuss ideas with other people. Sure you can mimic a procedure, but for conceptual understanding there has to be a development and refinement of ideas. As Joel Patterson mentions above, students need and benefit from the interpersonal contact with teachers.

  9. Dan,

    ===
    A computer graded my responses to every single question in eighth grade.

    That means I was never asked, “Why?” or “How do you know?” Those are seriously important questions but computers can’t grade them and Khan Academy didn’t ask them.
    ===

    If Khan or anyone else were to use “hinge point questions” [http://goo.gl/3j2hnh], would this not provide a way (of sorts) of machine-marking multiple choice questions that are of the “Why?” or “How do you know?” type?

    Seb Schmoller

  10. Hi Seb, a company called Zaption does an effective job embedding hinge point questions into YouTube videos. But hinge point questions that take the form of multiple-choice questions, however competently they anticipate distractors, constrain student thinking and discourse in ways that’ll always count against them in my ledger.

  11. What platforms like Khan Academy do effectively is assess students’ basic understanding of foundational concepts and feed progress to parents/teachers, giving visibility to each child’s individual strengths and weaknesses. Assuming reliability in this basic form of measurement, it can guide educators to probe content in greater depth with their students.

    Rich problem-solving as described above is not yet assessable in an automated way, which positions these platforms as a supplemental learning tool; an enabler of deeper and more meaningful learning experiences.

    I welcome the suggestion of making students’ thinking visible as far as possible, even – indeed, especially – if it can not be automatically evaluated. This serves the widely stated purpose of educational technologies as fostering deeper connections between teacher and learner.

    (Full disclosure: I work for Whizz Education; an online adaptive math tutor for 5-14 year-old students. Alongside the tutor we provide a suite of Teachers’ Resources as a way of recognising and supporting the need to push beyond the core knowledge and skills we are able to capture through the automated tutoring model.)

  12. As a teacher who tries to use Khan Academy with her students (and fails, because students are resisting using it) I wonder if the next step of development for KA could be to develop the coach aspect so that it enables a dialoague between coach and student regarding the concepts and questions the student is working on. In that way, KA would be supporting teachers – rather than trying to replace them – a variety of tasks, some of which could be very open ended and rich. KA could conceivably even allow coaches to design their own tasks, to supplement the ones provided by KA. As an educator, I would welcome such developments above further attempts to increase the scope of automatic assessment.

  13. Hi Junaid,

    I think your claim:
    What platforms like Khan Academy do effectively is assess students’ basic understanding of foundational concepts
    >>>
    is what Dan and others here are questioning.

    Are you making the claim that “basic understanding of foundational concepts” is the same thing as ‘being able to reliably and correctly compute?

    It is quite obvious that what KA does is assess procedural fluency. Can the student correctly and reliably compute …..? Yes/no?

    But, for me, it’s not at all obvious that what KA does is assess understanding. I generally think of understanding of as being able to address the ‘why’ questions that Dan has noted are missing. This is true of even the most foundational concepts [counting, addition, …]

  14. Very fun analysis, in both this post, and your previous one. I also love data driven analyses like these. But I’ve also worked on EdTech products that involve both automatic and non-automatic grading. I’m all about improving teaching practices and pedagogy.

    But I have to disagree with you at one point:”In my mind, Khan Academy could do one simple thing to improve itself several times over:

    Ask questions that computers don’t grade.”

    In fact, I think that would severly make the product worse. It’s easy to just say “Link students to coaches to do non-machine grading!” but you are calling for a tremendous amount of work in that statement, and are basically calling for the KA to turn into an entirely different kind of product. Additionally, if we agree coaching is a good thing, is virtual coaching through a website the best way to do it? Maybe an in-person coach would be even better? So maybe the KA should provide those- oh, yeah, those are teachers and they already exist. And that’s also why Khan has called for teachers to supplement KA with teaching, and not use it on its own. SBAC can make non-machine graded items (i.e. writing in an answer with words), because they have huge resources that they are able to put towards human grading those items, not become they have better technology.

    To come back to the original point- I think technology products should always focus most on what technology products are made to do well. They shouldn’t try to focus on doing replicating an interaction or on higher-order skills that might best occur face-to-face. But we as educators should also not be solely relying on technology to provide all of education’s needs. There’s no silver bullet here- even CC itself argues for an integrative approach of tackling a problem from different angles.

  15. Kevin:

    you are calling for a tremendous amount of work in that statement, and are basically calling for the KA to turn into an entirely different kind of product.

    True.

  16. It’s not as restrictive as an organic foods
    restaurant, because not all of your ingredients have to be ‘certified organic.
    For women, you can include jewelry like a pair of earrings that consist of clusters of peridot or amethyst to
    evoke the sense of luscious green or black grapes.

    Another German entrepreneur converted his creative skills of engraving
    toothpicks, to start an uncommon business selling
    engraved toothpick souvenirs.

  17. As an early childhood teacher, I am particularly troubled with this level of instruction from KA. I am also troubled by the near Carte Blanch acceptance of this platform by administration and the public in general. My emerging theory on this acceptance is that many adults don’t feel all that comfortable with their level of mathematical understanding (kind of lots of research supporting this!), so when they view the videos they are coming with their own background knowledge and the videos fill those gaps. So the light goes on as they say, “Wow…ok, now I understand it!” (See Joel’s hypothesis above).

    As teachers we OFTEN forget we look at students’ understanding from our own point of view; we assume there is this basic knowledge, so of course what we present will make sense. Then we teach a procedure; students get a correct response (algorithms work, after all), and we move on. This works pretty well for most multiple choice tests…but then when we ask for an explanation, critical thinking, or a logic based problem is presented…kaboom. Then we cry, “These students don’t have number sense!” Nope, they don’t because we never taught it at a deep level. Why? Too many reasons to list here. I pray Common Core can help alleviate some of this lock-step memorizing the procedure thinking. I am afraid it won’t because the curriculum to teach this way isn’t out there in full force (yet!).

    I don’t know where we are headed in technology in education, specifically in math. (By the way, technology is a really irritating word to me…a pencil is a piece of technology.) I believe there is great hope with adaptive AI cognitive tutors based on the learning sciences like perceptual control theory, flow, skill theory, and Bayesian knowledge tracing. God bless the developers who want to take this on in a world of crow-sourced apps that have all the bells and whistles crying out like the lights of Vegas. My cry is for VCs to invest in companies producing this type of work. Gates has given $$$$ to Kahn…and the truth is, I think if they will be honest and self-reflective, they have the ability to design such products.

    Dan, I like your work with gaming and instruction…it takes a skilled teacher to scaffold the dialogue that will take a student from merely playing a game to dissecting the learning.

    You might want to check out this blog (yes, I responded there too!) as well as this EARLY number sense, narrow scope of instruction, math app. I have used it and have conducted action research in several classrooms of my students. Waiting for developers to fund follow on apps (NO, I do not work for them or represent them…just giving it my teacher EXPERT stamp of approval!) (Blog is two parts)

    http://theeducationscientist.blogspot.ch/2014/02/khan-academy-and-illusion-of.html#.VLSxZivF-Sp

    http://www.nativebrain.com