tl;dr — Khan Academy claims alignment with the Common Core State Standards (CCSS) but an analysis of their eighth-grade year indicates that alignment is loose. 40% of Khan Academy exercises assessed the acts of calculating and solving whereas the Smarter Balanced Assessment Consortium’s assessment of the CCSS emphasized those acts in only 25% of their released items. 74% of Khan Academy’s exercises resulted in the production of either a number or a multiple-choice response, whereas those outputs accounted for only 25% of the SBAC assessment.
Introduction
My dissertation will examine the opportunities students have to learn math online. In order to say something about the current state of the art, I decided to complete Khan Academy’s eighth grade year and ask myself two specific questions about every exercise:
- What am I asked to do? What are my verbs? Am I asked to solve, evaluate, calculate, analyze, or something else?
- What do I produce? What is the end result of my work? Is my work summarized by a number, a multiple-choice response, a graph that I create, or something else?
I examined Khan Academy for several reasons. First, because they’re well-capitalized and they employ some of the best computer engineers in the world. They have the human resources to create some novel opportunities for students to learn math online. If they struggle, it is likely that other companies with equal or lesser human resources struggle also. I also examined Khan Academy because their exercise sets are publicly available online, without a login. This will energize our discussion here and make it easier for you to spotcheck my analysis.
My data collection took me three days and spanned 88 practice sets. You’re welcome to examine my data and critique my coding. In general, Khan Academy practice sets ask that you complete a certain number of exercises in a row before you’re allowed to move on. (Five, in most cases.) These exercises are randomly selected from a pool of item types. Different item types ask for different student work. Some item types ask for multiple kinds of student work. All of this is to say, you might conduct this exact same analysis and walk away with slightly different findings. I’ll present only the findings that I suspect will generalize.
After completing my analysis of Khan Academy’s exercises, I performed the same analysis on a set of 24 released questions from the Smarter Balanced Assessment Consortium’s test that will be administered this school year in 17 states.
Findings & Discussion
Khan Academy’s Verbs
The largest casualty is argumentation. Out of the 402 exercises I completed, I could code only three of their prompts as “argue.” (You can find all them in “Pythagorean Theorem Proofs.”) This is far out of alignment with the Common Core State Standards, which has prioritized constructing and critiquing arguments as one of its eight practice standards that cross all of K-12 mathematics.
Notably, 40% of Khan Academy’s eighth-grade exercises ask students to “calculate” or “solve.” These are important mathematical actions, certainly. But as with “argumentation,” I’ll demonstrate later that this emphasis is out of alignment with current national expectations for student math learning.
The most technologically advanced items were the 20% of Khan Academy’s exercises that asked students to “construct” an object. In these items, students were asked to create lines, tables, scatterplots, polygons, angles, and other mathematical structures using novel digital tools. Subjectively, these items were a welcome reprieve from the frequent calculating and solving, nearly all of which I performed with either my computer’s calculator or with Wolfram Alpha. (Also subjective: my favorite exercise asked me to construct a line.) These items also appeared frequently in the Geometry strand where students were asked to transform polygons.
I was interested to find that the most common student action in Khan Academy’s eighth-grade year is “analyze.” Several examples follow.
- Instead of just asking for the solution of a system of linear equations, for instance, Khan Academy asks the student to analyze how many solutions the system would have.
- Instead of just graphing a function, Khan Academy asks the student to draw conclusions from the graph of a function.
- Instead of just asking students to create a table, Khan Academy presents the table and asks students to draw conclusions.
Khan Academy’s Productions
These questions of analysis are welcome but the end result of analysis can take many forms. If you think about instances in your life when you were asked to analyze, you might recall reports you’ve written or verbal summaries you’ve delivered. In Khan Academy, 92% of the analysis questions ended in a multiple-choice response. These multiple-choice items took different forms. In some cases, you could make only one choice. In others, you could make multiple choices. Regardless, we should ask ourselves if such structured responses are the most appropriate assessment of a student’s power of analysis.
Broadening our focus from the “analysis” items to the entire set of exercises reveals that 74% of the work students do in the eighth grade of Khan Academy results in either a number or a multiple-choice response. No other pair of outcomes comes close.
Perhaps the biggest loss here is the fact that I constructed an equation exactly three times throughout my eighth grade year in Khan Academy. Here is one:
This is troubling. In the sixth grade, students studying the Common Core State Standards make the transition from “Number and Operations” to “Expressions and Equations.” By ninth grade, the CCSS will ask those students to use equations in earnest, particularly in the Algebra, Functions, and Modeling domains. Students need preparation solving equations, of course, but if they haven’t spent ample time constructing equations also, those advanced domains will be inaccessible.
Smarter Balanced Verbs
The Smarter Balanced released items ask comparatively fewer “calculate” and “solve” items (they’re the least common verbs, in fact) and comparatively more “construct,” “analyze,” and “argue.”
This lack of alignment is troubling. If one of Khan Academy’s goals is to prepare students for success in Common Core mathematics, they’re emphasizing the wrong set of skills.
Smarter Balanced Productions
Multiple-choice responses are also common in the Smarter Balanced assessment but the distribution of item types is broader. Students are asked to produce lots of different mathematical outputs including number lines, non-linear function graphs, probability spinners, corrections of student work, and other productions students won’t have seen in their work in Khan Academy.
SBAC also allows for the production of free-response text while Khan Academy doesn’t. When SBAC asks students to “argue,” in a majority of cases, students express their answer by just writing an argument.
This is quite unlike Khan Academy’s three “argue” prompts which produced either a) a multiple-choice response or b) the re-arrangement of the statements and reasons in a pre-filled two-column proof.
Limitations & Future Directions & Conclusion
This brief analysis has revealed that Khan Academy students are doing two primary kinds of work (analysis and calculating) and they’re expressing that work in two primary ways (as multiple-choice responses and as numbers). Meanwhile, the SBAC assessment of the CCSS emphasizes a different set of work and asks for more diverse expression of that work.
This is an important finding, if somewhat blunt. A much more comprehensive item analysis would be necessary to determine the nuanced and important differences between two problems that this analysis codes identically. Two separate “solving” problems that result in “a number,” for example, might be of very different value to a student depending on the equations being solved and whether or not a context was involved. This analysis is blind to those differences.
We should wonder why Khan Academy emphasizes this particular work. I have no inside knowledge of Khan Academy’s operations or vision. It’s possible this kind of work is a perfect realization of their vision for math education. Perhaps they are doing exactly what they set out to do.
I find it more likely that Khan Academy’s exercise set draws an accurate map of the strengths and weaknesses of education technology in 2014. Khan Academy asks students to solve and calculate so frequently, not because those are the mathematical actions mathematicians and math teachers value most, but because those problems are easy to assign with a computer in 2014. Khan Academy asks students to submit their work as a number or a multiple-choice response, not because those are the mathematical outputs mathematicians and math teachers value most, but because numbers and multiple-choice responses are easy for computers to grade in 2014.
This makes the limitations of Khan Academy’s exercises understandable but not excusable. Khan Academy is falling short of the goal of preparing students for success on assessments of the CCSS, but that’s setting the bar low. There are arguably other, more important goals than success on a standardized test. We’d like students to enjoy math class, to become flexible thinkers and capable future workers, to develop healthy conceptions of themselves as learners, and to look ahead to their next year of math class with something other than dread. Will instruction composed principally of selecting from multiple-choice responses and filling numbers into blanks achieve that goal? If your answer is no, as is mine, if that narrative sounds exceedingly grim to you also, it is up to you and me to pose a compelling counter-narrative for online math education, and then re-pose it over and over again.
49 Comments
Dan Meyer
December 4, 2014 - 8:44 am -Here I am livetweeting the experience.
Jeremy Babel
December 4, 2014 - 9:00 am -Thank you for taking the time to perform this analysis. Very insightful. There may be some good uses for KA but primary method of instruction is not one of them.
Bowen Kerins
December 4, 2014 - 9:03 am -Whatever you do, don’t give that man an asymptote.
Do you think any of the difference can be accounted for by the number of exercises required? If SBAC had to show 400+ problems, including multiple problems for each standard, would those 400 problems still have the same variety of action and production?
Dan Meyer
December 4, 2014 - 9:08 am -Bowen:
Maybe. But when you marry novel technology and good pedagogy, you can run and re-run that combination quite a few times to useful effect, as I found in the transformations exercise set.
I also nearly included the following in my limitations: “Who knows if these problems are representative of SBAC’s complete exam?” Maybe they’re only showing the flashiest, Common Core-iest stuff. But I reasoned that would undermine to a serious degree the point of releasing items. These items mean to tell stakeholders (teachers, students, etc.) “this is what’s coming.”
Kathy Mytton
December 4, 2014 - 9:12 am -Thank you Dan, I have always liked the KA as a skill based tool for a study session tutorial based on skills, however, I agree, it did not allow students to critically think.
Steve Cates
December 4, 2014 - 9:18 am -I agree that it’s largely a picture of what can be done on the computer in 2014 but at the same time I wouldn’t expect that the production would be identical to the SBAC. A student taking the SBAC is at the summative stage. A student going through the KA curriculum is taking a lot of formative steps that would need to be baby steps like multiple choice and fill in number.
Tom Hoffman
December 4, 2014 - 9:25 am -On the ELA side, I’m skeptical about the usefulness of verb analysis in looking at standards and exercises. It seems like another instance of Campbell’s Law. Once standards or activity writers know which verbs they are supposed to use to show “rigor,” they tend to sprinkle them liberally.
I’m particularly dubious about CC ELA’s use of the word “analyze,” because it seems to me that in a lot of cases, in practice, an “analysis” is just a detailed description.
Andrew D
December 4, 2014 - 9:26 am -Thanks for sharing. This goes back to your analysis from your TED talk. Teaching is boiled down to steps lined out for students and we monitor how well they step over cracks. I think if we view mathematics as a language, KA is the mathematical Rosetta Stone. (Not good or bad, just is.) Conversation and writing are critical in teaching mathematics well.
Kenneth Tilton
December 4, 2014 - 11:21 am -I am sure Khan has great engineers, but given that their site still offers a primitive maths editor and still cannot check intermediate steps in a solution, the conclusion that they are best of breed at Algebra might be worth reexamining.
As for instructional design, if I ask for a hint it just tells me a step of the solution each time I ask for a hint until the answer is produced. I was hoping hints would be just that, subtle reminders of prior instruction requiring our scholars to actively work out the point of the hint.
Given Khan’s capitalization I am sure Khan will get there in time. I did see a mention in some press coverage that they were planning on strengthening the practice component.
Most of all I am concerned by the either/or being posed. Automated practice is not antithetical to dread avoidance or discovering the magic of math. To the extent that immediate feedback helps kids sort out a subject challenging to many, it breeds success and a positive attitude (towards their own ability as well as math).
And Common Core does demand procedural skill as well as deeper understanding.
As for math being fun, the beauty of automated practice is the class time and teacher energy freed up for development of things other than procedural aptitude, which I agree only a teacher and the social environment of the classroom can provide.
Derek Oldfield (@Mr_Oldfield)
December 4, 2014 - 1:22 pm -Dan, & others…
First Dan, I think you’re conclusions are very accurate and I very much agree that the bulk of Khan Academy is calculations and analysis. I’ve watched a lot of your activity since first viewing your Tedtalk video over a year ago. Math classes do need a makeover! I’m an 8th grade math teacher using Khan Academy as one tool of use in my classroom. My response here is that if teachers are instructing their class entirely via Khan Academy (combo of vids & exercises), I’m certain those students aren’t “constructing” and “arguing” nearly enough. Even Sal himself would give equal importance to what students do on Khan Academy vs what they do when they aren’t on Khan Academy. KA or even the act of blending instruction is just as much about liberating teachers to provide higher quality experiences for their students as it is about the technology. I don’t use KA every day but I make a concerted effort to monitor the verbs my students are engaging in during KA and outside of KA. I try not to repeat. If my students are churning through exercises that are mostly computational, I try to provide experiences that require a higher level of thinking during the days my students aren’t on KA.
I’m responding honestly hoping for an answer or some support… Is the act of calculation and computation outdated at this point? Learning is never linear, I’m aware, but where on the path would calculation and computation fall in relation to argue, analyze, and construct? Do they have to appear at all? What skills require proficiency in calculation & computation in order for students to argue, analyze, and construct? I’ve been struggling for some time now mostly because I feel terribly ill equipped to provide my students more opportunities to argue, analyze, and construct. All I did in school was calculate and compute. Hours of HW calculating in high school and in college. (That’s no justification for me or anyone using those methods, just evidence that I have little exposure to math teachers instructing predominately using higher ordered verbs.) To what ratio should classrooms spend on calculation vs 3 act math tasks? My students (8th grade) can’t multiply to 12s without a calculator, they can’t add/subtract integers quickly & correctly, and they have zero knowledge of fractions especially any operations involving fractions. Perhaps my issue is a vertical issue involving all grade levels in my county. I don’t know. I’d love to see more students engaged in higher ordered problem solving. I honestly don’t know how to build or construct a 2nd floor bathroom when the foundation is crumbling and the first floor isn’t complete.
Any thoughts?
Jason Dyer
December 4, 2014 - 3:06 pm -Do you know how the free-response stuff on SBAC is going to be graded? Is it an army of interns, or the teachers themselves grade random blindly assigned students from different parts of the state, or some sort of automated process?
(If it’s the last one, I’ll have to say “good luck”.)
Timfc
December 4, 2014 - 3:43 pm -Two things:
1) I think you’re overly generous of KA’s vision for teaching. The tasks and selection that you’ve described appear to align with what I think most (not the fine folks who show up here) teachers do.
There’s some analysis of high school texts that are interesting, Thompson, Senk and Johnson (2012) claimed:
Quote:
Overall, opportunities to engage in proof-related reasoning increase as students move through the school mathematics curriculum, increasing from about 3% in Algebra 1 to almost 8% in Precalculus textbooks, averaging less than 6% across courses and topics. The percent of exercises addressing an argument for a general case was approximately equal to the percent requesting an argument for a specific case.
End quote:
Bieda, et al (2012), found that only 3.7% of the activities in 7 different upper elementary texts (ages 9-11) asked for reasoning and proving.
Lithner’s analysis of college calc exams shows that one can earn 90% via procedural fluency. Mesa just came out with a paper about college calc that argues that there’s a much wider range of item-types assigned, including those that call for novel thought.
That is… it seems like KA is aligned with standard practice?
2) Your analysis (in tweets) reminded me of Grant Wiggin’s critique of the NY 8th grade math test and the weird fixation on scientific notation:
https://grantwiggins.wordpress.com/2014/11/24/failure-the-8th-grade-nys-common-core-math-test/
Maybe KA was working, partly, from that?
Sam Wineburg
December 4, 2014 - 4:01 pm -Great points Dan. For a similar conclusion about what gets tested in history see my “crazy for history ” journal of American history , 2006
Shelbi
December 4, 2014 - 9:39 pm -Actually, Smarter Balanced does show 400+ problems. There are over 1000 in the item specifications posted online (zip folders available here http://www.smarterbalanced.org/smarter-balanced-assessments/ under the item and task specifications tab. These specifications are scheduled to be re-posted later in the spring since everything we have done to date has been either a pilot or field test and we are now reacting to all of that research to refine our approach to operational assessment, which includes kicking out problems that didn’t work or had other flaws.
As for scoring of open-ended responses, Smarter Balanced states are using slightly different approaches provided they meet established reliability standards. I have not heard of any states using Artificial Intelligence (AI) scoring in year 1, but I suspect this will continue to be an area of research as we acquire more student responses to open-ended questions (I personally find AI scoring studies fascinating — especially trying to analyze the characteristics of the problem to determine why it may or may not be AI scorable, and how to ensure that the task purpose is not compromised to preserve cheap scoring). Our states have committed to hand scoring up to 4 items for grades 3-5, 5 items for grades 6-8, and 7 items for high school (on a 30-35 item test). This is a significant change over prior assessments, and allows the assessments to attend to the skills outlined in any good set of college and career ready standards.
Michael Buescher
December 5, 2014 - 5:43 am -Is there perhaps one simple conclusion here?
The computer is a great tool for some tasks – especially for practicing routine skills. The teacher is necessary for the management of higher-level analysis tasks. The computer is great if we use it to free teacher time for more advanced activities. The computer will never replace the teacher.
I don’t know if Khan claims to be presenting a set of videos and exercises that can be used INSTEAD OF an 8th grade class – in which case, this shows they’re not. Or whether they are presenting tools to SUPPORT an 8th grade class – in which case, hooray for them.
Kevin Hall
December 5, 2014 - 5:45 am -Dan, I agree with your analysis, but I think KA’s work to deliver high-quality practice on these simpler question types (numerical & multiple choice) will drive the publishing industry to finally begin delivering on SBAC-style open-ended questions. After all, who would pay for a textbook or program that does numerical practice and multiple choice, when Khan does the same thing for free and perhaps better? The only way for publishers or ed tech companies to make money will be to deliver something beyond the lower-level stuff offered for free on Khan. Couldn’t it shift the entire industry, supporting outfits like Mathalicious or Desmos?
So I actually hope that Khan keeps its focus for now on improving its current materials, rather than developing a whole wing of people to make SBAC-type stuff. Lot’s of KA’s current materials need a lot of work–for example, some of the existing hint sequences are just atrocious.
Naomi
December 5, 2014 - 6:08 am -I wanted to thank you for this analysis, because I do regularly use Khan Academy with my students. That said, I think that its primary usefulness is for (a) remediation and reviewing skills from earlier grade levels and assessment of student readiness, and (b) individual practice when an instructor isn’t present. I use it for homework, and even as homework I tend to assign objectives associated with grades lower than the students’ current grade. So, my 6th grade students will be encouraged to work on 3rd and 4th and 5th objectives in Khan–but objectives related to what we’re currently working on at the 6th grade level in class. I’ve gotten some parent questioning about what I recommend, but I like to assign homework that fills in the gaps that we know so many of our students have, and that isn’t overwhelming. (Does this 3rd grade mean they were supposed to learn this in 3rd grade? Why are you doing this now?) And I leave students with the flexibility to work on harder related objectives should they finish the ones I’ve already recommended, or easier ones should they struggle with the ones I’ve recommend. All this to say that I think we need to approach Khan Academy as a tool to supplement classroom activities and differentiate, and a tool to empower students to take responsibility for their education, but not as a primary mode of instruction. Technology just isn’t going to be discussing or arguing about math–that’s something that (at least for now) can only be accomplished with interpersonal interaction. (And, heck, if we’re going to put a bunch of 8th graders together to do math, they really might as well be doing some of it together!)
Jami Packer
December 5, 2014 - 6:21 am -My children’s schools are using a program called Think Through Math as a supplement to their class instruction. The little exposure I’ve had to it seems a good bit superior to Khan’s coursework, but I’d be very interested to see a breakdown similar to this of those questions. I think it would align much better to the CCSS and SBAC. Any computer-based instruction is going to come with its limitations, in my opinion. But there are certainly steps that could be taken to improve it.
Derek Oldfield (@Mr_Oldfield)
December 5, 2014 - 6:57 am -https://www.youtube.com/watch?v=vmtgz95ZBbE This is a good piece referencing what some here have already pointed out. KA is a tool built to help teachers carve out time by allowing the computer to do what it can do much quicker and more efficient than a human can do: assess proficiency on verbs like compute and calculate. What teachers should do with the extra time? That’s what Dan has devoted much of his time developing resources… In fact, I’d much appreciate someone sharing even more resources to me!
Fitzpatrick
December 5, 2014 - 7:52 am -I find that this analysis faulters when it reaches its conclusion. The entire article is about assessment, and then concludes with assessment of instruction.
Accross the nation, Khan academy video instuction is utilized in reverse lesson format to breifly introduce students to a concept prior to entering the classroom, where true assessment, critical thinking, and performce tasks can then occur.
The Khan academy questions are more a skills test or quick “did you understand the process” test. True assessment and critical thinking would still occur in the classroom.
The videos are excellent and a positive enhancement for teachers. As a math educator, I use the videos to both enhance lessons in the classroom or prepare students for lessons. The Khan Academy program was not developed as an assessment tool, which this article clearly believes.
James Key
December 5, 2014 - 9:19 am -Hey, Dan. Good critique of KA. Regarding this quote: “Khan Academy asks students to submit their work as a number or a multiple-choice response, not because those are the mathematical outputs mathematicians and math teachers value most, but because numbers and multiple-choice responses are easy for computers to grade in 2014.”
Can’t we say exactly the same thing about SBAC and other computer-based tests? What am I missing?
Ana
December 5, 2014 - 9:52 am -Really good detailed analysis. Nice work!
I came across this saying from Einstein sometime back –
“Any fool can know, the point is to understand.”
Right from Kindergarten, I noticed that my daughters were not learning concepts in school. They were just doing drills after drills in the hope that they would eventually make sense of the math. My daughter was falling behind with that style of teaching. At the same time, she was reading at a level two grades higher.
I am a computer engineer, and a lecturer. Not being good in Math was never an option when I was young. It is not an option for my kids either. I built her some games that provided insight into the math and connected it with objects they see and games they play, making it more real for them. I eventually published those games for other parents too. She not only caught up, but is far ahead of her class now. What’s more important, she actually enjoys the math.
My older one is in 3rd grade now. When their teacher told the class they would start on multiplication, she was so excited! But, the rest of the class groaned. I was so sad to hear that kids are learning to dislike math this early.
Recently, I built similar way of learning Fractions. My daughters help me with testing, so, she ended up playing all the games – for older grades too. Afterwards, I noticed that her understanding of fractions was different – deeper. She would apply fractions in ways I hadn’t thought of.
I haven’t used khan academy with my daughters. They do more than enough math practice drills at school. I am careful about what kind of exercises I ask my daughters to do at home. Drills can just as easily kill their love of learning, as help them learn.
Alicia Carter
December 5, 2014 - 10:03 am -Hi Dan. Love this look at Khan! I am in a bit of a unique situation this year and would love some feedback. After many years in a traditional middle school math classroom, I am supporting a group of Middle Schoolers who are homeschooled and the bulk of their math instruction occurs independently at home. We are running a math program where our 32 students (grades 7 and 8) range in skill level from 1st grade to high school. We are currently using Khan Academy as a centerpiece for instruction as the students are not on campus in a traditional classroom most of the time; they do come to campus two hours a week for some basic skill instruction and review. It is the ability of students to watch video instruction when they need help that makes this powerful in my placement.
Khan has been great for differentiation, but I do have share of the concerns you’ve raised here. In my traditional classroom, I augmented the use of Khan with my direct instruction, collaborative math challenges, and project-based learning, but that just isn’t an option here. I want to give my students the best shot at learning and mastering skills in a meaningful way, but must work within the constraints I face. My question is, do you have any suggestions for other programs or materials that might help me design a better program? It’s the independent instruction component that is just so difficult to construct here. Any thoughts?
Pawan
December 5, 2014 - 11:15 am -Dan,
I generally agree with your conclusion about the state of the technology in 2014 being reflected in Khan Academy’s Math exercises. It probably also reflects how Math has been taught traditionally.
However, I think that the narrow filter used in this post should not be used to denounce what they are doing at a larger scale. They are providing access to the world and not just to US students with a lens of common core. And they provide access to many subjects in addition to mathematics.
We can certainly assess the aptness of the content for various purposes and audience. It’s pretty easy to find a few where this content is quite useful. For example, I have a motivated kid who gets the math quite well (I ask the probing questions to make sure she really gets it). When it comes to practising the calculation skills, I would send her over to Khan Academy or similar sources. Other motivated folks can also find some value here.
I am not looking at KA as the only or primary source of math education. For those who have not much else, this is still a valuable resource.
Keilah Martin
December 6, 2014 - 1:19 pm -Interesting. As a new teacher it’s good to have this pointed out so I know where to direct my efforts in conjunction with Khan Academy.
I’d like to note that many textbooks, training programs, and teachers don’t have a firm grasp on how to elicit thorough analysis and argumentation from students, so Khan Academy is not unique in this deficit.
Amy Roediger
December 9, 2014 - 7:16 am -I learned today that The College Board has formed a partnership with Khan Academy. When students receive their scores next year, they will also get linked to a Khan account so they can remediate skills that were found to be deficit areas. A College Board rep described it as personalized test prep for SAT/ACT. Who wants to volunteer to do this type of comparison between PSAT and KA?
Kevin Hall
December 9, 2014 - 9:58 am -It looks like Khan Academy is working on handwriting recognition, which would probably open up the range of answer modes and reduce dependence on multiple choice.
I say that because they’ve added MyScript as a supporter on their “supporters” page:
https://www.khanacademy.org/about/our-supporters
Dan Meyer
December 9, 2014 - 10:53 am -Clearing up a few items.
Derek asks what he should do about his students who find the acts of calculation and solution challenging. Are those acts outdated? Why not use Khan Academy for that kind of practice?
I don’t mind reading all these people who use Khan Academy in highly selective ways, mindful of its incompleteness, using it only for targeted supplement.
All that said, I don’t think completing three of these exercises in a row is the most interesting multiplication practice for a student. Or the most accurate assessment either. (Does anybody think that three correct in a row means a student has reached any level of mastery?)
It turns out there are lots of interesting ways to pair mathematical processes with fluency practice. I’d look to these Tiny Math Games or Jo Boaler’s Fluency Without Fear paper.
Timfc raises the point that much of the North American textbook market focuses on procedural skills like calculation and solution also. Granted. I’m not going to defend lousy curriculum.
James Key:
I may be missing your meaning. I’d argue that KA is exploiting the ease with which a computer can parse selected and objective response data while SBAC is asking computers to parse data that requires more engineering creativity. Nothing is stopping KA from replicating (or even improving on!) SBAC’s work. But KA is taking the wide path towards coverage while SBAC is taking the narrow path towards better math.
Kenneth Tilton
December 11, 2014 - 10:22 am -Call me (seven months) slow, but there is a *great* reason for comparing Khan Academy with CCSS, viz., their exclusive patnership: http://techcrunch.com/2014/03/05/khan-academy-gets-major-partnership-to-close-rich-advantage-in-college-test-prep/
So there it is, the College Board, Common Core, and Khan Academy, a nice tidy bundle.
I watched Sal’s video on one SAT question, requiring identifying the greatest rate of change from a chart. Instead of looking for segments with steep slope and checking those he tediously calculates the rate for each segment (except horizontals).
I would have liked a reminder of how slope steepness maps to rate of change, using that to identify the two apparent candidates (in the example given), then careful calculation of those ratios. That said, I do not in fact know which of us is closer to CCSS, maybe Sal has that right.
But that was just the video. The practice question itself had four possible answers and one hint (which just told me the answer). Again, I would have liked the first hint to remind me about steepness and rate of change.
As Dan notes, how far they can go with the practice answers is another question. In this case, the SAT question itself was multiple choice, so no loss there. If they want to go further, software could easily parse “1:04 to 1:08” and have them type in a blank.
Erik Gunderson
December 11, 2014 - 3:19 pm -As I practiced what looked like a pre-Windows 95 archaic SBAC, I asked, is this really it? I am deeply disappointed to think about the mass-production, high stakes, and capital that went into the interface we see here https://login4.cloud1.tds.airast.org/student/V42/Pages/LoginShell.aspx?c=SBAC_PT. Khan does much more than just ask students to solve problems, like others have said, it builds confidence in students and has changed the motivation amongst my low income student population. They love the mastery challenge. When was the last time a student said: “Can I have some review problems of things we did last month- or from the beginning of the year?” It has never happened in 10 years of teaching. When kids say: “In 15 hours I can do more Mastery Challenge problems!”- which is music to my ears.
I am hopeful from Kenneths link- that a Khan/SBAC relationship make a more user friendly, 2014 (heck- I’d settle for 2004) interface. And I agree and appreciate Dan, your analysis of Khan. I hope they read this and integrate more summative big picture problems for students to respond to- and I believe they will.
Eric Kamander
December 12, 2014 - 8:13 pm -I’d like to respond to this section:
“We’d like students to enjoy math class, to become flexible thinkers and capable future workers, to develop healthy conceptions of themselves as learners, and to look ahead to their next year of math class with something other than dread. Will instruction composed principally of selecting from multiple-choice responses and filling numbers into blanks achieve that goal?”
I think the answer is actually yes. I think Khan Academy is largely responsible for my children’s love of math and all that entails. While they are only in 6th and 2nd grade, they are both many years above their grade levels in math, and this is largely due to their practice on Khan Academy.
They started using Khan Academy before it aligned with CC, and they ignore the grade levels, which I believe to be tragically arbitrary anyway, and follow the knowledge map, which KA has attempted to hide. Even if the hints may give some answers away, they don’t help them “master” the skill. I watch my kids do the computations, usually on paper, when they can, in order to get to their answers. And when they can’t they watch the videos. Eventually they “get it” and move on. The confidence they’ve developed by always being able to stay ahead of the curve has allowed them to enjoy math on their own terms. In addition they’ve learned to embrace a growth mindset, seeing for themselves how quickly they can master new skills all on their own.
Even when my 6th grader, who volunteers as a peer tutor for 6th through 8th graders, didn’t make the school math team (only 4 8th graders did) he knew that if he wanted to next year he would just have to work a bit harder at it, but that it was attainable.
Because they can enjoy math, they think about math. They often think about the world in mathematic terms. In a world where schools tend to herd students towards the center Khan Academy has been an excellent source of self-paced and personalized learned them helps them excel.
Noirin Foy
December 12, 2014 - 10:14 pm -I agree that Khan Academy does not give the full common core experience, but it has its uses especially for Math Practice 6 “Attend to Precision”. In the past, teachers used one textbook to teach a math course and added very few additional resources. I do not look at Khan Academy as replacing the textbook and providing all parts of the curriculum for all students. To use a cliché, it is a useful tool in the teacher’s toolbox.
Justin Helps
December 13, 2014 - 3:31 pm -Speaking for KA, we’re proud that we’ve been able to put broad, interactive coverage of K12 math online for free, but we certainly understand that we’re still far from reaching the potential of our platform.
One contributor to the prevalence of numerical and multiple choice responses on KA is that those were the tools readily available to us when we began writing content. Our set of tools continues to grow, but it takes time for our relatively small content team to rewrite item sets to utilize those new tools.
In addition to utilizing our technology properly, we’re continually going over our existing content to make the progression more pedagogically sound.
We’re looking to hire brilliant question writers to help us reach higher. :)
https://boards.greenhouse.io/khanacademy/jobs/32840#.VIzI32TF_JU
P.S. Dan, I absolutely love this critical feedback. If there are any other thoughts you’d like to share, I’d love to hear ’em.
HT
December 14, 2014 - 5:14 pm -My school looked into Khan Academy, and I like the ideas of videos attached, but at the time the order of topics wasn’t user-friendly (or teacher-friendly) and it wasn’t aligned to our state’s standards, so we moved into IXL.
Is IXL perfect? No. It doesn’t have nearly enough analysis questions, and it’s weak on geometric constructions, for instance. But it offers practice on about 90%+ of the topics we cover, and it does allow for the writing of entire equations.
How it handles the graphing of lines and points (like that reflections question you did that was so much fun) was our selling point for it. You can graph systems of equations and then write the solution you find, or be asked to choose whether there exists one solution, no solution or infinite solution (depending on the skill.)
The program handles student input of whole-line text of equations, inequalities, laws of exponents, and has them graph inequalities on number lines well. Students can even create the graphs of linear inequalities, where you graph the line, then shade above/below, and click for a dotted or solid line. Pretty impressive!
The Virginia SOL has a lot of Technology-enhanced items (free response and “select all the choices” questions) and the students last year reported that IXL helped them prepare for this challenge. I saw its impact first-hand on my CFAs too, where students stopped me to check that their answer was “grammatically correct” since they know the importance of simplifying fractions and not leaving off those important negative signs.
Are you planning on reviewing IXL’s curriculum at some point? I don’t want this to sound like an infomercial, just that we enjoy this program in place of workbooks/textbooks for the feedback it provides (I can spot where students are in their own progression through the practice, and they know when they’ve mastered a skill or need to keep on working on it).
Chris
December 15, 2014 - 3:24 pm -i was wondering if for your doctoral you did more of these analyses, and also if you contemplated using existing frameworks or codes for the coding?
Also, I don’t have time to look at all of them, I was wondering why https://www.khanacademy.org/math/cc-eighth-grade-math/cc-8th-solving-equations/cc-8th-age-word-problems/e/integer_sums for example is deemed ‘calculate’ (Am I looking correctly?) because it seems to be more than that: three consecutive numbers together are X.
But I guess this will end up in your thesis?
Kenneth Tilton
December 15, 2014 - 3:25 pm -I think there is a middle ground in multiple choice of which few might be aware, in which deep assessment of student learning can be achieved without human involvement and without AI parsing of student responses.
Back in the day at Clark University a rather extraordinary (and hugely popular) philosophy professor gave terribly hard exams that were multiple choice (!), I suppose so he would not have to read and score 150 essay-based exams.
Each question was a masterpiece, and each choice by itself required mastery of substantial content. The one I recall was in the context of Leibniz: “c. You can take the nomad out of a monad, but you cannot take the monad out of a nomad.”. (Don’t ask, I forget.)
Mind you, this was quite hard work for the professor so it is not a free lunch.
FWIW.
Rob
December 16, 2014 - 3:37 pm -Interesting article… but Meyer’s analysis is skewed in that it analyzes what the students do in relation to the CCSS. It does not take into account: i) who the students are; ii)why they enroll; or iii) what they are trying to accomplish.
Bottom line: would I pay private school tuition to enroll my kids in a school with the open standards of Khan Academy as a preparatory school for post secondary education? No.
Would I use the Khan Academy as a cost effective supplement to cement the mechanical applications of concepts taught and then applied to real-world problems by trained and paid eighth grade teachers?
Absolutely.
Lori Cargile
December 24, 2014 - 11:20 am -Hello Dan and the math community,
I am also doing my dissertation work on Khan Academy. As said by Derek in comment 10, KA is not intended to be an all-in-one product. Sal Khan’s vision, as expressed in his book “The One-World Schoolhouse: Education Reimagined,” is that KA be used to efficiently provide an environment for teaching procedural fluency not higher order skills. The goal is for KA to supplement instruction as a replacement for inefficient teacher lecture. Using class time more productively should lead to more opportunities for collaborative activity and projects where argumentation and justification and other higher order skills are taught.
Sal Khan say, “As I hope is clear by now, it was never my vision that watching computer videos and working out problems should comprise a kid’s entire education. Quite the contrary. My hope was to make education more efficient, to help kids master basic concepts in fewer hours so that more time would be left for other kinds of learning. Learning by doing. Learning by having productive, mind-expanding fun” (Khan, 2012, p.149-150).
I think your analysis of KA presumes that math teachers may use KA as a primary teaching tool rather than as a supplement as Khan recommends.
Lori
Donna
January 24, 2015 - 9:21 am -I understand what you are saying and agree with you. Unfortunately, there aren’t enough resources out there yet that demonstrate what would align to the CCSS (using technology).
The only great examples I have seen are the exercises that have been created with Desmos. I have seen your Penny Circle and Function Carnival Activities, and Fawn Nguyen’s Des-man activity, plus the other activities like Water Line and Central Park and these are amazing. I have seen Mathalicious’s activities using Desmos as well and believe those are beautiful examples of Real-World Math.
What we need is for YOU, Dan Meyer, to team up with Desmos and create the next big thing!
David
January 25, 2015 - 3:57 pm -I think these KA questions are fine. (I didn’t understand the Pythagorean example; it seems to me that after step 1 and step 2 you’re done.) They teach mechanics. It’s very important to master mechanical skills, the grammar of mathematics, before trying to do creative work.
CC makes the mistake, a big one, of asking for wishy-washy argumentation that cannot graded clearly or evaluated. The mathematical form of argumentation is the proof. Traditional mathematics trains students until they are ready to do proofs. Those who don’t make it that far are better off doing their argumentation in other classes and can become lawyers but they won’t become mathematicians. They’ll still be able to calculate their grocery bill and even do their taxes with the math they did learn. Math is not the only place where logical thought is taught.
The CC standards calling for argumentation in English are useless. It seems an effort to make mathematics something different from mathematics. I applaud KA for not mucking up their curriculum with it.
Kenneth Tilton
January 25, 2015 - 9:22 pm -@David “The mathematical form of argumentation is the proof.” Well said. I saw a CC-oriented lesson complaining about teaching division by a fraction by saying invert the divisor. Me, I thought division by X was indeed syntactic sugar for multiplication by the reciprocal of X!
CCSS is concerned kids might get the right answer for the wrong reason. OK, but more will get the answer for the right reason and not be able to explain it coherently, or their explanation might be fine but elude the examiner. This problem with natural language is one reason we have math!
I am reminded of one of my rules for working on other people’s computer programs: first, delete all the comments. Code is unambiguous, we coders are inarticulate geeks.
btw, do not celebrate too soon on KA vs CC. The CC-aligned SAT now links directly to KA from test results: https://www.khanacademy.org/sat Of course there is no way KA can parse argumentation, so no danger of them going there.
David
January 27, 2015 - 5:45 pm -Wonder why SAT is willing to give such an exclusive to KA and nobody else. What saintly thing did KA do?
I don’t love KA. I find it’s generally missing higher level concepts and inspiration, so it should fit into the new SAT where the idea is that nothing is “tricky”. But at least it doesn’t promote the idea that those who can learn math do not get too far ahead of those who cannot.
I guess colleges are just going to start looking for other evidence of IQ if the SAT stops providing that.
Steve
February 5, 2015 - 2:08 pm -Definitely agree that Khan Academy underemphasizes argumentation. I understand the need to focus on multiple choice-type questions, but there’s no reason why a properly worded multiple choice question can’t force students to construct an argument in their head.
Eric
March 30, 2015 - 11:40 am -One of the problems in our educational system today is that students do not get ample time to master basic calculating skills through practice. We definitely want students to become analytical thinkers and critical thinkers (evidence-backed), but they need to have basic skill mastery, too. Otherwise, there is no foundation for higher level skills. I’m not sure if they do this, but if KA can help provide more skills mastery, that is excellent! Why not incorporate “the best of all possible worlds” and use CCSS, KA and more?
R. Craigen
May 28, 2015 - 10:28 pm -Interesting that you focus on Khan Academy, which is an *entirely free, self-help* approach to mathematics education, Dan — I look forward to your similar critiques of all the commercial systems that have a lock on classroom teaching, cost our system enormous amounts of taxpayer dollars and *also* are not particularly well-aligned with CC.
It seems the bee in your bonnet, however, has little to do with CC itself — it focusses narrowly on a certain ideology pertaining to the value of assessing what some refer to as “process standards” (a misnomer — they tend to refer only to cognitive processes): things that may be going on in a student’s mind, rather than what they can concretely accomplish on paper with the tools they’ve learned.
All large-scale studies comparing cognitive-focussed instruction in elementary school versus skills-focussed instruction generally come to the same conclusion: cognitive-focussed instruction is a nasty failure, not only at inculcating skills, but also at fostering the understanding upon which it purports to focus.
The epitome of these studies, which happens also to be the largest comparative educational study in history (and also the most ironically-named) is Project Follow Through — for those unfamiliar with this landmark study, Athabasca University has helpfully posted a summary here: http://psych.athabascau.ca/html/387/OpenModules/Engelmann/evidence.shtml
So … if this is the only way in which Khan fails to align with CC I say “hurray for Khan”! Because many of the commercial texts in use fail in other, significantly more damaging ways, such as failing to teach the standard algorithms of arithmetic and/or failing to provide sufficient consolidation practice for the mastery of said procedures. The cumulative effect of such lack of mastery, and the frustration it infuses in children, do much damage.
It is, indeed, important that children learn to demonstrate mathematical results. This must be done with children who have mastery of fundamentals. It is not (I emphasize *not*!) a clever way to *teach* the fundamentals — this is unsupported by any significant and empirically valid research; yet many of said texts blissfully build this presumption into their methodology from the start.
Students who have mastered mechanics and facts in early education have an easy time learning the cognitive content of later years in math education — because they needn’t dwell on trivial details that needlessly clutter up working memory, creating a mental logjam. Students who have not mastered this earlier material are almost universally destined to fail.
That is the beauty of Khan. Sure, there is more to do on the cognitive side in math education. That’s what live teachers are for. That is the bread and butter of those of us who teach university math. That is why high school math teachers are generally expected to have far more mathematics in pre-service training than elementary school children; because this is a highly mathematically challenging thing to do. Students are not helped when teachers with no background in the subject, themselves often struggling with the basics, are asked to teach, and assess, these sophisticated skills. And it does those teachers no service to place them in that situation.
Dan Meyer
May 29, 2015 - 8:47 pm -Thanks for the comment. I’ve seen this movie too many times, though, so let’s fast-forward. You cite Project Follow-Through, or maybe Sweller or Hattie’s meta-analysis. I’m supposed to be dazzled by PFT’s sample size but I notice all these studies are predicated on definitions of “discovery learning” that literally no mainstream learning scientist stands behind. Maybe I forward you Silver’s response to Sweller, but you’ve already read it. Maybe I invite us both to explore the vast expanse between Sweller’s preference for relentless direct instruction and Project Follow-Through’s total lack of teacher intervention, an expanse where we’ll probably find some good instruction.
But instead let’s do this. Let’s imagine we sat down for coffee and wrote down all of our favorite verbs from 8th grade math — the best actions a student should be able to perform at the end of their year. We rank their importance also. We shelve our disagreements about when and how students should learn how to do those verbs. We agree that summatively, to be stamped proficient in 8th grade math, students should be able to do them.
I’m guessing the overlap of our two lists would surprise us. I’m willing to bet that your list would contain more breadth and depth than Khan Academy’s list of skills. I know mine would.
I want students to do more satisfying work than Khan Academy requires. More consequentially than what I want, though, is the fact that this assessment consortium requires that breadth of work. That’s the problem.
Stan
May 31, 2015 - 4:25 am -I wonder what Alan Turing would say?
Given the technology we have today could a computer based assessment tell you if a student is proficient in grade 8 math?
Of course this depends on the definition of proficiency and if that definition includes being able to explain to another human why a particular solution or method works then it is likely much easier for a human than a computer to assess this.
But taking an alternative tack in Canada the CEMC Gauss competition is used to rank high proficiency in math at grade 8. Is it likely that the top performers on this contest are not proficient at grade 8 math?
Perhaps they are not as proficient at explaining their work as a grade 8 teacher but my guess is the top performing grade 8 students perform well above the average grade 8 teacher on the Gauss competition.
Also despite being simple multiple choice questions the top performers find these to be very satisfying. I am guessing but I would bet many find it more satisfying to get the right answer to a challenging problem than to explain their reasoning used in a simple problem.
Math competitions after grade 8 do evolve to include both multiple choice and written answers. But I wonder if at the early grades some educators are putting an unwarranted emphasis on the math skill they hold most dear, explaining an answer, rather than one kids enjoy, getting the right answer to a challenging problem. At grade 8 which of these do you need to assess?
In summary if you removed all assessment of students ability to explain their solution at grade 8 and below and replaced it with assessment by more challenging problems what would be the result on the students further math education?
(I am being careful here to ask a question about assessment. This says nothing about how much interest students have in understanding math. But just as students very able to read enjoy reading for its own rewards I am betting those very capable of doing challenging math problems enjoy learning about it. Just a guess.)
Kenneth Tilton
June 1, 2015 - 12:01 pm -@Stan: I am reminded of my college roommate who lured me by example into computer programming and later became an IBM Fellow specializing in operating systems but for the life of him could not explain to me how on earth “x = x + 1” made sense (in Fortran).
Indeed, we like students to teach others once they have some skill precisely because it requires greater mastery of the skill. But is this higher bar useful in assessing math skill?
My approach as a teacher was to shoot for procedural competence and create opportunities for any to soar higher, thinking many simply will not get any higher (and I am sadly awaiting CCSS’s proof of that).
Agreed also: throw them those “extra credit” problems if one wants to push them further, do not take them into the realm of pedagogy/exposition — do we make skilled writers diagram a sentence before admiring it?
The pro-explanation crowd is all in a panic that kids may have memorized something, but I do not think they have thought it through: I can memorize chess positions, but to recognize and execute a pin or a fork requires understanding the moves and weights of pieces regardless of chess position. No math student ever memorized all possible expressions and how to transform them.
Kevin Hall
November 7, 2015 - 1:57 pm -Here’s a way to improve Khan Academy by making it less adaptive. Thought you might be interested.