It’s great, first of all, that Khan Academy has all their student exercise code on GitHub for everybody to see. I don’t know any other adaptive system that does that. I figured there had to be a better way to reward them for that transparency than the criticism and judgment I’m about to post here, so I made them a badge also.

Their code illustrates the different ways good math teachers and good programmers try to figure out what students know.

Take proportions, for instance. Here is the code that runs beneath Khan Academy’s proportions assessment.

In each of the dozen files I’ve reviewed, Khan Academy first generates some random numbers that meet certain criteria. In the proportions assessment, they call for three random unique integers between 5 and 12. No decimals. No negatives. No zeroes.

var numbers = randRangeUnique( 5, 12, 3 );

Then they use those numbers to generate exercises. With proportions, they insert an “x” randomly into that list of numbers. The final order of that list determines the proportional relationship that students will have to solve.

numbers.splice( randRange( 0, 3 ), 0, "x" );

But good teachers are more than random number generators. They create exercise sets that increase in difficulty, that ask students to demonstrate mastery in different contexts, all because proportions are conceptually difficult but procedurally simple. It’s extremely easy for students to get by on an instrumental understanding of proportions alone. (eg. “All you hafta do is multiply the two numbers that are across from each other and divide by the number across from the x.” Boom. It’s badge time.) It’s especially easy when the only thing that changes about the problem is the random numbers.

But forget good teachers for a minute. Let’s look at the bar set by various standard-setting organizations. Here is what you have to do to demonstrate mastery of proportions on a) Khan Academy, b) the California Standards Test, c) the Smarter Balanced Assessment.

Khan Academy

You’ll do a handful of problems just like this, with different random numbers in different places.

California Standards Test

Smarter Balanced Assessment

The difficulty and value of the assessments clearly increases from Khan Academy to the CST and then Smarter Balanced. (I’m hesitant to guess how well a student’s score on the Smarter Balanced Assessment will correlate to all her practice on Khan Academy.)

Here we find a difference between good math teachers and good programmers. The good math teacher can create good math assessments. The good programmer can make things scale globally across the Internet. The two of them seem like a match made in math heaven. Just get them in a room together, right? But the very technology that lets Khan Academy assess hundreds of concepts at global scale – random number generators, string splices, and algorithmically generated hints – has downgraded, perhaps unavoidably, what it means to know math.

2012 Dec 13. Peter Collingridge points out in the comments that Khan Academy has a proportions assessment comparable to the California Standards Test. If they have anything similar to the Smarter Balanced Assessment, please let me know.