Blog monster hungry. Feed me.

I’m not big on the retroactively “sorry I haven’t been blogging” posts. I’d rather proactively explain why I’m not going to be around here for the next several months.

It *isn’t* for lack of interest in math education or for lack of interesting things *happening* in math education. For instance:

- My research group had a conversation about successful teacher professional development with Tom Carpenter, lead author of one of the most successful teacher professional development programs ever. Lots to share there.
- There were two especially useful sessions at CMCN14 I’ve wanted to share for months.
- RER published a meta-analysis of computerized feedback (useful summary) that I’m sure would find us all in a productive fight.
- Lots of
~~snark~~trenchant analysis of a current Khan Academy initiative. - Two curriculum confabs.
- Lots of great classroom action to talk about.

But my dissertation hearing is scheduled for mid May. I’m in the middle of data collection with lots of writing and analysis ahead and I’m sure I need to become a bit more ruthless in managing my time and writing.

So I’ll see you on Twitter (can’t quit *that* obviously), at NCTM and other conferences. But I won’t see you around here for a few months.

Please use this as an open thread to talk about whatever while I’m off dissertating. Also here’s all the great classroom action I haven’t written about over the last twelve months. Plenty of food for the blog monster there.

]]>Here is the temperature in the United States today (Fahrenheit):

So basically business is bad. No one wants your frozen treat.

So what do you do? You lower prices. An across the board cash discount? Maybe. But if you’re Gelato Fiasco, you institute The Frozen Code:

On each day that the temperature falls below freezing, we automatically use The Frozen Code to calculate a discount on your order of gelato dishes. [..] You save one percent for each degree below freezing outside at the time of purchase.

But how do you write this code using the language of variables that your pricing system understands? (Click through for Gelato Fiasco’s answer.)

How would you set this up as a mathematical learning experience for your students?

[h/t reader Nate Garnett]

*This is a series about “developing the question” in math class.*

**2014 Jan 8**. Updated to add this important exchange with Gelato Fiasco on Twitter.

@Veganmathbeagle @ddmeyer At -68 we would take 100% off, but we would likely not be able to open. Wind chill is not a variable.

— Gelato Fiasco (@gelatofiasco) January 9, 2015

**Featured Comment**

John:

]]>With the clear correlation between temperature and number of chain gelato shops, CLEARLY the temperature causes more gelato shops to be built…or does the number of gelato shops cause the temperature to decrease…am i right?

Here is one of the five, taken from Scott Farrand’s presentation at CMC North.

Here are some points in the plane:

(4, 1), (17, 27), (1, -5), (8, 9), (13, 19), (-2, -11)

(20, 33), (7,7), (-5, -17), (10, 13)Choose any two of these points. Check with your neighbor to be sure that you didn’t both choose the same pair of points. Now find the rate of change between the first and the second point. Write it on the board. What do you notice?

From Henri Picciotto’s review of Farrand’s session:

Students are stunned to learn that everyone in the class gets the same slope. This sets the stage for proving that the slope between any two points on a given line is always the same, no matter what points you pick.

In an email conversation with Farrand, he proposed the term “WTF Problems” because they all, ideally, involve a moment where the student exclaims “WTF”:

Set up a surprise, such that resolution of that becomes the lesson that you intended. Anything that makes students ask the question that you plan to answer in the lesson is good, because answering questions that haven’t been asked is inherently uninteresting.

These seem like essential features:

- These problems are all brief. They slot easily into an opener.
- They look forward
*and*backward. They fit right in the gap between an old concept and the new. They review the old (slope in this case) while setting up the new (collinearity). - Students encounter an unexpected result. The world is either more orderly (the slope example above) or less orderly (see problem #2) than they thought.

And the *weirdest* feature:

- They require the teacher to be cunning, actively concealing the upcoming WTF, assuring students that, yes, this problem is as trivial as you think it is, knowing all the while that it isn’t.

When did they teach you *that* in your teacher training?

It’s striking to me that the history of mathematics is driven by the explanations following these WTF moments:

- We knew how to divide numbers. We didn’t know how to divide by zero. Enter Newton & Leibniz explanation of calculus.
- We knew how to find the square roots of positive numbers, but not negative. Enter Euler’s explanation of imaginary numbers.
- We knew what Eucld’s geometry looked like, but what if parallel lines
*could*meet. Enter the explanation of hyperbolic, spherical, and other non-Euclidean geometries. - There are lots of WTF moments that
*haven’t*yet been explained.

In school mathematics, though, we simply *give* the explanations, without paying even the briefest homage to the WTFs that provoked them.

What Farrand and you and I are trying to do here is restore some of that WTF to our math curriculum, without forcing students to re-create thousands of years of intellectual struggle.

So help me out:

**Have you seen other problems like these?****Who else has written about these problems?**I believe we’re talking about disequilibrium here, which is Piaget’s territory, but I’m looking for writing local to mathematics.

**Featured Comments**

David Wees cautions us that the effect of these problems depends on a student’s background knowledge. If you don’t know how to calculate slope, the problem above won’t surprise, just confound. I agree, but the same is true of textbooks and nearly every other resource.

Michael Pershan worries that the “twist” in these problems will become overused, that students will become bored or expectant. (Clara Maxcy echoes.) I demur.

Dan Anderson offers other examples. As do Mike Lawler, Federico Chialvo, Kyle Pearce, Jeff Morrison, and Michael Serra.

Franklin Mason critiques my math history without (I think) critiquing my main point *about* math history.

Scott Farrand, whose presentation at CMC-North inspired this post, elaborates.

Ben Orlin summarizes the design of these problems in four useful steps.

Terri Gilbert summarizes this post in a t-shirt.

**Featured Tweet**

]]>WTF-based Learning (WTF-bL for short) in the math classroom http://t.co/cptgRZvo7J pic.twitter.com/X6qX4HAHxE by @ddmeyer

— Nacho Santa-María (@nacheteer) January 11, 2015

Here were my new blog subscriptions in December 2014, some of which might interest you.

**Steve Wyborney**posted his animated multiplication table, a very thoughtful tool highlighting interesting patterns for young math students. So I subscribed.**Steve Leinwand**has a blog! I’d subscribe if I knew how. [**BTW**. Commenter Sadler found the feed.]**Thinking Math**is a group blog written by four elementary math educators. Go encourage them to post more.**Kassia Omohundro Wedekind**blogs about elementary math education also (subscribed!) and posts interesting observations and analysis to her Twitter feed (followed!).- Khan Academy’s
**crack data science team**has a blog, which might be Khan Academy’s most valuable contribution to my life so far. Fancy tools, sharp analysis, well-written. - Off
**Anthony Carabache’s**post, Education Should Step Away From Apple Devices, he seems like a cautious, thoughtful technologist, which is my favorite kind.

What did you fill your head with in December?

]]>

The Interviewgenerated roughly $15 million in online sales and rentals during its first four days of availability, Sony Pictures said on Sunday. Sony did not say how much of that total represented $6 digital rentals versus $15 sales. The studio said there were about two million transactions over all.

**Featured Comment**

Stas, with a zinger for the ages:

Probably they took advice from this guy.

Angela Ensminger:

Maybe an opening question to this problem would be:. Do you think Sony had more rentals or more sales? This could lead to some interesting discussions before actually solving the problem.

[h/t Math Curmudgeon]

]]>Students generally prefer video games to our math classes and I wanted to know why. So I played a lot of video games and read a bit about video games and drew some conclusions. I also asked my in-laws to play two video games in front of a camera so we could watch their learning process and draw comparisons to our students.

These are the six lessons I learned:

- Video games get to the point.
- The real world is overrated.
- Video games have an open middle.
- The middle grows more challenging and more interesting at the same time.
- Instruction is visual, embedded in practice, and only as needed.
- Video games lower the cost of failure.

**Featured Comments**:

Tim brings storytelling to the conversation:

As one of those weird AP Lit and AP Calc teachers – and a gamer – I think “story” is key in video gaming. Psychologists (like Willingham) and sociologists talk about the “story bias” of the brain. Nearly all long video games have a heavy story element. You are a character embedded in a story, be it open-ended or scripted. So often when I’m frustrated with bad game design I’ll push through because I’m committed to the story. So often when I finish the “missions” I give up on the well-designed “side-quests” because the story has rushed out of the game and it’s just a task-garden again.

I’ll play Angry Birds for a few minutes. I’ll play Temple Run till I beat my friend’s score. But I won’t put 20 hours into a game until I find a story I want to be invested in. (In the same breath, I’ll say that – in the sense of “story” that Willingham uses it – Angry Birds and Temple Run have their stories, too. Far more than many “story” problems in math books like to pretend that have.)

Not sure how you get rich story into math. How to become characters whose adventures we become invested in, not the scripted Jane who is trying to maximize the area of his pasture or the open-ended John who is trying to find a good way to estimate the number of people in a photo.

Anyway – the first lesson I learn from video games is: humans will spend hours on a good yarn.

My Panama Canal metaphor was just a joke from the onset so I had to admire Joshua Greene’s continued debunking.

]]>Two follow-up notes, including the simplest way Khan Academy can improve itself:

**One**. Several Khan Academy employees have commented on the analysis, both here and at Hacker News.

Justin Helps, a content specialist, confirmed one of my hypotheses about Khan Academy:

One contributor to the prevalence of numerical and multiple choice responses on KA is that those were the tools readily available to us when we began writing content. Our set of tools continues to grow, but it takes time for our relatively small content team to rewrite item sets to utilize those new tools.

But as another commenter pointed out, if the Smarter Balanced Assessment Consortium can make interesting computerized items, what’s stopping Khan Academy? Which team is the bottleneck: the software developers or the content specialists? (They’re hiring!)

**Two**. In my mind, Khan Academy could do one simple thing to improve itself several times over:

Ask questions that computers don’t grade.

A computer graded my responses to every single question in eighth grade.

That means I was never asked, “Why?” or “How do you know?” Those are *seriously* important questions but computers can’t grade them and Khan Academy didn’t ask them.

At one point, I was even asked how m and b (of y = mx + b fame) affected the slope and y-intercept of a graph. It’s a fine question, but there was no place for an answer because how would the computer know if I was right?

So if a Khan Academy student is linked to a coach, *make a space for an answer*. Send the student’s answer to the coach. Let the coach grade or ignore it. Don’t try to do any fancy natural language processing. Just send the response along. Let the human offer feedback where computers can’t. In fact, allow *all* the proficiency ratings to be overridden by human coaches.

Khan Academy does loads of A/B testing right? So A/B test this. See if teachers appreciate the clearer picture of what their students know or if they prefer the easier computerized assessment. I can see it going either way, though my own preference is clear.

]]>California Math Council’s conference in Monterey, CA, last weekend was the best conference PD I’ve ever experienced. Your mileage may have varied depending on your session choices (or whether you were even there) but every. single. element. fell into line for me.

**Great evening keynote with Tony DeRose of Pixar**. (Shorter version here.) I love keynotes that are*just*outside, but not*too far*outside our discipline.**An excellent pick of four sessions on Saturday**. There were at least three great picks in every block. Painful choices. I went out for a few names I knew would be fun (Lasek, Fenton, Stadel). But I also ventured out for a name I didn’t recognize (Barlow) and learned an enormous amount about math teaching as well as about how to talk with math teachers about math teaching. I’ll share some details in a later post, which was supposed to be*this*post until I got all breathless about the conference itself.**The Ignite sessions on Saturday evening were best-in-class**. They were*all*entertaining and interesting, which is unusual enough, but three of them drew standing ovations. Five minute talks. Standing ovations. A standing ovation off of five minutes. Don’t worry. I’ll make sure you see them later.**The community.**I get such a charge off the crowd that assembles on the Monterey Coast annually. I walked around Point Lobos with mentors, broke bread with peers, and met lots of new teachers from local programs. One of the keynote presenters and I both gave talks we had already given elsewhere and we both noted how charged up the crowds were, how great the vibe was, relative to those other venues. No idea why, but I’ll take it.**The venue**. Unbeatable.

So great job, California Math Council. Everybody else: be sure to sign up to present and attend next year.

]]>I’d install whiteboards on every vertical surface in the room. I’d make sure I had a good document camera. And I’d probably purchase video capture equipment, a hard drive, and a microphone so I could record my lessons. That’ll probably get you close to $1,000.

I felt clever recommending old-school whiteboards with a new-school technology grant. But then I put the question out on Twitter and *everybody* suggested the same purchase:

@ddmeyer do the room in whiteboards, manipulatives, document camera

— Alex Overwijk (@AlexOverwijk) December 8, 2014

@ddmeyer Whiteboard paint and huge supply of markers.

— Zachary Herrmann (@zachherrmann) December 8, 2014

@ddmeyer Whiteboards all around classroom, variety of manipulatives, document camera, colour paper, graph chart paper & candy for students

— Mylene Abi-Zeid (@myleneabizeid) December 8, 2014

@ddmeyer white boards around the room and a glass wall down the middle to write on (because it would be awesome)

— Patrick Brandt (@pabrandt06) December 8, 2014

@ddmeyer white boarded room + layered & movable whiteboards on top. markers. money for laminating std quotes & other student work in colour.

— Jim Pai (@PaiMath) December 8, 2014

@ddmeyer Big whiteboards around the room and a ton of markers, a doc cam, chart paper and markers.

— Mattie B (@stoodle) December 8, 2014

@ddmeyer Whiteboards w/markers, 2 iPad minis (for video, pictures, projection of Ss work, & whiteboard app for Ss to talk through thinking)

— Kevin Lawrence (@kalawrence9) December 8, 2014

@ddmeyer group-able desks, whiteboard space vert and horiz, projector, some way to hand write stuff on computer

— Dan Anderson (@dandersod) December 8, 2014

@ddmeyer whiteboard markers! Maybe some extra 2×3' whiteboards. Or use it to start a student run business with profits for the classroom.

— Maria Kerkhoff (@MsMariaAK) December 9, 2014

@ddmeyer I would cover as much wall space with whiteboards as I could.

— Annie Vallance (@AnnieVallance) December 9, 2014

Crazy, right? What would *you* buy?

$1,000 isn’t nothing, but there are *lots* of organizations giving away that sum and more to teachers. I have it on some authority that The Mathematics Education Trust has trouble some years giving away their (fairly substantial) grants. “Not enough qualified applicants,” I was told. So get out there. Get some cash. Get those high-tech whiteboards.

**BTW**. I think we can trace some of this recent popularity of whiteboarding to Peter Liljedahl, an associate professor at Simon Fraser University. Liljedahl gave a presentation at the Canadian Mathematics Education Forum on whiteboards, which he called “Vertical Non-Permanent Surfaces,” which is why I’m looking forward to finishing graduate school.

**Introduction**

My dissertation will examine the opportunities students have to learn math online. In order to say something about the current state of the art, I decided to complete Khan Academy’s eighth grade year and ask myself two specific questions about every exercise:

**What am I asked to**What are my*do*?*verbs*? Am I asked to solve, evaluate, calculate, analyze, or something else?**What do I**What is the end result of my work? Is my work summarized by a number, a multiple-choice response, a graph that I create, or something else?*produce*?

I examined Khan Academy for several reasons. First, because they’re well-capitalized and they employ some of the best computer engineers in the world. They have the human resources to create some novel opportunities for students to learn math online. If *they* struggle, it is likely that *other* companies with equal or lesser human resources struggle also. I also examined Khan Academy because their exercise sets are publicly available online, without a login. This will energize our discussion here and make it easier for you to spotcheck my analysis.

My data collection took me three days and spanned 88 practice sets. You’re welcome to examine my data and critique my coding. In general, Khan Academy practice sets ask that you complete a certain number of exercises in a row before you’re allowed to move on. (Five, in most cases.) These exercises are randomly selected from a pool of item types. Different item types ask for different student work. Some item types ask for *multiple* kinds of student work. All of this is to say, you might conduct this exact same analysis and walk away with slightly different findings. I’ll present only the findings that I suspect will generalize.

After completing my analysis of Khan Academy’s exercises, I performed the same analysis on a set of 24 released questions from the Smarter Balanced Assessment Consortium’s test that will be administered this school year in 17 states.

**Findings & Discussion**

**Khan Academy’s Verbs**

The largest casualty is argumentation. Out of the 402 exercises I completed, I could code only three of their prompts as “argue.” (You can find all them in “Pythagorean Theorem Proofs.”) This is far out of alignment with the Common Core State Standards, which has prioritized constructing and critiquing arguments as one of its eight practice standards that cross all of K-12 mathematics.

Notably, 40% of Khan Academy’s eighth-grade exercises ask students to “calculate” or “solve.” These are important mathematical actions, certainly. But as with “argumentation,” I’ll demonstrate later that this emphasis is out of alignment with current national expectations for student math learning.

The most technologically advanced items were the 20% of Khan Academy’s exercises that asked students to “construct” an object. In these items, students were asked to create lines, tables, scatterplots, polygons, angles, and other mathematical structures using novel digital tools. Subjectively, these items were a welcome reprieve from the frequent calculating and solving, nearly all of which I performed with either my computer’s calculator or with Wolfram Alpha. (Also subjective: my favorite exercise asked me to construct a line.) These items also appeared frequently in the Geometry strand where students were asked to transform polygons.

I was interested to find that the most common student action in Khan Academy’s eighth-grade year is “analyze.” Several examples follow.

- Instead of just asking for the solution of a system of linear equations, for instance, Khan Academy asks the student to analyze how many solutions the system would have.
- Instead of just graphing a function, Khan Academy asks the student to draw conclusions from the graph of a function.
- Instead of just asking students to create a table, Khan Academy presents the table and asks students to draw conclusions.

**Khan Academy’s Productions**

These questions of analysis are welcome but the end result of analysis can take many forms. If you think about instances in your life when you were asked to analyze, you might recall reports you’ve written or verbal summaries you’ve delivered. In Khan Academy, 92% of the analysis questions ended in a multiple-choice response. These multiple-choice items took different forms. In some cases, you could make only one choice. In others, you could make multiple choices. Regardless, we should ask ourselves if such structured responses are the most appropriate assessment of a student’s power of analysis.

Broadening our focus from the “analysis” items to the entire set of exercises reveals that 74% of the work students do in the eighth grade of Khan Academy results in either a number or a multiple-choice response. No other pair of outcomes comes close.

Perhaps the biggest loss here is the fact that I constructed an equation exactly three times throughout my eighth grade year in Khan Academy. Here is one:

This is troubling. In the sixth grade, students studying the Common Core State Standards make the transition from “Number and Operations” to “Expressions and Equations.” By ninth grade, the CCSS will ask those students to use equations in earnest, particularly in the Algebra, Functions, and Modeling domains. Students need preparation *solving* equations, of course, but if they haven’t spent ample time *constructing* equations also, those advanced domains will be inaccessible.

**Smarter Balanced Verbs**

The Smarter Balanced released items ask comparatively fewer “calculate” and “solve” items (they’re the least common verbs, in fact) and comparatively more “construct,” “analyze,” and “argue.”

This lack of alignment is troubling. If one of Khan Academy’s goals is to prepare students for success in Common Core mathematics, they’re emphasizing the wrong set of skills.

**Smarter Balanced Productions**

Multiple-choice responses are also common in the Smarter Balanced assessment but the distribution of item types is broader. Students are asked to produce lots of different mathematical outputs including number lines, non-linear function graphs, probability spinners, corrections of student work, and other productions students won’t have seen in their work in Khan Academy.

SBAC also allows for the production of free-response text while Khan Academy doesn’t. When SBAC asks students to “argue,” in a majority of cases, students express their answer by just *writing* an argument.

This is quite unlike Khan Academy’s three “argue” prompts which produced either a) a multiple-choice response or b) the re-arrangement of the statements and reasons in a pre-filled two-column proof.

**Limitations & Future Directions & Conclusion**

This brief analysis has revealed that Khan Academy students are doing two primary kinds of work (analysis and calculating) and they’re expressing that work in two primary ways (as multiple-choice responses and as numbers). Meanwhile, the SBAC assessment of the CCSS emphasizes a different set of work and asks for more diverse expression of that work.

This is an important finding, if somewhat blunt. A much more comprehensive item analysis would be necessary to determine the nuanced and important differences between two problems that this analysis codes identically. Two separate “solving” problems that result in “a number,” for example, might be of very different value to a student depending on the equations being solved and whether or not a context was involved. This analysis is blind to those differences.

We should wonder why Khan Academy emphasizes this particular work. I have no inside knowledge of Khan Academy’s operations or vision. It’s possible this kind of work is a *perfect* realization of their vision for math education. Perhaps they are doing exactly what they set out to do.

I find it more likely that Khan Academy’s exercise set draws an accurate map of the strengths and weaknesses of education technology in 2014. Khan Academy asks students to solve and calculate so frequently, not because those are the mathematical actions mathematicians and math teachers value most, but because those problems are easy to assign with a computer in 2014. Khan Academy asks students to submit their work as a number or a multiple-choice response, not because those are the mathematical outputs mathematicians and math teachers value most, but because numbers and multiple-choice responses are easy for computers to grade in 2014.

This makes the limitations of Khan Academy’s exercises understandable but not excusable. Khan Academy is falling short of the goal of preparing students for success on assessments of the CCSS, but that’s setting the bar low. There are arguably other, more important goals than success on a standardized test. We’d like students to enjoy math class, to become flexible thinkers and capable future workers, to develop healthy conceptions of themselves as learners, and to look ahead to their *next* year of math class with something other than dread. Will instruction composed principally of selecting from multiple-choice responses and filling numbers into blanks achieve that goal? If your answer is no, as is mine, if that narrative sounds exceedingly grim to you also, it is up to you and me to pose a compelling counter-narrative for online math education, and then re-pose it over and over again.