[Help Wanted] Does The Medium Matter?

2011 June 13: No more help needed on this one. Thanks for piloting this study for me, team.

If you’re still in school, if you still have students around, I’d be thrilled if you’d help me answer a question that’s nagging me.

If you pose this question to a student:

It took Michael Benson six minutes to run one mile in a timed trial at his school – his fastest time ever. How long would it take him to run a 26-mile marathon?

Does the student multiply six by 26 getting 156 minutes? Or, more sensibly, does the student multiply six by 26 getting 156 minutes and then add some time to compensate for the fact that Benson isn’t going to be able to maintain his fastest time for one mile over 25 more of them?

My speculation is that math class pretty effectively conditions that sensibility out of a student by the fourth or fifth grade. It’s very difficult to succeed at math class if you don’t train yourself to ignore entropy, gravity, friction, and a million other factors mitigating these tidy word problems.

But does the medium matter? If you present those two sentences in black and white on paper, are the results worse than if you present them in a medium that’s more befitting the context? Like a video? Or a newspaper clipping?

How You Can Help Me

  1. Print out equal numbers of these four pages. They’re four different versions of the same question.
  2. Shuffle the four versions and pass them out randomly, in equal numbers, to your students. (Doesn’t matter to me if this is calculus or sixth-grade math. The more classes the better.)
  3. Tell them you can’t answer any questions about the problem. They should write those questions down. Let’s leave calculators off the table. Ask them to do their best and to be sure to write down their reasoning.
  4. Give your students as much time as they need to do this one problem.
  5. Collect the papers.
  6. Get the papers back to me:
    • via email: dan@mrmeyer.com
    • via fax: 831.325.0095
    • via mail: PO Box 429, Mountain View, CA 94040

I’ll need to know the letter grade of each student, too. And the course they’re taking. Can I get any volunteers to raise a hand in the comments? I’m not going to pretend this isn’t a huge favor. You can consider me in your debt.

2011 June 7. To clarify, would you be sure to send the forms back without the student names. Feel free to just attach numbers to the top. (“Student #1,” “Student #2,” etc.) Or anonymize them in some other way. I only need some way to link the student’s response to her overall class performance.

About 
I'm Dan and this is my blog. I'm a former high school math teacher and current head of teaching at Desmos. He / him. More here.

101 Comments

  1. wish I had a classroom full of kids. That sort of rhyme between how the original problem – or your hypothetical statement of it – eliminates all the real world stuff in order to teach “math” and how you’re isolating the “math” in order to (maybe; hopefully) show how the real world stuff absolutely inflects it is pretty darned elegant.

  2. Would it be helpful to have physics classes? I have a small sample size, but I’ve got them until Tuesday.

  3. I agree that in math we factor out the friction, but that question is pretty darn misleading too.

    “How long would it take him to run a 26-mile marathon?”

    Does anyone else feel like “at this rate” is implied at the front end of that question? Or that, since an exact answer seems also implied, that we must go on his mile speed because everything else would be conjecture? If the question was, “What would be a reasonable/believable time…” or something of the like, I think it would more closely match the latter format.

    All that griping aside, yes, the second format does push the real world. But answering it becomes far less mathematical and far more intuitive, doesn’t it?

  4. Mark Betnel

    June 6, 2011 - 7:43 am -

    I don’t have a class full of kids, but I have a prediction that none of the four sufficiently turns on the “No, really, I want a _real_ answer” framework and turns off the “Ignore everything but the statement” filter.

    The last one, maybe, but it would take a smart-aleck.

  5. I’m pretty sure that most of my students would interpret all versions of this problem as the equation \frac{6 {\rm min}}{1 {\rm mi}} = \frac{x {\rm min}}{26 {\rm mi}} in disguise if the context was math class.

    In a physics class, the same students would tell me that there’s no sensible numerical solution as one would need to make additional assumptions regarding the velocityvs time function.

    Sadly, math class remains the only place where people buy 60 water melons and no one wonders why.

  6. Maybe students will respond with more care outside of math class. Like in an English class, or social studies.

    I have a hunch the answers would be much better if kids could talk to one another in small groups.

  7. Graeme Campbell

    June 6, 2011 - 7:58 am -

    Wait…letter grade? You mean the yearly mark of the kid that wrote the paper? Not sure I can do that….

  8. I make liberal use of words like “approximately,” or even better, “Roughly how long would it take him to run a 26-mile marathon?”

    I don’t know if I could help with this project, but I’d sure like to see the resulting data. I think it would be a safe bet that in a fairly traditional algebra class, there will be a negative relationship between the quality of this question’s answer and the final course grade of the student.

  9. Erin Couillard

    June 6, 2011 - 8:33 am -

    My grade six class in Calgary will take a stab at this. Hopefully by mid-week.

  10. Yes, but how much additional time should you add? When I used to job 6 and 8 miles a day on alternate days, I ran them at about 6 minutes/mile. However, I didn’t run a single mile any faster than the 6 or 8 miles. I simply had one speed regardless of distance.

  11. I don’t like the “six minutes — his fastest time ever.” Is this talking about a one-mile race or what? This, for me, is the crux of the problem and what puts in the realm of psuedocontext.

    I’d rather dump the picture of the finish (burst of speed and all that implies) and include one of the kid in the middle of the race.

    Now use a split time, something that all runners are familiar with and is quite useful. “From here to there on the course is a mile and Michael completed that portion in 6 minutes. How long did he probably take to finish the 26.2 mile race?”

    Just for comparison, the world record is 2:04 for the 26.22 miles, a 4:43 pace.

  12. I’ll get more detailed info to you soon, but on the first go-round:

    19 students
    10th and 11th graders, mostly 10th
    very low achieving Alg 1 class
    average grade in class C- (I used SBG or whatever you want to call it)
    1 out of 19 factored in decreasing mile time over 26 miles, everyone else came up with 156 min, except for those who had multiplication errors

  13. Cherra Olthof

    June 6, 2011 - 9:13 am -

    Grade 8 class in Southern Alberta. Filled with both academics and struggling learners. We will do it. So do you want me to “grade it” before I send it?

  14. Hadley Wickham

    June 6, 2011 - 9:43 am -

    If you want to release this data in any way you really need to get IRB approval. It’s generally pretty easy for anonymous results.

  15. Macon Lowman

    June 6, 2011 - 9:47 am -

    I have loved reading (and receiving tips) from your website! I will definitely do this with my 6th graders tomorrow.

  16. I will happily assist you with my 125 students (8th grade). My experience is that almost every student wants to add minutes but “knows” this won’t produce the “right” answer. Sadly, though, it will produce the right answer…

  17. Dan, will you accept scanned copies of originals in PDF form or do you really need the originals? Thanks.

  18. Thanks, everybody, for the generous help.

    @Ms. Poodry, I’m interested in how other classes – physics, included – perceive the mathematical model.

    @Tim, it isn’t in my interests to bother myself or any of you with a strawman. My experience is that textbooks use the future tense in these problems, inviting speculation on an event in the future. Check out the four versions of the problem. Two of them use the future tense. Two of them ask the student to fill in a blank describing an event that already happened. I split those trials across the media. We’ll find out if that wording matters or not.

    @curmudgeon, I’m not remotely clear how this is pseudocontext.

    @Cherra, feel free to grade them for your own use, if you like. If you wouldn’t mind leaving the student work blank, though, I’d be obliged.

    @Laura, let me encourage you to have that conversation with your students afterwards. (ie. “how many of you wanted to put down a different answer but didn’t feel like it would be correct?”) I’m interested in that discussion.

    @Michael, definitely. PDFs are probably the most preferable format.

  19. After the magic that “Graphing Stories” created in my class this is the least I can do for you! And I know how much work Graphing Stories is for you now so consider us even! :)

    I’m glad I read this tonight – I have the last class tomorrow – I’m on it!

    Thanks!
    Julie

  20. I still have a classroom full of 8th grade math students. Would be happy to help. Will do this in tomorrow and Weds classes.

    Would the current grading period grade be enough for each? I probably won’t get to scan them till the weekend since I have to do them one page at a time. Will that work for you?

    Cheers!
    Dvora

  21. I will do this with my 24 kids this week after we come back from overnight camp (gahhhhhh…) on Wednesday. I’m very eager to see what they come up with.

  22. @Dvora, the current grading period grade would be great. And the weekend is fine. Don’t hurry, actually. I’ll be analyzing the results over the summer. Thanks!

    @kt, thanks a mil!

  23. Eric Biederbeck

    June 6, 2011 - 4:25 pm -

    My 6th graders will take this on as well! I’ll have them by the weekend for you!

  24. I’m pretty sure sending both the students’ names and their current letter grade in the class would be violating privacy laws here in Canada.

    Maybe remind people to either anonymize it, or skip the letter grade.

  25. Hi Dan,
    I’m in the southern hemisphere, so I’m on break at the moment. In two weeks I’ll be back in the classroom and I think this is a great thing to start the class with. I’ve got 120 students in different classes from 7th grade to 10th grade. I can easily turn the answers into a PDF and send it to you. My questions are:

    Would getting the results on the 23rd or 24th of June be too late for you to be useful?

    Do you care about results from other parts of the world? If you’re specifically after US data then I won’t bother.

  26. First, Dan, I commend you for using a picture of a kid whose real name is Forrest and not spinning it into a “Run, Forrest, Run” reference. (Sorry – After being around small-school Colorado sports for most of the last decade, I recognized the school right away!)

    As for the math: For the students who answer 156 minutes I’d like to see how they handle a follow up question like: “How long would it take Michael to run 100 miles? A million miles?” Perhaps that verges on the kind of step-by-step textbook handholding you generally would like to avoid, but if it’s all it takes to get students to realize and fix their own mistake then I’d say it’s worth a shot.

  27. Hey Dan, happy to try this out with my class.
    They’re a class of 22 12yr-olds studying Maths in the Scottish system. Is that ok for your trail?

  28. This is only marginally related, but Dan, are you familiar with Marshall McLuhan? I can’t imagine you’re not, although I’m only familiar with him because he (and his maxim “The medium is the message”) are mentioned regularly in one of my favorite books, Neil Postman’s “Amusing Ourselves To Death”. In short, the medium matters way more than we think it does.

    “Amusing” was written in 1985, about how television is (was) turning all aspects of our culture into entertainment. Even though it was written over 25 years ago, so much of it still speaks to the way new media affect (infect?) our culture in ways we don’t even notice. I really can’t recommend it highly enough (if I haven’t already).

  29. Jan van Hulzen

    June 7, 2011 - 5:41 am -

    Why not make it plain that the context is not the issue, just ask them questions like:

    Popeye lifts a pebble with one finger, how many fingers does he need to lift a train?

    I think the real issue here is to make them question the question instead of answering it. Mathematical or otherwise. I would love if my son would ask me why the coyote orders all that elaborate stuff to catch the road runner if he could just as easily order a pizza.

  30. Dan, I’ll preface my comments by saying that I may be taking this out of context, if so, sorry. I come from a background of instructional technology. The question, “Does the Medium Matter” has been answered long ago. In the 80’s I think. The two researchers who argued opposing views were Richard Clark and Bob Kozma and this line of research was referred to as “Media effects” research. The findings are really timeless and applicable today. Personally, I have always taken Clark’s side of the argument. He offers the analogy that media is nothing more than a grocery truck carrying the the goods. The goods in this sense are the instructional strategies that you employ. So employing media to help kids learn math is still dependent on the learning strategies that teachers employ when using the media. For instance, a simulation helps in visualization but it still boils down to the instructional strategies that the teacher employs related to that simulation. Again, if I misinterpreted the question, sorry.

  31. James McKee

    June 7, 2011 - 7:33 am -

    Dan,
    I’ll do it with my Algebra I classes this Thursday and mail them before the weekend. However, I don’t think I can include their letter grade on mine either. Privacy and all that….

  32. Dan,

    Would grades without names solve the privacy issue and still give you the data you want? You don’t really need the students’ names, right?

  33. Let me clarify quickly (and I amended the main post, also) that I don’t care that “Sarah Scofield” has a C+ in your class. Please don’t tell me that. I care that the person who completed a given test has a C+. Does that make sense?

  34. Sue Popelka

    June 7, 2011 - 9:58 am -

    Done and in the mail. Students enjoyed this and out discussion afterwards. Also did “how many tickets ont he roll” today, our last day of class.

  35. lfarrington

    June 8, 2011 - 6:32 am -

    If I had just ONE more week of school I would have time to do this… but it looks like you found plenty of volunteers. I’m curious and anxious to hear the results of your analysis.

  36. Dan,
    I had the discussion…most wanted to put that down but didn’t. Almost all knew/accepted he would slow down and not run full tilt to the end of the marathon but decided that we were looking for an “answer” not a generalization, in spite of the fact that there was an opportunity to explain – not multiple choice…

  37. As someone who is athletic, this question bothers me a little bit. I have looked at enough lap times for cyclists to know that times are not strictly increasing (that is, velocity is not strictly decreasing). While I recognize that a marathon will be run more slowly than one’s fastest mile, the dynamics of human performance are complicated and we are all going to make assumptions that simplify the problem to one we can deal with. I’m not sure that I’m even convinced that realizing that you should take your fastest mile time and multiply it by some factor (which will be different for every runner) to get your average per mile time for a 26 mile race is all that mathematically interesting.

    I suspect that at my urban charter school few students would consider fatigue as few students are involved in endurance athletics. To conduct an ad hoc study and not consider the context (school and students) in which the study is conducted seems like a grave mistake.

    What if you asked the same question but with a car traveling on a local road that has significant traffic?

  38. Marcie Shea

    June 8, 2011 - 8:29 am -

    I only have 40 students left at this point in the year, but I will give it to them. They are honors physics students in a variety of math classes. I will include my class info and their math class info on the paper.

    How would you like them sorted when we send them back to you? By version? By student grade? By answer?

  39. You’ll be getting 30+ from my 7th grade classes in Minnesota. It was interesting discussion afterwards. Amazing how little they apply reason to real life situations.

  40. From me, you’ll be receiving 5 sections total:
    2 of 8th
    1 of 7th
    2 of 6th

    I’ll have them all finished up by next Tuesday, 14 June, and e-mail them. Keep us posted on results!

  41. In the mail.
    I found the gender difference in response interesting, even though Dan did not ask for that.
    Given that girls “do school” better.

  42. I think the problem with this type of question is that while it’s “real world” (meaning it happened to someone), it’s not “real world for the students.” What does that mean?

    Consider buying loaves of bread a local baker. A loaf costs $3.50. How much would 15 loaves coast? First, how often do people buy 15 loaves of bread at once and if they do, wouldn’t they at least ask about a volume discount? People do buy bread, but this type of problem discourages rational thinking and results in relatively meaningless computation.

    Likewise, the running problem removes the element of personal experience (except for the students who are also runners and get the fatigue factor). I think a different approach might be to take students outside and have them run a mile and then estimate how long it would take them to run 2 miles, 5 miles, a marathon, etc. After doing that, maybe they can then address the 6 minute mile question with both additional questions (are we assuming constant speed or should we factor in for fatigue, etc.) and active thinking. And the kid that points out that s/he would die before the end if they tried running a marathon gets extra credit for bringing reality back to math.

    I realize that’s doesn’t answer your question about the medium, but maybe the fact that it’s still a worksheet is the problem we should be addressing first. :)

  43. Yeah, see, there have been a number of really good strategies raised here and elsewhere for how to make the context more meaningful to students – is having the students run a mile one of those? – how to drag this problem away from math-land (where entropy, friction, etc., do not exist) and back to Earth.

    The point of the study, though, is that those things aren’t usually done, perhaps leaving our students with a distorted sense of math’s application to the world around them. It does us little good to test the best-case scenario. It’s more interesting (and useful) to test the common-case scenario.

  44. There are lots of interesting mathematical modeling problems. I am not sure that students who have not made the leap from repetitive addition to multiplication are ready for real world models, which are more complicated.
    I don’t have a problem with “add some” for the marathon. A mathematical model would have some secondary function multiplying the primary function. The “add some” idea is more like statistics, where you take a statistical wild-ass guess (SWAG) and then see how likely you are to be right.
    I still think we are discussing the difference between pure and applied mathematics. Students need to learn the rules of mathematics, just as they need to learn the rules for written English (if they want to be able to use them). They might want to be able to move on to applied math, or pure math, or figure out the mortgage, just as they might want to write a novel or a scientific paper or leave a note for the milkman.
    It’s the reason why “calculator solutions” are not germane. You haven’t learned how to use the math, you haven’t learned to model a real application – the exercise had no value. IMHO.

  45. I will have ~80 7th and 8th grade students do this on Tuesday. I still have them for 2.5 weeks!
    We have looked at individuals jumping jack rates over time in the class before, so it should be interesting.

  46. Sorry if this comment is reaching you a bit late Dan.

    It seems like your hypothesis is (a) changing the medium matters and (b) the mechanism by which changing the medium will matter is that fewer students will fall back on learned habits of ignoring entropy, gravity, friction, etc.

    It’s not clear to me that your experiment tests hypothesis (b). Also, wouldn’t any arbitrary method of “alienating” the student (in the theatrical sense of alienation: http://en.wikipedia.org/wiki/Distancing_effect ) produce a similar effect? Put a different way, does the medium only matter because it’s different/novel compared to the norm?

  47. I’ve got a class set of grade 8 responses for you. Two kids thought beyond the basic answer. Very interesting to see their marks — one kid with 89% and one with 63%. Also interesting to see the number of A+ students who DIDN’T use any reasoning.

  48. Well I gave these out to my 38 8th grade math students last week (will work on the scanning this week and will then get to you) and over all only 2 students had any thoughts about the runner slowing down, but only wrote about it and did not calculate with it. Neither are my top students either. I was so proud of them.

    I think they are so used to these types of problems from books and standardized testing that they did not even think about the true context of the situation… something I plan to remedy for next school year.

  49. Matt Fletcher

    June 14, 2011 - 6:36 am -

    Two classes of grade 7 students (52 kids in total) and NOT ONE student dared to go beyond the basics of 26×6=156 minutes. No one even mentioned the idea of the runner slowing down – and there are a few kids in each class who just took part in a cross country running event!

  50. Ze'ev Wurman

    June 14, 2011 - 7:00 pm -

    Sorry to come late to this party, but much of this discussion strikes me as meaningless.

    Whether one form or another of the question has a significant effect on the outcome is a question for psychometricians, and even a half-baked one would know that (a) 20, 40, or even 100 kids without much control over the administration can tell nothing meaningful to no one, and (b) these type of results, even when applied to a much larger population and under good controls cannot tell much *except* for the actual empirical differences for these specific examples, if any. So much for Dan’s “experiment.”

    As to various discussion over the mathematical, rather than psychometric, value of the results, I would simply point to the initial instructions, where Dan clearly stated that the grade doesn’t really matter (“calculus or sixth-grade math” in his words). Unless someone truly believes that students in calculus class know no more math than sixth graders, this should tell everyone this is not about the math, or any high fallutin’ “math reasoning.” The only “math” here is a third grade integer multiplication and one can reasonably assume that most kids get that part.

    The simple (and the only one, really) answer to this question is “greater than 6*26.” The kids know this much — ask them any of the questions mentioned by a few commenters above, and that would be easy to ascertain. What the kids also know is that some fool of a teacher asks them this question, so the only *mathematical* answer that teacher must be after is 6*26. So they will hand it to him, perhaps hoping that he will finally get back to his job, teaching them math rather than stupid quizzes.

    Guys — what you see is the result of your own training the kids to give stupid answers to stupid questions. Many, often most, questions students are asked are ill posed and the students learn to guess what the teacher wants rather than to apply serious reasoning or serious math. You simply see the results. It is not the students that are the fools here.

  51. Ze’ev: As to various discussion over the mathematical, rather than psychometric, value of the results, I would simply point to the initial instructions, where Dan clearly stated that the grade doesn’t really matter

    I “clearly state” nothing of the kind. I’d like as many grade-level cohorts to participate as possible. A student’s prior knowledge will almost certainly be significant here. The only way to test that hypothesis is to apply the experiment to grade levels indiscriminately and then control for that level in the regression model.

    Ze’ev: these type of results, even when applied to a much larger population and under good controls cannot tell much *except* for the actual empirical differences for these specific examples, if any. So much for Dan’s “experiment.”

    As my father used to say, “you can’t reason with anybody who rejects a priori the results of a study without seeing data or an effects analysis.” (Okay, my dad never said that. Doesn’t mean it isn’t true, though.)

    Ze’ev: Guys – what you see is the result of your own training the kids to give stupid answers to stupid questions.

    Wait. So the results of the study do matter now, but only if they implicate teachers, not the “world class” standards you helped author in California and influence elsewhere. Those standards have nothing to do with what’s included in our textbooks, valued in their answer keys, and consequently taught in our classes.

  52. Ze'ev Wurman

    June 14, 2011 - 9:49 pm -

    Dan,

    It seems we are not communicating. Let’s try again. Responding to your three points in order:

    1. The level of the math procedure needed for this problem is about third grade. That’s not really disputable. So nobody expects a priori almost anyone to fail on what you call “prior knowledge,” assuming you refer to mathematical prior knowledge. Sure, you can test it, but what you are testing is in all likelihood not the procedural skill. And whatever the results will be, you cannot draw any strong conclusion as you don’t have information about the students are (background variables) or how the actual administration happened. (Their prior grade is such a weak proxy that I don’t even want to go there).

    2. Here we discuss the psychometric characteristics of the four test variants rather than the math aspect. If you had any experience with that you’d know how weak results of a few dozen uncontrolled administrations are. GIGO. I’d give “your father” a bit more sense than you seem to, and I would expect him not to ignore a study construction and instead try to analyze its results however flawed the study was. If the study is badly constructed on its face, there is little sense to analyze its results. GIGO again.

    3. Here you simply engage in ad hominem attack. My “your own training” was generic to the teaching profession rather than specific and addressed the fact that my analysis of the problem, as I explained, shows that the outcomes will not depend on any mathematical difficulty (which is quite trivial) but rather on students trying to guess what will satisfy their teachers.

    Bottom line, this is a badly constructed “study” of anything I can imagine. And hence the results will be meaningless.

  53. Perhaps a separate post will provide some background.

    In a few studies of the NAEP math items in recent years, mathematicians found about 20-25% of the items mathematically incorrect. Errors ranged from missing information, through unstated assumptions, to simply wrong answers. Yet psychometrically these items “performed well” — proficient students provided “correct” answers (i.e., what item writers deemed correct), less proficient student performed less well on them, etc.

    This seems to me a clear demonstration that to a degree we train students to guess what we want them to answer, rather than value correct thinking.

  54. Ze’ev: I cannot know your intentions, but I do know that for me the tone of your first comment came off as hostile-not exactly useful in promoting a healthy debate of the reliability and validity of this “study”. That said, I will assume good intentions. I too am hesitant to make any conclusions based on the few comments and my gut that most students will compute 26*6, but I join Dan in my surprise of your seemingly thorough rejection of any potential worth without dan sharing any data or making any conclusion himself(never mind the fact that last time i checked this is not a peer reviewed journal). That said, let’s assume you’re right and this study is unreliable and invalid. What concerns me most about your two comments is the blanket dismissal (at least that’s my perception) of any usefulness of this blog post (that is, after all, what it is). If nothing else, it seems that a number of people are now reflecting on their own practice and thinking about how students respond to questions they are asked in the medium of the math classroom (however this question was administered). I think that is valuable, especially if the results are surprising to the teacher (not so different from analyzing that last unit test on quadratics). Strikingly, it seems you do believe (or aren’t surprised) that many students will give 26*6 as an answer as they will simply try and “satisfy their teachers”. If that is in fact the case, that’s fascinating. Even if we disagree on the root cause of students blindly ignoring reality, even if we disagree on how frequently students are expected to ignore reality in psychometrician-and-researcher approved curriculum and tests, can we at least agree that dan’s study (would you feel better if it were just called an exercise to promote reflection) isn’t wasted bandwidth?

  55. I’ve read at least one if the NAEP studies you refer to…for some context, most of the “mathematically incorrect” answers are more “mathematically overly simplified” answers. If my memory is correct, something like a question on a 7th grade tests ignores possible imaginary answers. I see this as a very different kind of “mathematically incorrect” than the question posed in this post. To use an extreme example, no one believes we should be teaching 5 year olds set theory in order to allow them to conclude that 1+1=2. That said, it seems that most people would agree that students of all ages should read this question and go, wait, I don’t have enough info to answer this question precisely. The point is that ‘mathematically correct’ is a human construct.

  56. Russell Collins

    June 15, 2011 - 12:06 am -

    Dan: The experiment proposed here is applicable all the way through to “competitive” collegiate courses where a student’s presence in the course implies some level of reasoning ability uncommon among their peers.

    Anecdote: Freshman Calculus-based Physics class at the University of Washington circa 1998. Vladimir Chaloupka is doing his best to balance his desire to remind his class that reasoning skills are more important than simply parsing the text of the question for the minimum information necessary to arrive at the “Right Answer(TM)”. His department in the University uses his class to winnow the pool of aspiring students down because there are simply too many freshman (thank you, permissive admissions standards) for the higher level classes so each group needs to get hammered down by curved grading standards.

    The class is full of students mostly aware of the curve’s implications for their education: It doesn’t matter what you learn if you don’t learn it better than 70% of the class. Literally they’re in a system where a high school education that taught them how to play the education game (i.e. the stupid answer to the stupid question matters because they are the ersatz gatekeepers to progress) is TO THEIR ADVANTAGE. Any reasoning in their content area is a subordinate result measured against competition.

    You could try to reason that if the system were designed well enough that you can’t compete well without learning content reasoning. But the fact is, that while it is possible to imagine such a system, it remains a red herring. Such a system does not confer any value inherent with achievements against a static target such as a standard (no matter how well-designed, and well-intentioned that standard may be). A perfect standard does nothing to affect a competitive system’s selection process based on a limited number of available seats per term. You could be applying for a place in that class with a group of geniuses who negate any achievement you made against the standard simply by out-achieving you.

    So what was poor Dr Chaloupka’s distress all about? To begin with, he had just been the bearer of odd tidings 6 weeks prior when he returned our midterm tests to us, heavy in heart. He slyly admitted to the class that our grades were “embarrassingly high”. You see, our class was supposed to be designed to be difficult enough for students to average only about a 50% correct answer rate on the test and we crushed it. We averaged 75% correct. His dean was livid. We were too smart for our own good (or too good at appearing smart on standardized test questions).

    Here we were 6 weeks later engrossed in the course final exam. His curriculum standardized across the department. His grading curve had been dutifully adjusted downwards to give surpassingly good work (remember those 75%’s? Yeah, we kept kicking ass) artificially lower marks so that my 2.9 would have been any other term’s 3.5. But something surprising was about to happen. You see, the brilliant Russian pipe-organist of a physics teacher in him could not resist playing a subtle trick on his class. He asked a question with vague data. We were told a woman surmised a pipe on a pipe organ in front of her was roughly 3 times her height. We were given other dimensions but we weren’t told specifically what her height was.

    It is sad to me that I observed more than 20 students head to the front of the class with frustration on their faces hoping to get him to clarify that question only to return to their seats exasperated and dejected, occasionally furious, that he would have the audacity to tell them not only had he given them more than enough information for that question but that he now regretted spending the quarter giving them TOO much information. We were to calculate the note that pipe would produce. Not the frequency. The note. This is subtle because the chromatic scale sets targets for frequency that define certain notes while frequency itself is continuous and infinite, notes are discrete and few. Therefore a range of reasonable guesses at the height of the woman would produce frequencies in the close vicinity of a specific note (given the other hard data provided). This is the same concept in light where we’ve arbitrarily determined certain arbitrary, discrete wavelengths are Red, Blue, Yellow, etc.

    To get that desperately sought “RIGHT ANSWER (TM)”, a student needed only the merest reasoning skills, hardly even the skill, even just the bravery to resort to it. Comparatively it is such a rudimentary tool; so much less useful, sophisticated, and well-developed by this point than a student’s wrote memory.

    That must have been a depressing quarter for the old Russian. First, to have his students shine so brightly on a midterm as to have his dean call him out on not being able to control the grades effectively and squash the prescribed proportion of dreams regardless of individual mastery, this must have been tough medicine to swallow. The spoonful of sugar may well have been the sweet delight an attentive teacher gleans from a class full of brilliant scholars, thirsty for wisdom. That sugar was surely soured when our corporate performance on the final exam was encumbered by the inculcated blindness to so low-hanging of fruit that stepping blindly forward you would brush your face against the laden bough on this tree and have it for the taking at the cost of just a single step.

    Ze’ev: Wil Wheaton has asked you to observe a standard of etiquette on the Internet that your post fails to conceal. I don’t envy you your work. California schools are a tumultuous rookery of hydra. Driving improvement in any production line that complex and defective by design is Sisyphean even on a good day. That being said, you forget that Math is no more the ends than a hammer is. They are tools. You’d do no more good teaching a student only the mechanism and design of a hammer than you would the mechanism of multiplication. Instead the value is in asking them to imagine and realize its use. To master the tool is to free the mind from the fetters of merely sufficing with assembly. A system that promotes by competition cannot directly engender such mastery because it never asks you to imagine what you can do with your tools, merely construct a better widget than the next guy. Such a system would have encouraged the Wright Brothers to remain bicycle repairmen.

  57. Avery,

    Rereading my first post, I tend to agree with you. It does come across as hostile. I was annoyed with the underlying assumption of this exercise, and with what I presumed — admittedly always risky — will be the “lessons” drawn, with little or no justification.

    First, the facts. We are given four variant of a problem that needs trivial math skills, and whose difficulty stems solely from the non-mathematical ambiguity of the question. More specifically, two variants ask explicitly about what seems like the *number of minutes,* and two others are a slightly less restrictive when they ask “how long would it take,” although the form still implies a number.

    Now, essentially all kids above age10 know that one cannot run at a given speed forever. If you don’t believe me, ask any whether it will take him to run 200 yards exactly twice as long as it takes him to run 100 yards. Kids run these distances in PE by then, and they know the answer. Some responders already said as much.

    Yet the kids mostly answer 6*26. Why? They clearly know the operation needed. They clearly know that one slows down over longer distances. They know the don’t need any fancy formulas here. So why?

    Any normal kid will look at the question and come up with the feeling the question — in all its variants — expects a *precise* answer. Well, there is no other data to provide a precise answer, except 6*26. So, their thinking goes, the teacher is an idiot for asking a clearly unrealistic question here but, what the hell? If that’s what he wants, that’s what he’ll get. End of story.

    If a kid is a bit more “brave” he will — at best — say “longer than 26*6” based on his non-mathematical experience of real life. But there is no obvious place-holder for this answer in at least two variants, so the kid know he’s pushing it. If the teacher is smart, perhaps he’ll get it. If not, he’ll be marked down. Why risk it?

    Now I read the some of the comments and I find “students not using reasoning” and similar. Bull. They did. It is the form of the question that dictated their responses.

    So my annoyance was that people may conclude that “students can’t apply their math.” Perhaps. But this badly designed experiment certainly can’t tell us one way or another.

    But I shouldn’t have let my annoyance show. Sorry.

  58. Russell Collins

    June 15, 2011 - 12:31 am -

    Ze’ev: (part 2 and more on-topic) I am really only concerned that a study in tools usage could be summarily dismissed as “meaningless”. Nothing is meaningless to the curious. Einstein found the beachstroller’s buoyancy on sand relative to the amount of water in the sand meaningful though he didn’t construct a formal investigation with blinds, regression controls, and peer review.

    Instead of pedantry and deigning his blogged hypothesis “hoist by its own petard”, try re-examining it and suggesting a satisfactorily rigorous academic deconstruction of its components into more testable pieces.

    “My speculation is that math class pretty effectively conditions that sensibility out of a student by the fourth or fifth grade.”

    Factors I see ripe for consideration:
    1) Is it the “math class” environment, pedagogy, curriculum or something more broadly systemic?
    2) What is the threshold for effective conditioned removal of that sensibility? Under what circumstances is that relevant and manifest in student comprehension/test performance?
    3) Is the Fourth or Fifth Grade significant? If so, how?

  59. Ze'ev Wurman

    June 15, 2011 - 1:00 am -

    @68 Avery: No, many of the errors are of a very simple nature, nothing fancy, nothing to do with complex numbers. Often one of those pattern problems when they expect you to guess the Kth member based on some unstated assumption of a linear (or quadratic) progression. If I recall, one even had some relation between (young) age and weight exhibiting some such pattern, asking about a table entry few years ahead. If you were to take them at their word, you would believe that when you get to the age of 40 or 60 you would weigh many hundreds of pounds (smile). There was a NAEP validation study circa 2008-9 that had some examples. Tom Loveless (Brookings) also had a similar study some time back.

    @71 Russel: I find value in not trying to over-analyze flawed results. It consumes enormous amount of time and the analysis will be speculative anyway. It is much easier to design a study properly up front. In our case, mixing a study of the impact of variant posing, with the study of student’s ability to apply & interpret math in terms of reasonability, was a bad idea. Each could have been studied much easier in isolation.

  60. Research is not born like Venus, complete and on the half-shell. Why this experiment (no quotes) is worthwhile to me is that it offers an opportunity to notice. Dan asks for some interesting data points that extends what he has noticed in his daily practice. When the data comes in, then – if something interesting is noticed – it might be time for someone to conduct a thorough and rigorous experiment. This is a step along the way, and Dan was, in my opinion, very careful about not claiming implications out of proportion with what he was doing.

    If critics wish to be helpful, they could, of course, suggest ways to better the design of this initial data gathering. I think it’s indicative of a new type of teacher action-research, using crowd sourcing to improve the data gathered. Instead of just in your own building, someone like Dan has the ability to gather a remarkably broad collection of data.

    That all said, I don’t think that this is that far off from the research that does regularly get published in its experimental design. Not the biggest studies, but the run of the mill stuff.

    It also meshes with my experience. I would expect more experienced students to catch the naive assumption. And more of any level the more realistic the news story looks. This is a bit of a psychometrician’s question, but the testing climate has made us all amateur psychometricians.

    My thanks to the people responsible for that.

  61. Russell mentions Vladimir Chaloupka. Vladi is truly a wonderful man. You should listen to this song about what Vladi has to day about the universe:

  62. In our case, mixing a study of the impact of variant posing, with the study of student’s ability to apply & interpret math in terms of reasonability, was a bad idea. Each could have been studied much easier in isolation.

    @Ze’ev: I feel like you’re misunderstanding the study here. The point of the study is the notion that one idea (adding in context) will help alleviate the other problem (interpreting how reasonable the math is). Hence it wouldn’t make much sense to isolate them. It’d be like testing alleged visual/aural learners with only a visual test.

  63. What I think is neat about Ze’ev’s post is that he seems to be calling into question the idea that some contexts don’t elicit reasoning.

    Seems that we’ve got a bit of a dichotomy set up:
    answer “26*6” = not reasoning
    answer “definitely more than 26*6” = reasoning

    The idea then is that context supports kids to reason if it’s good context, and hampers their ability to reason if it’s not (and by context I mean the whole milieu of math class).

    But the timing of this conversation is perfect… Check out this article in the NYTimes: http://www.nytimes.com/2011/06/15/arts/people-argue-just-to-win-scholars-assert.html?_r=1

    The premise of the article is that reasoning is not an evolutionary adaptation to get closer to the truth or accuracy or the best ideas. It’s a social adaptation. We use reasoning to win arguments.

    By that standard, kids are reasoning no matter what, thinking what answer will get me what I need in this context. In math class that would be a good grade. In the real world, it would depend on why you wanted to estimate his time.

    So I’d argue that it’s plausible that in both cases (straight multiplying and multiplying & adding a slow-down factor) the kids are reasoning, along the lines of:
    “what counts as a good answer in this context? how can I best generate it?”

    That’s a mathematically valid question. We want kids to answer this question differently if they’re in charge of handicapping the marathon runners, if they’re placing bets on the marathon, if they’re estimating what time to get to the finish line to see the kid cross it, etc. So “in math class” is just another one of those contexts, and it’s hard to say what would be a better “math class” answer than 26*6.

    If the question is, “what media help kids do the sorts of mathematical reasoning about estimation and what’s close enough that they will need to do everywhere that’s not math class?” then I think this is a decently-formed and interesting experiment.

    If the question is, “what media helps kids reason mathematically” then Ze’ev’s post raises the question, how can we say that 26*6 is not a well-reasoned answer? Jean Lave’s research about how kids see learning math in math class speaks to this idea…

    For teachers who wonder what to do about this: a teacher friend of mine invented the “star fishy” a little star symbol to draw on your paper when you’ve answered a question but you know “something’s fishy here…”. We’ve noticed that kids with wrong or unreasonable answers often know “something’s up” but without being able to recover from that fishiness to get a answer they’re more confident in, they stick with the wrong answer and cross their fingers. I wonder how many kids would give this problem a star fishy?

  64. Now I read the some of the comments and I find “students not using reasoning” and similar. Bull. They did. It is the form of the question that dictated their responses.

    I agree, and think this is important. In fact, kids almost always use reasoning to come to their answer–however wrong it is. The interesting part of teaching is getting to the root of faulty reasoning. The interesting part of this conversation is deciding which is the faulty reasoning (if it is inn fact an either/or), reasoning that you’re in math class, so 26*6 must be the answer the teacher is looking for OR reasoning that the runner will slow down so the answer will be greater than 26*6.

    So my annoyance was that people may conclude that “students can’t apply their math.” Perhaps. But this badly designed experiment certainly can’t tell us one way or another.

    I didn’t read this. I never saw this as being about whether students understand the mathematical content d=rt, but instead about the students playing (or not playing) the school game.

    How’s this for a study design:
    I call N randomly selected people. Half get the following question: Hi. I’m hoping you could answer the following question. Michael Benson ran a mile in six minutes – his fastest time ever. How long will it take him to run a marathon? The other half hear the following: Hi. I’m a math teacher and was hoping you can answer the following question. Michael Benson ran a mile in six minutes – his fastest time ever. How long will it take him to run a marathon?

  65. I have one substantive thing to add to the conversation, which is that Guy Brousseau (French mathematics education researcher and theoretician) fleshed out some of the theoretical issues at play here with the concept of the didactical contract.

    The basic idea is that students and teachers have entered into a tacit contract in which students know that teachers ask certain kinds of questions and expect certain kinds of answers, for which students will receive certain kinds of rewards.

    Posing ill-formed or incomplete questions-or disrupting the medium through which these questions are posed can be seen as violations of that contract. Yet students will tend to do their best to honor the contract.

    The classic example is something of this form.

    There are 26 sheep and 10 goats on a ship. How old is the captain?

    Through the didactical contract, students are typically inclined to offer answers to such questions (and to use the given numbers to arrive at these answers) rather than to point out that the question makes no sense.

    So in a sense, Dan is trying to figure out under what conditions he can release students from this problematic contract.

    That seems like valuable work to me.

    And I’m guessing that the data he collects here is not his dissertation data. So let’s not hold it to that standard, shall we?

  66. Ze'ev Wurman

    June 15, 2011 - 1:18 pm -

    @78 Avery:
    I never saw this as being about whether students understand the mathematical content d=rt, but instead about the students playing (or not playing) the school game.

    Well, I obviously did not. Nor did some of the responders. However, you may be right. Let me try to roll with it.

    You suggest two variants:

    (a) I’m hoping you could answer the following question. Michael Benson ran a mile in six minutes – his fastest time ever. How long will it take him to run a marathon?
    (b) I’m a math teacher and was hoping you can answer the following question. Michael Benson ran a mile in six minutes – his fastest time ever. How long will it take him to run a marathon?

    Both forms seem to me identical in the critical element How long will it take him to run a marathon? and both have the uncertainty reduced to the same “I’m hoping that,” which anyway comes across more as a form of address than as introducing uncertainty. So, in my judgement, those are no better than the original forms suggested by Dan and at best may try to tease out empirical, but non-essential (i.e., psychometric), differences.

    If I were to design such, I would use something along the following three forms (two open ended, one MC):

    (a1): Michael Benson ran a mile in six minutes – his fastest time ever. How long will it take him to run a marathon (26 miles)?
    (b1): Michael Benson ran a mile in six minutes – his fastest time ever. Can you estimate about how long will it take him to run a marathon (26 miles)?
    (c1): Michael Benson ran a mile in six minutes – his fastest time ever. How long will it take him to run a marathon (26 miles)? (i. 26*6 minutes; ii. less than 26*6 minutes; iii. more than 26*6 minutes; iv. can’t say*.)

    Incidentally, I suspect that forms b1/c1 will have somewhat similar distribution of responses, while form a1 will have mostly 26*6 as an answer. The difference between b1/c1 responses will be able to tell us something about how substantive is the difference between CR and MC forms.

    And I would need at least a dozen or more of this type of item-sets before I’d dare to draw any conclusion, for dissertation or for my own use.

    * I would probably also experiment with two MC variants, one with “can’t say” as the fourth choice, and one with something truly unsupportable like “26*6*3.” “Can’t say” is, after all, a somewhat reasonable answer.

  67. Ze’ev: I think we just might be interested in different questions (and now I’m not sure what dan was originally interested in). I’m interested in learning more about when students reason “well this is what school/teacher wants so i’ll answer 26*6” and when they reason “I can’t answer this question precisely so i’ll either guess something greater than 26*6 or just say there isn’t enough info”. I wonder if simply adding the phrase “I’m a math teacher” would change things. And yes, the beginning of the interview protocol could be cleaned up but im not plsnning on submitting this proposal anytime soon. It seems that you’re wondering if changing the phrasing (adding the word estimate, for example) or the medium (multiple choice).

  68. cheesemonkeysf

    June 15, 2011 - 3:42 pm -

    @79 Christopher Danielson – I was going to write a response about the ‘didactic contract’ issues (Brousseau) here but you have articulated them beautifully.

    I would only add a few clarifying points for those who haven’t been following the French-language and U.K.-based investigations of this and its importance for this discussion.

    The didactic contract is the socially defined agreement between teachers and students about what kinds of knowledge, learning, and understanding will – or will not be – acceptable in the math classroom.

    As Nunes and Bryant point out (they are developmental psychologists in the U.K. who study children’s mathematical development), we are discovering that children’s social and emotional conception of mathematics has a HUGE impact on how kids approach mathematical situations. The math classroom itself is a social situation, and most kids define this socially constructed world as having an inside and an outside which rarely, if ever, meet.

    In the inside world – the life they know inside the math classroom – they attempt to apply seemingly arbitrary rules from a bewilderingly large catalog in an attempt to please the authority figure and avoid humiliation, pain, or emotional devastation. This amounts to an attempt by discouraged math learners to imitate certain ritualized actions demonstrated by the teacher in an attempt to please the authority figure (“succeeding”) and avoid the pain of not-succeeding (“failing”)

    Meanwhile, in the world outside the math classroom, in their daily lives, many of these same kids will often exhibit an openness to becoming curious, to relying on their own powers of observation, reasoning, and representation, and to being receptive to exploring and experimenting with mathematical thinking that they check at the door of the math classroom.

    In other words, for most kids, the social and emotional dynamic of the math classrooms comes down to two things: seeking pleasure and avoiding pain.

    And this dynamic finds expression for many kids in a conditioned attitude of ‘learned helplessness,’ in which they abandon their own trust in their own powers of observation and thought in favor of trying to recall the rules they believe the teacher wants them to apply.

    To them, the social system of the math classroom sends kids the clear message that math class is the place where they should check their powers of observation and reasoning at the door in favor of whatever memorized procedure will please the authority figure within (i.e., the teacher).

    Those who succeed in pleasing the teacher will do well — and may also learn some mathematics.

    Those who do not succeed in pleasing the teacher will be trapped between compliance and confusion, locked in a cage and subjected to a barrage of questions whose answers must be drawn from a bewildering catalog of rules/formulae/procedures that must be correctly reproduced in order to receive a food pellet and avoid devastating electrical shocks.

    So basically, SBG, WCYDWT, and ANYQ’s are necessary (but not sufficient) tools and conditions for renegotiating the existing ‘didactic contract.’ And that will require that we also find ways to (a) help kids notice their conditioned responses of panic and learned helplessness when those feelings arise and (b) help them to rebuild trust in their own powers of observation, reasoning, and representation so that they can get back in the game and learn mathematics in ways that make sense to them — i.e., in ways that are grounded in their own reality-based observations, experience, and intelligence, rather than in various educated non-experts’ opinions and ideas about what children must “learn” in order to achieve mastery over mathematics.

    For more information presented in a non-embattled way, see Nunes and Bryant, Children Doing Mathematics (1996) and Anne Watson’s fabulous articles, “Key Understandings in School Mathematics” (parts 1, 2, and 3), published in Mathematics Teaching 218, 219, and 220.

    We are not training parrots here. Not on our watch.

    – Elizabeth (aka @cheesemonkeysf on Twitter)

  69. Oh, maybe I did this wrong. I told my students that I’d be happy to give them credit for completion, if they really wanted (it turned out that only one person cared) but I was not assessing their knowledge, they were answering the question to help out some guy they and I don’t know.
    Most of my kids chose not to do so, but the ones who are obviously bright had no problem writing down their thoughts about actual times. And the good girls who try so hard but have no spark of imagination did exactly as expected. But none of them did it to please me, as I explicitly told them it was not for me.
    As has been observed by others, announcing oneself as a math teacher tends to put a dampener to any social occasion, so I would expect some reaction to Avery’s declaration.

  70. Apologies to all for my absence from an interesting thread. Christopher Danielson has my proxy w/r/t the rigor & validity of this study. I’m not pretending this is something that it isn’t. I haven’t yet analyzed any of the returns but I can report that all participating teachers (eighteen so far) have reported results that enlightened their own practice. So there’s that, at the very least.

    Avery has my proxy w/r/t to the point of the study. (ie. not math attainment.)

    Ze’ev: And I would need at least a dozen or more of this type of item-sets before I’d dare to draw any conclusion, for dissertation or for my own use.

    Are all of your hypothetical item-sets text-based? I’m anticipating most students will be able to correctly multiply six and twenty-six. That isn’t the point of this pilot. I’m curious how different media can disrupt the classroom contract. Why is text our default media for assessing an inherently visual problem? If students see a photo of someone running (or video of same) what does that trigger? How does that affect their interpretation of the same question? None of your hypothetical test items help me get at that, if they’re all text-based.

  71. Ze'ev Wurman

    June 15, 2011 - 5:52 pm -

    Dan,

    The sample item set I suggested was indeed textual. As was, I might say, yours. In my opinion, the illustrations in two of your variants were meaningless. If at all, I would suspect they would act as distracters.

    In early grades I would go for reducing the linguistic load even further, something like “if the fastest you can run a mile is 6 minutes, about how long do you think it will take you to run 26 miles?” But with higher grades it is a bit of a lesser issue. (Note: using “you can run” as opposed to “Michael Benton” may have significant psychometric impact too, but that’s for another day.)

    As to visuals, find proper problems that can take advantage of them. This often involves geometrical shapes. And make the visuals clear and minimalistic rather than detailed and colorful. Research seems to support effectiveness of less cluttered and more abstract illustrations — this way they will be helpful rather than distracting. In any case, your problem was inherently not visual. Count the words in each variants, if you doubt me. In fact, you used more words in the so-called “visual” variants. (smile)

  72. I’m curious about how other people would “solve” this problem. My approach was to look up the record times for the marathon and the one mile run (124 and 3.72 minutes, respectively) and solve the proportion x/124 = 6/3.72.

  73. @Ze’ev: What is your stake in chopping down this exercise? The blog title is “Does the Medium Matter?” – that is a clear desire to see if the proposed choices of question presentation alter answers from students. I don’t remember Meyer claiming that this was a question of the highest mathematical worth or relevance. At best, he will see whether answers range based on the media used from students anywhere from sixth grade to calculus. At worst, he doesn’t make any definite conclusions about the effect of quesion wording and presentation. At least, he is getting students to engage in some thinking, probably some conversation, and the whole activity can take 10 minutes. Can you really criticize that to the extent that you have attempted? Maybe we’ll engage some students, maybe we’ll discover some avid runners, maybe we’ll get to know our students a little better. God forbid. It’s not in the curriculum.

    On a slightly unrelated note, you started out rude, then apologized, now you seem to have reverted to rude. What, exactly, upsets you so much? That Meyer’s curiosity is flawed in your eyes? At least he is doing something, making efforts to understand and improve teaching. A teacher would not be quite so critical of someone’s efforts.

    Your argument that visuals should be minimal, geometric and less colorful made me laugh. I picture a dusty, 1970’s textbook.

  74. Oh! I think I see what’s angering you, Ze’ev. The straight-text version of Dan’s problem, if it were presented on a test, would leave many test takers feeling tricked. That’s a horrible feeling, thinking, “If I knew you wanted me to estimate, I would have done that. I thought you /wanted/ a stupid answer!*”

    As someone who probably has a lot of stake in writing assessment items that cue kids to do what we want to assess, I can see how the feeling that Dan is deliberately trying to trap kids into doing the “wrong” thing would be upsetting.

    BUT! Reading your thoughts on pictures made me realize… the care that we put in to writing perfectly formed test questions with no “distractors” could be part of what contributes to kids being in this terrible state of math class being only (as cheesemonkeySF so eloquently put it) avoiding pain and seeking pleasure by trying to memorize and guess.

    When we are being incredibly careful to write questions that follow a particular set of rules, that’s what kids learn to attend to and memorize and guess based on. It’s brilliant learning on their part — if they learn the rules of the test writers, they avoid pain and get good grades. They learn: I need to ignore the diagram, I need to look for key words, I need to cross out distractors, etc. They don’t learn, I need to pay attention to the context, I need to make reasonable estimates, I need to use what I know about this kind of situation to check my thinking. Sadly, what they learn is how to “do tests” and not how to “do math.”

    So there need to be some times when we explicitly break the rules with kids. Say, “this is not your standardized-test-writing-grandaddy’s math!” In this kind of math you need to pay attention to pictures and stories, use what you know, attend to friction and air resistance and getting out of breath and stopping to tie your sneaker and… In this kind of math Dave’s method of solving a proportion is as reasonable as he can justify it to be, and so is Dan’s method of saying, “definitely more than 26*6,” and questions like, “why do we care?” are fundamental questions, not distractions.

    It would be great news if a picture could do that. I would be sad, but not totally surprised, if kids have learned (from taking too many well-written tests of the very kind that you are arguing for, Ze’ev) that pictures have nothing to do with math. And so they try to extract Dan’s text-based question from the picture.

    2nd BUT! I also think that when we are assessing kids on standardized tests (something which the National Research Council has finally set is not an effective driver of school reform; maybe this is one part of why) we DO have to be really careful not to write mean trick questions. The text-based question Dan posed would be an evil one to put on a test, and I hope no one here is arguing for its validity!

    It’s just that we do need to learn to ask different kinds of questions off of tests, and we need to do experiments just like this one to learn what helps kids to cue in that this is a different kind of question. It’s kind of like how we had to learn to write good test questions by studying how kids answered them, we have to learn how to write good THIS IS NOT A TEST questions by studying how kids answer them.

    Executive summary: neither question Dan asked is the ideal one for assessing estimation and “real-world math” skills on a standardized test. They aren’t designed by the rules of that game. They shouldn’t be judged on how well they do that. Instead, they are a test to see if kids can get out of standardized test question mode when the question explicitly breaks the rules of that “language game” and how far the question has to go to break those rules before kids switch thinking caps.

    *Though estimation is something we haven’t done such a great job teaching kids to do thoughtfully. Many kids, when told to estimate, do the problem and then change trailing digits to 0s

  75. Ze'ev Wurman

    June 16, 2011 - 5:22 pm -

    @87 ML Dan:

    “now you seem to have reverted to rude.”

    ?

    “Your argument that visuals should be minimal, geometric and less colorful made me laugh. I picture a dusty, 1970′s textbook.”

    Each one to his/her own imagination, but you don’t need to go so far back. The UCSMP translation of Kodaira’s excellent Japanese program from the 1980s, or of the “Primary Mathematics” Singapore books from the 1990s would do. Incidentally, both led their respective countries to the top of TIMSS.

    You may also consider recommendation #3 in this IES Practice Guide. Some excerpts:

    To enhance learning, teachers should choose pictures, graphs, or other visual representations carefully. The visual representations need to be relevant to the processes or concepts that are being taught. For instance, a picture of a high school football player whose football helmet has been scarred by lightning is interesting, but it may well detract from learning about how lightning works.

    Graphics do not have to be completely realistic to be useful. Sometimes a more abstract or schematic picture will best illustrate a key idea, whereas a more photorealistic graphic may actually distract the learner with details that are irrelevant to the main point. For example, students may learn better about the two loops of the human circulatory system (heart to body and heart to lungs) from a more abstract visualization of the heart chambers than from a realistic illustration of the heart. Animations may sometimes add interest value, but a well-chosen sequence of still pictures is often as, or more, effective in enhancing learning.

  76. I agree that part of the reason students are answering 26×6 is because they think that is what is expected for an answer. If their life depended on it, they would give a different answer. Perhaps different presentations, such as a time lapsed video of Benson running in this case, will effectively prompt students that we are truly looking for an answer that makes sense in terms of the real world context.

    However, I don’t think that the lack of sensible thinking is unique to application problems. How many times have you received answers that made no sense in the context of the mathematics involved — like an algebraic solution not making any sense in terms of the corresponding graph — or a positive value when the answer must clearly be negative? If you ask the student whether the answer makes sense, many times they can tell you that it doesn’t. Why didn’t they ask themselves if the answer made sense in the first place? Do students feel that they are expected to reflect thoughtfully on their answer and the mathematics involved? Or do the feel they are expected to write down some equations to show work and put a box around an answer? How do we prompt students that we are really looking for sensible answers to all questions?

  77. @luke
    Yours was my favorite comment so far in all of this mess. I teach 6th and 7th graders and am FOREVER asking them to stop and consider if their answer makes any sense! I cannot seem to get this across to them. Is this the stage in math when they first have to actually think about whether or not their answer makes sense? Regardless of the outcome of this study, it has made me eager to push my students harder to reason. I want them to think about the math and not just accept things bc it is in a book or they want to give the “right” answer to a teacher. Maybe I will start regularly posing questions of hilarity for them like the “how many fingers would he need to lift,…” Maybe that is how I can train them to think “beyond the calculations”.

  78. I think that perhaps one fundamental flaw in Mathematics education that is clearly exposed by this “experiment” is that students are not exposed to the possibilities of solutions that occur in such “real” mathematics. Speaking of my own students, they enter my class believing that there can be only one “right” answer AND that the answer is the ONLY thing that is important and not necessarily the solution or process of developing a solution.

    They fail to connect the math that they do with something in the real world that is being modeled by that math and therefore never consider the unreasonableness of there solution, though computationally correct.

    At my local university, I have heard that one of the early grade math workshops that they conducted a local study of the effect of the current California State Mathematics Standards for K-2 and found that the focus is teaching students to rely too much on formulas and algorithms and not enough developing the fundamental concepts that support those formulas and algorithms. As a result many more students are leaving elementary school without knowing their multiplication facts or even recognizing that 2×3 can be represented also by 2+2+2 or 3+3 or any other way. Such important connections are not being made by students, locally; perhaps elsewhere.

    So, by not having the experience of connecting the math that a student dose with modeling something tangible, how can we expect a student to think outside the box? It doesn’t fit in their formulas and algorithms, which was definitely evident in the data that I provided Dan from my classes, then they will never see a different solution or recognize that their solution is unreasonable or doesn’t make sense, and therefore they conclude that math is hard or confusing. In reality, math is much more than just “number crunching!”

  79. @Michael

    Noting that you recognize this issue, would you also be able to offer what have you done to evolve this thinking in your students.

    It would be helpful to me if you can offer insight on how you go about addressing this situation in your classroom how often etc. I will assume some kids get it quicker than others, are you able to bring it back on an individual level until this child comprehends the concept fully.

    Have you been able to measure the results and see if it was retained once they entered the next grade level and have their scores gone up?

    Thanks

    @Julie R regarding how saying to a 6th grader stop and think if it makes sense!,

    IMO I think its important to help them developed the way to think about how to go about deciding if something makes sense or not.

    To an adult, its easy to point to the fact that a 6 min mile was the fastest run by Benson and we can naturally assume that for him to maintain 26 in a row is not going to happen at the same speed.

    It may seem silly but some kids may not realize this fact, additionally I have found that it also depends on how you communicate, the !! pts leave me to think you are expressing frustration in the classroom (hard not to sometimes), kids who don’t get it will just keep quiet because they feel dumb and don’t want to look stupid in front of others.

    Possible solution: Consider having the kids in the class run around the track/school (a measured distance that has them reach anaerobic threshold or max effort) 4x without stopping for recovery and time each lap.

    They will feel the effect on their body’s and see the time for each. Ask them if they ran 22 more times what they think would happen.

    You can also collaborate with the gym teacher for this activity and connect the dots later or next day in the classroom.

  80. @GregM
    “Possible solution: Consider having the kids in the class run around the track/school (a measured distance that has them reach anaerobic threshold or max effort) 4x without stopping for recovery and time each lap. They will feel the effect on their body’s and see the time for each. Ask them if they ran 22 more times what they think would happen. You can also collaborate with the gym teacher for this activity and connect the dots later or next day in the classroom.” I love this and could easily make it work at my school – plus, my classroom faces the soccer field so it’s very convenient.

    As far as the ! goes, I am the crazy, !!, energetic type so !’s appear often in my life and classroom. I would not be frustrated with the 26 miles at a 6mph pace answer as it does require deeper though, but AM often frustrated when I get (5)(0.5) = 25 and such. But, I am frustrated with myself, not my students because they know what I have taught them. It is those times that I think, how am I teaching wrong and how can I fix it? For them to not think, “Is my answer reasonable?” means that I have not deeply instilled in my students the correct way to think about math. What I would like to know is how as a teacher do I do this with 6th grade students? Because just telling them to do it (over and over and over) obviously isn’t working.

  81. @Julie R

    You and everybody else need to be commended for your commitment and willingness to share. I think this is what these forums should also include, if something is not working, what does?

    How to make them care about trying to figure this out as opposed to just being checked out.

  82. One of the things we want to discover by asking this is which question students will answer: “What answer do you think your math teacher wants you to give?” or “What do you really think?”

    But if they choose to answer the second question and say it takes 26*6 minutes will we assume they answered the first question? How will we know that they really don’t understand the context of the problem?

    Our goal should be (and I know Dan’s is) to work toward the day when students don’t expect their teachers to ask questions that are answered with a single operation.

  83. Looking at this question (the fourth version) with some pre-geometry students today, one student asked me “do you want the ‘math’ answer or the real answer?”

    I think that’s fairly telling of how students have been trained to be wary of context in math class.