## [PS] Assessment

April 2nd, 2011 by Dan Meyer

It's bad enough when you're trying to *gin up interest* in math by way of pseudocontext. It's worse when you're trying to *assess* math by way of pseudocontext. If the student isn't interested in math *by now*, what do you think an assessment is going to do?

If your students miss these problems, how certain are you they really misunderstood the mathematics? How certain are you they weren't distracted by the problem design?

on 02 Apr 2011 at 6:51 am1 RachaelThough it might not test ‘math’, it does test a student’s ability to pick out and discern a pattern. This is a key skill students need to achieve in any of the sciences, the humanities and life in general. I often think that these kinds of skills need to be taught in a variety of courses, in the way those skills apply in that area.

anyhoo, I agree that pseudocontext is the downfall of keeping a child’s attention. but I think its too easy to mix up ‘pseudocontext’ with ‘not obvious skills’.

on 02 Apr 2011 at 8:42 am2 AndyI agree the shirts are unnecessary and silly, but is it really that distracting? This example seems pretty harmless. If that causes students to get it wrong, then maybe they really aren’t ready to advance.

on 02 Apr 2011 at 8:44 am3 LindseyThis one doesn’t bother me like most of the others (pseudocontexts) because this is something that many students could encounter in real life (jerseys that have numbers on them). To add to what Rachael says about looking for patterns, this problem does resemble someone looking for patterns in a realistic situation. I could see someone actually looking at sports jerseys and looking for patterns in the numbers. Why not encourage that from our students? Why not suggest to them that they could look for patterns in things that don’t seem inherently mathematical?

on 02 Apr 2011 at 4:05 pm4 MyleneHm. I agree that there is a perfectly good reason to look for patterns in numbers everywhere: because it’s fun. But my first thought when I saw this item was “what is the meaning of numbers when they are on jerseys?” That made me think of how each sport has different numbers the are conventionally not used; famous players known by their numbers; retired numbers; Greztky choosing 99 as an homage to Gordie Howe; etc. All of which has caused me to waste test-taking time. If the skill being assessed is “pick out patterns despite distractions,” then the item is working. If the skill being assessed is “pick out patterns”, then the item is biased against people who find meaning in jersey numbers. If the goal of the item is “help students realize that picking out patterns can be done anywhere just because it’s fun”, well, putting the numbers on jerseys does not inherently make this fun for someone who does not already think so (and those who do already think so don’t need the jerseys).

Maybe this item would have a different meaning if it weren’t on a test. “Think of a non-math-class situation where you see numbers all the time. Could you find patterns in those numbers?” might make for an interesting conversation, though I’m not sure it would be the best tool for any particular job.

on 02 Apr 2011 at 4:49 pm5 serafinathe wording of this question is terrible as well!!! “which of the following do all of these shirts have in common?” does that sound awkward or what?

on 02 Apr 2011 at 4:50 pm6 Chris SearsI teach ACT prep classes for high school students. On the practice test there are a few silly word problems on the test. My favorite is from a series of problems about a swimming pool. The problem starts with “The directions say to install the ladder at a 75 degree angle from the ground.” I tell the students that if they find directions that say that, they should send it to me.

on 02 Apr 2011 at 5:04 pm7 NathanFor me, the points is “why even have the shirts in the first place?”

Do they add anything to the question? If yes, it’s not obvious to me. If no, then why are they even there?

I don’t see how having just the numbers is worse. Furthermore, as Mylene has pointed out (and I think what Dan is implying) this question hurts people who find real-life meaning in jersey numbers, as it has nothing to do with patterns that would naturally arise from those numbers.

on 02 Apr 2011 at 7:10 pm8 Matt McCreaMy bigger worry is that its distracting from the initial point of the assessment question. Take away the context; what is the question asking? Of course, it wants you to discern some sort of number pattern or find what property the numbers share, but it is entirely possible that the student could walk away from the problem being too distracted by the jerseys themselves to be able to solve the problem. Would a wrong answer really tell us that the student didn’t know the skill? Or does the problem confound it to the point that we can’t tell if it was the distraction or the patterns that tripped them up?

I’m all for putting some complexity into a problem and making the kids grapple with it, don’t get me wrong. I also happen to be a person that looks at just about anything and tries to find a pattern in it, but as Mylene says, suddenly being forced to look at a problem through this lens isn’t going to open anyone’s eyes to the beauty of math, it’s just going to make those on the fence and past it see another example of forcing math into a situation that they don’t see the utility in.

on 02 Apr 2011 at 8:57 pm9 Bowen KerinsThis is a horrible, horrible test problem, context or not. Others have commented well about how important it is for assessment items to actually be about assessing something.

Good assessment items work because the problem, choices, and reasoning are clear. A colleague who does this much better than I can said that in good assessment items “students should get the right answer for the right reason, and the wrong answer for the right reason”. (This is one reason why I have stopped using “none of the above” as a correct answer to a multiple-choice question, since students get it right if they mess up for any reason besides the ones you’ve thought of.)

But this assessment item has a horrible potential: it is possible, and relatively simple, for a student to miss the “12″ then answer choice B based on what information they think they have. If there were no shirts and just a list of five numbers, this would not be possible.

So the shirts waste time for all students, and potentially cause some skilled students to get the wrong answer. This needs to go, along with the far too many assessment items like this — and by “like this” I mean problems with ways for skilled students to get wrong answers for non-mathematical reasons. We can deal with removing the ones with idiotic contexts later…

on 03 Apr 2011 at 5:28 am10 Jan van HulzenThis sounds like the “numbers” racket, one of my favorite Calvin and Hobbes comics. Although Calvin gets distracted quite a lot i think it is spot on

http://www.gocomics.com/calvinandhobbes/1990/02/09/

http://www.gocomics.com/calvinandhobbes/1990/02/10

jan

on 03 Apr 2011 at 5:30 am11 Jan van HulzenThis sounds like the “numbers” racket, one of my favorite Calvin and Hobbes comics. Although Calvin gets distracted quite a lot i think it is spot on

http://www.gocomics.com/calvinandhobbes/1990/02/08/

http://www.gocomics.com/calvinandhobbes/1990/02/09/

http://www.gocomics.com/calvinandhobbes/1990/02/10

jan (edit: first comic included)

on 03 Apr 2011 at 6:13 am12 NoraWhoa! This is a horrible problem/questions. It’s horrible for all the reasons listed above, but it’s multiple choice too. How often in life are we presented with a multiple choice question? Usually we have to create the choices in which to pick.

Perhaps the question would be better if it asked for a common trait in all the numbers and lose the shirts. That’s just plain dumb.

on 03 Apr 2011 at 1:02 pm13 Peter“Mister, they’re all on the same team?”

As Nora said, just leave it open and let students respond; some might have said the quote I shared.

Why do jersey numbers need anything in common? If you play a sport you know that except for football, the numbers have no rhyme or reason. There’s no relevance to jersey numbers having something in common.

There are many other possibilities of similarity these numbers have beyond the choices given, e.g., the sum of their digits are less than 9.

Maybe a more relevant question would be, “My soccer team needs to have 5 jersey numbers that are multiples of 6, and one of them has to be greater than 84.”

on 03 Apr 2011 at 4:02 pm14 Bowen KerinsI think most, if not all of us, agree that open-response questions are better for truly assessing students’ understanding. The reality of testing, at least in the US, is that high-stakes multiple-choice tests are very common: SAT, ACT, AP, state tests, game shows. If we never give students multiple-choice questions they will be less prepared for these tests — right or wrong, that’s the landscape.

As a multiple-choice question, this problem actually has some pretty good distractors (the buzzword for “wrong answers”). It’s the shirts and the problem design that make it a mess. It’s a Where’s Waldo search buried in the middle of a math test.

on 03 Apr 2011 at 4:27 pm15 Emily KissnerI think it’s not just the one question that’s a problem for students. It’s the cumulative effect of 10 pages with these kinds of problems! Each problem has a new context. It’s a lot of cognitive work for kids who might already be struggling with reading or attention difficulties. As my fourth graders worked on state test sample items, they actually tried to carry information from one problem to the next. Whew! That made for some strange situations.

on 03 Apr 2011 at 6:55 pm16 Matt McCreaPeter,

Why would your soccer team need to have numbers that are all multiples of 6? Moreover, why would one need to be greater than 84?

Jersey number have no business here, multiple choice or not. Number patterns are a beautiful mathematical topic on their own, and if they are to be assessed, then it’s best to just leave them on their own instead of trying to force a context. Unless there’s something natural about the number patterns this question is assessing, then just assess the concept on its own.

The worst part about all of this is that I’ve never seen a student that doesn’t jump on number patterns on their own. Consistently, in all populations I’ve looked at, number patterns tend to be a relatively high scoring topic compared to everything else, especially in the lower grades where you’d see this type of problem. If you’re looking for a way to teach this, just write the numbers on the board and challenge the kids to find something that they have in common.

on 03 Apr 2011 at 7:06 pm17 Peter PriceSpot on again, Dan!

I agree with you on your first point: not only is this an example of a lousy problem, it is written for a time when students’ performances are most critically required: when they are being assessed. Of all the times, this is one where unnecessary distractions need to be removed.

If identifying the mathematics involved in an everyday context is what is being tested, then let’s test it. But as others have said here, if the intention is to find out if students understand number classifications and mathematical vocabulary, then ditch the shirts and the pseudocontext.

I find Emily’s comment particularly pertinent: Year 4 students are presumably not yet familiar enough with this sort of nonsense to recognize that school math questions aren’t supposed to mean anything, or to connect to other questions, they exist in a world of their own. Trying to make them mean more than they do will only lead to frustration.

on 03 Apr 2011 at 8:02 pm18 RichardMarcus du Sautoy’s football (soccer) team has prime number jerseys only, and for good reason.

The context here is derinitly unnecessary and offputting and may introduce bias into the resposnes, depending on the students’ particular feelings about sport. Some anxiety may be induced by the clearly nonesensical selection of numbers – what sport has this sort of number?

The original comment I have to agree with – we should assume and encourage a fascination with numbers in and for themselves, not dress them up in odd contexts in order to moitivate.

on 04 Apr 2011 at 6:12 am19 ShariThe skill being assessed by the problem is whether or not students understand terms like odd, factor, and multiple. I don’t see it being about patterns at all. It’s more about describing a set of numbers.

Is it not possible for a team to have jerseys with numbers that happen to be able to be described as a factor or multiple of another number? When I look at winning lottery numbers, I look for these types of relationships all of the time. No, it doesn’t always happen, but it is interesting to me to see if there is a relationship at all.

Honestly, I don’t see the jerseys as distracting. While they may not be the best context, they make the question look less threatening while allowing the numbers to be larger and to be separated by more space. This gives students the opportunity to consider each number separately, rather than gloss over them in a list. If they were just shown in a list, how should they be shown–from greatest to least, least to greatest, random order? If random, wouldn’t that be distracting?

I’m curious, how would you assess student understanding of factor and multiple in a real context in a multiple-choice format? (This question was taken from a state test and, unless you want to see money available for schools drop even more, state tests will have to stay in multiple-choice format.)

on 04 Apr 2011 at 7:56 am20 Bowen Kerins@Shari Some decent context questions to assess factors and multiples…

Which of these measurements is equal to an integer number of feet: 40 inches, 50 inches, 60 inches, 70 inches?

You gave 90 raffle tickets to some friends. Each friend received the same number of tickets, with none left over. Which of these could be the number of tickets each friend received: 12, 24, 30, 60?

I’m sure others could do better. The first requires kids to additionally know there are 12 inches in a foot, and the second is maybe pseudocontext-ish, but they assess the right skills.

I actually like the shirt problem’s question and choices, but the potential for student error and confusion needs to go. If the shirts weren’t there, and it was just the numbers, it would be fine.

on 04 Apr 2011 at 11:20 am21 Matt McCrea@Bowen The raffle ticket problem could be made a bit better by switching up the order of the information. Put yourself in the ticket distributor’s shoes; are you really going to give them out, know that they each got the same number, know that you didn’t have any left over, and know you started out with 90 tickets but not know how many you gave to each person? Say this instead -

“You have 90 tickets to give out to your friends. You want each friend to receive the same amount of tickets, and you don’t want to have any left over. Which could be the number of tickets you give each friend? 12, 24, 30, 60″

I know it’s subtle, but the information and setting is half the battle in setting up problems.

on 04 Apr 2011 at 11:42 am22 Shari@Bowen and Matt — I’m still having problems with the ticket problems. Wouldn’t you know the number of friends? So you’d just divide 90 by the number of friends.

The inch problem assesses the ability to convert measurements more than testing and understanding of factors/multiples.

Neither of the questions proposed uses the term “factor” or “multiple”. I think it is understanding these terms (along with even/odd, greater than/less than) that the question was trying to assess. I see the question assessing the ability to choose an accurate description of a set of numbers using mathematical terms.

on 04 Apr 2011 at 2:09 pm23 BreedeenTwo points on the design of this problem:

1) my guess on the “what were they thinking” line is that they wanted to separate the numbers from one another visually. This can actually be helpful with certain learning differences. The shirts put a contextual “box” around each number, thereby physically and visually separating it from its neighbors. Why the shirt instead of an actual box…? I have no idea.

2) a lame attempt at giving the problem a context in which shirts with numbers on them makes sense: The math club at Anywhere Middle School made shirts for each team member. If these are the shirts pictured below, what is the name of the team? a. “The Odd Numbers”, b. “Greater Than 20″, and so on…

It’s still completely lame, but the shirts now make sense in terms of the question being asked. It’s not a retribution though, as there really is no reason why this problem needs to have shirts in the first place. This is why test and textbook designers should not have unrestricted access to clip art.

on 04 Apr 2011 at 3:51 pm24 Matt McCrea@Shari You’re right, you likely would have that piece of information, although I suppose you could argue that maybe you give it to some friends but not others. Either way, I’m not satisfied with the problem anymore, so thanks for pointing that out.

In all honesty, are you able to find a genuine context for “factors?” In my thoughts, I’ve only been able to come up using factors in the context of making division easier to compute and prime factors, but of which are abstract concepts. In that case, why bother with a context? Just test the concept on its own, and don’t kid the kids out of the chance to realize that sometimes, math for math’s sake isn’t a bad thing.

on 04 Apr 2011 at 6:43 pm25 Dan MeyerFWIW I’m happy to find an example of pseudocontext that divides our group somewhat. Divisive problems cut to the heart of the matter better than the problems which we unanimously jeer at.

To those who have submitted some variation on the theme of, “C’mon, it’s isn’t that bad” or “

I’mnot distracted by the uselessness of the jerseys,” I say, yeah, I’m in total agreement, except it isn’t about us. Our tolerance is ever so much higher than theirs. And nothing is neutral. If it doesn’t add to the problem, it subtracts.I’d like to highlight the line

Matt McCreahas been repeating throughout this thread: that the jerseys are an implicit admission thatnumbers are not fun or interesting on their own.and later:

That you can’t enjoy math for its own sake is one of the two sad, false signals pseudocontext sends our students.

on 05 Apr 2011 at 7:51 pm26 Amy E.The next question: what is the question trying to assess? If it’s some kind of naive set theory (“some,” “all,” “none,” “not none”) etc., I agree that there are tons more interesting kinds of questions, even in multiple choice form.

on 06 Apr 2011 at 4:53 am27 JonI think the real answer is that the shirts are translations of a kind. Their congruence is the key to the answer. Maybe it’s the translation along a sine curve! The problem would have been better had the numbers been placed in braces and students asked to identify the large set of numbers for the illustrated subset. Boxed assessments are always good ?!?