It all depends on what you want from the question. Question 1 is actually better than the other 2 at determining if a student can determine the relative size of numbers (although, with any multiple choice question they have a 25% chance of guessing the correct answer, so a far better question for this purpose would be to have them write it ala #2). If a student gets #3 wrong you would need a whole set of other diagnostics to determine where they went wrong… do they think 02583 is a number? Did they give an even number? did they not know that there was a smaller 5 digit # to be found? Who knows? On the other hand, # 2 and # 3 are better at many other things. So as with any problems, the question has to be a response to “What do you want to know and why?” Is #3 far more interesting, will a student learn more by doing it? I think so, but 3 questions out of context are pretty hard to evaluate.

I think the MA test has better designed questions in general, but comparing questions on one topic from each test smacks of politicization (don’t identify the states if you simply want to compare the questions). We in the math community especially should appreciate the need for thoughtful and objective analysis and resist stooping to this level.

A test for the entire population should check for understanding at both a high level and a relatively low level. It make sense to compare high level question on each test (if they exist) and to compare low level questions on each test. Hear we are comparing what was probably an easier question on the CA test with what was probably not one of the easier questions on the MA test.

The follow up question in the MA test seems out of place and clumsily worded. You just asked a kid to do a task that clearly indicates that they understand place value. If you want to check how well a kid can articulate their understanding of a topic, make it a separate question were it is very clear what you want them to explain. Understanding a topic and being able to articulate that understanding are not at all the same thing.

I like the Singapore question better because it is more concise and requires students to think about two different ideas (which is often the case with their qustions).

Great comparison.
I’m nit-picking at the Singapore question, but if they are going to bold and underline “odd” they should also bold and underline “5-digit” as it can equally trip kids up.

Ah, but CA is also trying to test if kids read the instructions correctly, “… from greatest to least.” I don’t know about 3rd grade math textbooks, but my 6th grade text has all [that I could find] questions asking “from least to greatest.” Tricky.

Thanks for posting the other questions. I don’t think the source demonstrates much higher order thinking.

My comment about needing both stems in part from thinking about the interview with john sweller you linked to earlier. He claimed that no one had been able to show that any curriculum improved problem solving skills, so we should just teach basic schema.

My thought is that we hardly ever test for improvement in problem solving or higher order thinking. In order to do it, we would need to test both basic skills and complex ones, come up with a predicted correlation, and then see if it changes after trying to teach problem solving.

I think tests usually focus on one or the other, but I do believe the sweller is correct in saying it is hard to solve complex problems without the basic skills. Hence, we should test both and see where the shortcomings/improvements are.

If the assessment items are something to learn from, the textbooks from Singapore are even better.

http://www.singaporemath.com/

Especially at the K-8 level (up to algebra and geometry), these are valuable resources.

re:Ebaer

You made me think… If we’re going to be doing assessment, and we’re going to be doing it on computers, then why not ask problem #3. If they get it wrong, then give them other problems to narrow down which specific skills they’re having trouble with (or if it was just carelessness).

It would be pretty cool, but what a brutal test to take — each time you struggle with a problem, you’ll be faced with several related problems.

Seems to me that there’s no compelling argument in favour of multiple-choice questions over short-answer ones in testing any aspect of math, except for *very* basic procedural skills.

Dave (12), no doubt Singapore has something to learn from, but here’s an interesting diagnosis of that singaporemath.com site you mention, http://math.berkeley.edu/~giventh/diagnosis.pdf

Roy, that’s kind of the point. It’s not possible to form 10001 with the given digits.

Keep in mind, folks, that the dropout rate in high school in some California public school systems is 60%.

I agree with Ebear. What do we want from each question? What standards or objectives or goals or whatever are these questions tied to? Are they all the same?

Also, at what level are all three full tests designed to give valid scores or ratings? (individuals, classroom, school, state?) Are the tests meant to be formative or summative? I would think this makes a difference if you expect to get diagnostic information for individual students.

Finally, I’m curious how the MA and HK items are scored. I’ve scored open response items for KY (almost 20 years ago), and it was an interesting, although unintentional professional development experience. It was the beginning of my obsession with looking at student work. I’ve also seen open response items on local CA tests for which students could not get the top score unless they used a method specified in the corresponding standard. The choice of how to reliably score these items is a big deal.

Jason brings up a good point about the released test questions from CA possibly being cherry picked, but I think the document itself demonstrates that the questions were chosen because they were indicative of the way the standards would be assessed.

“In selecting test questions for release, three criteria are used: (1) the questions adequately cover a selection of the academic content standards assessed on the Grade 3 Mathematics Test; (2) the questions demonstrate a range of difficulty; and (3) the questions present a variety of ways standards can be assessed.”

Fawn also points out the “tricky” nature of the instructions. I’d also add that Mass. and Hong Kong use the word “smallest” as opposed to CA’s use of “least” and “greatest.” I wonder what the consequences of word choice may be.

I find the language in the standards themselves quite interesting in that Mass. has placed an emphasis on multiple representations as well as understanding yet CA simply wants students to perform a specific task.
Massachusetts:

3.N.1 Exhibit an understanding of the base ten number system by reading, modeling, writing, and interpreting whole numbers to at least 100,000; demonstrating an understanding of the values of the digits; and comparing and ordering the numbers.

California:

3.NS1.2 Compare and order whole numbers to 10,000.

Keep in mind that CA Standard 3.NS1.2 isn’t considered a “key standard” in that it is a support standard for the larger 3.NS 1: Students understand the place value of whole numbers. However, if you look through the entire document, there aren’t many (if any) questions assessing a thorough understanding of the content. And in my opinion, this has had a detrimental effect on how math is actually being taught in CA.

@David:

I find the language in the standards themselves quite interesting in that Mass. has placed an emphasis on multiple representations as well as understanding yet CA simply wants students to perform a specific task.

I read from one essay (I don’t have the link offhand, alas, but I’m pretty sure it was Wurman from the essay Dan linked to) the fact that the CA standard is easy to read and the Mass. standard (and the Common Core standard) is hard to read makes the latter evil.

@David: Yes, I think the idea that “easy to read” somehow equals good is ridiculous in this context.