Get Posts by E-mail

The Sealed Envelope

We all suffer from confirmation bias and earlier I wondered how the hardcore edutechnophiles would rationalize findings that the US fourth graders who used the most classroom tech performed the worst on standardized math exams?

My best attempt to crawl inside their heads:

Standardized exams are inaccurate assessments, moreover, they assess irrelevant skills.

Do I have that right?

Standardized Exams Are Lousy Assessments

I can't spend much time on this assertion. Exam validity varies from state to state, content area to content area, and I can't speak for the fourth grade assessment, but if California's Algebra and Geometry assessments are any indicator, the ground just fell out beneath you. They're extremely challenging and extremely fair assessments of prescribed coursework.

Standardized Exams Assess Irrelevant Skills

I can't cozy up to that first objection but I spend a lot of free time wondering if we've prescribed the wrong coursework, especially after the concluding paragraph from Roger Schank's recent post on math education at The Pulse:

I know this is a hopeless fight, but algebra really matters not at all in real life and the country will not fall behind in any way if we simply stop teaching it. That is not a fact, it is just a former math major’s, UT graduate’s, and Computer Science professor’s, point of view.

There is truth to this, I'm positive. There are studies to be conducted and evidence to be found concluding that, along with the current buffet line of alternative education options, we'll move to an a la carte mathematics curriculum, even while textbook manufacturers, math teachers, their unions, and parents fearful of the global boogeymen dig in their heels and pull.

Careful, Though

Depending on the strength of your biases, that last one is an ugly rationalization, one which says the only tasks which matter are the ones which involve technology.

It's worse than, "When you have a hammer, everything looks like a nail." It's, "When a hammer is your favorite tool, the tasks which don't involve nails are pointless."

There is a very rich, very nuanced conversation to be had here about the math students need (one which I am trying, and failing, to resolve internally with my foundering Feltron Project) and then there is a separate conversation, one which I will find very boring.

The Rationale Which Few Will Cop To

It will impress me a great deal if one of the usual tech proponents steps up to suggest that the implementation of technology in a majority of these classrooms simply sucked, that tech use in math classrooms simply isn't "there" yet. That'd be totally reasonable and somewhat courageous given the forum.

Courage Aside

It's obvious to me that this discussion will go nowhere — idealistic tech coordinators and their traditional colleagues permanently gridlocked — if we don't first resolve the question, "Just what exactly are we supposed to teach here?" and then select some best practices, everyone agreeing to be cool in advance if those practices involve wikis or no. 2 pencils, 'cause we're sure to find both1.

But these emotional appeals to decency and child welfare, my criticisms of which have clogged this blog for the last time for at least a week, treat education's sore throat with a colonoscopy, demonizing a whole lot of decent educators without much result.

  1. I reckon I'd quote some passages from Understanding by Design here if I'd ever read it.

16 Responses to “The Sealed Envelope”

  1. on 27 Mar 2008 at 10:19 pmTMAO

    Gotta tell ya, I dug Understanding By Design. Not a bad bathroom book.

  2. on 27 Mar 2008 at 10:34 pmdan

    *sigh* It’s on a long list.

  3. on 28 Mar 2008 at 2:14 amrdsc

    Don’t know if I’m a tech proponent but I totally agree with “the implementation sucked” theory – and it’s only going to get worse as PC prices drop. It’s the easiest thing in the world to fill a class up with hardware (“hmmm – £200 umpc – we’ll have a hundred”).

    Ok – it’s running Linux – and we’re a Windows shop – it’s running Firefox and the student portal doesn’t work with Firefox – we’re running MS Office and most of our teachers can’t get there heads round converting between different MS file versions – lets throw OpenOffice at them.

    So the senior manager who suggested this drove the programme manager who was going to have to implement it rushing to me in a flat cold sweat as she could see her teaching programme descending into an endless, pointless struggle with equipment that no-one had even *begun* to think about how they could actually integrate into her teaching. That’s routine.

  4. on 28 Mar 2008 at 4:58 amTom

    I’ll second “the implementation sucked” theory with a side of the teaching in general sucked.

    As I’ve wander from state to state (NY, VA, AL so far) and district to district I’ve become more and more depressed by what’s going on in way too many classrooms*. Granted my sample is not huge but if it scales at all then many kids are learning in spite of what’s happening to them in school.

    I see way too many petty tyrants, way too many teachers who dislike children, and way too many teachers who could make juggling high explosives boring. The worst thing is that so many are quite comfortable wherever they are and have desire to improve.

    I’m not saying it’s the majority, but the teachers who suck really do have an unpleasant ripple effect on everyone- students and teachers.

    I’ll also say that my own teaching has shamed me at times. I’ve failed miserably and done things that really depress me now. I’ve had carefully crafted lessons I spent weeks on utterly fail. The only difference between myself and the teachers I label as sucking is that I looked at the mess I made and tried to figure out some way to improve it.

    So, at least in my own mind, I don’t suck.

    I’ll mail you the check for therapy just let me know the address.

    *let’s say around 10% although your mileage my vary

  5. on 28 Mar 2008 at 4:58 amJenny

    I’ll second the support of the sucky implementation theory. I work really hard to use technology well in my classroom, when I use it. And I know that sometimes it sucks.

    I’d also be curious to know how they defined technology use. And who answered that question. Were teachers asked? Did they feel pressure towards a certain answer? I have too many questions about it.

    Finally, as to the tests. I have to say that our state math tests, at least for fourth and fifth grade, aren’t awful, but certainly don’t test as well as I’d like. My 5th graders did a ton of work with Geometer’s Sketchpad and ended up with much deeper understandings of concepts in our study of geometry. I think it might make some of the multiple choice questions more confusing to them because they can justify ‘wrong’ answers as right. (I think they are still smart enough to figure out which answer is the ‘best’ one, but it certainly will throw them a bit.)

    Slightly off topic, I’d be curious to know more about your thoughts (all of you high school math teachers) about what/how we currently teach math. Are we pushing students into algebra too soon? My students are on track to take it in seventh or eighth grade. Should we be building a better understanding of algebra in elementary school? Are we trying for a curriculum that is too broad and not deep enough? As an elementary school teacher I feel like math is an area we don’t do well. We (elementary teachers) have a lot to learn.

  6. on 28 Mar 2008 at 5:30 amRoger

    algebra really matters not at all in real life and the country will not fall behind in any way if we simply stop teaching it.

    What algebra teaches is not a set of formulas to memorize or obscure techniques to solve contrived situations (“A train leaves Buffalo at 6pm…”). Rather, what a student should take away from an algebra course is a methodology — a process to look at real-world problems and translate them into a more universal grammar that facilitates solving.

  7. on 28 Mar 2008 at 7:45 amDina

    Not to poop in the punch, but I’m wondering why we’re even entertaining this conversation, based on such crappy data.

    Seriously. If you’ve got a dozen well-trained educators on this blog asking double the amount of critical questions about this cute little NAEP Excel doppleganger, most of which questions completely debunk not nuanced “implications,” but the very parameters of one’s x (math curriculum being assessed) and y (use of technology) values– I mean, come on. The *X and Y* values?!? Doesn’t that indicate on its face that we should throw out the whole damn graph?

    Let’s do that, shall we? Then we can talk about Dan’s super question about relevant curriculum, which, by the way, springs like Aphrodite from the foam of Dan’s brain, and ironically is not supported– or unsupported– in the slightest by the data he posted.

  8. on 28 Mar 2008 at 7:52 amDina

    Sorry, that was supposed to be “not supported” and “supported.” Is that my purse over there? I’ll just retrieve it now. Thanks.

  9. on 28 Mar 2008 at 8:21 amChris Lehmann

    I’m going to pull my own experience in here to call out the “implementation sucked” theory.

    My experience in a portfolio-based school which did not do much in the way of traditional test-based assessments meant that I saw kids who could write incredible essays, deconstruct complex pieces of literature and powerfully represent themselves in all four of the major NY state standards in English (Reading, Writing, Listening, Speaking.)

    However, what they rarely did was score all that high on the NY State English Language Arts test. In fact, often the scores on those tests were not at all correlative to what I saw as my students’ abilities. And this wasn’t just true in my class, but in classes throughout our school. And I’ve worked with enough teachers and seen enough classrooms to feel qualified to say that our classrooms — while certainly capable of improvement — didn’t suck.

    In fact, progressive, project-driven schools often score lower on standardized tests despite the fact that if you look at other metrics (first year college GPA, attendance in HS, student / parent satisfaction surveys), they are often considered exemplary schools.

    Now, I know that progressive != technology, so I’m not quite arguing the same thing, but I think there *may* be an analogous situation here. Is there some bad teaching with technology? Yes… of course. Is the bad teaching in technology so much more widespread than bad teaching in non-technology infused classrooms as to explain the test scores, I am unconvinced.

    Data-driven decision-making is at its most dangerous when we look at incomplete or poor data to make big decisions.

  10. on 28 Mar 2008 at 9:24 amsandy

    I thought that everyone else would say this, so I didn’t post — but now I see that no one else saw it this way so must stick my toe in the water.

    For a long time now, there has been grant money available to low-end Title One schools to buy computer labs for individualized practice of basic skills in reading and math.

    Ergo, the schools who score lower and got these grants will be on the higher end of math “technology use” because the kids do practice sessions in the computer lab — individualized practice targeted where each kid needs it, which you can call drill-and-kill but math skills do need to be practiced!

    The tech-use-in-math didn’t *make* them score lower. They are using the stuff *because* they scored lower. Couldn’t this explain it?

  11. on 28 Mar 2008 at 10:03 amdan

    Sandy, my first school caught one of those grants for just that reason. Forgot all about it until now. Thanks.

    As I recall, my precal students and I used that lab to develop regression models for predicting (eg.) the number of Starbucks worldwide in 2050.

    The remedial classes used it for completing pre-programmed math drills.

    I’m not sure what to make of that.

  12. on 28 Mar 2008 at 10:33 amJeff Wasserman

    What algebra teaches is not a set of formulas to memorize or obscure techniques to solve contrived situations (”A train leaves Buffalo at 6pm…”). Rather, what a student should take away from an algebra course is a methodology — a process to look at real-world problems and translate them into a more universal grammar that facilitates solving.

    Just had a very similar conversation with a kid in our at-risk-kids-need-very-special-support-study-hall room, who is missing a ton of math homework with two days remaining in the third quarter. We were talking about how math class isn’t about learning the formula, or solving for x, or whatever you people do, but about training your mind to solve problems and focus on the details. I told him that 13 years out of high school, I couldn’t tell him where to begin on solving an algebra II problem–I haven’t looked at that stuff since ’93–but I do credit my math teachers with giving me a way to practice slowing down, checking my work, and being confident in the solution to a problem.

    Then again, I’m an English teacher, so I might not know what I’m talking about here.

  13. on 28 Mar 2008 at 10:57 amdan

    Nah, that’s right on in a lot of ways. The problem with attacking Algebra right at its real-world throat — stripping it of the x’s, y’s, deltas, etc. — is that without comfortability in abstraction, kids will never come close to Calculus.

    I know, big whoop, right.

    But Calculus is a thing of beauty. Some of the most beautiful work I’ve ever done. I never understood the world as well as I did after those two semesters Sophomore year.

    So that’s (one of) the problem(s). In secondary math ed, we have all our kids on a really onerous train ride towards an awesome destination. If you disembark early, you incur a lot of loss. In fact, you’d have been better off on a train with a less spectacular destination and a really pleasant journey.

  14. on 28 Mar 2008 at 11:33 amJason Dyer

    The only math students at our school using computers “every day” are the ones using PLATO — that is, doing remedial lessons or catching up on missed credits.

    We do have a good number of computer labs, and a set of roaming laptops, but even given that it’d be logistically impossible in our standard classes to be using the computer “every day”.

    The statistical difference in the other categories is too minimal to comment.

  15. on 28 Mar 2008 at 5:30 pmA. Mercer

    “I can’t spend much time on this assertion. Exam validity varies from state to state, content area to content area, and I can’t speak for the fourth grade assessment, but if California’s Algebra and Geometry assessments are any indicator, the ground just fell out beneath you. They’re extremely challenging and extremely fair assessments of prescribed coursework.”

    In this post:
    I show how out of alignment the standards and the teaching materials are in elementary, including fourth grade, the level you cited in your prior post. This means that how students do on the state test is very dependent on how well the teacher supplements the text. HOWEVER, the assessment you are looking at in that chart is a special National test, not the state, so I think there is a little apple/orange comparison going on. The fact is that the text/materials are out of alignment with state standards and testing (which is high, a tweet I got on that post from someone on the east coast indicated she was teaching similar concepts to her 10th graders), but the students still aren’t learning enough to do well on the national standards test.

    It’s difficult to get mixed up in all the layers, so I’m not going to beat you up any more about this. I could find myself in a similar position commenting on High School standards. I think it’s worth consider what the data tells us, but it’s also worth taking a hard look at the data itself.

    I think the points about HOW computers are used in Title One schools made earlier are spot on. The tech gap between schools is closing, but much of what poorer students do is testing, and drill oriented, while more well-off students get to create and discover. I can safely say that I’m one of the few teachers doing content creation type work on the computer with students in this SES bracket.

  16. [...] dy/dan » Blog Archive » The Sealed Envelope working hard to make it look easy. (tags: assessment education technology) [...]