Justin Reich pours a little water on Carnegie Learning’s claims that their Cognitive Tutor program “doubles” a year’s worth of Algebra learning:
But where I’m flummoxed is how we are supposed to provide practicing educators with the tools to evaluate these kinds of findings. I know you can’t sell a curriculum product or a newsletter with headlines like “HUGE STUDY PARTIALLY VALIDATES ALGEBRA PROGRAM, PARTIALLY DOESN’T.” I don’t expect Carnegie Learning to build a web site that says “Major study shows no significant impact of Cognitive Tutor in middle schools!” But it also isn’t clear to me who in the system is incentivized to provide disinterested, broadly-accessible, readable summaries of important studies that help educators make careful decisions with scarce resources based on careful interpretation of existing evidence.
Raymond Johnson writes a fascinating six-part series outlining the politicking and deliberation that led to NCTM’s 1989 standards:
Despite the risk of bearing the responsibility for the Standards total estimated cost of $258,000 former Executive Director James Gates claimed “the proposal [to fund the Standards] was not submitted to either NSF or the U.S. Department of Education, so that no claims could be made that the federal government had funded the development of curriculum and evaluation standards.” In addition, the self-funding of the Standards and the decision to not write textbooks, as had been the case during the new math era, afforded the working groups relative independence from textbook publishers.
Tom HoffmanAugust 29, 2013 - 12:10 pm -
I just can’t believe any existing program could be a revolution hiding in plain sight given the current testing regime. If there was one middle school in Rhode Island that doubled their math proficiency in a year after adopting a piece of software, OTHER SCHOOLS WOULD BUY THE SOFTWARE. It would all be perfectly clear.
Mary DoomsAugust 29, 2013 - 12:37 pm -
Back in June I wrote two posts regarding Carnegie Learning asking if it was a Game Changer. One of the founders of Cognitive Tutor responded. I was also interested in learning what John Hattie had to say on the topic so I posed the question to him in a follow uppost.
Hattie’s response was, “… It is tough out there making changes and even changes of d> .20 can be worth striving for.”
I’m just a middle school teacher mind you, not a statistician. But I was interested in Carnegie’s claims because the effect size wasn’t “significant”–in my mind. Again I’m not a statistician, nor am I knowledgeable about what is technically deemed significant.
And this is precisely the point Justin Reich makes.
I welcome more conversation on this topic. Over the summer I read a comment on Diane Ravitch’s blog of how one teacher was directed to implement the program merely as a proctor in her 8th grade class (She did not use the word “proctor”, but that’s how I think she wanted it to be interpreted.) The high school department chair thought the program was all that.
I’m looking forward to the comments.
Jason DyerAugust 29, 2013 - 2:49 pm -
Just wanted to mention:
I’m using the Carnegie books this year (my entire district bought new books for Common Core). Reviews have been pretty mixed; a good amount of it is a more heavily scripted Boaler-esque curriculum but the students are currently used to lecture-and-take-notes so I have some Algebra 2 students who are melting down. The freshmen have been less indoctrinated so appear to be having a better time with Algebra 1.
The Cognitive Tutor questions look exactly like the PARCC questions (same user interface, even down to the color scheme). The same company owns both.
Kevin HallAugust 29, 2013 - 4:07 pm -
Regarding this: “But it also isn’t clear to me who in the system is incentivized to provide disinterested, broadly-accessible, readable summaries of important studies that help educators make careful decisions with scarce resources based on careful interpretation of existing evidence,” I would say the What Works Clearinghouse is what you’re looking for. Some would also say RAND.
Kevin HallAugust 30, 2013 - 6:02 pm -
Steve Ritter has written an interesting reply to Justin Reich’s post. See it here: