I used the Carnegie Learning software with a classroom full of students once a week on Thursdays. After 10 weeks of the mind-numbing random-guessing inducing software, I gave it up. The students figured out how to game the system, but it took me a few weeks to figure out how they were doing it.

The most valuable attribute a class has is the group of people coming together in a physical space. If you give that up, the students might as well stay at home.

Every one of these learning labs I’ve encountered or heard about has droves of people who hate it, and a small minority that love it. There is an adult version of this as well, which undergraduate students who require remedial mathematics are forced into at Virginia Tech and about 100 other schools across the USA. See http://davidwees.com/content/math-emporium-walmart-higher-education for more details on the success of that program.

Also recommend, see the research on Bennie using IPI. n=1 but it is still a pretty damning study of personalization in mathematics education: http://www.wou.edu/~girodm/library/benny.pdf

Thanks for bringing up this mostly overlooked downside of personalized learning. I read something from the http://edtechnow.net/ blog recently that really struck a nerve – a quote from William Cory, Assistant Master of Eton, who wrote in 1861:

“You go to school at the age of twelve or thirteen and for the next four or five years you are engaged not so much in acquiring knowledge as in making mental efforts under criticism.”

It’s that ‘mental efforts under criticism’ piece, that structured classroom discussion where your thoughts are challenged where higher order learning takes place.

I do think these online learning resources play an important part in supplementing the learning or providing a “level of shared experience and knowledge” as described in your article.

I see frequent use of the terms “personalization” and “individualization” in the comments and sense an over-reaction. Further the video defines the learning lab as “differentiation”. True differentiation requires formative assessments and then adjustment to meet the student’s needs. Just as the opening segment showed, our education system is based on that Henry Ford model, “any color as long as it is black”. And that is definitely not working.

Aren’t those learning labs really the same, one size fits all? Where is the personalization (differentiation) for not only ability, but learning modality and interest? Where is the inidividualization of product, process or content?

If we follow Tomlinson’s model of differentiation, we can still maintain the whole class, but still attend to the individual needs of our students. There is much to be said and explored in expanding our concept of the traditional schoolhouse for 21st century learning away from the current industrial model with bells, rows, and age group grade levels.

So why do we in eduation keep looking for and holding forth to one magic answer. As an District Instructional Systems Specialist for Math, my vocabulary has become so full of acronyms (PBL, STEM, EDP, DI, 5e’S, etc) I fear I may soon lose my voice.

What I have come to see as a common thread in the various approaches is engagement. CAI provides an engaging practice platform, but in spite of recent programming advancements and sophistication, still doesn’t help the student “make sense” of the math. Classroom discourse is necessary and essential as students share, discuss, and critique their’s and other’s reasoning. The task now is to get a nation of teachers to see it.

Thanks for letting me rant.

P.S. When all else fails blame the unions.

It’s important to assess thing like this not only in terms of how effectively they teach math, but also in terms of what they teach children *about* math. The learning lab teaches children that math is a solitary activity, wherein one clicks at things on a computer until the computer approves.

Even if it taught math brilliantly, what it teaches kids about math seems pretty damaging. Much better to teach that math is the collaborative, exploratory, iterative, playful creation, communication, and comprehension of abstractions.

Whenever I hear an “or” debate (personalized learning by computer OR whole class learning by teacher, for example), I am frustrated. Clearly, there are good things and bad things about both. Is it possible for us to think about how the good parts together can make a better whole? Instead of choosing one over the other, how can ideas be combined to benefit students? What’s wrong with personalized learning by teacher with the support of technology? Or whatever other hybrid name you’d like to come up with.

Teachers naturally take good ideas from each other and make them work in their own classrooms or schools. Why, at a higher level, can’t we accept that there is no magic bullet that’s always the best? It’s ok to figure out what works and run with it, even if it’s not someone’s pure idea as they implemented it.

Glad to see ST Math rated a “isn’t bad” from Dan, and happy to hear that at least someone’s listening (see #10 Max’s comment above) when I babble away at NCSM.

I’m not going to comment on Rocketship’s implementation of ST Math specifically, but I do think there are one or two points of interest about ST Math in general that are worth adding to this discussion.

In the blog post Dan says, “…but computers constrain the universe of math questions you can ask down to those which can be answered with a click and graded by a computer.” This is entirely true for about 99.999% of computer-based math programs. They are generally some kind of multiple-choice type thing which does not encourage student exploration, curiosity, or strategic thinking. [I note with interest David’s comments on Carnegie above which I would thoroughly agree with from my experience both in and out of the classroom at LAUSD].

Most instructional software is really practice software. It’s somewhat useful for students to practice or brush up on material they’ve already learned (ALEKS is an example of this), but it’s not very good at actually teaching new ideas – don’t even get me started on the Khan Academy and video lectures! To learn new ideas, students need to be challenged, they need to figure things out for themselves, they need to make their own neural connections – and classroom discussion guided by an excellent teacher using well-designed instructional materials is a great way for this to happen… BUT – sadly that is NOT what happens in a large number of classrooms.

So the question becomes, how can we – at scale – provide those challenging learning experiences for ALL students (not just those lucky enough to be blessed with an excellent math teacher)?

And this is where ST Math is so powerful: There is practically no multiple choice, and the entire program is about solving visual math puzzles (which do transition to symbolic modes). Students develop strategic thinking skills, develop intrinsic motivation for mathematics, and build the schema needed for number sense. It is also an incredible tool for all teachers to use as a way of stimulating student discussion by having them explain their thinking and solution strategies in the classroom.

For a discussion of some of the neuroscience and theory behind the program you might like Dr. Matthew Peterson’s TEDx talk about “Math Without Words” http://bit.ly/pfXL6Q, or a TEDx talk I did recently “The Geometry of Chocolate” http://bit.ly/YtLCy8

I recently went to CA and visited Mosaic within the Rocketship School group. While I agree with the thinking posted here about the learning lab and have the same concerns addressed about their use of technology the school had a lot of good things going for them too that are not reflected here. Yes, the learning lab is set up for students to be there for an hour for 30min spent on reading and 30min spent on math. Yes, the facility was run worked by aide salary people and I couldn’t get concrete information if the lab was truly organized for each student to be working at different levels based on their needs. The day we were there we were told the lab at Mosaic was being redesigned and they couldn’t release what that might look like. I don’t think their teacher’s are paid more because their students spend an hour at a computer. Their school day is longer than many schools, 1.5 longer than mine. The teachers are young but are investing lots of time for professional development and meetings beyond the school day. They also have class sizes of 30, K-5. They help teach PE and the school does not have music or art.

Some of the great things I saw involved high parent involvement, support, and understanding of the school’s mission and test results. I saw great workshop models for reading and writing, best practices going on in the classroom with real teachers. I heard about PD that is needed in every building in our country, they care about growing their teachers. I wrote several post about my visit, please feel free to stop over and read them. The learning lab is not ideal but that is not the only thing happening at Rocketship Schools. I’m sad to think this is what they are claiming or getting some fame from.

http://enjoy-embracelearning.blogspot.com/2012/12/research-and-development-project.html

http://enjoy-embracelearning.blogspot.com/2012/12/research-and-development-student-growth.html

http://enjoy-embracelearning.blogspot.com/2012/12/research-and-development-engagement.html

http://enjoy-embracelearning.blogspot.com/2012/12/research-and-development-personalization_14.html

I don’t know of any math software that allows for graded text entry, for instance, so right there we leave out constructing and critiquing other peoples’ arguments.

Why not pay one of these low-cost workers to grade text, then? That seems like it could be just a slight increase in costs.

“The learning lab teaches children that math is a solitary activity, wherein one clicks at things on a computer until the computer approves.”

Perhaps not TERRIBLY different from the way many math classes operate, if you simply substitute “teacher” for computer in the second instance so that we have, “Math classes teach children that math is a solitary activity wherein one writes or says the answers to computations until the teacher approves.”

Hi Nigel,

I would love to read the research that supports your statements above, particularly:

“Students develop strategic thinking skills, develop intrinsic motivation for mathematics, and build the schema needed for number sense.”

Could you direct me to this?
Thanks in advance.

Like Nigel, I think it’s a pretty dreadful mistake not to examine the actual software. We’ve been looking at the same stuff — and yes, almost all of it is presentation of math equations with students providing what they think are the answers. It’s practice — and not practice designed to develop concepts.
Saying it’s “one size fits all” because it’s the same software program is making many the grand assumption. If the software provides many options, then that metaphor is like saying clothing is “one size fits all” because it comes from the same store. If there’s a fitting booth that takes an image of my body and finds clothes that will fit it, then it’s not “one size fits all” any more.
I’d love to see discussion of whether the transition from visual puzzles to symbolic math is effective. I suspect that it just might be. People aren’t discussing that, though… they’re deciding whether or not Computers Are Effective. If the computer programs that spit out symbolic equations and ask for answers aren’t effective, it might be because of that delivery, not because it’s Being Done On A Computer.
Finally, I *think* I”m supposed to look at students gazing at a screen and assume they are not engaged. I can be awfully still when I’m engaged in thinking. Is there a better measure?

Hi Blaise (#24). Thanks for your question. On the subject of research, you may find this study particularly interesting [Spatial Temporal Mathematics at scale] (note this is an interim report part of a multi-year study – still ongoing – funded by the I.E.S. and conducted by UCI) http://www.gse.uci.edu/CRCL/documents/AERA%202010.pdf

Additionally there is a wealth of efficacy data you can find here:
http://mindresearch.net/cont/research/re_ResultsAtScale.php Note the key idea here: these are studies not of small scale implementations in one or two classrooms but at scale across entire school districts.

One of the biggest ideas to get our collective heads around here is that just as all teachers are not “created equal”, neither are all computer programs. Just as Dan’s integration of technology (with his 3act lessons) actually has the potential to increase student motivation, curiosity, and critical thinking, so do other innovative approaches using new tools. One of the major problems I see when looking at most technology is that far to often it is little more than a replication of the dry, ineffective, “tell you everything and ask you to repeat it” teaching practices that sadly I saw in all too many classrooms during my five years as a trainer of math teachers and coaches in LAUSD.

I was intrigued by Siouxgeonz’s point about how do we assess engagement (and productive engagement) in solo computing tasks. It made me wonder, “what is the role of guessing in learning a mathematical concept?” as well as “what is the role of practice in learning a mathematical concept?” and finally, “what characterizes productive guessing?” and “what characterizes productive practice?”

It seems clear from the article and from the study that David shared (in Comment #2) that guessing is a strategy students use on computer programs, much like they do in math class, and they wait for the computer or teacher to give them feedback about their guesses. Sometimes, that strategy is effective. For example, guessing or estimating is part of Riley Lark’s ActivePrompt and part of the 3 Act structure of problem solving that Dan is developing. It serves as formative assessment, increases engagement, creates a “low floor” task, and more.

On the other hand, guessing is part of the trouble that the famous “Benny” gets into in the study David cited. He guesses and gets feedback about right/wrong that leads him to come up with his own mental model of what leads to right answers… a mental model that breaks down pretty quickly but which he persists with, even in the face of negative feedback.

So what makes guessing productive? One thing might be the kind of feedback students get. I know that ST Math’s visual puzzles provide visual feedback showing how students’ wrong answers fail. Here’s an example to try for yourself: http://mindresearch.net/media/edu/demoFolder/demo/games/index.html

Does feedback that shows why an answer is wrong lead to more robust mental models than feedback that just tells if the answer is right or wrong? How do hints or further questions (without showing the right/wrongness) play into this? That seems like an actionable question, and one that will matter for computer-based learning software and teacher-based learning experiences.

Similar questions could be asked about engagement. What does meaningful engagement look like? Can it be silent? Wordless? When and why does engagement matter? One way to think about engagement is to think about intellectual need, (which Dan refers to in “The Necessity Principle” https://blog.mrmeyer.com/?p=14871 and which comes from research by Fuller, Rabin, and Harel). When do students feel like they *need* to know how to figure something out? And that they need to get good at figuring it out?

One worry I have with gaming is that accomplishing the goal (scoring points, hearing the reward noise, moving Jiji) becomes the focus of the activity, the thing students need to do. The need becomes social or reward based. On the other hand, in order to accomplish the goal, they need to get good at the mathematical task, whether it’s estimating fractions, visualizing rotations in space, or understanding what happens when integers are operated on… And the ST Math program doesn’t prescribe how to get good at it, just poses the task.

I’ve been (slowly) reading the really good book, “Beyond Constructivism” by Lesh and Doerr which gives a theory of math learning that’s about building and improving mental models. One argument they make is that model building has a social component — good models are robust in the face of others’ challenges, and easy to communicate to others. I find it notable that students, when they are making good use of the ST Math games, seem to have a spontaneous need to talk about them (if you watch the first Ted Talk that Nigel linked to, there’s an anecdote of an autistic student who begins talking in order to talk about how he plays these games).

I wonder if one way to make students’ guessing and engagement more productive, especially guessing during and engagement with mathematical puzzle games, is to encourage conversation about what they are doing. When students have to explain how they solved the task, using language, and make another student understand, they are refining their mental model. When they have to decide whose way is better and whose way will always work, they are improving their mental model and perhaps adapting a better, more robust one. It may be that the student talk about the computer game is as important as getting good at the game!

And then, what is the role of the adult in the room at that point? Are they gathering data? Assessing? Guiding? Facilitating? Sugatra Mita’s Hole in the Wall computing experiments suggest that good listening is all it takes to make communication productive — that with a decent task and lots of requirements for students to talk, a patient listener who asks, “how do you know?” and “wow, how did you do that?” can be enough to spur lots of learning!

What if the Learning Centers continued to use math software that posed rich puzzles, but took away the dividers and headphones, made a 3:1 student:computer ratio, and asked their low-skilled workers to help the students take turns, be polite, explain how they know, and tell why they think they’re right. Also equip the teachers and students with flipcams, screen capture software that caught audio too, or cell phone cameras, in which they recorded their conversations “whenever there was a good argument or they thought they were about to figure something out.”

You keep the money-saving features of the lab, need fewer computers, and students are engaged in discourse and model-building, and they get meaningful feedback from peers and the software.

Max (30)
Guessing in the 3 Act has one big difference. It starts with give me an answer that is obviously high/low. So it isn’t a pure guess at all but an estimation. Suddenly, all students have subsequent guesses that are constrained by obvious wrongs.

As to the changes in the computer lab, what if they also included a highly paid trained math teacher to record observations of student discussions? Then followed up with targeted lessons in an actual classroom?

In 2013 it’s not that hard to collaborate on a computer. In fact it’s kind of fun and engaging in itself. And useful. Heck, we do it all the time on blogs and twitter. Look no farther than Patrick Honner’s “More Equalateral” problem (http://mrhonner.com/2012/11/12/which-triangle-is-more-equilateral-2012-edition/).

The problem here isn’t the banks of computers, it’s the solitary nature of them. It’s embarrassing that they (slash we) haven’t fostered the kind of creative collaboration that builds appreciation, confidence, and conceptual understanding.

Dan, thank you for the post and this: “It takes a lot of skill to ask students to explain how they know what they know and why they they’re right, and then to do anything constructive with those responses.”

I couldn’t agree more. I believe that this is exactly why it’s so difficult (impossible?) for any software to do what highly effective teachers can do. Not that anyone here suggests one replaces the other, but it’s important to note that the higher salaries afforded to teachers at Rocketship come at a detriment to their kids’ learning in the isolation labs.

I’m the Curriculum Director at DreamBox Learning, and we’ve been partners with Rocketship for the past several years. They use our adaptive math software in their Learning Lab as well. This is a great conversation, and I have 3 points to add to the dialogue.

1. Planning Backward. Like the thousands of other elementary schools who use our math software, Rocketship is working to figure out how to strategically use classroom time as well as technology to improve student learning. All schools should be planning backward that way – looking at the learning outcomes they want for students and then deciding to hire teachers, purchase materials and technologies, and set school schedules that will cause those outcomes (see “Schooling by Design” by Wiggins & McTighe). Rocketship’s commitment to review and revise their structures — especially given their high profile and media scrutiny — is a healthy thing for their organization. The K-12 district I worked in before joining DreamBox also eliminated elementary computer labs a couple of years ago and put the hardware back into the classrooms for similar reasons. But we weren’t on PBS.

Regardless of what one thinks about Rocketship’s model overall, it’s important to note here they do have math classes every day where students engage in conversations and problem solving. They supplement that classroom math instruction with math software in the Lab. The software, like the classroom instruction, is a means toward the same curricular ends. Because students have learning experiences both on- and off-line, there’s no underlying message that math is a solitary activity.

2. Software Quality. I’d agree with Nigel’s point that nearly all math software out there follows the “watch then practice” or “memorize and drill” formats. In these formats, students passively receive explicit instruction and have no opportunity to develop original ideas of their own and on their own. DreamBox isn’t like that. It’s adaptive, but not IPI. I’ve written several pieces explaining how our adaptive learning platform is informed by Freudenthal & Fosnot, not Skinner (one article: http://bit.ly/U0U42R). But you can also see for yourself. We have many K-5 lessons available for anyone to play and experience as if you were a student (3rd-5th grade here: http://bit.ly/NanvgR). I’d encourage you to try them. Click the ‘hint’ button. Get some problems wrong – our teachers write lessons that give students targeted feedback in specific situations. Watch how students use mathematical models and develop mathematical vocabulary. Then decide if you think kids are developing mathematical thinking. We’ve also formatted many of these digital tools into free Interactive Whiteboard tools so that classroom teachers can use them to support rich conversations in class (K-5 tools here: http://bit.ly/5d7yo6). Some of my favorites are Multiplication with the Open Array (a good precursor to partial products and the Distributive Property), Division with Remainders, and Multiplying Fractions.

3. The CCSS Practices. It’s important to note that students in primary grades are still learning to read and write. So text entry isn’t even a possibility, and it’s best for students to construct and critique arguments verbally and with pictures and manipulatives in class. But software can help students acquire the vocabulary, number sense, and mathematical models with which to make their arguments verbally. And I think technology is uniquely suited to strongly support the CC Practice Standard: “Look for and make use of structure.” I’ve written about it here: http://bit.ly/VH7vGk. Even in a small class with a great teacher, it’s challenging to be sure every student is getting better at independently looking for mathematical structures. That’s difficult to assess. Peers and teachers can, and do, help students “make use” of structures; but in doing so they often hinder students from developing the habit of “looking for” structures as mathematicians routinely do.

William’s comment brilliantly expresses my disappointment with almost all adaptive math systems: the student’s “interaction” is determined by the prescribed plan of software. There is no authentic feedback, only canned responses written by a software developer, which often does not change to fit a student’s learning style. Any educational program which reduces, or ignores, the role of collaboration is destined to let its students down.

Is anyone working on a system where a student, upon logging in, is presented with an appropriate online collaboration with other students? Having students work through a challenge together in an online environment would be exciting to me. The sophistication of the problems presented could somehow be determined by a student’s participation, responses, and persistence. This is where some of the applets Dan throws out now and then, along with newcomers like ActivePrompt, excite me as potential means to allow for real collaboration.

The great myth is that technology alone causes education to become more interactive and engaging. Teaching, done well, has always been interactive and engaging. The great gifts technology has brought us are opportunities for collaboration and connectivity. Sadly, the adaptive systems I have seen miss this mark. When somebody starts thinking realistically about the possibilities of collaboration in education technology, I’ll be ready to buy stock in their product…or a job…that would be cool too…

Just wanted to add, Dan, that I don’t disagree that making use of student thinking is hard. In fact I think it’s hard beyond the level of what individual teachers are called to do, and crying out for more research and the creation of more tools for individual teachers to use — I think math teachers need a lot more support for seeing & fostering emerging mental models and developmental trajectories within students’ creative problem solving work (and a lot fewer curricula and tools that restrict students’ developing mental models).

My goal was to imagine that the school had a need for kids engaging in academic work unsupervised by teachers, and figure out how to make that time involve thoughtful guessing (more constraints and estimation, less randomness, as Brendan points out), real engagement, collaboration, problem-solving, etc. and be useful to the teachers. My best thought so far was: good tasks (check — and the computers do help with that), collaboration (require students to work together, end the headphones), explicit focus on good communication & reflection (train low-skilled workers to prompt for it even if they can’t make use of it), and capturing data other than clicks (using the videos). It still requires skilled teachers to take the time to review the data and figure out how to make use of it, but it might allow kids to do meaningful math and more effective practice and link lab time with class time, without requiring lab time to be actively run by teachers. Is it worth it? I guess it depends on your school’s needs. If it frees the teachers up for lesson study, or reviewing student work, or team planning, it might be.

Surely there must be research that measures the attention span of a student working on these CAI programs? If not, you would think someone would find a way to measure the actual amount of “thinking” that takes place while a student is “working.” Some students, who are predisposed to staring at a screen, might be getting all the correct answers, but there may not be any actual “thinking” going on. Other students, who are getting answer wrong, might actually get more out of it, because the computer is challenging them. At the same time, there has to be some sort of “burnout” factor that shows that there are diminishing returns when it comes to engagement: perhaps the first 15 minutes are the most productive, and the balance of the 45 minutes shows a constant diminishing of interest and engagement.

Does anybody know of this research?

It looks like a warehouse for children. A cubefarm.