One of the most fascinating pieces to come out of the winter break was this segment from PBS NewsHour’s John Merrow on the Rocketship charter network.

The video distills into ten minutes all the most interesting angles on Rocketship — its high parent involvement, its high teacher salaries and professional development, its morning “launches,” and the segment pays special attention to Rocketship’s “Learning Labs,” which Merrow describes as “lots of computers and kids, no teachers.” (Watch that part of the segment.)

This aspect of a lot of charter and for-profit schools should make us all very uneasy. Rocketship can afford to pay its teachers more because, for one hour each day, the students are plugged into computers, boxed into cubicles, and tutored intermittently by low-skill, hourly-wage workers. Rocketship spruces up its lab with lots of primary colors but it can’t shake comparisons to a call center.

This is “differentiation,” says John Merrow, and it’s true that the students are working on different tasks, but at what cost? The students don’t interact with their peers or their teachers. The math program, ST Math, isn’t bad but computers constrain the universe of math questions you can ask down to those which can be answered with a click and graded by a computer. The promise of personalization, of perfectly differentiated education, has forced Rocketship to make dramatic concessions on the quality of that education. It’s a buffet line where everyone chooses their own flavor of the same gruel.

Merrow’s documentary team wasn’t persuaded of the Learning Lab’s merits:

The Learning Lab saves schools lots of money but there’s just one problem: they’re not really working. A problem we saw is that some students in the lab do not appear to be engaged. They sit at their computers for long periods of time, seemingly just guessing.

What’s remarkable is that the Rocketship staff is *also* unpersuaded of the Lab’s merits. One principal says, “If I had to guess, I’d say you come back in a year, you won’t see a Learning Lab.” Another says, “Next year we’re thinking of bringing the computers back to the classroom.”

This isn’t any kind of small pivot, something Rocketship can gloss over with a sunny press release. Throughout Merrow’s segment, the teachers, the principals, and the charter CEO all spoke of their commitment to innovation. We should commend them for innovating *away* from technology when it’s ineffective, especially given their particular location (Silicon Valley) and time (2013). That just isn’t easy.

**BTW**: Mike Caulfield suggests that personalization is hostile to the kind of whole-class conversation we know to be valuable:

Indeed, structured classroom discussion has one of the highest effect sizes in Hattie, much higher than mastery learning. But it’s really difficult to have a classroom discussion (or group activities that foster student discussion) without some level of shared experience and knowledge. I’m curious if this fact might lie behind much of the surprising failure of computerized adaptive learning systems.

**2013 Jan 09**. Edsurge got Rocketship CEO John Danner on record. The Learning Labs are staying:

Online learning is integral to our model…The Learning Lab is not going away, rather we are working to integrate its key components directly into our classrooms under the guidance of our incredible teachers and staff…I think Merrow probably just happened to focus on an isolated incident and wanted to bring it up as it is always a valid concern with online learning. We continue to work on the data integration piece and this pilot doesn’t change the importance of that. Our teachers continue to get more robust data from the Learning Lab and are eager for us to work towards a fully integrated and real-time system.

**2013 Jan 25**. MindShift reports that Rocketship is, indeed, moving the computers back to the classrooms.

**Featured Comments**

I read something from the http://edtechnow.net/ blog recently that really struck a nerve – a quote from William Cory, Assistant Master of Eton, who wrote in 1861:

“You go to school at the age of twelve or thirteen and for the next four or five years you are engaged not so much in acquiring knowledge as in making mental efforts under criticism.”

It’s that ‘mental efforts under criticism’ piece, that structured classroom discussion where your thoughts are challenged where higher order learning takes place.

It’s important to assess thing like this not only in terms of how effectively they teach math, but also in terms of what they teach children *about* math. The learning lab teaches children that math is a solitary activity, wherein one clicks at things on a computer until the computer approves.

Not only should we be concerned about what students are learning about math based on this experience but what they are learning about computers as well. I’m sure the majority of schools are not doing a much better job of offering elementary students the opportunity to use computers as more powerful tools rather than skill-practice machines, but most don’t have kids doing so quite this much. If we want students who will explore, innovate, challenge ideas, we have to help them see more possibilities than simply answering questions and being told right or wrong.

One simple filter, Jungian type, tells us that over half of all children aren’t going to be energized by an hour at a computer screen. Extraversion and Introversion in personality type terms involve how we are energized. All of us can do both, but one is preferred and the other is draining. Further, even if the Introverts like the computer lab, they still need the stimulation of discussion, learning to express their ideas and question those of others. Since a good portion of school is still set up for more Introverted activities, adding interventions that require more Introversion makes it a very, very long day for the Extraverts–and they just might start talking and moving when you least want them to.

“The learning lab teaches children that math is a solitary activity, wherein one clicks at things on a computer until the computer approves.”

Perhaps not TERRIBLY different from the way many math classes operate, if you simply substitute “teacher” for computer in the second instance so that we have, “Math classes teach children that math is a solitary activity wherein one writes or says the answers to computations until the teacher approves.”

Out of character, writing this sort of stuff is *hard*. It’s hard for actual live human beings to understand how students are modeling the math in their head and respond accordingly. Poor Jennifer [DreamBox’s computerized teacher-avatar –

dm] just repeats her instructions. If I were a student who didn’t understand place value, I might walk away from this unsure about my own multiplication facts, that were good.Jennifer might help me more if she knew about some common errors (and maybe that sort of thing is going on in the background, invisible to the student?). Like Dan, I don’t want to be a luddite, and if the computer is better than people, we should go for it. But computers have a long way to go.

Much of teaching is empathy – being able to see the world through the eyes of a person who doesn’t know the things you know. It’s being able to communicate with someone who sees the world differently than you do. There are a thousand ways that live, in person communication can cultivate and encourage that empathy in teachers. For programmers who are at arms length, cultivating that empathy is double difficult and important.

Jennifer just asked me if I’d like to continue working, ’cause it took me a while to write this. I think my answer would be “no”?

So just as you imagined this hypothetical student in a DreamBox lesson, I think it’s valuable to imagine this same student entering a classroom without the support of a technology like DreamBox:

The multiplication standard algorithm is a fifth grade Common Core standard, so let’s assume the student is a fifth grader who doesn’t understand place value. This student transfers into a new school and math class on the day after the teacher introduced the algorithm. Does the teacher know the student doesn’t understand place value? If not, how will that information be acquired? Once it’s known that the student lacks place value understanding, should the teacher continue teaching the algorithm lesson even though the student is clearly not ready for it? If not, what does the student do during math class?

Too often, the student is taught the algorithm right then because there are simply too many logistical and resource constraints that limit what even the best teacher is able to do in that situation. It’s no certainty that the student will meet grade level standards by the end of the year, and the inherent challenges of this reality end up being a huge strain on both teacher and student. I’m empathetic to both of them. And the tens of thousands of others in the same situation. These are the teachers and students we’re trying to help.

## 53 Comments

## Ben

January 7, 2013 - 9:17 amSounds like we have a new educational debate worthy of holding it’s own against the age old “whole language vs. phonics” debate. Whole class vs. personalization is a conversation that I’m hoping we can sew up a bit more neatly and come to some nice middle ground where we recognize the value of individual exploration and work while still bringing our experiences back to the whole class collective.

In other words, what a lot of really great teachers have done their entire careers; practice a concept, learn from mistakes, share with the group, repeat.

## David Wees

January 7, 2013 - 9:42 amI used the Carnegie Learning software with a classroom full of students once a week on Thursdays. After 10 weeks of the mind-numbing random-guessing inducing software, I gave it up. The students figured out how to game the system, but it took me a few weeks to figure out how they were doing it.

The most valuable attribute a class has is the group of people coming together in a physical space. If you give that up, the students might as well stay at home.

Every one of these learning labs I’ve encountered or heard about has droves of people who hate it, and a small minority that love it. There is an adult version of this as well, which undergraduate students who require remedial mathematics are forced into at Virginia Tech and about 100 other schools across the USA. See http://davidwees.com/content/math-emporium-walmart-higher-education for more details on the success of that program.

Also recommend, see the research on Bennie using IPI. n=1 but it is still a pretty damning study of personalization in mathematics education: http://www.wou.edu/~girodm/library/benny.pdf

## Brian

January 7, 2013 - 10:09 amI am curious that the news clip mentioned the learning lab as the linchpin that allows Rocketship to afford things like higher salaries, academic deans, and response to intervention. If they get rid of learning labs next year, how are they going to continue to afford these services?

I’m not sure if they said or not, but are students in the learning lab on computers for an hour straight, or is it broken up over the course of the day? I can’t imagine how boring that must get if the students have to do that for an hour at a time everyday. No wonder they get bored! As a classroom teacher, I might teach a specific subject for an hour, but there is a lot of variety of interactions and work within that hour usually.

I’m disheartened to hear that they have eschewed the arts and humanities to make more time for “core” academics. It’s an unfair burden to expect parents to make up for this, especially since not all students will have equal access as a result.

## Clyde Boyer

January 7, 2013 - 10:09 amThanks for bringing up this mostly overlooked downside of personalized learning. I read something from the http://edtechnow.net/ blog recently that really struck a nerve – a quote from William Cory, Assistant Master of Eton, who wrote in 1861:

“You go to school at the age of twelve or thirteen and for the next four or five years you are engaged not so much in acquiring knowledge as in making mental efforts under criticism.”

It’s that ‘mental efforts under criticism’ piece, that structured classroom discussion where your thoughts are challenged where higher order learning takes place.

I do think these online learning resources play an important part in supplementing the learning or providing a “level of shared experience and knowledge” as described in your article.

## Chris Robinson

January 7, 2013 - 10:49 amMy concern with the push towards “personalization” or “individualization” of students’ education is the fact that I have yet to see an educational institution (read departments of education) that don’t ask districts to teach grade-level standards, regardless of where the students’ math abilities are. To go along with that, they also require standardized, grade-level assessments that are often used to determine teachers’ and districts’ effectiveness. So where are the “personalized” or “individualized” assessments at?

## Paul

January 7, 2013 - 11:58 amI see frequent use of the terms “personalization” and “individualization” in the comments and sense an over-reaction. Further the video defines the learning lab as “differentiation”. True differentiation requires formative assessments and then adjustment to meet the student’s needs. Just as the opening segment showed, our education system is based on that Henry Ford model, “any color as long as it is black”. And that is definitely not working.

Aren’t those learning labs really the same, one size fits all? Where is the personalization (differentiation) for not only ability, but learning modality and interest? Where is the inidividualization of product, process or content?

If we follow Tomlinson’s model of differentiation, we can still maintain the whole class, but still attend to the individual needs of our students. There is much to be said and explored in expanding our concept of the traditional schoolhouse for 21st century learning away from the current industrial model with bells, rows, and age group grade levels.

So why do we in eduation keep looking for and holding forth to one magic answer. As an District Instructional Systems Specialist for Math, my vocabulary has become so full of acronyms (PBL, STEM, EDP, DI, 5e’S, etc) I fear I may soon lose my voice.

What I have come to see as a common thread in the various approaches is engagement. CAI provides an engaging practice platform, but in spite of recent programming advancements and sophistication, still doesn’t help the student “make sense” of the math. Classroom discourse is necessary and essential as students share, discuss, and critique their’s and other’s reasoning. The task now is to get a nation of teachers to see it.

Thanks for letting me rant.

P.S. When all else fails blame the unions.

## Chris S

January 7, 2013 - 12:13 pmI was critical of adaptive learning systems for years. But, my eight-year-old son, who has a learning disability, comes home and wants to do ALEKS. So, I had to give them a second thought.

If they are designed well, they can be a useful resource for simple procedural skills. I can see them helping to build kids computational, keyboarding, or grammar skills. Of course, kids really need rich context and engaging curriculum. But, a ‘structured discussion’ is limiting if if a kids is missing background knowledge, can’t process auditory information well, or is shy. Even good teachers talk too damn much as well.

I would be upset if my kids did ALEKS in school for more that 20 minutes a day. But, I would recommend good adaptive learning systems as a tool in a curriculum, especially with home access.

## William

January 7, 2013 - 12:40 pmIt’s important to assess thing like this not only in terms of how effectively they teach math, but also in terms of what they teach children *about* math. The learning lab teaches children that math is a solitary activity, wherein one clicks at things on a computer until the computer approves.

Even if it taught math brilliantly, what it teaches kids about math seems pretty damaging. Much better to teach that math is the collaborative, exploratory, iterative, playful creation, communication, and comprehension of abstractions.

## Max

January 7, 2013 - 1:38 pmI think ST Math is related to the Mind Research Institute — I recognized the cute little penguin (Jiji?). I remember a conversation with Nigel Nisbet, a Senior Mathematics Specialist at MRI, at NCSM last year. He mentioned that in future iterations of their software, he wants to get rid of sounds so that kids don’t have to wear headphones. The idea being that in an ideal implementation of their program, the students are talking. Mostly to each other. In explaining how they figured out how to solve the problems (how to get Jiji across the screen) the kids learn a lot, and the teachers learn a lot about their kids’ thinking. Nigel said they were actually trying to figure out PD to help teachers learn from how their kids talk informally about the math they’re doing, and feed that talk back into math class conversations. It sounds like the teachers are right on with their desire to get the computer work back into the classroom, or at least to improve the pipeline between what kids do on computers (working together!) and what they do in class.

Note: I have no professional relationships with ST Math, or the Mind Research Institute. I like to play their demos when I go to conferences.

## Patti

January 7, 2013 - 2:30 pmWhenever I hear an “or” debate (personalized learning by computer OR whole class learning by teacher, for example), I am frustrated. Clearly, there are good things and bad things about both. Is it possible for us to think about how the good parts together can make a better whole? Instead of choosing one over the other, how can ideas be combined to benefit students? What’s wrong with personalized learning by teacher with the support of technology? Or whatever other hybrid name you’d like to come up with.

Teachers naturally take good ideas from each other and make them work in their own classrooms or schools. Why, at a higher level, can’t we accept that there is no magic bullet that’s always the best? It’s ok to figure out what works and run with it, even if it’s not someone’s pure idea as they implemented it.

## Scott

January 7, 2013 - 4:25 pmThank you for posting Dan. While I don’t agree with the learning labs at all, I do like the innovation and ability to easily change practices if something isn’t working.

## Nigel Nisbet

January 7, 2013 - 4:46 pmGlad to see ST Math rated a “isn’t bad” from Dan, and happy to hear that at least someone’s listening (see #10 Max’s comment above) when I babble away at NCSM.

I’m not going to comment on Rocketship’s implementation of ST Math specifically, but I do think there are one or two points of interest about ST Math in general that are worth adding to this discussion.

In the blog post Dan says, “…but computers constrain the universe of math questions you can ask down to those which can be answered with a click and graded by a computer.” This is entirely true for about 99.999% of computer-based math programs. They are generally some kind of multiple-choice type thing which does not encourage student exploration, curiosity, or strategic thinking. [I note with interest David’s comments on Carnegie above which I would thoroughly agree with from my experience both in and out of the classroom at LAUSD].

Most instructional software is really practice software. It’s somewhat useful for students to practice or brush up on material they’ve already learned (ALEKS is an example of this), but it’s not very good at actually teaching new ideas – don’t even get me started on the Khan Academy and video lectures! To learn new ideas, students need to be challenged, they need to figure things out for themselves, they need to make their own neural connections – and classroom discussion guided by an excellent teacher using well-designed instructional materials is a great way for this to happen… BUT – sadly that is NOT what happens in a large number of classrooms.

So the question becomes, how can we – at scale – provide those challenging learning experiences for ALL students (not just those lucky enough to be blessed with an excellent math teacher)?

And this is where ST Math is so powerful: There is practically no multiple choice, and the entire program is about solving visual math puzzles (which do transition to symbolic modes). Students develop strategic thinking skills, develop intrinsic motivation for mathematics, and build the schema needed for number sense. It is also an incredible tool for all teachers to use as a way of stimulating student discussion by having them explain their thinking and solution strategies in the classroom.

For a discussion of some of the neuroscience and theory behind the program you might like Dr. Matthew Peterson’s TEDx talk about “Math Without Words” http://bit.ly/pfXL6Q, or a TEDx talk I did recently “The Geometry of Chocolate” http://bit.ly/YtLCy8

## Shaun Errichiello

January 7, 2013 - 5:32 pmI am wondering if there are schools using computer based models and software that encourage group discussion and problem solving?

## Mandy Robek

January 7, 2013 - 6:06 pmI recently went to CA and visited Mosaic within the Rocketship School group. While I agree with the thinking posted here about the learning lab and have the same concerns addressed about their use of technology the school had a lot of good things going for them too that are not reflected here. Yes, the learning lab is set up for students to be there for an hour for 30min spent on reading and 30min spent on math. Yes, the facility was run worked by aide salary people and I couldn’t get concrete information if the lab was truly organized for each student to be working at different levels based on their needs. The day we were there we were told the lab at Mosaic was being redesigned and they couldn’t release what that might look like. I don’t think their teacher’s are paid more because their students spend an hour at a computer. Their school day is longer than many schools, 1.5 longer than mine. The teachers are young but are investing lots of time for professional development and meetings beyond the school day. They also have class sizes of 30, K-5. They help teach PE and the school does not have music or art.

Some of the great things I saw involved high parent involvement, support, and understanding of the school’s mission and test results. I saw great workshop models for reading and writing, best practices going on in the classroom with real teachers. I heard about PD that is needed in every building in our country, they care about growing their teachers. I wrote several post about my visit, please feel free to stop over and read them. The learning lab is not ideal but that is not the only thing happening at Rocketship Schools. I’m sad to think this is what they are claiming or getting some fame from.

http://enjoy-embracelearning.blogspot.com/2012/12/research-and-development-project.html

http://enjoy-embracelearning.blogspot.com/2012/12/research-and-development-student-growth.html

http://enjoy-embracelearning.blogspot.com/2012/12/research-and-development-engagement.html

http://enjoy-embracelearning.blogspot.com/2012/12/research-and-development-personalization_14.html

## Jeff Layman

January 7, 2013 - 9:01 pmAt the end of the day, most educators ought to be able to take a look at that call center picture and come to the conclusion that it’s not going to make kids jump out of the car with the forgot-to-kiss-my-mom excitement about school that you get to see with that age group. That alone ought to throw up some red flags about whether or not personalization is worth it.

## Dan Meyer

January 7, 2013 - 11:10 pmThanks everybody, and especially Clyde and William, whose commentary I kicked up to the main post.

Patti:I’m not sure that this is an either / or situation, but read William’s comment above. When a computer program tells students that math is something you watch someone else do and then mimic, it is sending students a

contradictorymessage to one that says math is something you can construct and learn on your own. Those messages don’t harmonize nicely.Nigel Nisbet:From what little I’ve seen of ST Math, its universe of math question types is larger than most, and that’s great. But grab the CCSSM practice standards — or any list made by mathematicians describing what mathematicians do — and notice what even top tier personalized learning systems can’t accommodate. I don’t know of any math software that allows for graded text entry, for instance, so right there we leave out constructing and critiquing other peoples’ arguments.

I’m not saying a single personalized learning program should do

everything. But I want to be mindful of the costs, which are considerable.## Jenny

January 8, 2013 - 3:13 amNot only should we be concerned about what students are learning about math based on this experience but what they are learning about computers as well. I’m sure the majority of schools are not doing a much better job of offering elementary students the opportunity to use computers as more powerful tools rather than skill-practice machines, but most don’t have kids doing so quite this much. If we want students who will explore, innovate, challenge ideas, we have to help them see more possibilities than simply answering questions and being told right or wrong.

## Michael P

January 8, 2013 - 6:46 amWhy not pay one of these low-cost workers to grade text, then? That seems like it could be just a slight increase in costs.

## Jane Kise

January 8, 2013 - 7:52 amOne simple filter, Jungian type, tells us that over half of all children aren’t going to be energized by an hour at a computer screen. Extraversion and Introversion in personality type terms involve how we are energized. All of us can do both, but one is preferred and the other is draining. Further, even if the Introverts like the computer lab, they still need the stimulation of discussion, learning to express their ideas and question those of others. Since a good portion of school is still set up for more Introverted activities, adding interventions that require more Introversion makes it a very, very long day for the Extraverts–and they just might start talking and moving when you least want them to…

## Michael Paul Goldenberg

January 8, 2013 - 8:43 am“The learning lab teaches children that math is a solitary activity, wherein one clicks at things on a computer until the computer approves.”

Perhaps not TERRIBLY different from the way many math classes operate, if you simply substitute “teacher” for computer in the second instance so that we have, “Math classes teach children that math is a solitary activity wherein one writes or says the answers to computations until the teacher approves.”

## Dan Meyer

January 8, 2013 - 9:12 amReally helpful commentary from

Jane,Jenny, andMPG, all of which I’ve popped to the top of the post.Michael P:Not a bad idea. But they’re low-skill. Reading a student’s justification and leaning on it in the right ways is a high-skill activity.

MPG:This is a fair point and I want to be on record over and over again: if your math teacher is worse than a computer (or if you have no teacher) go with the computer. But a lot of the rhetoric around digital personalized learning gets really Platonic really fast — like we’ve reached the summit. And I want to make sure we keep the costs of those systems in mind.

## Blaise

January 8, 2013 - 9:32 amHi Nigel,

I would love to read the research that supports your statements above, particularly:

“Students develop strategic thinking skills, develop intrinsic motivation for mathematics, and build the schema needed for number sense.”

Could you direct me to this?

Thanks in advance.

## Michael Paul Goldenberg

January 8, 2013 - 9:53 amThanks for featuring my comment. I was more terse than I intended to be, however. I should have made clear that I completely get your point about the computers and concur, and found the comments several folks made here to be insightful and spot on. It’s just sad that actual humans often don’t operate all that differently from the way the computers do “instruction,” and convey such a similarly dry notion of what can happen in math class. I’m having so much fun working with a group of 4th to 8th graders four days a week for an hour a day in which we actually do math. Because it’s at a private (and progressive) school, I have my hands untied for the first time in my entire educational career, and it’s wonderfully liberating.

Of course, my students DO have to compute, but in the service of figuring things out and making sure that I’m not the only one in the room who CAN compute. :) But that computation is rarely, if ever, the point of what we do; it’s a means to an end. I wish every kid in the country could have such experiences with mathematics, and no computer is going to do that for them, at least not in and of itself.

## Siouxgeonz

January 8, 2013 - 10:43 amLike Nigel, I think it’s a pretty dreadful mistake not to examine the actual software. We’ve been looking at the same stuff — and yes, almost all of it is presentation of math equations with students providing what they think are the answers. It’s practice — and not practice designed to develop concepts.

Saying it’s “one size fits all” because it’s the same software program is making many the grand assumption. If the software provides many options, then that metaphor is like saying clothing is “one size fits all” because it comes from the same store. If there’s a fitting booth that takes an image of my body and finds clothes that will fit it, then it’s not “one size fits all” any more.

I’d love to see discussion of whether the transition from visual puzzles to symbolic math is effective. I suspect that it just might be. People aren’t discussing that, though… they’re deciding whether or not Computers Are Effective. If the computer programs that spit out symbolic equations and ask for answers aren’t effective, it might be because of that delivery, not because it’s Being Done On A Computer.

Finally, I *think* I”m supposed to look at students gazing at a screen and assume they are not engaged. I can be awfully still when I’m engaged in thinking. Is there a better measure?

## Siouxgeonz

January 8, 2013 - 10:47 am… and honestly, do students need to interact with people all the day? I fondly remember doing that thing called reading and learning a lot from it… by my little self.

This post feels like an English writing assignment where the students are assigned to defend an opinion, which is an entirely different task than honestly appraising information and forming an intelligent one.

## Nigel Nisbet

January 8, 2013 - 11:05 amHi Blaise (#24). Thanks for your question. On the subject of research, you may find this study particularly interesting [Spatial Temporal Mathematics at scale] (note this is an interim report part of a multi-year study – still ongoing – funded by the I.E.S. and conducted by UCI) http://www.gse.uci.edu/CRCL/documents/AERA%202010.pdf

Additionally there is a wealth of efficacy data you can find here:

http://mindresearch.net/cont/research/re_ResultsAtScale.php Note the key idea here: these are studies not of small scale implementations in one or two classrooms but at scale across entire school districts.

One of the biggest ideas to get our collective heads around here is that just as all teachers are not “created equal”, neither are all computer programs. Just as Dan’s integration of technology (with his 3act lessons) actually has the potential to increase student motivation, curiosity, and critical thinking, so do other innovative approaches using new tools. One of the major problems I see when looking at most technology is that far to often it is little more than a replication of the dry, ineffective, “tell you everything and ask you to repeat it” teaching practices that sadly I saw in all too many classrooms during my five years as a trainer of math teachers and coaches in LAUSD.

## Brendan

January 8, 2013 - 11:36 amAs a young man I was suckered into taking a lot of jobs selling crap. The one thing all of these jobs had in common was the cheerleading camp. Every meeting started and or ending with chanting, clapping, and chearing. We were expected to learn chants and junk. I usually quit within a few weeks with a bad taste in my mouth. I usually lost money even if I did manage to sell something.

## Max

January 8, 2013 - 11:36 amI was intrigued by Siouxgeonz’s point about how do we assess engagement (and productive engagement) in solo computing tasks. It made me wonder, “what is the role of guessing in learning a mathematical concept?” as well as “what is the role of practice in learning a mathematical concept?” and finally, “what characterizes productive guessing?” and “what characterizes productive practice?”

It seems clear from the article and from the study that David shared (in Comment #2) that guessing is a strategy students use on computer programs, much like they do in math class, and they wait for the computer or teacher to give them feedback about their guesses. Sometimes, that strategy is effective. For example, guessing or estimating is part of Riley Lark’s ActivePrompt and part of the 3 Act structure of problem solving that Dan is developing. It serves as formative assessment, increases engagement, creates a “low floor” task, and more.

On the other hand, guessing is part of the trouble that the famous “Benny” gets into in the study David cited. He guesses and gets feedback about right/wrong that leads him to come up with his own mental model of what leads to right answers… a mental model that breaks down pretty quickly but which he persists with, even in the face of negative feedback.

So what makes guessing productive? One thing might be the kind of feedback students get. I know that ST Math’s visual puzzles provide visual feedback showing how students’ wrong answers fail. Here’s an example to try for yourself: http://mindresearch.net/media/edu/demoFolder/demo/games/index.html

Does feedback that shows why an answer is wrong lead to more robust mental models than feedback that just tells if the answer is right or wrong? How do hints or further questions (without showing the right/wrongness) play into this? That seems like an actionable question, and one that will matter for computer-based learning software and teacher-based learning experiences.

Similar questions could be asked about engagement. What does meaningful engagement look like? Can it be silent? Wordless? When and why does engagement matter? One way to think about engagement is to think about intellectual need, (which Dan refers to in “The Necessity Principle” http://blog.mrmeyer.com/?p=14871 and which comes from research by Fuller, Rabin, and Harel). When do students feel like they *need* to know how to figure something out? And that they need to get good at figuring it out?

One worry I have with gaming is that accomplishing the goal (scoring points, hearing the reward noise, moving Jiji) becomes the focus of the activity, the thing students need to do. The need becomes social or reward based. On the other hand, in order to accomplish the goal, they need to get good at the mathematical task, whether it’s estimating fractions, visualizing rotations in space, or understanding what happens when integers are operated on… And the ST Math program doesn’t prescribe how to get good at it, just poses the task.

I’ve been (slowly) reading the really good book, “Beyond Constructivism” by Lesh and Doerr which gives a theory of math learning that’s about building and improving mental models. One argument they make is that model building has a social component — good models are robust in the face of others’ challenges, and easy to communicate to others. I find it notable that students, when they are making good use of the ST Math games, seem to have a spontaneous need to talk about them (if you watch the first Ted Talk that Nigel linked to, there’s an anecdote of an autistic student who begins talking in order to talk about how he plays these games).

I wonder if one way to make students’ guessing and engagement more productive, especially guessing during and engagement with mathematical puzzle games, is to encourage conversation about what they are doing. When students have to explain how they solved the task, using language, and make another student understand, they are refining their mental model. When they have to decide whose way is better and whose way will always work, they are improving their mental model and perhaps adapting a better, more robust one. It may be that the student talk about the computer game is as important as getting good at the game!

And then, what is the role of the adult in the room at that point? Are they gathering data? Assessing? Guiding? Facilitating? Sugatra Mita’s Hole in the Wall computing experiments suggest that good listening is all it takes to make communication productive — that with a decent task and lots of requirements for students to talk, a patient listener who asks, “how do you know?” and “wow, how did you do that?” can be enough to spur lots of learning!

What if the Learning Centers continued to use math software that posed rich puzzles, but took away the dividers and headphones, made a 3:1 student:computer ratio, and asked their low-skilled workers to help the students take turns, be polite, explain how they know, and tell why they think they’re right. Also equip the teachers and students with flipcams, screen capture software that caught audio too, or cell phone cameras, in which they recorded their conversations “whenever there was a good argument or they thought they were about to figure something out.”

You keep the money-saving features of the lab, need fewer computers, and students are engaged in discourse and model-building, and they get meaningful feedback from peers and the software.

## James McKee

January 8, 2013 - 11:36 amI am distressed by a couple of things, closely related, in this concept. The first was the documentary team’s likening of children to cars. The factory model of education has been dogging us for years, and although we all know we need to change it, we can’t seem to break that model. The public tends to think of children as widgets on an assembly line, that need to have certain knowledge “installed” at certain times.

The other thing that bugs me is the lack of humanities in these schools. For some children, that may be where their passions lie, and where their ultimate success in life may come from, but they’ve been sacrificed on the altar of math and reading scores. It’s the factory again. It’s one more way that we are trying to make all kids come out the same. One of the benefits of Ford’s assembly line was consistency: every Model T came out exactly the same.

My worst fear is that one day in this country we will actually be successful in doing this to our children.

## Brendan

January 8, 2013 - 11:54 amMax (30)

Guessing in the 3 Act has one big difference. It starts with give me an answer that is obviously high/low. So it isn’t a pure guess at all but an estimation. Suddenly, all students have subsequent guesses that are constrained by obvious wrongs.

As to the changes in the computer lab, what if they also included a highly paid trained math teacher to record observations of student discussions? Then followed up with targeted lessons in an actual classroom?

## Mary Dooms

January 8, 2013 - 12:05 pmOn January 3, Diane Ravitch posted an article about the Rocketship to Nowhere http://dianeravitch.net/2013/01/03/rocketship-to-nowhere/

“These are schools for poor children. Not many advantaged parents would want their children in this bare-bones Model-T school. It appears that these children are being trained to work on an assembly line. There is no suggestion that they are challenged to think or question or wonder or create.”

And they call this reform.

## Marshall Thompson

January 8, 2013 - 12:47 pmIn 2013 it’s not that hard to collaborate on a computer. In fact it’s kind of fun and engaging in itself. And useful. Heck, we do it all the time on blogs and twitter. Look no farther than Patrick Honner’s “More Equalateral” problem (http://mrhonner.com/2012/11/12/which-triangle-is-more-equilateral-2012-edition/).

The problem here isn’t the banks of computers, it’s the solitary nature of them. It’s embarrassing that they (slash we) haven’t fostered the kind of creative collaboration that builds appreciation, confidence, and conceptual understanding.

## Dan Meyer

January 8, 2013 - 2:17 pmSiouxgeonz:In the NewsHour video, they refer to students who

aren’tlooking at screens, the ones who are looking blankly off in the distance, as being unengaged. Has anyone in this thread suggested that a student looking at a screen means she’s unengaged?Great stuff from

Max, as usual. One quibble:It takes a lot of skill to ask students to explain how they know what they know and why they they’re right, and then to do anything constructive with those responses.

## Fawn Nguyen

January 8, 2013 - 5:30 pmDan, thank you for the post and this: “It takes a lot of skill to ask students to explain how they know what they know and why they they’re right, and then to do anything constructive with those responses.”

I couldn’t agree more. I believe that this is exactly why it’s so difficult (impossible?) for any software to do what highly effective teachers can do. Not that anyone here suggests one replaces the other, but it’s important to note that the higher salaries afforded to teachers at Rocketship come at a detriment to their kids’ learning in the isolation labs.

## Michael Paul Goldenberg

January 8, 2013 - 7:32 pmI think Dan and Fawn have hit one of the biggest nails squarely on the head.

There’s something going on with all the computer-aided instruction, computer-adaptive testing, and other example of technology abuse coupled with child abuse that so many deformers either miss or don’t care about. I certainly understand wanting to give kids more individuated/individualized help and attention than can generally be given in the typically large (and, particularly in poor districts, rapidly growing) student to teacher ratio classrooms. That’s one reason that class size really does matter (I’m teaching a group of 13 mixed age (4th through 8th graders) this year, and the small class size (maximum is 14) really gives me vastly more sense of each student than I’ve ever been able to get in math classes with 25 or more students). The problem is that the stuff that computers do well isn’t really what students need more of, which is mindless drill (or, as Dan makes clear, the opportunity to avoid the drill by random guessing).

The kind of incisive questioning, hint-giving, and a host of other teacher-moves that matter seem well outside the bounds of what computers can do, and the level of AI that would be necessary for them to approach that kind of thing seems quite far out of reach, if attainable at all. Computers have a hell of a time with facial recognition, from what I’ve read, let alone interpreting the emotional and other signals kids’ faces and body language offer that humans use (if they’re paying attention) to adjust their interactions with other humans. Human teachers do this sort of thing so frequently and so automatically that it may not appear from the outside that they’re doing it at all (and, to be honest, I believe that many teachers do not, which is why THOSE teachers may appear relatively easy to replace with machines).

For my money, however, what we want is not to get more machines in to supplement the work of machine-like humans, but more TEACHERS in who have the skills and motivation to interact like humans with other humans.

So at the risk of annoying people (and with the same caveat that Dan gave – I am most definitely NOT a Luddite), let me suggest that trying to get computers to do at all what, sadly, a lot of humans who are supposed to be teaching children seem ignorant of or indifferent towards is a bad investment of effort, time, and money. Computers are really useful for a lot of things in math class, but teaching isn’t one of them. What we should be investing in, in this regard, is vastly better teacher recruitment, training, professional development, mentoring, etc, including getting teachers to develop skills at evaluating student thinking, student errors, and crafting strategies and methods for taking the information students’ talking, thinking, writing, etc., offers up and feeding back skillfully into the student’s thinking cycle by asking carefully-constructed and carefully-timed questions and offering equally-well-developed hints.

In other words, let’s invest in human resources to accomplish human activities. What a concept!

## Tim Hudson

January 8, 2013 - 11:02 pmI’m the Curriculum Director at DreamBox Learning, and we’ve been partners with Rocketship for the past several years. They use our adaptive math software in their Learning Lab as well. This is a great conversation, and I have 3 points to add to the dialogue.

1. Planning Backward. Like the thousands of other elementary schools who use our math software, Rocketship is working to figure out how to strategically use classroom time as well as technology to improve student learning. All schools should be planning backward that way – looking at the learning outcomes they want for students and then deciding to hire teachers, purchase materials and technologies, and set school schedules that will cause those outcomes (see “Schooling by Design” by Wiggins & McTighe). Rocketship’s commitment to review and revise their structures – especially given their high profile and media scrutiny – is a healthy thing for their organization. The K-12 district I worked in before joining DreamBox also eliminated elementary computer labs a couple of years ago and put the hardware back into the classrooms for similar reasons. But we weren’t on PBS.

Regardless of what one thinks about Rocketship’s model overall, it’s important to note here they do have math classes every day where students engage in conversations and problem solving. They supplement that classroom math instruction with math software in the Lab. The software, like the classroom instruction, is a means toward the same curricular ends. Because students have learning experiences both on- and off-line, there’s no underlying message that math is a solitary activity.

2. Software Quality. I’d agree with Nigel’s point that nearly all math software out there follows the “watch then practice” or “memorize and drill” formats. In these formats, students passively receive explicit instruction and have no opportunity to develop original ideas of their own and on their own. DreamBox isn’t like that. It’s adaptive, but not IPI. I’ve written several pieces explaining how our adaptive learning platform is informed by Freudenthal & Fosnot, not Skinner (one article: http://bit.ly/U0U42R). But you can also see for yourself. We have many K-5 lessons available for anyone to play and experience as if you were a student (3rd-5th grade here: http://bit.ly/NanvgR). I’d encourage you to try them. Click the ‘hint’ button. Get some problems wrong – our teachers write lessons that give students targeted feedback in specific situations. Watch how students use mathematical models and develop mathematical vocabulary. Then decide if you think kids are developing mathematical thinking. We’ve also formatted many of these digital tools into free Interactive Whiteboard tools so that classroom teachers can use them to support rich conversations in class (K-5 tools here: http://bit.ly/5d7yo6). Some of my favorites are Multiplication with the Open Array (a good precursor to partial products and the Distributive Property), Division with Remainders, and Multiplying Fractions.

3. The CCSS Practices. It’s important to note that students in primary grades are still learning to read and write. So text entry isn’t even a possibility, and it’s best for students to construct and critique arguments verbally and with pictures and manipulatives in class. But software can help students acquire the vocabulary, number sense, and mathematical models with which to make their arguments verbally. And I think technology is uniquely suited to strongly support the CC Practice Standard: “Look for and make use of structure.” I’ve written about it here: http://bit.ly/VH7vGk. Even in a small class with a great teacher, it’s challenging to be sure every student is getting better at independently looking for mathematical structures. That’s difficult to assess. Peers and teachers can, and do, help students “make use” of structures; but in doing so they often hinder students from developing the habit of “looking for” structures as mathematicians routinely do.

## William

January 9, 2013 - 6:48 amI had a go at the Multiplication Standard Algorithm. I’m confused about how place value works. My multiplication facts are pretty strong, though, so maybe they’ll carry me through. I’m proud of how well I learned the times tables.

Jennifer (my teacher?) tells me that I’m supposed to “Answer this partial product to help me solve 3122 x 97”. Right. It looks like 2 and 9 are highlighted, so I’ll multiply those together and type it in. I typed in 18 and hit done. Huzzah! It was correct. I guess what I’m supposed to do is multiply the highlighted digits down in the big multiplication problem.

Now 2 and 9 are highlighted again (I guess it’s a different 2? Does that matter?), so I’ll type in 18 again. Oops. Jennifer tells me that I’m multiplying “nine by two tens”. I don’t see any tens anywhere though? There’s a “9 x 20” to the left of the problem. I wonder what that’s for? Maybe I should get a hint.

Jennifer reminds me that “I’m multiplying nine by two tens”. Same thing she told me at the beginning! I should get another hint. “I have no new hints. Remember, you’re multiplying nine by two tens.” Shoot. Still don’t see any tens. I’ll try 18 again.

Oops.

Jennifer tells me that “Twenty times nine is one hundred and eighty.” Where did twenty come from? She never said anything about twenty before!

Well, that one’s done. Now one and nine are highlighted. Not sure what to do. Maybe if I just keep hitting done Jennifer will tell me the answers?

Yep! Success. Ooh – I got a new hint! She told me to make sure I’m multiplying the highlighted digits and something about place value. “1” and “9” are highlighted. I know how to do that. It’s nine. I’ll type that in. Nine and done.

Aw, that wasn’t right either. Now Jennifer’s going to do it for me. I guess it was 900? But the number I’ve got down at the bottom is still 98. Not sure what’s going on with that.

—

Out of character, writing this sort of stuff is *hard*. It’s hard for actual live human beings to understand how students are modeling the math in their head and respond accordingly. Poor Jennifer just repeats her instructions. If I were a student who didn’t understand place value, I might walk away from this unsure about my own multiplication facts, that were good.

Jennifer might help me more if she knew about some common errors (and maybe that sort of thing is going on in the background, invisible to the student?). Like Dan, I don’t want to be a luddite, and if the computer is better than people, we should go for it. But computers have a long way to go.

Much of teaching is empathy – being able to see the world through the eyes of a person who doesn’t know the things you know. It’s being able to communicate with someone who sees the world differently than you do. There are a thousand ways that live, in person communication can cultivate and encourage that empathy in teachers. For programmers who are at arms length, cultivating that empathy is double difficult and important.

Jennifer just asked me if I’d like to continue working, ’cause it took me a while to write this. I think my answer would be “no”?

## Bob Lochel

January 9, 2013 - 3:15 pmWilliam’s comment brilliantly expresses my disappointment with almost all adaptive math systems: the student’s “interaction” is determined by the prescribed plan of software. There is no authentic feedback, only canned responses written by a software developer, which often does not change to fit a student’s learning style. Any educational program which reduces, or ignores, the role of collaboration is destined to let its students down.

Is anyone working on a system where a student, upon logging in, is presented with an appropriate online collaboration with other students? Having students work through a challenge together in an online environment would be exciting to me. The sophistication of the problems presented could somehow be determined by a student’s participation, responses, and persistence. This is where some of the applets Dan throws out now and then, along with newcomers like ActivePrompt, excite me as potential means to allow for real collaboration.

The great myth is that technology alone causes education to become more interactive and engaging. Teaching, done well, has always been interactive and engaging. The great gifts technology has brought us are opportunities for collaboration and connectivity. Sadly, the adaptive systems I have seen miss this mark. When somebody starts thinking realistically about the possibilities of collaboration in education technology, I’ll be ready to buy stock in their product…or a job…that would be cool too…

## Chris S

January 9, 2013 - 4:19 pm@Bob Lochel, you write…

“Is anyone working on a system where a student, upon logging in, is presented with an appropriate online collaboration with other students? Having students work through a challenge together in an online environment would be exciting to me”.

That type of system is called a “joint problem space.” If you google that term, you’ll find some research and products.

## Max

January 9, 2013 - 4:29 pmJust wanted to add, Dan, that I don’t disagree that making use of student thinking is hard. In fact I think it’s hard beyond the level of what individual teachers are called to do, and crying out for more research and the creation of more tools for individual teachers to use — I think math teachers need a lot more support for seeing & fostering emerging mental models and developmental trajectories within students’ creative problem solving work (and a lot fewer curricula and tools that restrict students’ developing mental models).

My goal was to imagine that the school had a need for kids engaging in academic work unsupervised by teachers, and figure out how to make that time involve thoughtful guessing (more constraints and estimation, less randomness, as Brendan points out), real engagement, collaboration, problem-solving, etc. and be useful to the teachers. My best thought so far was: good tasks (check — and the computers do help with that), collaboration (require students to work together, end the headphones), explicit focus on good communication & reflection (train low-skilled workers to prompt for it even if they can’t make use of it), and capturing data other than clicks (using the videos). It still requires skilled teachers to take the time to review the data and figure out how to make use of it, but it might allow kids to do meaningful math and more effective practice and link lab time with class time, without requiring lab time to be actively run by teachers. Is it worth it? I guess it depends on your school’s needs. If it frees the teachers up for lesson study, or reviewing student work, or team planning, it might be.

## Tim Hudson

January 10, 2013 - 12:05 amWilliam – thanks for taking me up on the offer and documenting your experience with one of our lessons. I appreciate that you played from the perspective of a hypothetical student.

The first and most important point I need to make is that a student who doesn’t understand place value would never be presented with the multiplication algorithm lessons in DreamBox. The algorithm lesson you played follows hundreds of prior lessons that develop models and concepts of numbers and operations that are needed before it’s possible to understand the algorithm. When I was a classroom teacher, I knew the important things that happened yesterday, last week, last month and last year that impacted a day’s lesson. So when my principal dropped in to observe for a few minutes, I knew she wouldn’t have that full context. In this instance, your analysis is from a viewpoint similar to my principal’s. That certainly doesn’t diminish your questions or my principal’s questions. I just need to provide more context.

Our teachers at DreamBox have extensive, successful classroom experience. They – not programmers – determine what hints and support are given to students in certain circumstances. Based on what we know students have learned prior to the algorithm lessons, we give hints that may seem thin or minimal. They might not always be perfect, but they do address common errors. There’s also a validity issue with the hints. During a test, when a student comes up to your desk and says, “What does this question mean?” there are only so many things you can say before your guidance invalidates the assessment. You are correct that there are assessment data underlying the DreamBox experience that students don’t see, but do affect what happens next.

Your questions, critique, and analysis were thoughtful. But I feel like we’re talking about different things. I never said computers were better than people, that we consider DreamBox to be empathetic, or that classroom experiences with peers and an empathetic teacher are unnecessary. All I’ve said is that to guarantee high quality learning for all students, we need to design and select learning experiences strategically – including technologies that are grounded in sound pedagogy. An earlier comment in this thread mentioned Tomlinson’s work on differentiation, which I hold in high regard. But her requirements for true differentiation are impossible for any teacher to achieve without technology support. There’s not enough time, data, or energy. DreamBox helps with differentiation.

Having spent 10 years in public schools as a high school math teacher and K-12 math curriculum director, I’ve met and worked with many kids just like your hypothetical student. And I co-authored a chapter in an NCTM book about Intervention (book description: http://bit.ly/UO8Q4t, chapter excerpt pdf: http://bit.ly/VJV9zf) (I get no royalties). So just as you imagined this hypothetical student in a DreamBox lesson, I think it’s valuable to imagine this same student entering a classroom without the support of a technology like DreamBox:

The multiplication standard algorithm is a fifth grade Common Core standard, so let’s assume the student is a fifth grader who doesn’t understand place value. This student transfers into a new school and math class on the day after the teacher introduced the algorithm. Does the teacher know the student doesn’t understand place value? If not, how will that information be acquired? Once it’s known that the student lacks place value understanding, should the teacher continue teaching the algorithm lesson even though the student is clearly not ready for it? If not, what does the student do during math class?

Too often, the student is taught the algorithm right then because there are simply too many logistical and resource constraints that limit what even the best teacher is able to do in that situation. It’s no certainty that the student will meet grade level standards by the end of the year, and the inherent challenges of this reality end up being a huge strain on both teacher and student. I’m empathetic to both of them. And the tens of thousands of others in the same situation. These are the teachers and students we’re trying to help.

## Robert Berkman

January 10, 2013 - 6:17 amSurely there must be research that measures the attention span of a student working on these CAI programs? If not, you would think someone would find a way to measure the actual amount of “thinking” that takes place while a student is “working.” Some students, who are predisposed to staring at a screen, might be getting all the correct answers, but there may not be any actual “thinking” going on. Other students, who are getting answer wrong, might actually get more out of it, because the computer is challenging them. At the same time, there has to be some sort of “burnout” factor that shows that there are diminishing returns when it comes to engagement: perhaps the first 15 minutes are the most productive, and the balance of the 45 minutes shows a constant diminishing of interest and engagement.

Does anybody know of this research?

## Jason Dyer

January 10, 2013 - 8:12 amJohn Danner’s comments don’t sound too promising (“it’s more a matter of getting teachers an interface they like than a data problem.”). Is the complaint from the PBS interview really about “teacher interface” or even “data”?

## Dan Meyer

January 10, 2013 - 2:51 pm@

Robert, Ryan Baker out of WPI has a rubric for “off-task behavior” in educational software. Other people use seat sensors to measure fidgeting. I don’t know much about any of this but those are some crumbs if you want to head to Google.## Christian

January 11, 2013 - 12:47 pm@Max re Feedback. Look into Hattie & Timperley – The Power of Feedback. There is quite a lot of research regarding feedback, also with ICT.

## Susan

January 23, 2013 - 10:29 amIt looks like a warehouse for children. A cubefarm.

## Brendan Murphy

February 1, 2013 - 11:30 amReaders of this post might like to learn of the changes happening at Rocketship and its founder John Danner. I think he might be an interesting person to watch in the time to come. He should even consult with Dan and perhaps Dave Major.

http://www.mercurynews.com/peninsula/ci_22493868/rocketship-education-founder-john-danner-leaves-charter-school