Stephanie Simon, reporting for Reuters on inBloom and SXSWedu:
Does Johnny have trouble converting decimals to fractions? The database will have recorded that – and may have recorded as well that he finds textbooks boring, adores animation and plays baseball after school. Personalized learning software can use that data to serve up a tailor-made math lesson, perhaps an animated game that uses baseball statistics to teach decimals.
Three observations:
One, it shouldn’t cost $100 million to figure out that Johnny thinks textbooks are boring.
Two, nowhere in this scenario do we find out why Johnny struggles to convert decimals to fractions. A qualified teacher could resolve that issue in a few minutes with a conversation, a few exercises, and a follow-up assessment. The computer, meanwhile, has a red x where the row labeled “Johnny” intersects the column labeled “Converting Decimals to Fractions.” It struggles to capture conceptual nuance.
Three, “adores” protests a little too much. “Adores” represents the hopes and dreams of the educational technology industry. The purveyors of math educational technology understand that Johnny hates their lecture videos, selected response questions, and behaviorist video games. They hope they can sprinkle some metadata across those experiences – ie. Johnny likes baseball; Johnny adores animation – and transform them.
But our efforts at personalization in math education have led all of our students to the same buffet line. Every station features the same horrible gruel but at its final station you can select your preferred seasoning for that gruel. Paprika, cumin, whatever, it’s yours. It may be the same gruel for Johnny afterwards, but Johnny adores paprika.
Featured Comment:
Enjoyable games/activities in general are difficult to create, especially in any quantity. Learning and teaching are complicated and personal by necessity. The combination is exceptionally difficult. [..] It’s just not realistic for this to happen on any timetable or method I’ve seen proposed.
2013 Mar 11. Michael Feldstein links up this post and wires in a comprehensive “Taxonomy of Adaptive Analytics Strategies.”
Precious moments:
First of all, the sort of surface-level analysis we can get from applying machine learning techniques to the current data we have from digital education system is insufficient to do some of the most important diagnostic work that real human teachers do.
Then there are those systems where you just run machine learning algorithms against a large data set and see what pops up. This is where we see a lot of hocus pocus and promises of fabulous gains without a lot of concrete evidence. (I’m looking at you, Knewton.)
And guess what? Nobody’s been able to prove that any particular theory of learning styles is true. I think black box advocates latch onto video as an example because it’s easy to see which resources are videos. Since doing good learning analytics is hard, we often do easy learning analytics and pretend that they are good instead.
41 Comments
Brendan
March 5, 2013 - 3:08 pm -I was at a similar presentation at ICE last week. I also spent some time in a few session s with Scot McLeod. He suggests we define personalization and individualization of education differently.
Peronalization would be a personal map to a specific outcome such as inBloom suggests.
Individualization would be allowing children to take part in their own education. Or at least to influence it.
I would suggest reading his slides and visiting his site for more information.
http://dangerouslyirrelevant.org/workshops/ice2013
http://learni.st/users/33030/boards/13765-2013-sai-region-5
Curmudgeon
March 5, 2013 - 3:12 pm -Know what’s even more terrifying?
All that Big Data is in the hands of teachers. Sure, some of it is important to my teaching algebra but not all of it. His psychological profile, transcripts, home-life issues … everything is slated to go online and be accessible … accessible at the touch of a button.
Think about what happens if my password is compromised …
Or I intentionally change information …
Or the secretary decides to “fix” her daughter’s grades …
Or a student gets in to “mess with people” …
If one person knows a secret, it’s a secret.
When two people know, it’s no longer a secret.
When every teacher in every school in the district can read and see, then it’s public knowledge.
That’s rarely a good thing.
Mylène
March 5, 2013 - 4:01 pm -The computer “struggles to capture conceptual nuance.” Kind of like Johnny does with fractions.
I can’t decide if this means that teachers need no more insight than struggling students… or if it means we expect students to achieve no more understanding than a computer.
Mary Dooms
March 5, 2013 - 4:29 pm -@curmudgeon
Diane Ravitch addressed the FERPA issue in this post (http://dianeravitch.net/2013/03/03/your-childs-data-now-online/)
One link in her post has already been provided by Dan. I’m glad to see multiple perspectives on this issue: from legal/ethical to edupreneur to best practice.
GregT
March 5, 2013 - 4:54 pm -Ouch. Even setting aside the concerns about privacy and the danger of pigeonholing people into categories (big enough issues in their own right) – if everything is tailored to the individual, how are they ever going to develop new interests?
In other words, to continue with the analogy above, what if I go somewhere that they DON’T serve paprika with their gruel? Am I going to starve? Because I only understand gruel with paprika, not this cumin nonsense and now it’s too hard and just get me my damn paprika already. That’s the way I learned it!
Moreover, wouldn’t you get TIRED of paprika eventually? I like potato chips. But I don’t eat them CONSTANTLY, partly because I’m old enough to know that’s bad for me, but also partly because I’m sure I’d get sick of them after a while and want some variety. Do I have the option for variety? Or am I only ever going to get baseball analogies because I happen to be good at swinging a bat? Yeesh.
Bruce James
March 5, 2013 - 5:15 pm -…and don’t forget that our tailor-made specimen likes baseball, but doesn’t think critically about it before 4pm. Maybe the ump can throw in a few probability questions between pitches.
Michael Garcia
March 5, 2013 - 5:16 pm -I love all of these comments.
Addressing the “conceptual nuances” is the bottom line for educators, not edtech startups. I would love to serve our kids healthy, fresh, and tasty meals that have all the necessary spices included, so they don’t need to add salt or anything else. The meal should be good enough!
Let’s work on finding the right ingredients and methods of cooking that ultimately allow our kids to enjoy the meal upon being served!
Justin Reich
March 5, 2013 - 5:25 pm -@brandon Will Richardson uses the terms personalized learning and personal learning. Similiar distinction: http://blogs.edweek.org/edweek/edtechresearcher/2012/08/separating_the_personal_from_the_personalized.html
Craig
March 5, 2013 - 5:43 pm -In the history of terrible ideas, this database ranks up there with Homer Simpsons “Nuts and Gum”. Seriously, why the need to teacher-proof everything?
Don’t know what to say when standing in front of a class? Use this scripted lesson plan, we’ll tell you what to say!
Don’t want to waste time getting to know your students? Here, a database with all the information you need to know!
Craig
March 5, 2013 - 5:50 pm -I wonder what company is setting up and administering this database, or designing scripted teaching approaches… nice big chunk of profit to slide around the table….
crazedmummy
March 5, 2013 - 5:53 pm -Thanks, GregT. My son was a “dinosaur boy,” and I was encouraged to put him into the Montessori program, where he could do all his learning through the lens of dinosaurs. I said no, he had to be made to learn about other things.
As an adult, he’s not a paleontologist, but he is able to learn things on his own, and enjoys a variety of interests. There’s a reason for school: it’s not to turn out human robots, or thinking machines, or cogs for the wheel of industry. It’s so we can expose people to new ideas, new experiences, new thoughts. A 9 year old inner city kid does not have enough world experience to decide that s/he will never need to speak Chinese, or program a computer, or run a wood saw, just because right now s/he thinks s/he’s going to be a basketball star.
Jeff Layman
March 5, 2013 - 9:54 pm -I’m with GregT. The quickest way to kill this kid’s adoration of animation is to shoehorn it into some decimal-to-fraction conversion. That sort of connection needs to be organic, something computers probably won’t figure out how to do for a while.
Jeremy Bell
March 6, 2013 - 4:44 am -I think you guys are taking this a little out of proportion. Criticizing an idea is one thing, but sounding “ludistic” is another. =)
The one-to-one is severely flawed, but I think it’s a worthy goal to attempt.
Computers can’t intellectually challenge a student? Have you guys played a xbox 360 game lately? I’m pretty sure it’s already been done.
David Patterson
March 6, 2013 - 5:33 am -Make sure the computer keeps up with Johnny’s performance in baseball. Johnny may not want a slew of baseball questions the day after he allowed the winning run to lose the championship.
Chuck
March 6, 2013 - 6:45 am -Here’s a crazy question – how important is it that Johnny can convert fractions to decimals. Is anyone asking if the skills we are set on teaching are still the ones kids still need to learn?
Ian Rae
March 6, 2013 - 10:07 am -Computers should not be trying to model the students. The students should be trying to model the computer. Computers are good at always-available, endlessly-patient demonstration of a concept in several different ways, and providing simple exercises or tests. That’s enough to be quite useful.
I agree with Dan. Let the teachers diagnose conceptual issues.
Sandy
March 6, 2013 - 11:28 am -Parents should be frightened to death about what this data warehouse on their children means to their future. Imagine all the “pre-existing” conditions that could prevent their child from taking an advanced course, or getting into a particular college, or later, getting a job. Data is sold. Any developer who identifies the company purpose as educational will have access to this data.
a different Dave
March 6, 2013 - 12:41 pm -I’m going to take a different direction: if I like baseball, I might actually be interested in using a baseball themed game to learn something.
The problem is that this is shockingly difficult (or at least, we haven’t well established how to do this). Enjoyable games/activities in general are difficult to create, especially in any quantity. Learning and teaching are complicated and personal by necessity. The combination is exceptionally difficult.
Within a relatively limited and well-defined area, Dan has developed a couple incredible ideas — things like the 3 act format and the really cool online class lessons (“draw a perfect square”, etc). Those of us who have watched along the way have seen how difficult it’s been for a gifted and exceptionally focused teacher to put together even this relatively small handful of excellent ideas into somewhat repeatable formats. 101questions has taught us that even with those ideas well defined, there’s plenty of room for “almosts” and missteps.
So now, we’re supposed to believe that a team of people who never participated in these amazing discussions that are undeniably the cutting edge of secondary math education are going to create a complete, multi-course curriculum chock-full of effective activities that are at the intersection of games and learning?
No. It’s just not realistic for this to happen on any timetable or method I’ve seen proposed. If we had twenty-five Dans and ten years and an unshakable method to keep politicians, business-people, and anyone else who isn’t a teacher out of the mix, I’d be all ears.
Peter Price
March 6, 2013 - 9:45 pm -I must be naive. I keep hoping that technology entrepreneurs will look for ways to help teachers do their job better, not replace them.
It seems to me that the attempt to “automate” teaching and thus exclude teachers from the equation has been going on since I started using a filmstrip projector in the 1970s. Every technology seems to be touted as a way to make education “fool-proof”, where the “fool” is, in some quarters, believed to be the teacher.
The capacity of non-educators to believe that a machine could do a better job than a teacher seems limitless. And the desire to keep trying, insatiable.
Craig
March 7, 2013 - 5:55 pm -If you like baseball, it doesn’t mean you like baseball all the time. You might have lost a game the night before, or seen your favorite team lose. A computer can’t tell what your mood is.
A person can.
Scott
March 8, 2013 - 7:30 am -Great quote from Dan Meyer in Minnesota from Spring 2012, “Start with perplexity and end with the standard.” Yes! But how do we track it, over several years, and provide information to intervention teachers for students who remain at ‘perplexed’.
Christian
March 12, 2013 - 5:38 am -Agree most with Jeremy Bell and a different Dave. It is difficult, maybe even impossible under the conditions. But turning “Since doing good learning analytics is hard, we often do easy learning analytics and pretend that they are good instead.” around: should we not try this because it’s so hard to do good analytics. Just as I don’t believe in dumbing down maths because maths supposedly is too difficult for children, I believe we should try.
Kevin H.
March 12, 2013 - 9:19 pm -Whoever figures out how to actually solve the learning analytics problem effectively will make billions of dollars off the technology. Given the potential profit, I think it’s an economic inevitability that, if the technology is possible, it will be developed. It seems to me that if IBM can make Watson win Jeopardy, then effective personalization is also possible. That doesn’t mean the companies out there now have figured it out–they haven’t yet. It also doesn’t mean that schools will be able to replace teachers with computers. It just means that some form of personalized computer-assisted instruction will become a normal component of classes once the technology reaches some tipping point of effectiveness.
Fionna
March 13, 2013 - 3:31 pm -Wow, I am glad I stumbled onto this site… and for so random a reason. Thanks Jeremy Bell for using the term “ludistic”.
If you are all teachers, then I was the student that you really wanted to hate, or even beat within an inch of my life. Never boring though, because I made it my mission to defy any gimmick that tried to maneuver me into the expected or preferred answer. I lived for discussion, and I would debate any point whether I agreed with it or not, just because I was a wretched and contemptuous little punk.
I wish I could say that I have changed, I haven’t, not much anyway. I have support for my arguments now.
I almost made a wrong assumption on your comment, Ian Rae, when you said to let the students model the computers. I thought you were going a little Skinner and Walden Two for a minute… too much time studying psychology for me, then I realized you were more on the George Boole path.
I think technology is one of the best tools for education, if we could just get past the social science aspects of teaching. Let a kid learn for crap sakes… most of them want to learn, and most of them know how they learn… if they knew what the desired end result was, they could find their own way there.
I took a physics course that tested me on my ability to navigate the course website more than my ability to navigate the laws of the universe… really… basic physics for the non science major hasn’t changed enough to warrant the $175 price tag for a textbook that was useless without the registration code for a moronic website that was of very little help for me.
Testing seems to be more about adherence to assigned reading than it is the assimilation and comprehension of information. I only hope that the teachers who read this keep that in mind.
George
March 14, 2013 - 4:58 pm -I’m confused by the response on this thread. I read the article and re-skimmed it to make sure… there wasn’t a single mention about computers replacing teachers. Why is everyone acting so defensive about how computers replacing teachers… of course they can’t. These people are probably naively over optimistic about the influence their data will make in the classroom, but come on, they’re trying to sell it. I’ve been in more than one math class that features only an overhead and feels like a ’80’s time capsule. Although I’m sure there’re districts in Silicon valley where teachers are being threatened by computers, they are outliers. The norm is a classroom in which the most advanced technology must remain, according to strict classroom rules, in students’ pockets and packs.
Kevin H.
March 15, 2013 - 2:37 pm -@Dan Here is a direct quote from the Michael Feldstein post you linked to:
“Carnegie Mellon’s OLI and Carnegie Learning are probably the two most widely known examples of products with inner loop adaptivity. There is decent and growing empirical evidence that these systems work to teach students, but they are time-consuming and therefore expensive to design, require relatively high levels of skill, and we don’t really know how to do them well for knowledge domains that aren’t either highly procedural or highly fact-driven (like writing, for example).”
In other words, adaptive learning can work, and does have a future. But only for organizations that are going to take a LOT of time to do it right. Once they get it right, though, they’ll make a metric crap-ton of money off of it. So in my opinion, it’s only a matter of time. Carnegie Learning, for example, isn’t there yet, but they have improved dramatically in recent years.
Dan Meyer
March 18, 2013 - 6:14 am -You give Feldstein kind of a loose paraphrase, Kevin, adding in the “crap-ton of money” and leaving out the important piece that “we don’t really know how to do [adaptive learning] well for knowledge domains that aren’t either highly procedural or highly fact-driven.”
A stricter interpretation of Feldstein would probably say that, if this can be done across a math curriculum, we have no evidence it can be done for the pieces that aren’t fact-driven or procedural. And that’s a lot of important math.
Kevin Hall
March 18, 2013 - 8:33 am -@Dan–I did not leave out piece about these systems not working for non-procedural skills in my post. And I completely agree that agree that nobody has demonstrated an adaptive system that would be very good at helping students with modeling practice or other non-procedural elements of the curriculum.
My intention was to respond to your critique that a computer can’t figure out what mistake you’re making, because it only checks your final answer. Programs with inner-loop adaptivity do, in fact, check each step of your work. Before too long, I they might even be better than a teacher at helping individual students identify their mistakes and correct them, because as as teacher I can’t even sit with each student for 5 min per day.
FYI for everyone out there, there is a NEW FREE program that incorporates elements of this: https://mathtutor.web.cmu.edu/ . Curious what this community thinks of it. As always, Dan, let me reiterate that I’m a big fan of your work and the work of other commenters on this blog.
Brendan Murphy
March 18, 2013 - 11:14 am -Thanks Kevin I did take some time to write my opnion of you website.
http://www.philosophywithoutahome.com/blog/2013/03/18/review-of-math-tutor/
Kevin H.
March 18, 2013 - 12:07 pm -I did not make the website–I’m a high school teacher. I replied to your comment here:
http://www.philosophywithoutahome.com/blog/2013/03/18/review-of-math-tutor/#comment-870
Don Byrd
July 15, 2013 - 10:53 am -I have only a modest amount of experience as a math teacher; I lasted less than two years — less than one year, if you exclude student teaching — before scurrying back to academic informatics/software research. But I scurried back with a deep interest in math education, and my academic work has always been close to the boundary between engineering and cognitive science. Anyway, I think Kevin H. is way too optimistic about the promise of computer-based individualized instruction. He says “It seems to me that if IBM can make Watson win Jeopardy, then effective personalization is also possible.” Possible, yes, but as Dan says, the computer “struggles to capture conceptual nuance.” Success at Jeopardy simply requires coming up with a series of facts; that’s highly data based and procedural. The distance from winning Jeopardy to “capturing conceptual nuance” is much, much greater than the distance from adding 2 and 2 to winning Jeopardy.
Kevin also says that “before too long, [programs with inner-loop adaptivity] might even be better than a teacher at helping individual students identify their mistakes and correct them, because as as teacher I can’t even sit with each student for 5 min per day.” I’d say it’s likely programs might be better than teachers at that “before too long” only if you think of “identifying a mistake” as telling Joanie that in _this_ step, she didn’t convert a decimal to a fraction correctly. It’ll be a very long time before a computer will be able to say _why_ she made that mistake, and thereby help her correct her thinking.