Category: tech enthusiasm

Total 114 Posts

Big Online Courses Have a Problem. Here’s How We Tried to Fix It.

The Problem

Here is some personal prejudice: I don’t love online courses.

I love learning in community, even in online communities, but online courses rarely feel like community.

To be clear, by online courses I mean the kind that have been around almost since the start of the internet, the kind that were amplified into the “Future of Education™” in the form of MOOCs, and which continue today in a structure that would be easily recognized by someone defrosted after three decades in cold storage.

These courses are divided into modules. Each module has a resource like a video or a conversation prompt. Students are then told to respond to the resource or prompt in threaded comments. You’re often told to make sure you respond to a couple of other people’s responses. This is community in online courses.

The reality is that your comment falls quickly down a long list as other people comment, a problem that grows in proportion to the number of students in the course. The more people who enroll, the less attention your ideas receive and consequently you’re less interested in contributing your ideas, a negative feedback loop which offers some insight into the question, “Why doesn’t anybody finish these online courses?

I don’t love online courses but maybe that’s just me. Two years ago, the ShadowCon organizers and myself, created four online courses to extend the community and ideas around four 10-minute talks from the NCTM annual conference. We hosted the courses using some of the most popular online course software.

The talks were really good. The assignments were really good. There’s always room for improvement but the facilitators would have had to quit their day jobs to increase the quality even 10%.

And still retention was terrible. 3% of participants finished the fourth week’s assignment who finished the first week’s.

Low retention from Week 1 to Week 4 in the course.

The organizers and I had two hypotheses:

  • The size of the course enrollment inhibited community formation and consequently retention.
  • Teachers had to remember another login and website in order to participate in the course, creating friction that decreased retention.

Our Solution

For the following year’s online conference extensions, we wanted smaller groups and we wanted to go to the people, to whatever software they were already using, rather than make the people come to us.

So we used technology that’s even older than online course software, technology that is woven tightly into every teacher’s daily routine: email.

Teachers signed up for the courses. They signed up in affinity groups – coaches, K-5 teachers, or 6-12 teachers.

The assignments and resources they would have received in a forum posting, they received in an email CC’d to two or three other participants, as well as the instructor. They had their conversation in that small group rather than in a massive forum.

Of course this meant that participants wouldn’t see all their classmates’ responses in the massive forum, including potentially helpful insights.

So the role of the instructors in this work wasn’t to respond to every email but rather to keep an eye out for interesting questions and helpful insights from participants. Then they’d preface the next email assignment with a digest of interesting responses from course participants.

The Results

To be clear, the two trials featured different content, different instructors, different participants, and different grouping strategies. They took place in different years and different calendar months in those years. Both courses were free and about math, but there are plenty of variables that confound a direct comparison of the media.

So consider it merely interesting that average course retention was nearly 5x when the medium was email rather than online course software.

Retention was nearly five times greater in the email course than LMS.

It’s also just interesting, and still not dispositive, that the length of the responses in emails were 2x the length of the responses in the online course software.

Double the word count.

People wrote more and stuck around longer for email than for the online course software. That says nothing about the quality of their responses, just the quantity. It says nothing about the degree to which participants in either medium were building on each other’s ideas rather than simply speaking their own truth into the void.

But it does make me wonder, again, if large online courses are the right medium for creating an accessible community around important ideas in our field, or in any field.

What do you notice about this data? What does it make you wonder?

Featured Comments

Leigh Notaro:

By the way, the Global Math Department has a similar issue with sign-ups versus attendance. Our attendance rate is typically 5%-10% of those who sign up. Of course, we do have the videos and the transcript of the chat. So, we have made it easy for people to participate in their own time. Partipating in PD by watching a video though is never the same thing as collaborating during a live event – virtually or face-to-face. It’s like learning in a flipped classroom. Sure, you can learn something, but you miss out on the richness of the learning that really can only happen in a face-to-face classroom of collaboration.

William Carey:

At our school now, when we try out new parent-teacher communication methods, we center them in e-mail, not our student information system. It’s more personal and more deeply woven into the teachers’ lives. It affords the opportunity for response and conversation in a way that a form-sent e-mail doesn’t.

Cathy Yenca:

At the risk of sounding cliché or boastful about reaching “that one student”, how does one represent a “data point” like this one within that tiny 3%? For me, it became 100% of the reason and reward for all of the work involved. I know, I know, I’m a sappy teacher :-)

Justin Reich is extremely thoughtful about MOOCs and online education and offered an excellent summary of some recent work.

2018 Oct 5. Definitely check out the perspective of Audrey, who was a participant in the email group and said she wouldn’t participate again.

2018 Oct 12. Rivka Kugelman had a much more positive experience in the email course than Audrey, one which seemed to hinge on her sense that her emails were actually getting read. Both she and Audrey speak to the challenge of cultivating community online.

Orchestrate More Productive Mathematics Discussions with Desmos Snapshots

Let me describe a powerful teaching tool we just released and the company values that compelled us to build it.

First, let’s acknowledge that statements of values are often useless. Values are only useful if they help people make hard decisions. Our company values should (a) help educators decide how we’re different from other math edtech companies, (b) help us decide how to spend our limited time in the world. So here is one of our values:

We believe that math class should be social and creative – that students should create mathematics in every form and then share those creations with each other and their teachers.

Many other companies disagree with those values, or at least they spend their limited time in the world acting on different ones. For example, many other companies think it’s sufficient for students to create multiple choice and numerical responses to express their mathematical thinking and to share those responses with a grading algorithm alone.

Our values conflict, and the result is that other companies spend their time optimizing adaptive grading algorithms while we spend our time thinking about ways to provoke mathematical creativity that algorithms can’t grade at all. We may both work in “math edtech” but we are on very different paths, and our path recently led us to a very thorny question:

What should teachers do with all these expressions of mathematical creativity that algorithms can’t grade?

Let’s say we ask students an interesting question about mathematics or we ask them to define a relationship and sketch its graph. That’s good math, but the teacher now has dozens of written answers and sketches that their computers can’t grade.

Other math edtech software offers teachers scarce insight into the ways students think mathematically. We offer teachers abundant insight which is a different kind of problem, and just as serious. We’ve spent months building a solution to this problem of abundance and we likely would have spent years if not for one book:

Mary Kay Stein and Margaret Smith’s Five Practices for Orchestrating Productive Mathematical Discussions.

Smith and Stein describe five teaching practices that promote student learning through summary discussions. Teachers should (1) anticipate ideas students will produce during a task or activity and then (2) monitor student work during class for those ideas and others that weren’t anticipated. Then the teacher should (3) select a subset of those interesting student ideas, (4) sequence the order of their presentation, and then help students (5) connect them.

In our classroom observations of our activities, we noticed teachers struggling to select student ideas because there were so many of them streaming from the students’ heads into the teacher’s dashboard. Sometimes teachers would make a note about an idea they wanted to select later, but when “later” came around, the student had already developed the idea further. So then we saw teachers take screenshots of that idea and paste them into slide software for sequencing. Smith and Stein’s recommendations are already ambitious and our software was not making it easier for teachers to enact them.

So we built “Snapshots.”

If you see interesting ideas at any time during an activity, press the camera icon next to it.

Then go to the “Snapshots” tab.

Sequence the ideas by dragging them into a collection.

Add a comment or a question to help students connect their classmates’ ideas to the main ideas of the lesson.

Then press “Present.”

We tested the tool ourselves during a summer school session in Berkeley, CA, and also with teachers around the country. What we’ve noticed is that students pay much more attention to discussions when the discussion isn’t about a page from the textbook or a worked example from the teacher but about ideas from the students themselves.

It’s the difference between “Let me tell you about a really useful strategy for multiplying two-digit number” and “Let me show you some useful strategies from around the class for multiplying two-digit numbers. They’re all correct. Decide which seems like less work to you.”

Here are some of our other favorite uses from the last month of testing.

Match the diagram to the expression.

Which of these answers are equivalent? How do you know?

Values help us all decide how to spend our limited time in the world, and nobody feels those limits quite like classroom teachers. Teachers frequently, and with good cause, evaluate new ideas and innovations by asking, “Does my class have time for this? What will we have to skip if we do this?”

Your decision to spend your limited class time talking about your ideas, your textbook’s ideas, or your students’ ideas is a loud expression of your values. Students hear it. We hope your students hear how much you value their mathematical creativity, explicitly in your words and implicitly in how you spend your time. You bring those values. We’ll keep working on tools to help you live them out in your classroom every day.

The Desmos Teaching Faculty Is Hiring!

2018 Jun 16: We will close this particular posting Saturday, June 23, 11:59PM Pacific.

My team at Desmos is hiring!

You should share that link with anyone who might be a good fit for the work. Alternately, if you think you’re a good fit for the work, you should guard that posting with your life, share it with nobody, and start thinking about your cover letter.

Why you should apply is really simple:

Desmos is the best place to do great work in math edtech right now and for the foreseeable future.

Here are six reasons I’m pulling out of muscle memory. I’m not even thinking about them. Ask me in ten minutes and I’ll give you six more just as fast.

  • Teachers and students love our work. Check our Twitter feed. Also we just wrapped up a pilot study of 44 teachers using our activities and the results exceeded all of our expectations.
  • Desmos folk are enormously talented in their own areas, curious and humble in all the others. So while my team didn’t come to Desmos having studied the same fields as our software developers and product designers (or vice versa) we’re conversationally fluent in each other’s work and humble about the limitations of that fluency. That disposition results in extremely enjoyable and productive collaboration.
  • Great work-life balance. Startups are notoriously unfriendly to families but all of the full-time folk on my team have a couple of kids or more. Each one will tell you they love Desmos’s flexibility to do their best work at negotiable hours and locations.
  • Everyone at Desmos is really satisfied with how we handle meetings and remote work, according to an internal company survey earlier this month. That’s uncommon.
  • Strong financial position. While other edtech companies take on as much venture capital as they can, mortgaging their ability to make important decisions for themselves, Desmos has worked hard to minimize its reliance on outside investment. The result is that my team has had time and freedom to make decisions, first, based on what works for math students and, second, based on what we can sell. (Example: we decided to invest heavily in making our graphing calculator accessible to vision-impaired students because we thought that reducing an impediment to mathematical thinking sounded like a really good idea. Afterwards, we turned that work into contracts with eighteen states with more on the way.)
  • Glassdoor reviews that speak for themselves.

We’ve spent several years tuning up our model for math, education, and technology. We studied it over the last three months and various aspects have clicked right into place. Demand is heating up for that work so we’re looking for people to help us build.

So please check out the posting and think about applying or sending it to someone you know.

BTW. If the formatting of the job posting seems atypical, it’s because we spent a lot of time discussing Lever’s blog series on reducing hiring bias. It would be easy to write a list of required credentials based on our mental profile of an ideal candidate. But that mental profile would be extremely susceptible to implicit and explicit biases. Lever received more responses from a more diverse group of candidates when they focused less on their credentials and more on what they’d need to know for the work and what they’d do at different milestones in their first year.

For example, an earlier draft of our posting required “at least five years of teaching experience at grades 6-12,” which isn’t bad as far as credentials go, but we realized it’s really just a proxy for the first four bullets beneath “What you should show up ready to teach anyone on your first day.”

  • What a day in the life of a public middle- or high-school teacher looks like in the United States.
  • The major challenges of technology integration in US classrooms from the perspective of both students and teachers.
  • What separates a great math lesson from a lousy math lesson.
  • What separates great classroom technology from lousy classroom technology.

We’re grateful to Lever for opening up their hiring practices to the public.

Must Read: Larry Berger’s Confession & Question About Personalized Learning

Larry Berger, CEO of Amplify, offers a fantastic distillation of the promises of digital personalized learning and how they are undone by the reality of learning:

We also don’t have the assessments to place kids with any precision on the map. The existing measures are not high enough resolution to detect the thing that a kid should learn tomorrow. Our current precision would be like Google Maps trying to steer you home tonight using a GPS system that knows only that your location correlates highly with either Maryland or Virginia.

If you’re anywhere adjacent to digital personalized learning – working at an edtech company, teaching in a personalized learning school, in a romantic relationship with anyone in those two categories – you should read this piece.

Berger closes with an excellent question to guide the next generation of personalized learning:

What did your best teachers and coaches do for you—without the benefit of maps, algorithms, or data—to personalize your learning?

My best teachers knew what I knew. They understood what I understood about whatever I was learning in a way that algorithms in 2018 cannot touch. And they used their knowledge not to suggest the next “learning object” in a sequence but to challenge me in whatever I was learning then.

“Okay you think you know this pretty well. Let me ask you this.”

What’s your answer to Berger’s question?

BTW. It’s always the right time to quote Begle’s Second Law:

Mathematics education is much more complicated than you expected even though you expected it to be more complicated than you expected.

Featured Comment


I have come to believe that all learning is personalized not because of what the teacher does but because of what’s happening inside the learner’s brain. Whatever pedagogical choices a teacher makes, it’s the student’s work that causes new neural networks to be created and pre-existing ones to be augmented or strengthened or broken or pruned.

Scott Farrand:

I’ll accept the risk of stating the obvious: my best teachers cared about me, and I felt that. Teaching is an act of love. A teacher who cares about each student is much more likely to, in that instant after a student responds to a question, find the positive value in the response and communicate encouragement to the student, verbally and nonverbally. And students who feel cared for are more likely to have good things going on in their brains, as described by SueH.

Desmos + Two Truths and a Lie

I’m absolute junk in the kitchen but I’m trying to improve. I marvel at the folks who go off recipe, creating delicious dishes by sight and feel. That’s not me right now. But I’m also not content simply to chop vegetables for somebody else.

I love the processes in the middle – like seasoning and sautéing. I can use that process in lots of different recipes, extending it in lots of different ways. It’s the right level of technical challenge for me right now.

In the same way, I’m enamored lately of instructional routines. These routines are sized somewhere between the routine administrative work of taking attendance and the non-routine instructional work of facilitating an investigation or novel problem. Just like seasoning and sautéing, they’re broadly useful techniques, so every minute I spend learning them is a minute very well spent.

For example, Estimation 180 is an instructional routine that helps students develop their number sense in the world. Contemplate then Calculate helps students understand the structure of a pattern before calculating its quantities. Which One Doesn’t Belong helps students understand how to name and argue about the names of mathematical objects.

(Aside: it’s been one of greatest professional pleasures of my life to watch so many of these routines begin and develop online, in our weirdo tweeting and blogging communities, before leaping to more mainstream practice.)

I first encountered the routine “Two Truths and a Lie” in college when new, nervous freshmen would share two truths about themselves and one lie, and other freshmen would try to guess the lie.

Marian Small and Amy Lin adapted that icebreaker into an instructional routine in their book More Good Questions. I heard about it from Jon Orr and yesterday we adapted that routine into our Challenge Creator technology at Desmos.

We invite each student to create their own object – a circle graph design in primary; a parabola in secondary.

We ask the student to write three statements about their object – two that are true, and one that is a lie. They describe why it’s a lie.

Here are three interesting statements from David Petro’s circle graph design. Which is the lie?

  • The shaded part is the same area as the non shaded part.
  • If these were pizzas, there is a way for three people to get the same amount when divided.
  • If you double the image you could make a total of 5 shaded circles.

And three from Sharee Herbert’s interesting parabola. Which is the lie?

  • The axis of symmetry is y=-2.
  • The y-intercept is negative.
  • The roots are real.

Then we put that thinking in a box, tie a bow around it, and slide it into your class gallery.

The teacher encourages the students to use the rest of their time to check out their classmates’ parabolas and circle graphs, separate lies from truth, and see if everybody agrees.

Our experience with Challenge Creator is that the class gets noisy, that students react to one another’s challenges verbally, starting and settling mathematical arguments at will. It’s beautiful.

So feel free to create a class and use these with your own students:

2018 Feb 6. I added eight more Two Truths & a Lie activities on suggestions from y’all!

BTW. Unfortunately, Challenge Creator doesn’t have enough polish for us to release it publicly yet. But I’d be happy to make a few more TTL activities if y’all wanted to propose some in the comments.