101questions: Behind The Scenes

[cross-posted to the 101questions blog]

We’re one week into 101questions and the early feedback has been encouraging. For a certain kind of warped individual (ie. my kind of individual) the experience seems to be, in a word, addicting. It’s also fun to find a non-trivial Swedish contingency jumping aboard. The more effective use we make of visuals, the more we can include learners who speak English as a second language, if they speak it at all.

After a week, 500+ registered users have uploaded 300+ photos and videos which have provoked 10,000+ questions across all users, including a number of unregistered users I haven’t counted. (The analytic component of site administration is right in my wheelhouse, as you can see.) We even have a registered troll, which means we’re halfway to a full-fledged online community.

Here’s a description of where 101questions came from, the problems it tries to solve, and a few notes on where it might go.

Where It Came From

I piloted the idea online in webinars and face-to-face in workshops. I tweaked the constraints and the implementation and arrived at an exercise that teachers found both challenging and fun, which seemed like the right combination. Teachers liked rating photos and videos as perplexing (or not). That same feedback on their own photos and videos helped them improve their eye for perplexity.

I introduced it on Twitter as #anyqs. You’d post a link to a photo or video (hereafter called “the first act”) and ask for questions. That implementation was good for a time, but ultimately very problematic.

Problems 101questions Tries To Solve, In Order Of Importance

Here’s the biggest:

The feedback to your first act is proportional to the quantity of your Twitter followers.

yeah, well that works great with almost 7500 followers. Less well with 4. That arent math teachers.

I have the most followers of anyone who has contributed to the #anyqs tag. I also get the most responses to my photos and videos. That correlation extends all the way down to people with a dozen followers who get very few responses in spite of their work being thoroughly perplexing. That’s a pity.

At 101questions, your first act goes into a huge pile along with mine and both of ours are served up randomly to other users until it gets 100 responses.

People post whatever they want and tag it #anyqs.

I’m talking about full web pages, long, meandering videos, Flash applets, etc. There is a place for all those things, but they all miss the design of the exercise: one photo or one minute of video.

At 101questions, your attempt to upload anything outside of those constraints will get you an invitation to revise and resubmit.

Tweets are fleeting. Perplexity should endure.

We don’t have a record of all the perplexing photos and videos you’ve posted on Twitter. Many of the #anyqs participants likely couldn’t dredge up their own contributions. I’ve saved all of them locally, but that takes a lot of diligence and they’re basically lost to the wind for everybody else. Along those lines, it’s also hard to know if someone has already posted a particular first act.

At 101questions, your contributions are stored in a database and logged in your profile. (Here’s mine.) The application also checks to see if a particular link has already been uploaded and, if so, points you to it. There is a bookmarking feature. You can save first acts for later.

It’s hard to know if you’re bored by my first act or if I just missed you.

I wish there were a “Skip It, I’m Bored” button attached to my #anyqs submissions on Twitter. If responses to my first act are light, I may infer it wasn’t perplexing, but sometimes I wonder if I just queried my followers at the wrong time.

At 101questions, there is a “Skip It, I’m Bored” button.

There isn’t any way to filter for quality.

What was the best photo posted last month? Which people post the best material most consistently? Where can I find their photos and videos? How are we defining “best” anyway? Those questions can’t be answered within our Twitter pilot.

At 101questions, I’ve set up a metric called “perplexity” which amounts to the likelihood your first act will provoke a question. (Technically, it’s the number of questions that have been asked about your first act expressed as a percentage of total skips and questions. 75 means three quarters of everybody who has seen your first act have asked a question about it.)

People post material because it seems vaguely connected to a discipline, not because it provokes a question.

“Interesting” isn’t the same as “perplexing.” “Engaging” is a different animal also. It’s easier to dazzle a student with fireworks than to provoke her to wonder a question. When I’m unperplexed by someone’s #anyqs material on Twitter, I’ll often tweet back, “What question did that photo make you wonder?” In my perfect world, I’d see your own question alongside the first act you uploaded, but only after I submitted my own, so my question is raw and unbiased by yours.

At 101questions, the upload page has fields for a link and a title. Then a blank for your question.

It’s difficult to see other people’s questions about a first act.

If someone tweets a first act I find perplexing, I often want to know if it perplexed other people and, if it did, the questions they asked. That’s difficult on Twitter.

At 101questions, everyone’s questions are logged beneath each first act.

Where This Might Go

Tagging. Searching. Commenting. Top ten lists for “today” and “the last week,” not just “all time.” A mobile application. The ability to submit files from your phone or computer, not just links. Complete mathematical stories, not just the first act. If we’re working on circumference tomorrow, I’d like to go to 101questions, find a list of complete mathematical stories for “circumference” sorted from most perplexing to least, and then download it to my hard drive. Those features will be expensive to develop and sustain. The core feature – getting 100 responses to your first act – will always be free but I may invite you to pay community membership dues for access to the fancier stuff.

Way, Way Behind The Scenes

One of the most annoying features of edu-punditry is how quickly our gurus decide they’ve done absolute everything they can to help us understand and accomplish their vision for learning. They write their blogs, publish their books, tweet their tweets, and give their speeches. Having decided they’ve done everything possible to help us wrap our brains around ideas that are obvious to them, their last recourse is to snark, sarcasm, hectoring, and irrelevancy.

In reality, their messages can almost always be clarified, made easier, more fun, and less expensive. I want nothing to do with that culture of punditry. I can be clearer. I can find new metaphors. I can publish in more media. And I can create tools to make these practices easier. That’s 101questions.

I'm Dan and this is my blog. I'm a former high school math teacher and current head of teaching at Desmos. He / him. More here.


  1. Feedback:

    I love that it’s so simple. At the same time, I’ve got a feature request. Sorry!

    I’d look at the videos and often, I could tell what the question was meant to be but I could also see how the video could have been posed better. There was nowhere (obvious) for me to give that feedback to the creator.

    For example, I saw a video with someone filling a glass tube with scoops of sand. The obvious questions are ‘how many scoops?’ and ‘how long to fill it?’ – but I had no reason to care. I wanted to know why they were filling that tube up: are they making a sandcastle? Do they want to know how long it takes to make each turret?

    Anyway, I’d love a way to give feedback to the creator aside from just the question. Of course it might not be in line with your vision for the project, but I thought I’d throw it out there.

  2. I’d also like to see a way to rate each question (just click the appropriate level of stars). There are some stimulants that were pretty boring or mediocre that I wrote a question to because the question came just as easily to me as clicking “I’m bored.” Or I wasn’t truly bored but wasn’t perplexed either. If you had a ranking system than you could more easily show “Top 10 lists” etc. or when people search for say circumference it could be ranked on this rating system rather than your current one where I might have missed an amazing opportunity but the people who saw it would rank it very highly and perhaps save that stimulant. A skip could be the same as zero stars even.

  3. Thanks for the feedback, everybody.

    @Alex, one of my next orders of business is to add a comments field to the end of each first act. Hopefully that will be a sufficient medium for users to give and receive feedback.

    @Mike, in workshops I’d ask teachers to rate their own questions on a scale from one to four. Four was “I’ve gotta know the answer to my question.” One was, “I’m kinda interested.”

    Taking the median of those ratings resulted in a similar ranking as calculating the perplexity score off a binary perplexed / bored score. There just wasn’t much difference. Not enough to justify the extra interaction of clicking the star rating, anyway, which may not seem like much, but we’re already up to 10,000 questions.

    I welcome any and all feedback but I’ve said it somewhere else and I’ll say it here: I’d rather hack off my middle finger than add another user action to the homepage. Not to say that might not be necessary at some point. I just like the simplicity of the thing.

  4. How might you deal with first acts that obtain high perplexity scores, but where there’s no strong consensus towards a single question? I don’t know how likely those will be to crop up but it seems possible. That kind of first act might either need revision or be just perfect, depending on how you want to use it, but it definitely seems like it should be in a different category from a first act that strongly prompts a single question in most viewers.

    One possibility that occurs to me is giving the contributor a way to tag questions within a single post, merging the ones that they consider to be hitting on the same idea, after all 100 questions come in. Then you could get statistics and rankings on these “curated” posts.

  5. “will always be free but I may invite you to pay community membership dues for access to the fancier stuff.”

    Thanks Dan for acknowledging that quality needs funds to sustain – you have a voice in this online ed tech community and coming from you it has a legitimacy that the textbook publishers struggle to achieve.

    Also thanks for the effort to mainstream the edu-speak.

  6. At times, I have clicked the

    Skip it, I’m bored

    not because I was in fact, bored, but rather because I was in a public place where it would have been rube to play audio files and I was just looking for a picture….. just an observation. Fantastic work!

  7. I love the site…it’s addicting!!

    One thing I think would be a plus is a way to comment on what others have written once you have posed your question yourself. After all, isn’t half the fun of these activities the discussion they start?

  8. Great data Fawn… I would love to try it with high school students sometime..maybe if I can squeeze it as a brain warm-up or something.

    I’ve been thinking this site may just be as useful to teachers and adults who are just interested in math as it is to students, perhaps even more useful. For example I saw one picture that immediately had me going through math in my mind far too advanced for high school students, and yet, I believe there’s value in that. If math is not a living thing for the teacher, something constantly challenging them, how are teachers ever going to inspire that in their students? To me, it’s not even so much as using the anyqs or 3-acts format as seeing math as a living thing useful in the world around us. Passion and enthusiasm is infectious.

  9. Johanna Langill

    March 26, 2012 - 7:15 pm -

    I’d be really curious to see if there’s a correlation between the brevity of the questions and their perplexity score. One of my most immediate, visceral responses was also the shortest. There would probably be quite a bit of “noise” in naming things, but I think if the media is clear and focused the “it” is clear (my short response was “Will it fit?”).

  10. Love this site! I find the things that grab teachers attention and focus like this do the same for our kids. If I have a hard time getting excited about a math problem how much more difficult it’ll be for my students. Tools like this will be how we get students to see the value in mathematics. Thanks for a great idea!

  11. Well said Dan. I think you have hit your targets and have a highly successful site/idea on your hands. I can’t thank you enough for creating 101qs.com. As always, thanks for listening to the constructive feedback!

  12. You’ve got a great way of fiddling with the questions, and reworking them into some sort of strategic solution, Dan. The way you’ve analyzed each of the components of the #anyqs movement, and attempted to infuse it with your influence, yet open it for others to take it where they want is great.

    As for further reflection, I noticed a couple of things popping up as I continue to explore and enjoy the site.

    A growing number of videos use the “bleep” effect to block out a part of important audio in an attempt to make the material more perplexing. While I typically don’t find that to make the question any more perplexing, as it usually just makes me focus on what the “one” answer is that’s being sought, I find it quite humorous:

    “There’s enough plastic here to circle the globe *BLEEP, BLEEP* times and *BLEEP*” – That just makes me chuckle, and the cultural understanding of “bleeping” audio overpowers my curiosity. We’ll just chalk that one up to immaturity.

    What really does concern me is that a lot of the video might fall under certain Creative Commons licenses. While I know that you will certainly be wary of making sure that contributor’s material isn’t going to be abused, I wonder if you’ve given enough thought yet into how to monetize certain aspects of 101qs, without violating any of the “non-commercial” licenses that submissions may be tied to.

  13. @Johanna, I have a list of questions I’m looking forward to digging into once the database gets large enough. (Maybe 12,000 questions is large enough.) For instance, I’m interested in popular question stems. (“How many?” “How far?” etc.) I had wondered about the correlation between video length and perplexity score. I’ll add yours to the list, also.

    @Ben, two things. 1) I won’t be selling people’s videos. I’ll be selling a hosting service for their videos. 2) This isn’t a new problem. When someone uploads a video to YouTube for which they don’t hold the copyright, the DMCA says the burden is on the copyright holder to alert YouTube with a takedown notice. Same goes for 101qs. Anyone can ask me to take down any first act for which they can prove they hold copyright. I don’t need to worry about it until they do.

  14. @monika, I would love to get kids on the site, asking questions and offering first acts of their own. I am totally unequipped to deal with the legalities of that kind of policy so in the short term I’ve excluded them in the terms of use. That will be a temporary measure until I can figure out the liabilities.

  15. “Interesting” isn’t the same as “perplexing.”

    That’s right. Interesting has a positive spin and is the first step into delving into the act of problem solving. Frankly, I don’t like being perplexed because I usually associate it with frustration as in this definition:

    1. (of something complicated or unaccountable) Cause (someone) to feel completely baffled.
    2. Complicate or confuse (a matter).

    I understand what you after with this, but I don’t think you are going to get a lot of takers for this outside of your current audience and without your personal explanation. Maybe another less perplexing word for this?

  16. 1. I love the site! I have found it addicting as well as an incredibly useful way to think about my own “act ones” that I have used in the past.

    2. An observation: I shared the site with my wife who is not a math educator. She as well was intrigued by what might come up next as she went from image to image. However as I was silently looking over her shoulder I noticed something. There were some images/videos that she was presented with that I also had responded to. Many of them very quickly invoked a question for me as my mind immediately started thinking about bounds and estimates, but I noticed that those questions did not come to her. For many of them she either clicked the bored button or had a question that was worlds apart from mine.
    I’m now wondering when I look at the perplexity whether they are perplexing to math teachers or a representative sample of the public which my math classes are more likely to be. Data on who (Math intesive profession, Other) responds to the images might make the difference when trying to predict success and flop in the classroom.

    3. I love the site and the value of its content.

  17. The site is a great tool for teachers and students. Looking through 101Questions, helps train my brain to look for questions. Also, I do wonder how “math textbook” programmed am I. Subconsciously, am I only relating to these photos and videos to how I have came across them in a textbook? Some times in the back of my mind I think, “Oh I have seen this type of word problem before.” I tend to spit out the same questions, however I try to resist and think, “What could I ask differently.” With that said, the familiar question are still good. The challenge can be to work from that familiar base and ask new questions. Furthermore, what I have experienced in a textbook a student might have experienced in real-life. This is key, because then a student might relate their real-life experiences to a mathematical event.
    I am trying to further promote the idea of 101qs by posting photos through facebook. Hopefully, they will see a photo I post and then want to go to the 101qs website. http://www.facebook.com/media/set/?set=a.348390148547294.96925.202241209828856&type=3 I hope this okay.
    By they way, where could I post pictures, so that I would have link to post them on 101qs.com?

    Thanks for all of your great work.

  18. A couple of notes:

    – The site looks awesome. Great design. On a big screen. I tried browsing it on my phone, and it choked on the videos. I apologize to those people for the Skips.

    – Some of the pictures are interesting, but my first question isn’t a math question. If I used this in the context of a math classroom, I could expect a bit of that bias, but I think part of the perplexity is that it shouldn’t be forced. So I’m going with non math questions for those prompts I find not boring, but also not math perplexing.

    – Seems the biggest winners are the area/volume/scaling problems. Proportions and unit conversions come next. I like your idea of tagging by standards, but don’t know if that’s enough incentive for people to fill in the holes.

  19. Thanks for the feedback.

    Some of the pictures are interesting, but my first question isn’t a math question. If I used this in the context of a math classroom, I could expect a bit of that bias, but I think part of the perplexity is that it shouldn’t be forced. So I’m going with non math questions for those prompts I find not boring, but also not math perplexing.

    That’s great. The site doesn’t specify “the first mathematical question that comes to mind” or “the first academic question that comes to mind.” (I hip-checked Colin Graham about this early on.) To insist on either of those qualifications would be to admit that the tools of mathematical analysis are only useful for answering questions that arise in math classrooms, when in fact our tools have broad appeal. “How many pennies are there?” is a question that the woman on the street might have. We can help her there.

    Seems the biggest winners are the area/volume/scaling problems. Proportions and unit conversions come next. I like your idea of tagging by standards, but don’t know if that’s enough incentive for people to fill in the holes.

    I think you’ve identified the popular candidates. (Though if I see another photo with a lot of little things in it, I swear …. ) The purpose of tagging won’t be to drive people to fill the gaps between them, though. The design of the site lends itself best to applied math (which means polynomials and rational expressions will be scarce) and even then only applied math questions that arise in response to visual stimuli (which means a lot of statistical questions won’t show up). I’m still trying to figure out the jurisdiction of the site (what it’s good for, when, and for which people) but it may be rather narrow.

  20. Paying a membership due would be a pleasure. Your materials have motivated a nation wide perspective shift. You deserve compensation! Thank you!

  21. I’ve been thinking about the perplexity metric, especially seeing some conversations on twitter about the scores of certain entries. I’m still not sure that the current measure works for a ranking. I feel it’s good for separating the wheat from the chaff, but not more than that.

    Then I thought, what if you could put more than one question for each entry, if you had more than one? I think a picture of video that evokes 10 different questions from some people would be just as perplexing as one that evokes 1 question from most people, if not more so.

    I feel like the current measure isn’t a measure of perplexity, but clarity. I can clearly see what’s going on easily, and so I can ask a question easily, and move on. I’ve definitely had examples where I can think of a question and so I put one in, but I don’t think the image is that great (so I could also have said “I’m bored.) But if something is perplexing, it should be grabbing in a different way.

  22. The challenge for anyone suggesting an alternate quality control metric is to imagine the implementation on the screen. What does the user do? What does the screen look like? How does it affect the user’s experience? In this case, how does the user add more questions, what does that look like, and how does it affect the user’s experience? None of these questions are remotely clear to me. Certainly, the thing isn’t perfect, or even all that nuanced. It’s the best compromise I can identify right now, though. At least until you pass me a sketch of how users add more than one question.

  23. Well, I suppose that depends on how willing you are to add more button, since I know you like the very simple portal. What I imagined was a little text link or button beneath the text box that said something like “Add additional question” and it would, when pressed, create a second text box below the first where you could add a second question. And then you can click as many times as you feel you need to get out all your questions. That way we still have the questions formed before being able to look at what other people say.