I have not done the lesson, but have tinkered around with the dummy dashboard. I have to agree that dashboards are tough to design and have a bunch of ideas that worked for me and a bunch that would work for me.

I really like the idea of displaying a minimal amount of data per student and having automated filters for categories of evidence. The UI is really a clean operational view for teachers on the run. I have not seen anything of this quality yet online.

That being said, I would add a “bank” of variables at the top of the page that teachers from when building their lesson plan the night before. This would allow for a variety of objectives for the lesson. Some additional variables to consider for me would be low estimate, high estimate, variance from curve of best fit, a visual range with the guess as a point (like a number line) and other statistical measures. The types of evidence that are currently used for filtering do not really make sense to me. If other students already tested out large circles, I would use their data and not worry about creating my own. This to me reduces the value of the circle information or not collecting data. I would like to know if students changed away from quadratic, but do not know what to do with the data about students changing in general. I would also like a classroom dashboard to toggle to. This could include range of circle sizes and data points. Whole Class questions could be included to promote a wider span of circle sizes.

Ok, so I would also like a Ferrari. I know many of these things require a lot of work. I love the effective combination of technology and math and see it far to infrequently.

-Eric

Hello, Dan,

I’ve done a fair amount of research on dashboards, and have built a dashboard or three as well.

Most dashboards suck because they make too many assumptions about what people coming to them will want to know (and your point about managing a class of 30 – or 5 classes of 30 – is right on). Understood this way, most dashboards are one way conversations that provide tools for people to filter down.

I’ve come to the perspective that dashboards are really an focused search tool – rather than telling you what you need to know, a good dash will let you ask questions about the information it contains. Some of this is text-based search, and some of this is filtered (based on facets that are either structured into the apps architecture, or inferred from student responses) – but in any case, a good dash will reflect the assumption that success can look different for different students.

I also like the idea of a dash tailored per-problem. Looking at this from an application design perspective, it would be pretty straightforward to allow instructors to set up custom categories on a per-problem basis.

RE: ” They require an understanding of pedagogy” – absolutely. This is a level of nuance largely absent from what passes as Edtech.

RE: “and a willingness to commit developer-hours to material that won’t scale or see significant reuse outside of one lesson.”

Once the use case is defined, it becomes easier to abstract out types of filters that could potentially scale. In your example, you have a set of filters that appear to be based on numerical comparisons, and some that are potentially based on classroom observation. But really, once the pedagogical goals are defined, it becomes possible to trap for the conditions that would allow for better questions to be asked of the dashboard. But if we can define the conditions we want to examine (good initial estimate, unreasonable answer, collecting a certain number of data points) the dash can bend with the needs of the problem, and the code behind the dashboards can become more general.

I agree with the emphasis on the need for much more fine grained customization to context than is generally regarded as necessary or economically feasible.

On SchoolTool we’re stuck at the opposite extreme, since our mission is to make something adaptable to a wide range of school systems, so I know the pain of having to make everything overly generic.

In fact, trying to get to generic solutions as quickly as possible is often a false economy, since then you tend to just make things that don’t work well enough for anyone to really use.

One thing that I’m constantly struck by is the extent to which different disciplines, grade levels, etc. have such different requirements for this kind of thing. You’d need a completely different approach in English class.

In particular, I don’t know how you’d break down something like this: “Evaluate the advantages and disadvantages of using different mediums (e.g., print or digital text, video, multimedia) to present a particular topic or idea.” At least, halfway sanely. And I don’t mean because it is too vague, but because it implies a level of specificity that just isn’t there.

Great post & great job, Dan & Desmos! Keep the good work!

Just thinking out loud, in case I stumble across anything helpful:

It seems like things could be kept streamlined to just the most actionable information. What steps of a teacher’s process can be simplified/helped by surfacing the right aspects of the data?

1) What problems were most common across each class? Actionable: what concepts do I need to revisit with the entire class? (I like looking through the filters, but ultimately this is what I want to know.)

2) What are some interesting things to say when we talk about this activity? “Some students did it this way, some did it this other way, why do you think that was. Everyone estimated low, except for Mary who estimated very close but a little high — do you remember how you came up with your estimate Mary?” (I think browsing through your custom filters is the best way to accomplish this right now, and that’s probably good.)

3) Which students are having major struggles? Actionable: who needs individual attention or tutoring? (Again, I like the ability to browse through everyone, but I especially want an idea of who needs help the most.)

4) What standards are good, bad, or stale? Actionable: I magically have some extra class time (ha!), what should I do with it? What have we not practiced in a long time (or at all)?

I think I want the ability to get that information easily, but I don’t necessarily want a screen that just answers those 4 questions with no other information.

And something that I’m kind of on the fence about: a “generate some small groups” feature. — This sounds cool, but would get used rarely, I think, and would clutter the interface the rest of the time. The idea is for the tool to automatically generate small groups in a way that you specify — either somewhat balanced mixed ability groupings, or groupings of similar ability level. Based on one problem or many problems. Then you could use it for teams, individualized homework or instruction, project groups, etc. I’d want a way to very easily display this on a computer projector and maybe to save a given generation to a file. (This might be entirely outside the scope of what you’re envisioning, though.)

I’m also thinking about how I’m going to use this. Am I going to sit at the computer during planning? Am I going to have it open on a wifi-enabled tablet while I walk around the classroom? Are there times when I want to be able to show this on the computer projector? Am I going to use this as a reference while I write down a rough lesson plan on paper, or am I going to print this out and then use the print out while I type lesson plans on the computer? Thinking about what use cases you prefer might help define what you want to show on dashboard screens.

I’m on the fence about whether I would want to know how my students compare with others. If 90% of nationwide students used big circles, but 90% of my students did NOT make any big circles, is that useful or significant enough to have pointed out to me? It seems interesting, but ultimately I’m always aiming for all students to understand as much as possible, so it might be a red herring.

There’s just so much here, it’s so hard to be succinct. But I’ll try.

I appreciate the shift in data you propose here. I don’t need to see time spent on task: how many quadratic equation questions a student happened to answer from your bubbles, or your idea of “proficiency”. I can see that on a daily basis if I am doing my professional best. Rather, I would like to know how my students repond to complex tasks. What comments they make. The vocabulary they use. And I would like it all in one place.

This is my first year trying some “flipping” in my AP Stats class. Along with short videos, I have a Google form where students respond to quick prompts. It keeps them “honest”, and I get some data about misunderstandings I need to confront if my next class meeting. In my last video, I asked students to describe a data distribution. As a class, we looked at some (anonymous) responses, and critiqued them. Such a valuable activity to look at writing of peers in class! Examining student thoughts via your interface would be an equally valuable discussion.

At the same time, I am reminded of Dylan Wiliam’s work, who proposes real, meaningful feedback as a means of formative assessment. And this is where I think your work is the most helpful. While many adaptive systems propose to help students along the way, they are often mis-interpreted as summative assessments, through their similarities to traditional grading terms and mechanisms. What I would like to see is my ability, as a teacher, to repond to student attempts and responses and provide real feedback.

For example: this year, many of my students are submitting math works on Edmodo. I have found that I am providing much more detailed feedback, simply because the technology makes it easier for me to type and construct helpful responses than before (my handwriting is awful), and that I have an archive of my comments. Migrating student responses into a similar system from your interface would be equally compelling and attractive to me, and helpful in providing real feedback to students.

I don’t believe I have done justice here to all of the work you and the Demsos team have done here. Keep at it…and can I buy some stock in your company please?

@Dan, curious you think what a dashboard would like to assist teachers with implementing SBG. Or do you think no dashboard would work for that?

The Discovery Assessment platform does a decent job of providing a respectable response to your first and third points. While I don’t have a screenshot to offer (although I will have many in about a month), it tries to break student data down into relatively simple color coded indicators of which skills students are deficient at. While it can provide all of the minutia like time spent on problem, most teachers using the data prefer to stick with the simple indicators of which skills students need work on. I haven’t personally used it in a classroom, but the teachers in my district that have all used it enjoy the connections that it makes to resources inside of Discovery Education’s website, and how easy it make it to assign certain tasks to the students.

As this now sounds like an advertisement for Discovery Education, I’ve also had many teachers just take the data and use it with existing interventions (non Discovery-provided) and teaching strategies within small groups in their classrooms.

@Dan, but remember that even with an expert teacher such as yourself using SBG, you were only 47% accurate: https://blog.mrmeyer.com/?p=2877 . I would venture to say that KA is now more accurate than you were. And, as you say at the end of the post you linked to above (https://blog.mrmeyer.com/?p=1558 ), you were putting in 58-hour weeks implementing that. If that’s considered standard operating procedure, SBG will never really take off.

Your point about KA being on their thirteenth-ish attempt to define mastery is well taken, however. Part of the problem was their initial reluctance, for reasons I couldn’t understand, to engage with anyone from the researcher community. I assumed it had to do with a Silicon Valley mindset that they could figure out anything on their own. So they ended up getting all excited about training a machine-learning logistic model, when the rest of us were looking at them and thinking, um, did you just rediscover Item Response Theory? (See the comments on this blog post: http://david-hu.com/2011/11/02/how-khan-academy-is-using-machine-learning-to-assess-student-mastery.html )

But I think they are past that attitude now–they seem to be more aware of the need to read and respond to the literature. I mean, you might not agree with everything they do, but they clearly read your blog, and I think they take suggestions and feedback seriously.

I’d also venture to guess that their model is now pretty good, since lots of modeling methods seem to get close to the theoretical maximum predictive accuracy, according to this article from the recent conference on Educational Data Mining: http://www.educationaldatamining.org/EDM2013/papers/rn_paper_04.pdf ). Check out sections 4 and 5 of that paper for the main results and a summary of the problems in educational data mining that we still don’t know how to solve. I won’t pretend to have read or understood the methods section…

Not about the dashboard specifically, but about the problem. Looking at the results through the dashboard, I noticed that a lot of my students’ reasons for choosing the quadratic model over the others was “because it fit the points.” I wonder, then, if it would be worth including a cubic model, or a generic power model y=ax^n with a slider for the exponent. Then we could really get into why a quadratic would be better than any other model. Because I bet they could make a cubic model fit, depending on the data that comes up.

Given what Mr. Bosma shared from his attempts at using this system, I wonder…

Do yall have the capability to just go ahead and provide students digitally with the follow up prompts you’ve written for teachers to discuss with students in person?

I understand that you’re trying to facilitate teacher-student interactions, but I don’t think giving students the opportunity to consider those extension questions/comments on their own would detract from the potential in-person conversations.

On the other hand, if the teacher does end up with too many fires to put out dealing with tech issues, explaining directions, or whatever… at least the students would have a chance to re-consider looking at circles of a different size (or what have you).