## Video-Based Assessment In Science

August 15th, 2011 by Dan Meyer

I met Greg Schwanbeck at Apple Distinguished Educator sleepaway camp last month. He teaches science. I teach math. We set those differences aside and found a connection. I use multimedia in my *curriculum*. Greg uses video for *assessment* in a way I found compelling.

Let’s say he wants to assess the *impulse-momentum theorem*, which is the theorem that explains why boxers roll *with* punches rather than *against* them. ie. If you double the duration of the impact, you halve its force. (Cut me some slack here, science-buds. Everybody knows I have no idea what I’m talking about.)

He gave me permission to share with you three versions of the same assessment of a student’s understanding of the impulse-momentum theorem. Let me invite you to *assess the assessments* in the comments. List some advantages and disadvantages. Ask yourself, “What is each option really assessing?” Greg will be along shortly to offer his own commentary and to *assess your assessment of the assessments*.

**Option 1**

An 80 kg stuntman jumps off of a platform high in the air and lands on an airbag. The stuntman hits the airbag with an initial velocity of 45 m/s downward. 0.1 s elapses between the moment the stuntman first touches the airbag and the moment the airbag completely deflates and he comes to rest. Assume that the maximum force that the stuntman can experience and survive is 39200 N. Does the stuntman survive the fall?

**Option 2**

A stuntman jumps off of the top of a crane extended high up in the air. Below him is an airbag–a large inflatable cushion that has a thickness of 3 meters. When the stuntman comes into contact with the airbag, the impact deflates the airbag over a period of time, compressing the airbag from 3 meters thick to 0 meters thick while slowing him down to a stop. Explain, making reference to the impulse momentum theorem, why the stuntman is able to survive.

**Option 3**

Explain, making reference to the impulse momentum theorem, why the stuntman is able to survive the jump.

on 15 Aug 2011 at 6:18 am1 Joe DonahueIs the answer to #3 because he didn’t miss the bag?

on 15 Aug 2011 at 6:32 am2 NickI teach elementary science, so the impulse momentum theorem isn’t something have to I deal with- but I’m intrigued with the idea of video-based assessment, so here goes:

Option 1:

This question is a yes/no closed response, so it seems like it would only require a simple application of a formula. I did countless problems like this in undergrad engineering, and know from personal experience that it’s quite easy to apply formulas to questions like this and get correct answers without any understanding of the underlying concept. So this question is assessing computational ability only- not everything we’re looking for as a science teacher.

Option 2 and 3:

Insufficient information is given in both of these to simply plug and play with the equation (I believe) and the questions are asking to explain rather than solve. This *should* provide evidence of actual understanding- however I know that this evidence is often limited by the thoroughness of the student’s response. For example, what if the student doesn’t actually apply the formula of the impulse-momentum theorem in their answer, but only discusses the general idea. Ideally, as a teacher I would like to see that students understand both the concept and application of the formula- so including both types of questions on an assessment would be helpful.

Comparing Options 2 and 3- it’s interesting that the video version introduces the problem details a little more subtly, because the student will be focused on the image of the jumper as well. It doesn’t explicitly explain how the airbag compresses over a period of time, which would be a big hint for students that we’re talking about an integral. Option 2 still reads like a word problem, even though it’s asking for an explanation. So I think students would be more likely to give a mathematical response to Option 2, whereas with Option 3 you may only get an explanation of the general concept.

I’m not sure it’s possible to effectively assess both the computational skill and the conceptual understanding aspect in the same question- so I would personally use two questions (like Option 1 and Option 3) to address both of these areas.

on 15 Aug 2011 at 6:34 am3 NickSince we’re on the topic of science- does anyone know of any good blogs out there on science teaching?? I’ve found plenty of great stuff on math ed thanks to Dan’s links- but not much in the way of science!

on 15 Aug 2011 at 7:47 am4 Melanie BaccaSo one video is used to assess the last question. While in some circles this may be uniques, I don’t really see it as anything out of the ordinary.

Having students create video responses as answers…now that would be phenomenal.

on 15 Aug 2011 at 8:57 am5 AWFOption 1 assesses whether a student can pick out the pieces needed to use in a math formula.

Option 2 assesses whether the student can pick out the pieces needed to write a definition of the theorem.

Option 3 assesses whether students can determine what is happening when a stuntman hits an airbag and then explain the physics of his survival. A thorough student might draw a picture, explain the data needed to plan for the stunt and explain the theorem behind the stuntman’s survival.

on 15 Aug 2011 at 9:15 am6 gasstationwithoutpumpsI have 15 science education blogs on my blogroll (right-hand sidebar of my blog). They mostly have similar pedagogic styles to the math education blogs I read, but they have more focus on inquiry and they use video more directly as data for everyday labs, rather than as inspiring stories the way Dan does.

on 15 Aug 2011 at 11:04 am7 NickThanks gasstationwithoutpumps. Great list of sci ed blogs, I appreciate it!

Melanie: I have had students do their own video assessments in the past- and it’s a great way to get oral assessment data (much better than the traditional oral presentation in front of the class). Beyond having students simply explain something on video though- creating a video might be more effort than it’s worth in terms of assessment value.

AWF: I’m not sure you’re going to see that big of a difference between responses between Options 2 and 3. Only the medium is different- essentially they are asking the same question while providing different details. I do think the visual aspect of the video will place less emphasis on the mathematics of the question, but I’m not sure that Option 3 will elicit “better” responses.

Hopefully the creator Greg has some data to share from using these questions? That would be interesting to digest!

on 15 Aug 2011 at 1:54 pm8 DonI think #1 is a “can you get there from here?” assessment. Good for determining whether someone can grind numbers but not great at assessing whether someone really understands how the numbers relate.

Two’s a little hand-holdey for me. That’s something I’d expect more in a discussion/introduction to the concept than in assessment. If you need that much coddling to get to the core point here – which I’d say is that it’s all about the delta – then the student getting the answer right doesn’t tell me much.

I like three but I wouldn’t give it to a student who didn’t reasonably have an expectation of what sort of answer would pass muster. As written “because it slows him down” would seem to barely quality. “Because it slows him down more slowly than hitting the hard ground” is better.

I think I’d ask for some observations about something you see in the video that support your answer, hoping to hear something about how thick the bag is and I’d be excited to hear something about all the visible vents on the bag that allow it to deflate at a particular speed.

on 15 Aug 2011 at 4:30 pm9 Greg SchwanbeckHello everyone! I’m really enjoying hearing what you think about my use of video in assessment. I thought I’d share with you my take on what each of these options are *really* assessing, and why I think the video option is best.

Option 1: This problem misses the mark. First, it manages to take an exciting thing like a freefall stunt and make it, as we say in Boston, wicked boring. Second, it’s really not testing the students’ conceptual understanding at all–it’s just testing to see if they can properly identify which number represents t, which number represents m, etc., then correctly pick the equation that relates these quantities, and then execute some algebra. In short, unless I’m aiming to test algebra skills and equation recall, I’ve failed. Solving it correctly does not demonstrate physics understanding.

Option 2: This version is much better at probing for conceptual understanding than option 1, but it is still lacking. First, it’s quite wordy. I’m not trying to test students’ reading comprehension skills, but this question is. It’s possible that a student who completely understands the impulse momentum theorem but has poor reading comp skills or poor command of the English language could blow this problem. It’s also likely that a student pressed for time on a test is going to see this huge block of text on the page and just skip it.

Second, it kind of gives the answer away. I have to state in the problem that time elapses as the stuntman collides with the airbag. It really wouldn’t be fair to leave this out–that would be putting to much of an emphasis on a student’s ability to read a story and thoroughly mentally picture the scenario. Again, this isn’t a reading comp test. Because I have to explicitly state the fact that time elapses during the collision, I have to give the students a huge hint that the answer hinges on the role of time. As a result, students with good test taking abilities or problem solving skills could be able to deduce the correct answer, without really having a *true* understanding of the concept.

Option 3: It takes less than a minute to show the video and the students instantly see the whole picture. (I’ll also show it a couple times again throughout the testing period on mute.) The students are prompted to make reference to the impulse-momentum theorem, so they can’t just say something like “’cause he landed on an airbag” and, technically, have answered correctly. But they aren’t given any hints by the video. To answer correctly, they really need to understand that survival is possible because the time of the collision impacting the airbag is much larger than the time the collision would be if the stuntman just hit the parking lot. Correctly answering this question confirms deep understanding of the impulse momentum theorem. A teacher could even drop the “making reference to the impulse-momentum theorem” part if s/he thought it gave away to much, and replace it with something to the effect of “explain in a way that demonstrates your knowledge of physics.”

on 15 Aug 2011 at 4:35 pm10 Greg SchwanbeckMelanie: I’ll be the first to admit that using video for assessment in this way isn’t unique–even Jeopardy has “video” daily doubles. But video’s strength as an assessment tool is not a function of uniqueness. Rather, its strength lies in its potential to quickly and thoroughly probe for conceptual understanding. What I find most attractive about using video for assessment is the simplicity of it–this is something that any teacher can do, even those with a limited technical background or under heavy pressure to hit all the standards in time. Organizing and executing a project in which students create their own video-answers is for the select few teachers who have technical expertise and the flexibility to integrate it into their curriculum. But using video for assessment is for the masses.

Nick: No data to share yet, unfortunately. I’ll be formally collecting data on this sort of thing during the upcoming school year though, and I’ll keep you posted (if you remind me!) I’m right there with you with what you said about needed two questions to assess both conceptual and mathematical understanding. My tests and quizzes have a mix of both. I’d like to see examples of some quality questions that manage to assess both, I’m sure they’re out there.

Joe: You didn’t “[make] reference to the impulse-momentum theorem,” so you would get a zero and a little frowny face next to your answer. Kidding aside, I should mention that the students have seen good, thorough answers to video-based conceptual questions modeled for them before such questions show up on quizzes and tests.

on 15 Aug 2011 at 4:53 pm11 Frank NoscheseHi Greg,

Great video! I have a collection of “real-world” YouTube physics videos I use for similar purposes. They can be used for introduction (your video makes a great hook, as well as assessment), discussion, whiteboard problems, exam problems, etc.

You can find the collection here:

http://sites.google.com/site/winfailphysics/

I wrote a blog post about it here:

http://fnoschese.wordpress.com/2010/07/21/win-fail-physics-an-introduction/

I would love to collaborate to expand the collection.

on 15 Aug 2011 at 5:04 pm12 Larissa PowersI teach middle school level physics and chemistry and I think this is a fantastic approach to problems that students might otherwise find difficult due to the presence of formulae, words, numbers, etc. (yes, I’m serious). The video assessment presents the situation in the same way the student would confront it in the real world and would help them to achieve understanding beyond a basic mathematically-based knowledge of the situation. I find that students understand a lot more about physics than they think they do, mostly based on what they’ve experienced outside of class. It’s often the numbers, equations, and vocabulary that mucks things up for them.

on 15 Aug 2011 at 6:10 pm13 Joshua SchmidtI’m already worried because I have no idea what the impulse momentum theorem is so I had to google it right away. However, I will try to break it down

1) I feel like the assessment here is searching for a student to figure out the formula needed to solve the problem. I don’t have the formula, therefore I can’t solve the problem (hence I think it being weak assessment.) It’s also a yes or no question meaning a student is ENTIRELY CORRECT OR INCORRECT.

2) We are out of essay type concerns but still in formula type solutions. This is the one toughest for me because I have no idea what information is pertinent.

3) Best for people like me who have no idea what they are talking about. Using the video allows me to spit out an answer with some semblance of information even though I’m lost. As a teacher, you have to be strong to give this type of assessment to sift through the useless and get to the pertinent.

Assessment is a weak point for me. I have to think about this whole setup some more.

on 15 Aug 2011 at 7:23 pm14 Megan GoldingOption 1 calls for all that meaty-looking math that generally passes for understanding. It’s traditional — in both a good and bad way. Kids who practice at solving traditional problems can pass traditional tests. Achievement within the system is something my students rarely experience and I don’t take lightly. But wow, a kid good at repeating algorithms sure can convince you he knows some physics.

Option 2 over-explains. Sure, there’s a real application in there but it’s buried in all those numbers. I was bored reading the problem.

Option 2 loses the meaty math, which I think is a shame. I’ve struggled with problems like this because kids often stop at the conceptual answer. How can we lower a framework of “make some reasonable assumptions and give me some kind of numeric answer” onto this problem? Man, that would be the best of all possible worlds.

on 15 Aug 2011 at 7:28 pm15 luke hodgeI had to google “impulse-momentum” as well. I assume we are dealing with the (perhaps unrealistic) constant mass & force version: F dt = m dv

What would you think of the following answers?

1) The force is applied for a long time because of the air bag. The i-m theorem is F dt = m dv, so the stunt person will survive.

2) The stunt person takes some time to stop after they land on the air bag.

3) The force on the stuntman will be relatively small because the stunt person slows down gradually due to the airbag.

4) The i-m theorem is F = m dv/dt, and the airbag increases dt, and therefore decreases F.

5) The i-m theorem is F = m dv/dt, and the airbag increases dt, and therefore decreases F. If the stunt person landed on the sidewalk, they would stop almost instantly – dt would be very small.

6) The airbag reduces the acceleration experienced by the stunt person.

7) A given change in momentum (stunt person coming to a stop) can be accomplished by a large force over a short amount of time (side walk landing) or a smaller force over a longer amount of time (air bag landing).

When we have already discussed similar problems in class (i.e. dt increases so F decreases), I find it very difficult to tell whether answers like some of those above are simply students parroting back explanations without much understanding or students with good understanding that just are not very articulate

on 15 Aug 2011 at 7:46 pm16 Frank Noschese@Megan: I think you can have it both ways (Options 1 and 3) if you simply add “Provide both a conceptual explanation and a back-of-the-envelope calculation. Be sure to describe any approximations and simplifying assumptions you made in your analysis.” What do you think?

The textbook I use in AP Physics C (“Matter and Interactions”) frequently asks students to do these types of analyses. Here’s an example straight from the text:

“Two students who are late for tests are running to classes in opposite directions as fast as they can. They turn a corner, run into each other head-on, and crumple into a heap on the ground. Using physics principles, estimate the force that one student exerts on the other during the collision. You will need to estimate some quantities; give reasons for your choices and provide checks showing that your estimates are physically reasonable.”

In my experience, kids freeze up when they see these types of questions initially. It takes practice for them to feel comfortable making up reasonable values and assumptions.

on 15 Aug 2011 at 9:39 pm17 Annie CGreg & Dan – following this conversation and, now, both your websites makes me feel honored to be a member of your profession. This Language Arts teacher salutes you… and strives to continue asking pedagogical questions like these in my subjects.

on 16 Aug 2011 at 5:49 am18 The VirtuosiI like option 3 the best, but as some have pointed out, it may be necessary to add the world ‘quantitatively’ to the prompt.

on 16 Aug 2011 at 3:16 pm19 Ross CooperGreg,

Excellent job! I love your explanations behind the three potential problems that you provide. Now, I am wondering how I can make this work on the elementary level (fourth grade). For instance, how could I do this for a topic like place value?

Ross

on 16 Aug 2011 at 4:17 pm20 Megan Golding@Frank: Thanks for the tip to encourage “back of the envelope calculations” to show conceptual & mathematical understanding.

@luke: Yes, in many of your “answers” I see students who kind of understand the material parroting back something I said in class. I call this the spaghetti method — throw a bunch of ideas on the test and see what sticks. This is where I favor Frank’s idea to include back-of-the-envelope calculations to apply the math to their understanding of the video.

On the aspect of video-as-assessment, I say “bravo”. But then, the deck was stacked in favor of video, wasn’t it? For the sake of assessment, I need to put this video in each kids’ hands to play and replay as needed. When they need to. Hmm, I wonder what tech I could do this with most economically.

on 17 Aug 2011 at 2:59 am21 Mr. PatrickDan & Greg –

I teach woodshop/science/technology in a school for neurologically-different kids and I’m going to give your three options my decidedly un-scientific sniff test.

Option 1 becomes nearly impossible for students with neurological issues – students with major reading comprehension issues will have trouble matching formulas to numbers, parsing the importance and relationship between those numbers and then just solving the equation. I could probably train my students to answer this type of question correctly – but if I change the pattern, my students’ neurological differences will trip them up, no their understanding of the physic concept.

I have similar issues with Option 2. Various writing disorders, such as dysgraphia, turn smart, articulate scientists into reluctant learners. Again, my assessment method runs into a neurological difference before it runs into a student’s understanding.

Option 3 gives the students a visual prompt to respond to in a low key way – written essay, orally if the time allows & even a physical demonstration. In this coming year, I’ll be asking for a physical demonstration of the principle (the wonderfulness of small class sizes). An example of the impulse-momentum theorem in woodshop: A student trying to hammer a nail into a board which is held in their free hand, not secured to a workbench/ground/etc. It is nearly impossible to hammer a nail into a board that way because your hand moves with the hammer blows, extending the time of impact and lessening the amount of force transferred to the nail head. My middle-schoolers would make that connection.

–Mr. Patrick.

on 17 Aug 2011 at 6:24 am22 Greg SchwanbeckLuke: Here’s how I’d grade your answers, out of 4 points.

1) The force is applied for a long time because of the air bag. The i-m theorem is F dt = m dv, so the stunt person will survive.

1/4. You don’t mention that the long time of impact results in *less* force. Stating the formula without further explanation is insignificant. In general, if you apply a force of a longer period of time you have a greater change on momentum.

2) The stunt person takes some time to stop after they land on the air bag.

0/4. The stunt person would also take some time to stop if they hit the pavement. You need to talk about the differences between these times and what the implications for force are.

3) The force on the stuntman will be relatively small because the stunt person slows down gradually due to the airbag.

2/4. You’re correct, but you’re not making reference to the I-M theorem. Add another sentence to the effect of “The stuntman will experience a large change in momentum as he comes to a stop, a change which according to the I-M theorem is equal to (average) force times time, so the larger t is the smaller F will be.”

4) The i-m theorem is F = m dv/dt, and the airbag increases dt, and therefore decreases F.

4/4. Though this answer is brief, it’s clear you’ve been able to precisely describe the mathematical/conceptual basics of the physics going on in the video.

5) The i-m theorem is F = m dv/dt, and the airbag increases dt, and therefore decreases F. If the stunt person landed on the sidewalk, they would stop almost instantly – dt would be very small.

4/4. Only thing that I would consider adding is “dt would be very small, making F very large.” to the end, but is a 4/4 answer already in my book.

6) The airbag reduces the acceleration experienced by the stunt person.

0/4. This is true, but you need to explain *how* it reduces the acceleration–and explanation that would involve the I-M theorem–and also how lower acceleration translates into reduced potential for injury.

7) A given change in momentum (stunt person coming to a stop) can be accomplished by a large force over a short amount of time (side walk landing) or a smaller force over a longer amount of time (air bag landing).

4/4. It’s fine that you don’t state the I-M theorem as a formula, because you’ve stated it with words. Your additions in parentheses connect the formula to the video, demonstrating understanding.

on 17 Aug 2011 at 6:27 am23 Greg SchwanbeckFrank: Thanks so much for sharing the win/fail physics resource–this looks really useful! You might be interested in the collection of videos I use in my class throughout the year, which can be found here: http://schwanbeck.westwood.wikispaces.net/Videos

Also, I like your example of a conceptual & back-of-the-envelope mathematical question. I’ll have to think about using more of that sort of question in my classes. And you’re right on when you say students need practice/modelling before they feel comfortable with similar “less helpful” questions.

on 17 Aug 2011 at 6:35 am24 Greg SchwanbeckRoss: Here’s an example of a video you might be able to use regarding place value/units: Verizon Math Fail (http://www.youtube.com/watch?v=ANDk0SWzplo)

You could use this video to introduce the topic, by showing it at the beginning of the unit and facilitating a discussion about just where the miscommunication between customer and customer service rep is coming from.

For assessment, you could use this video at the end of the unit, asking students to explain who is in the right–customer or customer service rep–and why, and/or determine how much the customer would think he owes and how much the representative would think he owes if some other given quantity of data was downloaded.

on 17 Aug 2011 at 7:02 am25 morrowmathAt the risk of being the simplistic one, I did not immediately identify time as the variable salient to the jumper’s survival. In other words, while I understand the idea of literally “rolling with the punches,” the result of reduced force due to increased time of contact was not immediately apparent to me. I also don’t think this would be a unique initial response, either in this problem or in other video presentations. Thinking of the CCSS, there is an element of understanding that is necessary before the application of a mathematical or even a scientific structure is possible.

Perhaps this is the difference between an assessment and an initial task…

on 17 Aug 2011 at 9:14 pm26 CCPhysicistGee, I wish I had gotten here earlier.

Assessing them, I’d say each has its role and you have foolishly listed them in order of increasing complexity.

Option 1:

This is a primitive problem, one that I would put in on-line homework to be sure they have done the reading. What makes it the most basic of problems is that it gives you all of the numbers needed to apply Fdt = mdv, and nothing else. No distractors of any kind.

Option 2:

I think this tests the concept, but not conceptual understanding. What I like about this version is that it has an excellent distractor in that the only number given is irrelevant to the impulse-momentum theorem. It dangles bait that suggests work-energy (or even work-impulse) in front of the student. Although you mention time, the student has to CHOOSE between time and distance when analyzing the problem. Force is not mentioned at all, so its importance has to be recognized by the student.

True, it is too wordy, but that can be fixed. Could even have a drawing.

Option 3:

The only difference between Option 2 and Option 3 is that the student has to identify all of the key variables from knowing the theorem, not from the quantities given in the problem itself. (BTW, that could make this version easier because students should start from the theorem rather than the data, so they won’t waste time trying to find some way of using the distance given in Option 2.)

I won’t go so far as to say that it does a better job of testing conceptual understanding, because it tells them which principle to apply. A better question would simply ask why the stuntwoman survives, letting them choose the one that is most relevant. After all, an expert would look at the question and say “this is usually used as an impulse-momentum problem with a known speed, but conservation of energy is the best way to determine the ratio of the height of the crane to the height of the air bag if I know I can land on my back.”

Comment:

No mention of your rubric. Is choosing the height of the crane part of your ideal answer?

Off topic digression:

Firing an automatic weapon is a much cleaner example, and these days I usually have several students (both male and female) in any given class who have extensive experience with fully automatic weapons and can share with the class their experience of average force resulting from a series of impulses.

on 22 Aug 2011 at 1:53 pm27 Telannia NorfarI think by the amount of answers on this post, there should be more discussion of video assessments. Just a thought Dan :).

I think everyone has made some very valuable comments. However, I think there are times we are missing the forrest because we are looking at the trees. For example, discussing how the student can not answer the question correctly is a tree. Any student can give an answer that is not detail enough if you don’t say that is not what you are looking for in an answer. When was the last time you gave an assessment and you didn’t explain what was acceptable?

Basically, I think the forrest is the power of video to remove cultural or reading biases and achieve evidence of conceptual understanding. I teach in an inner city high school where there are 28 countries represented. In addition to this high english as a second language population, most students are two to three grade levels behind in reading. However, reading and cultural bias should not be a factor in determining of the mathematical or science concepts. We should strive for assessments that eliminate these factors. While video removes these filters, it also demands a demonstration of conceptual understanding that we were not able to achieve with a word problem. I don’t know about anyone else, but this makes me EXCITED!

I know Meagan wanted to problem to require the students to show the meaty math but I think it actually does. Too often in math class, we have students complete the steps of math without ever explaining what the answer means. What better way to get to the meat of math then to apply what the answer. Why not take it a step further and have the students explain how a change in certain numbers changes the outcome of the stuntman!

I love this blog!

on 31 Aug 2011 at 6:03 am28 NickI know this thread is dead, but since it inspired me to try video assessment out myself, here’s what I came up with: http://scientificteacher.com/2011/08/31/video-assessment-killed-the-clip-art-star/