The Earth goes around the Sun-- every day!

June 14, 2012 8:16 AM

Khan Academy and the Effectiveness of Science Videos is an 8 minute video by Derek Muller that offers some skepticism as to the usefulness of science videos that only teach the facts without investigating existing misconceptions. TL;DW? Here's a 1 minute 29 second version. Too brief? Here's his PhD thesis.
posted by gwint (32 comments total)

It is ironic that Muller has decided to teach us about this via a YouTube video. This is academic concern trolling.

I'm extremely skeptical that his experimental data would lead to his conclusions wrt to the Khan Academy. The model of the experiment does not follow the same learning model as Khan Academy. His experiment had a small sample size (20). It did not dive into the motivations of the learners, nor did it include the peer to peer interaction on Khan Academy. It also failed to provide the meaningful reward system (badges) enjoyed by Khan students. Furthermore his materials lacked the peer review and ratings which are found in Khan Academy materials. Thus we are left only with his subjective review of material quality pertaining to their factual content, not the delivery.
posted by humanfont at 8:46 AM on June 14, 2012 [2 favorites]

This is really really interesting and an important discussion to have.

Does the conclusion - students don't learn watching videos because they have misconceptions about the topic at hand - apply to class room learning and other contexts? Because just by watching the video I didn't get why videos (heh) are more passive than other forms of learning.

The idea that confusion and being confronted with problems actually are engines that drive understanding keeps popping up in different fields of communication and philosophy in the form of breakdowns. Heidegger talks about breakdowns and present-at-hand and in IT the pretty influential Understanding Computers and Cognition by Winograd and Flores dedicates lots of space to the subject of breakdowns as vital to language and learning. It's refreshing to see this idea in a very practical context and more plainly explained, however.

Thanks gwint, for posting this.
posted by Foci for Analysis at 8:57 AM on June 14, 2012 [1 favorite]

A while back I found an interesting comparison of Khan vs. the academic literature on how students learn here. Key pull-quote:

Khan (along with most of the general public, in my opinion) has this naive notion that teaching is really just explaining. And that the way to be a better teacher is to improve your explanations. Not so! Teaching is really about creating experiences that allow students to construct meaning.

posted by kiltedtaco at 8:58 AM on June 14, 2012 [12 favorites]

I don't see the irony as much. What he's saying isn't "science videos suck", he's saying that they need to quite obviously correct common misconceptions. I think he applied that theory well to his video, because he talked about the misconception that science videos sans-correction worked, and then proved that idea incorrect. I think it's great that we're analyzing our forms of education in this way! It gives me hope.

As an aside, it's a little terrifying that their confidence levels increased after they watched the video, but remained (significantly) just as wrong. It makes me lose a little faith in our education system... O.O
posted by pcrsweetness at 9:00 AM on June 14, 2012

Oh god I hate to double post, but I just realized the difference between 'passive' and 'active' teaching: passive is explaining, lecturing, and so on. Active is interacting with the students and correcting their misconceptions on an individual basis. This is why the video is not necessarily a great medium of teaching: the teacher should be tailoring the learning experience to the student, because everyone has different misconceptions: a video can't possibly correct every misconception that's out there!
posted by pcrsweetness at 9:06 AM on June 14, 2012 [1 favorite]

The misconception is that education is a good thing and should be improved, but the goals of our leaders (and those selling us products) isn't knowledge, but loyalty. A misconception that's hard to give up is loyalty in action.
posted by Obscure Reference at 9:09 AM on June 14, 2012 [1 favorite]

I think we need to move beyond static videos to more interactive mediums and tools that indeed center around experimentation and taking things apart. Bret Victor has been working on many promising projects that use basic web technology (HTML, JavaScript) to create interactive learning tools.

There are lots of people and projects working on creating better learning tools but so much of the work is fragmented and happens in isolation. Would love to see someone put the puzzle together and create a single, easy to use (easy to confuse?) web platform that could be re-use for any type of learning.
posted by Foci for Analysis at 9:14 AM on June 14, 2012 [5 favorites]

Bret's stuff has exactly the same characteristics as Khan's: too much faith in explanation, not enough in confusion and reconstruction.
posted by migurski at 9:19 AM on June 14, 2012 [1 favorite]

"Students think they know it" is the real kicker here. I can't possibly count how many students I have had tell me that they're going to miss class for a whole week, but "it's OK, because I've taken this class before". You're taking it again why? Oh right, because you failed it the first time. That means you have no idea what's going on.

I am also not surprised by the student who says the thing like "In the video it said the ball is slowly decreasing in force so therefore it stops at one point then comes down". Nevermind that the force is downward and constant, pay attention to this: the ball is decreasing in force. This student doesn't even have the vocabulary to be talking about this subject. This problem is all too common.

Also, the question "did humans live at the same time as dinosaurs" is a loaded one, since some fundamentalists believe this to be the case, regardless of overwhelming evidence to the contrary.

I'm trying my best. I spend a lot of time coming up with engaging and thoughtful ways of tackling mathematical ideas, but too many students lack the ability to pay attention and listen to what I'm saying. When I have students get a question like "Spell l'Hôpital correctly" wrong on a quiz, there is little I can do for them. I did put that on a quiz, yes the word l'Hôpital was written there, yes, they did spell it incorrectly.

People do not like feeling confused. Too many students do not understand that this painful feeling is the beginning of learning. Helping people to realize that and understand that learning is supposed to be painful (at first) is important.

You need to embrace that pain. That means you're doing something right.

I like how the confused guy at the end laughs after he realizes the right answer to the question about the Earth going around the Sun. That's the right reaction! Joy, happiness, that you figured something out! I love seeing that.
posted by King Bee at 9:36 AM on June 14, 2012 [7 favorites]

An important factor here is that Khan's stuff is probably most often accessed as a supplement to homework. So students are generally already in a confused state, trying to construct a solution to a particular problem, when they go to access the Khan explanations. This is when explanation has a chance of actually being worth something. The student's hands are already dirty, and they're invested in figuring out how to concretely get through a process.

I'm currently skimming through the thesis...

Research: Muller's unhappy that there haven't been many very rigorous studies of multimedia in education, but a big part of the reason for that is that it's nigh-impossible to build a well-controlled education study. The majority of studies I've seen suffer from low sample sizes with poor controls; there's way too much going on between classrooms, and the effort involved in putting together a class makes it really hard to justify making numerous slightly different curricula to try out research claims.

It is good to consider, though, that with web-based education systems there's a much greater potential for A/B testing across the population of students, since we get to deliver up content on an individual basis instead of to 30-300 people at a time. Web-based learning tools are thus much more open to good research than human-centric systems.

MULTI-Media: It's important to consider that there are various media in use here. In particular, traditional lecture and instructional labs are themselves media of education!

There are absolutely areas where technological solutions are better than human solutions. Grading drill-like math problems is an obvious example; in the WeBWorK system, students get instant feedback on their solutions and multiple tries to get a correct answer, thus making it possible to learn while focused on a problem. This is in stark contrast to the human-based system of handing in a stack of problems and waiting a week or two to get results; by the time you know something went wrong, you're working on a whole different set of ideas. And indeed, there are studies showing that WeBWorK improves grades, though - as usual - the studies have small sample sizes with poor controls. OTOH, for concept-based or proof problems, webwork simply can't do the kind of grading humans can. So I tend to use a mix of written and web-based problems in my classes; it's not an either-or situation.

Again, Khan works well because it provides access to applications when it matters most: when students are stuck on a problem. There are human-centric ways to get a similar (and more interactive) effect, by using discussion sections as problem sessions, or by setting up 'math labs' where students can work in a big room with on-call tutors. These human-centric solutions are great, too, but a bit more costly, and generally not as on-demand as Khan videos. (Can't go to the math lab at 3am the day before your assignment is due...)

Ultimately, it's best to have some base theory of learning, and examine the available tools in light of that theory. The goal should be to develop an ensemble of tools, human and technological, to best encourage learning, rather than to consider the human/technological divide as a dichotomy. It's not.
posted by kaibutsu at 10:00 AM on June 14, 2012 [3 favorites]

This, to me, is part of the reason that I enjoy the C.G.P. Grey videos so much, that, probably because they are as much argument as education, they spend time dealing with misconceptions and opposing views and deconstructing them before replacing them with something truer/stronger.
posted by Navelgazer at 10:09 AM on June 14, 2012

Great video, thanks. I recently completed a slightly technical MA degree and experienced again the pain of confusion and the pleasure of getting it right... but most of all, I found the thing that made some classes hard was that it was hard to know what I understood and didn't understand. When you are being lectured to or reading a book, a valuable service a lecturer/author can provide is clues to the difficulty of the material, in terms of raw concentration power required to understand something and how intuitive it is. This is part of what

E.g.: "This concept is actually simple because it's just like [intuitive comparison to something you already understand]." or "Pay extra attention here because the math is complex."
posted by ropeladder at 10:17 AM on June 14, 2012 [1 favorite]

yea, video/problem sets aren't the same thing as classroom/lab/field experience and khan would agree -- he's trying to supplement classroom learning, not provide a substitute for it -- so maybe muller will help educators address misconceptions, but it seems like the common denominator here is still for society to invest more in good teachers and/or increase kids' access to them (like finland).

isn't knowledge, but loyalty

or debt bondage...
posted by kliuless at 10:49 AM on June 14, 2012

in the WeBWorK system, students get instant feedback on their solutions and multiple tries to get a correct answer, thus making it possible to learn while focused on a problem.

My experience with WeBWorK and similar online systems is that students begin to fret much more about getting things right in a minimal amount of time than actually focusing on the problem. Just as technology allows for marvelous things like instant feedback, it also allows for efficient reporting, and students are all-too aware of this. Even with infinite tries and assignments that weren't graded, students were reluctant to enter potentially incorrect solutions because they didn't want their attempts to be logged.

Working as a classroom assistant, many times I'd be asked during office hours if a solution "looked right". Even after explaining that the system accounts for common mistakes and will provide extra help if it thinks you need it, students would refuse to type their answer in the box and hit "submit" unless I had personally vetted it.
posted by RonButNotStupid at 11:09 AM on June 14, 2012

Khan (along with most of the general public, in my opinion) has this naive notion that teaching is really just explaining. And that the way to be a better teacher is to improve your explanations. Not so! Teaching is really about creating experiences that allow students to construct meaning.


I think that the writer of that statement is expressing a number prejudices about Khan and reaching conclusions not necessarily supported by empirical data. See also: Learning Theory and an article on Constructivism for a general overview on this subject.
posted by humanfont at 11:15 AM on June 14, 2012 [1 favorite]

Is L'Hôpital the most common hatted "o" the American college student is likely to see? I cannot think of a single other instance offhand. Also I had to cut and paste it because I do not have the faintest idea how to type it.
posted by bukvich at 11:22 AM on June 14, 2012

If they had left off the "hatted o", I wouldn't have counted it wrong. I'm talking people writing shit like "el hospeetal" or some such garbage. I wish I still had the quizzes so that I could tell you exactly what they wrote.
posted by King Bee at 11:26 AM on June 14, 2012

Wow. Those guys are probably level 25 World of Warcraft rulers.
posted by bukvich at 11:39 AM on June 14, 2012 [1 favorite]

I totally agree that for many average teachers, teaching is merely explanation, and this is the bludgeon that destroys passion, creativity and profound understanding.

The real functions of education, in my opinion, are in order of importance
• the transmission of enthusiasm,
• the development of the student's intuition, and
• presenting a foundation.

The problem is that teachers have big classes, little time, or didn't have a good education themselves. All people, especially children, are naturally curious, but this is destroyed in the majority of classrooms that are obsessed with hammering back the student's own way of thinking to make way for the one way the educator understands a thing.

Instead, the first goal of an educator in the course of presenting a foundation of a subject should be to transmit her passion, and second to satisfy the curiosity of the student as he explores the subject himself. This exploration is fraught with many confusing moments, as described in King Bee's comment, but the student pushes through because of his awakened passion and his liberated curiosity.

It is precisely because educators have paid no attention to passion and the student's intuition that student's fail. Unfortunately, after years of bad education it is hard to revive passion and develop atrophied intuition.
posted by esprit de l'escalier at 11:39 AM on June 14, 2012 [5 favorites]

King Bee: "I'm trying my best. I spend a lot of time coming up with engaging and thoughtful ways of tackling mathematical ideas, but too many students lack the ability to pay attention and listen to what I'm saying. When I have students get a question like "Spell l'Hôpital correctly" wrong on a quiz, there is little I can do for them. I did put that on a quiz, yes the word l'Hôpital was written there, yes, they did spell it incorrectly."

Wha? Why were you quzzing them on the spelling of (copy-paste) l'Hôpital?! This is engaging or thoughtful? Also, did you word the question is such a way that they knew you weren't trying to throw them a curve ball? Because, if I saw "Spell l'Hôpital correctly" on a quiz, I'd assume that "l'Hôpital" must not be the right answer.

I mean, there must have been some more context to this question and it must have been presented a heck of a lot more clearly to your students than what you're give us, right?
posted by Reverend John at 12:52 PM on June 14, 2012 [2 favorites]

No this is not an example of something engaging or thoughtful. I wanted to throw them a bone on a question they would definitely get right. I was wrong.

We were discussing l'Hôpital's rule, and how to spell and pronounce the name in class, because when you don't pronounce and spell words correctly, you occasionally get ignored by people who believe you to be less intelligent than you actually are. This is one of those "here is a fact and you must know it" things, not a thoughtful or engaging thing.

My point was that I can't hope to be successful with my thoughtful and engaging questions when they can't even answer the "spell l'Hôpital" question correctly.

Also, did you word the question is such a way that they knew you weren't trying to throw them a curve ball?

The quiz item was, word-for-word: Spell "l'Hôpital" correctly.
posted by King Bee at 1:11 PM on June 14, 2012

Well, not to get off on too much a derail, but, its hard for me to pin the blame on the student for getting that question wrong. I mean I wouldn't judge your skill as a teacher based on that one example, but it doesn't seem like a good question. I think it would just scream "trick!" to a student, leading them to wrack their brain for the "right" answer, and of course reject the one in front of them.

Because the alternative explanation is that they literally can't extract *any* piece of information from a question no matter how simply presented. Does that seriously seem likely? How did they get into calc without that skill?

Or, alternatively, what do you think it says about the expectations your students have developed from their experiences in previous classes that they don't trust you in test situation when you ask them a question like that?
posted by Reverend John at 1:27 PM on June 14, 2012

I think it would just scream "trick!" to a student, leading them to wrack their brain for the "right" answer, and of course reject the one in front of them.

I'm never out to trick anyone.

what do you think it says about the expectations your students have developed from their experiences in previous classes that they don't trust you in test situation when you ask them a question like that?

It tells me they've had horrible teachers who do try to trick them.

I just wanted them to have to write "l'Hôpital" out at least once in their lives, and the only way to cajole students into doing things sometimes is to offer points for them.

How did they get into calc without that skill?

Oh, oh my. This is too much of a derail to get into. I've had calculus students who quite literally did not know how to add 1/4 to 1/3. They'd keep coming up with 2/7, then desperately trying to convince me that "this is how they learned it". The fact that they can't spell a word when it is written in front of them shouldn't have surprised me at all.
posted by King Bee at 1:34 PM on June 14, 2012 [2 favorites]

Let me second King Bee. I too gave calculus students who cannot do arithmetic, even with a calculator.
posted by wittgenstein at 2:00 PM on June 14, 2012 [1 favorite]

bukvich, the other example I can think of is Le Châtelier's Principle* in chemistry. Possibly also some French historical figures, although I can't think of any very well-known ones off the top of my head.

* Which one of my professors inevitably misspelled as "Le Chatlier". I obsessively crossed it out and wrote the correct spelling in the handouts, and then, as a further act of protest, put circumflexes on all of my vowels.
posted by beryllium at 2:20 PM on June 14, 2012

To second and defend King Bee's approach, a college political science professor of mine did that. The first few questions on the final were "spell the word legislature" and "spell the word bureaucracy" and a few other words. Then we were marked down for mispelling those same words on the rest of the test. This followed several mentions throughout the semester that it was important to spell words correctly in order to be properly understood.

I still had to look up bureaucracy in order to write it here though, so does f-all for long term retention.
posted by holyrood at 3:06 PM on June 14, 2012 [1 favorite]

I had a similar experience to the “L’Hôpital” thing in my sophomore biology class in high school, maybe worse. The test question was to draw a single, complete amino acid from memory. Mr. Wong said that same question would be at the end of every single exam through the end of the year until everyone in the class got it right once, and it stayed for the remaining half-dozen exams. I guess someone just really didn’t want to bother to get the dumb thing right.
posted by migurski at 3:16 PM on June 14, 2012 [1 favorite]

Just to continue this derailment, and to second wittgenstein, for a moment...

You'd definitely be surprised about freshman calculus students and their inability to do arithmetic. Regarding calculators, I've actually found that students are, on average, worse at arithmetic if they're allowed calculators.

I've also witnessed students with calculators on tests adding zero to running sums they were calculating (and not in the context of taking an average with a smart calculator that might remember how many terms have been summed).
posted by monolith at 6:33 PM on June 14, 2012

True story, I have entered 10 x 10 into a calculator and happened to be caught doing it by my AP biology teacher who was walking by at the time. It was mortifying/funny.
posted by en forme de poire at 6:41 PM on June 14, 2012

Is there a name for (or further study of) the phenomenon of learning being obstructed by the student's preconceptions, as described in the video?
posted by Terheyden at 8:12 PM on June 14, 2012

King Bee: "How did they get into calc without that skill?

Oh, oh my. This is too much of a derail to get into. I've had calculus students who quite literally did not know how to add 1/4 to 1/3. They'd keep coming up with 2/7, then desperately trying to convince me that "this is how they learned it". The fact that they can't spell a word when it is written in front of them shouldn't have surprised me at all."

Ugh, well its hard to disentangle all of the different problems that different students have, but this example actually argues that they do have the ability to extract information from the problem, contrary to your assertion with the l'hopital spelling question. The student who answered 2/7 quite clearly parsed the two fractions and added their numerators and denominators. Their solution was wrong but their ability to get the information from the question wasn't.

Furthermore, they weren't even all that far off the mark to assert that that was "how they learned it". It seems extremely likely that they're confusing fraction multiplication with fraction addition, and that they remember successfully applying the process of "perform the operation to the numerators to get the numerator of the answer and then perform the operation to the denominators to get the denominator of the answer". They probably just didn't understand that their rule only applied to multiplication, but they weren't wrong to remember that there was a rule like that.

And this gets right back to the topic of this post. The best response to that student would be to demonstrate to the student first, why this is the wrong answer, possibly with some concrete examples, then to present the right procedure that would give the right answer, and finally to explain why the right procedure works for addition (and also to contrast that with the reason that the "other" procedure is right for multiplication but wrong for addition, which I guess would boil down to the fact that division is the inverse operation to multiplication, so basically you could rewrite it all as multiplication, but with addition you'd need a common denominator, comparable chunks, before you could even sensibly talk about addition).

You'd need to understand their misconception and address it. And in this way I think maybe video teaching could eventually surpass all but the most experienced teachers. It's hard for a teacher to consistently diagnose what a student's mistake is right in the moment and figure out the ideal way to address that mistake, but by refining the video, addressing common misconceptions as they're identified in students who have learned from previous versions of the video you might eventually arrive at a near-ideal presentation that most people would understand while seeing their own misconceptions addressed and corrected. Pair it up with exercises so students can do the work and find their own areas of weakness and return to the appropriate video and fix them and you might eventually give students the tools to learn science and math more independently than they do now.

I think this guy makes some good points and I would hope that Khan and others like him would learn from him.
posted by Reverend John at 8:49 PM on June 14, 2012

Reductive thought here, but is this basically - illusory correlation and the difference of hearing/seeing facts vs actual comprehension?
posted by Rocket Surgeon at 10:16 PM on June 14, 2012