Students Learn as Well from Software as from Live Teachers

http://www.insidehighered.com/news/2012/05/22/report-robots-stack-human-professors-teaching-intro-stats

by mhb • 14 years ago

52 33 14 years ago

teach• 14 years ago

First, the title should be changed. The source article is titled "Score One for the Robo-Tutors" or "Machine Learning: Score One for the Robo-Tutors", not "Students Learn as Well from Software as from Live Teachers".

Secondly, this is an interesting result, but not too surprising. Working through exercises on your own with software that presumably identifies and targets area of weakness ought to put the onus of learning on the student.

On the other hand, many college students sitting in a lecture don't engage the material themselves to the same degree and instead rely on the professor to "teach" them instead attempting to learn it on their own.

In fact, I'm sort-of surprised the machine-learning students didn't do better than the lecture-listening ones.

icegreentea• 14 years ago

A slightly more cynical view might be that the learning outcomes that we're getting is really as good as you can expect given the material + general mix of students + time involved.

Honestly, I would be more interested in seeing the comparison between variation in time/scores for hybrid + traditional formats. My hunch is that the smartest people are significantly dragging up the hybrid scores. My personal experience in class is that professors tend to teach at a rate acceptable to the top ~75% of the class, leaving the top ~5% feeling like class is moving like a glacier. And then there's the fact that each student typically gets stuck at slightly different points, so if a lecturer is to clarify each of these points, there's a LOT of redundancy being thrown around.

tedmiston2• 14 years ago

> many college students sitting in a lecture don't engage the material themselves to the same degree and instead rely on the professor to "teach" them instead attempting to learn it on their own.

On the other hand, material in a medium to large lecture is rarely custom tailored to individual students. If half of the class doesn't understand what slope represents in a supply curve, the experience is ruined for the more gifted students desiring to learn the finer points of economics. Once a lecture regresses into basic concepts you already know, at the loss of interesting finer points, it's quite easy to zone out and lose focus. When the "instructor" to student ratio is 1:1, custom tailoring can really benefit those outside the norm.

Even more, the student is driving the educational vehicle. It's quite empowering to be able to skip over videos about concepts I already know, especially when I can confirm my thoughts with a basic exercise (e.g., a multiple choice question, or short code prompt).

Cushman2• 14 years ago

> First, the title should be changed. The source article is titled "Score One for the Robo-Tutors" or "Machine Learning: Score One for the Robo-Tutors", not "Students Learn as Well from Software as from Live Teachers".

Where does this idea come from that the title should be the same? Granted, the current title isn't quite perfect either -- "Study Reports Software Speeds Up Teaching" would be more accurate -- but it's at least more informative, and certainly no more editorial, than "Score One for the Robo-Tutors".

tokenadult3• 14 years ago

Where does this idea come from that the title should be the same?

It is suggested although not mandated by the Hacker News guidelines:

http://ycombinator.com/newsguidelines.html

"Please don't do things to make titles stand out, like using uppercase or exclamation points, or adding a parenthetical remark saying how great an article is. It's implicit in submitting something that you think it's important."

. . . .

"You can make up a new title if you want, but if you put gratuitous editorial spin on it, the editors may rewrite it."

The retitling in this case seems to have been effective for eliciting more discussion, and I appreciate the various interesting comments here. I regret cases where I submit an article with an original title, only to find out that the same article was submitted hours before with a retitled submission title, one that is missed by Hacker News participants and does not elicit discussion. Most professionally published articles have well written headlines, and it is often hard for HN participants to improve upon those.

yummyfajitas6• 14 years ago

So far that research has failed to persuade many traditional institutions to deploy the software -- ostensibly for fear of shortchanging students and alienating faculty with what is liable to be seen as an attempt to use technology as a smokescreen for draconian personnel cuts... the resistance of many professors to prefabricated courses that they did not invent and cannot modify to their own needs and tastes.

This is a perfect illustration of why non-profit education is broken. There is no principal, so the agents are running the show. (See https://en.wikipedia.org/wiki/Principal-agent_problem if you are unfamiliar.)

In a competitive market dominated by for-profit principals, this wouldn't be an issue. The effective principals would order their agents to cut costs and they would, allowing these principals to dominate the market by cutting costs.

ippisl• 14 years ago

Even in for profit market there's a big problem of adopting technology due to fear of losing revenue(the innovator's dilemma).

And isn't the higher education market in the u.s. open to competition from for-profit universities ? isn't the problem that for profit universities have a bad reputation vs non-profits so employees prefer people who went to non-profit universities ?

yummyfajitas6• 14 years ago

Legally yes, the higher education market is open to for-profit competition. But as you note, reputation effects and history make the education market anything but competitive.

pantaloons2• 14 years ago

The problem isn't a non-profit for-profit one, it's that you can't have a competitive market for what should (ostensibly) be a free and universal service.

Broken it may be, but I certainly prefer free and inefficient education to the alternative.

yummyfajitas6• 14 years ago

...it's that you can't have a competitive market for what should (ostensibly) be a free and universal service.

Why not? How does your normative point (that education should be free) relate to the positive point (that education isn't currently competitive) at all?

Also, I'm curious which service you believe should be free? Education, evaluation or credentialing?

gavinlynch2• 14 years ago

"...ostensibly for fear of [...] alienating faculty with what is liable to be seen as an attempt to use technology as a smokescreen for draconian personnel cuts."

If the product educates students just as well, and at a better price point, why would the cuts be "draconian"?

I would deem it intelligent business and a better use of government resources.

patio11• 14 years ago

Because schools are in the business of employing well-organized constituents and educating students is a not-unwelcome industrial biproduct.

tokenadult3• 14 years ago

Agreeing with patio11's point at this same reply level, as the school system is currently structured, it is only incidentally about providing education to children. The system is mostly about guaranteeing employment to a privileged group of providers, as an economist of education (one of the founders of the study of economics of education) recognized: "The education system is a formalised, bureaucratic organisational structure and, like any bureaucratic organisational structure, it strives for maximum autonomy from external pressures as its cardinal principle of survival. While ostensibly devoted to the education of children, teachers, school administrators and local education officers must nevertheless regard parents acting on behalf of children as a force to be kept at bay because parental pressures in effect threaten the autonomy of the educational system. . . . I would hold that the stupefying conservatism of the educational system and its utter disdain of non-professional opinion is such that nothing less than a radical shake-up of the financing mechanism will do much to promote parental power." -- Mark Blaug, "Education Vouchers--It All Depends on What You Mean," in Economics of Privatization, J. Le Grand & R. Robinson, ed. (1985)

rprospero• 14 years ago

To take a more favorable response, think about how programmers respond to off-shoring.

If you can replace one crappy American programmer with an excellent programmer from Wakanda for less money, that's just good business and I've never seen a good argument against that.

However, every programmer has heard the horror tales of companies firing their best (and highest paid) programmers. These programmers were not replaced with equally skilled Wakandans for less pay, but rather the cheapest code monkeys that Latveria has to offer. It saves a bunch of money this quarter, but causes problems in the long run because you didn't want to pay for quality.

In the same way, I'm suspicious of some of the motives here. I don't doubt for a second that software instructors can improve the learning process. I'm working on my own platform to help my students with some of their math anxiety. This is a good idea and is the path forward.

However, imagine a university with three stats lecturers. The first two have twenty years of teaching experience and are well liked by their students. The third one was just hired in as a temp to cover the excess course load and mumbles through her lectures. With the new software, the administration should get rid of the mumbling temp and now let all the students benefit from the good professors' lectures. On the other hand, the university can save $40k by firing one good lecturer and keeping the mumbler. As a final option, the university could fire both of the experienced lecturers, have an overloaded mumbler give a course that's even worse than the one she was doing before, and save $160k. I'd argue that the first option was progress, the second poor planning, and that the third set was draconian. I also believe that the third choice is exactly the one that the administration will make. That's why I'm nervous about this software.

tedmiston2• 14 years ago

draconian personnel cuts

This is certainly a concern for non-tenured, non-research faculty. The plain vanilla instructor may really have something to worry about.

joe_the_user• 14 years ago

Uh you think?

How well do you think American society and institutions were doing during the hay day of not-for-profit education versus how well do you think it's doing now? Does your criteria include providing for all citizens and having a robust public debate?

It seems more likely to me that the marketizing measures of today have done a lot to break the community building aspect of higher education but that for-profit education hardly promises anything better. Suuure, blame the government for the scams that constitute for-profit institutions now but without the student loan scheme, they wouldn't exist at all.

yummyfajitas6• 14 years ago

...hay day of not-for-profit education versus how well do you think it's doing now?

Could you be more specific about what periods you are referring to? Personally, I would characterize now as the hay day of not-for-profit education - we have far more of it than ever before.

Suuure, blame the government...

Huh? I was blaming principal-agent problems in nonprofits, and suggesting they are less of an issue in for-profits (though it is still an issue even there). What does this have to do with the government?

endlessvoid94• 14 years ago

Students need a guide, not an assistant.

gavinlynch2• 14 years ago

Software can't guide?

excuse-me2• 14 years ago

My university was one of the pioneers in self directed learning.

It was during the black death when there was a shortage of lecturers and many students wanted to go away from the city to learn remotely in their own homes

So they came up with these portable reading tablets, a sort of p-book, which contained the lecturers words and could be read anywhere, even in bright sunlight. Eventually large stores of these were collected together in data repositories- a bit like tape libraries but with human robots.

- they also invented exams when they realised it was cheaper to examine a 100 students at once and outsource the marking than have the professor viva each student individually.

zipdog• 14 years ago

Yes, they should be testing software-based learning against other forms of private (self-directed) learning, as well as (face-to-face) instruction.

Remote-instructor learning through software differs markedly from the same through books in a variety of ways. ie the software can provide more compelling extrinsic motivations, but can also provide increased distractions (both if used on a computer with internet access, and through the affordance of screen-based tools).

So its good not to treat this as an either or problem (does software work as well as face-to-face) because there's a number of other considerations (is the face-to-face personal or lecture hall, is the software highly interactive with immediate feedback, etc

excuse-me2• 14 years ago

Except the software is almost always a Powerpoint-ized version of the book with "an interactive multimedia experience" consisting of either a diagram being drawn piece by piece by an animated gif, or a video clip in the corner with a talking head reading the lecture notes.

zachh• 14 years ago

As interesting as the finding is, it's not exactly surprising that college students learn as well from software as professors. What I want to know is whether K-12 students (especially the lower end of that) learn just as well. My intuition is no, that early childhood education will always require some amount of human educational support, but I'd love to see research confirming or denying as much.

nerdfiles• 14 years ago

If the AI system compiled their Facebook IDs before class (we expected school IDs to Facebook, so why not vice versa? Why shouldn't we expect Facebook and unis to advertise together? -- aren't bars validating age against FB IDs by now anyway?), it could determine an upper limit on analogical connections which might be possibly not included in the set of students corresponding to those IDs. In this way, an AI system can make inferences into successful analogical connections which might similarly function as a growth method for shared knowledge.

As opposed to _generating_ successful anthropological-demographic predictions of linguistic behavior? Shouldn't students be "liking" professors' "asides" anyway?

tokenadult3• 14 years ago

I read this interesting article and then shared with friends on Facebook who are higher education teachers. Thinking about exactly what the study in the submitted article shows, in view of having had a child recently take an introductory statistics course, I think that the result may be rather limited to the course described, an introductory course in statistics for students who will not go on to major in statistics. This result may not generalize to other subjects (although, for reasons suggested by other commenters, it is important to do the research and find out what teaching methods are effective for all subjects).

Most introductory statistics courses are crap, going wrong right at the start by hiring the wrong instructor and specifying the wrong textbook.

http://statland.org/MyPapers/MAAFIXED.PDF

Statistics, the "science of data," is actually an endlessly fascinating subject, from the introductory level on up, but few statistics courses are taught in a manner that requires anything but a robot to present the material, and those statistics courses are so dull that they don't encourage student engagement. In fact, statistics gets into some deep issues about the nature of correct inference right from the beginning,

http://escholarship.org/uc/item/6hb3k0nz#page-1

and it occurs to me that a really, really good statistics course might be a lot harder to automate (although this would be a worthy challenge for artificial intelligence researchers) than the bleak general run of introductory statistics courses.

As a good human statistics teacher, and perhaps even a good automated statistics teacher, would say, further research on this point is needed. One of the best examples I have seen of application of artificial intelligence (software) to teaching of a foundational subject is the ALEKS

http://www.aleks.com/

online K-12 mathematics course, used by a variety of higher education institutions to supplement remedial courses, and used by an increasing number of secondary and elementary schools and self-taught learners of all ages to learn mathematics and some closely related subjects. (ALEKS has a statistics course

http://www.aleks.com/about_aleks/course_products?cmscache=de...

that mostly covers the dull issues rather than the cool issues in statistics.) I like ALEKS a lot for my children learning mathematics and for myself and my wife learning basic principles of sole proprietorship accounting. The ALEKS learning spaces model

http://www.aleks.com/about_aleks/research_behind

is based on some very interesting research, and I have previously recommended that the Khan Academy developers assimilate that research base as they continue developing the Khan Academy courses.

Based on real-world examples, it just might be that automated presentation of many forms of course content will be more efficient (resulting in more learning for fewer dollars in less time) than current in-person models of instruction. And everyone should be happy about that. There will always be a role for in-person teachers, and they do their best work if they use their comparative advantage of being able to do some things better than machines, even if everyone acknowledges that some machines do some kinds of teaching better than most human beings. Let the research continue, until we have an abundance of good data.

cdcox• 14 years ago

This is like saying "People find food made by machines as tasty as human made food" and failing to mention that you only did the test at McDonalds

Looking at the original study:

These were intro stats classes with 100-1000. These classes are already basically a teacher lecturing, minimum engagement, simple material, no ability to have complex homework, and from the looks of things not in the student's major specialization (so super 'in-depth' learning is unnecessary)

Universities have realized for years that they are not very good at these types of classes. They've tried to move these courses online or encourage students to take these at community colleges. But, you don't (or shouldn't) go to university for these classes, you go to the university for the more specialized upper division training in your department. I would be more interested to see if a high level, small, for majors course could be replaced by a similar program. It's worth noting the even the Khan Academy classes are still the 'factory' classes, he has very few 'specialized' classes in more complicated subjects.

georgieporgie• 14 years ago

My attitude toward automated teaching is the same as my attitude toward automated cars: you won't beat the best humans (in the near future), but you'll probably rival or beat the average humans (mediocre training, tired, grumpy, etc.)

I had enough teachers who were barely able to grasp the material themselves, vindictive, or emotionally unfulfilled in their personal lives, that I'd rather have learned from a book, computer, and videos. But hey, maybe dealing with crazy or incompetent educators is an important socialization step for young people...

Students Learn as Well from Software as from Live Teachers

Related Stories