Online Education's Dirty Secret - Awful Retention

http://rein.pk/online-educations-dirty-secret-awful-retention/

by pkrein • 13 years ago

197 159 13 years ago

kvb• 13 years ago

Thankfully the article is much better than its title. Low retention is not a problem per se - since there's no cost to signing up for (and subsequently dropping) these online courses, it's to be expected that retention won't be great. However, I agree that retention could probably be improved among the marginal subscribers by using some of the techniques covered in the post.

tkahn62• 13 years ago

I usually sign up for a bunch of classes just so I can access the material at a later point when I have time or when I'm interested. I'm pretty sure that in most cases Coursera locks you out of old material unless you were enrolled. I don't follow along with the class.

Synchronous learning, IMO, is an anachronism. We have video recording technology, practically infinite bandwidth, and near speed of light communication. The idea of synchronous learning, where you have five thousand teachers teaching the same Calculus I class every year, is predicated on a world where these technologies don't exist.

As an aside, for this reason I think the collegiate model is going to completely change in the near future as people start realizing how absurd it is. The real value professors provide is having someone to answer questions (and in a lot of cases, professors don't even take questions during lecture). And even that functionality can and is (in my experience) mostly replaced by other students in the class or students who have already taken the class.

Put the material up, and let communities form around it. I don't need the professor to answer my questions, all I need is the relevant forum (in the original sense) to ask.

Retric• 13 years ago

Only terrible calculus teachers/class setups don't respond to student questions. Until you can do the same thing with a video there is going to be a gap. Traditional collage classes are generally based on the idea that ~1/3 of the time is synchronous learning in a classroom with a teacher 2/3 is self study. Have a 3 hour class expect 6+ hours of homework / projects on average. With that said, most classes don't spend a lot of time on questions, but they occur at the right time as in right in the middle of a lecture or right after it.

Presumably online videos could let you cut down on some class time, but even 1 hour after watching a video your going to forget some of your questions and much of the context of other questions you remember.

PS: If you took calculus in a setting of 500 people where you could not ask questions you still got ripped off even if there was a teacher in the room.

tkahn62• 13 years ago

You can do the same with a video. You pause it, get on a relevant IRC channel or internet forum and ask "what does she mean by X at 23:51?" Or you Google the term or process you were confused about and find someone else's explanation (I have personally done this loads of times). Or you turn to the person you're watching it with and ask. Or you call up your study group on skype. Do those two latter things usually happen now? No, but they easily, easily could if this type of learning became mainstream.

Is having a professor there that you can ask immediately in the moment (and interrupt the lecture for the other students who understood the material) worth $1000 per credit hour?

rjtavares• 13 years ago

That's why I prefer Udacity's approach: they have the tradicional weekly classes when they open the course, but after that they're open enrolment. Start and finnish anytime you want.

Unfortunately, they have much less courses available, and they're all more or less computer science related.

michaelt• 13 years ago

  I usually sign up for a bunch of classes just so I can 
  access the material at a later point when I have time or 
  when I'm interested.

How many of those classes have you done the work for?

Personally I find the synchronous model helps me complete the course, as it removes the temptation to put things off.

scott_s• 13 years ago

I think the synchronous model makes it easier to have small communities of students working through the same thing at the same time, helping each other.

khet• 13 years ago

That may be but I personally find it extremely difficult to juggle my everyday working life with an online class. I would much rather download and watch a video lecture on the way to work.

MaysonL• 13 years ago

I've signed up for over a dozen, finished three, learned significant amounts from, and done significant work in, about 5.

summerdown22• 13 years ago

Just thought I'd add a data point in the opposite direction. I've signed up to two Coursera courses and two Udacity ones, and it's the Udacity ones I'm sticking with. I've finished Cryptography already and am 1/3 through learning python in CS1.

There is something dispiriting to me about a synchronous model. Once I can't keep up I tend to drop it as a failure, knowing there's no point in working through something that might be removed at any point. I also think it's the way I respond to the "don't break the chain" apps. Once life gets in the way and I do break the chain, I feel like a failure.

I'm quite happy to motivate myself however, so Udacity's model of giving the student however long they need, works perfectly.

spikels2• 13 years ago

I have taken classes at several top private and public universities and except in a few smaller, more advanced classes few questions were asked during the professor's lectures. And only some of those questions were helpful to the majority of the class. Having discussed this with many other students I believe this is typical, at least in US universities. (I have heard it is even worse in European and Asian universities where interruptions are discouraged. Is this true?)

I believe this is due to the constraints of the tradional class format. It is simply impossible to both cover the material and answer very many questions in typical length of a class. This means that if the class has more than a small number of students their questions simply cannot be answered during the lecture.

This is why many larger classes are divided into sections with labs run by grad students. These smaller forums are are better for discussion but their quality depends on finding grad students who are good at teaching. This is also fundamentally not scalable.

Online learning has the potential to break thru this limitation. The recent wave of MOOCs are a huge improvement over the online classes of just a few years ago. I believe there is still lots of room for improvement from both tailoring the class to the individual student (e.g fast vs slow pace) and A/B testing which has only just begun to be applied.

Interestingly some of these improvements can be applied to tradional classes some of which are already using online discussion forums and automated grading. A hybrid approach (in person lectures and discussions with full online support) will probably provide the best overall quality but will be more expensive than a pure online approach and not practical for people who can't physically attend classes.

majani• 13 years ago

It's a reality that for most people, learning gets tedious after a while. The synchronous learning helps a lot of people drag themselves back into class when they're not really "feeling it."

Tycho• 13 years ago

Same with me - I signed up to loads of these things because I loved the idea. Turns out I didn't have time to actually do any of them, but it didn't cost me anything to sign up. I may at a later date search my inbox and access the material.

It's not really a retention problem if you offer something for free and tons of people have a look without making much commitment. Having said that, probably the only relevant statistics about these courses are how many people did them and completed the assignments/tests.

fudged71• 13 years ago

Exactly this. I was almost offended when the course material was taken down or that I was 'locked out'; it seemed like it went against the whole idea of the online learning initiative. Quizzes and exams, okay, but learning materials should always be accessible!

I used the database course to supplement a database course that I was taking at the same time, and it was not just a stark difference in quality but also fantastic that I could pick the pieces of material that I needed to learn better and watch those lectures at my speed and how I wanted!

It's important to me that all course materials are available at the start of the course, and until at least a year after I take the course. Otherwise give me an easy way to download the course materials as a bundle to use in the future.

viveutvivas• 13 years ago

The way that Udacity does it is perfect for me -- work through at your own pace. Coursera has a couple of self-paced classes, I believe, and I wish that more of the courses had that option.

I think the majority of MOOC course developers want to run their online courses as closely to their university versions as possible, since that's what they're used to working with. It also provides a handy way for them to go off-duty, in a sense, if the course has a finite end date. Using their current pacing structure (which is incredibly difficult, as the OP points out, for people that aren't full-time students) allows the teachers to do something other than devote themselves solely to the course, assuming they don't want to just post an archive and leave it alone -- which would meet a lot of people's needs, but misses the whole teacher-student interaction, which is pretty much missing from MOOCs anyway. If they want to provide an environment that's like a classroom, with students interacting with the instructors and with each other, you kind of need everybody at the same pace. It would be nice for us if they could slow that pace down, but that would probably increase the workload for the instructors.

We're still early in this game. I'm glad that so many professors have been willing to invest the time into developing the courses, and I understand why they are currently set up to be conveniently structured for them. I think we'll start to see some improvements if/when the money appears in the MOOC game. Once it's no longer basically charity work for the instructors, there will probably be more efforts to work around student schedules.

goostavos• 13 years ago

I agree completely on the Udacity model allowing you to work through at your own pace, as well as keeping the course open After it finishes. The strict scheduling thing with other sites drives me mad!

In addition to having a normal job, I'm a traditional university student, so any of the online classes I take are simply out of interest in the subject. I dip into the class when I have time, almost like a leisure activity. I just don't currently have the time to fit in more strict course work on top of my already over loaded class schedule and work week.

manifold• 13 years ago

My problem was mainly the availability of the materials. Codecademy and duolingo give you access to as much as you require from the start and you can go through as quickly as you like. The university driven sites limit access to so much per week (though I'm not sure how courses will operate the second time) and demand you stick to their schedule, though granted this may be due to their need to peer-review the more demanding assignments.

Unfortunately my free time isn't available in nice predetermined six-week chunks, but even if I am able to catch up three weeks or more in a weekend the courses gave a very negative vibe about continuing to progress as soon as you miss a single one of their deadlines (i.e.- "you missed our deadline for this multiple-choice computer marked test, so your effort no longer counts"). I've 'failed' several coursera sessions in the fourth or fifth week for that reason.

Timetabling seems a very traditional educational view, and it contrasted sharply with codecademy and sites like duolingo where I spent Jan and Feb learning the basics of new languages - computer and human. I finished the courses I took because I did them at my own pace.

nolite• 13 years ago

really great points in this article. I hope that Coursera sees this and really takes it seriously. I love what they're trying to do, and have signed up for about 30 classes myself that I totally intended to study, and just couldn't finish a single one. And its true, artificially imposing the traditional college course model just does NOT make sense in the online context. Why are there deadline for problem sets, and test schedules? Fix that, and it'll change everything

johnwatson11218• 13 years ago

One of the things to that I found confusing with the coursera classes was that each one seems to have its own set of rules for dealing with homework assignments, late homework, and partial credit. At one point I was signed up for two classes at the same time and I assumed they had the same policy for when homework was due. It wasn't a big deal but there were several times I would have a free hour or so and it was a task to figure out the best way to use that time. I think it is a positive thing that the courses are trying out a lot of different approaches to see what works best but this reminded me of the parts of academia that I didn't really care for.

jurassic• 13 years ago

There will continue to be low retention until there is an incentive to finish. Without the credentialing piece, I don't see a problem with low retention. I think it's awesome that people dip in and out as their needs and interests change. In every class I ever took at MIT/Caltech there were at least a couple lectures I would have been happy to skip. MOOCs enable that.

I hope there will be more short and tightly focused online courses in the future instead of monolithic traditional-length courses we have today. I think that'd go a long way to improving the apparent retention rate.

futhey• 13 years ago

Since the barrier to entry for enrollment is zero, compared to traditional education, directly comparing them is rather unfair.

pamelafox• 13 years ago

(Coursera engineer here) It's great to see thoughtful critiques like this.

Thankfully, we have a data analytics team now (3 out of our 17 engineers), and they are studying our retention statistics and the factors that affect it. They're also running A/B experiments to see what increases it and getting some interesting results.

We do see some big advantages of the timed model for learning, particularly in classes with peer-to-peer grading and evaluations, but there's obviously a big desire for the self-study mode, which is enabled for a few of our classes currently.

We're also introducing things like Signature Track, which some students sign up for just to encourage themselves to make it to the end (and it seems to work for many of them).

We'll keep experimenting to see what makes students both happy and successful. :-)

mikevm• 13 years ago

Nice to see you here!

I do hope that you will open up most (if not all) of your courses for self-study mode. There are a few courses I really wanted to take, but due to the lack of time I really couldn't commit myself at the time they opened (I wanted to do them in the summer instead!).

I also think this guy makes valid points: https://news.ycombinator.com/item?id=5469909

jahmohn• 13 years ago

Hi, Pamela. I am a long-time user of Coursera and I love the site.

I have feedback for you as regards retention of students.

The number one thing is due dates and late penalties. If I sign up for a course and work through the first few weeks and get everything done, and then I miss a deadline because life happens, suddenly I am no longer able to get a 100% in the class. I do not mind getting an A- in a class if it happens naturally because the material is difficult, but losing my A+ or A because I am 12 hours late on an assignment ruins the experience for me. I usually drop a class when this happens and wait for it to come around again.

A counter-example to prove the rule: Robert Sedgewick's Algorithms course. The last time he gave it, he had late penalties and I only had time to get through the first two weeks of the class without getting behind. This time around, there were no late penalties. Being enormously busy with other things, I was not able to keep up with the class as it went along. However, because I lost no credit for doing things late, I was able to complete the entire class in the final 10 days of course and I got an A. I have no problem staying up all night to finish interesting programming assignments and quizzes, but I need to be allowed to do this on my schedule.

I strongly encourage you to encourage your professors to do away with late penalties. It should be up to us as students to determine what our work schedule is. I know for some courses, those that use peer-evaluation, this is impossible, but many more could take this route than currently do.

I agree that Signature Track does have the effect of encouraging one to keep up with the class.

robryan• 13 years ago

For online learning the grades aren't likely to matter, people are mainly doing it to get the knowledge. I can see though if you distilled it down into telling people what you got, say a B, won't reflect how well you actually learned the content if the B was only the result of late handins.

On the other side of the coin though people are relying on the strictness of handin times in order to motivate them to get through the course in a reasonable amount of time.

Timed based on a final date to finish everything is a compromise but wouldn't work for everyone, as some would loose motivation after falling too far behind. So maybe there needs to be a few deadlines throughout the course, but not a weekly one.

The only course I have done I ended up missing a week handin as well, I think most working people would be in the same boat.

olalonde• 13 years ago

Genuine question: why do you care so much about your grade?

bbbbbbbbb• 13 years ago

With work, family and everything that goes around, it is very hard to be in time... Standford Class2go gives you half credits if you are late for all the quizzes and exercises but the final exam counted triple so you could skip a few things here and there and still work it out(you had to study them for the exams though!).

I love the fact that there is a due date, it forces me to do it and a certificate is very rewarding too (I tend to give up the courses that give me no "credits")...

I think in general,the schedule are too tight though.

zura2• 13 years ago

For me, the killer feature would be the availability of textual/reading material corresponding to 100% of the content provided by videos.

So successfully completion of the course won't require watching the videos.

mark_l_watson2• 13 years ago

You may want to rethink that. For most of the classes I have taken the lectures were very enjoyable. I like the frequent quiz breaks also. The videos have captions if you like them.

riggins• 13 years ago

I'm curious about enrollment trends.

Having taken online courses from the start my sense is that enrollment/participation is slowing. The bulletin boards of some classes are pretty dead on the first run.

Its not crazy to think that the people most likely to take the classes would participate right away and finding the next batch of students will be harder.

vdp• 13 years ago

A relevant graph: http://www.katyjordan.com/MOOCproject.html

IMO for a class with tens of thousands people signed up, 5% retention rate is not really that bad.

tsiki• 13 years ago

I agree with the sentiment, but I wonder if having a low barrier of entry is really the solution, or does having a low barrier of entry average out to a low level of initial commitment. It'd be interesting to see if an online course for which you pay for would be better at retention. From a business point of view, they could even offer to give you the money back if you finish the course, or use some other retention trick, like have people calling you to create some social pressure if you seem to be falling behind.

ghaff• 13 years ago

There is something to be said for class participants having some "skin in the game" so to speak. But I think at this stage, the companies/organizations involved are most focused on getting the most people to experience the courses--which having a fee would certainly diminish. As someone else wrote, I expect that these will become more modular over time. It's only natural that they're mimicking traditional product classes to start--but it's hard to believe that's necessarily the optimum chunk size, especially for those doing this as a relatively casual part-time thing.

Paul_D_Santana• 13 years ago

Hah, I was about to post the same idea, about paying for a course and getting the money back on completion. Maybe even redistribute the funds from prior students who hadn't finished the class to students who complete the class and go back to help others. Get paid for answering questions, kind of like a university TA.

Another idea I had was that students who completed a course, and achieved a certain level of proficiency, would get access to a special forum. This forum could be shared by potential employers.

gingerlime• 13 years ago

I totally agree that commitment is a major part of any educational success, and paying for something drastically increases our commitment and motivation.

At kenHub[1], we offer anatomy online training for medical students, physiotherapists etc. Since we are working in a collaboration with the Charité University[2], we offer students at the university the same content other students pay for, completely free. Our statistics shows around a factor of 8x in engagement of paying vs. non-paying students[3].

[1]https://www.kenhub.com [2]http://www.charite.de/en/charite [3]http://imgur.com/5eG3teP

waterlesscloud3• 13 years ago

The author says courses shouldn't have deadlines, but then talks about retention that's measured as of the deadlines.

I signed up for the Hinton neural network class when it was almost over, and completed it well after it ended.

Was I "retained"? Did I not count as completing the class?

Just because there's deadlines doesn't mean you have to finish it by the deadline.

henrik_w• 13 years ago

Taking a Coursera course is hard work. I've taken 3 courses so for (one on databases and two on algorithms) over the past year and a half. This is about as much as I can manage (I work full time and have a family), even though I would like to take more (so many interesting to choose from). But the main reason for this limit is the amount of work it takes, not how the course is presented, or how well written the mails are.

For me, the pace of the course is actually a plus. If there were no deadline, I simply would not get around to doing it. I remember when I first found all the MIT courses on-line (several years ago). I really wanted to take some of them, but because there was no schedule and no deadline, I never got around to taking any of them. It's only with Coursera that I have actually taken (and finished) any.

I've written about my experience of all three courses on my blog. The latest was Algorithms: Design and Analysis, part 2 http://henrikwarne.com/2013/02/18/coursera-algorithms-course...

denzil_correa2• 13 years ago

One of the main problems the author has with Coursera or EdX is that it is "too fast". To address this problem, I urge the author to try Udacity [0]. You can set learning at your own pace.

That said, not all material on Coursera is "amazing". Some of the classes have very high completion rate like Functional Programming with Scala has a 19.2% completion rate [1]. Similarly, the class taken by Andrew Ng on Machine Learning is fantastic. However, many of my peers had bad reviews on Daphne Koller's Probabilistic Graphical Models class. Last year, I myself registered for one of these MOOC's and found that half of the course was good while the other half was quite bad - both halves had different professors.

At some point of time, universities would have to realize that great researchers do not make great teachers. Some excel in both - researching & teaching while some in just one of those two fields.

PS - Other problems on reengagement do stand though.

[0] https://www.udacity.com/ [1] http://www.katyjordan.com/MOOCproject.html

DennisP2• 13 years ago

Koller's class is difficult, but it's a graduate-level CS class with almost all the same material as the class at Stanford. I thought she taught it well. It was rough, and I didn't make it through the whole thing the first time, but I'm back to try again.

denzil_correa2• 13 years ago

    Koller's class is difficult, but it's a graduate-level CS class

A graduate level class is not a guilt-free pass to make it "difficult". A course no matter what level, should be engaging and challenging. It need not be difficult to be an engaging course.

fcorr• 13 years ago

I'm taking two Coursera courses at the moment - Calculus One and Pattern-Oriented Software Architectures for Concurrent and Networked Software. I'm very pleased with the material and presentation in both, but I don't think I could handle more than two such (relatively difficult) courses at any one time.

Codecademy and Udacity allow you to freely dip in and out of lectures and assessments, whereas Coursera courses demand that you deliver quizzes/assignments every week. It's hard to recommend one approach over the other -- with weekly deadlines I find I have more of an incentive to engage in the lectures and course materials, but the added failure conditions can push students to abandon the course.

mikecane• 13 years ago

I've not tried these. Do they all require you to sit at a desktop PC or are they tablet-friendly or tablet-customized?

tunesmith• 13 years ago

For the past year, I've only been seriously interested in three Coursera courses - Scala, Ng's Machine Learning, and Probabilistic Graphical Networks.

All three overlap at the same time!! And they don't give assurances of when they will be offered again, or even if. Argh. I've started Scala but doubt I'll be able to continue if it I attempt PGN again. This happened to me last fall and I ended up completing none of the three.

Plus, it is apparently "known"... somewhere... that certain courses are easier if you take other courses first... but good luck finding that information when you want to refer to it.

waterlesscloud3• 13 years ago

I agree the scheduling makes it difficult. I'm doing the NLP class right now, and scheduled to start the PGM class in a couple of weeks. I'm worried about the overlap, but I'll make it work out.

But I also wanted to do the Scala course. I didn't start it because I know I'd not be able to keep up with all three when they overlapped for a few weeks. It's a shame, but I had to prioritize.

mani27• 13 years ago

Even if 5% people(say 5000 people) complete the course, whats wrong in that ? 5000 is still a big number. Many Prof's wouldn't have taught that many students all their life. I have myself completed 7 courses from edX,udacity and coursera. Each of these have a different approach. Udacity for beginers while edX and coursera have some advance courses and I found these courses a great supplement to my course work at my school.

jonathanjaeger• 13 years ago

True, that's a nice number. But you have to keep in mind many of these startups, while doing something noble, still need to make a big business (often they're VC-backed). Increasing retention rates is essential for business and for making online education viable. I'm not sure if 5% retention makes the model work or if you need >10%.

queensnake2• 13 years ago

Also, their mindshare: "We're changing the world, 100K under-served people signed up!" vs, 5K who finish and most have at least a Master's already anyway :)

jiggy2011• 13 years ago

Does it matter if not everyone watches right until the end and completes the final coursework?

I want to know as much as I can about different subjects but I want to immerse myself in them to different levels.

Sometimes I am happy just to stick a few videos on in the background while I do something else, so I get the gist of what something is about and know what to focus on later if I need more information.

I don't necessarily care if I could pass an exam or not.

Turing_Machine• 13 years ago

Another article that equates "MOOC" with "online education".

Sigh.

walrus• 13 years ago

I'm more likely to finish a few chapters of a textbook than a series of videos:

1. Videos are usually too long. There isn't a natural break to stop and try things out. (Udacity does a good job with this one.)

2. Information density isn't constant. With text, this isn't an issue: I naturally adjust my reading speed if I hit a sparse or dense area. With video, I have to either live with it or constantly re-adjust the video speed.

nbouscal2• 13 years ago

I don't see how something can be a "dirty secret" when everyone already knows about it. This is not news, and I think the author is well aware of that and just wanted an eye-grabbing headline.

As for courses being too fast: they're college courses. That's a core part of their value proposition. If they were slower, or did not follow a rigid schedule at all (like the "better" examples presented in the article), they would be a fundamentally different product. The author simply doesn't understand the concept behind MOOCs, and probably isn't their intended audience. He would be better served watching lectures on Youtube (I don't mean that sarcastically, there are fantastic courses available there).

Synchronous learning is not, as another poster claims, an anachronism. It simply isn't necessary or valuable for everyone. For some, myself included, it is a significant benefit. That is the market that Coursera and edX are targeting, and one shouldn't criticize a company simply because one isn't a member of their target market.

lawnchair_larry5• 13 years ago

Or should you not criticize dissenters just because it happens to fit you? There could be a better way for more people, regardless of your personal experience.

nbouscal2• 13 years ago

There is a substantive difference between the service offered by iTunes U, Youtube, and countless others, and the service offered by Coursera and edX. This difference corresponds to a difference in learning styles (synchronous v. asynchronous, as it has been phrased in this thread), and it doesn't make sense for any one product to target two diametrically opposed learning styles. What does make sense is the approach that Coursera has actually already been taking: offer both as separate products. You can take classes on schedule, or you can take them off schedule for no certificate. Complaining about the synchronous model because you are an asynchronous learner simply doesn't make any sense.

michaelochurch2• 13 years ago

The issue with Coursera is the synchronous model. The material's good and the selection's excellent, and they deserve a lot of props for solving such a critical problem. College life is sanitized. Real World interjections are rare enough, at that age, to be handled case-by-case when it comes to extensions and such. Adult life is messy and complicated-- sick parents become more common, job demands fluctuate-- and synchronous education is just brittle. I didn't find Coursera courses to be "too fast". The paces were fine, so long as I didn't have fires in the rest of life.

It's not a "dirty secret", though. One should know and expect that. If anything, low retention is a good thing insofar as it means that the courses are demanding and people who aren't dedicated trickle out.

I feel like we have three problems to solve in online education. I'm sure there are plenty more, but 4 stand out right now:

(1) Hidden node discovery. You're 23. You just learned Python. You want to be a Data Scientist in 3 years. How do you get there? "Data Scientist job" is one node, and there are a bunch of prerequisites to those (hidden nodes) and prerequisites to those. How do you navigate this network? The 23-year-old programmer doesn't know where those hidden nodes are. In other words, the Google for Learning and Development.

(2) Forward learning. Recommendations. Things that would be interesting to a person that she doesn't know she wants to know, because she doesn't know that it exists. Since there's a lot of investment here (you're not just buying a book and possibly reading it, but anticipating putting 50+ cognitively intense hours into a course) it would be nice if the service gave indications as to why it was making those rec's.

(3) Interactivity and (buzzword warning) "gameification". When you haul out an 800-word machine learning textbook, you often have to go for a long time (hours) without the "kick". There's a flat array of 50 exercises of which 25 are easy, and 25 are really hard and will take a long time (they might be worth doing, but they aren't quick) and it's hard to pick which ones to focus on. The programming exercises in texts often don't get your creative juices flowing. Not a lot of people can get through long spells without feedback, and the skill of "making your own feedback" seems to be losing ground in our distracted culture.

(4) "Social". Online study groups. This is Big and I don't know how it's going to evolve. How do we keep quality control in place and make sure that our automated expert discovery mechanisms work?

btilly5• 13 years ago

...in our ADD culture.

As someone who has personal interaction with the disorder (I'm not officially diagnosed, but probably have it, my son has it, several siblings, nieces and nephews as well), I hate this phrase. Because people use it to mean the exact opposite of what the actual disorder is.

People with ADD or ADHD (different names for pretty much the same thing) do not lack focus. To the contrary, a primary characteristic is extreme focus. Instead we lack the ability to focus on what we are told to focus on.

Society labels this "distractibility" because the person does not focus on what those around would want them to focus on (following instructions, getting dressed, doing homework, etc) but instead goes back to what interests them (bugs, dinosaurs, math, etc). But in terms of ability to get fascinated by something, then follow through on complex tasks, people with ADD or ADHD are significantly more capable than the general public.

michaelochurch2• 13 years ago

Interesting. That sounds a lot like me.

You make a really strong point. Going back and changing it.

jcheng2• 13 years ago

That's an interesting and appealing description, but sounds pretty different than what Wikipedia has to say about ADD. http://en.m.wikipedia.org/wiki/Attention_deficit_disorder#se...

jcmontalbano2• 13 years ago

"We lack the ability to focus on what we are told to focus on," or indeed in general I find that I lack the ability to control what I'm focusing on.

So I reduce distractions and work in a black box. With only one task in front of you, you get it done.

rtpg• 13 years ago

I have been diagnosed with ADD, and I know I can't trigger that focus, even for things I want to do. I sometimes fall into extreme focus mode, but it's usually on "accident".

mshron• 13 years ago

I think Coursera has struck a very effective balance between synchronous and asynchronous learning. Once you sign up for a class, you're able to go back and watch videos, get assignments, even submit assignments for automated grading.

For those of us who find the synchronous style to be a sufficient motivator, it's a great benefit. For people who fall behind but want to cover the material, they're welcome to do so. The grades don't actually matter, remember? Coursera is already "gamified." It's just that the game is done so well that most of the time we're not aware of it.

StavrosK• 13 years ago

I agree with this. I wouldn't be motivated enough to follow 4 courses if it weren't for the synchronous model.

jacalata• 13 years ago

That's certainly not true for all Coursera courses. I took one in February - if I go back and look at it now all the material is gone (although the lecturer put the videos on youtube) and there is nothing but a flat description of what the course was. I'm actually signed up for a course now that I found out about half way through and didn't have time to do on the spot, so I'm looking for a way to download all the material because I fully expect it to disappear once the course 'finishes'.

demian2• 13 years ago

That's key.

MOOC have organized material and information in classes, but now it needs to also curate "tracks".

hipsters_unite• 13 years ago

> "If anything, low retention is a good thing insofar as it means that the courses are demanding and people who aren't dedicated trickle out."

This. You need gumption to succeed in a traditional university course so surely the same will apply to a MOOC with adequately difficult content?

mathattack• 13 years ago

I think you hit it right in the head with the synchronous nature. It's ignoring a great feature of the new medium. Ignoring it would be like putting an encyclopedia online without links.

I wish the author would have discussed Udacity.

VLM2• 13 years ago

Or like putting the encyclopedia online and then, in 2013, deleting articles to "save space".

dpmehta02• 13 years ago

Strongly agree with all of michaelochurch's points, especially #1. Frankly, it baffles me that "career tracks" weren't (and still aren't) a core component of the MOOC product offering. (I actually applied to YC S13 with a solution to points 1-3.)

Duke researchers who actually taught a course on Coursera put together a comprehensive report on their experiences. In it, they specifically address the drop-out issue. I highly recommend it: http://dukespace.lib.duke.edu/dspace/bitstream/handle/10161/...

amirhhz2• 13 years ago

#1 resonates strongly with me, too, as I've also been thinking about the best way to put MOOCs into tracks/streams that lead to an end goal (e.g. become a data scientist) and engage learners that way.

I'd love to talk to someone who's on the same page, do you think I could pick your brains on this? If so, a contact email is on my profile page :)

cliftonmckinney2• 13 years ago

I think this is correct, for the most part. Call me a dirty capitalist, but I think that especially for problem no. 1 the solution probably needs to involve--in a significant way--those who have the most incentive to see it solved. Generally, the incentive for students comes from curiosity, desire to learn new things, and maybe the desire to broaden career options, but not really from "profit," at least in the short term.

But, the companies that have immediate need for a data scientist--those are the folks who have a vested interest in the problem being solved. The trick I think will come down to getting companies involved without having them mucking them up like most "corporate training" is these days.

I guess that's a bit of what's missing for many of the online options so far--a direct "if you do this this will happen" value proposition for the students. Right now it's all driven by our curiosity and a desire to learn, which is great and drives many of us to sign up, but sometimes doesn't trump all the "life" that gets in the way after the coursework gets tough.

amirhhz2• 13 years ago

Took the words right out of my mouth. Have you had any experience with getting companies/employers involved in something like this? I've been thinking about how MOOCs would fit into streams (or tracks as someone else here said) that lead a learner to an end goal e.g. data scientist, and companies are the most qualified to come up with the criteria for meeting such goals.

lifeisstillgood2• 13 years ago

(1) Editors not google. If we invent the google for L&D all it will do is read the blogs of practioners of the subject, and syllabus writers, then page rank them.

Why not cut out the middle man and have academics and practionsers actually recommend the courses, and guide people through. Pretty much the only thing we cannot automate right now is real good editing it is sorely needed in journalism, books, what to watch on tv and education courses.

(2) see editing

On both the above points I get the feeling you worry / hope that we shall need to automate huge swathes of this, but I think education is an intensive production - akin to books or films. And so would be due similar levels of review attention.

(3) I think that's not the needed approach - everyone learns at different paces and gets stuck at different points. With maths one approach is large numbers of gradually changing mini tests - if you keep making the same mistake that area is reviewed and revisited till you get it right. I know an Bristol based company doing this now in schools, the success rate - hard to say but the approach seems solid.

Repitition seems to be the mainstay of human learning.

(4) I don't understand- what is automated expert discovery systems? Study groups are an inherently emergent social phenomena - humans are really good at those, so this would be my least worrisome issue - an online calendar and Skype meetings as an example.

trhtrsh• 13 years ago

> Why not cut out the middle man and have academics and practionsers actually recommend the courses, and guide people through.

And then we can use an artificial intelligence algorithm to combine the recommendations of all the academics and practitioners, to get an index of recommendations across all fields.

Call it PractictionerRank, or PeerRank, or BookRank, or....

avenger1232• 13 years ago

I really like your point (1). It actually coincides with the same idea that I have had but you've described in a good way, the "Google for Learning and Development".

I am wondering why this can't be an actual web application? Have different topics, (ie. Learn Web Development, Python, Javascript) and actually have a listing/review of all the credible sources on the internet that one can use to do the learning. It would just be a listing but have others who learned the material, describe how they did it and have them review the resources.

It's come to a point where this is really needed. Just typing "learn javascript" on Google is just not good enough.

Is there something like this out there already?

zabramow• 13 years ago

I'm a big fan of Codeacademy myself. But, as someone who doesn't come from a technical background, I feel like I'm often completing many of their lessons, but not internalizing the information or actually learning how to code.

I don't plan on giving up, but I'm curious if others have had that experience or if I'm the only person who's figured out how to use Codeacademy without actually learning to code.

hkon• 13 years ago

An anecdote, I followed this course this year, https://www.coursera.org/course/growtogreatness and the number of student increased every week.

The guy presenting, Ed Hess, was very skilled and interesting and the material was very relevant. I also enjoyed the discussion of various homework on the course forum.

te• 13 years ago

MOOC providers could enhance engagement and retention by providing audio-only content in addition to or even instead of video-based content. I've got a couple hours per DAY where I crave quality audio content but am physically unable to watch videos at that time. In contrast, I've only got a couple hours per WEEK that I can watch videos.

eah138• 13 years ago

This is an interesting approach. I think if they paid more attention to where and when exactly people would engage with content they could increase retention. I also have more time for audio than video and would love lectures of some sort delivered to my mobile. This could create a sort of immersion experience when combined with video and exercises.

KC8ZKF• 13 years ago

My mother used to tell me that my eyes were bigger than my stomach when I took food more food than I could eat. I have the same problem with MOOCs.

The future me always has more time, is always more dedicated to learning, is always more focused. So I sign up, and when the class starts I'm no longer the better, stronger, faster, future, me, I'm just me.

Still I manage to finish some, and learn a little from the ones that I drop, so I don't see it as a problem.

sefk• 13 years ago

I don't necessarily think this is a bad thing.

For Stanford's database class that just finished we had 64 thousand register, but only 20,000 actually do some work, and then close to 5k actually take the final or get a statement of accomplishment.

My blog response: http://sef.kloninger.com/2013/03/online-ed-retention/

I'm the engineering manager on Class2Go, Stanford's open-source MOOC platform. Check it out: http://class2go.stanford.edu/

vixen99• 13 years ago

I was one of those students (great course!) and you're right. So, folk sign up on the spur of the moment and then drop out! Big deal. Where’s the dirty secret? Was there any cost to signing up on the part of the student or the instructors? Not at all. Why then probe further into what would be a vast range of personal reasons for not completing the course?

We have to ask ‘Why do people make cost-free momentary decisions and then fail to act on them?’. Do we think this an important question? I suggest most of us have much better things to do with our time than agonize over this.

gfodor• 13 years ago

Learning is hard. Self-study is very hard. Incentives matter.

The main incentive to power through a Coursera course is personal enrichment. This is not the reason most people go to college.

darrellsilver2• 13 years ago

Pacing is incredibly important to motivation. The thing is, there's no single answer. Even in traditional, full-time, in person classrooms where everyone puts in the same class hours, natural ability, different skills at the beginning and better attention mean that students will always learn at different speeds.

The ideal model is one of personalized education. What's kind of shocking is that online education so far has basically ignored this altogether. There's been zero investment in ways to continually pace and encourage students to learn at a speed that is both challenging and within reason.

This isn't too surprising: it's a hard problem, and it's much easier to have users paying a subscription fee forever (for those services that are paid) and blame themselves when they fall behind.

At Thinkful (http://www.thinkful.com/) we see evidence that our learning model, which pairs experts and students together much more like you'd expect from tutoring, sees 10x the retention and completion as other online course options.

lawnchair_larry5• 13 years ago

This is not only a shameless plug but is actually a load of crap.

darrellsilver2• 13 years ago

Please elaborate. Unless "load of crap" fully articulates your argument. If so, thanks for the feedback.

sopooneo• 13 years ago

At a fitness center, personal trainers serve the role not just of instructing their clients, but of renewing motivation. That task is absolutely dependent of physical proximity. I feel like this dynamic is being completely ignored in most discussions of online education.

spikels2• 13 years ago

Everyone needs to understand that the low completion rates are not as bad as they at first seem. The way to try out these classes for free is to sign up. There is no obligation at all: completely voluntary, no cost, no time commitment and it will not show up on permanent record.

These completion rates are actually more like conversion rates for free trials. As many of us know these are almost always quite low. How many people actually ever read books they "Look Inside" on Amazon? How many people finish long articles on the web? Now many people sign up for paying accounts after free trials of your new website? Not very many.

They are not the same as dropout rates in high schools or universities despite what some online education haters say[1]. Like any disruptive technology the existing players are threatened so you need to pay attention to the source of the criticism.

Instead completion rates are a metric that can be used to improve the class such as the suggestions here. The developers of online classes should and are using A/B testing to improve completion rates. However like almost all conversion rates they will likely remain low.

[1] http://mitsloan.mit.edu/shared/ods/documents/High-Costs-of-F...

UnoriginalGuy• 13 years ago

That's because online education tries to simulate traditional education.

Back awhile ago free online lecture videos (e.g. Berkeley) were simply uploaded to a page and anyone could go watch them without signing up for a class or otherwise jumping through hoops.

This resulted in me watching a few random lectures about subjects I know little or nothing about. Like econ classes, engineering, science, or similar.

This was good because it was short bursts of information without any hassle, commitment, and similar.

These days everyone has locked their content behind sign up/registration pages, and you're expected to commit to a tradition semester of a class in order to get a worthless certificate of completion.

They also dictate the speed at which you learn. No more learning at your own pace, no, you have to do one a week for as many weeks as it goes on for like it was a traditional class.

And why? Why indeed. Why is online education simulating University education when it actually has no relationship to it? You aren't getting a University qualification by completing a "class," there is no reason why a "class" has to be X number of weeks or you should have to complete the rereq's in order to take it.

Places like Khan Academy have got this right. I can go to Khan Academy right now, click on a video series, watch a few bites of information and then stop when I'm ready/bored. Coursera's class system is just pointless, stressful, and annoying.

If Coursera was transferable then it would absolutely make sense to do. But that doesn't look likely and while that isn't the case they're just making access to learning material more difficult with no obvious benefit to anyone.

shawabawa3• 13 years ago

> I can go to Khan Academy right now, click on a video series, watch a few bites of information and then stop when I'm ready/bored.

You can. But you wont.

The sad fact is that deadline motivate people. Without having deadlines and structure coursera would have gone nowhere.

lawnchair_larry5• 13 years ago

This claim is baseless. Please reserve it for after various MOOCs have tried different models and the data supports it.

It's possible and even likely that the schedule contributes to motivation for many, but my experience has been the opposite and I'm probably not unique.

lawnchair_larry5• 13 years ago

I agree with many of your points but the main reason I use Coursera is because it's an actual course, with structure and quizzes and assignments. I personally find Khan Academy to be useless, and I've tried to use it (Edit: useless for me, but I do think what he has created is fantastic). I think I would find it useful as a supplement to an existing course, and that's what it was originally designed as. Very few people learn anything of moderate complexity without doing exercises throughout.

adaml_623• 13 years ago

With regards to your use of the word 'worthless' relating to the Coursera certificate of completion.

Many university courses aren't significantly different to Coursera so I think that if the courses keep up their standards and enough people have knowledge of them then it is possible that that certificate will not remain worthless.

We were discussing Coursera at work last week and I think that if a job candidate appeared with Coursera completion certificates on their CV then it wouldn't be looked at unfavourably.

demian2• 13 years ago

"Awful" is kind of relative.

We are talking about a new model, low "retention" seems to be normal for free clases wich have absolutely no barrier of entry.

I have taken 5 courses, only finished 2 "on time". For the other 3 I downloaded the material, got some exercises on sites like MIT OCW, and finished them at my own pace.

graycat2• 13 years ago

I believe on-line learning can work.

Why? Three reasons:

First, a 'dirty little secret' of the US software industry is how much of the learning from the beginning of electronic computing to the present has been just from individuals teaching themselves from books, e.g., K&R on C, Lippman on C++, Ullman on database, Sedgewick on algorithms, and on-line, e.g., Microsoft's MSDN site, StackOverflow, etc., essentially independently without courses, lectures, problem sessions, credits, homework, tests, etc.

Or since just K&R, ..., StackOverflow, etc. have been responsible for so much learning so far, then 'the bar is low' and on-line courses should do even better.

Second, in my experience in technical subjects, pure and applied math, mathematical physics, some topics in electronic engineering, e.g., surrounding the fast Fourier transform (FFT), and software, with a class or not, nearly all the learning (in my case) took place from study, alone, in a quiet room, from good materials just on paper. My 'educated guess' is that on-line learning can't replace such learning but can help stimulate more of it.

Third, for a researcher in applied math and software, and also likely some other fields, one of the main 'work items' is to take recent books and papers and work through them much as working through advanced course materials in such subjects.

Back to my case, in the fields I worked hardest on and did the best in, math, physics, and software, starting in the ninth grade, through my Ph.D., and in my career to the present, I did nearly all the work with relatively little contribution from teachers. E.g., in plane geometry my teacher was the most offensive person I've ever known in education, and I slept in her class and refused to admit doing her assigned homework. Instead, I worked every non-trivial problem in the book including the more advanced supplementary problems in the back where she never made any assignments. Then, after working all those problems, no wonder, on the state test in the subject, I did fine: I came in second best in the class; the guy who beat me also beat me by a few points on the Math SAT -- we were 1, 2 in the school. Net, my approach to learning plane geometry worked fine.

I never took freshman calculus. The college I started at wanted me to take some math that really was just a review of what I'd covered in four years of math in high school. So, a girl in the class let me know when the tests were, and I showed for those. The teacher said I was the best math student he'd ever had. Meanwhile, I got a good calculus book and started in, worked hard, and did well. For my sophomore year I went to a much better college and started on their sophomore calculus and did fine. Yup, never took freshman calculus.

When I went to graduate school, I took a problem with me and had an intuitive solution. My first year had some good courses (one was just terrific, from a star student of E. Cinlar now long at Princeton) and gave me what I needed to turn my intuitive solution into a solid math solution; I did that in my first summer, independently; and that was the research for my Ph.D.

I continued that way in my career: E.g., in a software house working for the US DoD, I saw a problem in a specification, got Blackman and Tukey, 'The Measurement of Power Spectra', and read it carefully enough to see what was wrong with the specification and how to fix it. Right: without courses, lectures, problem sessions, ....

As far as I can tell, nearly all the technical content on HN, StackOverflow and other Internet fora is from people who taught themselves in similar ways with little or nothing in courses, lectures, problem sessions, .... And that's part of what researchers have to do and is just part of getting tenure as a research professor.

So, since so much work is being done by essentially independent study now, just by not making things worse on-line courses should be able to look successful.

But I see some problems with the on-line materials I tried:

(1) The video quality just sucked. I couldn't read the board. That meant I couldn't copy what was on the board and study it. Bummer.

(2) The sound quality was not good enough.

(3) The course materials, e.g., on paper or in PDF files, were from not good enough down to just missing.

(4) Sadly the quality of the course content was too low; apparently the main reason was the desire to make the course more 'appropriate', that is, 'easier', for 'the common man in the street'. But, omitting material 'waters down' the course content and, really, for a good student, requires that they fill in the gaps for themselves -- bummer.

E.g., I looked at the course by Stanford professor Ng on 'Machine Learning'. What I saw were weaknesses (1)-(4) above. For more, (A) a lot that he was doing was maximum likelihood estimation but with far too little explanation and justification; so, I would have had to have run off and studied maximum likelihood estimation on my own. So, again I was on my own to do some independent work, trusting Professor Ng that somehow maximum likelihood estimation was better than it has long seemed in the statistics community. (B) He mentioned the 'maximization' to be done via following gradients, and that is an overly simplistic and not very promising approach to maximization -- the standard, first problem is that spend nearly all the computing time moving in directions nearly orthogonal to the direction really should be moving in.

So, from (A) and (B), I concluded that for a good course I would have had to have taken his lectures just as topics to be investigated, gone to good materials elsewhere, collected good details, and written my own text. That's his job as a professor, not my job as a student. His field, 'machine learning', didn't look worth that much work for me now. I've done some serious work in several cases of applied math that could be called 'machine learning' as much as his material, and I'm left without much respect for his material.

I looked at the course 'Probabilistic Graphical Models' taught by Daphne Koller. Since I very much liked a course by a star student of E. Cinlar, maybe I should like Kollar's course. Sorry, I didn't -- the quality looked too low. Better quality from Stanford? Sure, K. Chung, H. Royden, D. Luenberger (his 'Optimization by Vector Space Methods' is a beautifully, even elegantly, done one mile long applied math dessert buffet), D. Knuth.

For courses in 'how to code', that is, introductory material in software, gotta be kidding! 'Coding' alone is easy; it's just, pick a language, learn the basic syntax, and write if-the-else, do-while, call-return, allocate-free, etc. It's easy but doesn't take one very far. So, don't get very far with thousands of Web pages of documentation at MSDN on .NET, ASP.NET, ADO.NET, administration of SQL Server, IIS, Windows Server, etc. or the equivalent in the Linux world.

I have two broad conclusions:

First, traditionally in academic material in technical subjects, the author and the student 'reached' to each other, and they connected at a well written textbook. So, the author went far enough toward the student to prepare a good text -- and the best texts are terrific. And the student reached far enough toward the author to make do with little or no more than a good text. It's how I learned plane geometry, freshman calculus, theoretical, applied, and numerical linear algebra, everything I learned about statistics, most of what I learned about advanced calculus, signal processing and the FFT, stochastic optimal control, artificial intelligence, ..., and everything I learned about software. E.g., it's heavily what worked for me.

The on-line community will have to face the fact of this 'reaching'. In effect, on-line learning requires the professor to do more work of a kind that promises not to be well rewarded by tenure and promotion committees. So, net, so far a student should still reach mostly for one of the best texts, on paper or PDF.

Second, the situation will 'settle out': The professors will come to understand what minimum quality is needed, and the students will realize that the learning is not just a spectator sport, is not like watching a movie, and still requires nearly all the traditional work from a book or PDF file. Then the courses will get better; the students just looking to watch a movie won't sign up; and the course completion rates will increase.

pkrein2• 13 years ago

I really like your idea of "reaching" here. I think that in the MOOC model, the role of the MOOC is to help the professor extend their reach towards the students. I think the MOOC should bring the content marketing & designe experts to the table with the professor and help put together the front-end of the class.

Part of that means helping the professor craft every email to the class to maximize engagement. In fact, the professor probably shouldn't even be doing _any_ of that administrative stuff. The MOOC should handle all of that, including the skills required to "extend the reach towards the students" as you aptly put it.

I think Coursera and EdX can do much better at using their design and content marketing talent to make these classes more accessible and successful.

eah138• 13 years ago

> I think Coursera and EdX can do much better at using their design and content marketing talent to make these classes more accessible and successful.

There's an interesting balance here. Professors I've talked to bemoan the lack of 'support' from Coursera (usually the universities provide a pot of money for profs to use themselves). But at the same time they would balk at some marketing guru coming in an 'spinning' their course. Profs are used to large amounts of freedom in their courses and aren't used to incorporating feedback from the outside into their plans.

It's a tough line to walk I think: freedom vs accessibility.

graycat2• 13 years ago

Some of what you are saying is the way to solve some of the problems I mentioned, e.g., poor video and sound quality.

Some more of what you are saying, e.g., about marketing, design, and e-mails to students, definitely would help. However, I assumed that such things were asking too much of poor schools and that, really, the HN readers who dug through K&R, etc. on their own with no marketing, design, encouraging e-mails, etc. didn't really need such nice hand holding.

But I will say, at some crucial points for me, some teachers did play a crucial role: The college I graduated from then had a quite good math department, and some of the teaching aimed at me personally in some of the courses was just excellent and got me going so that I could continue on to 'the next level' in math via independent work. Then for on-line learning, as progress is made on the first, and easiest work, many of the students will need some 'personalized TLC' or risk getting off track, slowing down, and stopping.

Part of such TLC will be overviews: That is, it needs to be more clear to students what the value, purpose, or promise is of various parts of the course.

Another part of TLC is that students need to write some things, e.g., homework exercises, term papers, something like lab reports, and have their work read by an expert and 'corrected'. E.g., a place to learn nicely polished skills in writing about functions, maps, mappings, relations, etc. is in a theorem proving course in abstract algebra where a good professor actually corrects some student homework. Then when work with functions, maps, ..., in computing, will get the notation and terminology right; without such TLC, often won't get those things right. E.g., earlier today I read a little about hashing passwords, and the material mentioned a function "between the hashes and passwords". No, "between" is not appropriate. The usual description, and appropriate, is a function f from the set of hashes to the set of passwords. To be clear and not just picky, function f may not define a function g from the set of passwords to the set of hashes because there may be a password p such that there is no hash h so that f(h) = p. That is, the function f may not be 'onto', i.e., 'surjective' as at

     http://en.wikipedia.org/wiki/Surjective_function

The level of clarity and precision available, useful, and often important is high, but students commonly need some in-person, TLC teaching in such things.

There's an old lesson in 'education': Get the student able to continue on in learning on their own. For that, usually some 'personalized TLC' is needed at a few crucial points in a student's development. To be really successful, the on-line approach will have to include some of that TLC also.

johnminter2• 13 years ago

The hard truth is that the student gets out of a class what they put into it. I have finished 3 Coursera classes, two of which gave certificates. All three were well worth my time, but I worked my tail off to complete the challenging assignments. I found the support staff (TAs) helpful in the forums. Life does get in the way some times but all three coures had at least a week to finish an assignment, typically two and the option of applying late days on quizzes.

The biggest thing I noticed were the clueless forum posts by students who obviously had not read much of the provided material. Two of the classes had material similar to ESR's "How to ask questions the smart way" and this was largely ignored. I think we are seeing the fall out of the distraction of modern society. Sadly, tl;dr doesn't work with complex technical material.

That said, it was a privilege to interact with motivated fellow students. Their input in the forums helped eliminate misconceptions.

By the way, the statistics for our Data Analysis class were similar to those from the OP - from the instructor: "There were approximately 102,000 students enrolled in the course, about 51,000 watched videos, 20,000 did quizzes, and 5,500 did/graded the data analysis assignments."

andrewem2• 13 years ago

My wife also wanted to take that Introductory Human Physiology course. She's a freelance medical translator, so improving her knowledge of that subject matter will have a direct positive impact on her career. She's not among the people who Daphne Koller said "never really intended to seriously take the class in the first place" - she bought the suggested books ahead of time, marked it all down on her calendar, and planned out time to work on it.

What she found was that the course was way more difficult than the description suggested, and proved to be well above what she could understand with her current background. She watched the first video and found that it was going to take way more time than the course description predicted, because she would have to try to fill in a tremendous amount of background knowledge that the teachers presumed. On the plus side, from the course forums she found about resources (videos, etc.) that were more at her current level, and has been working through these.

The comments made elsewhere about the synchronous model for these courses is probably right for her case - she had to wait a number of weeks in order to find out it wasn't the right class for her. This is what you expect when you're in college, at which point you either drop the class and maybe find a different one to take, or tough it out in a class that's not as interesting or useful as you'd hoped. I hope online courses can do better than this.

eah138• 13 years ago

> What she found was that the course was way more difficult than the description suggested, and proved to be well above what she could understand with her current background. She watched the first video and found that it was going to take way more time than the course description predicted, because she would have to try to fill in a tremendous amount of background knowledge that the teachers presumed.

It's quite possible that this is a function of having so little open content out there right now. I can imagine a future where the background your wife needed would be easy to get from the platform itself, so that the profs' demanding teaching could serve audiences both with and without that background. Right now many of these courses must feel like skyhooks but once they're embedded in a web of knowledge (an earlier commenter called these 'nodes') they'll be more accessible i think.

There's still a huge problem of collecting and sequencing these nodes and I think many students will need direct contact with instructors of some sort to do this.

andrewem2• 13 years ago

I agree with your points, and thanks for putting them so clearly. The traditional solution for this is to have each university course specify its prerequisite courses, and for students to talk to the professor before the class starts if they're not sure it's appropriate. The latter happened to my friends and I much more during freshman year, when we weren't sure how our high school classes would prepare us, though it didn't always work perfectly - I knew lots of people who re-took calculus or a low-level foreign language class after having had it in high school, and found it way too easy.

Maybe in 5 or 10 years there will be enough online courses available that it would be clear that the Duke physiology class is harder, and some other class would be a good prerequisite to it. Or maybe there will be smaller modules that would fill in gaps for students who were close to ready for the class, or maybe it'll all be in smaller modules that you'll assemble into a coherent whole based on your current knowledge.

jimbobimbo• 13 years ago

I did one Coursera course, where I'd finish Q&A tests. But the course I'm taking now requires me to do some video/audio assignment, share it with peers and provide peer critique. Sorry, but I have a life, so I'm there just for the content.

eah138• 13 years ago

tl;dr: What's at stake here is the proper roles of ed tech in education. MOOCs are media, only one part of the ecosystem that is developing here, and I think that finding a central role for great educators is key.

Coursera and other MOOCs are media properties that monetize viewership (thorugh certificates rather than ads). As media properties, michaelochurch and others are right that the synchronous model isn't really going to work in the long term.

I think ed tech should be about connecting teachers with the content they need to teach (and I'd include self-educators amongst 'teachers' here). Students need synchronous access to teachers, but those teachers don't necessarily have to have created that content just now. This suggests that online education might look more like a 1:100 teacher-student ratio rather than 1:100k. Nothing's more motivating and engaging than a human being.

Personally, my best experiences have come from two sources: self-study from great materials and guided study from great teachers. This suggests that a role for ed tech might be connecting great teacher with great content rather than broadcasting teacher-as-content. Personalization, engagement, motivation, etc all come for free if we empower great teachers rather than seek to replace them.

It's misguided to think that some professor from a great school can best deliver the content to all audiences. Imagine great open content accessible to every teacher for use in their own classrooms. With an inverted model of teaching, the lets them push content out to the students for self-study and then have synchronous classes where they, the teacher, can motivate and customize the delivery of tough concepts for their particular group of students.

silverlake3• 13 years ago

We are following this model with LearningLine (www.learninglineapp.com). We offer an optional "class" format where an instructor is available to answer questions and review homework. With most eLearning, when students hit a topic they don't understand, they have to figure it out themselves. Some will quit when they hit this wall. Our instructors are there to help students keep going. No more web searches for help, just send a question and an instructor will answer and even get on Skype to explain in person.

eah138• 13 years ago

Interesting. I see you have a lot of courses on offer and are 'powered by DevelopMentor'. Are you affiliated with them or serving as a portal/add on to their existing content?

Jormundir2• 13 years ago

You should title this article "How to cheapen the online education experience". You may be right, that edX and Coursera require a big commitment, but that's what learning requires. Do you want to play games for an hour, or do you want to advance your skills and understanding?

(I've completed the Codeacademy Javascript track, and a full Coursera class (livin' in the 5%, wooo!)).

You say Codeacademy got it right, but I think you're dead wrong. As other articles today have pointed out, at best you're going to learn syntax on codeacademy, and the most basic programming principles, but you'll be miles away from being a capable programmer. The jump from the online editor with tiny little exercises to setting up your own environment and programming your own project is huge. Many of their lessons give misleading or outright incorrect information, not to mention teaching you bad practices.

What this article says is the huge barrier to entry, is not a barrier to entry to online education, but a barrier to entry of learning. What the author proposes is solutions to hide the barrier of learning, by gamifying the platforms and making them more "fun" with a disregard for the depth of effectiveness, and the consequences of pursuing such techniques. (Obviously I'm delving into opinion land, but I strongly believe gamification cheapens and platform and doesn't produce the deep engagement necessary for learning. This is my assessment from my experiences, I'm not going to tell someone their wrong for feeling differently about this. But I think if you wanted to create a deeper experience from gamifying education, it would take a tremendous effort, far different from the typical "Wow you completed 5 exercises! ZOMG KEEP GOING!" joke of rewards that other websites have.

I don't want to rail on the online education effort either, but this is plain cheap thinking. Coursera and EdX are doing an excellent job, but they certainly have many problems to surmount. They've definitely solved the problem of higher education accessibility, now the problems they have are in the effectiveness of education, and on administration and grading that gets the f#$% out of the way of learning. But making it "easier to learn"? I call utter bulls$%t.

asafira• 13 years ago

I think I agree with most of your points, but disagree with your disapproval of "easier to learn". The presentation of material really makes a gigantic difference on how well you learn the material (if at all). While I agree that coursera and edX shouldn't hasten towards "gamification" of any sort, I think there is plenty of work in making the classes of better quality, and I think that _will_ make it easier to learn.

I was taking a course on coursera where I thought the professor was talking too quickly --- so I actually slowed it down to 75% the pace in the video. Awesome! I was happy to be able to do that.

Otherwise though, I don't think the course was presented that well. For one, lectures were shitty recordings of slides; I was looking at nothing but text and static images for entire lectures (never seeing the professor's face), and they were blurry, hard to read, and just hard to understand. Of _course_ that makes it harder to learn! I have to focus on figuring out what the hell is written there! And there was no (free) textbook for the course.

Anyway, I think the point I am just trying to make is that while the author of this article might be suggesting to Coursera to take steps in questionable directions, I DO think a lot of those courses could be better polished, and that would make it easier to learn from them.

Jormundir2• 13 years ago

Very agreed. There's certainly much these online courses can do in the lecture quality department alone. (Maybe they just need more quality control instead of a rush to fill their sites with content). I think you can do a lot to make the learning process much smoother and more streamlined. My opposition to the suggestions in the article is on the grounds that the author is searching for a shortcut to the learning process (or maybe just wants a really addicting, very shallow introduction to the topics), which is exactly opposite of what these sites are aiming to achieve.

I'm all for making these platforms more fun, engaging and rewarding. The more the better. But my main point is that the focus, the most important aspect of education is the process of learning and that often requires prolonged focus (to follow along and venture all the way down the rabbit hole of a subject), deep engagement (so your brain can make all of the connections, tie all of the strings together), and interest. The authors suggestions don't really reinforce the process of learning (as I know), they may even weaken it by creating distractions, giving rewards for too little work (obviously subjective, but being subjective you're going to help some people sometimes, and hurt some people sometimes) and getting students to focus on the rewards rather than their understanding, and rather than building implicit positive feelings about their accomplishments.

kvasan• 13 years ago

Things work differently for different minds, I find that Courseras model of commitment works better for me. I tried 2 Udacity classes which are self-paced, I actually did quite a lot of the modules but somehow I moved on. On Coursera you have to live with the class and at least try to make all assigments and such on time. I think one thing that works good for me is the feeling of having a special oportuinty and "if i dont do this I will be left behind in the new age". Dont get me wrong I would love to be the kind of person I thought I was; searching the web, finding places and things most people would not assume were there and then putting them together. But I need the feeling of being on a "quest" and building up a illusion of "if I make this, things will change". Latter rambling is maybe more general, my point is some people (me) just cant handle the notion of almost force-feed overflow of accesible information and thinking that everybody doing it, so why bother (if the retention in the article is true that should be instrumentaly good for my crazy notion). Sorry for the crazy rant, if only the mind lacked biases. Maybe I should atleast try those mentioned in the article instead of being overpowered by strange thoughts.

tokenadult• 13 years ago

I've bookmarked this interesting analysis to add the link to my personal website. I especially like the comparisons among differing brands of online courses.

Before I read the fine article (which I recommend you read too), I was going to react just to the headline. And that reaction is still there. The reason we (correctly) criticize lack of retention in online education is that online education makes apparent metrics that show the lack of retention. But I would suggest, for learned discussion by all the participants here, that most forms of classroom learning, and especially compulsory secondary schooling in most of the developed world, suffers from a very bad lack of engagement on the part of the students. (Some of the comments posted here before this comment tell stories about that.) A high school student may attend classes often enough to avoid being expelled and gain a diploma, but the student may operate in "regurgitate and forget" mode the whole time he is in high school, not really remembering anything he supposedly was taught. The one kind of deep learning many high school students engage in is learning how to pretend to be studying. If online education develops better demonstrations of learner acquisition of knowledge and skill than mere seat time in a classroom, it may be a powerful force for exposing the dirty secret of classroom education.

newobj• 13 years ago

Seems a bit early to optimize for retention. Or to have dirty secrets. Or real suffer almost any kind of analysis at all.

droithomme• 13 years ago

Another week, another article complaining that teaching a class which 5-10 thousand people successfully complete, and which cost less than $10 per successful student to provide, is an abysmal failure.

Meanwhile, the schools that the authors hold up as more successful examples of education have 20-500 successful graduates per session offered, nearly all of whom are idle young wealthy white people living in rich western countries, and who pay up to several thousand dollars per session each for this privilege.

steveinflow• 13 years ago

I don't think low retention is a secret and I don't think it's a bad thing- it shows that there's no barrier to showing up but there is a barrier to completion. That's what we want- equal opportunity, without giving completion away unearned.

For my part I sign up for more classes than I could possibly complete. Sometimes I just want to take a peak at the material to see if it grabs me. And sometimes I just want to get familiar with a subject without going all-in, picking and choosing the lectures that are most relevant to me.

People shouldn't feel like they're hurting Coursera's/EdX's/Udacity's stats by dipping their toes.

jcmontalbano2• 13 years ago

This article exposed me to a novel usage of "churn".

http://en.wikipedia.org/wiki/Churn_rate

laughfactory• 13 years ago

Just to throw my hat (opinion) into the ring...

I've tried pretty much all of the "MOOC's" out there, and so far the only two I've stuck with were Codecademy and Udacity. Why? Precisely for the reasons that are mentioned at the article which started this discussion. The format of Edx and Coursera is just all wrong for most people who DO want to learn something but have a busy life that is not conducive to a traditional college class.

Besides which, all too often, Coursera and Edx just throw a college class online (lectures, course notes, homeworks, quizzes, and tests) and call it a day. Then they have arbitrary deadlines--which I acknowledge might help some people but the first time I got a deadline email I decided that it wasn't for me...no matter how interested I was in the class topic. I'm already married and a father who is working part-time while doing a Master's program in Computer Science. So I don't have huge chunks of time available for traditional course work (in addition to my Master's coursework). I have little bits of time every day which I use effectively to progress through the Codecademy and Udacity coursework. Slowly but steadily I'm getting through both and making substantial progress.

But, since Coursera and Edx require huge blocks of time committed every week, and a tight timeline to complete the course, I just don't find it's workable. Which is sad since they seem to offer some tremendous material.

In short, I think that Coursera and Edx (and probably colleges in general) should learn a thing or two from the highly modularized approach of Udacity and Codecademy.

I find that my learning retention and interest remains much higher with both Udacity and Codecademy because they present a little material (little bite-size chunks) which slowly build into more and more challenging material and intersperse "challenges" or "quizzes" throughout--rather than the traditional approach which--by necessity--was far less responsive.

The fact is that online learning doesn't just reach potentially numerous people that wouldn't have otherwise been exposed to such educational opportunities at such a low price (currently free), it potentially allows for a radical rethinking of the traditional education model.

I think that it is sheer ignorance to say, as some have in the comments I've read (and as--apparently--Coursera maintains), that those who don't finish a course just aren't interested enough or motivated enough. This is ridiculous. The very same (allegedly "unmotivated" or "undisciplined" or "uninterested") people may be successfully working their way through Codecademy or Udacity. And yes, it may take them a while, but eventually they'll complete a Codecademy "track" or a Udacity course. And they'll do it one bite-size chunk at a time...and sometimes, say on a weekend, they'll plow through a whole bunch of bite-sized chunks.

When Coursera and Edx adopt a more bite-sized format which isn't oriented around arbitrary deadlines, I'll check them out again. Until then I'll stick with Codeacademy and Udacity.

Bottom-line: the standard model is broken. Lectures and homeworks and quizzes and tests are poor ways to educate. For many years they were the best we could do within the technological constraints we faced. But now we can do better. Heck, I'd encourage everyone working the bleeding edge of online education to think "outside the box" and try new approaches. While the best new approaches I've seen have been those utilized by Udacity and Codecademy, that's not to say that there aren't even better methods--methods better at retaining learners (keeping learners engaged), AND methods which excel at increasing retention of material.

tansey• 13 years ago

I saw a talk from Andrew Ng a couple of weeks ago where he was asked about the low retention rate. He countered that 40% of people who submit the first homework actually complete the class. I would say that is pretty great retention rate.

Personally, I've signed up for dozens of Coursera courses but I've only completed one (Game Theory). It's not necessarily that the other courses were lacking the ability to keep me enrolled, but rather I never planned to try at them. I sign up to most classes to see the videos and get surface-level understanding of a subject.

mark_l_watson2• 13 years ago

That was a well thought out article, a good read, and I don't agree with most of it. I think Cousera is Amazing.

The only thing I don't particularly like is the inflexibility on turning in assignments. I completed five classes but did not pass one because something happened work wise that caused me to turn it several assignments late and that really knocked down my score. Great class though.

I have also started four other classes that I dropped after 1-3 weeks. I am still glad I was exposed to some of the material for those classes, so still a positive experience.

Accepting the figure that 95% of students drop classes on average, I would like to know if most of those people in general felt their time had been worthwhile spent. That is the important question.

DennisP2• 13 years ago

I actually often "never really intended to seriously take the class in the first place." Coursera doesn't let you view the materials unless you sign up for the class, so I sign up for every class I might be interested in browsing a little. Then I can view the materials at leisure, even long after the class is finished.

On the other hand, I mostly completed the first algorithms course and Model Thinking, and while Daphne's class kicked my ass the first time, I'm already reviewing to tackle it again and try to finish it this time.

It is true though that the time commitment is horrendous, especially for Daphne's class, which estimates 15-20 hours per week! If I fall behind, I may finish slower than their schedule and miss the homeworks and final exam, in which case I'll officially be an incomplete but I'll still have gotten through all the material.

In short, retention is an irrelevant metric for online classes, especially for Coursera which has such a strong incentive to sign up "unseriously." My suggestion would be to keep accepting and grading homeworks and exams at any time, and give extra props on the certs for completing on time with the other students.

zallarak• 13 years ago

The title of this article is very exaggerated and needlessly so. The only reason you see awful retention numbers is because it's so easy to sign up. Even if I'm only partially interesting in a class, I'll sign up. I often sign up for 4 or 5 classes that I'm interested in and will finish 1 or 2 per term, just so I don't miss the chance to sign up later/have access to course materials.

If you have issues with the class, that's a personal problem and you simply need to work on your willpower.

timedoctor• 13 years ago

I think a major problem with retention is that the course is not a recognised qualification.

The next step for online education is get a real qualification that improves your job prospects through places like Coursera.

Also I think there are a bunch of other hacks such as collaborative motivation (getting a group of members to commit to each other to complete the course and automatically reporting to each other when they have completed each section where everyone in the group has collective responsibility to help each person in the group to complete the course.

tsumnia• 13 years ago

Retention rates are low in everything, not just online classes. Ultimately, its how the student reacts to the plateau. Anyone in any area of teaching, I'd highly recommend reading "Mastery" by George Leonard. In the book, he talks about 3 of learning types that can stifle progress: the Dabbler, the Obsessive, and the Hacker (not to be confused with the IT world's rough definition).

* The Dabbler will pick up tons of things, "overjoyed" about the newness that they are learning. Eventually, they hit their first plateau and the 'newness' becomes boring and they'll seek out another activity to experience the same 'newness' high. Maybe they'll come back to it later and see if they can progress again.

* The Obsessive will seek out perfection on this new activity, pouring over every YouTube video or article to figure out how to be better. When they plateau, they'll try harder; hoping for the same results. If the return isn't there, the Obsessive will look for another activity.

* The Hacker will actually ride out the plateau potentially indefinitely. In some cases, this will create progress eventually, but in others, it can become the cap of their learning capability (hence, 'a hack'). They just don't put the extra effort needed to progress further.

While all these types are negative, two-thirds drop the activity for something new, something they might be a natural at. From a business perspective, a monthly subscription favors Hacker-type customers; where a 'per course' pricing scheme favors Dabblers.

Steering back to online courses, I think the issue isn't 100% the site's fault; but that certain students will join for the excitement, but fan out eventually (Leonard goes on later to discuss how to better approach each of these types as the Teacher). I've signed up to all of the sites in the article, along with Udacity (where I'm currently taking the Web Dev course). For Udacity and edX, I've completed the courses I've signed up for; however at Duolingo, I stopped after the Basics 1 in Spanish. I want to learn Spanish, but the drive and time isn't there for me.

Jumping back to 'Mastery', Leonard talks about 5 keys that can help promote mastery and can be used to retain students: Instruction, Practice, Surrender, Intentionality, and The Edge.

* Instruction is obvious: have great teachers(not just through credentials, but are generally good AT teaching), something these sites offer tenfold.

* Practice is where I felt edX's course fell through. It was a lot of lecture, but the Finger Exercises were more the tradition 'regurgitation' of knowledge from traditional classes. Duolingo and Udacity (at least the Web course) seem to have this concept taken care of. Duolingo offers different ways to practice and at least with Web Development, you can to see the fruits of your labor with each class.

* Surrender is more for the student than the instructor, but talks about accepting you aren't a Master as something and so sometimes you need to take two steps back to take one step forward (in Aikido, we refer to this a 'Kihon Waza', or practicing the basics). So many people have the mindset of "If you're not doing it this way, you're doing it the wrong way", that this can stifle learning. As an instructor, thinking up new and creative ways to practice the basics can help alleviate this.

* Intentionality talks about visualizing the action, and while focused more on physical activities, could be utilized online. Give the students a view at the end results/expected return on their time. Sure, its dangling the proverbial 'carrot on a stick' in front of them, but it can help them sees where the fruits of their labor. If you can mix this with some of the Practice key, you could get better results (IMO).

* The Edge tip-toes the line between endless practice and pushing the envelope (though Leonard says it isn't). For a new student, this can be a little harder, considering they are still learning; but creation of 'outside the box' milestone/projects could benefit from this. The student's amassed this bit of knowledge; make them use it in a less than cookie cutter way. This would be a great undertaking by online courses, due to how do you 'personally' grade someone's submission, especially overtime? Maybe setting it up in a 'there is no right/wrong answer' way, namely because if a student takes a risk, they shouldn't be penalized for it.

Like I said, I'd recommend the book. I teach community college currently, and while most this is anecdotal opinion, I feel like it's helped me maintain engaged students.

nnq• 13 years ago

Why do people think retention is a good thing?! I think retention is actually bad even for classical education.

The lower the retention rate, the more the likelihood that they are following lots of courses at the same time or multitasking with something else. This means that people are "exploring" more! I think that "exploratory learning" has always been stifled by classical organized education and is one of the reasons why I hate most academic environments (yeah, they're cool if you're in one of the top 10% unis or in a "privileged" position", but not for the rest of cases...). I believe that, after a certain level of baseline knowledge, in any field, it's actually more important that someone learns "what they want/need/have inclination for/find more interesting" than that they "learn more"! Maybe more time exploring and less time actually uploading things to your mind is better (not "productive", just "better", and I said "fuck productivity" a lot lately because I found that it just doesn't lead to better anything). All the new ways of doing education make exploration easier (as in you can explore a lot without really "wasting" that much time), and we should take advantage of this!

auctiontheory• 13 years ago

Coursera courses are supposed to be college courses. College courses are a lot of work, and you have to get your reading and assignments done on time. This should not come as a surprise.

CodeAcademy (also wonderful) is a completely different animal. It is not comparable to Coursera.

What you're calling a bug (aka "dirty secret") of Coursera is actually a huge feature: that we can all, whatever our goals or our budget, sign up and check it out. Quite unlike an class at MIT or Duke, which is only available to a pre-selected few.

In short, the MOOCs, and specifically Coursera, are awesome. We are so lucky to have this opportunity. Quit yer bellyachin'.

jeffdavis• 13 years ago

The article seems to imply that, in an ideal world, all of these student would stay throughout the course. But I don't think it's clear that the churn is bad.

There is some lower bound to the amount of time that it takes to learn something (absent some educational revolution). Many people aren't even willing to pay that lower bound simply because they have other priorities in life. But they do like to sample (which is good), and potentially finish out a course if nothing else gets in the way.

Online courses are great because they lower the cost of sampling, so we shouldn't be surprised that there are more samplers, and fewer people finishing out the courses. If 1000 people complete an online course, then that's great, even if 99000 people signed up and disappeared a week later.

That being said, I'm pleased to see the specific criticisms offered in the article, and I hope they lead to a better balance for more people.

donniezazen• 13 years ago

Online education has no similar incentive as formal education has. And one of the most important aspect of online education for me is to learn what I find interesting.

lessnonymous• 13 years ago

I find most of the comments on this post fascinating. Probably partly due to the audience, and partly due to the slant the author took on the topic.

His bait is retention .. and that's something HN readers understand from a business POV. But retention isn't his issue. Retention is a problem for Coursera et al.

His problem (and mine) is that we (personally) found it hard to stay the course. And Coursera don't help us.

I've a full time job, two side projects, a small business and a family. I'm about to start the Gamification course and I probably wont finish it because I don't have time. Not because I don't want to, but because the course expects all students to spend 4-6 hours per week studying. Those weeks would be rare for me.

How hard would it be to offer a 2 hour-per-week version? Or whatever commitment I'm able to make? Then, when the inevitable happens, let me hit the 'pause' button.

michaelfeathers• 13 years ago

We could come up with a bunch of excuses about why online education will always have worse retention, or we could figure out ways to fix it.

Or we could ask whether it is a problem. I'm not sure that it is.

I don't think the numbers are surprising or unnatural at all. The thing to compare them against is not traditional education but rather the patterns of all people who take the autodidactic route.. every person who has ever picked up a book and not finished it. Even then, the comparison isn't perfect.

In fact, we could argue that low retention numbers are actually a good thing for non-credit courses. It means that the content is challenging and focused, and that people are able to discover early whether it is or isn't for them or cherry-pick the content that they need and move on.

If there are usability problems, fix them. But if participant retention got dramatically higher, I'd worry.

hojoff79• 13 years ago

I think the main issue is a matter of motiviation for the classes. And I do not mean people being lazy, but the original impetus behind signing up for these classes.

For the Coursera classes I did sign up for, I did it because I am interested in the subject area, know almost nothing currently and wanted to gain some functional understanding. I would say of the classes I signed up for, and probably 50% of those I have seen all the lectures. But then I go off and incorporate into side projects to test out / make sure I understand the materials. I have never turned in an assignment, (almost) never posted on a message board and never opened a final exam. Why should I? I am going straight to the application of the content into my projects, I've already achieved my goal of understanding and expanding my capabilities.

What reasons would someone have for wanting official completion of the course? On most courses, you do not gain any officially recognized credit, the certificate has limited (if any) commercial marketability for someone... so again, why would I want to go through the hassle of officially completing the course?

Two takeaways from these thoughts are the following: 1) How many people like myself have utilized the content of the course but are not seen as retained due to the fact they do not partake in the academic completion side? 2) What potential motivators can exist in the future to drive completion of the academic side of the coures? (college credit is the first one that comes to mind, but still does not help motivating many users who are above that age). What other drivers could come into being to incentivize?

Online Education's Dirty Secret - Awful Retention

Related Stories