Want to Remember Everything You'll Ever Learn? Surrender to This Algorithm

http://www.wired.com/medtech/health/magazine/16-05/ff_wozniak?currentPage=all

by mshafrir • 17 years ago

258 87 17 years ago

btilly4• 17 years ago

In grad school I managed to take advantage of the pacing effect in an educational setting. I was teaching linear algebra. What I did was make the homework incremental - 1/3 of homework on today's material, 1/3 on the previous week, and 1/3 anything in the course. Those thirds were in increasing order of difficulty.

I also started every class with a question/answer period. The rules were simple, the questioning will last at least 10 minutes, and you don't want me to ask the questions. :-) Anything that had come up in the questions that seemed to be a point of confusion was sure to be added to the next homework set.

I won't go into what else I did with that class, but the end result is worth thinking about. First note that I gave a ridiculously hard final. Other grad students who saw it thought that the class would bomb. Secondly they aced the test. What do I mean by aced? Well I had a bonus question which fellow grad students thought nobody would get. 70% of the class got that question, and a good fraction were over 100% on the test. So they must have studied hard, right? Nope. I ran into some students several months later. They told me that they tried to study for the final and stopped after a few minutes because it was useless, they knew everything. And several months later they still knew much of the material cold!

The thing is that none of what I did was very radical. The principles have been known for a century. Psychologists have been trying to get people to listen for that long. I learned about it in the 80s from a university course I watched on TV. (British Columbia had a TV channel devoted to lectures for correspondence courses.)

Yet, despite how dramatic the effects are, nobody listens and nobody takes advantage of it.

kiba• 17 years ago

This remind me of a drawing book called "Drawing on the Right Side of the Brain". It teaches people how to draw realistically using techniques like contour drawing among other stuff.

However, middle school art teachers seem stuck in their little art project. I am not sure about high school, but it doesn't look like they teach the skills.

cesare4• 17 years ago

> It teaches people how to draw realistically using techniques like contour drawing among other stuff.

A great book. One of the exercises is to copy a picture turned 180 degrees, so that you're drawing exactly what you're seeing without the interference of your mind's prior knowledge of the subject.

diiq• 17 years ago

Not in high school, and not in collegiate art school.

I have a copy that I pass around via samizdat, because the mandatory drawing classes here are useless to students without the basic principles.

(Also worth noting: 'The Memory Book' by Lucas and Loraynne. Now back in print, an falling-apart copy got me through high school. I can still memorize a deck of cards in under ten minutes.)

gila• 17 years ago

I took art in high school and my teacher used this technique, although getting kids to adhere to the style wasn't very easy at first.

I guess my teacher had an advantage with a psychology degree though.

tocomment2• 17 years ago

If you made this story into a blog post, people would listen. (I'm actually serious, it's an interesting story and a lot of people might benefit from hearing more of the details and the thought process behind it.

btilly4• 17 years ago

I'll think about it. I won't get around to it in the near future though, sorry.

Alex3917• 17 years ago

"If you made this story into a blog post, people would listen."

That's what I hate about people. What makes people pay attention isn't the ideas themselves, but whether it looks like you worked hard in writing it. I've made comments on HN that no one paid any attention to, and then turned them into blog posts and had them get to the top of Reddit or Digg or whatever. Exact same ideas, all I did was visually change the text to make it look more professionally laid out, and carefully rework some of the phrasing to make it sound better to the ear. (Plus add a couple really good first sentences and a good headline.)

I've found that if you actually leave a couple minor mistakes in your blog post it gets more votes, because it look like you're thinking at the edge of your intellectual ability, and people like that. It's the exact same thing Seth said in his post about the Chris Bliss Diss video:

http://sethgodin.typepad.com/seths_blog/2006/03/its_how_you_...

"Today, I got a video, featuring Jason, who just might be the best juggler I have ever seen. Same music, similar routine. Except... five balls. Not three, five. Infinitely more difficult. And Jason makes it look easy.

The thing is, even though I know how much more difficult Jason's routine is and how skilled he is, the very ease of his delivery makes it less likely an audience would give him that same ovation. Interesting how important effort seems to be."

gherlein• 17 years ago

It is a blog post!

elai• 17 years ago

Could you go into more details into SRS for math? "Fact Memorizing" type systems seem to fit well with things like supermemo (languages, law, history, biology, a bunch of sciences), but not so much with things like math (other than memorizing formulas). And did you really add homework questions like find the kernel of this subspace in the first few weeks of the course?

btilly4• 17 years ago

I'm not sure what details to go into. Being reminded of something as we're forgetting it pushes that thing into longer term memory. It also internalizes the lesson. This applies to both rote facts and intellectual ideas. If you think of it as fact memorization, then you're missing the real power.

There are a lot of things you need to learn to master a mathematical topic. You need to learn definitions and terminology. You need to learn key theorems. You should be able to apply procedures and derive formulas. At a higher level you need to learn available proof techniques and the key ideas behind important proofs.

Of course learning these things intellectually is not quite sufficient. You need skills as well. However failing to learn these things will keep you from learning the topic. And that is where most students fall down.

This is particularly true in subjects that build on themselves like mathematics. If you missed an important point earlier, you may develop a bigger misconception about a current topic that will make it impossible to learn a future one. Anyone who has tutored students should be aware of this.

As for what to review, students did not get the same questions over and over again. Instead I gave them questions from the same section over and over again. Since I only asked a few questions from the current day's section, there were always plenty of questions left to go back to for future homework sets. As a bonus doing many kinds of questions at the same time makes connections obvious. When a single homework set asks you to solve a set of equations, find a basis for a vector space, and invert a matrix, you see how they tie together.

gwern4• 17 years ago

Speaking from my personal experience: you can get very far just by dumping a few hundred questions/examples/small-problems into your favorite SRS. That's so many that your mind, rather than memorize them, will instead learn just how to do them. This works as well for math or CS as for languages.

I've proposed for Mnemosyne 2.0 that there be a card type which is just some Python code which is evaluated: http://groups.google.com/group/mnemosyne-proj-users/browse_t... The idea is that if you want to learn, say, multiplication, you would write some Python code that generates 2 random ints (as the question) and their product (as the answer), and now the user must solve it. You get much the same benefit as if you generated several hundred problems by hand or by script, but without cluttering your deck or risking memorizing some.

stevenj• 17 years ago

I think I learned more about retention (for me personally) by reading your comment than by reading the article.

Thanks for sharing.

imajes• 17 years ago

I wish you were my teacher. That's a theory of education I've been pondering about for ages. Glad to see it formalized like this. Please consider running for office and fixing how we go about learning. :)

btilly4• 17 years ago

Actually you should be glad that I'm not your teacher. I set a number of goals for that course around learning, and forgot to set any around making the students happy. I met the goals I set but...

I'm sure that if I had stayed in academia I would have sorted out those kinds of issues. However it is clear to me that I'm much happier not being an academic.

adinobro• 17 years ago

You might want to have a look at Pimsleur. The theory described is almost the same except Pimsleur was studying language learning... http://en.wikipedia.org/wiki/Pimsleur_language_learning_syst...

ghshephard• 17 years ago

What an excellent Hacker News Post. Please do write up a bit more about your method, what your students got out of the course specifically, and whether you've had success with it in other places. You have a clean, readable, and interesting writing style.

I think we can all recall a teacher or teachers in our life who truly made a difference in teaching us something - sometimes _despite_ our interest (or lack thereof) of learning it.

I was forced in University to take a course in Propositional Calculus (it was a breadth requirement for computing science) - it was mid-day so I attended the class (unlike early morning classes in which my attendance was abysmal) - I went in with zero desire - no coding, or even math. But, the timing was right so I just sat in the class.

The instructor spent thirteen weeks walking us through Reductio Ad absurdum proof, tautologies, conditional proofs, etc.. He gave us homework assignments that built on earlier knowledge, and just as we were getting into new material, he'd throw some earlier sequent proofs at us to solve and refresh our memories.

Here is the thing - I had _no_ interest in this course, the only thing going for it was it's timing, and repetition. I was _killing_ myself trying to learn integration by parts, linear algebra, discrete math, spending 5 hours outside of class for every hour in class trying to master those other course, and basically just sitting in this one distraction course. I was a C+ student in the math courses (after _massive_ effort) - I walked out of the Propositional Calculus course with an A+.

Sixteen years later I look back at University and can honestly say that the most useful course, that I _continue_ to have rock solid knowledge of PLUS use almost every single hour of the day was that stupid Philosphy 210 Propositional Calculus course. For the life of me I can't remember much of graph theory, and wouldn't be able to invert a matrix if you held a gun to my head, but, I'll sit in a meeting and see a ton of possible alternative and immediately start applying Disjunction elimination to break us out of a log jam.

There really is something to this spacing method.

joe_the_user2• 17 years ago

It could also be that propositional calculus is dead simple - or at least that has always been my impression of it. Calculus and linear algebra work on complex problems in a large domain. Propositional calculus has a small domain and fairly simple theories.

I think in general, the knowledge domain is important. Linear algebra is a very unified field by itself but when you get courses that involve a grab bag of algebra, linear algebra, trigonometry and calculus, I think the "stew" they present is incredibly hard to comprehend on a high level and students wind-up either failing or learning by a horrible kind of rote.

Uh, this is from biased personal experience. I indeed only taught one college math course when I doing my math MA, it was "Trignometry". It was a complete failure. I still blame the "grab bag" quality of the material but perhaps I was to blame instead.

TobiasCassell2• 17 years ago

Hey OP, was your technique influenced by Mihaly Csikszentmihalyi's Flow? I'm referring to the relationship between ability level and challenge. Do I see similarities with your clearly effective teaching style and some of the elements Video Game designers have almost unanimously implemented to great success?

butterfi• 17 years ago

I'm not sure "the relationship between ability level and challenge" is what MC means by flow, if that's what you're saying.

Q: "What is "the flow experience" and what does it have to do with motivation? A: The flow experience is when a person is completely involved in what he or she is doing, when the concentration is very high, when the person knows moment by moment what the next steps should be, like if you are playing tennis, you know where you want the ball to go, if you are playing a musical instrument you know what notes you want to play, every millisecond, almost. And you get feedback to what you're doing. That is, if you're playing music, you can hear whether what you are trying to do is coming out right or in tennis you see where the ball goes and so on. So there's concentration, clear goals, feedback, there is the feeling that what you can do is more or less in balance with what needs to be done, that is, challenges and skills are pretty much in balance."

http://www.edutopia.org/mihaly-csikszentmihalyi-motivating-p...

fburnaby• 17 years ago

I'm just about to start lecturing the tutorials for a linear algebra class next week. The prof that is running the course taught it to me 4 years ago and is the reason I'm now in grad school. She did more or less the same thing that you mention, and it definitely works. I remember all of this material from back when I took the class; it's still familiar and I use it all the time.

But despite being very useful for a half to two-thirds of her students, there is always about 1/3 of the class who does terrible in this course. How did you get these people to keep up?

Anyway, I'm glad you posted this. It's a good pick-me-up motivator as I'm about to be trying to do the same thing!

cgs2• 17 years ago

I've toyed with the idea of keeping a practice journal, but I've always been too lazy to stick to it. Your comment has inspired me though! This way, I can go back X number of days and review whatever I was working on then.

xccx• 17 years ago

is this smart.fm?

http://www.techcrunch.com/2009/09/10/cerego-raises-3-4-milli...

nwinter2• 17 years ago

Yes. It used to be called iKnow, and now iKnow is just an app on smart.fm, which is run by Cerego.

juvenn4• 17 years ago

I think a good self-teacher would be better a teacher than who just hold some fancy degrees. Since good self-teachers always know what students need, and where they failed to grasp the essential, and most importantly how to get them out of obstacles.

And I bet you must be a good self-teacher. I wish I have ever had you as my teacher!

antipaganda3• 17 years ago

There is one glaring fault that a self-taught teacher may have; people learn in different ways. What works for one may not work for all. So the self-taught teacher may not realise that their own personal methods are not necessarily valid for everyone.

eob2• 17 years ago

What is a "self-teacher"?

The advantage of holding "fancy degrees" is usually that you have been exposed to a wide variety of viewpoints and insignt that self-taught people might have. People tend to study what they are interested in. Unfortunately, this can produce self-taught academics with horse blinders on.

Imaging you only got your news from Reddit & Huffington Post, saying you were becoming a self-taught news commentator. You'd probably end up incredibly informed... about liberal viewpoints, perhaps not realizing that a large portion of the country doesn't share them. Getting taught in a formal education system (providing the system works) would expose you to a wider variety of views.

morphir• 17 years ago

I recently heard that newly educated teachers come to schools with a strong desire to teach their specialties. And to transmit some of their true passion over to their students. But since being a teacher (here in Norway) means they have to administrate and organize the kids so much, they have little energy left for teaching. Meaning the teacher appears as a boring, pale fuck (not very inspiring). This is of course a organizational problem, and not a pedagogical one.

hyperbovine2• 17 years ago

Otoh I have always assumed pretty much everyone is pale in Norway. :-)

scharan• 17 years ago

Please do a more detailed version, if it makes sense to do so. You actually managed to say a lot in this little space. I had a teacher too who did this on a small scale - the first 5 mins was a summary of the previous calls. The first day of each week he summarized the entire chapter or section in the first 10 mins. This bird's eye view of the material went a long way in keeping the concepts 'live',

msluyter• 17 years ago

Seems related to this Wired article that recently was reposted on Reddit:

http://www.wired.com/medtech/health/magazine/16-05/ff_woznia...

Bottom line is that to maximize retention, there's an optimal time for being re-exposed to information: right before you're about to forget it. Seems like re-covering the previous material in the class may have accomplished this.

wsprague• 17 years ago

The question of why is there so much crappy education is interesting. I think part of the answer is that society is well-served by the failings of the educational system in that widespread educational crappiness helps support the class system. I think it would be fairly straightforward to turn 90% of the population into well educated upper middle class types, using techniques like btilly describes...

... But if we did that, who would drink Coors and drive forklift for Walmart and not complain about it?

So I think that a huge function of the educational system is to educate a large part of the population badly. I think that the teacher training system serves this is as well, by selecting for mediocre teachers and then making them more mediocre via training. Additionally, in the schools themselves a lot of effort is made to classify young people into dumb and smart categories, usually unfairly (http://en.wikipedia.org/wiki/Pygmalion_effect), and this classification stamps them for the rest of their lives and creates a population of hopeless lower class workers.

If there are SOME good teachers in the system like btilly, then SOME kids go on to get out of their class, which is perfect in that (1) we need to claim that it is possible to pull yourself up by your bootstraps, and (2) we need to recruit SOME kids to grow up to fill management positions, but not too many that they all can't find careers. Historically, underemployed educated people are the people who become union leaders.

The beauty of it is that the people who fail think it is their fault!

(Sorry to rant about the educational / class system in general, rather than the topic of how to teach advanced math, but I couldn't stop myself.)

Call me paranoid, but there you go.

seanb• 17 years ago

Interesting idea.

"But if we did that, who would drink Coors and drive forklift for Walmart and not complain about it?"

Robots? If the education system elevated more of the population, there would be a shortage of blue-collar workers, more pressure for system automation, and more well-educated minds to develop the robots. Of course, there would be no shortage of immigrants to fill the blue-collar positions, but perhaps in the long term, a "better" equilibrium would be reached.

kalid• 17 years ago

That's really awesome. I'm a huge proponent of finding the real, actual insights and lessons behind a subject (plug: I blog at betterexplained.com).

The dirty little secret of education is that it's "ok" to cram and forget. Most classes reward that; education is about "being able to do something once" vs. actually remembering and applying it.

I hope you write more about your findings!

nostrademons• 17 years ago

When I was learning to play the violin, my teacher had a very strict method for going through the repertoire. You had your working piece, which was supposed to be hard - it was stretching the abilities of what you could do as a violinist. You had your polishing piece, which was your previous working piece where you were finessing all the fine points of technique and musicality. You had review, which was all the other pieces you'd learned so far. And you had your preps, which were little passages (4 measures or so) from your working piece that were so hard that you played them over and over again, way slowed down, until you got them right and could speed them up and incorporate them into your working piece.

I've tried to do something similar with my work so far - a working codebase that I'm just learning, a polishing codebase that I basically know my way around in, and various tweaks and bugfixes that I have to do for previous projects. It's a bit harder in the corporate world though - while Google engineers are given a lot of freedom to pick their projects, they're still subject to the needs of the business, and sometimes a project will come up that's such a great opportunity for professional visibility that I'd want to take it even if it involves working hard on something I already know well and dropping the polishing of stuff I just learned.

martincmartin• 17 years ago

It's sad that professors are promoted for research, rather than either research or teaching. It amazes me that the private U.S. universities that are most highly ranked for undergraduate education are the ones with the best research.

pw• 17 years ago

I'd suggest the book Why Don't Students Like School? (http://www.amazon.com/Why-Dont-Students-Like-School/dp/04702...) to those interested in applying the findings of cognitive science to their own learning. I'm finding it very useful in that regard.

araneae• 17 years ago

I'm wondering what type of university you were at that let you write a final. The most flexibility given to grad students in introductory courses I've seen is, at most, to write a quiz or two. Most don't even let you write homeworks.

jedliu• 17 years ago

Actually, at my department (a "top-10" school for CS), for each course we TA, we're expected to write at least one problem set and come up with at least one question for each exam given. There are also half-courses on intro to Unix or intro to C++ that are taught entirely by grad students.

I suspect the OP was teaching an intro-level course, designed to get freshmen up to speed on basic linear algebra, filling any gaps they might have had in their high school curriculum.

cesare4• 17 years ago

Once you've finished the article, there's a lot of reading material on the SuperMemo website:

http://www.supermemo.com/english/contents.htm#Articles

Really interesting stuff. The website and the software user interface (at least judging from the screenshots) are terrible, though.

billswift• 17 years ago

You beat to it. I've had the site on my favorites list for months, it's got more information than the article, but it wasn't anywhere near convincing enough to get me to actually spend any money. Maybe I'll go through it again and see.

cesare4• 17 years ago

Have you tried Anki? (it's open source, see GiraffeNecktie comment). I've just downloaded it. There are also some decks shared by people (mostly for natural languages) that you can download from the app itself.

GiraffeNecktie• 17 years ago

This article is written as if SuperMemo were the only software that uses spaced repetition (SRS) algorithms for learning. Actually there are quite a number of other programs out there including some that are free or open source. I've used SuperMemo for years on my Palm and found it very useful for learning French but the software (at least for the Palm) hasn't been updated for many years and doesn't give the user any control whatsoever over how the program is used. For learning Chinese I've moved to a commercial product called Pleco on the Palm which is basically a dictionary but also has elaborate SRS functionality, it's really light years ahead of SuperMemo. Other more generic SRS programs include Anki and Mnemosyne. Unlike SuperMemo, it's easy to migrate your flashcards among these other applications. One other point, SRS is a huge help to learning vocabulary but learning words is only one part of mastering a language.

cesare4• 17 years ago

Thanks. I was just wondering if there was a similar open source project.

Anki: http://ichi2.net/anki/

Wikipedia entry about this technique: http://en.wikipedia.org/wiki/Spaced_repetition

sandGorgon• 17 years ago

Mnemosyne is actually very good

http://www.mnemosyne-proj.org/

it works on Win, Mac and Linux - written in Python

asdf1011• 17 years ago

Anki is a great spaced repetition tool; despite only using it for 20 minutes each day to remember the Japanese Kanji, it has resulted in me remembering a lot of characters (but English -> Kanji and vice versa).

If you're looking to memorise a lot of data, it's definitely worth a look (runs on Mac, Windows, and Linux).

mwerty2• 17 years ago

I tried anki and while I liked it, I want to be interrupted while working on code/browsing to test/reinforce. Is there an app that does that?

hyperbovine2• 17 years ago

+1 for Anki. I loaded up a list of GRE vocabulary words in Anki, studied it for three months, and got a 1600. I probably spent two hours studying for other portions of the test. Takeaways: 1) SRS works; 2) the GRE is a joke :)

rgrieselhuber• 17 years ago

There is a company based here in Japan called Smart.fm (formerly iKnow) made up of really great people doing similar work.

klipt• 17 years ago

There's at least one maker of language courses (Pimsleur) which use the pacing effect. Unfortunately they're audio only, which at least for me isn't nearly as effective as a combination of audio and writing.

jwecker• 17 years ago

I've used these- really brilliant. Takes the pacing effect but makes it conversational so it really seems to flow. The trade-off, of course, is that it has no idea how well _you_ have memorized various things (i.e., being prerecorded audio, it is generalized).

tocomment2• 17 years ago

Is there someone I can try this out for free? Are there ready made flashcards for something useful to learn?

tigerthink2• 17 years ago

As mentioned above, Anki and Supermemo are two free implementations. As for ready-made flashcards, here are some I made for a calculus class:

http://www.spicyelephant.com/decks/523 http://www.spicyelephant.com/decks/577 http://www.spicyelephant.com/decks/659

You probably have to register for an account if you want the site to apply supermemo-type techniques. I don't recommend using the site in the long term because it will eventually force you to register for $20.

gwern4• 17 years ago

Supermemo isn't really free - maybe you meant 'Anki & Mnemosyne'? That's the big 2 in the Free SRS space.

r70002• 17 years ago

You can try my free implementation here: http://flashcarddb.com. You won't be forced to pay for it to keep working.

nwinter2• 17 years ago

This is what we do at http://www.skritter.com for Chinese and Japanese. Spaced repetition is definitely still in the geek realm, though. Even after automating almost everything (vocab creation, addition, even grading), there are still a lot of people who don't want to practice on the computer, or aren't as interested in efficiency of the learning process. I think as spaced repetition moves away from raw SRS like SuperMemo and toward integration with classrooms, textbooks, and other software that provides content as well, we will start to see more widespread adoption and recognition of its power.

There was a recent thread here about whether SRS will evolve to appeal to non-programmers: http://globalmaverick.org/archives/169-repetitio-est-mater-s...

idigthought• 17 years ago

Is there a good os x and iphone application for this? I'm looking particularly for one that supports images and LaTeX.

gwern4• 17 years ago

There are a number of SRS clones already produced for the iPhone in the store (Mnemosyne, I understand, works fine on OS X); there's also a Mnemosyne clone in early development for the iPhone.

cgs2• 17 years ago

One thing I've never bothered to do is learn an API of X language really well. Because, I thought, I'll just look up whatever I don't know. But I'm constantly stopping to look things up and it really slows me down. Now I'm beginning to think there's really some value in having a strong command of an API. Has anyone tried using an SRS tool to help with this? Or is it simply a matter of not having enough experience with X language?

chimariko• 17 years ago

I am not sure about SRS for learning API of a language. However I have recently started to use mind maps for that. Whenever I come across a new language or library I take an A4 and a pen and put down all basic concepts. It helps to grasp the big picture of the system and to see the interconnections of different blocks of code. I prefer mind maps to the standard linear up-down way of writing stuff down because the schematic representation utilized in mind mapping eases the twist between the way information is stored on paper and (presumably) in the human memory.

All in all I find spending some time on studying docs useful in contrast to the pure work-by-doing approach.

gwern4• 17 years ago

Actually, the SuperMemo site covers just this topic: http://www.supermemo.com/articles/programming.htm

tl;dr: if you will spend more than 5 minutes looking it up or trying to remember it over the next 10/20 years, then it's a net gain to memorize it in a SRS.

I find this plausible, but I'm not sure 5 minutes is right - I understand from Anki users (Anki's statistics include 'how much total time ever have I spent looking at/on this particular card') that a card will cost you more than 5 minutes, and more like 10 minutes.

gfodor• 17 years ago

iPhone version coming out in a few weeks. (I emailed him about this.)

Want to Remember Everything You'll Ever Learn? Surrender to This Algorithm

Related Stories