A Year of Spaced Repetition Software in the Classroom

http://lesswrong.com/r/discussion/lw/mfm/a_year_of_spaced_repetition_software_in_the/

by eurg • 11 years ago

198 58 11 years ago

hiq2• 11 years ago

For those interested in memory and learning and who want some hints on why SRS work, Make It Stick[1] is probably a must-read.

[1] http://makeitstick.net/

roflmyeggo2• 11 years ago

Great book. Interleaving contexts, spaced repetition, etc. - all great concepts.

The most important takeaway for me was accepting that the tougher something is to remember, the greater the benefit will be when we actually remember it.

cJ0th3• 11 years ago

I overestimated the effect of spaced repetition. For six months, I used Anki every day to recap Chinese characters. At the end of the term I was perfectly prepared for the test. Afterwards, I didn't study my flash cards again. Only a few weeks later I found out that I've forgotten most characters or at least some of their details. I expected that after six months of daily practice (about 20 minutes per day) I would have stored at least some characters in my long term memory.

jliechti1• 11 years ago

What method did you use to initially learn the characters? If you just used Anki, I could see your result as making sense.

I used Heisig's Remembering the Hanzi to learn the characters and then would practice them in a space repetition system. A few years later I can still write most of them. His method has you create a story and make mental image associations for each character. Eventually you forget the stories over time, but the character usually sticks.

cJ0th3• 11 years ago

What method did you use to initially learn the characters?

We were introduced to the characters in class and subsequently learned how to write them. So after being introduced to a character I repeated it on paper for a couple of lines. I repeated this exercise from time to time. I also read the stories of the text book out loud and listened to the recording of the texts. But as I've pointed out in my other post, you have to have context to really learn a language. Just the fact that I used several types of learning doesn't mean that they weren't kind of shallow.

The technique you've delineated above certainly sounds like a better approach.

hiq2• 11 years ago

Anki is great to help learning languages, however I think you have to make some special effort to actually use the words (in full sentences) when you learn them if you want to be able to reuse them in the future. One-to-one flash cards only helps for basic information retention. For example I find kanas much easier to learn than kanjis, as the former translate perfectly as plain syllables in western languages, while the latter can be far more subtle.

Also after six months, if you're successful in your reviews, Anki should not expect you to review your (old) cards before several weeks, so it's strange that you forgot most of them for your test if it occurred in that range. It can also depends on the type of test I guess, how was it?

dllthomas• 11 years ago

"Anki is great to help learning languages, however I think you have to make some special effort to actually use the words (in full sentences) when you learn them if you want to be able to reuse them in the future. One-to-one flash cards only helps for basic information retention."

Try a different Anki deck, maybe. I've been studying Polish, and a good portion of my cards are phrases or full sentences.

mjklin2• 11 years ago

The kanjis are far more subtle

In other words, they have to be memorized. No connection between writing and pronunciation.

cJ0th3• 11 years ago

Also after six months, if you're successful in your reviews, Anki should not expect you to review your (old) cards before several weeks, so it's strange that you forgot most of them for your test if it occurred in that range.

The test went well. It was some weeks after the test that I began forgetting.

however I think you have to make some special effort to actually use the words (in full sentences) when you learn them if you want to be able to reuse them in the future.

yeah, I absolutely agree. I think context is key. Nothing beats being forced to use a language.

mjklin2• 11 years ago

I find that in my language learning, when I have a question about a vocabulary word or a grammar point, I have to go back in my memory to the social situation in which I first heard it. Whether on TV or in conversation. I personally don't get the same hit to my memory from flash cards, though it may just be me.

tokenadult• 11 years ago

I began my study of Chinese in 1975. I got to be good enough in reading Chinese to set the ceiling of scores obtained in an experimental administration of a test for Chinese language for speakers of English. (The test was not further developed by the test developers, but I saw the summary of norming administration scores.) I worked for quite a few years as a Chinese-English interpreter for government-sponsored people-to-people visits to the United States. I was just reading Chinese-language newspapers via Google News just before bopping over here to Hacker News.

Simply put, the biggest single mistake learners of Chinese make in learning Chinese characters is learning them only in isolation and not also by reading connected Chinese-language text in which Chinese characters appear in "compounds" (meaningful Chinese words). The full argument for doing this is developed by the late John DeFrancis (a brilliant language teacher and textbook author) in the front matter of his book Beginning Chinese Reader,[1] which is still a very worthwhile book decades after it was first published. DeFrancis built in a lot of spaced repetition in the book lessons, back in the days before "spaced repetition" was a trendy term, but he crucially also makes sure the lesson texts illustrate the various ways that Chinese characters enter into compounds to form high-frequency words in the modern Chinese vocabulary. Try it if you'd like to more Chinese in the most efficient way possible.

[1] http://www.amazon.com/Beginning-Chinese-Reader-Part/dp/03000...

P.S. for a more extensive discussion of how to learn natural human languages effectively, see a comment of mine from about two years ago that was very popular with other readers of HN:

https://news.ycombinator.com/item?id=6302276#up_6302816

mraison• 11 years ago

I'm really glad to see such an encouraging and useful comment on HN.

I'm right in the middle of the John DeFrancis series (just started Intermediate Chinese Reader volume 2). These books seem to be almost forgotten and I haven't found any modern equivalent, so I was starting to feel slightly lonely in my quest for Chinese reading fluency :) I started them after trying

* regular flashcards: not very effective for me, as all the accumulated knowledge would disappear very quickly after I stopped practicing

* "modern" graded readers: they're nice, but far less structured than the DeFrancis' series, and they often stop at a beginner/intermediate level.

What's truly unique about the DeFrancis' series compared to other graded readers is that the texts are adapted to the vocabulary lists, rather than the other way around. Therefore characters and compounds can be introduced logically, in a coherent manner: for example each new character comes with a cluster of words that use this character, making the whole much easier to remember. The downside is that the texts often feel contrived and boring, but if you're serious about learning to read it feels much more effective and gratifying on the long term. The approach used by most other readers is to start from a simple and enjoyable story, and then adding a glossary containing the more advanced words in the story. But although the story itself is more natural and enjoyable for the reader, the vocabulary itself is much harder to remember.

It would be awesome if someone was willing to put in all the hard work it took to craft the DeFrancis' series, and write something similar with a more modern and less academia-focused content.

visarga2• 11 years ago

I used Pimsleur to learn Japanese. They use a kind of SRS where words don't get quizzed directly, but embedded in small phrases. Using words in context made me more sure of myself and I was immediately able to use them in conversations or understand them in videos. With each phrase more than one concept could be refreshed. I think it is essential to put small phrases in order to develop the vocabulary.

Also, for absolute beginners this method has the advantage of forming a solid understanding of the core of the language (those 1000 words that see over 50% usage in regular conversation) and also learn accent really well - cards being in audio format. It feels as if I were a small child learning his native language.

dfan• 11 years ago

Yeah, the "downside" of spaced repetition is that you can't stop and expect it to stick (unless you have other means of keeping your memory fresh, such as actually using a language you've been learning). I have some subjects in my stack of cards (10,014 and counting!) that I'm not really so interested in anymore, but I keep them around because I'm paranoid that in a few years I'll want to pick them up again and will have to start from scratch.

It's not really so much a downside as an unrealistic expectation, like expecting to keep weight off after you've stopped dieting.

laichzeit02• 11 years ago

I expected that after six months of daily practice (about 20 minutes per day) I would have stored at least some characters in my long term memory.

Yeah, it doesn't work that way with SRS. If you're not actively using what you learned with your SRS (like say using Chinese on a daily basis) of course you're going to forget it! It's the same with bodybuilding: You have to constantly train your muscles and eat enough calories or they will shrink back to what they were before. It's not like you can gym, get as big as you want and then think if you stop you'll retain whatever you built :)

pella• 11 years ago

classics:

Sivers: "Memorizing a programming language using spaced repetition software" ( 2013)

https://sivers.org/srs ( HN: https://news.ycombinator.com/item?id=5015183 )

---

Janki Method: Using spaced repetition systems to learn and retain technical knowledge.

http://www.jackkinsella.ie/2011/12/05/janki-method.html

https://www.oxbridgenotes.com/articles/janki_method_refined

---

Gwern: SPACED REPETITION : http://www.gwern.net/Spaced%20repetition

pcote3• 11 years ago

At the last programming job I was fired from, I was deemed to be a slow and all around poor performer. About a year ago, I started doing my own version of Jack Kinsella's Janki method. As a result, I've leveled up to what some call a "full stack" web engineer.

SRS is a fun way to learn or else I wouldn't have kept at it so long. I would even go as far as saying it's made my personal projects a little bit easier for me to execute on. However.....

What I don't know is how applicable this is in the "real world". SRS systems don't translate well in the "Learn-in-5-days-or-die" approaches to IT training. As for benefits for work-related development and maintenance, I don't have a clue one way or the other.

asimilator3• 11 years ago

I'm surprised that memory or memorization would be the key (for some people? for yourself?) to learn a programming language.

I've studied a number of different subjects in my life, from foreign languages to computer science to math, etc. I clearly see the benefit of SRS and necessity of memorization in foreign languages (vocab is so critical, and there's really no way to avoid straight memorization of it), but I don't see how you could effectively encode your knowledge of a programming language on to flash cards. And I'm not sure why you'd want to when you can easily look up things in the documentation or on Stackoverflow.

1123581321• 11 years ago

I used to be very excited about spaced repetition, but haven't taken it as seriously in several years. Yes, it is superior to more ordinary methods of quizzing vocabulary, and I would still use it if I had to pass a vocabulary/fact test on a subject I didn't care about.

As a method of developing real competency in a subject or practice, though, it offers none of the benefits of immersion as it keeps the student just on the edge of full engagement (and if we add immersion, we lose the time-savings that spaced repetition offers.) It introduces a fragility into the student's schedule that make lapses in 'discipline' more likely (not everyone has a teacher and regular subject periods to keep them on task.) For anything beyond basic key-value learning, it requires too many contortions and restatements of the subject -- to fit the format -- to be feasible for many students/subjects. The packaged decks available online help but usually don't meet the needs of the specific course or subject concentration the student requires. We can introduce variability to space how to solve a problem instead of a 'what' answer, but then we need to consider muscle memory, developing intuition and other things that the spaced repetition model does not address.

flowlikeh2o• 11 years ago

This is why SRS is a tool but not a complete solution. In medical school, there is such an enormous amount of material to commit to memory (specifically for board exams) that a lot of students find SRS very helpful. It is possible (I did this for a little bit in the beginning) to rely on it too much and therefore not be able to make the connections needed, be it on an exam or in real life. There's an art to making good flashcards- knowing which context to test a fact in, how to give enough context but not too much, etc. Also, there is little point to memorizing cards without understanding the material first. If I struggle with a card, I know I need to review the material, develop a mnemonic, etc.

danmaz74• 11 years ago

I think SRS is useful to learn "data", not competences. Learning data has it own usefulness.

grovulent2• 11 years ago

I disagree with this in part. I'm using spaced repetition to learn maths - and without it I just wouldn't progress at all. My natural aptitude isn't great.

Can you say a little bit more about what you mean by immersion? Do you mean - say, having a teacher who can show you a problem from multiple angles, allowing you to triangulate your understanding by asking numerous questions... etc? Things like this?

I do agree that in terms of developing immediate understanding - having access to this environment allows you to obtain understanding quicker. And I agree that it doesn't cohere well with spaced repetition learning.

However, the problematic thing with learning with this way is that unless you go on to use it daily, you lose almost all of it. So all you manage to do is pass tests throughout your life - never developing a full suite of knowledge that you can deploy. What an enormous waste of time!

Given that the energy needed to learn/recall an individual card drops exponentially over time, over the long term this is a vastly more efficient process for retaining information.

The only question is whether it can be used to learn complex and abstract ideas. I believe it can.

Here's how I typically progress with learning a difficult, abstract mathematical concept or problem.

First pass - I'll rote memorise a solution.

Then after a couple of weeks - I'll typically forget various parts of the solution. More often than not these will be sub-problems that I don't completely understand. So I'll create new cards that will provide training on these smaller sub-tasks. In the meantime, I'll rote learn the solution to the larger problem again.

What I find is that no matter how many times I rote learn a complex solution, eventually I typically forget it. But the smaller, easier concepts stick. Eventually my rote knowledge gets swapped out progressively by the smaller units of understanding and I end up being able to work through the solution without remembering it as a whole at all.

This is a frustratingly slow process in the beginning - particularly if you are starting at the very beginning of a new field (as I am with maths). But in my experience it's worth it. For the first time in my life I feel I'm actually really learning a topic in a way that I'll truly own.

voltagex_• 11 years ago

Do you have an example of some cards you're using? Was there an existing set or did you develop them yourself?

laichzeit02• 11 years ago

I've found SRS (with Supermemo) is indispensable for learning vocabulary of languages in which immersion is not possible or too impractical e.g. ancient Greek/Latin.

As you say, it requires discipline; but if something is important you'll find the time, if not you'll find an excuse.

qznc• 11 years ago

Sounds like an interesting startup idea: SRS for classrooms

Probably not a millionaire-in-5-years or one-man-lifestyle business. More of a improve-the-world kind.

cthalupa2• 11 years ago

I've been using SRS to learn 日本語 for some time now. Prior to this, my ability to remember kanji was basically nil. I could spend tons of time writing it over and over again, and nearly immediately forget how to write it, and shortly after, forget how to read it.

I switched to using Anki, and the reading at least became quite cemented - I still struggle with writing at times, but it's a significant improvement over what it was previously.

Daily study is hugely important - if I go out and have a crazy weekend, I hate myself Monday, and my study session becomes 3 or 4 times longer than it would otherwise.

MichaelGG• 11 years ago

Have you looked at "Remembering the Kanji"? I was skeptical when reading "Remebering the Kana", but halfway through it I found that despite the silliness, it worked. I tried the Kanji book for a bit and was fairly surprised how well it was working.

cthalupa2• 11 years ago

I'm not a huge fan of the Heisig method. I started off with it, but I realized that a lot of the time, the key word I was learning for the kanji had absolutely nothing to do with any meaning or reading for it.

What I currently do is learn new kanji via Kanji in Context, and then create cards for them in my anki deck as I have learned them.

hudibras• 11 years ago

Recommendation: "The Kodansha Kanji Learner's Course" by Andrew Scott Conning is a modern version of RTK. Same concept, but better organization, more kanji, and a more natural layout and learning order.

roflmyeggo2• 11 years ago

As someone who has dabbled in both biology/chemistry and now computer science academically, my thoughts on flash cards and spaced repetition software has changed over the years.

On one side you have the biology/chemistry guys who tend to use a lot of flashcards, and have a reputation for "just memorizing" facts. Then on the other end you have the physics/computer science guys who often refer to themselves as "lazy" for lack of memorizing and have a reputation for only focusing on conceptual ideas.

It seems to me that the people who succeed utilize a mixture of both approaches ("memorization" and "conceptual understanding"). I'm not sure when these two approaches suddenly became mutually exclusive.

As an example, although there were some classes in Biology that required a heavy amount of memorization, without the flash cards it would have been very hard for me to make the conceptual connections during lectures, while reading, etc.

Knowledge and understanding is kind of like a spider web, the more you have, the more you are able to grasp.

Purely focusing on conceptualizing the topics at hand has never worked out as well as I had hoped.

tfigment• 11 years ago

The only class I ever used flash cards was in Organic Chemistry. I do not think they helped long term memory at all as I recall next to nothing from that. If anything the act of creating the cards helped more than anything. I think this might be where the joining of the two sort of happens as you indicate.

To further that point, during exams some classes allowed 1 sheet of paper with anything you can put on it. I created some super dense cheat sheets in my time and I am fairly sure that I never referenced them but once or twice in 4 years of school. But again the active of creating them and sorting out important and unimportant information was key in solidifying knowledge for the long term. If someone handed me those cards or sheet they would have been useless to me in the long run is at least how I see it.

The other thing I learned in school was anything I crammed for I knew I was not going to learn. It may have been enough to get through the next couple of days but it was not going to stick. I took some summer school courses and found it to be a waste as it was too much like cramming as it was so condensed.

I guess I did well in school by using something like SRS which was allowing time between study sessions for information to settle and allow subsequent review after time. There is probably more than one reason for university courses to be on alternating days as it probably facilitates this kind of thinking. The only thing is that SRS seems to be focused on much shorter time periods than a day or 2.

chucksmash• 11 years ago

What you say about the act of making the cards themselves rings true to me.

When I was preparing for the GRE in 2010 I first wrote down all the words and definitions from both the Princeton Review and Kaplan GRE vocab books on a legal pad. That didn't work as well as I'd hoped for quizzing (it's a pain to try not to read 90% of the page) so I wrote a really basic data entry script in Python that I used to input the words/defs and then a second one that would generate random questions based off of them (definition + 4 words as MC).

By the time I was done with typing them all in and debugging my little quiz script, I'd say I already had active recall on >90% of them. Having that working quizzing tool at the end to practice recognition was just icing on the cake - creating it in the first place was more useful to me.

patrickmclaren• 11 years ago

A couple months ago I put together a spaced repetition app (https://www.memoread.io/) aimed at less tech savvy students. At the time it didn't meet much success, although I may revisit it in the future.

bane• 11 years ago

There's two major parts to learning a language: recognition and production. SRS can be quite good as a tool to aid in recognition, but I've found it to be lousy for production.

This is a classic problem in language pedagogy.

Some old comment of mine from almost a year ago https://news.ycombinator.com/item?id=8510165

https://news.ycombinator.com/item?id=8251569

https://news.ycombinator.com/item?id=8253351

I think "completing the circuit" is among the most important aspect of educational theory. Recognize and produce needs to happen in any subject, and a good student recognizes that SRS is but one tool for recognition, but others are needed to complete that, and production is an entirely different side of the coin.

sxyuan• 11 years ago

Where my bright students might have been used to high Bs and low As on tests, they were now breaking my scales. You could see it in the multiple choice, but it was most obvious in their writing: they were skillfully working in terminology at an unprecedented rate, and making way more attempts to use new vocabulary—attempts that were, for the most part, successful.
Given the seemingly objective nature of Anki it might seem counterintuitive that the benefits would be more obvious in writing than in multiple choice, but it actually makes sense when I consider that even without SRS these students probably would have known the terms and the vocab well enough to get multiple choice questions right, but might have lacked the confidence to use them on their own initiative. Anki gave them that extra confidence.

Not that you're wrong, but the author also points out here that SRS can help production by reinforcing the student's knowledge and boosting their confidence in what they've learned when it comes to using it.

slaypni• 11 years ago

Last year, I have developed a JS library SM.js (https://github.com/slaypni/SM-15) which is based on the algorithm used in the latest SuperMemo.

I have also developed a web service (http://slaypni.github.io/flashward/) for memorizing Foreign words using the library. Through the experience of using it by myself, I felt that spaced repetition helped me to remember many words for learning English.

I am feeling that most (free) SRS are using relatively simple algorithms (for example, Anki is based on SM-2 which is developed in the late1980s). Although I am not sure SM-15 is better enough, there could be some room to improve it. It is nice if the library was helpful for some developers who create new SRS.

wahnfrieden• 11 years ago

Newer isn't necessarily better with SuperMemo algorithms. They introduced tremendous complexity without strong evidence of increased effectiveness, last I looked into it - it seems to create more corner cases to manage, if anything.

I'd be interested in materials I may be missing.

A Year of Spaced Repetition Software in the Classroom

Related Stories