The Unreasonable Effectiveness of Character-Level Language Models

http://nbviewer.ipython.org/gist/yoavg/d76121dfde2618422139

by woodson • 11 years ago

74 12 11 years ago

nbviewer.ipython.org

I did a project that used a similar technique, but mapping only a single state transition. For its simplicity it was very effective.

https://github.com/rectangletangle/atypical

JoachimSchipper• 11 years ago

HN discussion of the blog post this is a response to (890 points, 204 comments): https://news.ycombinator.com/item?id=9584325.

bcoates• 11 years ago

This is exactly the letter-based version of the "Dissociated Press" Markov-chain algorithm, right?

I'd suspect some of the perceived quality at higher orders (particularly for the source-code example) is just coming from the transition graph becoming sparse and deterministically repeating long stretches of the input text verbatim.

unhammer2• 11 years ago

Goldberg says parenthesis-balancing and indentation etc. would require lots of non-trivial human reasoning to implement, but I found https://news.ycombinator.com/item?id=9585080 's example even more amazing: the NN actually learns meter and parts of speech – I don't find it very likely that you could get a character-based markov-chain to generate completely novel grammatically correct constructions (with correct meter!) from such a small corpus, since you always have to weigh correctness (high order) up against ability-to-generalise (low order).

SilasX• 11 years ago

That same article (that you linked) showed how the RNN learned to balance parentheses.

foldr• 11 years ago

It's not true that it learns the meter. If you look through the large generated example you'll see that there's not much consistency in the number of syllables per line.

flipp3r2• 11 years ago

I also did a project that used a similar technique which scrapes text from 4chan, mapping N words to the next possible word.

https://github.com/skphilipp/humanity

transpy2• 11 years ago

Why 4chan?

flipp3r2• 11 years ago

I browse it myself, it has a really simple API and the content is refreshed quite fast on active boards.

wodenokoto• 11 years ago

People have been working for age trying to generate passable text by looking at n-grams for whole words. It is quite surprising to see that all along we could have done better by using a simpler model!

transpy2• 11 years ago

Yes, I guess computers are better than us at deciding how to organize characters in words, judging by the author's typos: 'liklihood', 'Mathematiacally', 'langauge', 'immitate', 'somehwat', characer', 'commma', Shakespear', 'characteters', 'Shakepearan'.

The Unreasonable Effectiveness of Character-Level Language Models

Related Stories