Show HN: My Leeds Hack Day project, a language detection API

http://polyglossy.com/

by lgeek • 16 years ago

9 15 16 years ago

polyglossy.com

zorander• 16 years ago

Uhhh..what?

http://imgur.com/LH2Hi.png

lgeek7• 16 years ago

Sorry, only supports English, German and French at the moment

zbanks2• 16 years ago

"Je t'aime" is wrongly classified as English. (Although understandable, since it's short)

How does this work? You claim it has "zero language knowledge," so how does it classify? Did it start out with nothing, and then train it on a corpus?

EDIT: nevermind, found your slides (http://polyglossy.com/presentation.pdf [pdf]) How big of a corpus did you use?

cemregr• 16 years ago

Good luck with your project, just wanted to say Google Translate API also has language detection, and gives you a degree of confidence. http://www.google.com/uds/samples/language/detect.html

petercooper2• 16 years ago

I was also at Leeds Hack Day. I didn't know you were working on this until the presentations but I wrote a language detection library for Ruby some time back: https://github.com/peterc/whatlanguage

It uses a slightly weird technique, though. Dictionary based and using a bloom filter for memory efficiency. Going forward, though, I plan to rewrite it to use a combination of n-grams and language "fingerprints."

lgeek7• 16 years ago

That's pretty cool! One of my friends suggested to use dictionaries and bloom filters but I've wanted to build probabilistic language models.

Do you have any accuracy stats? I'm guessing my approach might work better in some cases because the models include frequency information too. Did you experience significant accuracy loss when adding new languages? Anyway, I'll run it over my test data and compare.

lgeek7• 16 years ago

...and these are my results: http://imgur.com/9U6II.png I've only kept English, French and German for WhatLanguage. It looks like the Markov chains approach is indeed more accurate for shorter phrases but the dictionary approach gets slightly better for longer sentences.

petercooper2• 16 years ago

Do you have any accuracy stats?

No, but as you have noted, the method has the intrinsic property of being less accurate with fewer words and more accurate the longer the text. As my anticipated use was for documents over 10-20 words, this was OK. I expect the other techniques I outlined that I'm switching to to yield more accurate results across the board.

Show HN: My Leeds Hack Day project, a language detection API

Related Stories