Classifying Japanese characters from the Edo period

http://community.wolfram.com/groups/-/m/t/1221098

by soofy • 9 years ago

49 24 9 years ago

community.wolfram.com

wodenokoto• 9 years ago

Anyone know the work he refers to as the inspiration for this article?

titanix2• 9 years ago

If I understand correctly, the guy downloaded a data set, train it and get happy because it performs over 95% on the training set from the data set.

This is not interesting at all because the hard part is to actually segment and classify characters in a real text, and in real life they don’t conveniently come as 28x28 images. As someone with a few hours of training in reading that kind of texts the difficulties are:

- taking appart characters (hentaigana) that looks very close

- segmenting characters because size and length vary a lot

- differentiating between kana and kanji because the former are deritaves of the latter and they sometime look quite alike

So the experimenter should have started his post telling he didn’t know any Japanese instead of closing it as such (even if one can notice when he just use modern kana as classes).

chch• 9 years ago

Since the original article doesn't give a lot of context as to what these characters looks like in context, here is an image of a Japanese book [1] and some playing cards [2] to give context to the segmentation difficulties.

[1] https://wakancambridge.files.wordpress.com/2015/07/e8bbbde58...

[2] http://img03.aucfan.com/item_data/image/20150720/yahoo/u/u80...

microcolonel• 9 years ago

Hard enough for a literate human to segment.

reustle• 9 years ago

This is not interesting at all because...

Comments like this make me reluctant to post things I work on

mikekchar• 9 years ago

The OP's comment is very good because the discussion in the article is quite misleading about what they accomplished. It's also apparently self-serving; apparently marketing material for Wolfram. As your sibling's post makes extremely clear, classifying handwritten text from the Edo period is a much, much, much more difficult problem. In fact the samples they have taken for training are so unstylized that they might just as well have done an article about recognising modern hiragana hand writing. That would be very cool too, but would have gathered no interest because it is a well solved problem.

I have no problem with people doing fun projects and explaining how they did them. I have a bit of a problem with people apparently making exaggerated claims about their product's capabilities.

fenomas2• 9 years ago

I think the article is more mistitled than anything else.

I mean, it's a fine article for someone interested in how to train a classifier on an annotated dataset in Mathematica. It's just not very interesting for someone interested in how to classify Japanese characters from the Edo period.

peterburkimsher3• 9 years ago

I want to do the same for Chinese.

I have a collection of 1500 fonts, and finally exported all the PNGs for 75,000 characters. Now I need to pad, crop, and scale to make 28x28 (or 32x32, or 64x64, or another resolution).

Then I want to do the Machine Learning (Classify) step.

The article doesn't go into any detail about how to install Classify, how to import the training and testing data, and how to then actually run it. I watched the videos from CS231n because of my boss, but again, I'm still not really sure what to do practically.

If I have lots of folders of images, what should I do to build an OCR program?

visarga• 9 years ago

- load images, convert into array, call it X

- put the labels (character codes one hot encoded) in an array called y

- use a convolutional neural network to learn the mapping X->y (training)

- send new images to the network to make predictions

That's how you train all neural nets in supervised learning.

peterburkimsher3• 9 years ago

Just like the article, your comment is pseudo-code. Isn't there a tutorial for this?

contingencies2• 9 years ago

Sounds like a nice collection. I'd be keen to see any decent ones you have for rarer seal scripts.

peterburkimsher3• 9 years ago

I recommend FangzhengXiaozhuantiFont-TraditionalChinese.ttf for Small Seal Script and JDFZHUANF.ttf for Large Seal Script.

Both are supported by Pingtype.

https://pingtype.github.io

Sorry that it's slow to load; the fonts have to be downloaded from the server every time because I can't be sure if they're on the user's device. The default font is 12MB. Once it's loaded once, you can re-translate much faster - it's just the first run that's slow.

henrikeh• 9 years ago

Classify is a function in Mathematica which returns a ClassifierFunction.

Documentation for Classify: http://reference.wolfram.com/language/ref/Classify.html

As far as I could see they listed all the Wolfram Language code in the example to set it up.

pornel2• 9 years ago

The training set contains errors. The に row is especially messy: it's full of 小, ふ, and み.

ramchip• 9 years ago

Aren't these hentaigana?

http://www.book-seishindo.jp/kana/onjun_2.html#ni

pornel2• 9 years ago

Ahh, I see. These aren't supposed to be visual variants of に specifically, but are just any old forms/arbitrary other characters that were used in place of に (sort of like "V" being classified as "u" because of Roman texts).

Classifying Japanese characters from the Edo period

Related Stories