Ask/Poll News.YC: What is a good open source Bayes classifier?

https://news.ycombinator.com/item?id=124085

by fiaz • 18 years ago

35 36 18 years ago

news.ycombinator.com

I'm just doing a little research for a project and I thought everybody could benefit from the fact that we have happen to have an expert on the subject as a member.

I would also like learn from what other people have used as well so any input that others could give would be helpful.

------------

UPDATE: I started a poll thread below so that we can submit a project link and up-vote a link if we have experience or knowledge over one project versus another.

Thank you to everybody who replied, you are great!!!

jgrahamc2• 18 years ago

What sort of classification are you trying to do? Text, I assume. If it's text and you need something open source, or just a pointer to how to write one yourself, you could read the article I wrote for Dr Dobbs on this subject: http://www.ddj.com/development-tools/184406064

There are quite a lot of toolkits out there that do Bayesian things (take a look at libbow or Weka).

fiaz9• 18 years ago

I'm looking for something that is open source that I can start using right away with minimal configuration... I'm not smart enough to create my own, but clever enough to take somebody else's work and cater it to what I'm trying to do.

It's only for text processing.

vital_sol• 18 years ago

If Perl is ok, you can try these: http://search.cpan.org/search?query=Bayes&mode=all

bcater• 18 years ago

The one in BioPython is pretty good.

dreish2• 18 years ago

Whichever one you use, you'll probably need to edit the source to change the probability formula from (n_hits / n_total) to ((n_hits + 1) / (n_total + 2)). That's the correct formula based on an even distribution of probabilities (which is close enough to the actual distribution in most situations for this to be a huge improvement). I can never find a reference for this when I search for one, but you can verify it experimentally with a short program in your favorite dynamically-typed language, or a long program.

For example, if you live in a world with only black and white birds, but you don't know the percentage of each and have no reason to believe it's more likely 2% black than 70% black, or any other percentage, if you see two black birds fly by, that doesn't mean the next bird you see has a 100% probability of being black, but that's exactly the assumption most widely-used naive Bayesian classifiers make.

I modified SpamProbe to use (spam+1)/(total+2), and the results have been good.

jgrahamc2• 18 years ago

Sounds to me that you are describing Laplace Smoothing in a Naive Bayes classifier. This is a pretty standard technique for avoiding the problem that you are seeing where the probability comes out as 100%/0% because of a lack of information in the model.

dreish2• 18 years ago

Thanks, yes, that's what it's called.

All due credit to Laplace for the technique, but the word "smoothing" is making me wince, because it makes it sound as though this is some artificial approximation. For the assumption of an even distribution of probabilities, n+1 / m+2 really _is_ the exact probability of the event repeating. Like I said, you can confirm this experimentally with a quick program.

timr• 18 years ago

More generally, it's equivalent to applying a Dirichlet prior distribution, with uninformative parameters (i.e. all of the parameters on the Dirichlet are equal).

This is important, because while adding a single pseudocount to each column will prevent zero divisions, it's probably not reflective of the true distribution of values. If instead, you add pseudocounts using a Dirichlet where the parameters are set based on some prior knowledge, you can often improve the performance of the classifier (especially in low-count situations), without biasing the results unfairly.

jackbard2• 18 years ago

http://www.codeproject.com/KB/cs/BayesClassifier.aspx

This is a simple one built in C#/.Net. I fixed a significant bug and raised classification accuracy from 74% -> 96% in my noisy dataset (automobile accident claims). I emailed the author with a bunch of improvements (such as histograms) and some other tweaks but never heard back. Anyway, the bugfix is simple: take a look at category.cs. In TeachPhrase(), move m_TotalWords++ inside the test for "if (!m_Phrases.TryGetValue(phrase, out pc))".

What you want here is to count the # of unique words. The original code was counting the total # of times all words appear.This one change reduced classification errors by 3X.

Cheers, --Jack

jackbard2• 18 years ago

Oh, forgot to mention that you'll want to compute probabilities using logarithms to avoid underflow precision loss (which will introduce classification errors). example: wordValue = System.Math.Log((double)count / (double)cat.TotalWords);

ashu• 18 years ago

Andrew Mccallum's Bow library (CMU) is a very robust toolkit. http://www.cs.cmu.edu/~mccallum/bow/

Very high quality. Multiple people have used it for serious research.

fiaz9• 18 years ago

PROJECT LINKS/POLL

reply only to this message and vote on links below

fiaz9• 18 years ago

Ruby :: Classifier gem

http://classifier.rubyforge.org/

fiaz9• 18 years ago

Java :: jBNC Toolkit

http://jbnc.sourceforge.net/

dfranke• 18 years ago

CRM114

http://crm114.sourceforge.net/

henning2• 18 years ago

The one with the simplest interface I've seen is http://divmod.org/trac/wiki/DivmodReverend . Looks very easy to get started with.

If you want to get more serious, use Weka or Bow or YALE or something implemented in a reasonably fast language.

food792• 18 years ago

The Cadillac of bayes classifiers is CRM114--it can use classifiers that are far more advanced than naive bayes, such as clustering, or with hidden markov modeling.

henning2• 18 years ago

Beware, the source code to CRM114 ain't no freakin' Cadillac.

ketralnis• 18 years ago

I can not over-recommend crm114. I've been using it to classify some database entries and its accuracy is second-to-none, and its custom language makes working with strangely-stored data (like database entries) easy (after you learn the strange language)

fiaz9• 18 years ago

Is there any documentation that stands out for learning the alien language?

No doubt, I'll be combing through all of the CRM114 information on the website. Is there anything that is not referenced there that will be of use?

yawl• 18 years ago

I use Weka in my project. Btw, there is a book about Weka: http://www.cs.waikato.ac.nz/~ml/weka/book.html

jorgeortiz852• 18 years ago

+1 for Weka.

It's not just a Bayes classifier. It's got every machine learning algorithm you can think of. Once you get the file format right, you can drop-in any algorithm you want. It's amazing.

fiaz9• 18 years ago

How does Weka differ from CRM114?

jorgeortiz852• 18 years ago

I'm not familiar with CRM114, but upon cursory inspection it seems to be written in C and targeted mainly at document classification.

Weka is written in Java and implements all kinds of machine learning/data mining tools.

pg• 18 years ago

crm114

ivankirigin• 18 years ago

awesome logo

http://crm114.sourceforge.net/

fiaz9• 18 years ago

pg has spoken, and it is good...

Seriously though this is exactly what I need to accomplish my tasks (which is not SPAM filtering). I briefly looked at all of the other alternatives before diving deeper into the "CRM114 Revealed" book. I really wish I knew about this a few years earlier!!

simplegeek• 18 years ago

I'm not sure about this one but you should definitely look into Orange if you're considering Python. http://magix.fri.uni-lj.si/orange/

toplakm• 18 years ago

You could try Orange: http://www.ailab.si/orange

It is written in combination of Python and C. It can be used as a python module.

mig• 18 years ago

Use CRF for classifying/labeling.

http://crf.sourceforge.net/

Its extremely well written, easy to define features etc. Reasonably good support too.

herdrick• 18 years ago

Write it yourself. I will say that teasing out the features you want to score can be some work; does anyone here know if the CRM114 package helps with that?

Ask/Poll News.YC: What is a good open source Bayes classifier?

Related Stories