Random Forest vs. Neural Network

https://mljar.com/blog/random-forest-vs-neural-network-classification/

by pplonski86 • 7 years ago

136 55 7 years ago

eden_h3• 7 years ago

This is a very thinly disguised advert for the author's product, and doesn't really advance on the benefits of either approach, as it doesn't go into any depth on why Random Forests/NNs are applicable to each type of data provided.

They're both generalised solvers, but default Random Forests aren't the most common Forest these days - LightGBM/XGBoost are both using Gradient Boosted Forests by default, which would be a much more interesting comparison to a NN's Gradient Boosting.

turingbike2• 7 years ago

I don't know why it isn't as popular, but CatBoost should be on the list too https://catboost.ai/

pplonski866• 7 years ago

You are right, CatBoost is an amazing algorithm. However, you will be shocked when you will talk with many older senior data scientist, that never heard of Xgboost or LightGBM. CatBoost for them is far too new.

eden_h3• 7 years ago

I tried Catboost when it came out. It should be very popular, as working with categories is where a lot of people seem to fall down in Random Forests.

The 'typical' response is either to make them into numeric variable, so 1-3 for 3 categories, or to make an individual column for each one. The first approach makes sense for ordinals, but not so much for actual categories, and the latter makes it difficult to group categories when a group of two categories together has more predictive capability than any single group. I know that LightGBM did a lot of work in this to optimise testing groups of variables, as testing every possible group in a large set is very intensive.

When I tried Catboost in R, I remember it downloading a large binary to work with, which put me off considerably, and predicting with it was pretty fragile, even for R. I trust Yandex about as much as I'd trust Google, but it seemed 'odd'.

cheez• 7 years ago

I looked into the site but I didn't see how CatBoost actually accomplishes its goals compared to other gradient boosting algorithms. Is there a summary somewhere?

minimaxir3• 7 years ago

With the current low cost of cloud computing, there's no reason not to just try everything and see what happens (which is why AutoML has become more popular).

It's more pragmatic than trying to rationalize which framework is "best" for a given dataset, as the results are often counterintuitive.

pplonski866• 7 years ago

I don't like such brute-force approach. Even if you have low cost of computing the number of possible combinations of hyperparameters is huge! Google Cloud AutoML Tables solution cost is 20 USD per 1 hour of computing (I guess that's because of inefficient Neural Architecture Search algorithm). Running few ML experiment can easily end with huge bill.

oneshot908• 7 years ago

On the contrary, I think this is one of the biggest emerging blockades to progress in ML/AI research, especially in academia. It has always been more cost-effective to run ML algorithms on consumer HW such as GeForce GPUs and gaming CPUs. It's frequently even faster than contemporary cloud offerings when the consumer HW gets ahead of existing enterprise HW. And it's so effective that HW companies starting changing their EULAs and crippling previously available aspects of APIs to herd AI back into the datacenter where they seem to think it belongs.

And that IMO is a reinvention of the "Walled Garden" of academic HPC (ask any grad student begging and pleading for supercomputer time) which has always sucked and its new commercial incarnation is even worse because it's unclear how to get commercial cloud time on government grants.

OTOH it's fine for large shops like OpenAI, DeepMind, AWS AI, FAIR, MS Research etc because they have deep deep pockets. So if you're content with most future groundbreaking research coming from a small tribe of market leaders, well great, but I suspect innovation is already slowing down because of this.

dlphn___xyz• 7 years ago

this approach is also very inefficient- autoML takes hours searching for parameters when you could build the same model manually in a fraction of the time.

windsignaling• 7 years ago

Not to mention the article is full of grammar and spelling errors. Not a great look if you are trying to promote your product.

turingbike2• 7 years ago

The article recommends RF for tabular data because it is easier. In general I agree, but newer tools are making NN for tabular data as easy as can be... see, for example, fastai https://docs.fast.ai/tabular.html

ramraj074• 7 years ago

The biggest reason to use RFs is that with sufficient trees it's basically impossible to overfit your data. You also don't need to spend days optimizing your hyperparameters. Hence, if you need a quick model where time is tantamount, and you want to err in the side of caution, I feel like an RF is the best choice.

pplonski866• 7 years ago

You are right! The more trees in the RF the more stable the model is. What is more, you got feature importance for free!

yters5• 7 years ago

I would have assumed the opposite, with enough trees you are guaranteed to overfit your data? Boosting increases the VC dimension of the aggregate model, which makes it more prone to overfitting.

claytonjy4• 7 years ago

It's quite possible to overfit with an RF; if you grow deep trees on a self-similar feature set, 1000 trees can overfit just as much as 10.

pplonski866• 7 years ago

But still, NN has much more hyperparameters than RF and training good NN model requires far more experience.

ScoutOrgo2• 7 years ago

fast.ai has some pretty good defaults. The main HP to choose in their tabular learner are the neurons in the linear layers and the number of embeddings for each category.

meesterdude• 7 years ago

Been using the annoy library lately (https://github.com/spotify/annoy) after becoming frustrated with TF not behaving as I'd expect. There's another lib in python, nmslib, but I can't get seem to get it to work right - and the docs are crap.

Anyway, would love to pick someone's brain about this stuff to help fill in my gaps.

RocketSyntax• 7 years ago

No mention of backpropagation

vbarrielle• 7 years ago

Random forests can be applied to images. The RF algorithm only needs to be tweaked to split its trees by comparing the difference of two pixels to a threshold.

Random Forest vs. Neural Network

Related Stories