Why do tree-based models still outperform deep learning on tabular data? (2022)

by tosh • 2 years ago

212 109 2 years ago

I still don't get the impetus or desire to make NNs work better for tabular data. Regression works pretty well and is easy to interpret/diagnose/work with. GBMs work really well (given a few considerations) and is trickier to work with but nothing crazy. When I see all the fancy hijinks people get up to when applying NNs to audio/text/pictures I think it's really cool but also not something I'd want to have to do if I didn't absolutely need to when working with data out of a relational db. And anyways, how much of a benefit could it actually bring? GBMs are already capable of fitting and dramatically overfitting most datasets.

mrfox321• 2 years ago

When you need the best possible model, full stop.

E.g. finance

In a sufficiently competitive space, good enough doesn't cut it.

dchftcs• 2 years ago

Do you know of any shop that is running deep learning profitably?

Thrymr• 2 years ago

There is no such thing as "best possible model, full stop". Models are always context dependent, have implicit or explicit assumptions about what is signal and what is noise, have different performance characteristics in training or execution. Choosing the "best" model for your task is a form of hyperparameter optimization in itself.

asdff3• 2 years ago

These models usually have poorer fit though

dawnofdusk2• 2 years ago

The paper offers a reason why NNs working for tabular data would be good:

Creating tabular-specific deep learning architectures is a very active area of research (see section 2) given that tree-based models are not differentiable, and thus cannot be easily composed and jointly trained with other deep learning blocks.

Here is a second reason, from the paper

Impressed by the superiority of tree-based models on tabular data, we strive to understand which inductive biases make them well-suited for these data.

which is a great reason, because understanding the inductive biases of different learning/regression techniques gets us closer to a more general understanding of how to encode inductive biases in a generic learning algorithm.

hackerlight3• 2 years ago

My hypothesis is decision trees are more robust to nonstationary distributions. If the variance and means of the features shift dramatically, the model isn't going to blow up, because it's not additive.

In the domains where NNs work well (image processing and language), you're dealing with a predictable and stable distribution of values. Elephants might look a bit different in the train and test set, but you're not randomly getting 100x the variance of the input data. The decision tree just isn't going to care as much, because splits around the mean will lead to the same outcome.

Another hypothesis is that zooming into bivariable relationships is more important in tabular data. Neural nets are better at local and global context. But they struggle if all that matters is the relationship between two columns of data because of the additive nature. Large networks can figure it out due to model capacity, but then you'll run into overfitting.

melondonkey4• 2 years ago

At this point I wish every junior DS could read this paper and not come in to every problem with the new bright idea that they’re going to beat XGBoost with their DL architecture. Free promotion if they never say the words “latent subspace”

barrenko• 2 years ago

One of those juniors is going to do it once!

bbstats• 2 years ago

because smooth is better than jagged :)

mikkom2• 2 years ago

It's very important to note that this is from 2022. I'm not saying it's not true today but neural models have gotten much better in 2 years.

(I'm personally using NN models for predicting certain values for tabularly structured data and at least for my case, the NN works better than state-of-the art tree models.)

mjhay5• 2 years ago

Do you have any intuition you could share of why NNs work better in this case?

queuebert2• 2 years ago

Not the parent, but NNs typically work better when you can't linearize your data. For classification, that means a space in which hyperplanes separate classes, and for regression a space in which a linear approximation is good.

For example, take the circle dataset here: https://playground.tensorflow.org

That doesn't look immediately linearly separable, but since it is 2D we have the insight that parameterizing by radius would do the trick. Now try doing that in 1000 dimensions. Sometimes you can, sometimes you can't or don't want to bother.

mikkom2• 2 years ago

I assume it's because there are some very complex relationships and patterns that cannot be captured by decision trees. Tree models work better on simpler data at least that is my gut feeling based on previous experiments with similar data.

lerchmo• 2 years ago

There is some work with zero shot (decoder only) time series predictions by google and an open source variant. Curious to see how these approaches stack up as they are explored.

redox99• 2 years ago

In what way have models gotten better for tabular data? Can't think of any new technique since 2022.

jeffreyrogers2• 2 years ago

There has been some work on training on lots of different data sets and then specializing on the one you care about. But I think people were trying that approach pre-2022 as well.

_pastel• 2 years ago

Tooling around embeddings has improved. Creating and fine-tuning custom embeddings for your tabular data should be easier and more powerful these days.

__mharrison__• 2 years ago

Pretty sure it is still true today. Catboost rules the roost!

dawnofdusk2• 2 years ago

Paper seems interesting but I don't like the question title. I think the answer to the question would just be that tabular data is not fully in the "big data" regime yet so there is no reason a priori to expect deep NNs to do better. Factor in computational simplicity of tree-based models and I think the deck is stacked against deep learning from the start.

math_dandy2• 2 years ago

Do you know of any (families of) examples of tabular datasets of any size (you can choose what "big" means) where deep learning convincingly outperforms traditional methods? I would love some quality examples of this nature to use in my teaching.

Scene_Cast26• 2 years ago

Recommendation engines: search, feeds (tiktok / youtube shorts / etc), ads, netflix suggestions, doordash suggestions, etc etc. Also happens to be my specialty.

dcl• 2 years ago

Regression targets where extrapolation may be needed. Decision tree methods cannot extrapolate, the predictions are have to be a mean of a subgroup of the data.

Consider: Predicting how much a customer might pay by end of month, with information we have at the start of the month.

In this example, if a customer had a record $10m of open invoices due by EoM and the largest payment amount received in prior months of $5m, the decision tree cannot possibly predict the payment amount will be ~$10m, even when the best feature indicates the payment will be $10m.

There are some hacks/techniques which can maybe reduce this issue, but they don't always work.

Scene_Cast26• 2 years ago

I've worked on models trained on ultra-large tabular data. It still took substantial effort to beat tree models (custom architecture specifically for this particular domain, something I haven't seen elsewhere out in the open).

When tabular data is mentioned, one of the unspoken applications is finance. There, my guess is that one of the issues is that data is not very IID and thus latent "events" are fairly sparse. Combine that with the humongous amount of raw data, and you get models that overfit.

TimPC2• 2 years ago

I think there are certain types of tabular data that lend themselves naturally to tree models. But when you're talking about tabular data for finance I guarantee you very few hedge funds are running tree models for trading strategies. When your scale of data is the past X quarters of all stock prices and trade volumes you have enough data that you can fit an NN and there are a number of techniques you can use to reduce overfitting (large amount of data, good regularization, dropout, etc.)

Jensson3• 2 years ago

The tabular data I had at Google was exabytes, tree models still performed the best so I guess exabytes is small data then?

Scene_Cast26• 2 years ago

The team behind Yggdrasil tree library at Google was doing some interesting research into tree differentiability (and thus unlocking SGD & end-to-end learning for hybrid architectures).

Rexxar2• 2 years ago

Is it this library https://github.com/google/yggdrasil-decision-forests ?

Scene_Cast26• 2 years ago

Yep.

melondonkey4• 2 years ago

This is interesting. Are BART models differentiable? I haven’t looked closely at them but I would have thought for posterior sampling they’d have to be. BART has been around for a while, too

doubtfuluser• 2 years ago

Since this is from 2022, I’m wondering how “tabular foundation models” could change this. The incredible success of DL we see at the moment comes partially from foundation models learning on a lot of “semi-related” data an “understanding” of the behavior. Something similar has been explored in tabular data as well iirc.

So I would be curious to see latest DL results. On the other hand it is also the case that in most cases where DL based on foundation models is used, specific heavily tuned models outperform the generalistic models. And for tabular data there is a lot of experience how to make it great with tree based models.

dweinus• 2 years ago

What would these tabular foundation models look like? LLMs work as foundation models because the input is fixed in format (a sequence of text). Would the model be for a specific fixed tabular format?

scottyak• 2 years ago

One promising approach is to encode each feature key and feature value as embedding vectors, concatenate them into "feature tokens", then feed them into a Transformer (without positional encodings). This takes advantage of column-order invariance. See:

https://arxiv.org/abs/2403.01841 (ICLR 2024 spotlight)

moelf• 2 years ago

There seems to be differentiable tree models now that perfor somewhat better than e.g. XGBoost https://github.com/Evovest/MLBenchmarks.jl?tab=readme-ov-fil...

candiodari• 2 years ago

TLDR: Because tree-based models don't just outperform deep learning, they totally outclass deep learning on simple data.

But they don't scale with larger and more complex data. You cannot (realistically) make an LLM with XGBoost.

Kind of surprised how well Resnet and FT Transformer do though.

nickpsecurity• 2 years ago

I want to see more work combining them. Here’s an example I saw in one of the links in this thread:

https://arxiv.org/abs/1806.06988

It combines NN’s with decision trees.

huac• 2 years ago

the famous FB ads paper (from 10 years ago!) combines decision trees with a logistic regression and shows a significant improvement: https://research.facebook.com/publications/practical-lessons...

feel free to extend logistic regression to an MLP :)

MAXPOOL• 2 years ago

deep learning architectures have been crafted to create inductive biases matching invariances and spatial dependencies of the data. Finding corresponding invariances is hard in tabular data, made of heterogeneous features, small sample sizes, extreme values

Transformers with positional encoding have embeddings are invariant to the input order. CNN's have translation invariance and can have little rotational invariance.

It's harder to find similar invariances to tabular data. Maybe applying methods from GNN's would help?

queuebert2• 2 years ago

This is tangential, but that paper has some amazingly good plots for an ML paper.

usgroup2• 2 years ago

I'm not sure this is surprising. Say you were to glue together 10 datasets with the same 10 explanatory features and 1 response feature, but distributed very differently to each other. This would be no problem for tree based model because they'll conditionalise indefinitely to get a good fit. If the number of records is relatively small (say 10k) the dataset will be much too scarce for an NN to learn these discontinuities -- its like it has 1000 records per segment.

Similarly, tabular data is often of this nature. Its not i.i.d, it tends to cluster.

3abiton• 2 years ago

I wonder if that would be the case for graph based models too

skadamat• 2 years ago

When working with tabular data, there are very few situations where absolute model performance is the only criteria that's important. In practice, the following are equally as important:

- Explainability / debug-ability of models

- Effort to train, deploy, and manage NN models in production

- Capturing, collating, and organizing new & better datasets

- Local developer experience and human-model-iteration time

Building all of your software in C or Assembly will be faster and higher performant. But at what cost and with what tradeoffs? Building a website has a different set of tradeoffs than building a program for the Mars rover.

derefr3• 2 years ago

It's funny; as a regular non-ML programmer, the optimum for every one of those factors for "tabular data" would seem to me, to be to "throw the tabular data into a relational data warehouse, and ask your questions in the form of SQL queries."

Or, if the "tabular data" is heavily relationship-based, then possibly replace "relational data warehouse" with "graph database", and "SQL queries" with whatever querying language that graph DB is natively / most expressively queried in.

Of course, this is the most important implicit "equally important" factor, one that an ML dev would think goes without mentioning: the generality or "power" of the model in what questions it can answer. You can only make these trade-offs in the context of knowing what kinds of questions you want your model to solve for! If all your questions are quantitative ones, maybe the right "model" for you is an RDBMS!

---

Though, that being said... why can't a deep-learning model emulate the thing that an RDBMS does, "at runtime", as part of its "mental toolkit" for approaching problems? That would be the best of both worlds, no?

I know that LLMs in particular have been observed to have "emergent numeracy" above a certain training-set size. There is a step function in how they approach such problems, going from their only being able to answer arithmetic questions on numbers of bounded size, and sometimes getting the answers wrong (probably this is due to a memorization-based approach); to being able to answer arbitrary arithmetic questions on operands of unbounded size, and always getting the answer correct.

I would guess that that what's happening, is that they are developing a functional component of their network that works akin to an Arithmetic Logic Unit, operating not on tokens, but on tokens transformed into a "numeric register" representation that is amenable to having math done to it with stable, quantized, position-independent results. (Just like the functional component that human brains develop after seeing enough math problems... probably.)

Do you, as an ML dev, think it would ever be possible for any of the model architectures we're familiar with today, to be trained such that they would develop an analogous emergent functional component for handling tabular-data questions, by transforming its internal working state into relational-DB/graph-DB data structures — e.g. page-heaps of binary-packed row-tuples; B-tree indices; etc — and then manipulating the working state in that form, using learned algorithms applicable to that type of data?

It seems to me (possibly just because I don't know any better) that just as with numeracy, "being able to put the data into a different and better internal representation" is what would be needed for deep-learning models to become truly good at dealing with tabular-data problems.

But, unlike with numeracy, "thinking as if you were a relational database" is not something a single human would ever intuit how to do without being taught. Relational algebra — and the data-structures and algorithms to make it practical to have a Turing machine do said relational algebra — wasn't even a single intuition, but a conscious effort, of multiple humans, working together over years. I strongly doubt that there's any number of "tabular-data problems" that you could show a human being, that would result in them developing an intuitional ability to do what a relational database does with its memory to efficiently answer queries.

(I suppose we could give an ML model an RDBMS, and hardwire it to interact with it. I know there are hybrid ML + formal-logic systems. Are there hybrid ML + data-warehouse systems? Not where the model queries an external DB — while that can be done, it'd be only in the same "stop and do this" way that ChatGPT runs Python code, which wouldn't make it a thinking tool the way that the formal-logic proof engines are for hybrid ML systems. Rather, I mean that some data-warehouse execution engine could be embedded into the ML execution framework itself, deployed as part of the GPU shader-program to each tensor core, such that data-warehouse operations can be done as a native part of the network's per-node instruction-set. Anyone ever tried this?)

dist-epoch3• 2 years ago

ask your questions in the form of SQL queries

How do you know which questions to ask? This is what ML is good at, finding the right questions which classify the data.

jonathankoren3• 2 years ago

It's funny; as a regular non-ML programmer, the optimum for every one of those factors for "tabular data" would seem to me, to be to "throw the tabular data into a relational data warehouse, and ask your questions in the form of SQL queries."

It's doubly funny; as someone that comes from an ML background, and has developed and maintained multiple ML systems at multiple orgs, that I also think the answer very often is, "throw the tabular data into a relational data warehouse, and ask your questions in the form of SQL queries."

closeparen• 2 years ago

"throw the tabular data into a relational data warehouse, and ask your questions in the form of SQL queries."

You can ask SQL descriptive questions. Can you ask it for predictions? How?

otabdeveloper45• 2 years ago

LLMs don't have any "mental model". They are just text completion with a larger memory. This works for text (unsurprisingly), but for nothing else.

martindbp2• 2 years ago

Deep learning really shines when the input is raw and at a very low abstraction level: pixels, byte pair encodings etc. Using deeply learning for classification on tabular data is just needless complexity, as the variables are often at a very high abstraction level already. Also with tabular data there are generally not many spatial or temporal relationship between the variables, which CNNs and transformers excel at.

jobigoud• 2 years ago

We have things that can describe and explain why an image they have never seen is funny. That's pretty high level.

martindbp2• 2 years ago

Yes, what I meant was deep learning is great at deriving those higher level abstractions from low level raw data. Words can be seen as something in between, bag of words can be fairly effective at simpler tasks, but LLMs embed words into higher and higher abstractions.

andy99• 2 years ago

Also images and text have tons of recurring patterns that can be exploited to train big models with lots of data. There is an internet worth of each modality that at least generally can all contribute helping a model build up a better overall understanding.

There is no analog for tabular data, it's all different.

hresvelgr• 2 years ago

I wonder if tree-based models would outperform in situations concerning source code as well; given it's already quite structured. Going a step further and supplying and AST may be even more beneficial.

sgt101• 2 years ago

Isn't this just that trees are a natural compression of tables?

smthin smthin inductive bias?

hashta2• 2 years ago

I have a lot of experience working with both families of models. If you use an ensemble of 10 NNs, they outperform well-optimized tree-based models such as XGBoost & RFs.

padthai• 2 years ago

Which kind of ensemble? Because it cannot be as easy as a voting meta model of nn with same architecture/hyperparametres right?

knightoffaith• 2 years ago

Is this true? I hear that XGBoost tends to win Kaggle competitions for tabular data - how come you don't see NN ensembles winning instead? Or do they?

malshe• 2 years ago

I would like to know this too. From what I understand, it is XGBoost that rules Kaggle.

hashta2• 2 years ago

To both questions above, just simple averaging of the logits (classification) or raw outputs (regressions) usually works well. If I had to guess why people don't use this approach often in kaggle competitions is the relative difficulty of training an ensemble of NNs. Also, NNs are a bit more sensitive to the type of features used and their distribution relative to decision trees (DTs).

Ensemble models work well because they reduce both bias & variance errors. Like DTs, NNs have low bias errors and high variance errors when used individually. The variance error drops as you use more learners (DTs/NNs) in the ensemble. Also, the more diverse the learners, the lower the overall error.

Simple ways to promote the diversity of the NNs in the ensemble is to start their weights from different random seeds and train each one of them on a random sample from the overall training set (say 70-80% w/o replacement).

jhoho• 2 years ago

Can you recommend a resource on this for the curious learner?

up2isomorphism• 2 years ago

Because human-being heavily compress picture, sounds, but much less with tabular data.

GVaroquaux• 2 years ago

As an author, I'm happy to see this work here.

For those curious about what we have been up to on the topic of tabular learning, we have found a setting where deep learning does seem to bring sizable benefits (spoiler alert, it's about being able to pre-train, and transferring to new data works best when there are some strings to be recognized): https://arxiv.org/abs/2402.16785

In the above work, pre-trained tabular models markedly outperform tree-based models (including catboost, which is a very strong baseline).

As someone who has been banging on tabular data for years, I'm really excited about this development.

afiodorov• 2 years ago

A calculator outperforms deep learning on basic arithmetic tasks.

Tree-based models are extremely good at finding clustering patterns; they outperform trained humans at that, thus we have commercial applications such as fraud detection.

Deep Learning is most promising way of getting us to general intelligence. So far only known general intelligence, human intelligence, has many quirks at specific tasks and I think Deep Learning won't be any different. However Deep Learning models can recognise their own weakness and call tree-based model if they think that's appropriate.

Why do tree-based models still outperform deep learning on tabular data? (2022)

Related Stories