XLSTMTime: Long-Term Time Series Forecasting with xLSTM

by beefman • 2 years ago

231 53 2 years ago

In recent years, transformer-based models have gained prominence in multivariate long-term time series forecasting

Prominence, yes. But are they generally better than non-deep learning models? My understanding was that this is not the case, but I don't follow this field closely.

Pandabob• 2 years ago

While I don't have firsthand experience with these models, I recently discussed this topic with a friend who has used tree-based models like XGBoost for time series analysis. They noted that transformer-based architectures tend to yield decent performance on time series tasks with relatively little effort compared to tree models.

From what I understood, tree-based models can usually outperform transformers when given sufficient parameter tuning. However, models like TimeGPT offer decent performance without extensive tuning, making them an attractive option for quicker implementations.

techwizrd2• 2 years ago

In my aviation safety work, deep learning outperforms traditional non-DL models for multivariate time-series forecasting. Between deep learning models, I've had a wide variance in performance between transformers, Bi-LSTMs, regular MLPs, VAEs, and so on.

theLiminator• 2 years ago

What's your go-to model that generally performs well with little tuning?

montereynack• 2 years ago

Seconding the other question, would be curious to know

ramon156• 2 years ago

Now take into account that it has to be lightweight and DL falls shirt

nerdponx• 2 years ago

What are you doing in aviation safety that requires time series modeling? Weather?

dongobread3• 2 years ago

From experience in payments/spending forecasting, I've found that deep learning generally underperform gradient-boosted tree models. Deep learning models tend to be good at learning seasonality but do not handle complex trends or shocks very well. Economic/financial data tends to have straightforward seasonality with complex trends, so deep learning tends to do quite poorly.

I do agree with this paper - all of the good deep learning time series architectures I've tried are simple extensions of MLPs or RNNs (e.g. DeepAR or N-BEATS). The transformer-based architectures I've used have been absolutely awful, especially the endless stream of transformer-based "foundational models" that are coming out these days.

sigmoid102• 2 years ago

Transformers are just MLPs with extra steps. So in theory they should be just as powerful. The problem with transformers is simultaneously their big advantage: They scale extremely well with larger networks and more training data. Better so than any other architecture out there. So if you had enormous datasets and unlimited compute budget, you could probably do amazing things in this regard as well. But if you're just a mortal data scientist without extra funding, you will be better off with more traditional approaches.

rjurney2• 2 years ago

They aren’t so hot, but recent efforts at transfer learning were promising.

svnt• 2 years ago

The paper says this in the next paragraph. xLSTMTime is not transformer-based either.

Dowwie• 2 years ago

marketed as a forecasting tool, so is this not applicable to event classification in time series?

RamblingCTO• 2 years ago

I'd say that's kind of a different task. I'm not a pro in this, but you could maybe treat it as a multi-variate forecast problem where the targets are probabilities per event if n is really small?

jimmySixDOF• 2 years ago

Yes, I would be interested where this (and any Transformer/LLM based approach) is improving anomaly detection for example.

spmurrayzzz• 2 years ago

I can't speak for all use cases, but I've done a great deal of work in the space of using deep learning approaches for anomaly detection in network device telemetry. In particular with high resolution univariate time series of latency measurements, we saw success using convolutional autoencoders and GANs. These methods lean on reconstruction loss rather than forecasting, but still effective.

There is some prior art for this that we leaned on [1][2].

RE: transformers — I did some early experimentation with Temporal Fusion Transformers [3] which worked pretty well for forecasting compared to other deep learning methods, but rarely did I see it outperform standard baselines (like ARIMA) in our datasets.

[1] https://www.mdpi.com/2076-3417/12/23/12472

[2] https://arxiv.org/abs/2009.07769

[3] https://arxiv.org/abs/1912.09363

greatpostman2• 2 years ago

The best deep learning time series models are closed source inside hedge funds.

3abiton• 2 years ago

I think hedge funds, at least the advanced once, definitely don't use time series modelling anymore. That's quit outdated nowadays.

max_• 2 years ago

What do you suspect they are using?

rjurney2• 2 years ago

There are many ways of approaching quantitative trading and many people do employ time series analysis, especially for high frequency trading.

fermisea• 2 years ago

Most of the hard work is actually feature construction rather than monolithic models. And afaik gradient boosting still rules the world

energy123• 2 years ago

There is no such thing as a generally best model due to the no free lunch theorem. What works in hedge funds will be bad in other areas that need less or different inductive biases due to having more or less data and different data.

thedudeabides5• 2 years ago

cant wait for someone to lose all their money trying to predict stocks with this thing

nyanpasu64• 2 years ago

I misread this as XSLT :')

avidphantasm• 2 years ago

Same. I am old?

ThomasBHickey• 2 years ago

Me too (and yes, I'm old)

mikepurvis• 2 years ago

100% clicked thinking I was getting into an article about XML and wondering how interesting that was in 2024. Simultaneously disappointed and pleased.

antod• 2 years ago

Yup. And it's about transforms too.

optimalsolver• 2 years ago

Reminder: If someone's time series forecasting method worked, they wouldn't be publishing it.

dongobread3• 2 years ago

They definitely would and do, the vast majority of time series work is not about asset prices or beating the stock market

musleh22• 2 years ago

The Transformer model, despite being one of the most successful in AI history, was still being published.

logicchains• 2 years ago

It's a sequence model, not a time-series model. All time series are sequences but not all sequences are time series.

dlojudice• 2 years ago

Is this somehow related to the Google weather prediction model using AI [1]?

https://deepmind.google/discover/blog/graphcast-ai-model-for...

scellus• 2 years ago

No, Graphcast is a graph transformer trained on ERA5 weather reconstructions of the atmosphere, not a general time series prediction model. It by the way outperforms all traditional global point forecasts (non-ensembles), at least on predicting large-scale global patterns (Z500 and such, on the lag of 3–10 days or so). ECMWF has AIFS that is a derivate of Graphcast, they'll probably get it or something similar to production in a couple of years.

wafngar• 2 years ago

AIFS is transformer based (Graphcast is pure GNN) so different architecture and is already running operationally, see:

https://www.ecmwf.int/en/about/media-centre/aifs-blog/2024/i...

brcmthrowaway• 2 years ago

Wow, is there a way to apply this to financial trading?

musleh22• 2 years ago

If you have dataset in financial , I can try it for you

abrichr• 2 years ago

The paper links to their code:

https://github.com/muslehal/xLSTMTime

And their data, which includes daily exchange rates:

https://drive.google.com/drive/folders/1nuMUIADOc1BNN-uDO2N7...

localfirst• 2 years ago

time series forecasting works best with deterministic domains. none of the published LLM/AI/Deep/Machine techniques do well in the stock market. Absolutely none. we've tried them all.

dkga• 2 years ago

A part of my work is literally building nowcasting and other types of prediction models in economics (inflation, GDP etc) and finance (market liquidity, etc). I haven’t yet had a chance to read the paper but overall the tone of “transformers are great for what they do but LSTM-type of models are very valuable still” completely resonates with me.

uoaei• 2 years ago

Have you had the chance to apply Mamba to your work at all? Thoughts?

_0ffh• 2 years ago

Too bad the dataset link in the paper isn't working. I hope that'll get amended.

XLSTMTime: Long-Term Time Series Forecasting with xLSTM

Related Stories