Google's 200M-parameter time-series foundation model with 16k context

Wish they gave some numbers for total GPU hours to train this model, seems comparatively tiny when compared to LLMs so interested to know how close this is to something trainable by your average hobbyist/university/small lab

OliverGuy2• 4 months ago

Edit, it looks like the paper does

TPUv5e with 16 tensor cores for 2 days for the 200M param model.

Claude reckons this is 60 hours on a 8xA100 rig, so very accessibile compared to LLMs for smaller labs

EmilStenstrom2• 4 months ago

I somehow find the concept of a general time series model strange. How can the same model predict egg prices in Italy, and global inflation in a reliable way?

And how would you even use this model, given that there are no explanations that help you trust where the prediction comes from…

teruakohatu3• 4 months ago

What is not generally understood is that these models don’t predict egg prices or inflation in Italy.

They decompose a time series into trends, seasonality and residuals. That’s what they are actually modelling.

They cannot predict wars in the Middle East influencing inflation unless there is a seasonal pattern(s).

visarga• 4 months ago

ARIMA and ARMA models

d--b2• 4 months ago

The main issue is that people do use them to predict bitcoin prices intraday and that sort of things.

cybrox• 4 months ago

Wars in the middle east seem to have increasingly regular patterns tied to stock market opening hours, unfortunately.

perks_12• 4 months ago

I am not familiar with time series models, but judging from your answer, it would be necessary to feed long time series into this model for it to detect trends. What is a token here? Can it, for the lack of a better example, take in all intraday movements of a stock for a day, a week, a month, etc?

ReptileMan• 4 months ago

It is the Middle East. Wars are always in season. And supply is more than the demand.

lordgrenville• 4 months ago

That's what traditional time-series modelling does. This is a foundational model, which means it's just a neural network trained on lots of time series. (So maybe OP's question still stands? But it's the same question as "how can LLMs be good at so many different kinds of conversations?")

graemep• 4 months ago

Do these models predict on just a single time series then?

it is far more useful for predictions to look for correlations between time series. This is far more complex than looking for correlations in general because most time series trend up or down and therefore correlate.

amelius2• 4 months ago

What makes these models different from models used for e.g. audio?

Or other low-dimensional time domain signals?

jcelerier• 4 months ago

They cannot predict wars in the Middle East influencing inflation unless there is a seasonal pattern(s).

well...

a-dub• 4 months ago

ar(k) stuff, sure. that's old news. i would expect the newfangled stuff to be good at 0-shot learning of pre-event signatures spread across multiple series, at a minimum.

benob• 4 months ago

I would say:

- decomposition: discover a more general form of Fourrier transform to untangle the underlying factors

- memorization: some patterns are recurrent in many domains such as power low

- multitask: exploit cross-domain connections such as weather vs electricity

lovelearning• 4 months ago

My understanding is that the synthetic training data helps capture abstract time-series patterns that are common in all domains.

As they say in appendix 8:

We create the synthetic data to reflect common time-series patterns using traditional statistical models. We start with four simple times series patterns:
• Piece-wise linear trends (I), where the number of the piece-wise linear components is randomly chosen between 2 and 8.
• ARMA(p, q) (II), where 1 ≤ p, q ≤ 8 and the corresponding coefficients are generated from either a multivariate Gaussian or a uniform, then normalized.
• Seasonal patterns. In particular we create the sine (III) and the cosine (IV) waves of different random periods between 4 and max context length / 2 time-points and time delays.

If there were no such underlying patterns in the class of all time-series data, then even the idea of traditional time-series models would be fundamentally misplaced.

And since this is a transformer model, it also looks for patterns in the problem-specific input data at inference time, just like how the input context to an LLM influences its output's relevance.

eru5• 4 months ago

How can the same model predict egg prices in Italy, and global inflation in a reliable way?

How can the same lossy compression algorithm (eg JPG) compress pictures of everything in a reliable way?

cenamus• 4 months ago

It can't compress pictures of everything in a reliable way.

Text and anything with lots of high frequency components looks terrible

annie511266728• 4 months ago

It’s not really predicting “egg prices” or “inflation” — it’s mostly fitting patterns that happen to show up in those series.

The problem isn’t domain generalization, it’s that we keep pretending these models have any notion of what the data means.

People ask how one model can understand everything, but that assumes there’s any understanding involved at all.

At some point you have to ask: how much of “forecasting” is actually anything more than curve fitting with better marketing?

fjdjshsh• 4 months ago

"curve-fitting" has a long history (centuries old) and could be regarded more as a numerical method issue.

Rigorous understanding of what is over fitting, techniques to avoid it and select the right complexity of the model, etc, are much newer. This is a statistical issue.

My point is that forecasting isn't curve fitting, even thought curve fitting is one element of it.

nairadithya• 4 months ago

I don't know how I feel about LLM slop coming to HN.

JackeJR• 4 months ago

Actually it can. See https://youtu.be/FUQwijSDzg8?si=LWd5gVNYRd3HH9rJ

Or just search for the James-Stein paradox.

thesz• 4 months ago

  > How can the same model predict egg prices in Italy, and global inflation in a reliable way?

For one, there's Benford's law: https://en.wikipedia.org/wiki/Benford%27s_law

So, predict sign (branch predictors in modern CPUs also use neural networks of sorts), exponent (most probably it changes slowly) and then predict mantissa using Benford's law.

samuelknight• 4 months ago

I think that a model designed to ignore semantic chatter like financial news and deeply inspect the raw data is a very powerful perspective.

ludicrousdispla• 4 months ago

It's best to think of it as a giant tree, from which you can pick cherries.

strongpigeon• 4 months ago

When I worked on Google Ads, we used time series forecasting to compute the odds of an ad campaign reaching its goal (and to tell users how likely they were to hit them).

A ton of (unsophisticated) advertisers would just draw a line from zero to the number they are at today and project that line to the end of the month to forecast the amount of conversions/spend they were going to hit. This of course doesn't take into account various seasonalities (day-of-week, time-of-year, etc.) and gives you a pretty poor forecast. Compared to those, time-series forecasting is much more accurate.

Is it perfectly accurate? No, that's impossible. But when you can train a model on all advertising campaigns, you can give good 95% confidence intervals.

nurettin• 4 months ago

predict egg prices in Italy, and global inflation in a reliable way?

Easy, both go up.

wiradikusuma• 4 months ago

Also: https://github.com/Nixtla/nixtla and https://facebook.github.io/prophet/

mijailt• 4 months ago

And https://github.com/amazon-science/chronos-forecasting

ra• 4 months ago

This has been around a few months now, has anyone built anything on it?

jdthedisciple2• 4 months ago

Let me be blunt: Shannon would tell us that time forecasting is bullshit:

There is infinitely more entropy in the real world out there than any model can even remotely capture.

The world is not minecraft.

mikkom• 4 months ago

Yeah all weather forecasts are just magic

eru5• 4 months ago

And JPG doesn't work either..

kgwgk• 4 months ago

Whether forecasting is simple: it either rains or it doesn’t. 50/50 probability!

tgv• 4 months ago

Weather forecasts are notoriously iffy, and accuracy drops with time, but we understand the physics behind it (to a large extent). There's also a lot of fine-grained data available. For some arbitrary time series, there's only one data sequence, and the model is unknown. Extrapolation then becomes a lot more magical.

FartyMcFarter3• 4 months ago

Shannon would tell us that time forecasting is bullshit

If you're trying to forecast random data, then yes, it's bullshit. Otherwise you have a chance.

GTP2• 4 months ago

But, if you don't have the information required for a forecast, then the outcome can look random. We know the physics needed to predict the outcome of a dice throw, but, since to predict the outcome you would need a lot of information that you don't have, the output is random to you.

drzaiusx112• 4 months ago

Time series forecasting has proven useful in a number of different domains from weather to health monitoring. Sure you can easily over fit on the training data, but in general that's a data source/input problem where you need many high quality data sources to find the signal in the noise.

The world is chaotic sure, but there are still truths to be found in noisy time series data; saying that the world is too random to be knowable is a bit dismissive, no?

jdthedisciple2• 4 months ago

I agree when it comes to highly niche applications with a generous SNR.

Universal models though?

And I haven't even mentioned the fact that en mass forecasting ITSELF may influence the subject of forecasting.

bwfan123• 4 months ago

time forecasting is bullshit

for a model to be useful, it doesnt need to capture the behavior of a system. It only needs to capture signals which can be useful. For example, for a biased coin toss, a model is already useful if it can predict a little better than random.

dash2• 4 months ago

So the time series are provided with no context? It's just trained on lots of sets of numbers? Then you give it a new set of numbers and it guesses the rest, again with no context?

My guess as to how this would work: the machine will first guess from the data alone if this is one of the categories it has already seen/inferred (share prices, google trend cat searches etc.) Then it'll output a plausible completion for the category.

That doesn't seem as if it will work well for any categories outside the training data. I would rather just use either a simple model (ARIMA or whatever) or a theoretically-informed model. But what do I know.

Tarq0n• 4 months ago

If it works for predicting the next token in a very long stream of tokens, why not. The question is what architecture and training regimen it needs to generalize.

wtyvn• 4 months ago

I think I'm in the same boat as you are, in preferring more conventional approaches to time series analysis.

I'm curious as to how this would compare to having an actual statistician work on your data, because I feel that time series work is as much an art as it is a science. To start, selection of an appropriate timeframe is always important to ensure our data doesn't resemble either white noise or a random walk, and that we've given the response time of our data appropriate consideration! I find that people unfamiliar with statistics miss this point - I get people asking why I might use a weekly or biweekly timeframe for data when they reckon I should be using hourly or daily data. Selection of appropriate predictors is also important for multivariate time series and I have no idea how this model approaches that.

I also have questions about how interpretable the results outputted by this model are. With a more "traditional" model, I can easily look at polyroot or the [P/E]ACF, as well as various other diagnostic tools, and select a relatively simple model that results in a decent 95% prediction interval. I've always been very wary of black box models simply because I wouldn't be able to explain any findings derived from them well.

From skimming the blog post, is MAE all they're using for measuring the output quality?

raghavMultilipi• 4 months ago

This has been around a few months now, has anyone built anything on it?

magimas2• 4 months ago

we did some internal tests. The quality isn't bad, it works quite well. But it's essentially on the same level of an ARIMA model trained on the data just much bigger and slower.

So in my opinion it currently falls into a kind of void. If your use case is worth predicting and you put a data scientist on it, you're better off just training cheaper ARIMA models.

clarionbell• 4 months ago

That is disappointing. One would say that with all the budget and compute, Google would be able to create something that beats methods from 70s. Maybe we are hitting some hard limits.

Maybe it would be better to train an LLM with various tuning methodologies and make a dedicated ARIMA agent. You throw in data, some metadata and requested window of forecast. Out comes parameters for "optimal" conventional model.

emsign• 4 months ago

Can this finally break the stock markets?

GTP2• 4 months ago

The safe bet is no. Based on other comments, this would depend a lot on the specific trends you're trying to predict. But it wouldn't work for everything in the stock market.

kuu• 4 months ago

It would be nice to add (2024) to the title, this is not news (see: https://research.google/blog/a-decoder-only-foundation-model...)

mrklol• 4 months ago

Not directly 2024, there was a big update end 2025

croemer• 4 months ago

(2024)

pplonski86• 4 months ago

Can someone explain ELI5 how it does work? and how many data points it can read?

konschubert• 4 months ago

Let's say I have long time series of past solar irradiation and long time series of past weather forecasts. Can this model make use of weather forecasts for time X in the future to predict electricity prices in the future?

That is, can it use one time series at time X to predict another time series at time X?

Or is this strictly about finding patterns WITHIN a time series.

etrautmann• 4 months ago

The paper suggests it’s for forecasting. How this doesn’t just represent the relatively small number of training samples isn’t obvious to me. If most of the time series for training go up and to the right then I assume that’s what the model will (generally) do, but who knows.

htrp• 4 months ago

isn't this basically prophet?

staticautomatic• 4 months ago

No. Prophet is based on curve-fitting.

mikert89• 4 months ago

I'm willing to bet an intelligent LLM with a dataset and a pandas stats package could outperform this model by running its own experiments and making predictions

doruk101• 4 months ago

Instead of willing to bet, you can do it yourself and prove it. It is not like there is a ceiling for doing what you are proposing. I am willing to bet that you are wrong.

aris0• 4 months ago

Has anyone gotten this to run on MLX yet?

casey2• 4 months ago

Same with all tech scams, Even if you magically assume that they could solve their problem with this tech why on earth would they give it to the public, for free or for a price. Alphabet would just become the best quantitative hedgefund in the world.

technimad• 4 months ago

The Cisco Time Series model is inspired by this model from Google. This one is targeted at observability data and I can confirm it works great in that context https://github.com/splunk/cisco-time-series-model

Google's 200M-parameter time-series foundation model with 16k context

Related Stories