Everything is a linear model

https://danielroelfs.com/blog/everything-is-a-linear-model/

by nopipeline • 2 years ago

224 85 2 years ago

I find it irritating that the article mentions repeated measures, but does not try them, much less a mixed effects model. Yes, they are linear models with more parameters, but doing it in lm would be a special kind of madness

simulo• 2 years ago

I knew the linked-in-the-article https://lindeloev.github.io/tests-as-linear/ which is also great. A bit meta on the widespread use of linear models: "Transcending General Linear Reality" by Andrew Abbott, DOI:10.2307/202114

ivan_ah2• 2 years ago

Here is the Python port of test-as-linear developed by George Ho (eigenfoo): https://github.com/minireference/tests-as-linear/blob/bugfix...

I'm linking to my fork of it because I've added some fixes and filled in some of the missing parts (e.g. Welch's t-test).

LifeIsBio• 2 years ago

I read this article when I was in grad school 5 years ago. Absolutely love it and talk about it to this day.

It really makes me frustrated about the ways I was introduced to statistics: brute force memorization of seeming arbitrary formulas.

btdmaster2• 2 years ago

I thought nonlinearity was very important to be able to make a larger model better than a smaller one? Like so important that tom7 made a half-joke demo with it: https://yewtu.be/watch?v=Ae9EKCyI1xU

epgui2• 2 years ago

Linear models don’t need everything to be linear.

iamcreasy2• 2 years ago

I presume you are implying that linear model only mandates linear relationship between predictor and regression coefficients?

dist-epoch• 2 years ago

Well, you can create a non-linear model by piece-wise combining multiple linear models.

The famous ReLU non-linearity is just that - two linear functions joined.

nyrikki2• 2 years ago

same thing with any feed forward network too. They are all piece-wise linear in respect to inputs.

Layers reduce resource requirements and make some patterns easier or even practical to find, but any ANN that is a FNN supervised learning could be represented as a parametric linear regression.

Unsupervised learning, that tends to use clustering is harder to visualize but is the same thing.

You still have ANNs, which have binary output, which can be viewed through the lens of deciders. They have to have unique successor and predecessor functions.

Really this is just set shattering that relates to a finite VC dimensionality being required for something to be PAC learnable.

But the title of this is confusing the map for the territory. It isn't that 'Everything is a linear model' but that linear models are the preferred, most practical form.

The efforts to leverage spikey neutral networks, which is a more realistic model of cortical neurons, and which have continuous output (or more correctly the computable reals) tend to run into problems like riddled basins.

https://arxiv.org/abs/1711.02160

Obviously setting rectified linear unit at 0 = 1 resolves to differentiation problem, but many functions may not be so simple

Perhaps a useful lens is how TSP with a discreet Euclidean metric is in NP-complete while the continuous version is in NP-hard.

But it isn't that everything is linearizable, but rather that linearized problems tend to be the most practical.

btdmaster2• 2 years ago

I see what you mean. Though in my mind, and this is clearly subjective, piece-wise linear is at least less strict than a linear model. (With enough ReLUs, you could get arbitrarily close to a lookup table, which I think would be best described as a nonlinear model.)

tnecniv• 2 years ago

Nonlinear things start looking like linear things again in very high dimensions

nyrikki2• 2 years ago

Only when you dimensions are truly independent and that is a stretch. Really what you are saying is that you are more likely to find a field for your problem, and fields don't exist in more than 2 dimensions.

Consider Predator Pray with fear and refuge, which is indeterminate, and not due to a lack of precision but a topological feature where ≥3 open sets share the same boundary set.

https://www.sciencedirect.com/science/article/abs/pii/S09600...

General relativity, with 3 spacial and one temporal dimension is another. One lens to consider this is that rotations are hyperbolic due to the lack of independence from the time dimension.

Quantum mechanics would have been much more difficult if it didn't have two exit basins. Which is similar to ANNs and linear regressions being binary output.

(Some exceptions will orthogonal dimensions like EM)

dboreham• 2 years ago

Hmm...I thought everything was an Eigenfunction.

rzzzt• 2 years ago

Applying basic syllogism, Eigenfunction is securities fraud.

ggm• 2 years ago

If you bundle up enough securities of dubious value, it creates less dubious value higher than the sum of the parts.

jwilber• 2 years ago

Another fun stats X is really Y:

estimating the Area Under the Curve metric (AUC) is equivalent to the Wilcoxon-Mann-Whitney test!

https://rmets.onlinelibrary.wiley.com/doi/abs/10.1256/003590...

t_mann• 2 years ago

Statistics is more than hypothesis testing, but you'll get surprisingly far without straying too far from linear models - I remember a Stats prof saying 'most of classical Statistics is GLM [0]'

[0] https://en.wikipedia.org/wiki/Generalized_linear_model

lanstin• 2 years ago

Which means it really all just finding hyperplanes that are near the data.

sebastianavina2• 2 years ago

"Classification of mathematical problems as linear and nonlinear is like classification of the Universe as bananas and non-bananas. "

and everything turns around the same principles. For example dynamical models and PID controls.

yet solving a banana, is the only thing we really know how to do. So we end up fitting everything in our banana models.

Tommah• 2 years ago

Linear algebra was the first math class I took in undergrad. I thought the next one was going to be non-linear algebra! But it wasn't.

whatshisface8• 2 years ago

I disagree with the implication that linearity is an unnatural concept, it appears whenever the changes being studied are small relative to the key parameters that determine the system. Every system is linear for small perturbations. Even logic gates; in negative feedback they can form passable inverting amplifiers. In a place as big as the universe it is rather common for two things to be very different in scale and yet interacting.

SonOfLilit• 2 years ago

My favorite moment in university was in the first class of semester 2, where a prof said "lets look at a really small part of our thing, and assume we apply some force to it. This will make it stretch, lets assume the stretching is linear relative to the applied force". I raised my hand and asked "is this assumption supported empirically?" and he said "no, we know it's not always true, but if we don't make it we can't calculate anything".

At the time I was mad at engineers for being non-scientific, but after a few years I understood the deep wisdom in that. Nonlinear materials exist, and materials we use have nonlinear ranges. We just don't build things from those, because the math is too unwieldy. (except in very very specific edge cases where we spend a lot of money building a very limited thing)

3abiton• 2 years ago

It's true, until it's not. It's easy to make these claims, but would you bet your money on it?

ofrzeta• 2 years ago

I thought everything was a power function.

smitty1e• 2 years ago

I thought everything was a graph.

optimalsolver• 2 years ago

Nah, it's all just ifs and for-loops:

https://www.reddit.com/media?url=https%3A%2F%2Fi.redd.it%2F4...

sva_• 2 years ago

(2022)

chmaynard• 2 years ago

RSS feed: https://danielroelfs.com/blog/index.xml

matteoraso• 2 years ago

A cool thing about linear models is that they can be used to model non-linear correlations by using transformations. For example, an exponential function can be made linear if you just take the logarithm.

johnsutor• 2 years ago

See the topic of GLMs: https://en.wikipedia.org/wiki/Generalized_linear_model

Horffupolde• 2 years ago

Not all functions have a trivial transformation.

gwern• 2 years ago

Doesn't have to be 'trivial'. It could be a trillion-parameter MoE LLM spitting out 10,000-large embeddings. But a lot of that will act linearly and can, in theory, be put into your linear model easily & profitably.

richrichie2• 2 years ago

You can state and prove theorems with linear models. You can do inference and testing. This means papers and academics naturally love them. And therefore, they are everywhere.

Not the case with non-linear models. We need to throw computers at them.

whatshisface8• 2 years ago

I think that's an overstatement. You can consider the nonlinearity as a perturbation of the linear theory, study the topology of the solutions or their symmetries, or sometimes even find exact solutions, like exist for a handful of transistor circuits.

richrichie2• 2 years ago

If the model is y = a + b * x + e, then we can precisely state probability distributions of a and b given x and y for reasonably general assumptions about error e.

If the model is y = f(x) + e, where for example f is a neural net, there is not much we can say about the “quality” of parameters of f. That is, we can’t attach confidence interval. We will have to resort to expensive simulations and for most practical applications this is not workable (size of dataset).

What you say is useful in a different context. Local linearization is a very powerful idea.

cs702• 2 years ago

Maybe it's because to scientists and statisticians comfortable with linear algebra, everything looks like a linear model.

levocardia3• 2 years ago

I see a lot of comments here assuming "linear model" means "can't model nonlinearities." Absolutely not the case. Splines can easily take care of that. The "linear" part of linear model just means "linear in the predictor space." You can add a non-linear predictor easily via spline basis (similar/sometimes identical to "kernels" in ML).

My series of lm/glm/gam/gamm revelations was:

1. All t-tests and ANOVA flavors are just linear models

2. Linear models are just a special case of generalized linear models (GLMs), which can deal with binary or count data too

3. All linear models and GLMs are just special cases of generalized linear mixed models, which can deal with repeated measures, grouped data, and other non-iid clustering

4. Linearity is usually a bad assumption, which can easily be overcome via splines

5. Estimating a spline smoothing penalty is the same thing as estimating the variance for a random effect in a a mixed model, so #3 and #4 can be combined for free

And then you end up with a generalized additive mixed model (GAMM), which can model smooth nonlinear functions of many variables, smooth interaction surfaces between variables (e.g. latitude/longitude), handle repeated measurements and grouping, and deals with many types of outcomes, including continuous, binary yes/no, count, ordinal categories, or survival time measurements.

All while yielding statistically valid confidence intervals, and typically only taking a few minutes of CPU time even on datasets with hundreds of thousands / millions of datapoints.

carlthome• 2 years ago

This sounds really cool but was hard to digest for me as a ML Engineer who came into work just around deep learning and DNNs.

Is there some go-to practice material I could look at?

Splines I haven't touched since numerical computing exercises in school.

goosedragons• 2 years ago

Simon Wood's Generalized Additive Model book.

levocardia3• 2 years ago

If you know DNNs, you'll find the introduction of splines in Semiparametric Regression in R very intuitive - splines are introduced using what the authors call a "truncated line basis" but you already know it as the RELU function, just with a bias. Indeed, even the penalization of splines will look extremely familiar: it's basically just L2 regularization to induce smoothness.

You might also enjoy reading the "Neural Additive Model" paper from Hinton's lab, which is basically GAMs using a separate DNN as a "spline basis" for each input variable.

idiotsecant• 2 years ago

Widely known in the controls engineering world for the last hundred years as 'anything is linear if you zoom in far enough'

vitus• 2 years ago

Well, subject to certain constraints.

You aren't guaranteed that your equilibria behave the same in the linearization of a nonlinear system if your Jacobian has any eigenvalues with real part of 0.

https://en.wikipedia.org/wiki/Hartman%E2%80%93Grobman_theore...

new_username01• 2 years ago

It's amazing how many ideas I've seen being "rediscovered" these days from control theory.

kqr2• 2 years ago

Wait isn't that just an application of Taylor expansion which analysis has known about for several hundreds of years?

rcfox• 2 years ago

Alternatively, anything is linear if you add enough dimensions.

mjcohen• 2 years ago

In the 70's in CS grad school at USC, I wrote an adaptive least squares cubic spline fit routine. Kept subdividing intervals and fitting a least squares cubic spline in each interval until a criteria was met.

Fun times.

Don't have the code or writeup any more.

EvgeniyZh2• 2 years ago

"You can make linear model non linear by adding non linearity" is kind of vacuous statement, isn't it

ivan_ah2• 2 years ago

No no, linear just means some expression of the form

    y = b1*x1 + b2*x2 + ... + bp*xp

which is in the heart of the model (perhaps more specific description would be linear combination). Whatever you call it, it's just some quantity y, that is constructed via an additive process from components x1...xp, and each component is multiplied by some constant (the coefficient of the linear combination).

This linear combination "core" of the model can be used directly (in which case it is geometrically a hyperplane), but you could also use non-linear transformation of the inputs or the outputs.

E.g. x1 could be a spline basis function, or log-transformed data, or anything.

Similarly, the output y can be passed though some non-linearity to force a particular output (like in logistic regression). Hence the statement is not vacuous

2-718-281-8282• 2 years ago

Splines can easily take care of that.

within set boundaries.

usgroup• 2 years ago

GLMs are a non linear transformation on an output followed by linear modelling. They are referred to as “linear models” but you might as well then also consider NNs as linear models, or any model at all which ends with addition as the final step.

fho• 2 years ago

Can you recommend some resources to go from step 3 to 4?

So far I have been successfully using GLMMs where appropriate, but then jumped to implementing completely arbitrary models by fitting them to the data (plus bootstrapping).

But that takes ages for most problems as you say

2-718-281-8282• 2 years ago

isn't every non linear model fundamentally linear if it is run on a von neumann computer?

kqr2• 2 years ago

This is why I feel like I couldn't become a credible AI/ML consultant. I would just throw everything into a linear model, make some progress, and call it a day.

blitzar2• 2 years ago

99% of AI / ML consultants are just throwing everything into a linear model, make no progress, slap a big AI sticker on the box and call it a day.

You would have fitted right in.

ronald_raygun• 2 years ago

Wasn’t there an OpenAI paper where they showed that MM multiplication with the numerical imprecision of floats was enough to get general function learning?

You might not even need anything else…

funcDropShadow• 2 years ago

Everything is linear if plotted log-log with a fat magic marker (Mar's Law) [1].

[1] https://spacecraft.ssl.umd.edu/akins_laws.html

Everything is a linear model

Related Stories