Guide to Machine Learning with Geometric, Topological, and Algebraic Structures

https://www.arxiv.org/abs/2407.09468

by johmathe • 2 years ago

176 27 2 years ago

dpflan5• 2 years ago

The paper's references have some good ones for getting more acquainted with these subjects; this one being a nice dense one to start with:

- Geometric Deep Learning Grids, Groups, Graphs, Geodesics, and Gauges: https://geometricdeeplearning.com/

data_maan• 2 years ago

Don't read that. That's one of the most horrible papers I know, a hodgepodge of mathematical well-known concepts thrown together with some vague ideas how they connect. The mathematical parts are explained better in any undergraduate book. I'm not sure why the author felt the need to expound on what a manifold is when that has been done better in hundreds of other texts - literally.

And it lacks a definite conclusion: They don't prove anything, don't make any particular experiment, but just loosely talk about how these ideas might be relevant to machine learning.

I'm surprised that such highly cited researcher have produced such a paper. I would be embarrassed to be on it - and I'm embarrassed on behalf of the ML community that they are citing it.

dpflan5• 2 years ago

Do you have better references you could share?

funnygiraffe3• 2 years ago

Is geometric, topological, and algebraic ML/data analysis actually used in the industry? It is certainly beautiful math. However, during grad school I met a few pure math PhD students who were saying that after finishing their PhD they will just go into industry to do topological data analysis (this was about 10 years ago and ML wasn't yet as hyped up). However, I have never heard of anybody actually having success on that plan.

dpflan5• 2 years ago

I believe a use-case(s) receiving attention is drug design, protein design, chemical design, etc.

Here is a summer school by the London Geometry and Machine Learning group where research topics are shared and discussed. - https://www.logml.ai/

Here is another group, a weekly reading group on graphs and geometry: https://portal.valencelabs.com/logg

funnygiraffe3• 2 years ago

Thanks. That's certainly very interesting. Albeit it seems to me that the number of jobs doing geometric and topological ML/AI work in the drug or protein design space would be quite limited, because any discovery ultimately has to be validated through a wet lab process (or perhaps phase 1-3 clinical trials for drugs) which is expensive and time-consuming. However, I'm very uninformed and perhaps there is indeed a sizable job market here.

heyitsguay• 2 years ago

As someone who did an applied math PhD before drifting towards ML, it's worth pointing out that these applied math groups typically talk about applications, but the real question is whether they are actually used for the stated application in practice due to outperforming methods that use less pretty math. Typically (in every case i have seen) the answer is "no", and the mathematicians don't even really care about solving the applied problems nor fully understand what it would mean to do so. It's just a source of grant-justifiable abstract problems.

I would love to be proven wrong though!

llm_trw3• 2 years ago

I've had some success using hyperbolic embeddings for bert like models.

It's not something that the companies I've worked for advertised or wrote papers about.

jqgatsby• 2 years ago

Hyperbolic embeddings have been an interest of mine ever since the Max Nickel paper. Would love to connect directly to discuss this topic if you're open. here's my email: https://photos.app.goo.gl/1khCwXBsVBuEP6xF7

nborwankar• 2 years ago

Can you share what kinds of problems were conducive to hyperbolic embeddings in your experience. Also, separately, are you saying companies are using these in practice but don’t talk about them because of the advantage they give? Or am I reading too much into your last sentence.

fjork• 2 years ago

I don't think there's much use currently. But I kinda like the direction of the paper anyway. Most mathematical objects in ML have geometric or topological structure, implicitly defined. By making that structure explicit, we at worst have a fresh new perspective on some ML thing. Like how viewing the complex numbers on a 2d cartesian plane often clicks more for students compared to the dry algebraic perspective. So even in the worst case I think there's some pedagogical clarity here.

itissid• 2 years ago

One common theme I see in the paper(e.g. in protein folding) is:

"Identify what properties are important (geometry, algebra, topo) and which one is an useful prior and then "use" the guide to select an initial struct. This is probably harder than it sounds(unlike bayesian priors which are more forgiving for one to select, but quite like them in that they both require special assumptions)."

I wonder: could one use it to bring together certain multimodal data and a proposed network for a task? Like could one bring in sensor, map topology, urban topology, pictures which have certain properties and that help me use this guide to make a statement like : "Street data could be embedded with Sensor data to do ABC kind of inference using XYZ NNetwork structure because this paper suggests that is a reasonable thing to do"?

uoaei• 2 years ago

All machine learning is just embedding of various forms. If you have a way to translate disparate types of data into a common space, in ways that preserve inductive bias and information content, you can then combine them for downstream tasks.

mjhay• 2 years ago

I am 100% convinced that these kind of approaches will be what delivers ML research from the current resource-hungry and ungeneralizable status quo. Low-dimensional Euclidean geometry is special. Higher-dimensional Euclidean spaces are less special. Most real-life data is high-dimensional, not at all smooth, and possessing a structure you cannot call Euclidean with a straight face. Look at what works with tabular data (which is probably most of what practitioners work with in the wild). It's gradient boosted trees, not neural networks.

There is a fundamental mismatch between the data we usually work with and the spaces we shove it into. Tools from algebraic topology and geometry are old hat in physics. If anything, they should be even more useful in ML.

Davidzheng• 2 years ago

I heavily disagree with the statement "tools of algebraic topology is old hat in physics"

BenoitP• 2 years ago

Well, I consider Lorentz' work to be old hat. I can't find an older example though.

https://en.m.wikipedia.org/wiki/Lorentz_group

OutOfHere2• 2 years ago

Comment is heavily exaggerated in every way.

mistrial9• 2 years ago

some people on this thread are asking about jobs. The bigger picture here is that previously intractable problems are going to be solved with a new combination of math, data and compute.. there are lots of commercial cases that will change dramatically. How can individual people or small groups benefit from serious problem solving, economically?

funnygiraffe3• 2 years ago

Why are we so sure that a lot of "previously intractable problems" are/will be solved with this family of methods? (and I mean real-life/real-world problems, not toy problems constructed specifically to show the proposed methods in the best light possible in the research papers) Of course others above have pointed out drug or protein design as a potential area, but there still seems uncertainty as to the practical impact on the real world. Other than that, I don't see areas of impact for these approaches so far.

johmathe2• 2 years ago

One of the authors here. Thanks for your comment! While a lot of this research is theoretical and does not have immediate use cases, we have tried to summarize some of them in the last paragraph of the paper (VII. Applications of Non-Euclidean Geometry). See page 26. We present some in Chemistry and Drug Development, Structural Biology and Protein Engineering, Computer Vision, Biomedical Imaging, Recommender Systems and Social Networks and Physics.

OutOfHere2• 2 years ago

Note that the GPU hardware is setup for Euclidean matrix operations. Even if you had a deep structure learner, it won't necessarily help you if you have to go back to emulating it on Euclidean hardware.

Pseudomanifold• 2 years ago

Great selection of works, but I am missing a lot of references from topology in ML, with the article only assuming a very cursory perspective in terms of 'topology captures connectivity and/or continuity.'

Some works from my colleagues and me go a little bit deeper (no pun intended), for instance:

- Neural Persistence Dynamics: https://arxiv.org/abs/2405.15732

- Simplicial Representation Learning with Neural $k$-Forms: https://openreview.net/forum?id=Djw0XhjHZb

- A general review on topology in machine learning: https://www.frontiersin.org/journals/artificial-intelligence...

There are more things in topology and machine learning, Horatio, than are dreamt of in your article ;-)

johmathe2• 2 years ago

Thanks for the great pointers!

Guide to Machine Learning with Geometric, Topological, and Algebraic Structures

Related Stories