Why do we still flatten embedding spaces?

https://news.ycombinator.com/item?id=44628840

7 11 1 year ago

news.ycombinator.com

ost dense retrieval systems rely on cosine similarity or dot-product, which implicitly assumes a flat embedding space. But embedding spaces often live on curved manifolds with non-uniform structure—dense regions, semantic gaps, asymmetric paths.

I’ve been exploring the use of:

- Ricci curvature as a reranking signal

- Soft-graphs to preserve local density

- Geodesic-aware losses during training

Curious if others have tried anything similar? Especially in information retrieval, QA, or explainability. Happy to share some experiments (FiQA/BEIR) if there's interest.

PaulHoule3• 1 year ago

I've been bothered by this since before there were transformers.

Probably the most interesting function over t is G(t), that function Chomsky said was the grammar in that it is true if t is well-formed and false if it isn't.

G(t) over t is not a manifold because it is not continuous and its projection in the embedding space can't be continuous either. It boggles my mind, and leaves me thinking that it's not legitimate to work in the embedding space but it obviously works.

If you have two points in the embedding space which represent well-formed sequences and draw a line that interpolates between them you'd think that there would have to be points in between that correspond to ill-formed sequences. Intuition over high dimensional spaces is problematic, but I imagine there have to be structures in there that "look" like a crumpled up ball of 2-d paper in a 3-d space or are folded up like filo dough.

Intrinisical-AI5• 1 year ago

That's fascinating! — but I don't fully agree with the framing.

Using G(t) in the context of embeddings seems problematic, specially given the probabilitistic nature.

Example - Take a sentence with a typo but semantically clear and correct (let's suppose): "The justice sistem is corrupt."

G(t) = 0, right? But semantically, it's close to G(t) → 1.

Instead of focusing on exact validity —which for me seems too rigid for something as ambiguous and context-dependent as language— what if we focused on _approximate semantic trajectories_?

You wrote:

"If you have two points in the embedding space which represent well-formed sequences and draw a line that interpolates between them you'd think that there would have to be points in between that correspond to ill-formed sequences."

In my view, it's actually the opposite:

If the embedding model captures meaningful structure, and you account for geometric properties like curvature, local density, and geodesics — then the path between those two points should ideally trace semantically valid (even "optimal" - if that exists) reasoning.

The problem isn't that interpolation fails — it's that we're interpolating linearly in a space that likely isn't flat!

Thanks for your comment. Lmk what you think :)

PaulHoule3• 1 year ago

You're right that "systems competent in language" (either humans or LLMs) are able to accept and understand slightly wrong sequences but generate correct sequences almost all of the time. (Who hasn't made a typo when talking to a chatbot and had the chatbot ignore the type and respond correctly?)

Treating G(t) as a binary function works for linguists who need a paradigm to do "normal science" but Chomsky's theory has not been so useful for building linguistically competent machines so there have to be serious things wrong with that theory.

Still, the vast majority of sequences t are gibberish that is nowhere near being valid. If those gibberish sequences are representable in the embedding space and took up a volume anywhere near the numeric prevalance they have I can only imagine that in a (say) N=3000 embedding space there is something like a manifold that is N=2999 or N=2998 or N=1500 or something inside the flat embedding space -- that structure would be the non-flat embedding you're looking for or be an approximation to it.

It might be that it is not really a manifold or has different dimensionalities in different places or even fractional dimensionalites. For instance you'd hope that would geometrically represent semantics of various sorts as suggested by the graphs here

https://nlp.stanford.edu/projects/glove/ [1]

So I've thought a lot about sympletic spaces in higher dimensions where area has to be conserved over various transformations (the propagator) and maybe this has led me to think about it the totally wrong way -- maybe the flat embedding space doesn't devote a large volume to gibberish because it was never trained to model gibberish strings, which has to have interesting implications if that is true.

Something else I think of is John Wheeler's idea of superspace in quantum gravity where, even though space-time looks like a smooth manifold to us, the correct representation in the quantum theory might be discrete: maybe for points a, b there are the possibilities that: (1) a,b are the same point, (2) a is the future or b, (3) b is the future of a, or (4) a and b are not causally connected. So you have this thing which exists on one level as something basically symbolic but looks like a manifold if you live in it and you're much bigger than the Planck length.

[1] ... which I don't really believe, of course you can project out 20 points out of a 50 dimensional embedding into an N=2 space and have the points land wherever you want!

PaulHoule3• 1 year ago

https://nlp.stanford.edu/projects/glove/ [1]

But to get to that answer of "why do we flatten it?", we're not flattening it deliberately, the "flattening" is done by the neural network and we don't know another way to do it.

[1] ... which I don't really believe, of course you can project out 20 points out of a 50 dimensional embedding into an N=2 space and have the points land wherever you want!

bravesoul22• 1 year ago

I found your medium article on this subject by hunting around. It's very interesting. Hope you write more.

From a layman's point of view, say this is true and embeddings should be a manifold. Like a horse saddle or sphere for example. Then some adding of vectors the simple way won't make sense. On a sphere west in China plus west in US is double west but in 3d it's zero as they point in different directions. Is this sort of the idea?

Intrinisical-AI5• 1 year ago

Hey man! Thanks a lot for your support! Might sound like just common words, but honestly — knowing this helped or inspired someone really motivates me. Makes me feel a bit less like a madman hahaha.

About your question — I think I get your point. Here’s how I understand it:

In the hypothetical latent manifold, we might want to measure two different things:

1. Distance between two points: This wouldn’t be the usual Euclidean distance, because the space is curved. Like how the shortest path between two cities on Earth isn't a straight line, but an arc on the globe. That’s where geodesics come in — they’re the shortest paths constrained by the shape of the manifold.

2. Similarity between two vectors ("parallel transport") Instead of asking where the vectors point in ambient space (like cosine similarity in R^n), we (should) care about how their directions compare _on the surface itself_. So ideally, we’d compare them along the geodesic — parallel transport — to properly align their frames of reference before measuring any angle or similarity.

That’s the intuition I’m working with, anyway. Let me know what you think, and thanks again for your comment!

bravesoul22• 1 year ago

Thanks! I am not used to thinking in geodesics. And while I studied vector spaces it was a long time ago and forgotten most of it!

So to get the intuition. Your model we still are dealing with vectors like regualar searches and not positions?

This is the model in my mind. A 2D surface with a steep mound in hidden dimension 3 to the north. 2 vectors at 1 degree apart pointing toward the mound are likely to be more different than 2 vectors 1 degree apart pointing away from it. But you only see this if you are right on the foot of the mound. What if you are further away?

Anyway I think my understanding is off it may not be worth answering! but I can read more about it.

TXTOS• 1 year ago

Really love this question — been thinking about it for a while now.

We’ve been hacking on a different approach we call the WFGY Engine — it treats embedding space not as flat or uniform, but more like a semantic energy field. So instead of forcing everything into clean cosine ops, we track these “semantic residue structures” — kind of like how meaning resists compression when you flatten it too early.

We measure stuff like ΔS (semantic tension), λ_observe (viewpoint drift), and E_resonance (contextual energy), which lets us reason across curved or even broken spaces where normal LLMs kinda lose track. It’s a mix between geodesic logic and language field dynamics — weird but surprisingly stable.

A couple of early apps built on this — TXT OS, BlahBlahBlah, etc — ended up scoring 100/100 from 6 major AI models, which was a cool surprise. We’re still in early dev, but even Tesseract gave it a formal endorsement recently, which was huge for us.

Anyway, core engine is all open source if anyone’s curious: https://github.com/onestardao/WFGY

Would love to hear if others are exploring non-flat logic or weird manifold behavior in language space too.

can link the Tesseract thing if folks want

Intrinisical-AI5• 1 year ago

Love your project. And the mentiond metrics ΔS (semantic tension), λ_observe (viewpoint drift), and E_resonance (contextual energy) totally align with my mental model.

Also loved your phrase: "kind of like how meaning resists compression when you flatten it too early." That's a key point. Implicit in that is the assumption that flattening will eventually happen — and I agree (assumed for efficience in last term): Just like how we create flat maps of the Earth sometimes a flat projection is useful _as long as_ we understand the properties of the deformation and the underlying surface. If we don't, EEUU might seem a lot bigger than it's "represented", in our euclidean view.

That interpretation really clicks for me.

I'm currently exploring *geodesic-aware training methods for IR*, based on contrastive / triplet loss. Take a look if you're curious: [https://github.com/Intrinsical-AI/geometric-aware-retrieval-...](https://github.com/Intrinsical-AI/geometric-aware-retrieval-...)

Still recent and (very) experimental — but already done some initial tests. The goal is to construct a differentiable graph over the embedding space and propagate training signals end-to-end. Lots of work ahead, but preliminary results are promising (and adjusting things with o3 feedback ofc).

Are you open to collaborating or jamming on this? Would love to sync up — are you on Discord or Slack?

Thanks again for pushing this forward.

Why do we still flatten embedding spaces?

Related Stories