Intrinisical-AI

Absolutely — you’re on the right track! The intuition of the "mound" is actually quite powerful.

Let’s imagine the embedding space as a 2D surface with a third hidden dimension — say, curvature pointing ‘up’ toward a peak (like a saddle or a bump).

Then:

Two vectors pointing in the same compass direction (e.g., “north”) might diverge or converge depending on their local curvature.

If you try to compare them using Euclidean cosine similarity, you’d be blind to the underlying geometry.

What you'd want is to "parallel transport" one vector along the surface toward the other — adjusting its direction to the curvature — and then compare them.

In high-curvature regions, even small angular differences can grow rapidly. In flatter regions, directionality might dominate. So yes, context (i.e., local geometry and density) changes everything.

Right now I'm testing some of this with reranking signals based on Ricci curvature + soft k-NN graphs (preserving density), and trying out simple geodesic-aware losses. But it’s still early days.

Your comment actually gave me a better mental image of what happens when you’re “far from the mound” — maybe curvature becomes negligible, and naive methods recover. Thanks for that.

Wow! you brought up several deep ideas that deserve unpacking step by step (as if we were LLMs):

- On the manifold being “high-dimensional” (e.g., 2999): I got your intuition; the set of valid linguistic sequences is tiny relative to the space of all possible strings, but still enormously rich and varied. So the valid set doesn’t fill the whole space, but it also can’t live in a low-dimensional manifold like 20D. But I'm also not so sure about that: how many ways you have to give an accurate response? Hard to argue than many more than one. Hard to argue even one of them it's completly correct. _There must be some sort of "clustering"_.

- On domain-specific manifolds and semantic transitions: 100% agree with your idea that different domains induce distinct geometric structures in embedding space, and even that the idea of a "simple manifold", seems to optimistic. But what about "regions" with common (geometric / topological) properties? Eg: Physics should? form a dense structured region, and I guess there common patterns between the implicit structure of it's subspace, and the maths' one for example. The semantic trajectories inside each domain will follow specific rules, but patterns must exists, and also should be transitional zones or “bridges” between them. I relate the emergen abilities into LLMs with this (what are LLMs but transformers of vectorial representations, taken by "views / parts / projections" (e.g: multi-attention heads).

What if we hypothetize about chart atlases; multiple local coordinate systems with smooth transition maps? Maybe a patchwork of overlapping manifolds, each shaped by domain-specific usage, linked by pathways of analogy or shared vocabulary.. Even if this is the case (we only guessing), the problem it's that the computational costs, and interpretations are neither trivial.

- On GloVe and the projection fallacy: I take your point, you can always "cherry-pick" the most good loonking examples to tell your story haha

- On symplectic spaces: I don’t know enough about symplectic geometry :( Only think: you got me thinking about hyperbolic spaces where volume grows exponentially; anti-intuitive from an euclidean point of view

- “maybe the flat embedding space doesn’t devote volume to gibberish because it was never trained to model gibberish.”

I initially thought of this as a kind of "contraction", but that term might be misleading - but thinking about it I prefer the idea of density redistribution. Like a fluid adapting to an invisible container --> Maybe it's like a manifold emergence through optimization pressure indirectly sculpted by the model’s training dynamics.

- Wheeler Superspace: Again, I cannot follow you :( I guess you're pointing that the semantical relationships could be formulated as discrete.. BUT, as a non-physicist, I honestly can’t tell the (any?) difference between being modeled as discrete vs being discrete. (xD)

Thank for the deep response, Paul! Its a pleasure having this conversation with you.

Love your project. And the mentiond metrics ΔS (semantic tension), λ_observe (viewpoint drift), and E_resonance (contextual energy) totally align with my mental model.

Also loved your phrase: "kind of like how meaning resists compression when you flatten it too early." That's a key point. Implicit in that is the assumption that flattening will eventually happen — and I agree (assumed for efficience in last term): Just like how we create flat maps of the Earth sometimes a flat projection is useful _as long as_ we understand the properties of the deformation and the underlying surface. If we don't, EEUU might seem a lot bigger than it's "represented", in our euclidean view.

That interpretation really clicks for me.

I'm currently exploring *geodesic-aware training methods for IR*, based on contrastive / triplet loss. Take a look if you're curious: [https://github.com/Intrinsical-AI/geometric-aware-retrieval-...](https://github.com/Intrinsical-AI/geometric-aware-retrieval-...)

Still recent and (very) experimental — but already done some initial tests. The goal is to construct a differentiable graph over the embedding space and propagate training signals end-to-end. Lots of work ahead, but preliminary results are promising (and adjusting things with o3 feedback ofc).

Are you open to collaborating or jamming on this? Would love to sync up — are you on Discord or Slack?

Thanks again for pushing this forward.

Hey man! Thanks a lot for your support! Might sound like just common words, but honestly — knowing this helped or inspired someone really motivates me. Makes me feel a bit less like a madman hahaha.

About your question — I think I get your point. Here’s how I understand it:

In the hypothetical latent manifold, we might want to measure two different things:

1. Distance between two points: This wouldn’t be the usual Euclidean distance, because the space is curved. Like how the shortest path between two cities on Earth isn't a straight line, but an arc on the globe. That’s where geodesics come in — they’re the shortest paths constrained by the shape of the manifold.

2. Similarity between two vectors ("parallel transport") Instead of asking where the vectors point in ambient space (like cosine similarity in R^n), we (should) care about how their directions compare _on the surface itself_. So ideally, we’d compare them along the geodesic — parallel transport — to properly align their frames of reference before measuring any angle or similarity.

That’s the intuition I’m working with, anyway. Let me know what you think, and thanks again for your comment!

That's fascinating! — but I don't fully agree with the framing.

Using G(t) in the context of embeddings seems problematic, specially given the probabilitistic nature.

Example - Take a sentence with a typo but semantically clear and correct (let's suppose): "The justice sistem is corrupt."

G(t) = 0, right? But semantically, it's close to G(t) → 1.

Instead of focusing on exact validity —which for me seems too rigid for something as ambiguous and context-dependent as language— what if we focused on _approximate semantic trajectories_?

You wrote:

"If you have two points in the embedding space which represent well-formed sequences and draw a line that interpolates between them you'd think that there would have to be points in between that correspond to ill-formed sequences."

In my view, it's actually the opposite:

If the embedding model captures meaningful structure, and you account for geometric properties like curvature, local density, and geodesics — then the path between those two points should ideally trace semantically valid (even "optimal" - if that exists) reasoning.

The problem isn't that interpolation fails — it's that we're interpolating linearly in a space that likely isn't flat!

Thanks for your comment. Lmk what you think :)