atkindel

Your thinking here is pretty close to right, but it sounds like you may be a bit confused about the difference between "standard deviation" and "standard error". They're closely related, but it's very important to keep them distinct.

In your example, you have a random variable X that comes from some distribution parameterized by its mean m and its variance s^2. We can write this random variable as x ~ ?(m=1, s^2=1). Continuing your example, suppose we sample a sequence of n draws of this random variable X = [x1, x2, ... xn] and then take their sample mean f(X) = 1/n * sum(X). The value of the sample mean will vary depending on the sample we drew, so we want to know the properties of the sampling distribution of f(X), particularly as we increase n. The standard error is the standard deviation of the sampling distribution of f(X).

Your intuition here is right on; in fact, the reason why we like embeddings in the first place is that they address this problem! When we have some sparse high-dimensional vector representation of a bunch of content, we often find that we are devoting an entire dimension to very little information, so it would be convenient if we could approximate each of those vectors in fewer, denser dimensions without losing too much information. If all that we are interested in is how much the vectors align (i.e. correlate) with one another, embedding methods are one of the best ways to construct those lower-dimensional vectors, in the sense that we are guaranteed to find a solution up to whatever amount of approximation error we are willing to tolerate.

An added benefit is that if we're smart about how we construct the lower-dimensional space, we often find that it generalizes to unseen data better than if we'd used the high-dimensional representation, because a lot of the variation we're throwing away is specific to how we constructed the input data.

A really cool thing about embeddings is that we've known how to do this for a really long time --- for example, a landmark paper [1] on this exact problem was published in the 1930s!

[1] Eckart, C. and G. Young (1936), "The approximation of one matrix by another of lower rank." Psychometrika 1, p. 211-218. https://link.springer.com/article/10.1007/BF02288367