I agree with your take but interestingly it seems to be both an argument for, and against, calling it a spectrum. Humans are so good at adapting/conforming/masking, and we adjust ourselves toward a common accepted way of behaving, which further confounds the difficulty in understanding the problem.
I think the "spectrum" analogy has reduced stigma overall, especially toward people with poor social skills. But it isn't always helpful.