Ask HN: color sorting algorithm?

Imagine a pallete with N colors. You need to display them in some order that suits human eye best.

I seem to have problems with definition of what I want. Ideally, I'd like to see some gradients, e.g.: blue gradient from lightest to darkest, yellow gradient from lightest to darkest, etc.

The problem is that sorting by [H,S,V/L] or [H,V/L,S] doesn't seem to work for non-ideal palletes. The biggest problems are with shades of gray or something similar to shade of gray (when [R,G,B] components are almost equal) that appear in between gradients. Also, it should allow some deviation on hue (which would be invisible for human eye but surely visible for simple sorting algorithm), but my simple tries of generating a keys consisting of weighed sum of components failed.

What I also tried: building a matrix of RGB distance and then trying to sort colors keeping distance between two neighbours minimal; using combination of HSL and YUV components; sorting by [R,G,B], [H,S,L] keys and all the possible transpositions of keys.

What I haven't tried yet: hilbert curves, from what I see on pictures it won't help.

Suggestions, anyone?