Good on you, but this doesn't prove anything imo. People are almost always positively biased toward attractive faces due to the halo effect. Biologically/psychologically this is a result of one's facial bone structure, skin, and bodyfat levels.
So being shabbily dressed and having stubble is mostly irrelevant here. Maybe you were just much better looking than the other (poor) bloke.