I agree with you in general, I like my working remote, but it's the other guys working remote that I don't like. ;-)
Managing a distributed workforce, definitely has challenges. From what I have seen, onboarding new employees can be particularly awkward. It may be different in hip, new startups, but in my experience with less fashionable, old school organizations, it makes accountability, knowledge-sharing and organizational-political navigation more challenging.