This is classic libertarian thinking, and I used to subscribe to it, at least to some extent. The more you think about it, though, the more it doesn't hold up.
It's just undeniably true that individual freedoms exist only to the extent that society allows them to. To think that you can opt out of society is fantasy.
It's in everybody's best interest to make sure society works well for everyone, and that kind of thinking pushes you back to the political center.