Most people would think an entire town deciding to exclude minorities is morally repugnant
What? I'm not saying I don't. It's just irrelevant to the issue.
shouldn't be allowed.
That's like saying that "these people over here cannot be allowed to decide over their own neighborhood, because they might decide things that I find repugnant". Well, who are you going to trust with the power?