As a site owner you should probably have both. FB connect isn't "free": implementing integration probably takes more time than installing authentication plugin for Rails (and doesn't Django have built-in user management?)
But I think you should have Facebookless login if you care about early adopters: too many of them are indeed "outliers" you've identified above. I also think you should have facebookless option simply because it's good for the Internet: I wouldn't want to see so much of it being controlled by a handful of large corps.