The Impossible Job: Inside Facebook’s Struggle to Moderate Two Billion People

https://motherboard.vice.com/amp/en_us/article/xwk9zd/how-facebook-content-moderation-works

by dsr12 • 8 years ago

67 52 8 years ago

motherboard.vice.com

arcanus• 8 years ago

Not impossible, impossible for hoards of humans.

This is going to be a major use-case for AI, which can scale to billions of people.

CharlesW• 8 years ago

The real solution is probably hybrid (i.e. AI-assisted human moderation).

I've assisted with moderation for a bunch of communities over the years, and IMO the problem isn't that there's not enough humans — for every bad actor in a community there are many good ones — but that Facebook doesn't provide good actors and moderators with anything beyond the most basic tools imaginable. Had Facebook invested even 1% of their resources into this, I don't believe that Facebook would be the raging culture infection that it currently is.

chasd00• 8 years ago

It would be awesome if AI could handle the worst cases. From what i've read moderating the worst content taxes humans emotionally pretty heavily. Many have ended up in therapy because of it.

SllX3• 8 years ago

So I've never managed a subreddit or a Facebook page/group. I once managed a forum where my options were to Delete and Edit comments, but I only used Delete and would message the user why, it was a small forum though so if it were say, 3x or 5x larger I'm not entirely sure I could have moderated the entire thing by myself.

I'm explaining my ignorance on this subject matter because I want to ask you, and you seem like someone who might know, how do Reddit's and Facebook's tools compare? From my point of view, Reddit is pretty well moderated, each sub having its own rules that at least in the subs I frequent, are actually fairly well moderated. They have tools like automoderator and I've heard of subs I don't frequent preemptively banning users for having too much karma in subs they dislike. Probably overkill, but given the types of communities they are it seems to work for their purposes regardless of whether I agree with them or not.

If I look at a comment thread on Facebook, there doesn't really seem to be any kind of conversation happening, just "conversation". There's some basic threading now but even with that a Facebook "conversation" seems to just be one long string of human consciousness.

Now there is a difference in scale, Facebook has over a quarter of the Earth's population on its platform. That said, there is also a difference in scale between Facebook's resources and Reddit's, so is there some reason they choose not to develop new and better mod tools? Or are their mod tools cutting edge already and the percentage of humanity on their platform is just so large that even with cutting edge mod tools it just isn't enough and throwing more money at it doesn't make it any less hard?

notriddle2• 8 years ago

Reddit gives the masses downvotes, but those don't get used for moderation, they get used to make the echo chamber more impenetrable.

pixl972• 8 years ago

Pretty sure it's impossible.

You're assuming there is a global maxima in line with the local maxima. I personally don't believe its the case. Moderation practices across one group will not make sense to another, but when you try to moderate each group individually, said groups will point out inconsistencies between moderation in each group.

For example a user with 2 accounts

Group A is moderated for doing x

Group B is not moderated for doing x

Therefor the user believes the system is unfair, and will point that out to others spreading the idea of unfairness in the moderation system.

smt88• 8 years ago

Facebook didn't need much moderation until it started allowing strangers to enrage each other through public comments.

My experience on FB is that 100% of the value of the platform comes from parties I choose to engage with (friends, local businesses, groups), and almost all of that value is wiped out by having to see comments from trolls.

pasbesoin2• 8 years ago

Remember when "friends of friends" used to be a (readily accessible) setting?

Now, it's "friends" or "public".

Reflect on that, for a minute.

"Friends of friends" isn't/wasn't perfect, but it had a moderating effect.

A lot of this, is Facebook's own fault. Because greed.

dlhavema• 8 years ago

I'm guessing it was a lot harder for stuff to go viral when it was only "friends of friends" vs "public"...

but i remember reading somewhere they throttle delivery of messages to friends until they can gauge feeedback.. not sure if that was confirmed or a rumor.

javagram• 8 years ago

Didn’t privacy researchers show that “friends of friends” was misleading and effectively public because of how easy it is to become a friend of a friend ?

bufferoverflow3• 8 years ago

What do you mean by "trolls"? Someone who disagrees with you? Like half the country is "trolls"?

And what stops you from simply unfriending these trolls, if they are so toxic?

I'm not buying that narrative at all. What I think actually happened is that groups of people always had irreconcilable differences, but these groups were mostly physically separated, because people tend to form bubbles of like-minded people. But then FB came, and everyone's opinions became visible. Half of your coworkers that you went out drinking with turned out to be Trump supporters or antivaxxers, and that just makes you and them angry.

The solution is to stop demonizing the other side, and start having rational conversations. But good luck with that.

P.s. Calling them "trolls" will only drive them further away from your politics, you're helping Trump to get reelected.

emodendroket• 8 years ago

Anything short of enthusiastically agreeing with Trump someone will accuse you of being the reason Trump was elected. In one ear and out the other.

SllX3• 8 years ago

I'm not going to disagree with you, because what you say is spot on.

What I am going to do is point out that hey, maybe there are people out there that troll just to troll or troll for a particular cause, and there's a lot of them. When you have a few different sides that are generally isolated from each other and already struggling to have a conversation and throw trolls and paid trolls (astroturfing) onto the same platform, you don't have a platform that is conducive to these various sides having any kind of conversation because there are groups of people that are actively working to prevent this. It doesn't even take a very strong push to keep them apart either, they were already inclined to isolate themselves in meatspace, why would cyberspace be any different?

Facebook has more than a quarter of the Earth's population on their platform. Undoubtedly this is a hard problem to solve. An authoritarian approach is probably what they are going to end up having to go with because they're not a country or a religion, they're a bloody online advertising company.

komali22• 8 years ago

I completely disagree. I grew up as the outsider in the south - liberal ideology, atheist, etc. Through the Clinton, Bush, and Obama eras it was No Big Deal. I'd have spats with friends and neighbors, and they'd end friendly. In South Carolina I went from being called the "n-lover whiteboy" to convincing my white friends that my black friends are just as cool, and have good BBQs. We'd all disagree with eachother, but it'd be respectful. Online and off. This is when Facebook/Myspace is just coming out, and "normies" are learning about interacting online, and about how easy it is to forget about the face on the other side of the screen. Even there it wouldn't flare out too horribly.

Something's different, these last few years. Sometime around the tail end of the Obama era is when I began noticing it, that it wasn't a "political disagreement," it was a fundamental inability to reconcile values. It was "freedom of speech vs political correctness," it was "nazis vs antifa," it was "black lives matter vs all lives matter." People getting mass-defriended on facebook, me being mocked by my southern friends for my liberal values (that are the same as they've always been). Things have gotten far more divisive, and if I had to make a guess, I blame the political organizations paying trolls to disseminate divisive fake information across the internet, and prominent politicians engaging in disgusting divisive language (Duterte, Le Pen, Trump, Netanyahu, Erdoğan, ISIS high-production-videos).

donmcronald2• 8 years ago

Part of the problem is Facebook doesn’t want anyone to have a negative experience because those people disconnect, at least temporarily.

I can go on Facebook, praise Hitler, and get likes. I think that skews peoples’ perception of what’s acceptable and what’s not. Maybe Facebook needs a way to give negative feedback.

It’s such a hard problem though. The quantity over quality friend groups everyone has now makes it difficult to trust others. When I was growing up, if one of my good friends disagreed with me it was a good indication I needed to reevaluate my opinion. I think that’s been lost.

eksemplar• 8 years ago

I’m not sure that is really facebooks problem. If I saw you praising Hitler, I’d unfriend you, so it’s not like there isn’t social consequences on Facebook.

I think the real issue is the users. I mean, Facebook makes it easier to find people who are like you and believe the things you do, but Facebook didn’t make you believe the earth was flat.

The fact that people believe the earth is flat shouldn’t be blamed on social media, because it’s a much bigger problem than that. Those people have managed to grow up wrong, with terrible education and horrible life choices, and maybe they find each other in groups on Facebook but they sure weren’t made this way by the internet alone.

donmcronald2• 8 years ago

I think Facebook has some responsibility because they're such a large platform. The problem is constant positive reinforcement of everything, even things that are unacceptable. Part of the reason those people grow up "wrong" is because they're growing up on social media where the feedback they get is unnaturally skewed to agreed with them.

I don't think it's intentional, but I think it's going to be impossible for Facebook to solve with watered down feedback like the "empathy" button. They need a way for people to express disagreement, but it's going to be difficult to do that without reducing engagement, so it probably won't happen.

crunchlibrarian• 8 years ago

Nobody believed all this nonsense before Facebook/Youtube, dumb people had the good sense to defer to others. This is absolutely new and caused by these tech conglomerates, it is not inherent in people.

We optimized the hell out of user engagement, and created these closed loops where you just swim in nonsense for years. The marketing teams loves it, shareholders love it. Nothing foreign or unpleasant penetrates the engagement loop. Keep growing those numbers. What's the content? Hmm? who cares?

coldcode• 8 years ago

If Facebook's revenue was not tied to keeping people accessing the service enough to see ads to prop up the stock price, then they would have more options. Maybe they should consider charging instead of serving ads. But it would likely reduce their overall revenue.

wpietri• 8 years ago

Impossible only because they have underpriced the service. The notion that only good things will happen when you create a context and let humans interact is at best naive. In this case, I think it's willfully blind. I was talking recently with people who set up the first web-based chat; abuse was almost instant and they what we'd now call a doxxing within the first year. And they were hardly the first people to see abuse happen online.

In the real world, if you want to have something as anodyne as a parade, you have to provide for sufficient security to keep things safe. There's no reason that companies pocketing billions in profits should be exempt from this principle.

j-c-hewitt2• 8 years ago

The risk with this is that eventually it will erode Facebook's legal argument that it isn't responsible for the content that its users post to the network or share through Facebook Groups.

As it is there are tens of thousands or even more Facebook groups dedicated to some kind of either illegal, rights-infringing, or grey market activity in the US alone. The more that Facebook holds itself responsible for moderating and controlling that content, the less weight that Facebook's arguments that it is just a dumb pipe will hold. This has been going under-remarked relating to this cluster of issues.

Strongly doubt that FB or any similar company will successfully navigate this issue in the long run.

pitaj• 8 years ago

No company should be liable for the content users post, regardless of moderation policies.

telltruth• 8 years ago

They definitely screwed up how they handled the crises. Zuck & Sandberg's apology tour has made them responsible for any future liability or slip ups which are inevitable given the size of operation.

There were 4 separate issues

1. People posting fake news

2. Recommandation serving fake news due to similarity

3. Apps stealing data

4. Advertisers able to target vulnerable audience

They should have kept these issue absolutely separate and taking responsibility for all but #1. They should have stayed very firm on not accepting blame for what people decide to share on their platform. Instead they ended up accepting blame for everything and now they are screwed.

j-c-hewitt2• 8 years ago

Right. This is a giant multi-billion dollar target that is increasingly portrayed as a machine for the mass production of liability. People seem to be under the impression that internet companies putting themselves under the cloak of free speech came purely from 'idealism.'

The real reason is to avoid being Napster'd. How can FB turn around to a copyright holder and say "Ha ha, this content is totally not something we take responsibility for -- you should take it up with the user" if it is actively policing content on the platform with an army of staffers and actively shaping what people can and cannot write on the platform? That looks a lot more like editorial behavior than 'moderation'.

It's different when it's just automated systems that are dinging people for potentially posting copyrighted material or a snuff film. The system doesn't exercise human judgment on an active, day-to-day basis. To the extent that services like FB took a hands-off approach had much more to do with avoiding liability than any iron moral commitment to allow people to share flat earth and lizardman related content.

EGreg• 8 years ago

Why do social networks have to be so big and centralized?

Because we don’t have good open source software like Wordpress for blogs that people can install and manage their own networks on their own servers. And why? Because a social network is far more complex than a blog, including contacts integration, permissions, realtime notifications and so on.

https://m.youtube.com/watch?v=pZ1O_gmPneI

superkuh• 8 years ago

Notabug.io is doing good work in this area. It's a reddit-alike with a distributed architecture. It has federated nodes hosting the website but also the in-browser client distributes posts p2p between nodes in the federation. Additionally, voting is proof of work based.

It's the early days now and the lens/filter system for individual users hasn't rolled out yet but the idea and implementation are far ahead of anything else I know about.

pixl972• 8 years ago

Because of the 'network effect', networks are more powerful when they are larger.

fertilityrspctr• 8 years ago

The thing that makes it hard is differing cultural standards between parts of the world. You either acknowledge them and push moderation decisions down as far as possible to local authorities, users, and group moderators or (what they seem to be doing) build an authoritarian, culturally-imperialistic global AI censorship algorithm that makes people mad and (hopefully) causes them to leave.

jimnotgym• 8 years ago

The Impossible Job

Leads me to two possible conclusions

1) Facebook gets a free pass to ignore the law because 2bn members requires x million moderaters and that doesn't make economic sense, is hard etc.

2) Facebook has to abide by the law, and since it doesn't make economic sense...Facebook closes.

notriddle2• 8 years ago

The question is whether Facebook is a communications company or a media company. Are they like GMail, where they aren't responsible for everything that goes through their pipes, or are they like Vice, where they clearly are?

Facebook obviously wants the publicity of a media company and the legal protections of a communications company. Which is obviously stupid, but that's the weird situation we're in.

WorldMaker• 8 years ago

Facebook also obviously thinks of themselves as a tech company first and foremost, so the question of moderating and scaling moderating is considered almost solely from the standpoint of it being a technical problem.

The problem is a social and human one. Newspapers have traditionally handled it with editors and people and human policies. Facebook isn't inclined to explore editors and people and human policies, not because they don't scale or we don't know how to scale them (Wikipedia has scaled relatively okay, for what that is worth as an example; Newspapers have used editors and researchers for a long time), but that it's not an interesting technical problem to scale people.

Facebook even had people hired in some of these moderation roles for a while, but it was sexier for them to use essentially blackbox algorithms that turned out were easy to game by malevolent entities. Now they only seem to question "fixing" the blackboxes rather than finding solutions that use smart or trained people.

antidesitter• 8 years ago

Which law are you referring to?

kyle_v• 8 years ago

dont shoot the messenger

twblalock• 8 years ago

The impossibility is not just a problem of scale.

Even if Facebook does everything right, and moderates every single thing they aim to moderate, they will still be blamed by conservatives and/or the alt-right if they censor too much, and by liberals if they censor too little. There is no type of moderation that will make all sides happy.

Verdex_3• 8 years ago

Wait, so the conservatives want less censorship and the liberals want more censorship?

Is this one of those mirror universe things?

firemancoder• 8 years ago

To be fair it has flipped considerably, if you consider the extremists in each group.

In the 2000s the extremist conservatives wanted to censor everything (mostly due to religion) and liberals wanted freedom of expression.

Now it seems that extreme liberals want to censor "hate speech" which is absolutely valid, however the lines are moving or getting blurry to the point where they want to censor conservative ideas altogether, and conservatives want "freedom of expression".

I've seen this shift and I'm sure many others have as well.

siruncledrew• 8 years ago

Think about the scale of 2.2B users. Imagine how difficult it would be to moderate just America. Now imagine handling 6.8 x America. Or 1.6 x China or India. That is a massive undertaking for any centralized entity. Probably bigger than many countries national security efforts to feasibly implement. Is it really worth it to Facebook when their primary concern as a corporation is their bottom line?

dralley• 8 years ago

In which case the argument becomes

"If a platform/service provided by a corporation becomes large enough to credibly threaten the fabric of society and/or government, and said corporation won't tamp down on said problems, should they be allowed to exist?"

pimmen• 8 years ago

The smartest engineers in the world have been figuring out how to make people click on ads. If they have to figure out how to quell genocidal movements I think that's an improvement.

tvh• 8 years ago

For example, its revenge porn policy and recently created software tool—which asks people to preemptively upload nude photos of themselves so that they can be used to block anyone from uploading that photo in the future—was widely mocked.

This is hilarious. I understand the logic, but to believe that users will willingly submit nudes as a preemptive measure seems to widely underestimate the ramification of the user's decision-making process, and the reliability of Facebook when it comes to user privacy protection. Hey, let me upload my nudes to a Facebook software, they've been handling privacy really well so far, said no one ever.

The Impossible Job: Inside Facebook’s Struggle to Moderate Two Billion People

Related Stories