XCheck at Meta: Why it exists and how it works

https://blog.nindalf.com/posts/xcheck/

by nindalf • 4 years ago

337 134 4 years ago

blog.nindalf.com

otterley• 4 years ago

a system that exists and mostly works is preferred to a hypothetical perfect system that is never built.

At any sufficiently large scale - especially with a product you're not paying for - this is the best you're likely to get. Once you fully understand and embrace this, your stress levels about imperfection tend to go way down.

itake• 4 years ago

I wish they would explain what xcheck is and how it works before the talk about the problems with it. This reads like a blog post to other Integrity FB engineers that already know that system.

nindalf2• 4 years ago

It’s explained later on but I could be clearer.

XCheck is a system that supports tagging specific accounts with tags that exempt that account from certain integrity enforcement. In the post I give an example of a tag that used to exempt accounts from the fake account checkpoint. Applying these tags is usually a manual process and usually carefully vetted.

webmobdev2• 4 years ago

You yourself claim that you are only aware of Xcheck "features" since 2021,

   As of June 2021 no XCheck tag I knew of allowed a user to take down content at will.

Are you also claiming that there have been no changes, and no new features, added to XCheck since June 2021? Are you aware of the new social media regulations recently passed by the current Modi government of India? (To be credible, you need to honestly answer these questions and address it in your article too, as otherwise you are basing your assumptions on incomplete information, thereby misleading yourself and others.)

To clarify, one of the requirement of this regulation is:

   Under the new Intermediaries Rules, intermediaries must *complete* the takedown process under Section 79(3) of the IT Act, within 36 hours.

Source: https://internetfreedom.in/intermediaries-rules-2021/ (a more brief summary of the regulation is here - https://techwireasia.com/2021/06/indias-new-social-media-rul...)

Very simply put, large social media platforms are legally required to respond and delete content when requested by government representatives.

Under such a scenario, can you confidently say that XCheck hasn't been extended with new features, like a tag that allows a user to immediately flag posts that must be compulsorily taken down when the particular user reports it? It seems a very obvious extension when you consider the political demands that are placed on Meta from around the world, and in India in particular.

webmobdev2• 4 years ago

Yeah, it would be easier to understand your article if you start it with an introduction to what XCheck is what problem it is trying to solve.

localhost• 4 years ago

This is an excellent piece which really does a good job at explaining the nuance that exists in this challenging problem domain. Thank you for writing this. It's a great piece to point to whenever the next scandal gets "exposed" by click-seeking folks on the internet.

uoaei• 4 years ago

Are you saying that this article excuses data breaches?

ironrabbit• 4 years ago

The "scandal" that prompted this article is referenced with sources under the heading "Why XCheck is in the news". Neither the article nor the comment you're replying to says anything about data breaches.

sneak2• 4 years ago

It's good that it keeps getting exposed that a single organization without accountability decides what people are or are not allowed to publish on a platform to the people who explicitly subscribed to their publications.

The fact that moderation is a hard problem is orthogonal. The fact remains that there are different rules for different people, both for flagging and for content. The fact that there is more nuance than "two sets of rules" doesn't really change the fundamental problem that private companies now get to arbitrarily police publication by normal people.

I have the distinct impression that in 100 years the default mode is going to be "if it's legal you can post it, if it's not you can't, and if you have a problem with the boundary, take it up with the lawmaking authority and not us". We will look back at this time at the emergence of any-to-any publishing like the dawn of radio or TV or newspaper where a few rich guys got to decide what could or could not be published.

CursedUrn• 4 years ago

What makes you think something as unique and hard-fought as the First Amendment will be applied to the Internet when all trends point in the other direction, both on and offline?

wbl2• 4 years ago

https://www.getty.edu/art/collection/object/109FW5 (NSFW) is a beautiful picture, taken by a major 20th century photographer. It's also depicting a sex act that most people would never even consider.

Do you really want pictures like this everywhere on Instagram? So now we have a no sex acts rule. But then an art museum is hosting an exhibition of art, and that art is depicting sex. Maybe even pedophilic sex, like a vase in the Met that has an old man stroking a young teenagers hairless genitals on it. That's very likely child pornography, except that the law can't ban images of artistic value, and the image of the vase has a lot of value, and I doubt anyone would find it erotic. Should the Met not be allowed to use a picture of a bunch of vases to promote its exhibitions?

There's lots of reasons why reasonable seeming results are not possible from simple rules, and why it's legal, let's have it is not a great rule. Having a moderation system that addresses e.g. art studios and museums differently from other people is quite sensible.

yardstick2• 4 years ago

That works well for about two weeks until users figure out how to exploit this. They form groups that agree to coordinate to report content. Now reports become lower signal than before, but still somewhat useful. We use the reports, but try to limit exploitation.

What about human employees reviewing users reports, especially randomly picked samples from those that resulted in successful takedowns. If the content was found to be sound under the rules, penalise the reporters. Suspend their ability to make reports (probably in the form of silently ignoring them).

Ie some human moderation should still be done at this scale, but using a combo of random sampling and system-flagged-suspicious.

xani_2• 4 years ago

I remember Dota2 (video game) did this, people fake reporting had less reports and they counted for less, while people that were consistently reported players that got punished got more reporting power. Only one report per match "counted" too so a group of player couldn't gang on reports on one player.

Of course that could be abused as well,but you'd have to make a group of people that first got the good rating then reported same people and that would be significantly harder

qualudeheart• 4 years ago

So that’s why I was never banned from Dota even though I sucked at it.

WorkerBee28474• 4 years ago

Well that just sounds like Bayes's theorum with extra steps

coredog64• 4 years ago

This is Slashdot's meta-moderation system from 20(?) years ago? Logged in users would randomly be selected to check moderation, and accounts that abused their mod points were flagged. Granted, you probably couldn't do that today as you could see meta-moderation brigades out in the wild. Employees should be better, although there's still the chance that an employee might use their position to punish opinions they disagree with.

MBCook• 4 years ago

I remember that. After a while I was “randomly” chosen to meta-mod all the time. But I know others were almost never chosen.

Clearly the actions were reviewed so people’s percentage change could be adjusted based on seeming fairness.

yardstick2• 4 years ago

Yeah I remember slashdot days. Having employees do the meta moderation is key imo, and sounds like would need a second level of meta moderation to review samples of the employees actions.

I’m also very pro a pay for final review, where eg you pay $50-100 or something around that level of inconvenience to have a multi-employee review (each employee reviewing it independently, unknownst to each other, and taking the majority decision).

timzaman• 4 years ago

How is it people are allowed to blog about company internal implementation details and numbers barely a year after they leave?

tyingq• 4 years ago

Seems unsurprising to me. If you, as a company, want ironclad confidentiality from people after they leave, that wouldn't be free or automatic. None of what was shared seems to qualify as a trade secret, which would have some protection.

extr3• 4 years ago

I've never worked somewhere that didn't have me sign some kind of broad NDA-esque clause saying I wouldn't share proprietary or non-public info for a certain period after release (imo, completely reasonably). This blog post is literally nothing BUT non-public proprietary info. Like the parent I'm really surprised the blog author feels comfortable sharing all this. Even if it ended up being technically legal, I have to imagine this kind of thing is extremely frowned upon.

Kiro2• 4 years ago

Why wouldn't they?

extr3• 4 years ago

Have you ever read your employment contracts? I've never worked somewhere that would allow something like this to be shared so soon after departure.

photochemsyn• 4 years ago

Not to be unduly cynical, but any Public Relations 101 course will introduce the importance of third-party messaging, because statements coming from outside a company directly involved in some (apparent) scandal or other will always carry more weight with the public than an official corporate statement. Additionally, this kind of distance removes certain concerns about legal liability if more insider information is leaked later.

[edit] I have no knowledge or opinion on this story, never read the Wire, and never use Meta or any of its products. It does, however, seem rather clear that governments and corporations concerned with controlling message and image do view social media platforms as the most important battleground in the information wars these days.

numair2• 4 years ago

I think that the key takeaway from the entire Wire vs Meta fiasco is that there is a lot of absolutely weird Spy vs. Spy behavior going in the Indian political and media industries.

I was initially extremely sympathetic to the story presented by The Wire, because it's quite believable that Meta/FB would go to extreme lengths to try to distance themselves from such a situation, but the facts ... just ... don't add up. As Alex Stamos has noted, there is little debate regarding collusion between Meta's "government relations" people and policy groups, so that's not really much of a scandal -- it's not like Meta will deny that Modi was treated like a divine being when he visited their campus, so the idea that his people can easily call in a favor to crush a social media post won't really surprise anyone.

The bulletproof evidence with DKIM authentication and a video of a logged-in admin instance doesn't look so bulletproof after all, based on the credible reports from those who know how Meta's admin tooling actually looks and functions, and those with other DKIM authenticated emails from the fb.com domain.

So, the question is, what's the agenda here? Why would someone go through all of this effort, to create a scandal out of something that is not very far from the truth? What's the point of this entire thing? Maybe this is like Nick Denton and Gawker losing their entire business over the stupidest sex tape story ever; or maybe this is part of something else that requires domain knowledge regarding Indian politics and media to understand (do page views monetize so well that this mini-scandal is going to be super-profitable to The Wire? Highly doubtful, right? What could they possibly get out of this?).

The whole thing is just really weird. It's also a major distraction from the very real problems that Meta doesn't even try to hide. Meta's relationship with the governments in various countries -- including the United States -- is way too close for comfort, and absolutely toxic on multiple levels. If this story turns out to be fake news, it'll do a lot to help the company deny, deflect and discredit the next real scandal. I think this is what the wacky conspiracy theorists call "4D chess," but I don't think that's what's happening here.

The true story behind all of this is bound to be very strange, and very stupid.

DisjointedHunt• 4 years ago

I'd speculate this is part of an "Opposition research" move by a political party that might benefit.

The extreme lengths that these people seem to have gone to is shocking. Whatever it is, FB has the evidence on their servers as Alex Stamos points out since they created a Workspace instance.

nindalf2• 4 years ago

Based on my conversation with the editor of the Wire, he seems sincere in his belief in this source. He is a respected journalist with decades of experience so he wouldn’t trash that just for a few clicks.

My read is that he wants this story to be true so much that he’s ignoring evidence to the contrary.

As for why the source is doing this, I couldn’t say.

numair2• 4 years ago

Thanks, this is very helpful. It would be wild if this was a very carefully crafted campaign meant to bait The Wire into blowing itself up over an almost-true but totally fake news story. Now I really want to know where this all ends up.

sa1• 4 years ago

If the story is false, it's not just the source, at least the tech person making the videos at Wire is in on it.

ummonk3• 4 years ago

Fwiw Devesh Kumar has to be in on the fabrication given that he provided an impossible story as to why the email screenshots they initially posted showed 2021 (impossible because the day of the week didn’t match the displayed date, indicating the screenshot was fabricated).

It also takes a special level of suspension of disbelief to think that Andy Stone somehow writes emails in Indian English and that XCheck gives privileges to take down content from other users.

desi_ninja• 4 years ago

The wire is notorious for propoganda hit pieces in Indian political landscape. Based kn conversation is not enough merit when you see kind of reports the wire dishes out.

moomin• 4 years ago

I’m nitpicking, but the reason Gawker went bust was because someone went out of their way to make sure they did. Sure the tape was the proximate cause, but if it hadn’t been that, it’d have been something else.

nocoiner3• 4 years ago

Yup. Gawker was victim of a targeted assassination. Sure, their editor gave a truly disastrous deposition which probably directly caused them to lose that particular case more than any other single factor, but if it hadn’t been that case, it would have been the next fact pattern where a well funded opposition would have even a slim chance at nailing them to the wall.

There’s plenty Gawker did that I didn’t like or didn’t care for, but as a proponent of free speech, I see nothing positive about them losing that shit.

sangnoir• 4 years ago

Peter Thiel was going around with an open checkbook, actively looking for anyone who may have a viable case against Gawker. If it hadn't been Hulk Hogan's sex tape case, it would have been unlawful termination, or some other grievance that involved Gawker Media in some way.

Edit: lol, no wonder parent & gp were circumspect on names - I'm getting downvoted :'-D Anyhow, here are my sources:

https://www.theatlantic.com/business/archive/2018/02/hogan-t...

https://www.theatlantic.com/technology/archive/2016/05/the-b...

cavisne• 4 years ago

The first thing that comes in to mind is the Bezos leaked nudes.

At the time there was a story about the Saudi's being involved and some link to Trump.

Years later it came out that Bezos's investigator knew pretty early on that it was likely the brother of Bezos's girlfriend, however the National Enquirer was already in trouble for allegedly acting as a foreign agent for the Saudi's. By seeding the fake Saudi story they were able to apply huge pressure on the Enquirer.

Maybe something similar is happening here, Meta is under legal pressure somewhere on this topic. It doesn't have to be true, it just has to annoy the wrong politician/judge.

wiml• 4 years ago

This is a mildly interesting post only because of how it comes up to, but avoids ever talking about, the central flaw in the system it describes — they're conflating several things without ever really acknowledging that they're different. Theres "integrity", which is a Meta-internal term; verification of ID, which is a far deeper rabbit hole than they're going to go into; abuse; and undesirable behavior. They're all different things.

remus• 4 years ago

They're all different things.

Different but related, I would say. For example abuse of a service and undesirable behaviour on a service could be thought of as different points on a spectrum.

SilverBirch2• 4 years ago

It makes sense from a technical perspective why you'd build a system like XCheck. The problem is that once you build XCheck you actually need to say to your C-suite and PR department "Actually no, you can't say we treat everyone equally, we're manually going in and exempting certain people from scrutiny, and that's entirely within our discretion."

The problem isn't technical, it's legal - you can't build a system that operates one way and then publicly misrepresent it.

xani_2• 4 years ago

As facebook shows you can, and get away with it.

pjc50• 4 years ago

Legally, their TOS probably allows them to be completely opaque and arbitrary.

And any system which involves reporting from the public will need to build a "this account gets lots of false reports of category X, to save time assume they're all false" system. As well as "all the reports made by this user are bogus".

etchalon2• 4 years ago

There are no laws which say a company has to be transparent and accurate when discussing its moderation systems.

lmm2• 4 years ago

Given that people pay to advertise on facebook, any false statements they make about their moderation systems via the internet almost certainly constitute wire fraud.

nocoiner3• 4 years ago

As Matt Levine says, “everything is securities fraud,” and this is probably the area of greatest exposure for a company with a substantial degree of departure between their stated practices and their actual practices.

It’s not likely to be more than a nuisance to defend and not terribly expensive to settle, but an enterprising plaintiff’s lawyer could probably find a cause of action that passes summary judgment.

KrishnaShripad5• 4 years ago

"Actually no, you can't say we treat everyone equally, we're manually going in and exempting certain people from scrutiny, and that's entirely within our discretion."

Isn't that literally what Twitter does as well? Making exceptions for Politicians and Government Representatives? Or does HN's bias towards Twitter exempt it from any form of scrutiny?

nl5• 4 years ago

Isn't that literally what Twitter does as well? Making exceptions for Politicians and Government Representatives? Or does HN's bias towards Twitter exempt it from any form of scrutiny?

What a completely bizarre comment.

No one mentioned Twitter, there is no "HN" general viewpoint, and if I had to say I'd say most comments on HN about Twitter are negative.

I'd say this could be the worst case of "Whataboutism" I've ever seen, but it is such a weird thing to use "what about" regarding.

numpad0• 4 years ago

I don't think this has too much with celebrities, but about exempting "problematic people" from being repeatedly banned by algorithmic and applied AI systems. IOW, they don't have controls over internal mechanisms of so-called algorithms, and a separate suppression system is used to reduce harm.

crmd• 4 years ago

Legal will certainly advise leadership they can continue claiming that the system treats everyone equally, because scope of the word system includes the entire Trust organization people + automation, and “equally” can mean almost whatever Mark needs it to mean.

“Senator, thank you for the question. We treat everyone equally. To be clear this means every single user on our platform is subject to the same terms of service they agreed to when signing up.”

IncRnd2• 4 years ago

In other words, you'd lie to Congress. Being glib is not the same as being truthful.

SilverBirch2• 4 years ago

By the definitions you're using, Mark Zuckerberg personally logging in to the database and type DELETE FROM USERS WHERE "Policital_Affiliation_heuristic" == "Conservative".

So fine, if you want to define the words as meaning something they clearly don't, and want to try and fight that battle in the media... ok. But almost certainly the first thing that happens is something exactly like this - some nerdy engineering comes out and goes "Well no actually, we manually intervene for the people we like". And then you get to go back to Congress and explain to them why you clearly lied to them, and you're not going to get out of that by using Mark Zuckerberg's pure eye-watering levels of charisma.

lazide• 4 years ago

Why can’t they exactly?

8note2• 4 years ago

I think the tag based system infers everyone is treated the same. Everyone can get the tag.

That's opposed to an Id or different account types, where if you didn't register with a specific type, you cannot be granted that same exemption

DannyBee2• 4 years ago

Most carefully claim that the laws they write apply equally to everyone, not that they equally apply them to everyone

Which (and I hate to ever sound likey I'm defending Facebook) is quite common in legal systems.

pixl97• 4 years ago

One user type the law binds but does not protect, and another user type the law protects but does not bind.

concinds• 4 years ago

I see zereo issue with what Facebook is doing. They are indeed "treating users equally." To not have special measures in place protecting accounts that are orders of magnitude more likely to receive false reports would be unequal treatment, and would result in celebrities' accounts being very frequently taken down.

"Unequal treatment" would mean: these users are not subject to our policies. That's not the case here. XCheck just turns off the "we receive 100 reports, an algo shuts that account down" phenomenon. Humans review those cases manually, and the account can still be suspended or banned. How in the world could that be contentious?

smolder3• 4 years ago

Your stance, as described, doesn't make any sense. You're being willfully obtuse here. Equal treatment would mean that users are treated the same by the company. They are not, in the same situation.

NikolaNovak3• 4 years ago

What they are doing may or may not be necessary and practical. It may or may not be fair or right.

What it empathically is not, by definition, is "equal treatment". It's existence is epitome of unequal treatment.

oars• 4 years ago

I would love to work at a FAANG in the future to gain exposure to these interesting problems and systems at huge scale.

philjohn• 4 years ago

They're interesting problems for sure; it's equal parts fun, technically challenging, and then you get to throw in having to quickly react to adversarial responses.

1letterunixname2• 4 years ago

So I work at Meta and just got MetaMorphized s/@fb/@meta/g.

When I started, my personal account was banned twice on the condition of adjusting 2FA. It was sorted out. I had a past account that I could no longer access as email domains changed, so it looked like I was trying to maintain duplicate accounts. Why it triggered on adjusting 2FA didn't make sense. Then, it wanted me to verify my identity in a way that created a Catch-22 of wanting a login from itself or to a device that didn't have a valid login session. I set the options for better security and left it, rolled all of my personal passwords, eliminated public data, canceled unnecessary social media accounts, and enrolled in security features like Google's Advanced Protection Program. I'm hesitant to make changes risking getting locked-out again.

I recently spoke with someone in Readiness who works with hiring human verifier contractors around the world. A primary issue is scale. You would need to hire the entire world's population to moderate the content being produced. Still there will be bias and misinterpretation of sarcasm, and varying standards of acceptability and decorum. AI is a force-multiplier to an extent, but it takes human judgement to rectify mistakes and data to identify brigades, scammers, terrorists, and political manipulators seeking to exploit an imperfect system. Sadly, the humans with the best judgement typically have better career options that they couldn't be paid to do social media moderation. It can be made better, but the ultimate realization is with even great care, good intentions, and attempts at making things sensible and fair, there are always going to be mistakes. It's trying to minimize mistakes and not enable genocides, election sentiment manipulation, or product scams. It's doable to minimize mistakes, but it take persistent vigilance, wisdom about human nature, and creative solutions to deter and prevent harm while avoiding harming innocent persons. Mistakes are bad and disappointing and it feels bad when they happen.

In case anyone were wondering, the security is the inverse of Twitter's. Everything is logged and access requires a business purpose for a limited time, narrow scope, and approval to get that access. Almost no one has access to production data. PII is taken very seriously. There are no laptops with copies of user data. All laptops are encrypted, just in case, and for general principles. Password complexity requirements are insane. I can see my work/personal FBID user object in the graph, but as soon as I try to prod any links to other users, big warnings appear. There's an army of insane genius security researchers and practitioners who create and deploy defense-in-depth tools for broad and specific solutions to prod, corp, and endpoints that reduce our risks to being compromised, data being exfil'd, and security "oops"es from happening.

Work users who transit through certain "hostile" countries lose some security credentials and access. I'm actually wondering why laptops aren't spot-checked for malware implants and hardware/firmware modifications. I would assume employees with critical access who travel internationally with their work laptops and phones are prime targets.

PS: I wonder if people would pay $X / month (say $199) to have a high-signal social media service that requires a level of "vouching" invitation, names with faces profiles (not visible to the wider internet), sensible/proportional mediation and civilized feedback, politically-neutral, and free speech-loving to increase the sense of community and reduce the potential of anonymous bad actors. 37signals/basecamp accumulated research that showed that smaller communities with faces and real profile names lead to nicer interactions. I don't recall the source, but communities that are defended in terms of politeness and boundaries tend to endure while undefended communities drive away users and tend to disperse.

fmajid2• 4 years ago

Don't forget the terrible PTSD endured by the people tasked with eliminating child pornography, who are often contractors and not employees.

civilized• 4 years ago

Maybe after 5 or 10 or 100 reports, the system will automatically make the reported content invisible.

This explains why so much "moderation" on sites like Twitter seems to lack any rhyme or reason.

josephcsible3• 4 years ago

Normally such content would be taken down after sufficient number of reports were received.

I can't be the only one who sees a problem with this. No content should ever be taken down automatically just because a bunch of random people report it.

throwawaylinux4• 4 years ago

No content should ever be taken down automatically just because a bunch of random people report it.

Serious question, why not?

fragmede2• 4 years ago

Because you have no idea if those reports are at all genuine, or if the reporters met up elsewhere (online or off) in order to brigade and mass-report said content, with the intention of getting it taken down despite breaking no rules. Sometimes, the coordination isn't even necessary, it just needs to be the right target posting something online. (Eg more than a few people have gone and reported every post by a politician you like/dislike for hate speech and inciting violence.)

josephcsible3• 4 years ago

Because coordinated false reports aren't uncommon.

TheDong2• 4 years ago

What is your suggested solution?

Your options are:

1. Leave up child porn, revenge porn, hate speech, gore, fake and misleading stuff, etc for a long time, possibly forever, since a machine can no longer automatically take down mass-reported content

2. Hire so many human moderators that facebook goes bankrupt

3. Use untrained and unpaid/underpaid human moderators, such as strangers (the reddit model); go back to square 1 where volunteer moderators take down stuff they don't like with minimal oversight

4. Have so few users that a small number of human moderators can actually review every report, i.e. the hacker news model. Or actually, no, hacker news "dead"s posts based on just number of reports without a human seeing it, I take that back. I guess this is the approach forums, most smaller blog comments, and other quite small websites use.

Do you have another solution that isn't one of those? Do any of those solutions sound good for facebook? Better than what they have now?

josephcsible3• 4 years ago

Here's a possible system: replace "random people" with "people with a solid track record of previous correct reports". When something gets reported, human moderators categorize it as either "correctly reported, and should be taken down", "incorrectly reported, but possibly a misunderstanding or a borderline case", or "a clearly bad-faith report against content that no reasonable person would actually believe breaks the rules". Keep track of how many reports from each user end up in each category.

If almost all of your reports are in the first category, then you're considered trusted, and if enough trusted users report a post, then it can be automatically removed before a human moderator sees it. If you haven't reported anything before, or too many of your reports are in the second category, then your report only helps to get the submission in front of a human moderator and doesn't directly contribute to it being removed. If more than a handful of your reports ever end up in the third category, you get banned for abusing the report system.

asah3• 4 years ago

"The number of reports number in the millions per day. That's too many for human moderators"

Isn't Reddit the counterexample?

A dozen of us moderated a controversial sub with 2M users, no problem.

HiJon89• 4 years ago

With 2 million users, I imagine it would be on the order of thousands of posts per day, and maybe hundreds of reports?

sophiebits• 4 years ago

So for a few billion users, you need tens of thousands of moderators, assuming that ratio holds?

smolder3• 4 years ago

This is a very efficient moderation force. I'd be happy to get away with 10s of thousands per billion. That's still under 100 per million. Under 1 per 10,000.

Spivak• 4 years ago

2M “subscribed” users to a subreddit is about 10% or 200,000 daily viewers with about 1% of those commenting and 0.1% posting so 2,000 comments and 200 posts per day. Some subs will be more or less but that’s the magnitudes we’re talking about.

The equivalent situation assuming mods review every comment would be a subreddit with 1 billion users. If mods are only reviewing reports which is some small percentage of comments then adjust accordingly.

fullsend• 4 years ago

Very well put, it's not even close to the same ballpark. It would also be like moderating a 1 billion user subreddit where nearly every single post is off topic and comes from different users, whereas in subreddits a few super users often produce a huge percent of the content.

asah3• 4 years ago

more review every comment

Moderation workflow starts with user-reports on posts/comments. At scale, you can set a threshold as well, which dramatically reduces the number of items to review.

Obviously, scaleup the number of reviewers. Obviously, add some obvious algorithms: warnings, temporary/permanent bans, etc. ...

hokumguru• 4 years ago

Try multiplying your numbers by a few thousand, billions of people use lets platforms daily. Reddit user interaction is also very different in regard to celebrities.

mistermann2• 4 years ago

A dozen of us moderated a controversial sub with 2M users, no problem.

No problem except for people who may have been censored due to bias and error on behalf of moderators.

nocoiner3• 4 years ago

Sure, but bias is going to be a factor with any system of evaluation. It happens with human moderators, it happens with automated systems, it happens in corporate performance reviews and it happens among 12 of a defendant’s peers convened for a criminal trial. The existence of bias doesn’t pertain to whether or not a given system is scalable.

seattle_spring3• 4 years ago

Which sub? I think people could claim they're successfully moderating subs like the_donald and wallstreetbets. I would disagree.

whiddershins• 4 years ago

Sounds like binary flags that should be weights.

Zenbit_UX• 4 years ago

If I were designing a system like this, I'd likely implement a reputation system for reporters (users who report, not journalists).

When you report content for the first time your reputation score is 0.5, if the content is of a celebrity or public figure, your reputation will invisibly drop, enough of these and your score will be so low as to not be a useful flag when evaluating whether content is objectionable.

On the other side, if the content is reviewed as violating ToS, your rep will increase.

With a system like this the fake reports and brigading will quickly lose its power, eventually all users who abuse reports will be ranked so low as to not matter.

A threshold of rep score can be set before the content is auto removed. Maybe something like 25pts in order for it to be removed, and that's after adding together all the rep of the reporting users.

koyote• 4 years ago

I always thought that this is how it already worked.

That being said, the above can also be abused by creating a bunch of high rep accounts or opening up a black market of high rep accounts for sale. It's definitely a higher bar though.

DannyBee2• 4 years ago

In a system like this they will play both sides. They do, in fact. Set up accounts that produce content that violates TOS and accounts that report it. The reporting accounts will have their rep increased for correctly reporting tos violations.

Then they use the high rep accounts to brigade and fake report. Or sell them.

They are patient, so even if you make reputation go up slowly, they will just build a pipeline of account harvesting.

nonrandomstring• 4 years ago

The corruption in tech, perhaps inevitable or intrinsic, looks more and more like the forces that tore the Soviet Union and then Russia apart.

1) A "revolution" and promises of new fairness

2) A Nomenklatura

3) The rise of unassailable systems and faith in them

4) Widespread failure and instability of those systems

5) Attempts to stabilise by handing more power to privateers

6) Abuse and corruption run wild amongst new oligarchs

1letterunixname2• 4 years ago

I'm wondering if it's XCheck that locked-out my later account when it decided by zombie account (I don't have the email address anymore) was a "duplicate". The check condition it fired on was visiting the 2FA settings page. Some nice wf on-call fixed it from an oops task and I don't touch that page anymore. :)

XCheck at Meta: Why it exists and how it works

Related Stories