Show HN: Perceptual hashing tools for detecting child sexual abuse material

by faustomorales • 7 years ago

92 49 7 years ago

Hi HN! I'm a data scientist at Thorn, a non-profit dedicated to defending children from sexual abuse. We're excited to open source some elements of our perceptual hashing tooling as a Python package. We've tried to make it very flexible both for ourselves and hopefully also for others. Our aim with is to provide tools that (1) help more people eliminate child sexual abuse material from the internet and (2) assist with common tasks where perceptual hashing can be helpful (e.g., media deduplication). We hope you'll take a look, get some use out of the package (check out the example use cases), and even contribute feedback and/or code to make it better.

For more information on the issue, I urge you to check out our CEO's TED talk here: https://www.thorn.org/blog/time-is-now-eliminate-csam/

Documentation for the package here: https://perception.thorn.engineering/

akersten• 7 years ago

If I may, I'm curious about your thoughts on few things, in the context of use case #1 (abuse), not #2 (reduplication):

* What are the challenges surrounding verification that your system functions properly, given that the test material is illicit?

* Can you speak to the reliability of the system in a sensitivity/specificity kind of way? In other words, what are the false positive and false negative rates?

* Are you aware of any large organizations leveraging your solution?

* Do you feel that the availability of these tools obligates service providers to use them, either morally or legally?

nathanaldensr• 7 years ago

Wouldn't a human have to go through thousands (or more) of illicit images and classify them in order to train the AI?

faustomorales11• 7 years ago

Thanks for asking these important questions!

* What are the challenges surrounding verification that your system functions properly, given that the test material is illicit?

You're right that storing child sexual abuse material (CSAM) is illegal, unless you are the National Center for Missing and Exploited Children (NCMEC) or law enforcement. What is legal is to maintain a hash of known CSAM. NCMEC, Law Enforcement, and large tech companies maintain their own data sets of known CSAM hashes and, where appropriate, share them. The Technology Coalition [1] has more information on this. All that said, we can and do simulate the system to verify that it works properly using bench testing with non-illegal content [2].

* Can you speak to the reliability of the system in a sensitivity/specificity kind of way? In other words, what are the false positive and false negative rates?

The false positive rate in practice is very low. We set our thresholds based on bench tests with an expected false positive rate of less than 1/1000 (the thresholds vary based on which hash function was used). Different hash functions are more resilient to some transformations than others (e.g., cropping, watermarks, etc.).

For the false negative rate, it depends entirely on the kind of modification made to the image. For many common operations, it is close to zero.

* Are you aware of any large organizations leveraging your solution?

Thorn builds technology to defend children from sexual abuse, one of the products we build for this purpose is Safer [3]. Perception provides an easy way to get started using the Safer matching service. Safer provides a more robust and complete solution including handling a queue of content and reporting tools. Some organizations using Safer include Imgur, Flickr, and Slack.

But this technology (perceptual hashing) is used by many companies who don't use our tools. Our goal is just to make it easier for more people to get started.

* Do you feel that the availability of these tools obligates service providers to use them, either morally or legally?

Not being a lawyer or a public policy expert, what I can say is that the law, as I understand it, requires companies to report CSAM once they are aware of it. Working in this field I’ve learned two things pertinent to this question: (1) Most people don’t know how pervasive of an issue this is, and (2) There aren’t a lot of easy ways to start protecting your platform from this abuse. No one wants the cool new products and platforms they make to be used to abuse children. Privacy is important too, which is why solutions that preserve privacy and avoid leaking private information to third parties are critical, and perceptual hashing allows us to do both.

[1] https://www.technologycoalition.org/

[2] https://perception.thorn.engineering/en/latest/examples/benc...

[3] https://getsafer.io

EDIT: Line breaks

noneeeed• 7 years ago

I just wanted to take the opportunity to thank you for what you and your colleagues do. I honestly don't know if I could work in a field like that without just being overwhelmed by it, but I'm really glad that others can and do.

jsjohnst4• 7 years ago

Can you comment on how this compares to PhotoDNA? I looked at the readme and was surprised to not see mention of it.

faustomorales11• 7 years ago

Great question! PhotoDNA is the most well-known and supported hash function for this use case. And we do support hashing and matching with PhotoDNA in the Safer product. However, the PhotoDNA hash function is non-public so we cannot include it in an open source package. We support pHash as an open source alternative so that companies without PhotoDNA licenses can get started with hashing.

RcouF1uZ4gsC• 7 years ago

Was there discussion about the downsides of open sourcing the implementation? With an open source implementation, it becomes easier to test transformations that will change the hash without changing the image in ways a human can’t notice?

faustomorales11• 7 years ago

Great question -- all of the hashes we included in the package have been public for years (except for PDQ, which was open-sourced this year). So this package doesn't reveal anything new with respect to the algorithms themselves. What we add is an easy path to using at least one of them for the CSAM hashing / matching use case. Non-public perceptual hashes for this use case exist and, naturally, are not available in the package.

tailaiw• 7 years ago

Great work faustomorales. Thank you and your colleagues so much for this. Greeting from your old buddies ;)

faustomorales11• 7 years ago

Thanks, old friend! Hope you decide to give the package a peek and maybe lend us some of your Python chops. :)

probablypat• 7 years ago

Thank you for what you and your team do. Is there anyway I can help contribute to your organization?

faustomorales11• 7 years ago

Thank you so much for asking! Naturally, part of why we wanted to share this with the broader community was to make it so interested people can jump in and help out in the open. And I would be remiss if I didn’t mention the fact that we’re hiring! [1]

We can always use help raising awareness. Advocating for more survivor resources is a great place to start. Help spread the word through your social networks by connecting with us on Facebook and Twitter. Learn about even more ways to get involved and subscribe to our newsletter for general updates [2].

[1] https://thorn.org/careers

[2] https://thorn.org/join-us

concordDance• 7 years ago

help more people eliminate child sexual abuse material from the internet

Is that actually a good thing?

Less material seems like it would mean more motivation to produce more, which is the very thing we want to avoid.

tempaccount8354• 7 years ago

If you dare to look into this topic, please also take a moment to look to another child protection non-profit[1] that (quite unusually) considers human and civil rights and sex positivity as one of its core values.

You might feel uneasy about "sex positivity" being associated with preventing child sexual abuse. They also have a lot of other messages that might initially repel you. But I also think that "when you associate shame and guilt with sex, you are facilitating sexual abuse"[2].

Here is a relevant Twitter thread about CSEM content filtering[3] and the secrecy around it. Secrecy that got them ejected out of a public National Center for Missing & Exploited Children meeting merely for tweeting about what was being said.

This Show HN open source project by Thorn seems to be an enormous improvement on that front. Would it be resistant to adversarial hashing (false positives)?

In my opinion, Thorn focuses way too much on technological solutions, and has an outright hollow message beyond that. Looking at 10+ of their website's pages, they don't dare to try to confront or explain the actual child sexual abuse itself, but only its most visible ill effects.

[1] https://prostasia.org/about/

[2] https://prostasia.org/blog/the-weapon-of-shame/

[3] https://twitter.com/ProstasiaInc/status/1178783074328424448

claudiawerner• 7 years ago

I think it's good you posted this; the goals listed on the first link seem very reasonable to get behind, and it's something both I and researchers in child sexual abuse material (e.g. Amy Adler's paper, which is one of the most cited in the field) have noticed. The rhetoric of "think of the children" can be and has been used many times to silence sexual activity between adults, including fictional representations (such as manga and comics, some of which are illegal to possess in the UK), kinks (online ageplay roleplay is often targeted under obscenity law, despite being between two adults), and other cases.

In my view, the protection of children based on hard evidence showing causal links is paramount, not suspicion and anecdota (it was, in fact, anecdota given before Parliament which was used in the creation of the bill in England and Wales to illegalize lolicon manga) based on the "social harms" that seem obvious at first sight but are refuted by cultural anthropology (e.g Patrick Galbraith and Mark McLelland have come to very interesting conclusions regarding how adults consume fantasy material). There is no group to fight for the few thousand people caught under this law so far (according to the English VAWG report 2017), nor do I imagine their being. The organization you linked seems, at least, to have this sort of thing on the agenda.

pbalau• 7 years ago

From what I've read, this is supposed to catch the bad thing after it happened, not to prevent it. Then you say that 1 in 1000 is an acceptable error rate. 1 in 7 billions is not an acceptable error rate!

hnbroseph• 7 years ago

without a "minority report"-esque system, or preemptively putting the entire population in cages, you cannot prevent every ill of the world.

catalogia• 7 years ago

"Then you say that 1 in 1000 is an acceptable error rate. 1 in 7 billions is not an acceptable error rate!"

When systems like this return a positive match, that doesn't result in a summary execution. Rather, it prompts a deeper investigation.

ALittleLight2• 7 years ago

If you catch people who have done bad things you prevent the bad things they are likely to do in the future.

1 in 1,000 does seem a bit high to me, but the prevalence of the bad images matters. You also have to think about the consequences of false positives versus admitting child pornography. If 0.1% of innocent meme pictures get removed from a site incorrectly - that's not ideal, but how much sexual exploitation of children do you need to stop to make that worth it?

httpsterio• 7 years ago

Without reading the link, can this be used to help find child porn? If yes, are there any steps you can or are going to take in order to prevent this?

Nasrudith• 7 years ago

It only filters images already there - child porn isn't already among the chaff only thing it can find by definition would be false positives. Which could be an interesting vetting exercise in itself - feed in tons of cat pictures until it dings on one as inappropriate.

Technically you can always try cycling through image combinations match the hash/looks like child pornography and nothing could technically prevent it except that it would take a very, very, long time.

Even given adversarial networks I would be very surprised to get anything more than vague figures and I don't even expect the result to look human. In which case it is really generating instead of finding.

callmejorge• 7 years ago

This is great work. Not often we see posts about using tech directly for humanitarian efforts!

GistNoesis3• 7 years ago

How did you build, label and store your dataset ? What's the legal framework for manipulating such data ?

olliej• 7 years ago

Seriously this is a topic that I can’t imagine having to think about :-/

I really hope that there’s support in place for people who have to work on this, even if they aren’t exposed the actual abuse.

m0zg• 7 years ago

When I read about something like this, I can't help but wonder, how do you deal with error analysis and such? I personally would not be able to view such disturbing and revolting material. And if you don't do error analysis, how do you know this stuff actually works?

totoasticot• 7 years ago

Reminds me of https://github.com/CIRCL/douglas-quaid which also use SIFT-similar techniques in addition to perceptual hashing

amitport• 7 years ago

I don't get it. Sound nice but what exactly does Thorn do? what is this spotlight services they're providing? Seriously I've just spend time on their site but I don't get it beyond "using data". How they use it?

Lets say I have friends in a local DA office and I want to sell them this idea of using data they have. They just know that giving access to their data is a big legal headache. what exactly to I tell them to get them on-board with this? (exactly how the data / what data is handled?)

GistNoesis3• 7 years ago

How do you defend against semi transparent overlaying ?

Take an image you want to censor, overlay with a very transparent offensive image. Also publish the original image (or a second image with a different transparency value), and an in-browser extension can reconstruct the offensive image.

Your hash database will be flooded with wrong values.

lonelappde• 7 years ago

That seems like symmetric key encryption with extra steps.

GistNoesis3• 7 years ago

This is a general attack against perceptual hashing. It tries to achieve multiple objectives. Using stenography you could add any data in the image, like you would with encryption as you suggest. But here it is deeper.

By superposing a very transparent bad image with the good one, the perceptual hash won't be affected but the image would be still labelled as offensive by the poor guy labeling the data because the transparent offensive image is still visible. This mean that the good data will be marked as potentially offensive.

This allows anyone to target any individual user, or website. You take some of their published content, overlay offensive image and republish. The image will be reported as offensive, yet have a similar hash (i.e. a collision) to the good image which automated systems will pick-up as the original image being offensive and blacklisting its user or tanking down the website on search engine results.

Because this is sensitive data, by law it should be deleted as soon as it is detected which means the proofs get deleted automatically.

natch3• 7 years ago

What are some of the ways you can improve on the n-squared nature of these distance calculations at scale?

In your use case you have a limited set of hashes you are comparing against, but I'm thinking of the more difficult use case of comparing every image against every other image (the context is ML training and weeding out overly similar examples so that overfitting doesn't happen, not related to CSAM). This quickly becomes untenable if you get, say, millions or billions of images.

"Don't do that" is one answer but are there any other ways you have come across to mitigate that?

faustomorales11• 7 years ago

Hi there! Our example on deduplication [1] takes the case of deduplicating of the Caltech256 [2] dataset for the purpose I think you're describing, which I interpreted as avoiding duplicates in a dataset so that you don't have images that end up in both your training and test sets. Even though the dataset contains >30K images, you can still do this in memory (and find a handful of duplicates!) because each category is relatively small and you probably don't need to deduplicate images of trains with images of dogs.

On the same page, we mention two tools (specifically, FAISS [3] and Annoy [4]) to help with doing approximate search for scales where computing the distance matrix is impractical.

[1] https://perception.thorn.engineering/en/latest/examples/dedu...

[2] https://authors.library.caltech.edu/7694/

[3] https://github.com/facebookresearch/faiss

[4] https://github.com/spotify/annoy

natch3• 7 years ago

Yes looked at annoy previously and it is really impressive. Will probably revisit that as part of the solution here. You did interpret my use case exactly right. Thanks for the reply and especially for the links!

natch3• 7 years ago

One more thing: I tried the different hashes that would run on my system (some had a dependency on opencv-contrib that was not met, and which python -m pip couldn't find) and found they aren't what I'm looking for. They may be good at finding different versions of the same image, but what I want is more like environment similarity. So that two pictures taken on the same street would be closer than a picture of the street versus a picture of a forest. They don't seem to be tuned for this task at all. So, my search continues. Let me know if you have suggestions on this as well.

Show HN: Perceptual hashing tools for detecting child sexual abuse material

Related Stories