Expanded Protections for Children

https://www.apple.com/child-safety/

by robbiet480 • 5 years ago

230 365 5 years ago

apple.com

robbiet480• 5 years ago

Previous discussion at https://news.ycombinator.com/item?id=28068741

ComputerGuru6• 5 years ago

Which would still be on the front page given how many upvotes it had if people hadn’t, for some reason, flagged it.

Telemakhos2• 5 years ago

The previous discussion was based on a few Tweets. This is Apple's information release, and it is more specific. For me, the first thing on Apple's release is the most important: that they will begin censoring Messages content for all possible nudity, not just hashed CSAM, when the recipient is a minor. That's a step beyond the earlier discussion.

Perhaps this same approach can also soon be applied to filtering out the lies and misinformation of the enemies of the Party and to disrupt Emmanuel Goldstein's networks of treasonous domestic terrorists. After all, such misinformation is a plain harm and danger to IngSoc. That Apple is willing to eliminate nudity in private messages gives hope that the Party will soon be able to free all society from doubleplusungood differenthink.

echelon2• 5 years ago

This is what happens when you abandon individualism and replace it with corporate worship. When you make brands a part of your identity, you can't think about the real world damage being done.

If you're on the fence about Apple, and their lockdown of free and open computing wasn't enough, or their strong arming of the mobile commerce economy didn't do it, please let this sink in. This is where we're headed. 1984 is a plausible future.

Stop buying Apple and ask your representatives to break up the monopolies. This order came from the FBI and intelligence communities (and probably the CCP and respective organs in China).

Not all US lawmakers are beholden to this abuse of power, and they can help us put an end to it. Call them. And stop buying Apple.

Fight for your democracy and freedom, lest it be completely wrested from us.

hoppyhoppy2• 5 years ago

Also https://news.ycombinator.com/item?id=28075021

netr0ute6• 5 years ago

Sad, more spyware masqueraded behind "think of the children."

DSingularity15• 5 years ago

At some point it becomes amoral to do nothing.

jszymborski3• 5 years ago

That's a false equivalence. Surely there are things that are to be done (and in fact are done) that don't involve spying on all civilians.

netr0ute6• 5 years ago

To do nothing against what?

Zak3• 5 years ago

This is a very slippery slope. Users of iOS devices have already ceded general-purpose computing in favor of Apple being their sysadmin. Now Apple is using that control and access to act against the interests of some users.

The fact that it's only targeting users doing something almost universally considered heinous is irrelevant. If Apple can scan your encrypted photos to make sure they don't contain child abuse, why not scan your encrypted chats to make sure they don't contain a conspiracy to commit murder, or an arrangement to buy drugs, or a plan to reveal something embarrassing to the ruling party?

bob10293• 5 years ago

These efforts will evolve and expand over time.

I have no words.

DSingularity15• 5 years ago

I feel like the absolute positions are they easy ones to take. But be honest with me, would you not celebrate an outcome where a system like this eliminates the potential use of iOS for CSAM and in turn causing significant decreases in the abuse of children?

cultofmetatron2• 5 years ago

you seriously think they are going to stop at CSAM? that just their foot in the door. a Generation ago, they would have used this to identify lgbt people for instance.

echelon2• 5 years ago

This isn't about children! Children mean fuck all to these people. This is the surveillance state building more instrumentation and monitoring into the fabric of society. "Thinking of the children" is the time cherished tradition for pushing through abusive tech.

This will be used in Xinjiang and Hong Kong. And soon, it will be used against you.

The only relief I have is that Millennials and GenZ aren't having kids and many couldn't give two fucks about them "being protected" from imagined goblins at a cost to their convenience. Maybe this argument won't work anymore.

bob10293• 5 years ago

No I would not celebrate such an outcome, because I have been around long enough to be aware of and care about higher order consequences of decisions like these.

Have we already forgotten about where fear of terrorism has landed us?

Hamuko5• 5 years ago

No. Whatever minimal impact this will actually have does not justify the vastly oversized means that they will be accomplished with.

I would also not celebrate random warrantless house searches even if it managed to stop a couple of terrorists attacks in a decade.

anigbrowl• 5 years ago

While I share your view that child abuse is a serious problem that hackers and privacy advocates would prefer to wish away, the idea that child abusers' decision calculus is centered on the need to have an iPhone seems bizarre.

Anyone motivated to engage in that kind of behavior is just going to use a different device for their illicit material.

Syonyk7• 5 years ago

Don't shit in my hand and call it chocolate ice cream.

"We're going to scan your photos, on your encrypted device, to look for badness. Right now, we're going to claim that's only for the really icky people that nobody is going to defend, but, hey, once the tech is in place, who's to say we can't scan for dank memes and stuff?"

I think I'm done with Apple. Sad, really. I was hoping that their bowing to China with iCloud wasn't a sign of what's to come, but apparently it was. They had done such nice stuff with privacy too.

Demote my phone to a house phone and go without, I suppose.

DSingularity15• 5 years ago

Would you please describe a solution to finding CSAM which you would find acceptable? It seems to me that on-device scanning for known CSAM content is on the lower end of the privacy-violating spectrum.

jszymborski3• 5 years ago

No, the lower-end is infiltrating CSAM distribution networks the old-fashioned way.

gpm6• 5 years ago

Traditional police work that respects the principle behind the 4th amendment, no searches of private information, algorithmic or otherwise, directly by the government or otherwise, where you do not have probable cause.

Blikkentrekker2• 5 years ago

Child pornography is the boogyman used to compromise privacy.

There is significantly more illegal material of greater concern such as communication to plot terrorists attacks or companies ignoring safety regulations and leading to the deaths of many in doing so, and all that isn't scanned for either on the theory that it might exist, not that there is due cause for a search warrant or some similar mechanism and burden.

But child pornography is the one thing Anglo-Saxon culture is notoriously emotional about, and willing to surrender all it's freedom and privacy for. For a while terrorism counted among this too, but that seems to have fallen off.

hapless7• 5 years ago

There is no solution to searching private data that I am going to find acceptable.

* If the "target" database is secret, then government entities are free to use a "CSAM" database to monitor dissidents by adding their own items of interest to the target list.

* If the "target" database is public, the only way to validate its contents is to ... traffic in child sex abuse material. That's not great.

Basically, there is no way to create and validate this database in public view. I don't need yet another flavor of secret policing in America.

This doesn't seem like a problem that can be solved without compromising my privacy.

I would rather the problem went un-solved than allow the state to rifle through all of my private files to "prevent the distribution of child sex abuse material," knowing full well the state will define that "material" however they like, then use parallel construction to prosecute dissidents whenever they have gotten information illegally.

diebeforei4856• 5 years ago

Yeah. They can scan photos that are actually passed around, instead of something that's just in a photo library. That will likely result in 2-3 orders of magnitude fewer false positives.

gorgoiler• 5 years ago

Boosting the amount of influence schools have in childrens’ lives is something we could do better at.

More teachers, smaller classes, stronger relationships, better discipline, more love. I see too many teachers where I work who just don’t care any more.

Educating school leaders to educate their teachers to help children root out the family member that is abusing them. That there would be a pretty neat solution — and I really don’t mean to be inflammatory about this — than catching villains based on photo evidence of the abuse that has become systematic enough to be shared online.

It is very expensive and time consuming though. Teachers are hard to recruit (low pay, low quality job) which exacerbates the problem which makes it even harder to recruit them.

alexfromapex2• 5 years ago

Them having to get a warrant or some form of fair adjudication instead of a carte blanche of privacy invasion?

Syonyk7• 5 years ago

Find the websites distributing it, infiltrate them, generally do the legwork to find what's going on - which is exactly what they've been doing.

"But the children!" is not a skeleton key for privacy, as far as I'm concerned.

I reject on-device scanning for anything in terms of personal content as a thing that should be done, so, no, I don't have a suggested way to securely accomplish privacy invasions of this nature.

I'm aware that they claim it will only be applied to iCloud based uploads, but I'm also aware that those limits rarely stand the test of governments with gag orders behind them, so if Apple is willing to deploy this functionality, I have to assume that, at some point, it will be used to scan all images on a device, against an ever growing database of "known badness" that cannot be evaluated to find out what's actually in it.

If there existed some way to independently have the database of hashes audited for what was in it, which is a nasty set of problems for images that are illegal to store, and to verify that the database on device only contained things in the canonical store, I might object slightly less, but... even then, the concept of scanning things on my private, encrypted device to identify badness is still incredibly objectionable.

In the battle between privacy and "We can catch all the criminals if we just know more," the government has been collecting huge amounts of data endlessly (see Snowden leaks for details), and yet hasn't proved that this is useful to prevent crimes. Given that, I am absolutely opposed to giving them more data to work with.

I would rather have 10 criminals go free than one innocent person go to prison, and I trust black box algorithms with that as far as I can throw the building they were written in.

aaomidi6• 5 years ago

Yes. Find the one's producing it and use proper infiltration tactics.

Empower children to report abusers and protect them when they do.

wyager5• 5 years ago

Don't spy on my device. I literally don't care if that doesn't help some pet cause.

WindyLakeReturn3• 5 years ago

on the lower end of the privacy-violating spectrum

I think that's the thing, privacy minded people want solutions that aren't on the spectrum at all.

I can come up with some solutions for greatly reducing CSA that are really low on violating parental rights but they have little chance of gaining support because the only solutions people want are the ones that don't violate parental rights at all (except when parental rights have been suspended due to evidence of CSA being found).

throw1998149898• 5 years ago

Anything which

- directly prevents actual abuse of children in the physical world (obviously)

- fights the sale of images created through abuse (as this incentivizes abuse).

Those are the main underlying reasons to fight CSAM, besides the obvious horrificness of the stuff in itself.

Please also consider the second-order effects of an increasingly totalitarian society on the wellbeing of children.

tomjen32• 5 years ago

Not OP, but a warrant, upon probable cause, supported by oath or affirmation, and particularly describing the place to be searched, and the persons or things to be seized.

In other words, you don't get to yell think of the children and randsack peoples homes. If you don't have specific evidence suggesting that I am a criminal, you assume that I am not _and_ let me be.

r00fus4• 5 years ago

How about not doing it.

The "privacy" of a on-device scanning is meaningless when you upload to iCloud where they will send to law enforcement and an unaccountable nonprofit as soon as it matches something.

etchalon14• 5 years ago

Ah yes, the "slippery slope" argument, where evidence is never required, pessimism always wins over realism, and any counter-argument can be dismissed with "but they COULD!"

rsgrn• 5 years ago

You're right in one sense. In another, isn't this exactly what people said about intelligence agencies ("they could but they don't") and then... Snowden.

Hamuko5• 5 years ago

Considering that Chinese iPhone users already do not enjoy the same privacy features as non-Chinese iPhone users, I would not be at all surprised if Chinese iPhones are going to have a lot more "child porn" hashes to check against.

Syonyk7• 5 years ago

Do you trust your government to not abuse "But they could?" types of capabilities?

I don't. I've seen what the FBI claims, that they then magically find ways around the "impenetrable encryption" they claim they can't break (but then do), and I don't trust them to not require Apple to add hashes of "things they find problematic," and include a gag order with it so Apple can't report it.

I've worked computer security long enough to know that it's always worse than advertised, always will be abused, etc.

gouggoug• 5 years ago

It is a slippery slope argument, but wouldn't you agree that it's been proven by the past decade of events (e.g. Snowden) that we can expect governments to try (and succeed) to abuse originally well intentioned technologies to spy on anyone?

RIMR2• 5 years ago

I think we've learned from experience that tech companies will happily exploit anything they’re capable of exploiting, and this new capability has plenty of potential for exploitation.

Searching everyone's media for evidence of criminal wrongdoing, but setting only one example of the kind of wrongdoing you're looking for is very susceptible to an actual slippery slope, given that there are plenty of other criminal activities they could start looking for should they decide that it's part of their mission.

The only thing that makes CSAM so attractive to go after is how disgusting society feels that it is. Up next could be reporting drug-related texts of known convicts to parole officers. Next could be drug-related texts of everyone to police officers. Next could be letting the police officers make their own searches, where they find everyone talking about a certain kind of political organizing.

We should refuse to accept even the first instance of this kind of thing.

colordrops3• 5 years ago

The "slippery slope" argument is not a fallacy. If you were in 2001 and told people what the state of data collection and privacy would look like in 20 years, they'd call you an absolute conspiracy nutjob. It was a national news story then that a printer driver phoned home. Now look at where we are at, and every loss of privacy has had some seemingly reasonable excuse.

revolvingocelot7• 5 years ago

I remember when they put a fucking U2 album on everyone's phone. Remember that? Many found that suspicious, but I recall people like you saying "oh Apple can write, who cares, but they'd never read, you're being ridiculous".

I remember when an Apple auth server went down and no one could launch non-Apple applications because Apple needed to see the hashes of the things people ran on their computers.[0]

When will the slope be inclined enough for you?!

[0] https://mobile.twitter.com/llanga/status/1326989724704268289

hncurious• 5 years ago

You're right, but isn't the rapid expansion of censorship and surveillance the past year evidence that we are currently slipping down the slope?

spockz• 5 years ago

Just the other day in the Netherlands it came out that cameras placed with the strictest promise that they would only record license plates and only trigger when it matched with a known license plate now also capture persons in the car. And before that the systems were changed to store pictures for 28 days in order to look back. It also came out that the deletion doesn’t always work and that the obfuscation that is supposed to happen also doesn’t always happen.

The reaction of the police seems to be “why do you care about this? We have the data let’s use it. And we could use it to find those ~killers~ these people that allegedly just killed the most popular crime news reporter. So let us do our work.”

Just like software expands to eat cpu/memory/disk resources and people start living up to their salaries: whatever is possible will happen.

To me the slippery slope argument is perfectly valid.

longtom• 5 years ago

Edward Snowden's revelations provide very solid evidence that what can be done is being done.

Zak3• 5 years ago

Prior to this change, they couldn't. Now they could.

It's much easier to change the policy than to add a new mechanism. Governments, including the US government have previously attempted to force Apple to break into users' devices, and often apply gag orders to such attempts.

Now they could, and that's enough for me to be unwilling to keep sensitive information on an iOS device.

wyager5• 5 years ago

Slippery slope is a shockingly predictive heuristic.

hapless7• 5 years ago

Well, there's not really any slope here. Apple plans to have a target database of BAD FILES!!1! that will get your name handed to the government.

It's not subtle. No expansion to the program is necessary to violate your privacy or endanger dissidents.

randyrand4• 5 years ago

Slippery slope? What they already announced is abhorrent. Scanning all of our photos against our wishes?

Who cares what it's for? The fact that they are even doing this at all is a huge infringement to their customer's autonomy.

narrator2• 5 years ago

Who has access to update what are considered bad images and when to alert the authorities? This could very easily be made into an anti-wikileaks feature.

nulld3v• 5 years ago

Exactly, if some whistleblower is going leak a picture of some classified documents, the intelligence agency could simply upload a picture of the documents to the database. And bam, the journalist is arrested.

Moreover, if the database doesn't store actual images and instead only the perceptual hashes, it would be impossible to audit, even if the auditor has access to the database.

All the auditor would see is a bunch of hashes, they wouldn't be able to tell which hash actually represents CP and which hashes represent pictures of confidential documents.

stefan_2• 5 years ago

The next question is since this database is not exact binary hashes, but an ill defined "perceptual hash" - how many false positive images has this system surreptitiously extracted from customer hardware and shown to presumably human control personnel? Were these customers notified? What is the recourse for them?

PhotoDNA has been around for a while and you might think about it what you want, but they have never answered those very pertinent questions.

baybal2• 5 years ago

Lawsuit incoming? How this is possible if this seems to go 180° against them previously pretty much putting a privacy guarantee in EULA.

Will they announce an EULA change, reversing the guarantee, and freeze anybody who haven't clicked it through?

I thought Apple was one of a kind company brave enough to put such a guarantee in writing.

tmn2• 5 years ago

You’re already completely trusting them if you’re using their device? I mean I get the privacy concern, but it’s odd that this is the line if you were fine with things before. That said I’m all for ditching the smartphone and best of luck to you

wyager5• 5 years ago

If they outright lie about their device spying on you, the repercussions are much worse than them openly spying on you.

Syonyk7• 5 years ago

Historically, Apple has come down, hard, on the side of "Your device is your device, and where that assumption is violated, we will work to protect it." They have added increasing layers of hardware security around device encryption keys, have added the various mitigations for password guessing attacks, etc.

And when the FBI said, "Hey, can you write and sign a custom bootloader for this phone to bypass that stuff?" they told the FBI to pound sand and made the hardware security features stronger so even Apple couldn't break them.

And then they bowed to China regarding iCloud and in-country servers, which clearly are accessible to the government. And then this. Whatever claims they've been making about privacy are obviously now crumbling under some external pressure.

tomjen32• 5 years ago

At the limit you are trusting the sum of the people who are creating the software you use - and that includes drivers, firmware, etc. Linux is not exempt, but it may be easier to find issues in open source code.

Given this, Apple was not a bad person to trust. Yes they wanted you to pay them money, but were a) quite happy to deliver value in return, b) prepared to do lots of hard work to fight on your behalf and c) it was quite clear that you were their customer, not their product. Google is an advertising company, even if you pay them money you are always also the product.

I am seriously wondering what happened to Apple that they came up with this idea, it doesn't seem to be in their interests at all.

erhk7• 5 years ago

Im worried about the millions of peolle who dont read hacker news

alexfromapex2• 5 years ago

This is all a few months after touting themselves as a company focused on privacy. Decentralization is the only way to truly know you have privacy today I suppose.

nradov• 5 years ago

I appreciate that Apple is trying to find technical means to reduce child porn. That's a more important problem to solve than optimizing ad click rates. But my concern is that once the tools for local image scanning exist, Apple will come under pressure from authoritarian regimes like China, Australia, and Saudi Arabia to also search for images associated with lesser crimes or even just criticism of the government. It's tough for a company to refuse. Either comply with the orders, or risk having your business shut down and your local employees punished.

hapless7• 5 years ago

There is something like a 0% chance this program wasn't co-developed with representatives of the government entities you least want rifling through your data.

option_greek2• 5 years ago

You are stating it in reverse. Think about the children is always an excuse to get started with the next big things (like all the other things that you mentioned). So its step 1 of a grand plan on the lines of 1984 (probably to get some more markets to open for them by cozying up to governments).

aaomidi6• 5 years ago

The US is one of the main countries that spies on their citizens. I do wonder if Apple developed this to keep FBI happy. The same way they stopped on E2EE cloud backups to keep the FBI happy.

babyshake• 5 years ago

They need to at least have a canary clause that can be tracked to determine if they had complied with any such requests.

dimmke3• 5 years ago

I find discussions around this kind of stuff frustrating because often times what's actually happening always gets muddled by the hysteria, even here on Hacker News.

From the link: "Before an image is stored in iCloud Photos..."

This leads me to believe that only data that is going to get uploaded to their servers is going to be scanned. If anyone has a different interpretation or thinks I'm wrong, feel free to reply.

It's my understanding that all cloud services do this type of scanning, when they are technically able to.

But all data on iCloud is encrypted by default, so Apple can't scan for this kind of material once it is on their servers. Doing it on device before it gets encrypted and uploaded is the only place they even could do a scan like this.

Additionally, they make it clear in the article that there has to be more than one hit (they don't say the actual number) which would mitigate risk of hash collision false positives.

If this type of scanning makes you uncomfortable, you can just not use their cloud services.

I do agree that this is still not a good direction to go, even with all the precautions they've taken. But I had to do some digging to figure out what was actually going on, the comments/commentary made it seem like Apple is now routinely scanning all your photos/videos if you have an iPhone.

Once the code is there to do local scanning, it might make it easier for a zero day exploit to do phone scanning and grab data it might not otherwise have access to or for governments to force Apple to conduct scans of content on a phone when they ask.

erhk7• 5 years ago

This tool has nothing to do with child pornography. Youre reading the marketing paper.

AniseAbyss2• 5 years ago

You are right to worry. Once the tech is employed out in the field you literally cannot go back.

ogurechny6• 5 years ago

What makes you think that child porn is such an important problem, apart from rag articles on vague “Darknet”? Shouldn't all the phones get scanned for the evidence of tax evasion, a much more widespread crime? All those rich people dealing with offshore businesses have iPhones™, it's only natural to make Apple do their part as a honest member of society… I mean, honest corporation of society.

If you allow me to guesstimate wildly, most “child porn” these days, in a technical sense, is made by kids themselves having access to an internet-connected device with a camera. Sometimes it is extorted by despicable abusers, sometimes it is done for no one in particular, just for the perceived fame/popularity/likes on social services/etc. Big services have an army of moderating grunts to keep Victorian purity of blissfully ignorant common public intact, things are a bit different in poorer parts on the Net and on the Globe. Should we expect the naked selfie of a teenager sent to their significant other to automatically cause a police interrogation of the same teenager? What if the device is shared with older members of the family? Another man-made dystopia enabled by people who enjoy to express outrage over the racy stories in the media, and people who are too afraid to speak up.

In essence, Apple has introduced a software agent to signal whether you have files that belong the list someone provides. If I recall the scandal correctly, this is what Kaspersky allegedly (ab)used, and what other antivirus tools (including Microsoft's built-in and enabled-by-default Windows scanner that for some seemingly important reason nags all the time if uploading of files to Microsoft is disabled) surely enjoy offering to various agencies around the world.

I don't think you should worry about “China, Australia, and Saudi Arabia” so much, there's elephant in the room you don't like to notice.

Zababa• 5 years ago

I'm always a bit wary about child porn arguments. For example, the Australian police ran a child porn website for 11 months https://www.vg.no/spesial/2017/undercover-darkweb/?lang=en. A child protection agency spread pictures of child pornography http://saucenao.blogspot.com/2021/04/recent-events.html. At some point, I start to wonder about the intentions of the people behind these.

antpol• 5 years ago

forget about authoritarian regimes committing genocide, think about the children sharing pepe memes with ok signs. There are lots of pitchforks ready to go, and some of them already work withing Apple campus who are writing design docs on how to make Apple ecosystem even more safe

jimbob45• 5 years ago

Based on the press release, I don't think you're right. It doesn't appear to be all photos, just ones uploaded to iCloud.

It looks like it only triggers when you go to upload a photo to iCloud. In that case, it (maybe?) scans only that photo on your device and then decides if it's CSAM.

I guess Apple really really doesn't want CSAM on their servers? I don't know why they wouldn't just perform the scanning on their own server otherwise.

zionic4• 5 years ago

Apple’s press release is a lie.

iCloud stuff already isn’t E2E and they already scan.

This system is built exactly to surveil the entire contents of your mobile device, cloud enabled or not.

Apple has burned years of social capital in 2 days.

MisterTea• 5 years ago

I was hoping that their bowing to China with iCloud wasn't a sign of what's to come, but apparently it was.

It's likely that this tech originated from Apples capitulation to China and has been in place. So it's probably been in place for quite some time.

voldacar2• 5 years ago

These processes seem to only move in one direction. In 5 years this exact comment will probably make you sound like an "extremist" if you say it to a random person. "What, why wouldn't you be okay with apple searching through your photos?"

I see it everywhere and it literally seems like some kind of one-way entropic process. I can't think of anything that would reverse it. It would be like turning an omelet into an egg.

Is there something about modern life that just inescapably creates this complacent, servile temperament in most of the population? Or has it always been there and I'm just overthinking it? It's really depressing either way so I try not to think about it

throwaway0a5e• 5 years ago

I see it everywhere and it literally seems like some kind of one-way entropic process. I can't think of anything that would reverse it. It would be like turning an omelet into an egg.

Thomas Jefferson had some arboriculture advice that seems relevant to this kind of thing.

bonestamp2• 5 years ago

Is there something about modern life that just inescapably creates this complacent, servile temperament in most of the population? Or has it always been there and I'm just overthinking it?

I only have anecdotal evidence, but it seems like most people don't care about problems unless they affect themselves personally right now. Also, most of us (HN aside) don't tend to think seriously about how things will go wrong, we're generally optimists that give more weight to the good something will do.

0x000000001• 5 years ago

They're already scanning your photos to classify the images. That's how you can search for photos by descriptions. They also can OCR the text in images. Seems like a weird time to start getting angry about your privacy being invaded

akmarinov• 5 years ago

If only iCloud backups were encrypted…

Also this is US only? What about other children?

diebeforei4856• 5 years ago

The US has a rigid structure in place (NCMEC, etc), which other countries don't. Also, they have to start somewhere, and they understand the landscape of their home market best.

The US also has stronger laws so this is also about protecting Apple.

DaniloDias• 5 years ago

If you have young children, better not take pics of them, lest apple inc accuses you of being a predator.

tptacek5• 5 years ago

They got Mihir Bellare to review (and write a proof for) the private set intersection cryptography pieces of this.

https://www.apple.com/child-safety/pdf/Technical_Assessment_...

Apple's official proof was done in part by Dan Boneh:

https://www.apple.com/child-safety/pdf/Apple_PSI_System_Secu...

skinkestek10• 5 years ago

I think Moxie came up with the end-to-end encryption technology that WhatsApp and Signal uses.

It is great.

But it didn't prevent Facebook from trying to abuse WhatsApp users in every conceivable other way including some rather innovative ones as seen recently.

Also it didn't prevent Signal from releasing a desktop client with a really nasty XSS bug and the phone client hasn't exactly been without faults either.

You and Moxie are both much smarter than me but I foresaw and maybe even predicted (possibly so early it is under another handle) some of Facebooks actions while everyone else was just talking about how E2E-encryption was the big difference.

This feels more or less the same: the intent is honorable, the core code is clever and almost unbreakable - but you cannot really trust all of the actors you need to trust for the system to work.

We cannot read WhatsApp messages in transit. And, as specified this system sounds good. But WhatsApp happily uploads unencrypted copies to Google Cloud and/or iCloud without your consent if another participant enable backups, and I'm fairly sure we'll see this system containing huge opportunities for abuse by power hungry regimes as well as one or two nasty bugs that has the possibility to utterly destroy a few innocent persons lives.

tptacek5• 5 years ago

I'm not sure I really follow this (Signal Protocol is Trevor and Moxie; in particular, I think the triple-DH AKE is Trevor's invention) --- but I'm just here to point out the interesting cryptography angle. I wouldn't touch the policy argument here with a 10 foot pole.

pranau• 5 years ago

So what? When you don’t trust the actors running the system, it doesn’t matter if the crypto is perfect.

And right now, I don’t trust the maintainers of the CSAM database and Apple to do the right thing.

tptacek5• 5 years ago

So nothing? It's interesting that Bellare and Boneh are involved. That's the extent of my commentary.

zionic4• 5 years ago

Even then, say you trusted Apple.

Nobody signed up/purchased an iPhone trusting random 3rd parties/CSAM aggregators.

drenvuk4• 5 years ago

Thank you for the purely informational post and I'm currently reading through the second just to understand this more thoroughly but I can say already screw the proofs. Apple needs to stay off my devices with their crime scanning bs. It doesn't matter if this whole scheme ensures that Apple cryptographically keeps their hands clean. The effect is the same. They're on my device looking for things that can send me to jail.

These brilliant academics just don't care so long as they can publish an interesting paper I guess. Whatever, still reading.

tptacek5• 5 years ago

I don't think Bellare or Boneh are hurting for cites.

ncw962• 5 years ago

A summary of the photo scanning system:

- Only applies to photos uploaded to iCloud

- Matching against a known set of CSAM (Child Sexual Abuse Material) hashes occurs on-device (as opposed to the on-server matching done by many other providers)

- Multiple matches (unspecified threshold) are required to trigger a manual review of matched photos and potential account suspension

Telemakhos2• 5 years ago

You should reread the first section of TFA, titled "Communication safety in Messages." This goes beyond the scope of CSAM: they're scanning all Messages photos sent to or from a minor's account for any possible nudity, not just CSAM hash-matching.

eightysixfour• 5 years ago

It sure seems like this is two different techniques/technologies at work. There is a CSAM deterctor using a known database and then, it looks like, there is a separate model in messages for detecting pornographic content without a known database.

stu2b50• 5 years ago

Yeah but that just blurs the image and notifies the parents. Pretty low impact for false positives.

ncw962• 5 years ago

Sorry for the confusion — I was referring to just the CSAM hash feature that uploads results to iCloud.

There is also scanning for nudity in the Messages app, but those scans happen on-device and the photos stay on-device even if nudity is detected.

carnitas• 5 years ago

Read the article again. The on device scanning makes hashes on the local device

maratturaev• 5 years ago

Apple has been scanning iCloud for child abuse since (at least) Jan 2020 [1] [2]

[1] https://www.telegraph.co.uk/technology/2020/01/08/apple-scan... [2] https://web.archive.org/web/20200110193302/https://www.apple...

alpb• 5 years ago

Alternative title: Security researchers voice privacy concerns over Apple's new plan to scan users' data for child abuse images, saying governments will likely increase its scope https://www.ft.com/content/14440f81-d405-452f-97e2-a81458f54...

jszymborski3• 5 years ago

This will probably result in me ditching the iPhone next upgrade.

netr0ute6• 5 years ago

This makes me never get an iPhone until they remove this malware infection that they themselves installed.

TravisHusky2• 5 years ago

Wow, I am surprised Apple is taking this route. It's not like iCloud was a haven for sharing illegal content. I love the security features of iOS but honestly this may have pushed me to move to an Android device running GrapheneOS.

Honestly, any time there is a new policy to "protect children" it is almost always incredibly invasive and it always feels like there is some other motive and "protecting children" is used to scare anyone who tries to question it.

brightball2• 5 years ago

After reading the post and as a parent of two kids who are in middle school now...I'm pretty happy with what I see. I didn't expect to be based on the comments I read here before reading the article though.

I know a local family who has a daughter who's been in therapy for the last 3 years because she fell victim to the type of thing Apple is discussing in this post. They are firmly advocates for better parent education and oversight, sharing their experience so that other people can hopefully never have to deal with the same thing. They told us about an app called Bark[1] that's supposed to really help with a lot of this stuff and seems inline with what Apple is talking about here. I'm pretty happy to see it will be built in.

The Messages app will use on-device machine learning to warn about sensitive content, while keeping private communications unreadable by Apple.

All the parental controls in the world don't prevent the fact that getting your kids a phone in this day and age is a pretty terrifying experience if you know what type of things are out there.

When receiving this type of content, the photo will be blurred and the child will be warned, presented with helpful resources, and reassured it is okay if they do not want to view this photo. As an additional precaution, the child can also be told that, to make sure they are safe, their parents will get a message if they do view it. Similar protections are available if a child attempts to send sexually explicit photos. The child will be warned before the photo is sent, and the parents can receive a message if the child chooses to send it.

1 - https://www.bark.us/

webdood902• 5 years ago

getting your kids a phone in this day and age is a pretty terrifying experience

there is a really easy solution to this problem

brightball2• 5 years ago

Oh I know. I'm the parent who's kids constantly complain "everybody else has one" and keeps holding out.

threeseed• 5 years ago

There is nothing easy about it.

a) Many places are still going in/out of lockdown or are schooling remotely and so their phone is the only way for them to communicate with their friends. Depriving them of social contact is incredibly unhealthy and harms their development.

b) For better or worse apps like Tiktok are a huge part of their culture and the popular dances etc are often known by everyone. Being the only child who is out of the loop can cause serious isolation.

Children are growing and making them not feel like they are part of a social group is incredibly harmful and can have permanent effects in adulthood. Giving them a phone but monitoring their activities is likely to be the least harmful approach.

Zelphyr3• 5 years ago

My kids' school recommends Bark but, after looking into it, I felt I couldn't trust it. Unlike Apple, Bark appears to (at the time I reviewed them) transmit nearly everything the kid does on the phone to the Bark servers for review. That's a pretty terrible way of solving this problem, in my opinion.

zug_zug4• 5 years ago

I think the account-owner being able to turn-on or turn-off that type of capability is generally okay (though god knows if I was a 17 year old I'd certainly be switching to Droid for that reason alone).

It sounds like there's a second, unrelated thing going on which cannot be turned-off that reports images to apple that set off their warnings. The large concern there is that this type of techology could obviously someday be used to say report/delete all photos of police brutality. Since 20% of the world alone lives in China, I think the question of "How do we ensure a malicious authorities cannot use this technology against good users" is not really an afterthought but must be addressed head-on before people will buy-in.

r00fus4• 5 years ago

This iMessage feature is good, it's the hashing all of your photos for comparison against CSAM hashes that's the problem. How do you know if you've tripped the wire? What about false positives?

Given how free US Law enforcement is with violence, any potential threat of involvement with them makes me very very nervous.

Going to need to reconsider hosting my photos with Apple devices.

crackercrews• 5 years ago

Is the blurred-photo feature available to adults as well? I can imagine that some people who get unsolicited photos sent to them might want it.

Also, it looks like the blurring feature is limited to the Messages app. That's pretty easy to work around.

morpheuskafka• 5 years ago

The Messages app auto-blurring seems useful and respectful of the user, which is nice.

Additionally, the client side scanning seems very well-designed, but if iCloud Photos are not end-to-end encrypted, why are they going to such an effort to do this when they already have access to any image they want server-side?

ducadveritatem5• 5 years ago

The explanation is complicated but really fascinating. I think I understand it, but not well enough to explain it. Read the section entitled "What cryptographic tools are used in the implementation of the system?" in this write up about Apple's methodology.

https://www.apple.com/child-safety/pdf/Technical_Assessment_...

Also look at their full whitepaper here: https://www.apple.com/child-safety/pdf/CSAM_Detection_Techni...

My take is that performing the initial hash matching and encrypting the results in two separate layers on device prevents Apple from having meaningful knowledge of low (under the set threshold for being flagged) quantities of matches on a user account. This protects the use of the threshold as a way to further reduce false positives. For example they couldn't comply with a subpoena that said "Hey, we know you set a threshold of only flagging + reporting accounts with 50 image matches, but we want to see a list of all accounts with 10 or more matches because we think that's good enough."

This method lets them set and enforce a threshold to maintain their target false positive rate which they say is ~1 in 1 trillion accounts incorrectly flagged.

Disclaimer: I'm not a cryptographer and could be misunderstanding this.

walkedaway• 5 years ago

My kids are older teenagers now, but I wish Apple would have had some of this 5-8 years ago. Good on Apple for investing in helping real world problems and issues instead of investing in silencing opinions they disagree with in the name of "misinformation."

breadsticks3726• 5 years ago

As a young adult I have to ask this - you guys really fucked up this hard on defending the privacy landscape? How long do I have until a mini space hitler in my pocket that censors and snitches on my activities?

Your actions or lack thereof will have grave consequences for billions of people across generations to come.

cultofmetatron2• 5 years ago

damn, I was thinking of getting an iphone. Now I'm thinking of ditching the entire apple ecosystem altogether.

The idea that apple can scan files on my system without my consent is pretty sickening. I don't care what its purportedly used for. This is a slippery slope to all sorts of privacy violations.

oh_sigh2• 5 years ago

You're tacitly giving consent by using their device and services.

tick_tock_tick• 5 years ago

Well they are now reject that consent and never getting an iPhone.

rabuse• 5 years ago

Since when did we start allowing companies to own the actual products we purchased?

kyralis4• 5 years ago

Don't use iCloud photos and no scanning occurs. This is a condition of the service, not of the device.

Syonyk7• 5 years ago

For now. The technology to do on-device scanning will exist.

zionic4• 5 years ago

Apple is putting the entire database on every phone to scan and promising to only do it for certain situations. They can alter that deal at any time, you’d have to be a fool to think that isn’t their plan.

On-device scanning of “encrypted” cloud content where Apple has always held the key makes 0 sense unless your plan is total device surveillance.

diebeforei4856• 5 years ago

The first item (on-device nudity detection in conversations) is good and deserves some applause. It's also easy to implement, they could just take an off-the-shelf API and optimize the parameters to their needs, and has minimal privacy concerns because nothing is being reported to Apple. I'm honestly surprised this isn't common already.

It's the part about scanning people's photo libraries that folks are (rightly) concerned about.

RIMR2• 5 years ago

It's absolutely a noble cause, but if Apple is able to scan your iCloud content to figure this stuff out, then your iCloud content isn't secure.

variant• 5 years ago

And how often to noble causes stay noble?

userbinator3• 5 years ago

"The road to hell is paved with good intentions."

The gradual but steadily accelerating rise of authoritarianism scares me far more than terrorists, drugs, child abuse, and the pandemic.

Unless we push back mightily, it will be a question of when, not if, owning a general-purpose computer that's not controlled by the government or a company becomes discouraged, suspicious, and eventually illegal.

hypertele-Xii2• 5 years ago

Do you honestly think this, what I'm doing right now - posting on HN on a general-purpose computer, on an open source browser and server stack - will become limited to only licenced professionals?

EvanAnderson• 5 years ago

Yes. I absolutely do think that.

Companies holding interests in copyrighted works would love to see general purpose computers go away, replaced by "trusted" media players that can't make unauthorized copies or be used to make unsanctioned independent creative works.

Totalitarian governments would love to take away general purpose computers to prevent end users from removing surveillance and anonymity.

Companies who want to control software markets would love it if all software licensing transactions ran thru their "marketplace".

MonadIsPronad2• 5 years ago

This scares the heck out of me too. It just seems like such an obvious next step.

"Oh you run linux? Do you not trust the government, or are you trying to hide something?"

a13692099933• 5 years ago

There are no good intentions; they're just lying.

judge20202• 5 years ago

Apple is not an autonomous hive mind. The hundreds of people that built this and coordinated the deployment across multiple teams including management, deployment, QA, beta testing, machine learning, and SWE aren't lying and the majority of them certainly have good intentions.

ComputerGuru6• 5 years ago

The device creates a cryptographic safety voucher that encodes the match result along with additional encrypted data about the image.

"additional encrypted data" makes it sound like it's encrypted to keep your information safe, when it actually means "encrypted so you can't see what's being sent" and that "only Apple (and people/governments Apple shares decryption keys with) can decrypt it." And what does "additional encrypted data" even mean? I'm guessing it includes a thumbnail. It could include a list of people you shared the image with and their home addresses. The point is, you don't (and can't) know.

TheJoeMan• 5 years ago

Maybe related to totalitarian regimes like China wanting “origin tracking” which means to know who made the insert Xi meme here

aunty_helen• 5 years ago

Done with Apple now. I will never own a device that is a revolving door for law enforcement to look into my doings.

This constant yo-yo of privacy marketing and bad privacy decisions is too time consuming and mentally stressful.

I was waiting to update my 2017 MBP to a M1 16" when they come out and when my iphone 11 is coming up 2 years. These products will no longer be apple products and idgaf if I have to deal with linux desktop and jailbreaking an android to get it done.

arriu2• 5 years ago

I'm happy to see people held accountable for being scummy human beings but this does raise the question of where the line will be drawn and who gets to make that decision.

sascha_sl2• 5 years ago

This is brilliant misdirection. The crypto is great, but where and how your content is scanned was never the problem to begin with, or at least, a small part.

The accuracy of not just the detection rate (there are some outlandish claims of once in billion here), but also the accuracy of the NCMEC database are really the main concern, as well as Apple keeping this system limited to this specific scope.

Interesting aside: I once attempted to get access to PhotoDNA, essentially the only insurance against malicious actors abusing upload fields on your website to "digitally swat" you (as has happened to a twitch streamer with an open Dropbox folder), and there is no way you'll get access without a department of lawyers. Why is NCMEC is so protective of an API with rate limits and automated reporting features and then would let Apple ship a bloom filter.

hackinthebochs• 5 years ago

I remember reading at one point that perceptual hashes used in PhotoDNA could be reversed to reconstruct the image, which is why the database is closely guarded. I assume the bloom filter implementation significantly raises the bar on reversing.

sascha_sl2• 5 years ago

The basic version of PhotoDNA is an Azure API you pass all your images through directly. Even that is seemingly unobtainable.

karmicthreat2• 5 years ago

Fantastic, so is Apple going to notify me if someone is looking at the pics on my phone? No?

How about when Apple gets a NSL or other non-disclosable warrant and searches my phone with this tool? Still no?

Probably going to be a lot of leaked nudes by Apple soon.

PragmaticPulp• 5 years ago

I really don’t understand why Apple is doing this. The vast majority of their customers aren’t involved in any of these illegal activities, so it only provides potential downside through false positives.

I’m also struggling to imagine scenarios where a child predator is clever enough to acquire illegal photos without triggering any number of internet monitoring mechanisms (e.g. honeypots, server logs with their IP address) who would then turn around and upload those photos to their iCloud account. Doesn’t make sense.

This is a really strange move.

jareklupinski• 5 years ago

i'm scratching my head wondering why they bothered announcing it ahead of time?

why not just run the scans in the background... (honestly surprised they aren't already) it's not like it would be hard to omit this avenue as your lead during prosecution

Shank• 5 years ago

potential downside through false positives

I just spent the last hour of my life digging through the material. It seems like they've calibrated the system to have an expected false classification rate of one account in one trillion, per year. Based on the threshold secret sharing math, they won't know anything until a user has a significant collection of CSAM and the secret key is recovered.

I’m also struggling to imagine scenarios where a child predator is clever enough to acquire illegal photos

I don't want to be the bearer of bad news, but CSAM has been shared openly on the clearnet on places like 4chan for years. The internet is a pretty wide open search space. Many of the people who download CSAM don't do so from some sketchy underground website. This is why Tumblr, for instance, burned their platform to the ground: lots of people sharing CSAM with no way to detect and stop it (without costing Verizon a ton of money to work on the problem).

q3k• 5 years ago

I really don’t understand why Apple is doing this.

My guess is that this is a PR campaign to show that they're doing self-regulation. Perhaps related to the EU push against end-to-end encryption.

aerovistae• 5 years ago

It never occurred to me before reading this post that someday child porn will inevitably be completely vanquished. Open to contradictory views, but my thinking is that someday, eventually, AI will be advanced enough to identify an image's content unambiguously and without error, just like a human user.

No human could possibly be unable to correctly identify an image of child porn, and someday algorithms will reach that point too. And once they do....it's probably not a big leap from there for browsers and operating systems to start denying the images altogether, just blacking them out and preventing their transmission over networks.

The flipside is that it is impossible for even a human to tell the difference between a 17yo and an 18yo, and moreover impossible for a computer (and arguably for a human) to know whether the user of a device is sending images of themself (i.e. a 17yo sexting with their bf/gf) or whether it's exploitation. That's harder.

So for like this new post from apple, this is going to be pretty shitty for high schoolers trying to sext their boyfriends and girlfriends.

woofie11• 5 years ago

Goodness, I hope not!

I run GNU/Linux, and use encryption. Any mandatory technology which could scan my hard drive for child porn could scan it for:

- Antigovernment subversive materials

- Embarrassing things to blackmail me with

- Non-PC communications

- Right-wing/left-wing political materials

- And so on

I'd like to continue to have a computer I control, and I'd like my child to grow up with that freedom too. I'm much more worried about risks to my child's freedom and privacy than the (exceptionally rare but frightening) chance of exploitation for child pornography.

As a footnote, many algorithms able to correctly identify child porn would also be able to generate it.

https://en.wikipedia.org/wiki/Deepfake_pornography#DeepNude

dweekly• 5 years ago

To the latter point, a minor user transmitting sexual images of themselves may be breaking the law depending on the state and the age of the recipient, so an algorithm may not need to "special case" a "legitimate" reason for the transmission of a sexual image of a minor -- or if the algorithm did, it would need to be very complex to account for the state and ages of both sender and recipient. See e.g. Louisiana's sexting laws: https://www.criminaldefenselawyer.com/resources/teen-sexting...

randyrand4• 5 years ago

CSAM existed before computers too. Photographs can produced without computers even involved!

Jaxkr• 5 years ago

This is incredibly disappointing. The sick criminals that run child pornography rings are not storing their material on iCloud.

The "This could be sensitive to view" screen is downright Orwellian. This technology could be used to scan for ANYTHING, completely undermining user privacy. It might just be CP today, but tomorrow it could be screenshots of protest material, whistleblower content, or anti-government memes.

I cannot express how sad I am Apple has decided to do this. It doesn't protect children, it won't catch any pedophiles, but it certainly WILL be misused in the future and create a chilling effect on what (politically dissident) content people are willing to store on their phones.

arriu2• 5 years ago

Lol there is no way they will be scanning for just one type of thing. That would be a waste of computation. They will tag everything and decide if they want to use it later.

Scary times; for exactly the reasons you've called out.

cyral• 5 years ago

They already tag everything, just search for any object in your photos library (wether you are on iOS or even Android with Google Photos) and it will find it.

erhk7• 5 years ago

I can't wait until photos of marvel movies get deleted and you get a warning for breaking copyright.

ComputerGuru6• 5 years ago

Remember, Apple doesn't actually control the CSAM database (NCMEC). They almost certainly don't (and wouldn't want to) even have access to a reverse mapping between hashes and original images. A/The government(s) could (yes, theoretically - this is cryptography we are purportedly talking about, the onus is on them to prove they can't) easily slip in pictures/screenshots of political materials to have your account flagged. Even if it requires manual human review on Apple's end, the government could still (and has in the past) serve them with a warrant + gag order for "false" matches in the past.

Yes, it's all "they could" but with current technical solutions already providing some measure of protection against a corrupt or malicious government from cracking down on its citizens, anything that erodes from that freedom deserves to be held to such a high standard.

rdtwo• 5 years ago

I wonder if the tiananmen square photo matches one of the Chinese hashes. I bet a bunch of lesser known uyghur photo hashes pop against the database. Is it a publicly available hash database? I wonder if it’s possible to run known controversial stuff and see if it flags?

judge20202• 5 years ago

Such images of the Tiananmen square would have been wiped off the face of social media if so - every big social media company uses PhotoDNA or similar to catch likely matches before someone in the real world sees the illegal images on their platform.

zionic4• 5 years ago

In the press release Apple says they are using hashes of hashes.

Apple has no visibility into the original image that generated the hash, so the gov can compromise the list at the source and Apple would have plausible deniability.

r00fus4• 5 years ago

THE NCMEC isn't even "government". It's just a nonprofit. Here's the guy who founded it, and his controversies:

https://en.wikipedia.org/wiki/John_Walsh_(television_host)#C...

There is literally no oversight here. It's a misfeature and will be misused.

MrBuddyCasino• 5 years ago

„ In his book Tears of Rage, Walsh openly admits being in a relationship with 16-year-old Revé when Walsh was in his early 20s and aware of the age of consent being 17 in New York.[30] Critics of the Adam Walsh Act have pointed out that, had he been convicted, Walsh himself would have been subject to sex offender registration under the law which he aggressively promoted.“

Can’t say I‘m surprised.

celeritascelery• 5 years ago

Using another technology called threshold secret sharing, the system ensures the contents of the safety vouchers cannot be interpreted by Apple unless the iCloud Photos account crosses a threshold of known CSAM content. The threshold is set to provide an extremely high level of accuracy and ensures less than a one in one trillion chance per year of incorrectly flagging a given account.

Are they saying that they physically can’t access your iCloud until the threshold is reached, or just that they “promise not to”?

slaughtr• 5 years ago

If they can ever access it, then it's always "promise not to".

samrolken• 5 years ago

It sounds like this is a feature of the encryption used. Apple will not have enough of the key to decrypt anything unless there are enough vouchers which match.

ggreer• 5 years ago

Apple always has the decryption key for your iCloud data. When law enforcement requests an account's data, Apple decrypts the data and sends it over. Apple's law enforcement guidelines[1] explain:

iCloud content may include email, stored photos, documents, contacts, calendars, bookmarks, Safari Browsing History, Maps Search History, Messages and iOS device backups. iOS device backups may include photos and videos in the Camera Roll, device settings, app data, iMessage, Business Chat, SMS, and MMS messages and voicemail. All iCloud content data stored by Apple is encrypted at the location of the server. When third-party vendors are used to store data, Apple never gives them the encryption keys. Apple retains the encryption keys in its U.S. data centers. iCloud content, as it exists in the customer’s account, may be provided in response to a search warrant issued upon a showing of probable cause, or customer consent.

This happens to thousands of accounts every year.[2]

1. See page 11: https://www.apple.com/legal/privacy/law-enforcement-guidelin...

2. https://www.apple.com/legal/transparency/us.html

throwawaysea• 5 years ago

Safetyism, particularly with children, is the vehicle that is most often used to enable bad precedent-setting policies, because it can seem virtuous and morally acceptable in that single application. Here, Apple is using a "think of the children" argument to open the door on their intrusion into customers' private data and policing/moderation of the same. Google started doing this themselves recently (https://news.ycombinator.com/item?id=23275308) and it is just as unacceptable here. I guess that's the end of me using or recommending Apple products.

lph2• 5 years ago

This is weird. Apple's own announcement only talks about hash matching, but other reporting (e.g., [0]) talks about a system called 'neuralMatch' that's doing AI on user photos. To me, the privacy implications (and chance of false positives) seems quite different. Quite a discrepancy.

[0] https://www.zerohedge.com/technology/apple-plans-monitor-all...

ComputerGuru6• 5 years ago

No, Apple's announcement talks about machine learning to power the iMessage nudity detection algorithm. Machine learning in the context of ruining someone's life (and/or landing them on a government watchlist forever) is a huge no in my book.

lph2• 5 years ago

Ah, you're right, it does go into that later.

The reporting isn't helping, with sentences like 'the company is rolling out a new machine-learning tool that will scan iPhones for images that match certain "perceptual hashes"'... is that describing a system that classifies new photos, or is it talking about comparing hashes to a known-evil set?

hughrr2• 5 years ago

While I believe there is genuine good intent behind this I can’t support apparatus that can be repurposed and can be used to cut me off from my entire digital life due to an arbitrary review process.

Not only that I do not consent to using the device I purchased for such measures.

I need a grand reset to 2005 levels of technology at this point.

protoman3000• 5 years ago

Can somebody please explain what the big issue here is? I thought this only applies to iMessage and there are many other possibilities available to communicate securely.

Zelphyr3• 5 years ago

They are also scanning iCloud documents to match photos against known child abuse photos and reporting any that they find to law enforcement as well as locking you out of your iCloud account.

kyralis4• 5 years ago

No, they're potentially reporting after some high threshold of matches. They explicitly say that individual matches do not reach that bar, although they do not actually say what "high" is other than the statistical claim.

smnrchrds3• 5 years ago

They have created and implemented the technology to scan your personal phone for specific documents from a government database and alert the authorities if matches are found. If you don't consider this an issue, you trust your government way too much.

joshstrange3• 5 years ago

That is not true, they are only scanning photos uploaded to iCloud against the database. The only on-device things being done are detecting nudity in iMessage for minors and alerting the minor/parent. Also something about Siri and detecting searches for CSAM content.

Edit: Minor correction, the scanning on your photos does happen on-device but only if you are going to upload it to iCloud.

RegnisGnaw• 5 years ago

Note that:

1) its only scanned on upload to iCloud, so if you don't upload then its not scanned

2) (per another article, https://techcrunch.com/2021/08/05/apple-icloud-photos-scanni...): Most cloud services — Dropbox, Google, and Microsoft to name a few — already scan user files for content that might violate their terms of service or be potentially illegal, like CSAM.

So you really can't opt out unless you avoid all cloud photos

jeffbee2• 5 years ago

Just because Techcrunch says it's so, does not mean it is true. My understanding of other services is they might not scan contents that are not shared. Techcrunch does not have first-hand knowledge of this topic and the other clouds are not explicitly saying they do or do not.

randyrand4• 5 years ago

These efforts will evolve and expand over time.

Hard to make any absolute claims about how it works when they admit it will be changing.

wayneftw• 5 years ago

If those other services do it on their own computers I'm fine with that. I never expected privacy from them. I don't want my CPU, battery time and network bandwidth to be used for this.

yellow_lead• 5 years ago

It's kind of funny they announce this on the first day of Defcon. By next year's conference, hackers will have had a field day with this.

thih9• 5 years ago

I like that these features are being introduced with something similar to a press release and are being discussed in the open, as well as from the point of view of privacy.

It all may still end up being a slippery slope; but with this perhaps the chance is slightly lower (than if Apple introduced these changes without as much of a comment).

matrix• 5 years ago

This feature is a big plus for parents. It's hard to appreciate how hard it is to protect kids online if you don't have kids yourself. I get that those without kids will find it intrusive, but it sounds like these feature are opt-in.

I consider iOS to be the best platform for kids today. That said, Apple: if you're listening: please tighten up parental controls around time limits and re-loading apps!

karakot• 5 years ago

I have another idea to protect kids, let's say a kid is moving on a highway and it's speed according to a phone gps is 66mph in 60mph zone, wouldn't be it wise to notify the police about his parents braking a law and endangering a child? There are so many kids dying every year on our roads.[/s]

bdibs2• 5 years ago

Doesn’t this ease the worries from yesterday’s thread? They’ve taken steps to minimize false positives (one in a trillion accounts per year), and it’s not happening on private photos like rumored, only photos that are synced to iCloud (which is what they were doing server side already (?)). I don’t buy the slippery slope argument personally, if you want to get up in arms when they scan for politically motivated content, do so when they actually do that.

randyrand4• 5 years ago

These efforts will evolve and expand over time.

Increases the worry for me. I want to be invest in a company that cares for property rights.

bdibs2• 5 years ago

I took that to mean further protections for children, not expanding the overall scanning. Also isn't it you not caring about their property rights? Should Apple be forced to host CSAM on their servers? Should all businesses be required to?

Zelphyr3• 5 years ago

To everyone angry at this change, please take a moment to directly let Apple know how you feel. https://www.apple.com/feedback/

jpxw• 5 years ago

First off, I don’t think this is some evil plan to kill our privacy. I think this project is done with good intentions, if nothing else.

However I think this is an interesting question: how does Apple know that the hashes they’re supplied match CSAM, and not, say, anti-government material? How would they know if the people they got hashes from started supplying anti-government hashes? Apple will only be receiving the hashes here - by design, even they won’t have access to the underlying content to verify what the hashes are for.

obblekk• 5 years ago

I'll take an unpopular stance. This is the least bad way a backdoor scan could be implemented.

At the very least, this makes retroactive scans of a newly banned content difficult without explicit deployment by Apple, and explicit software update from a user. In Western countries, this friction is enough to ensure people will be able to file suit in courts to block the government.

In authoritarian governments, it's not, but then, nothing is.

wyager5• 5 years ago

If a user feels their account has been mistakenly flagged they can file an appeal to have their account reinstated.

Oh, joy, if a robotic bureaucrat nukes my personal data, I have the option to file an appeal that will probably never be looked at.

tobyjsullivan• 5 years ago

Apple has been pro-encryption and pro-privacy lately and is using that as their key differentiator, especially against Google.

Various governments, on the other hand, have been very anti-encryption over the past couple decades. Australia's anti-encryption law[0] is just one example but I'm sure there are many others.

This presents an ongoing threat to Apple's current strategy.

The most common arguments used by governments to justify anti-encryption legislation are child protection and anti-terrorism.

I wonder if this is a tactic by Apple to undermine that common argument and pro-actively protect their rights to employ end-to-end encryption and other privacy features.

[0] https://fee.org/articles/australia-s-unprecedented-encryptio...

diebeforei4856• 5 years ago

Apple's software is quite buggy. There have been multiple instances of Unicode bugs for example, where receiving a notification or text message containing certain Unicode characters would cause a kernel panic, boot loop, or other fun stuff.

People often posted messages containing these Unicode characters in Discord groups (tagging @everyone) for the lulz, because every iPhone user in that group would get a notification containing those unicode characters and then kernel panic.

It's only a matter of time until someone finds a bunch of false positives, spams them around for the lulz, and boom people's iCloud accounts are disabled.

Even if you believe the theory is mathematically sound, the implementation need not be.

breck• 5 years ago

"developed in collaboration with child safety experts"

Prove that. Release all git commits and emails and communication and who was involved in these features. I don't see any reason they would not do that, if this was about "child safety".

My guess is a portion of these "child safety experts" will have emails ending in "nsa.gov".

teekert• 5 years ago

If I understand correctly pictures of my naked children never match any known hashes so I should be ok (1 in a trillion ok that is? What is the expectation value for this?)? Not that I make such pictures but perhaps by accident. It’s not that I find it weird but god I’m afraid they end up beyond my control.

I don’t use iCloud (only for MS MFA backups), is this affecting me?

I self host everything, what happens when my account is suspended, does my iPhone work?

I sure hope we get to learn how effective this has been in say, a year? It’s quite something that my device is going to scan and check my pictures. I’d really like there to be a large, confirmed benefit for the children.

diebeforei4856• 5 years ago

1 in a trillion is not particularly impressive. Aren't there a billion iPhones? And probably 10,000 pictures on each phone on average (people take lots of selfies).

thw0rted6• 5 years ago

Well, since Talking About Statistics Is Hard, you have to look at the exact phrasing from the website: it "ensures less than a one in one trillion chance per year of incorrectly flagging a given account". So, each of those billion accounts has a 1:1T chance of false positive. If I remember my stat 101 correctly, that should translate to a 1:1000 chance of having at least 1 false positive on the planet during any given year. (And remember, even a false positive just means that a human reviews your photos, not that you get reported to the police.)

threatofrain• 5 years ago

The Messages app will add new tools to warn children and their parents when receiving or sending sexually explicit photos.
Messages uses on-device machine learning to analyze image attachments and determine if a photo is sexually explicit. The feature is designed so that Apple does not get access to the messages.

malwarebytess2• 5 years ago

Woah. I definitely don't want this. I don't want apple trolling through my data.

Terrifying. This will be used in other ways. Whenever you hear "protect the children" as an excuse to increase surveillance you know they're up to something horrible. The future is bleak AF.

nomel3• 5 years ago

Google has been doing this with their cloud services since at least 2014 (from a quick google search) and Facebook at least 2011. From talking with a founder of a popular image service, this was a requirement a decade ago, for liability reasons.

malwarebytess2• 5 years ago

I don't think you quite understand what Apple is doing.

This is absolutely going to lead to deeper use of this kind of anti-encryption tech. third party doctrine loophole to the 4th amendment -- pretty soon every document on your PC will be crosschecked for a litany of things the government deems illegal.

ogurechny6• 5 years ago

It's always been “think of the children” for three decades in IT, whether it was encryption, anonymity, or access to information. https://en.wikipedia.org/wiki/Four_Horsemen_of_the_Infocalyp...

Almost every government blocking system started for the sake of the children, then expanded for this or that reason. In some cases, it expanded before starting: notoriously, first sites blocked when Russian government censorship was introduced (to protect our kids from the dangers of the internet!) were well known media entities that had criticized Putin for a long time. The zest was that they were hurriedly blocked even before the relevant acts came into effect. Like, why keep the act after the deed is done? So now there is a government agency that has been blocking oh-so-edgy pictures of nooses on DeviantArt and similar content for years to pretend that they care about the children, and not execute censorship (and sell their extortion services to people who want to remove some information from the Web on the side). Whether that helped any kid is an open question.

DavideNL• 5 years ago

I think Apple has good intentions. However history shows the next step will be detecting less serious types of abuse, crime, etc. This system will expand slowly, step by step, until it is normalized again.

A sad say for privacy and freedom.

Obviously any child porn owners will stop using Apple devices, while the rest of us suffer the consequences.

I always wonder what would happen if Tim Cook will retire one day and be replaced by someone like Donald Trump. Once all the tools are in place...

olliej• 5 years ago

Can't wait for this to be extended for all other "criminal" content. In many countries that includes LGBT, pro-democracy, etc materials.

In the US weed is illegal at a federal level, so we can get that as well.

Woo!

And does anyone think this will actually catch any fucking pedos?

ryukafalz• 5 years ago

So my concern about this has less to do with the technology itself, and more to do with...

What happens when lawmakers make this mandatory? In such a way where the user can't be allowed to remove it?

If that ever happens, you've just outlawed phones/computers that allow their users to have ultimate control over the software running on them. No more alternative desktop/mobile OSes.

Oops.

zug_zug4• 5 years ago

This raises some questions. Suppose my hypothetical 17-year-old daughter does send nudes to her boyfriend, and suppose I as a parent am totally okay with this. If this technology isn't using exact-file-matches but heuristics on the file, is there some world where this consensual naked photo gets sent as "evidence" to Apple/NCMEC?

"Apple then manually reviews each report to confirm there is a match, disables the user’s account, and sends a report to NCMEC" -- So doesn't "reviewing reports" in this hypothetical mean looking at a child's nudes without consents in this case? The documentation left out the part where this file, stored entirely locally, gets uploaded to apple. Or does this only happen for iCloud?

shuckles2• 5 years ago

Only if your daughter’s nudes end up in NCMEC’s database, which happens if (among other things) your daughter confirms to NCMEC that they were generated through an abusive scenario and she would like them classified as such.

zug_zug4• 5 years ago

But that's using "perceptual matching" too isn't it? They say it's configured for a low risk of false-matches, though they don't detail their testing procedure. Presumably they would know almost immediately of false matches, maybe if the odds are really 1/trillion they should commit to disabling the system if any false matches happen at all.

Also, is this all configured on the server-side? Suppose Apple has a pervy employee who vouches my daughter's account "looks suspicious," would he/she then be able to override or "manually review" all her pictures? Would I be notified of any such review?

There are a lot of unaddressed questions here.

arthurcolle• 5 years ago

So can I just like disable updates right now and not get this malware when it comes out in the future, or will it be some evil OTA update?

joshstrange3• 5 years ago

These features are coming later this year in updates to iOS 15, iPadOS 15, watchOS 8, and macOS Monterey.

As long as you stay on iOS 14/iPadOS 14/watchOS 7/Big Sur then you will never get the on-device features. The photo scanning is happening on photos you upload to iCloud. Other than for malware I'm not aware of Apple ever pushing an OTA (in the sense that the user doesn't get to choose to install the update since almost all updates are technically "OTA"). Even the most recent Pegasus hack fix is in an update you can choose to install.

option_greek2• 5 years ago

I'm going to bet 100:1 that Android and Windows with Mac in tow are going to follow soon with their own implementations (though most likely technology supplied by three letter agencies) in next three years.

All that's needed is a law prohibiting unsecured computing devices by government to plugin any 'gaps' (cue: non tpm, locked down bootloader devices)

summerlight• 5 years ago

I'm not strictly against this if it can effectively help protecting children (which needs completely separate discussions of course). But Apple's usual privacy/security PR masquerade doesn't seems very compatible with this, especially with the sentence "These efforts will evolve and expand over time".

MonadIsPronad2• 5 years ago

Please, can we just be allowed to control what happens on the hardware we buy? Is that so great a demand?

aluminum96• 5 years ago

I'm going to contradict the rest of the commenters here: I strongly support this change. The ease of producing and profiting from child pornography has risen tremendously in recent years due to the advent of technology, and it's perfectly ethical for tech companies to account for their impact and attempt to mitigate it. There is no slippery slope from child porn prohibition to political censorship, as so many other commenters fear.

Mindwipe2• 5 years ago

There is no slippery slope from child porn prohibition to political censorship, as so many other commenters fear.

The UK ISPs implemented Cleanfeed, a joint database to block access to child abuse materials, many years ago. Your feelings were pretty common.

Since then Cleanfeed has expanded to censor copyright infringing sites by court diktact, and proposed legislation (which will, in some form pass) will effectively use it to block all sites not suitable for under 13 unless they implement age verification and where the guidelines can be updated at will explicitly for policial purposes by the Secretary of State.

So it is pretty hard to take your statement seriously where such slope exists and many of us are much further down it.

ogurechny6• 5 years ago

The ease of producing and profiting from child pornography has risen tremendously in recent years

Have I missed something?

Running a porn site with underage content and receiving big money from big payment processing systems stopped being an option in the early 2000s after some big busts. Then individuals would try it, and would try to use shady payment sites linked to other crimes. Then child porn, and even traces of past discussions of it, simply disappeared from open web (thanks to Google silently sugarcoating the search results, and hosting and ad businesses being increasingly picky about what they want to see). A typical person won't be able to find any of those producers (apart from obvious scams and honeypots) neither on open web nor on anonymous networks. Heck, even I, having some experience, probably won't be able to find anyone “selling child porn” even if I was told to do so and visit every possible place of discussion.

Though I agree that producing all kinds of media with modern technology got easier enormously. A dedicated handheld camera looks like a steam engine today.

erhk7• 5 years ago

Sexual predators and sex trafficking are trojan horse reasons for deeper momitoring and privacy violations. No one would argue against child pornograpy being bad and by transitive properties there is an attempt to make arguing against violations of your own privacy equally morally abhorrent.

_23sd• 5 years ago

My main question is whether the system makes it highly impractical to look for 1 to a small number (let's say 5) newly introduced images. We have the secret sharing threshold, how easy is it for the server to change this on the fly? Putting aside the principle for a moment (personally I don't like this because it's a cop in your phone, full stop), if we want to practically prevent government abuse to look for leakers (or whatever category you prefer) then it's helpful if Apple can't reasonably comply with an order to find someone who has just 1-5 specific images.

jwiley• 5 years ago

Who needs the NSO group?

Repressive regime TODO list:

1: create a child safety organization, or require an existing one to accept your images

2: add images of the children of dissidents (or journalists, or leaders of other political parties), photoshopped to be sexually explicit

3: dissident iphones informs on them. Apple turns the information over to the authorities in the host country

4: if Apple pushes back, threaten iphone sales. Or just improve your doctoring.

5: if Apple plays along or doesn't complain, insist on the ability to detect terrorists, criminals, etc. Again, threaten iphone sales, allow Apple to keep the agreement secret.

This may only work once or twice, but it's worth a shot! If you make it to step 5, you have a really bespoke, beautifully designed, Apple managed intelligence apparatus. made with love in Cupertino.

Maybe this is an overly cynical take, and in addition to the cryptography they have rock-solid, audited governance and internal controls that would prevent it and/or insider abuse.

Maybe localities with real data privacy laws (EU) will be able to offer protections to their citizens with fines big enough Apple will begrudgingly agree, so that a repressive regime can't target their citizens as well as citizens in the host country.

Maybe this isn't a slippery slope to more exotic forms of surveillance, like scanning your contact list for pedophiles.

ashleyn• 5 years ago

Upon reviewing TFA it appears this is intended for iCloud users as a privacy-focused alternative to CSAM-scanning the images you upload to iCloud on-server. Rather than leaving images unencrypted on-server so thay they can be checked, the images are checked on your own device prior to upload (to dramatically simplify it). The sell here is that CSAM checking is then done without revealing the image to Apple...unless it's CSAM, and there's enough of it.

iCloud was already scanning for CSAM, this time though they do it without seeing what you upload to the server.

This is essentially encryption with a backdoor, and the chief objection is basically the same as the Clipper chip/Skipjack. There are privacy issues with the idea of backdoored encryption, as the EFF pointed out. To counter misinformation that is being spread, it's not a little AI cop doing blanket scans on your image library and app caches - it is expressly intended as a part of iCloud CSAM detection.

But at the same time it seems trivial for Apple to expand it to that someday, since they just deployed a method capable of doing so trivially. Conceivably they can make the API so that it does a scan when an image is saved to disk. This is not currently what will be done but whether or not they will is anyone's guess.

Copernicron3• 5 years ago

This is completely unacceptable. Apple is giving themselves the ability to monitor and/or censor anything you do on your phone, no matter if it's end to end encrypted or not. It's not at all a stretch to go from monitoring iMessage for CSAM to monitoring it for gay pornography or any pornography at all. Or monitoring for anti-government sentiments.

Sure, they're only scanning for images right now. But there's nothing stopping them from scanning for other things, like Mein Kampf or the writings of Karl Marx. Or some other author who suddenly finds themselves unpopular with the government of the day.

kleene_op• 5 years ago

It's almost comical the tendency companies/societies/countries have to invoke absolutely abhorrent concepts in order to justify their future unethical behaviors and shield themselves from any valid criticism.

The fact that they raised the children card speaks volumes about the level of fuckery they're gonna deploy on this one.

wellthisisgreat• 5 years ago

Can someone who understands cryptography well enough please comment, if these hash comparisons can easily be extended to other areas such as contextual analysis of photos or texts?

For example would it be easy now to get to the hypothetical scenario where a text containing certain phrases will be flagged if some partner / regulator demands that?

bob10293• 5 years ago

I posted in this thread a little bit ago. I couldn't find anything coherent to say at the time. Now I have something to say.

I will be pushing for our current & future customers to use alternative device vendors as part of our product stack. We have developed a cross platform implementation of our iOS application recently, and this is a great opportunity to start putting it to use.

I don't know if Apple is paying attention, but there are enterprise customers out there paying them $300/yr for the permission to use their presumably-secure devices for purposes of running custom LOB applications. Our organization and product are part of the reason you have financial institutions making bulk purchases of iPads and other Apple accessories.

If Apple makes it impossible for us to provide assurances to our customers that no data is leaving the devices (aside from agreed-upon application protocols), then we have a serious problem. I am not putting myself in a position where I push a customer to purchase 1000+ iPads for their business if I will then have to answer to their auditors 6 months later on when sensitive customer documents hashes or whatever other horrible things start getting sent to the mothership as part of v2.0 of this shitty idea. Apple even said it themselves - "These efforts will evolve and expand over time.".

Is there a special order form if I need to purchase a non-bugged iPad for a classified government setting or very pissed-off enterprise customer? If not, our default recommendation is going to be Microsoft Surface and we will totally deprecate usage of Apple hardware over the course of the next 18-24 months. This is setting a completely unacceptable precedent, and most of our customers I have spoken with today also agree.

angch• 5 years ago

One new way for swatting.

Send or otherwise inject problematic images to the target, but make sure the target can't see it (image too small, part of larger collage), but their device does. The target gets flagged by Apple for having CSAM.

The target gets FBI'd.

73737373732• 5 years ago

Similar things happening in Europe: https://www.patrick-breyer.de/en/chatcontrol-european-parlia...

aparks517• 5 years ago

Does Apple have a legal or regulatory requirement to scan iCloud photos for CSAM? I poked around but didn't find a quick answer.

Reading through the two linked PDF's, I got the impression that they're aiming to use cryptographic techniques to meet a (possibly self-imposed) requirement to scan for CSAM while revoking their current ability to decrypt photos stored in iCloud. I guess they may want to revoke this ability so that they can no longer be compelled to hand over customer's photos except those for which the requirements of this new system are met.

I guess if it results in an increase of the privacy of iCloud content, that's nice as far as it goes. But it does skeeve me out to be reminded that my phone can (and does) paw through my stuff.

YeBanKo• 5 years ago

Until this I was typically defaulting to Apple products for privacy reasons. This seems like a major issue to me. This is an ideal spy tool, disguised as an privacy-preserving feature.

Any government of a country with a large enough market for Apple, can force Apple to include hashes hashes into localized versions of OS.

Expanded Protections for Children

Related Stories