iCloud Photo uploads approved: no material Thinks Different
August 8, 2021 3:28 AM Subscribe
This week, Apple announced and then explained (pdf) the measures they're taking to protect children (so: trigger warning) from grooming and to stop abusers from storing images of child sexual abuse in iCloud Photo Storage. Daring Fireball's Jon Gruber seems to have a balanced unpacking of the situation. The change is coming to iOS 15, with opt-in parental controls on messaging within family accounts and for uploads to iCloud Photo Storage. It is restricted to the USA for now -- among the questions raised are whether another state could insist that other material to be flagged for "Thinks Different" content.
The restriction to the USA reflects the CLOUD Act which arose when US federal law enforcement couldn't compel Microsoft to supply emails from a server located in Ireland. It kind-of forces USA-based companies to store data for USA people inside the USA, and so makes the legal process for compelling access much easier.
Apple avoid any need to comply with this if they disallow material to be uploaded to iCloud Photo Storage with this change -- so get to continue proclaiming the privacy of their ecosystem by virtue of "you own the device, we own the network services."
The restriction to the USA reflects the CLOUD Act which arose when US federal law enforcement couldn't compel Microsoft to supply emails from a server located in Ireland. It kind-of forces USA-based companies to store data for USA people inside the USA, and so makes the legal process for compelling access much easier.
Apple avoid any need to comply with this if they disallow material to be uploaded to iCloud Photo Storage with this change -- so get to continue proclaiming the privacy of their ecosystem by virtue of "you own the device, we own the network services."
Remember that whatever computing device you use, act like it's NOT yours and it does NOTHING to protect your privacy
unless you're running non-proprietary free software on it. Then you've got a fighting chance.
posted by flabdablet at 4:06 AM on August 8, 2021 [4 favorites]
unless you're running non-proprietary free software on it. Then you've got a fighting chance.
posted by flabdablet at 4:06 AM on August 8, 2021 [4 favorites]
In a very limited way flabdablet - there's (for example) no non-proprietary free software running most modem chips,and these things run mostly their own operating system - a computer within a computer. Heck, even the chip running the lightning connector on your iPhone is running its own small operating system, with no free alternative available. Your Intel CPU is running on a customized minix regardless of what free software you put on it...
I know I'm paranoid - I just wonder if I'm paranoid enough...
posted by DreamerFi at 4:12 AM on August 8, 2021 [10 favorites]
I know I'm paranoid - I just wonder if I'm paranoid enough...
posted by DreamerFi at 4:12 AM on August 8, 2021 [10 favorites]
prediction: the Chinese government will tell Apple to use THEIR hash database to check Chinese users and it will include a hash of tank man.
My theory was that the purpose of this whole exercise was to reassure the Chinese Communist Party that Apple (who depend on China both for manufacturing and a significant chunk of profits) is a reliable partner in maintaining Harmony in China, and did not need to be brought down to size like Tencent/Sina Weibo and other tech firms who started getting too big for their boots.
Insert other governments and relevant pictures for your favorite government
The next round of the Australian bipartisan national-security law ratchet will almost certainly mandate a database of hashes of “extremist materials”, along with draconian secrecy clauses about its content.
posted by acb at 4:16 AM on August 8, 2021 [12 favorites]
My theory was that the purpose of this whole exercise was to reassure the Chinese Communist Party that Apple (who depend on China both for manufacturing and a significant chunk of profits) is a reliable partner in maintaining Harmony in China, and did not need to be brought down to size like Tencent/Sina Weibo and other tech firms who started getting too big for their boots.
Insert other governments and relevant pictures for your favorite government
The next round of the Australian bipartisan national-security law ratchet will almost certainly mandate a database of hashes of “extremist materials”, along with draconian secrecy clauses about its content.
posted by acb at 4:16 AM on August 8, 2021 [12 favorites]
All iCloud data for Chinese users is already stored in China. It is hard to imagine a scenario where this update is what finally gives the Chinese government access.
posted by snofoam at 4:36 AM on August 8, 2021 [14 favorites]
posted by snofoam at 4:36 AM on August 8, 2021 [14 favorites]
Forgive me if I'm misunderstanding how this works, but what's to stop this automation from flagging, say, photos parents have taken of their children swimming or in the bath, or just in a general state of undress (which often happens with small kids)?
I have to say I think the questions about how this will be implemented to cut down on other "unsavoury" privately stored (but not privately owned, as discussed above) content are valid ones. You can already see echoes of this started by FOSTA/SESTA across all the big social media platforms. Upload an image, no matter what it is, that pings the anti-sex/anti-nudity algorithm to Facebook or Instagram and you'll be flagged or see your account shadowbanned or outright banned. Just in the last few weeks I've seen this happen to friends, artists (particularly queer photographers) and companies supplying packer products to trans men (because the algorithm can't be told not to flag something that looks like a dildo).
I keep wondering when we'll get to the point of being sick enough of all of this to stop it, but it's starting to feel like that won't happen. The internet will become a sterile, government-controlled, authority-mandated space all the way down to the photos you're allowed to store on your phone.
Time to invest in an analogue camera again..
posted by fight or flight at 5:12 AM on August 8, 2021 [10 favorites]
I have to say I think the questions about how this will be implemented to cut down on other "unsavoury" privately stored (but not privately owned, as discussed above) content are valid ones. You can already see echoes of this started by FOSTA/SESTA across all the big social media platforms. Upload an image, no matter what it is, that pings the anti-sex/anti-nudity algorithm to Facebook or Instagram and you'll be flagged or see your account shadowbanned or outright banned. Just in the last few weeks I've seen this happen to friends, artists (particularly queer photographers) and companies supplying packer products to trans men (because the algorithm can't be told not to flag something that looks like a dildo).
I keep wondering when we'll get to the point of being sick enough of all of this to stop it, but it's starting to feel like that won't happen. The internet will become a sterile, government-controlled, authority-mandated space all the way down to the photos you're allowed to store on your phone.
Time to invest in an analogue camera again..
posted by fight or flight at 5:12 AM on August 8, 2021 [10 favorites]
there's (for example) no non-proprietary free software running most modem chips,and these things run mostly their own operating system - a computer within a computer
There are at least green shoots. The cellular modem in the Librem 5 is a separate processor rather than being built into the main CPU and has no direct access to main memory, which renders it completely unable to interfere with end-to-end encryption performed by software running inside the free software OS.
posted by flabdablet at 5:15 AM on August 8, 2021 [7 favorites]
There are at least green shoots. The cellular modem in the Librem 5 is a separate processor rather than being built into the main CPU and has no direct access to main memory, which renders it completely unable to interfere with end-to-end encryption performed by software running inside the free software OS.
posted by flabdablet at 5:15 AM on August 8, 2021 [7 favorites]
Forgive me if I'm misunderstanding how this works, but what's to stop this automation from flagging, say, photos parents have taken of their children swimming or in the bath, or just in a general state of undress (which often happens with small kids)?
If I'm understanding it correctly, flagging of your pics would only happen if they are identical to hashed pics supplied by law enforcement. And only if you've uploaded them to iCloud.
posted by Thorzdad at 5:17 AM on August 8, 2021 [12 favorites]
If I'm understanding it correctly, flagging of your pics would only happen if they are identical to hashed pics supplied by law enforcement. And only if you've uploaded them to iCloud.
posted by Thorzdad at 5:17 AM on August 8, 2021 [12 favorites]
Yeah, it is only searching for known child sexual abuse images, only looking at images uploaded to iCloud, and there is a threshold for the number of images that triggers a response. It is designed to be narrow in scope and reduce the possibility of false positives as much as possible.
posted by snofoam at 5:25 AM on August 8, 2021 [7 favorites]
posted by snofoam at 5:25 AM on August 8, 2021 [7 favorites]
Ah, that makes a lot of sense, thanks! And I can see how the above concerns can easily become reality, irt certain non-government-approved images.
posted by fight or flight at 5:26 AM on August 8, 2021 [1 favorite]
posted by fight or flight at 5:26 AM on August 8, 2021 [1 favorite]
Forgive me if I'm misunderstanding how this works, but what's to stop this automation from flagging, say, photos parents have taken of their children swimming or in the bath, or just in a general state of undress (which often happens with small kids)?The discussion of this has been confusing because there are two separate systems: iMessage is getting a nudity detector for accounts flagged as children and iCloud Photos having a scan for CSAM. The first one is a machine-learning model and would make the kind of matches you're talking about attempting to recognize characteristics of new images which have never been reported before.
The second one is moving the CSAM scanning which already happens from the iCloud servers to the phone, and that uses a technique called “perceptual hashing”. Unlike those ML classifiers, a perceptual hash is designed to match exact images — not even separate photos of the same scene — but surviving some common manipulations (resizing, rotation, color to grayscale conversion, embedding in a frame or larger image, etc.). The idea here is that this system would only match images which have previously been reported and verified, but it does so on the device instead of when it hits the servers — hopefully as a prerequisite for enabling end-to-end encryption on iCloud Photos (which does not currently have that, unlike iMessage). Even very similar images do not trigger these matches — I've used more primitive versions of these tools before in a library context and the challenge was always dealing with images which had been modified enough to no longer match at a remotely usable degree.
Since the system is moving a scanning stage which already happens rather than adding a new scan for images which are not currently being scanned, the kind of scenarios DreamerFi and acb outlines don't apply yet. The critical question is how anyone would know whether something causes that to change and start scanning even images which are not being uploaded to iCloud, which would open up all of the scenarios where the men in black show up and tell Apple that they need to know which iPhones have images matching these hashes which, we triple-pinky-promise, are associated with the most heinous of crimes.
posted by adamsc at 5:32 AM on August 8, 2021 [19 favorites]
To be fair, triple pinky promises are really strong promises.
posted by flabdablet at 5:48 AM on August 8, 2021 [14 favorites]
posted by flabdablet at 5:48 AM on August 8, 2021 [14 favorites]
Not to abuse the exit window, I think this is a great example of Apple’s culture hurting them. This is an important problem to address — anyone building systems which move digital media for the public knows that those systems will be used to facilitate abuse on a scale which would give anyone pause — but doing it in the dark and then just announcing it without comment is exactly the opposite of how you’d build trust that a tool would only be used for it’s stated purpose.
These systems are inherently scary because they have to operate without informing the user (otherwise you’ve just built a tool to help abusers process their collections until they no longer match) and the nature of the material means they can’t be publicly audited. For most other types of content, you could use an approach like Certificate Transparency where all additions to the database are published in a public list for scrutiny but obviously you can’t publish the source images so it’s on the honor system that a group like NCMEC has confirmed the images are what’s claimed, and it’s not even easy to verify that the image hashes are just, say, NCMEC’s database and don’t contain, say, the Chinese government’s tank man hashes.
To expand the above, consider what happens if either the database is leaked or the system alerts the user: it turns into what we’ve already seen with antivirus signatures where someone malicious uses a disposable system to see which of the files they’re about to distribute trips an alert and they can either remove or modify that file until it stops happening. Apple’s design tries some fancy blinding techniques and doesn’t trigger an alert on the device to prevent that, which is good for the stated goal but by necessity stymies outside auditing. Having auditors reviewing alerts at Apple could change that but that raises the question of who and how you’d maintain enough people willing to cope with the trauma of seeing abuse imagery on a regular basis just to confirm that there aren’t abuses of power hidden in all of the accurate alerts.
Alex Stamos, formerly of Facebook and now at Stanford, has a long Twitter thread worth reading, especially this part:
These systems are inherently scary because they have to operate without informing the user (otherwise you’ve just built a tool to help abusers process their collections until they no longer match) and the nature of the material means they can’t be publicly audited. For most other types of content, you could use an approach like Certificate Transparency where all additions to the database are published in a public list for scrutiny but obviously you can’t publish the source images so it’s on the honor system that a group like NCMEC has confirmed the images are what’s claimed, and it’s not even easy to verify that the image hashes are just, say, NCMEC’s database and don’t contain, say, the Chinese government’s tank man hashes.
To expand the above, consider what happens if either the database is leaked or the system alerts the user: it turns into what we’ve already seen with antivirus signatures where someone malicious uses a disposable system to see which of the files they’re about to distribute trips an alert and they can either remove or modify that file until it stops happening. Apple’s design tries some fancy blinding techniques and doesn’t trigger an alert on the device to prevent that, which is good for the stated goal but by necessity stymies outside auditing. Having auditors reviewing alerts at Apple could change that but that raises the question of who and how you’d maintain enough people willing to cope with the trauma of seeing abuse imagery on a regular basis just to confirm that there aren’t abuses of power hidden in all of the accurate alerts.
Alex Stamos, formerly of Facebook and now at Stanford, has a long Twitter thread worth reading, especially this part:
Apple was invited but declined to participate in these discussions, and with this announcement they just busted into the balancing debate and pushed everybody into the furthest corners with no public consultation or debate.posted by adamsc at 5:58 AM on August 8, 2021 [13 favorites]
I have friends at both the EFF and NCMEC, and I am disappointed with both NGOs at the moment. Their public/leaked statements leave very little room for conversation, and Apple's public move has pushed them to advocate for their equities to the extreme.
While I don’t care to have Apple acting as an arm of Big Brother (no matter what their good intentions), if the government wants to act corruptly there are many many avenues it can take before abusing this system makes sense.
So, an annoying reminder that the Apple Terms and Conditions basically give them carte blanche, but a fairly minor risk as far as government corruption is concerned.
posted by Tell Me No Lies at 6:00 AM on August 8, 2021 [1 favorite]
So, an annoying reminder that the Apple Terms and Conditions basically give them carte blanche, but a fairly minor risk as far as government corruption is concerned.
posted by Tell Me No Lies at 6:00 AM on August 8, 2021 [1 favorite]
How likely is it that there will be accidental or deliberate collisions with those hashes?
posted by Joe in Australia at 6:28 AM on August 8, 2021
posted by Joe in Australia at 6:28 AM on August 8, 2021
Louis Rossmann and Daniel Smullen discuss Apple, hashing, & privacy (YouTube, 41m15s)
posted by flabdablet at 6:29 AM on August 8, 2021
posted by flabdablet at 6:29 AM on August 8, 2021
My favorite part of this, as someone who has never had any illusions about Apple is their internal memo dismissing concern of the "screeching voices of the minority". Think different.
posted by joeyh at 7:01 AM on August 8, 2021 [17 favorites]
posted by joeyh at 7:01 AM on August 8, 2021 [17 favorites]
Unpopular Opinion: I know all the issues with this, but I don't really have that much of a problem with Apple doing it. I think, realistically, this kind of data forensics to prevent crime are a fair thing for cloud services because you're not doing this on your hardware you are saving data on the company's hardware.
So to be fair to Apple, Google, Microsoft, and others who use this tech (let's be clear, the only difference is that Apple is applying it to encrypted communications. Google and MS already do this to all their users and have for years. Hell, Microsoft helped develop PhotoDNA) if someone is uploading child pornography to your servers, they have technically made you (the company) technically complicit in the crime unless you actually do something about it. This, this is doing something about it.
I'm not here to rehash all the very realistic, thoughtful, and scary arguments against it in this thread. I understand those, but I do think that capturing and stopping abusers is important. I know our justice system sucks, but so does letting abusers run roughshod over people's lives.
Also, yes, Apple are hypocritical jerks and they chose the wrong hill to die on, privacy, when they had no intent of taking it seriously at any point. They're just hoping most people are too non-tech-savvy to know any better.
posted by deadaluspark at 7:05 AM on August 8, 2021 [11 favorites]
So to be fair to Apple, Google, Microsoft, and others who use this tech (let's be clear, the only difference is that Apple is applying it to encrypted communications. Google and MS already do this to all their users and have for years. Hell, Microsoft helped develop PhotoDNA) if someone is uploading child pornography to your servers, they have technically made you (the company) technically complicit in the crime unless you actually do something about it. This, this is doing something about it.
I'm not here to rehash all the very realistic, thoughtful, and scary arguments against it in this thread. I understand those, but I do think that capturing and stopping abusers is important. I know our justice system sucks, but so does letting abusers run roughshod over people's lives.
Also, yes, Apple are hypocritical jerks and they chose the wrong hill to die on, privacy, when they had no intent of taking it seriously at any point. They're just hoping most people are too non-tech-savvy to know any better.
posted by deadaluspark at 7:05 AM on August 8, 2021 [11 favorites]
I think, realistically, this kind of data forensics to prevent crime are a fair thing for cloud services because you're not doing this on your hardware you are saving data on the company's hardware.
Except that they're going to do it locally now. On what was so far your device. Which means, as I said, that it's not really your hardware any more.
posted by DreamerFi at 7:09 AM on August 8, 2021 [5 favorites]
Except that they're going to do it locally now. On what was so far your device. Which means, as I said, that it's not really your hardware any more.
posted by DreamerFi at 7:09 AM on August 8, 2021 [5 favorites]
Kind of interested to know if this might pave the way for Apple to enable end to end encryption of iCloud accounts. If the justification for local hash-matching satisfies the law enforcement sector (enough that they can’t decry ‘but what of the children!’) then it /could/ allow Apple to finally turn on encryption for all data they store in iCloud.
I’d personally be happy with that trade off, and from Apples POV I can see it reducing their liability (what if someone really hacks iCloud badly) and of course the privacy marketing win.
Remember, at the moment they can scan every image, doc, everything that gets stored in the iCloud (which if you’re an average user— is everything) and of course hand that data off to any gov that asks.
posted by Static Vagabond at 7:13 AM on August 8, 2021 [2 favorites]
I’d personally be happy with that trade off, and from Apples POV I can see it reducing their liability (what if someone really hacks iCloud badly) and of course the privacy marketing win.
Remember, at the moment they can scan every image, doc, everything that gets stored in the iCloud (which if you’re an average user— is everything) and of course hand that data off to any gov that asks.
posted by Static Vagabond at 7:13 AM on August 8, 2021 [2 favorites]
Which means, as I said, that it's not really your hardware any more.
It never really was to begin with, but especially when you're working with a company like Apple, who want complete control over their hardware and software ecosystem.
Sure, Android has the same problems, but I can pop on a more secure OS if I really want to, like CopperheadOS, with all connections to Google stripped out.
If I'm worried about them scanning my chats I can set up and run my own Matrix server and run an instance of Synapse and still be able to communicate with other people in the Matrix federation without leaking tons of metadata about myself.
If I want a more secure email those are available, too. Self-hosting and federated services are where it's at.
I can block whole domains from being able to be pulled up on my network with a Pi-Hole.
These are off-the-shelf components, you can do even more with industry components.
But yeah, you can't escape being tracked in some way. MAC addresses are unique and baked into the hardware of every internet enabled device. *shrugs
Once again, yeah, stupid hill for Apple to die on, but it was expected by anyone who could tell all their messaging about "privacy" was just PR hogwash.
posted by deadaluspark at 7:15 AM on August 8, 2021 [12 favorites]
It never really was to begin with, but especially when you're working with a company like Apple, who want complete control over their hardware and software ecosystem.
Sure, Android has the same problems, but I can pop on a more secure OS if I really want to, like CopperheadOS, with all connections to Google stripped out.
If I'm worried about them scanning my chats I can set up and run my own Matrix server and run an instance of Synapse and still be able to communicate with other people in the Matrix federation without leaking tons of metadata about myself.
If I want a more secure email those are available, too. Self-hosting and federated services are where it's at.
I can block whole domains from being able to be pulled up on my network with a Pi-Hole.
These are off-the-shelf components, you can do even more with industry components.
But yeah, you can't escape being tracked in some way. MAC addresses are unique and baked into the hardware of every internet enabled device. *shrugs
Once again, yeah, stupid hill for Apple to die on, but it was expected by anyone who could tell all their messaging about "privacy" was just PR hogwash.
posted by deadaluspark at 7:15 AM on August 8, 2021 [12 favorites]
MAC addresses are unique and baked into the hardware of every internet enabled device. *shrugs
All modern OSes will allow you to spoof the MAC address. I'd be more worried about what lurks in some of the firmware blobs.
posted by jaduncan at 7:27 AM on August 8, 2021 [11 favorites]
All modern OSes will allow you to spoof the MAC address. I'd be more worried about what lurks in some of the firmware blobs.
posted by jaduncan at 7:27 AM on August 8, 2021 [11 favorites]
How likely is it that there will be accidental or deliberate collisions with those hashes?That’s one of the most important questions now: accidental collisions should be much lower risk than what you see with things like ML classifiers because these aren’t trying to match previously unseen images but the fallout could be quite high.
Deliberate collisions are more worrisome: they try to make that process harder but it could be ugly if someone found a way to make relatively normal looking images and a bunch of 4chan-types started trying to trick people into saving them or abused things like the WhatsApp auto-save feature. That’s pretty unlikely but it’s one of the areas which most needs outside stress testing.
posted by adamsc at 7:36 AM on August 8, 2021
Kind of interested to know if this might pave the way for Apple to enable end to end encryption of iCloud accounts. If the justification for local hash-matching satisfies the law enforcement sector (enough that they can’t decry ‘but what of the children!’) then it /could/ allow Apple to finally turn on encryption for all data they store in iCloud.
I kinda wonder why this wasn't announced at the same time as announcing encryption of all iCloud data. Although I guess only a tiny fraction of people care about the theoretical future ramifications of this change.
I think I am in the minority of people here who don't really see this as a big problem or a big invasion of privacy. (Alternately, probably with the vast majority of people in general who don't really think it is a big deal.) You could still have this material on your iPhone. The only thing you have to do to avoid having images scanned is not use iCloud Photo Library. I already don't use that.
posted by snofoam at 7:53 AM on August 8, 2021 [1 favorite]
I kinda wonder why this wasn't announced at the same time as announcing encryption of all iCloud data. Although I guess only a tiny fraction of people care about the theoretical future ramifications of this change.
I think I am in the minority of people here who don't really see this as a big problem or a big invasion of privacy. (Alternately, probably with the vast majority of people in general who don't really think it is a big deal.) You could still have this material on your iPhone. The only thing you have to do to avoid having images scanned is not use iCloud Photo Library. I already don't use that.
posted by snofoam at 7:53 AM on August 8, 2021 [1 favorite]
Deliberate collisions are more worrisome: [...] That’s pretty unlikely
If the classifier is actually moved onto the phone, though, then that makes it a lot easier for people to extract the model and use it to try to generate adversarial examples (innocuous [to humans] images designed to trip the detector).
posted by Pyry at 7:57 AM on August 8, 2021 [4 favorites]
If the classifier is actually moved onto the phone, though, then that makes it a lot easier for people to extract the model and use it to try to generate adversarial examples (innocuous [to humans] images designed to trip the detector).
posted by Pyry at 7:57 AM on August 8, 2021 [4 favorites]
Echoing what snofoam said, this sort of privacy issue goes along with any cloud storage one could potentially use. Whoever hosts the servers might decide to take a peek at your data.
On a side note, “Think Different” was a marketing campaign that started in 1997. Snarkily using the phrase stopped being original sometime around a week and a half after that. Just stop.
posted by Fleebnork at 7:59 AM on August 8, 2021 [4 favorites]
On a side note, “Think Different” was a marketing campaign that started in 1997. Snarkily using the phrase stopped being original sometime around a week and a half after that. Just stop.
posted by Fleebnork at 7:59 AM on August 8, 2021 [4 favorites]
You could still have this material on your iPhone. The only thing you have to do to avoid having images scanned is not use iCloud Photo Library
and trust Apple's triple pinky promise that they're not going to enable scanning for stuff that isn't about to be backed up on any of their own services.
I agree with the point raised in the Rossman and Smullen discussion I linked above, that if Apple leads the way on making on-device content scanning acceptable to consumers then all the other vendors are going to jump on the same bandwagon pdq.
posted by flabdablet at 8:00 AM on August 8, 2021 [6 favorites]
and trust Apple's triple pinky promise that they're not going to enable scanning for stuff that isn't about to be backed up on any of their own services.
I agree with the point raised in the Rossman and Smullen discussion I linked above, that if Apple leads the way on making on-device content scanning acceptable to consumers then all the other vendors are going to jump on the same bandwagon pdq.
posted by flabdablet at 8:00 AM on August 8, 2021 [6 favorites]
this sort of privacy issue goes along with any cloud storage one could potentially use. Whoever hosts the servers might decide to take a peek at your data
unless, as is the case for e.g. Keybase, everything they store as private is encrypted at the client, using open-source client software and keys that exist only on the client.
posted by flabdablet at 8:03 AM on August 8, 2021
unless, as is the case for e.g. Keybase, everything they store as private is encrypted at the client, using open-source client software and keys that exist only on the client.
posted by flabdablet at 8:03 AM on August 8, 2021
and trust Apple's triple pinky promise that they're not going to enable scanning for stuff that isn't about to be backed up on any of their own services.
By this logic, they are already doing it, so it's not a big deal that they've finally announced it.
posted by snofoam at 8:03 AM on August 8, 2021 [4 favorites]
By this logic, they are already doing it, so it's not a big deal that they've finally announced it.
posted by snofoam at 8:03 AM on August 8, 2021 [4 favorites]
known CSAM image hashes provided by NCMEC and other child-safety organizations
Is an exhaustive list of the providers of the secret unreviewable database too much to ask? Are there so many that it would make the PDF too large? Or does this leave wiggle room for other child-safety organizations such as [F]ind [B]oys [I]mmediately, [I]mpound [C]hildren [E]veryday, etc.
Forgive my skepticism, it's just that I've only seen the privacy ratchet turn in one direction during my lifetime.
posted by RobotVoodooPower at 8:32 AM on August 8, 2021 [5 favorites]
Is an exhaustive list of the providers of the secret unreviewable database too much to ask? Are there so many that it would make the PDF too large? Or does this leave wiggle room for other child-safety organizations such as [F]ind [B]oys [I]mmediately, [I]mpound [C]hildren [E]veryday, etc.
Forgive my skepticism, it's just that I've only seen the privacy ratchet turn in one direction during my lifetime.
posted by RobotVoodooPower at 8:32 AM on August 8, 2021 [5 favorites]
Y’all, please read at least the Daring Fireball post linked in the OP. It fully answers a lot of the questions asked so far on this thread about how this works in a technical sense; how Apple will check to protect against false matches; and so on. It even includes a discussion of how this might relate to end-to-end encryption of iCloud storage. And it does not let Apple off the hook about the real and valid privacy concerns.
posted by snowmentality at 8:37 AM on August 8, 2021 [10 favorites]
posted by snowmentality at 8:37 AM on August 8, 2021 [10 favorites]
I just recently started working in the privacy tech space a few months ago and one of the big surprises so far is how terrible Apple's reputation is as an employer of actual privacy professionals. That said, I've been reading the hysteria about this announcement online all week and been honestly a bit baffled by it. I'm not at all sure what Apple could have cooked up that would be more privacy-preserving than this and still comply with US law around CSAM. I guess people read the news about them pushing back on specific subpoenas or warrants or LE requests and extrapolated that they would just refuse to obey any and all laws that don't match their marketing?
I definitely agree it was a mistake to announce the ML-based detection feature and the hashing-based detection feature at the same time. Almost all the press coverage I've seen has been very muddled about who's using ML for what, when. And the idea that ML, which is definitely not only currently incapable of distinguishing a seventeen-year-old from an eighteen-year-old on sight, but will never be able to do that except by integrating with a number of other data sources that are not even slightly in scope for this launch, could be combing through your phone and calling the FBI about an innocuous picture of your kids is very disturbing! Luckily that's not what's happening. That Daring Fireball write-up is excellent and I thank you for linking it.
posted by potrzebie at 8:40 AM on August 8, 2021 [4 favorites]
I definitely agree it was a mistake to announce the ML-based detection feature and the hashing-based detection feature at the same time. Almost all the press coverage I've seen has been very muddled about who's using ML for what, when. And the idea that ML, which is definitely not only currently incapable of distinguishing a seventeen-year-old from an eighteen-year-old on sight, but will never be able to do that except by integrating with a number of other data sources that are not even slightly in scope for this launch, could be combing through your phone and calling the FBI about an innocuous picture of your kids is very disturbing! Luckily that's not what's happening. That Daring Fireball write-up is excellent and I thank you for linking it.
posted by potrzebie at 8:40 AM on August 8, 2021 [4 favorites]
This isn’t a classic ML classifier, however, and they’re putting some effort into making it harder to know whether you even have a hit:Deliberate collisions are more worrisome: [...] That’s pretty unlikelyIf the classifier is actually moved onto the phone, though, then that makes it a lot easier for people to extract the model and use it to try to generate adversarial examples (innocuous [to humans] images designed to trip the detector).
https://www.apple.com/child-safety/pdf/CSAM_Detection_Technical_Summary.pdf#page4
That does not, of course, prevent attacks but it won’t be an easy adversarial attack given the way NeuralHash isn’t trading false-positives for the ability to recognize novel images as aggressively as most image classifiers have to. I would find it quite plausible that it may effectively be impossible to produce a matching image which isn’t quite distorted or obvious porn, avoiding the “4chan spreads toxic meme-bombs” problem, but it seems more possible that someone could potentially generate something they could use to set traps for people who are looking for legal porn.
This isn’t a particularly new problem (PhotoDNA has been in use for years) but I really wouldn’t want to be in the business of hosting user-generated content without a healthy budget to both directly combat this and minimize the impact on your staff who have to get involved with that.
The key thing is that these are really important questions to have asked and answered before deployment. If nothing else, we have plenty of evidence that police departments often have a faulty understanding or willingness to abuse the error modes of imperfect tests and that this tends to reinforce existing social fault lines around race and class.
posted by adamsc at 8:40 AM on August 8, 2021 [2 favorites]
Also, the provisos (that they're only using specific hashes of the vilest possible material, and only checking images when they're about to be copied into iCloud) are not matters of technical capability but of policy. Expanding the set of hashes would be trivially easy (and, for example, a UK web filter originally intended to block child abuse imagery was extended to block images of counterfeit watches). Similarly, and more sweepingly, once the scanner is on every device, changing it to run every time an image is sent or received, or saved to the camera roll, would be trivial.
One step beyond that is replacing the list of hashes with a neural classifier trained to recognise the combination of nudity and juvenile appearance, which is more of a step, though given how ineffective a static list would be if the threat is abusers using their iPhones to take pictures, will be demanded soon enough, and possible enough. Once this is there, the phone can continuously check camera input for likely-illegal material and flag it. Which is both a huge escalation and an obvious (and arguably desirable) next step; after all, if it saves one child, surely it would be churlish to complain about the minor inconvenience of your nudes being reviewed by an abuse department. (Also, implicitly, decent people don't take nude photographs; if you want to be a pervert, you can pay the social price.)
posted by acb at 8:44 AM on August 8, 2021 [2 favorites]
One step beyond that is replacing the list of hashes with a neural classifier trained to recognise the combination of nudity and juvenile appearance, which is more of a step, though given how ineffective a static list would be if the threat is abusers using their iPhones to take pictures, will be demanded soon enough, and possible enough. Once this is there, the phone can continuously check camera input for likely-illegal material and flag it. Which is both a huge escalation and an obvious (and arguably desirable) next step; after all, if it saves one child, surely it would be churlish to complain about the minor inconvenience of your nudes being reviewed by an abuse department. (Also, implicitly, decent people don't take nude photographs; if you want to be a pervert, you can pay the social price.)
posted by acb at 8:44 AM on August 8, 2021 [2 favorites]
My favorite part of this, as someone who has never had any illusions about Apple is their internal memo dismissing concern of the "screeching voices of the minority". Think different.
This is such a perfect example of the problem in this debate. People are putting forth strident opinions without even reading or understanding the source material. If you had read it, you would know that Apple didn't use the phrase "screeching voices of the minority." NCMEC did. But that's not as inflammatory of a headline, so of course people claim Apple said it, and then just like a childhood game of telephone, it passes from poster to poster, morphing a little each time.
I don't particularly like what's going on here, but it's really hard to have an honest discussion in good faith if people keep twisting the words and who said them. Let's discuss the actual facts rather than turning it into a contest of tabloid headlines.
posted by primethyme at 8:54 AM on August 8, 2021 [19 favorites]
This is such a perfect example of the problem in this debate. People are putting forth strident opinions without even reading or understanding the source material. If you had read it, you would know that Apple didn't use the phrase "screeching voices of the minority." NCMEC did. But that's not as inflammatory of a headline, so of course people claim Apple said it, and then just like a childhood game of telephone, it passes from poster to poster, morphing a little each time.
I don't particularly like what's going on here, but it's really hard to have an honest discussion in good faith if people keep twisting the words and who said them. Let's discuss the actual facts rather than turning it into a contest of tabloid headlines.
posted by primethyme at 8:54 AM on August 8, 2021 [19 favorites]
Is there any evidence whatsoever that this sort of privacy invasion will make kids any safer?
I knew a guy who was a social worker ostensibly providing therapy to children in juvenile detention. Maybe he was helping some of the kids, but others he was sexually abusing and taking pictures for his own personal use. None of his pictures would have shown up in a preexisting database of such pictures. He accidentally left his laptop unlocked and a coworker accidentally found his the images and videos of the children and himself. He's in prison, but has no access to any resources that might help him control his urges.
The correlation between possession of child pornography and actual abuse is murky at best.
We should protect children, but should we give up our privacy to do something that may not make a difference to a single child?
posted by thedward at 8:57 AM on August 8, 2021 [3 favorites]
I knew a guy who was a social worker ostensibly providing therapy to children in juvenile detention. Maybe he was helping some of the kids, but others he was sexually abusing and taking pictures for his own personal use. None of his pictures would have shown up in a preexisting database of such pictures. He accidentally left his laptop unlocked and a coworker accidentally found his the images and videos of the children and himself. He's in prison, but has no access to any resources that might help him control his urges.
The correlation between possession of child pornography and actual abuse is murky at best.
We should protect children, but should we give up our privacy to do something that may not make a difference to a single child?
posted by thedward at 8:57 AM on August 8, 2021 [3 favorites]
I mean, maybe people are just spoiled by modern technology? I remember when you had to have photos developed by a "professional," which absolutely meant someone was looking at each and every photo you took, and the only thing that made them "professional" was they knew how to develop photos and you didn't. Plenty of those guys were fuckin creeps and didn't exactly have people watching them to make sure they didn't do creepy shit. Pretty sure at least a handful of those "professionals" were part of the old analog kiddie porn production pipeline.
Sorry, but I do think this system is better than that system. People used to have zero issues with total strangers seeing every photo they took, and by default, the lack of information surrounding those people (you didn't know if they're a creep, you didn't know if they're keeping copies of your photos for "science") means by default they are less trustworthy than a giant corporation who is actually being fairly forthright about what they are doing.
posted by deadaluspark at 8:57 AM on August 8, 2021 [6 favorites]
Sorry, but I do think this system is better than that system. People used to have zero issues with total strangers seeing every photo they took, and by default, the lack of information surrounding those people (you didn't know if they're a creep, you didn't know if they're keeping copies of your photos for "science") means by default they are less trustworthy than a giant corporation who is actually being fairly forthright about what they are doing.
posted by deadaluspark at 8:57 AM on August 8, 2021 [6 favorites]
How large is the CSAM hash database? If the scanning is done locally on your phone, does that mean several hundred megabytes are "wasted" on our already cramped phones?
posted by ymgve at 9:18 AM on August 8, 2021
posted by ymgve at 9:18 AM on August 8, 2021
How large is the CSAM hash database? If the scanning is done locally on your phone, does that mean several hundred megabytes are "wasted" on our already cramped phones?
I thought the hashing was done on the phone, not the DB comparisons.
posted by atoxyl at 9:26 AM on August 8, 2021
I thought the hashing was done on the phone, not the DB comparisons.
posted by atoxyl at 9:26 AM on August 8, 2021
How large is the CSAM hash database? If the scanning is done locally on your phone, does that mean several hundred megabytes are "wasted" on our already cramped phones?
A 256-bit hash is 32 bytes. 32 bytes times even 100,000 images is a touch over 3 megabytes.
After reading the paper I'm absolutely gobsmacked at how much care and effort Apple have gone to in order to keep as much information hidden as possible. Synthetic match vouchers even mean that Apple doesn't know how many violations have been made. All it does know is that it doesn't have enough to hit the decryption threshold and it's fucking ingenious.
I don't know if it's the right thing to do but damn I have to give them props for enabling matching CSAM while revealing so very, very little.
posted by Your Childhood Pet Rock at 9:28 AM on August 8, 2021 [13 favorites]
A 256-bit hash is 32 bytes. 32 bytes times even 100,000 images is a touch over 3 megabytes.
After reading the paper I'm absolutely gobsmacked at how much care and effort Apple have gone to in order to keep as much information hidden as possible. Synthetic match vouchers even mean that Apple doesn't know how many violations have been made. All it does know is that it doesn't have enough to hit the decryption threshold and it's fucking ingenious.
I don't know if it's the right thing to do but damn I have to give them props for enabling matching CSAM while revealing so very, very little.
posted by Your Childhood Pet Rock at 9:28 AM on August 8, 2021 [13 favorites]
I thought the hashing was done on the phone, not the DB comparisons.
From the Apple technical summary: "This blinded database is securely stored on users’ devices"
I also did a quick search and according to Interpol their CSAM database contained 2.7 million pictures and videos in 2018. So I guess it's not too bad for size use as long as the hashes aren't too long (I haven't seen any mention in Apple's spec of what size their hashes are).
There is also the big issue of "are we sure all the images in the database is actually of child exploitation and not, say, certain pictures of Tiananmen Square?"
posted by ymgve at 9:42 AM on August 8, 2021 [2 favorites]
From the Apple technical summary: "This blinded database is securely stored on users’ devices"
I also did a quick search and according to Interpol their CSAM database contained 2.7 million pictures and videos in 2018. So I guess it's not too bad for size use as long as the hashes aren't too long (I haven't seen any mention in Apple's spec of what size their hashes are).
There is also the big issue of "are we sure all the images in the database is actually of child exploitation and not, say, certain pictures of Tiananmen Square?"
posted by ymgve at 9:42 AM on August 8, 2021 [2 favorites]
So I guess it's not too bad for size use as long as the hashes aren't too long (I haven't seen any mention in Apple's spec of what size their hashes are).
I don't know what they're using, but if they were using an old MD5 hash, that's only 128 bits, which translates to roughly 43 megabytes for 2.7 million pictures.
posted by deadaluspark at 9:49 AM on August 8, 2021 [1 favorite]
I don't know what they're using, but if they were using an old MD5 hash, that's only 128 bits, which translates to roughly 43 megabytes for 2.7 million pictures.
posted by deadaluspark at 9:49 AM on August 8, 2021 [1 favorite]
There is also the big issue of "are we sure all the images in the database is actually of child exploitation and not, say, certain pictures of Tiananmen Square?"
Devil's Advocate here: Tiananmen Square happened in 1989, with a dearth of cameras to take photos of what happened. Meaning, there's only a handful of images of that day, and yes, with hashes, those could easily be hidden away from the world.
However, we're the modern world, with cameras everywhere, a million different photos and videos from all angles. It is becoming increasingly difficult for governments to capture, catalog and hash every single image that is damaging to their public image simply because of the scale of the thing.
Think about the 2020 police riots in the USA. There is SO MUCH video and photographic evidence of the pigs violently assaulting peaceful protestors. Do you really think the American government has the chops to find every single one those and add them to a hashed database, even if they wanted to? It's a tall fucking order, honestly. Machine Learning isn't up for that kind of task, yet.
So while I agree that it would be very useful in preventing discussion about, say, Tiananmen Square, I think it's a lot harder for them to hash and hide images from far more recent issues, because those issues are so widespread and photos are taken by so many people. The issue is absolutely scale, and while they might be able to hash and hide a lot of those images, they won't be able to hash all of them.
Also, as other people are pointing out, adding a shitload more hashes to the database will make the database size increase and start impacting device performance. When the hash database doubles in size overnight, people will have questions about those new hashes and where they came from.
posted by deadaluspark at 9:59 AM on August 8, 2021 [2 favorites]
Devil's Advocate here: Tiananmen Square happened in 1989, with a dearth of cameras to take photos of what happened. Meaning, there's only a handful of images of that day, and yes, with hashes, those could easily be hidden away from the world.
However, we're the modern world, with cameras everywhere, a million different photos and videos from all angles. It is becoming increasingly difficult for governments to capture, catalog and hash every single image that is damaging to their public image simply because of the scale of the thing.
Think about the 2020 police riots in the USA. There is SO MUCH video and photographic evidence of the pigs violently assaulting peaceful protestors. Do you really think the American government has the chops to find every single one those and add them to a hashed database, even if they wanted to? It's a tall fucking order, honestly. Machine Learning isn't up for that kind of task, yet.
So while I agree that it would be very useful in preventing discussion about, say, Tiananmen Square, I think it's a lot harder for them to hash and hide images from far more recent issues, because those issues are so widespread and photos are taken by so many people. The issue is absolutely scale, and while they might be able to hash and hide a lot of those images, they won't be able to hash all of them.
Also, as other people are pointing out, adding a shitload more hashes to the database will make the database size increase and start impacting device performance. When the hash database doubles in size overnight, people will have questions about those new hashes and where they came from.
posted by deadaluspark at 9:59 AM on August 8, 2021 [2 favorites]
> So while I agree that it would be very useful in preventing discussion about, say, Tiananmen Square, I think it's a lot harder for them to hash and hide images from far more recent issues, because those issues are so widespread and photos are taken by so many people. The issue is absolutely scale, and while they might be able to hash and hide a lot of those images, they won't be able to hash all of them.
In addition, governments would only know what images to block after they've spread across social media. This method can't be used to preempt the propagation of photos and videos of events.
posted by at by at 10:04 AM on August 8, 2021 [2 favorites]
In addition, governments would only know what images to block after they've spread across social media. This method can't be used to preempt the propagation of photos and videos of events.
posted by at by at 10:04 AM on August 8, 2021 [2 favorites]
My understanding is there is no public oversight of the database. What are questions going to do? Besides that's not how you do it. If I was in charge of the database and wanted to use it for nefarious purposes I'd be seeding it regularly with non CP image hash data to grow it over time and then if I wanted to find matches for a political image (whether for surpression or identification) I could just replace some of the previously entered data with new hashes.
posted by Mitheral at 10:09 AM on August 8, 2021
posted by Mitheral at 10:09 AM on August 8, 2021
>if I wanted to find matches for a political image ... I could just replace some of the previously entered data with new hashes.
Do you have any other superpowers you've not declared besides finding collisions in cryptographic hashes?
Hash collisions haven't really been addressed here, but a well-formed hash takes small differences in input data and makes them big differences in output. The thing we call the 'pigeonhole principle' (27 letters guarantees at least one double-up on the 26-character Latin alphabet) means that 256 bits or 512 bits of SHA-2 standard hashing will have multiple inputs that give the same hash when you've fed in gigabyte after gigabyte of input but it's hard: the use of this hash to find things that had results starting with 6-10 zeroes was a battle that burned much Chinese coal, destroyed computer part availability and made Bitcoin notorious.
You won't just replace the targets of these hashes without also breaking the security of the hashing algorithm.
posted by k3ninho at 10:29 AM on August 8, 2021 [1 favorite]
Do you have any other superpowers you've not declared besides finding collisions in cryptographic hashes?
Hash collisions haven't really been addressed here, but a well-formed hash takes small differences in input data and makes them big differences in output. The thing we call the 'pigeonhole principle' (27 letters guarantees at least one double-up on the 26-character Latin alphabet) means that 256 bits or 512 bits of SHA-2 standard hashing will have multiple inputs that give the same hash when you've fed in gigabyte after gigabyte of input but it's hard: the use of this hash to find things that had results starting with 6-10 zeroes was a battle that burned much Chinese coal, destroyed computer part availability and made Bitcoin notorious.
You won't just replace the targets of these hashes without also breaking the security of the hashing algorithm.
posted by k3ninho at 10:29 AM on August 8, 2021 [1 favorite]
I’m going to just point out that anyone wanting to comment on this should START by reading the explanation in Daring Fireball, because an awful lot of the concerns and predictions upthread are already unpacked, debunked, or explained in Gruber’s write up.
posted by caution live frogs at 10:31 AM on August 8, 2021 [13 favorites]
posted by caution live frogs at 10:31 AM on August 8, 2021 [13 favorites]
Gruber's write up is fine and informative, but it contains a lot of speculation and some plain contradictory material (e.g. "NCMEC is the only organization in the U.S. that is legally permitted to possess these photos" vs. Apple's "other child-safety organizations")
posted by RobotVoodooPower at 10:40 AM on August 8, 2021
posted by RobotVoodooPower at 10:40 AM on August 8, 2021
In addition, governments would only know what images to block after they've spread across social media. This method can't be used to preempt the propagation of photos and videos of events.Propagation, no, and they have other mechanisms for that in countries where it’s legally possible. However, it could be used to round up supporters of a protest movement or to de-anonymize people – if a photo or video is distributed through foreign media or activist groups they might be very interested in knowing who in the area had copies.
Again, this is currently all already possible to the extent that Apple, Google, Microsoft, etc. are willing to cooperate with the government in question so this isn’t an immediate huge change. My question would be more along the lines of “what would John Ashcroft have done if he’d had the option?” and how we’d know whether it was changing from only scanning iCloud uploads to everything.
A similar question arises with antivirus software: how would you know if your AV software had a signature update causing it to report arbitrary file contents? That’s already deployed widely and millions of people use products made by companies in other countries with different legal constraints.
posted by adamsc at 10:50 AM on August 8, 2021 [3 favorites]
I’m going to just point out that anyone wanting to comment on this should START by reading the explanation in Daring Fireball, because an awful lot of the concerns and predictions upthread are already unpacked, debunked, or explained in Gruber’s write up.
Ah, yes, Gruber, a truly unbiased source of info when talking about Apple.
posted by simmering octagon at 10:52 AM on August 8, 2021 [12 favorites]
Ah, yes, Gruber, a truly unbiased source of info when talking about Apple.
posted by simmering octagon at 10:52 AM on August 8, 2021 [12 favorites]
Also, if the database is just hashes of known CSAM, couldn't that have the unintended consequence of driving paedophiles to pay for having children abused live on camera, which is guaranteed to not hit a hash, and possibly driving the abuse rate up?
posted by acb at 11:03 AM on August 8, 2021
posted by acb at 11:03 AM on August 8, 2021
I wonder if Apple has thought about the implications of being required to maintain separate databases for each jurisdiction. For instance, I think cartoons and computer-generated images of CP are illegal in Australia but not in the USA: I seem to recall that someone here was prosecuted over explicit drawings of Bart and Lisa Simpson. If Australia demanded similar protection, that would mean tagging uploads geographically and reporting users based on their location. And what if, say, Saudi Arabia demands that Apple track copies of the blasphemous Jyllands-Posten Muhammad cartoons? It's just a mess of potential liability and regulatory requirements.
posted by Joe in Australia at 11:04 AM on August 8, 2021 [2 favorites]
posted by Joe in Australia at 11:04 AM on August 8, 2021 [2 favorites]
This method can't be used to preempt the propagation of photos and videos of events.
It is not known how new hashes are shared to devices. The provenance of some hashes is defined. Where others come from is left deliberately vague.
Obviously, no method can stop distribution of images of an event in progress, but this method could conceivably be used for augmenting near-real-time search and tracking of government targets, with quiet database pushes.
Google and Apple already make devices that share our location and proximity to others via metadata shared by telecommunications companies and apps. This extends the scope of distributed surveillance to photo and video captured by individuals.
In effect, our devices can be used to extend the degree to which we are already used to spy on one another without our knowledge or consent.
This is not entirely a hypothetical concern in states such as Texas, for example, which has already enacted Stasi-like anti-abortion law that an extremist federal Supreme Court could one day soon end up codifying.
Could this be used to target, say, gynecologists or other doctors whose travel patterns are deemed suspect? One can imagine other enemies of the state under a second Trumpist regime.
We live in a country with few real significant privacy laws, and companies like Google and Apple (and others) work with governments within that regulatory vacuum.
posted by They sucked his brains out! at 11:16 AM on August 8, 2021 [1 favorite]
It is not known how new hashes are shared to devices. The provenance of some hashes is defined. Where others come from is left deliberately vague.
Obviously, no method can stop distribution of images of an event in progress, but this method could conceivably be used for augmenting near-real-time search and tracking of government targets, with quiet database pushes.
Google and Apple already make devices that share our location and proximity to others via metadata shared by telecommunications companies and apps. This extends the scope of distributed surveillance to photo and video captured by individuals.
In effect, our devices can be used to extend the degree to which we are already used to spy on one another without our knowledge or consent.
This is not entirely a hypothetical concern in states such as Texas, for example, which has already enacted Stasi-like anti-abortion law that an extremist federal Supreme Court could one day soon end up codifying.
Could this be used to target, say, gynecologists or other doctors whose travel patterns are deemed suspect? One can imagine other enemies of the state under a second Trumpist regime.
We live in a country with few real significant privacy laws, and companies like Google and Apple (and others) work with governments within that regulatory vacuum.
posted by They sucked his brains out! at 11:16 AM on August 8, 2021 [1 favorite]
Also, if the database is just hashes of known CSAM, couldn't that have the unintended consequence of driving paedophiles to pay for having children abused live on camera, which is guaranteed to not hit a hash, and possibly driving the abuse rate up?
It would certainly be an interesting study to do. The systems such as PhotoDNA have been in use for years ny the likes of Microsoft and Google, and so we do have a time period in which we could study it. However, I don't actually think the vast majority of people actually pay attention to issues like this, those people including the vast majority of pedophiles.
I think about the January 6th terrorist insurrection when it comes to this a lot. Sure, all modern OSes allow you to spoof your MAC address, but the vast majority of the people who showed up to that did not even know what a fucking MAC address was and were doing dumb shit like deleting photos they had already posted online like that was going to solve it. Their MAC addresses were definitely logged by every cell phone tower and wifi hotspot they came into contact with. The massive amount of data collected on those people that day is nothing to sneeze at. Those people represent the lack of understanding of technology in America, and there's a metric fuckton of these people.
Another example was talking to a friend who is a middle school teacher, and him relating to me about when he was explaining smartphones and computers, and how students were shocked that a smartphone was technically a computer. They didn't even realize they function the same way, with microchips. Most kids just think this shit is fucking magic and don't even ask questions or think to.
I think this extends massively to the general populace. Most people won't even know Apple is doing this, despite the announcement, the furor, and the hand-wringing. Lots of pedophiles were caught thanks to tips from Microsoft and Google, and so it stands to reason that most pedophiles won't even know it's a thing and will continue being caught.
It will drive others to the darkweb and other sources, sure, but when it comes to the true technical ability to really secure your privacy online, there's just not that many people who are really up to the task. Hell, even the people in this thread (like myself) who are even halfway competent from a hobbyist perspective aren't really up to the task of truly securing our online privacy. Unless the pedophiles in question have oodles of money, industry connections, or literally are computer security experts themselves, the likelihood is that they will continue to use off the shelf components like iPhones and Androids and they will continue getting caught because of that.
People like things that are easy to use, and beyond that, most people don't have the time nor the technical capability to do all the difficult and computer sciencey stuff you need to do to secure your privacy online.
I just don't think it will actually drive that many to other ways of accessing it. Mostly because the modern era requires specialized knowledge, and so many people have such little knowledge of this that they don't even know it's a thing of truly understand how it works.
I mean, we had articles about this in 2014, when Google was revealed to be using this kind of software to scan gMail accounts for child porn. Did all the pedophiles stop using Google after that? No. So why would this be different suddenly?
posted by deadaluspark at 11:18 AM on August 8, 2021 [2 favorites]
It would certainly be an interesting study to do. The systems such as PhotoDNA have been in use for years ny the likes of Microsoft and Google, and so we do have a time period in which we could study it. However, I don't actually think the vast majority of people actually pay attention to issues like this, those people including the vast majority of pedophiles.
I think about the January 6th terrorist insurrection when it comes to this a lot. Sure, all modern OSes allow you to spoof your MAC address, but the vast majority of the people who showed up to that did not even know what a fucking MAC address was and were doing dumb shit like deleting photos they had already posted online like that was going to solve it. Their MAC addresses were definitely logged by every cell phone tower and wifi hotspot they came into contact with. The massive amount of data collected on those people that day is nothing to sneeze at. Those people represent the lack of understanding of technology in America, and there's a metric fuckton of these people.
Another example was talking to a friend who is a middle school teacher, and him relating to me about when he was explaining smartphones and computers, and how students were shocked that a smartphone was technically a computer. They didn't even realize they function the same way, with microchips. Most kids just think this shit is fucking magic and don't even ask questions or think to.
I think this extends massively to the general populace. Most people won't even know Apple is doing this, despite the announcement, the furor, and the hand-wringing. Lots of pedophiles were caught thanks to tips from Microsoft and Google, and so it stands to reason that most pedophiles won't even know it's a thing and will continue being caught.
It will drive others to the darkweb and other sources, sure, but when it comes to the true technical ability to really secure your privacy online, there's just not that many people who are really up to the task. Hell, even the people in this thread (like myself) who are even halfway competent from a hobbyist perspective aren't really up to the task of truly securing our online privacy. Unless the pedophiles in question have oodles of money, industry connections, or literally are computer security experts themselves, the likelihood is that they will continue to use off the shelf components like iPhones and Androids and they will continue getting caught because of that.
People like things that are easy to use, and beyond that, most people don't have the time nor the technical capability to do all the difficult and computer sciencey stuff you need to do to secure your privacy online.
I just don't think it will actually drive that many to other ways of accessing it. Mostly because the modern era requires specialized knowledge, and so many people have such little knowledge of this that they don't even know it's a thing of truly understand how it works.
I mean, we had articles about this in 2014, when Google was revealed to be using this kind of software to scan gMail accounts for child porn. Did all the pedophiles stop using Google after that? No. So why would this be different suddenly?
posted by deadaluspark at 11:18 AM on August 8, 2021 [2 favorites]
Could this be used to target, say, gynecologists or other doctors whose travel patterns are deemed suspect? One can imagine other enemies of the state under a second Trumpist regime.
No offense but they could have literally done this at any point during the actual Trump regime and nothing about what Apple is doing now actually would help you target their movements, especially when the government can already just buy all your movement data from a data broker with no warrant needed at all.
No need to get all "but what about if they had this power!" They did. They do.
EDIT: Just to be clear, I don't think it's good they have this power, just they already had that power a while ago. Once again, look at the Police Riots in 2020, they knew exactly where everyone was. They even arrested a woman who torched a cop car based on a shirt she was wearing which she bought on Etsy which they traced back to her through getting info from Etsy. They can do this already and they do and have.
posted by deadaluspark at 11:21 AM on August 8, 2021 [8 favorites]
No offense but they could have literally done this at any point during the actual Trump regime and nothing about what Apple is doing now actually would help you target their movements, especially when the government can already just buy all your movement data from a data broker with no warrant needed at all.
No need to get all "but what about if they had this power!" They did. They do.
EDIT: Just to be clear, I don't think it's good they have this power, just they already had that power a while ago. Once again, look at the Police Riots in 2020, they knew exactly where everyone was. They even arrested a woman who torched a cop car based on a shirt she was wearing which she bought on Etsy which they traced back to her through getting info from Etsy. They can do this already and they do and have.
posted by deadaluspark at 11:21 AM on August 8, 2021 [8 favorites]
Also, if the database is just hashes of known CSAM, couldn't that have the unintended consequence of driving paedophiles to pay for having children abused live on camera, which is guaranteed to not hit a hash, and possibly driving the abuse rate up?
Apple already decrypts iCloud photo data on demand of the courts. All of the major cloud photo providers (Google Photos, Amazon Photos) are the same so professionals have avoided any sort of cloud service since their inceptions. This makes it easier to catch tech illiterates and those who dabble.
Obviously, no method can stop distribution of images of an event in progress, but this method could conceivably be used for augmenting near-real-time search and tracking of government targets, with quiet database pushes.
Except Apple can't even see which image each voucher matches up to until the threshold has been reached along with synthetic vouchers which give background noise and stop occasional blips from being picked out. Going from the technical PDF, one needs a serious statistical spike above the baseline to be picked out.
posted by Your Childhood Pet Rock at 11:22 AM on August 8, 2021 [1 favorite]
Apple already decrypts iCloud photo data on demand of the courts. All of the major cloud photo providers (Google Photos, Amazon Photos) are the same so professionals have avoided any sort of cloud service since their inceptions. This makes it easier to catch tech illiterates and those who dabble.
Obviously, no method can stop distribution of images of an event in progress, but this method could conceivably be used for augmenting near-real-time search and tracking of government targets, with quiet database pushes.
Except Apple can't even see which image each voucher matches up to until the threshold has been reached along with synthetic vouchers which give background noise and stop occasional blips from being picked out. Going from the technical PDF, one needs a serious statistical spike above the baseline to be picked out.
posted by Your Childhood Pet Rock at 11:22 AM on August 8, 2021 [1 favorite]
No need to get all "but what about if they had this power!" They did. They do.
I tried to use words like "augment" and "extend" and others for that reason. I'm sorry I wasn't clearer.
posted by They sucked his brains out! at 11:25 AM on August 8, 2021 [1 favorite]
I tried to use words like "augment" and "extend" and others for that reason. I'm sorry I wasn't clearer.
posted by They sucked his brains out! at 11:25 AM on August 8, 2021 [1 favorite]
Did all the pedophiles stop using Google after that? No. So why would this be different suddenly?
This is actually one of the things puzzling me. The measures Apple announced will only catch people sharing images that have already been identified as CP. It won't catch people committing sexual abuse otherwise. It won't even identify people directly viewing CP online. Are there actually networks of pedophiles directly exchanging CP images nowadays? This sounds very 1980s.
posted by Joe in Australia at 11:33 AM on August 8, 2021
This is actually one of the things puzzling me. The measures Apple announced will only catch people sharing images that have already been identified as CP. It won't catch people committing sexual abuse otherwise. It won't even identify people directly viewing CP online. Are there actually networks of pedophiles directly exchanging CP images nowadays? This sounds very 1980s.
posted by Joe in Australia at 11:33 AM on August 8, 2021
I think cartoons and computer-generated images of CP are illegal in Australia but not in the USA
IIRC, in Australia, erotic photographs of small-breasted adult women are legally considered CP in some circumstances.
posted by acb at 11:33 AM on August 8, 2021
IIRC, in Australia, erotic photographs of small-breasted adult women are legally considered CP in some circumstances.
posted by acb at 11:33 AM on August 8, 2021
Are there actually networks of pedophiles directly exchanging CP images nowadays?
Yes. There really are.
posted by chrchr at 1:34 PM on August 8, 2021 [3 favorites]
Yes. There really are.
posted by chrchr at 1:34 PM on August 8, 2021 [3 favorites]
Hell, even the people in this thread (like myself) who are even halfway competent from a hobbyist perspective aren't really up to the task of truly securing our online privacy.
deadaluspark, I co-founded and ran an cybersecurity company for six years and worked with some of the smartest tech folks in the field of ethical hacking. I'm supposed to be more than halfway competent in this, and I know for sure I am NOT up to the task of securing my online privacy against nation state adversaries.
posted by DreamerFi at 1:58 PM on August 8, 2021 [8 favorites]
deadaluspark, I co-founded and ran an cybersecurity company for six years and worked with some of the smartest tech folks in the field of ethical hacking. I'm supposed to be more than halfway competent in this, and I know for sure I am NOT up to the task of securing my online privacy against nation state adversaries.
posted by DreamerFi at 1:58 PM on August 8, 2021 [8 favorites]
Are there actually networks of pedophiles directly exchanging CP images nowadays? This sounds very 1980s.Yes. The exact stats vary but this has over 20 million reports last year, most of which were from Facebook:
https://www.missingkids.org/content/dam/missingkids/gethelp/2020-reports-by-esp.pdf
Not all of them are hits from systems like this and the various parties involved have some incentive to show that they’re making a difference and/or do not need to be regulated but that’s a pretty large number even if you assume a hefty false positive rate. Microsoft reports something on the order of hundreds of thousands of unique images being recirculated.
posted by adamsc at 2:11 PM on August 8, 2021 [4 favorites]
A similar question arises with antivirus software: how would you know if your AV software had a signature update causing it to report arbitrary file contents? That’s already deployed widely and millions of people use products made by companies in other countries with different legal constraints.
@adamsc, you'll be so amused by this defcon video where a guy puts the EICAR test string in a QR code. For you non-technical folks out there, the simple explanation is interesting too. There's a piece of text called the EICAR test string that is used to test anti-virus engines. If antivirus software sees this text, say in a file, it is supposed to trigger and do the actions it would normally do when spotting a real virus. Combine this with the fact that most modern computers with a camera (such as a phone or PC with a webcam) continuously scan the image for barcodes and QR codes and if they see one, they try to parse it.
So this guy puts this anti-virus test string in a QR code, and shows it to many camera systems - for example the exit of car park that promptly shuts down all the exits of the car park, since the computer trying to decide if the car can exit throws an exception from the anti-virus software and basically stops working until somebody resets it.
So, yeah, curiously enough sometimes removing the anti-virus software from your computer makes it MORE secure.
Then there's the case where a contractor for an agency that develops and uses malware to, ehm, find terrorists puts some of these files on his own computer in a moment of brain fog where kaspersky notices "these files are weird, let me upload them to hq".
Keeping computers secure is hard. Very, very hard.
posted by DreamerFi at 2:22 PM on August 8, 2021 [17 favorites]
@adamsc, you'll be so amused by this defcon video where a guy puts the EICAR test string in a QR code. For you non-technical folks out there, the simple explanation is interesting too. There's a piece of text called the EICAR test string that is used to test anti-virus engines. If antivirus software sees this text, say in a file, it is supposed to trigger and do the actions it would normally do when spotting a real virus. Combine this with the fact that most modern computers with a camera (such as a phone or PC with a webcam) continuously scan the image for barcodes and QR codes and if they see one, they try to parse it.
So this guy puts this anti-virus test string in a QR code, and shows it to many camera systems - for example the exit of car park that promptly shuts down all the exits of the car park, since the computer trying to decide if the car can exit throws an exception from the anti-virus software and basically stops working until somebody resets it.
So, yeah, curiously enough sometimes removing the anti-virus software from your computer makes it MORE secure.
Then there's the case where a contractor for an agency that develops and uses malware to, ehm, find terrorists puts some of these files on his own computer in a moment of brain fog where kaspersky notices "these files are weird, let me upload them to hq".
Keeping computers secure is hard. Very, very hard.
posted by DreamerFi at 2:22 PM on August 8, 2021 [17 favorites]
The broader issue is that most users have no idea what’s being done with their digital images and even which companies have them. Until recently on iOS if you let an app have access to your camera and photos it had the ability to get everything. Even now I suspect that a lot of people just clicked allow all when they just wanted to share a single photo to Snapchat, Instagram, etc. Flickr, Google Photos and other services are often used to get around limited storage on devices; but of course that also lets Google mine your photos to build profiles to sell to advertisers.
This Apple thing is just the tip of the iceberg and we need to make sure people understand that instead of just thrashing Apple. We need to have stronger protections for our private moments and the data we store in the cloud.
posted by interogative mood at 3:47 PM on August 8, 2021 [2 favorites]
This Apple thing is just the tip of the iceberg and we need to make sure people understand that instead of just thrashing Apple. We need to have stronger protections for our private moments and the data we store in the cloud.
posted by interogative mood at 3:47 PM on August 8, 2021 [2 favorites]
How likely is it that there will be accidental or deliberate collisions with those hashes?
Apple claims one in a trillion per year. (Notably not per user per year)
I don't really have an enormous problem with the idea. The implementation is as privacy preserving as possible, after all. A single match tells nobody anything, not even Apple. The device sends back false matches to prevent them from knowing if there are any real matches until there are enough real matches to allow decryption. I do question how effective it will actually be, both in the sense of slowing the distribution of CP and the further abuse of children.
The whole thing has me beginning to think NCMEC is going down the same path MADD did. Despite the wide availability of digital cameras and camera phones taking billions of pictures a year, there are only 2.4 million known CP images? Kinda makes you think...
posted by wierdo at 4:16 PM on August 8, 2021 [1 favorite]
Apple claims one in a trillion per year. (Notably not per user per year)
I don't really have an enormous problem with the idea. The implementation is as privacy preserving as possible, after all. A single match tells nobody anything, not even Apple. The device sends back false matches to prevent them from knowing if there are any real matches until there are enough real matches to allow decryption. I do question how effective it will actually be, both in the sense of slowing the distribution of CP and the further abuse of children.
The whole thing has me beginning to think NCMEC is going down the same path MADD did. Despite the wide availability of digital cameras and camera phones taking billions of pictures a year, there are only 2.4 million known CP images? Kinda makes you think...
posted by wierdo at 4:16 PM on August 8, 2021 [1 favorite]
Enjoy managing your battery life with secret unkillable scanning processes running in the background.
posted by srboisvert at 7:21 PM on August 8, 2021
posted by srboisvert at 7:21 PM on August 8, 2021
Enjoy managing your battery life with secret unkillable scanning processes running in the background
To be clear this is running already on iCloud and, unrelated, on GMail. What they're doing is just moving this to the clientside which means someone doing the numbers found out that it cost $x/year in iCloud data center processing power to churn through all those images. They figured it would cost a certain amount to offload it clientside and that was less than they were paying over a certain time frame so they did it.
Given that this is looking at existing images, it would be odd to save them on your phone, wouldn't it? Plus if you somehow got one of these illegal images, which I'm sure are not easy to obtain given that everyone is looking for them you'd probably be in an environment where you're disconnected form iCloud and encrypting things to prevent a simple hashing algorithm from capturing it.
This feels like a backdoor POC to get into our devices more than it feels like it is actually doing anything to stop child pornography.
posted by geoff. at 8:20 PM on August 8, 2021 [1 favorite]
To be clear this is running already on iCloud and, unrelated, on GMail. What they're doing is just moving this to the clientside which means someone doing the numbers found out that it cost $x/year in iCloud data center processing power to churn through all those images. They figured it would cost a certain amount to offload it clientside and that was less than they were paying over a certain time frame so they did it.
Given that this is looking at existing images, it would be odd to save them on your phone, wouldn't it? Plus if you somehow got one of these illegal images, which I'm sure are not easy to obtain given that everyone is looking for them you'd probably be in an environment where you're disconnected form iCloud and encrypting things to prevent a simple hashing algorithm from capturing it.
This feels like a backdoor POC to get into our devices more than it feels like it is actually doing anything to stop child pornography.
posted by geoff. at 8:20 PM on August 8, 2021 [1 favorite]
Like a lot of things, if we could only trust people not to be awful when given more power over our lives, this wouldn't necessarily be a bad thing, but ... yeah.
posted by pulposus at 10:11 PM on August 8, 2021
posted by pulposus at 10:11 PM on August 8, 2021
>The measures Apple announced will only catch people sharing images that have already been identified as CP. It won't catch people committing sexual abuse otherwise. It won't even identify people directly viewing CP online.
The database looks to me like detection policing (which confirms that a thing has happened to get a robust conviction) rather than preventiative interventions (which engages in communities to know the people and redirect individuals to therapy and restorative justice).
posted by k3ninho at 11:53 PM on August 8, 2021
The database looks to me like detection policing (which confirms that a thing has happened to get a robust conviction) rather than preventiative interventions (which engages in communities to know the people and redirect individuals to therapy and restorative justice).
posted by k3ninho at 11:53 PM on August 8, 2021
I'm not making this a link, and I'm obfuscating the url a bit. For obvious reasons:
hxxps://drive.google.com/drive/folders/1c3-a8-XQ1X0CxXuFL0N3XVkglUD2OJS_
These harmless generated images have a neuralhash equivalent to those provided in the NCMEC database submitted for testing. Do NOT upload these harmless images to iCloud as Apple will assume its Child Porn (CSAM).
Because, yes, somebody already has made a tool that creates images that match a NeuralHash.
Well, there are several, that's just one of them.
And NeuralHash is in the latest beta.
posted by DreamerFi at 1:01 AM on August 9, 2021 [10 favorites]
hxxps://drive.google.com/drive/folders/1c3-a8-XQ1X0CxXuFL0N3XVkglUD2OJS_
These harmless generated images have a neuralhash equivalent to those provided in the NCMEC database submitted for testing. Do NOT upload these harmless images to iCloud as Apple will assume its Child Porn (CSAM).
Because, yes, somebody already has made a tool that creates images that match a NeuralHash.
Well, there are several, that's just one of them.
And NeuralHash is in the latest beta.
posted by DreamerFi at 1:01 AM on August 9, 2021 [10 favorites]
So either they go for security through obscurity, modifying the hash algorithm somewhat (which means that whoever is authorised to have the original images will have to be involved), they ignore the problem (and ship a feature any troll can prank) and quietly roll back enforcement of it, they roll back the filter altogether, or they upgrade to a more intrusive neural classifier that calculates the CP-potential of an image (and can be run live on the input of the camera, if so mandated).
posted by acb at 1:16 AM on August 9, 2021
posted by acb at 1:16 AM on August 9, 2021
somebody already has made a tool that creates images that match a NeuralHash
I thought that finding hash collisions was a Hard Problem unless you skimped on the address space?
posted by Joe in Australia at 1:27 AM on August 9, 2021
I thought that finding hash collisions was a Hard Problem unless you skimped on the address space?
posted by Joe in Australia at 1:27 AM on August 9, 2021
cryptographical hashes are indeed hard. These aren't crypto hashes, they are perceptual hashes. Their goal is to be resilient against rotation, crop, etc. Different kind of problem.
posted by DreamerFi at 2:05 AM on August 9, 2021 [4 favorites]
posted by DreamerFi at 2:05 AM on August 9, 2021 [4 favorites]
Even the most commonly used "crypto" hashing algorithms have known collision attacks. Thankfully, they aren't useful in the context of TLS, code signing, and the like because the tampering is obvious and changes the file size, making them relatively easy to mitigate.
Thankfully, the part of Apple's protocol that requires multiple hits before flagging an account and not having any way to see that there have been any hits at all until a certain threshold is reached would make it pretty robust against collision attacks so long as they remain at least moderately computationally expensive. If NeuralHash collisions can be generated reasonably quickly, on the other hand, it's ripe for whatever the CP-detection equivalent of SWATting is called and abuse by authorities who could potentially force Apple to disclose any flagged account immediately, before any internal review process takes place and use that flag to justify seizing and searching your shit.
posted by wierdo at 2:52 AM on August 9, 2021
Thankfully, the part of Apple's protocol that requires multiple hits before flagging an account and not having any way to see that there have been any hits at all until a certain threshold is reached would make it pretty robust against collision attacks so long as they remain at least moderately computationally expensive. If NeuralHash collisions can be generated reasonably quickly, on the other hand, it's ripe for whatever the CP-detection equivalent of SWATting is called and abuse by authorities who could potentially force Apple to disclose any flagged account immediately, before any internal review process takes place and use that flag to justify seizing and searching your shit.
posted by wierdo at 2:52 AM on August 9, 2021
Wait, NeuralHash doesn't do the obvious thing of putting its output through a final round of SHA or something?
posted by ymgve at 4:34 AM on August 9, 2021 [1 favorite]
posted by ymgve at 4:34 AM on August 9, 2021 [1 favorite]
Wait, NeuralHash doesn't do the obvious thing of putting its output through a final round of SHA or something?
Since the whole point of NeuralHash is that different images result in the same hash (rotate, mirror, crop, stuff like that) it's not too surprising it's not hard to deliberately generate hash collisions. What you're looking for is "this and only this image, yes I'm pointing at that file there, should generate this one hash" which is exactly what NeuralHash is designed NOT to be.
posted by DreamerFi at 4:51 AM on August 9, 2021 [3 favorites]
Since the whole point of NeuralHash is that different images result in the same hash (rotate, mirror, crop, stuff like that) it's not too surprising it's not hard to deliberately generate hash collisions. What you're looking for is "this and only this image, yes I'm pointing at that file there, should generate this one hash" which is exactly what NeuralHash is designed NOT to be.
posted by DreamerFi at 4:51 AM on August 9, 2021 [3 favorites]
Since the whole point of NeuralHash is that different images result in the same hash (rotate, mirror, crop, stuff like that) it's not too surprising it's not hard to deliberately generate hash collisions. What you're looking for is "this and only this image, yes I'm pointing at that file there, should generate this one hash" which is exactly what NeuralHash is designed NOT to be.
No, I mean, you put the output of NeuralHash into a normal cryptographic hash. This preserves the NeuralHash property of "same hash -> visually same picture" while at the same time making it computationally infeasible to generate a picture that matches the same final hash.
(Unless you have the original picture that generated the hash of course, then you can attack the intermediate step, but it's impossible for anyone without access to the source pictures)
posted by ymgve at 5:12 AM on August 9, 2021 [1 favorite]
No, I mean, you put the output of NeuralHash into a normal cryptographic hash. This preserves the NeuralHash property of "same hash -> visually same picture" while at the same time making it computationally infeasible to generate a picture that matches the same final hash.
(Unless you have the original picture that generated the hash of course, then you can attack the intermediate step, but it's impossible for anyone without access to the source pictures)
posted by ymgve at 5:12 AM on August 9, 2021 [1 favorite]
somebody already has made a tool that creates images that match a NeuralHash.
A quick look at the source code of the collision generation tool you linked to tells me that it's based on modifying a source image in such a way as to minimize a distance metric between that image's perceptual hash and that of the target hash it's trying to collide with. If the target perceptual hashes were not available - if, instead, the comparison databases held only cryptographic hashes of the perceptual hashes - then the information required to generate those distance metrics would not be available either.
Of course, if "match a NeuralHash" actually means "come very close to that NeuralHash according to some or other distance metric" as opposed to "be bit-identical with that NeuralHash" then a final round of crypto hashing would completely break the ability to perform that matching.
posted by flabdablet at 5:15 AM on August 9, 2021 [1 favorite]
A quick look at the source code of the collision generation tool you linked to tells me that it's based on modifying a source image in such a way as to minimize a distance metric between that image's perceptual hash and that of the target hash it's trying to collide with. If the target perceptual hashes were not available - if, instead, the comparison databases held only cryptographic hashes of the perceptual hashes - then the information required to generate those distance metrics would not be available either.
Of course, if "match a NeuralHash" actually means "come very close to that NeuralHash according to some or other distance metric" as opposed to "be bit-identical with that NeuralHash" then a final round of crypto hashing would completely break the ability to perform that matching.
posted by flabdablet at 5:15 AM on August 9, 2021 [1 favorite]
Having a database of hash(neuralhash(img)) doesn't stop attackers if they can get ahold of the original source images, because then although they can't invert the outer hash, they don't need to because they can just compute neuralhash(source_img). In this case the source images are themselves illegal, making it a case of "security through illegal training data".
posted by Pyry at 6:14 AM on August 9, 2021
posted by Pyry at 6:14 AM on August 9, 2021
Are we going to ban standalone digital cameras and encrypted ZIPs too? Or have operating systems that scan every file on our computers against a database of hashed CSAM?
Predators will adapt, and this technology will be abused for other purposes once people get used to it.
posted by snuffleupagus at 6:31 AM on August 9, 2021
Predators will adapt, and this technology will be abused for other purposes once people get used to it.
posted by snuffleupagus at 6:31 AM on August 9, 2021
No, I mean, you put the output of NeuralHash into a normal cryptographic hash.
If you think about it, that won't help. Identical inputs to the crypto hash produce identical outputs.
posted by SPrintF at 6:40 AM on August 9, 2021
If you think about it, that won't help. Identical inputs to the crypto hash produce identical outputs.
posted by SPrintF at 6:40 AM on August 9, 2021
I don't think the ability to trigger Apple's scrutiny would help an attacker: apparently the protocol would allow the decryption of the apparently-offending images, but not anything else. So the attacker sends innocuous triggering images to someone's phone; some poor bastard at Apple is tasked with examining those images; the images turn out to be innocuous; case closed.
Maybe an attacker could convolute a piece of legal porn so it resembles a triggering image and Apple's staffer mistakenly identifies it as CP - but an attacker with that degree of sophistication could probably obtain and use genuine CP to trigger an investigation.
No, to the extent that I'm worried about the security aspect it's that fundamental layers of iOS are apparently held together with packing tape and twine. I read that the recently-reported hacks of iPhones were triggered by crafted data sent via iMessage, which has lots of legacy code meant to interpret data packaged in various ways. This proposal provides Yet Another Opportunity for an attacker to force a phone to process arbitrary data and thereby corrupt the heap or whatever.
I'm also worried about the slippery slope which may lead to Apple enabling repression by regimes looking for excuses to suppress political rights, but it's hard to argue that in this case: apparently Apple is already scanning uploaded files for CP, so.
posted by Joe in Australia at 6:41 AM on August 9, 2021
Maybe an attacker could convolute a piece of legal porn so it resembles a triggering image and Apple's staffer mistakenly identifies it as CP - but an attacker with that degree of sophistication could probably obtain and use genuine CP to trigger an investigation.
No, to the extent that I'm worried about the security aspect it's that fundamental layers of iOS are apparently held together with packing tape and twine. I read that the recently-reported hacks of iPhones were triggered by crafted data sent via iMessage, which has lots of legacy code meant to interpret data packaged in various ways. This proposal provides Yet Another Opportunity for an attacker to force a phone to process arbitrary data and thereby corrupt the heap or whatever.
I'm also worried about the slippery slope which may lead to Apple enabling repression by regimes looking for excuses to suppress political rights, but it's hard to argue that in this case: apparently Apple is already scanning uploaded files for CP, so.
posted by Joe in Australia at 6:41 AM on August 9, 2021
Of course, if "match a NeuralHash" actually means "come very close to that NeuralHash according to some or other distance metric" as opposed to "be bit-identical with that NeuralHash" then a final round of crypto hashing would completely break the ability to perform that matching.
That's why the crypto-hash idea, although it is a VERY smart question to ask, compliments to @ymgve for coming up with it, won't work.
posted by DreamerFi at 7:03 AM on August 9, 2021
That's why the crypto-hash idea, although it is a VERY smart question to ask, compliments to @ymgve for coming up with it, won't work.
posted by DreamerFi at 7:03 AM on August 9, 2021
So the attacker sends innocuous triggering images to someone's phone; some poor bastard at Apple is tasked with examining those images; the images turn out to be innocuous; case closed.
Joe, Apple has to worry about "Denial of Service" attacks. They don't have an infinite amount of poor bastards.
posted by DreamerFi at 7:04 AM on August 9, 2021
Joe, Apple has to worry about "Denial of Service" attacks. They don't have an infinite amount of poor bastards.
posted by DreamerFi at 7:04 AM on August 9, 2021
So the attacker sends innocuous triggering images to someone's phone; some poor bastard at Apple is tasked with examining those images; the images turn out to be innocuous; case closed.
Somehow "I was flagged as a child pornographer but Apple decided it was a false positive" doesn't feel like a place I'd want to be, "case closed" or no.
posted by The Bellman at 7:16 AM on August 9, 2021 [2 favorites]
Somehow "I was flagged as a child pornographer but Apple decided it was a false positive" doesn't feel like a place I'd want to be, "case closed" or no.
posted by The Bellman at 7:16 AM on August 9, 2021 [2 favorites]
I don't think the ability to trigger Apple's scrutiny would help an attacker: apparently the protocol would allow the decryption of the apparently-offending images, but not anything else. So the attacker sends innocuous triggering images to someone's phone; some poor bastard at Apple is tasked with examining those images; the images turn out to be innocuous; case closed.
Apple can already decrypt all of the photos in the iCloud account. Same as Google. Same as Amazon. Same as most US based photo providers. Apple has end to end encryption on card transactions, home data, health data, the iCloud Keychain, maps search history/bookmarks, web search history/bookmarks, memoji, any payment methods, learned keyboard vocab (FBI can't tell if you got rid of ducking), screen time, siri stuff, wifi passwords, and any bluetooth keys for W1 and H1 devices. Photos are stored encrypted by Apple and are fair game for a court order.
The vouchers are a different thing that can be used by an automated process without having to touch the content on the server side. I assume they're trying to keep it client side so that if China turns up at the door of GCBD asking to scan en masse they shrug their shoulders because they're just hosting it and have nothing that can be turned on the dataset. China turns up to Apple US with the request and Apple tells them to GFY.
posted by Your Childhood Pet Rock at 7:32 AM on August 9, 2021 [1 favorite]
Apple can already decrypt all of the photos in the iCloud account. Same as Google. Same as Amazon. Same as most US based photo providers. Apple has end to end encryption on card transactions, home data, health data, the iCloud Keychain, maps search history/bookmarks, web search history/bookmarks, memoji, any payment methods, learned keyboard vocab (FBI can't tell if you got rid of ducking), screen time, siri stuff, wifi passwords, and any bluetooth keys for W1 and H1 devices. Photos are stored encrypted by Apple and are fair game for a court order.
The vouchers are a different thing that can be used by an automated process without having to touch the content on the server side. I assume they're trying to keep it client side so that if China turns up at the door of GCBD asking to scan en masse they shrug their shoulders because they're just hosting it and have nothing that can be turned on the dataset. China turns up to Apple US with the request and Apple tells them to GFY.
Could governments force Apple to add non-CSAM images to the hash list?If CSAM detection gets end-to-end encryption turned on for iCloud photos I could see that being a worthy compromise. I think the only thing that makes it even remotely ok is that it's only images going to the cloud and it's all perceptual hashing. iCloud photos are already potentially accessible to LEO and it's not like a human is going to be able to go through a trove looking for nudes to screenshot.
Apple will refuse any such demands. Apple’s CSAM detection capability is built solely to detect known CSAM images stored in iCloud Photos that have been identified by experts at NCMEC and other child safety groups. We have faced demands to build and deploy government-mandated changes that degrade the privacy of users before, and have steadfastly refused those demands. We will continue to refuse them in the future. Let us be clear, this technology is limited to detecting CSAM stored in iCloud and we will not accede to any government’s request to expand it. Furthermore, Apple conducts human review before making a report to NCMEC. In a case where the system flags photos that do not match known CSAM images, the account would not be disabled and no report would be filed to NCMEC.
posted by Your Childhood Pet Rock at 7:32 AM on August 9, 2021 [1 favorite]
Matthew Green has some excellent threads on Twitter.
There is an open letter against this which I have signed. I hope there is internal pushback from Apple employees.
This Stratechery article is also excellent.
I am concerned about the slippery slope of what this is expanded to cover in subject as well as what on the phone is included in scanning beyond iCloud uploads. I’m also concerned about attacks be it Pegasus, less sophisticated malicious apps that write to photos, or the scariest thought: What’s App which has the default that images sent to you are auto saved to you photo library.
posted by ridogi at 8:28 PM on August 9, 2021
There is an open letter against this which I have signed. I hope there is internal pushback from Apple employees.
This Stratechery article is also excellent.
I am concerned about the slippery slope of what this is expanded to cover in subject as well as what on the phone is included in scanning beyond iCloud uploads. I’m also concerned about attacks be it Pegasus, less sophisticated malicious apps that write to photos, or the scariest thought: What’s App which has the default that images sent to you are auto saved to you photo library.
posted by ridogi at 8:28 PM on August 9, 2021
From the "what could possibly go wrong" department:
Apple today held a questions-and-answers session with reporters regarding its new child safety features, and during the briefing, Apple confirmed that it would be open to expanding the features to third-party apps in the future.
posted by DreamerFi at 12:15 AM on August 10, 2021
Apple today held a questions-and-answers session with reporters regarding its new child safety features, and during the briefing, Apple confirmed that it would be open to expanding the features to third-party apps in the future.
posted by DreamerFi at 12:15 AM on August 10, 2021
Apple’s CSAM detection capability is built solely to detect known CSAM images stored in iCloud Photos that have been identified by experts at NCMEC and other child safety groups.
Which is just sidestepping the issue. How willing are NCMEC experts to stand up to the Federal Government bearing (Secret? FISA?) court orders? I'm guessing NCMEC doesn't have the super deep legal pockets Apple has and is 100% dependent on Federal government cooperation to legally maintain the child porn database (possession of which would otherwise be a crime).
posted by Mitheral at 11:46 AM on August 10, 2021 [1 favorite]
Which is just sidestepping the issue. How willing are NCMEC experts to stand up to the Federal Government bearing (Secret? FISA?) court orders? I'm guessing NCMEC doesn't have the super deep legal pockets Apple has and is 100% dependent on Federal government cooperation to legally maintain the child porn database (possession of which would otherwise be a crime).
posted by Mitheral at 11:46 AM on August 10, 2021 [1 favorite]
Next up: NCMEC demands hash-scanning functionality be added to the 'cp' and 'mv' Unix utilities. /s
posted by rhizome at 1:01 PM on August 10, 2021 [1 favorite]
posted by rhizome at 1:01 PM on August 10, 2021 [1 favorite]
It turns out that the algorithm was in the wild, and has been reverse engineered. Hash collision has already been demonstrated.
posted by NoxAeternum at 11:43 AM on August 18, 2021 [1 favorite]
posted by NoxAeternum at 11:43 AM on August 18, 2021 [1 favorite]
I came to post what NoxAeternum just posted.
posted by glonous keming at 5:51 PM on August 18, 2021
posted by glonous keming at 5:51 PM on August 18, 2021
I found this video to be very interesting: iPhone Neural Hash - SHOCKING AI Tech by privacy activist Rob Braxman.
posted by glonous keming at 9:58 PM on August 20, 2021
posted by glonous keming at 9:58 PM on August 20, 2021
« Older @thetrashwalker | No limbs, no problem Newer »
This thread has been archived and is closed to new comments
1) you have to trust Apple that it does exactly this and this only. And if you trust Apple, do you trust the government whose jurisdiction Apple is in?
2) prediction: the Chinese government will tell Apple to use THEIR hash database to check Chinese users and it will include a hash of tank man. Insert other governments and relevant pictures for your favorite government
3) prediction: the next Pegasus version will add the option to add a few images to your image library that match database hashes, giving law enforcement using pegasus a good pretext to arrest you.
Remember that whatever computing device you use, act like it's NOT yours and it does NOTHING to protect your privacy.
posted by DreamerFi at 3:53 AM on August 8, 2021 [33 favorites]