Windows AI feature that screenshots everything labeled a security 'disaster'

https://www.theverge.com/2024/6/3/24170305/microsoft-windows-recall-ai-screenshots-security-privacy-issues

by alwillis • 2 years ago

73 42 2 years ago

theverge.com

speedylight• 2 years ago

Windows 10 is a disaster, this is a catastrophe!

neilv2• 2 years ago

Sounds like great tech for forensics, surveillance, and legal discovery.

hsbauauvhabzb• 2 years ago

And employers.

MadnessASAP• 2 years ago

I don't think employers are going to be that interested in this. It's not like there's a shortage of tools to monitor employees usage of company owned devices.

Rather this seems like a giant security hole, I'm almost certain that my company would permanently disable anything like this on our computers and I'm sure we're not the only ones.

neilv2• 2 years ago

I suppose it depends what side of discovery they're on.

Oh, ha, if a company is under some kinds of investigation, could they be instructed not to disable this record-every-little-thing feature on any company Windows computer?

Finally, it's the year of Linux on the desktop! :)

TheLoafOfBread• 2 years ago

Interesting. Could be used to create fake alibi too - i.e. I will use innocent browsing of social networks logged by this feature as an evidence that it was not me who got access into server which should not be accessed in that specific time.

OrvalWintermute• 2 years ago

Dystopian hubris

m3kw9• 2 years ago

Screen shot that password creation screen

etchalon• 2 years ago

You mean a feature which records all user activity into a global, well-structured, explicitly searchable structure might be a ripe target?

Obviously unforeseeable and not at all a feature implemented rashly in a clever but fragile manner.

Phil_Latio• 2 years ago

The thing is: Even if one can disable this "feature"... Who guarantees that Microsoft will not enable it again by some update or suddenly introduce a remote version which is enabled by default?

Microsoft can not be trusted anymore because regulators failed in the past to properly punish companies for dirty behavior.

layman51• 2 years ago

Wasn’t there an open source software (called “Rem”) that someone posted about here not that long ago? It was basically the same thing as this Windows feature but it was for Mac.

SV_BubbleTime• 2 years ago

I’m not at all a “year of the Linux desktop guy”… at all… but seriously, I’m so very glad I dropped Microsoft last year. Even the idea here was a bridge too far.

lenkite• 2 years ago

This feels like some crazy joke taken from a fictional alternate universe. Or that I am dreaming. Slapped myself and checked - nope this is true reality.

transcriptase• 2 years ago

I am once again amazed at the inability for anyone within Microsoft to prevent or halt the development of things that nobody wants, needs, or could reasonably expect to be a good idea.

poisonborz• 2 years ago

Actually this is something everyone will want. Not by Microsoft, not built into Windows, but this is something that has to come in some form.

Slyfox33• 2 years ago

I don't want it.

rsynnott• 2 years ago

I'm increasingly convinced that at this point it is, for these companies, mostly about putting out the word "AI" in press releases a lot until such time as the markets lose interest; any actual utility is completely beside the point.

hulitu• 2 years ago

things that nobody wants, needs, or could reasonably expect to be a good idea.

Maybe for you. Sincerely, the NSA. /s

andrewflnr4• 2 years ago

It's not a good idea, but I don't see how security is a big part of the problem. In particular crying foul about the data being stored in the clear seems silly. How would encrypting it help, when the key has to be right there anyway? Is it that much worse than all the other sensitive data also sitting in the clear in, you know, regular files?

What's the plan, encrypt the data a public key where the private key is locked most of the time? Require a passphrase any time you want to read or process it? Does anyone think that UX will fly? Maybe there's some TPM magic that makes it work without also making it trivial to steal for an attacker who already has arbitrary code execution, but no one is citing that so far. I don't see the better design that everyone seems to think is obviously there. (Besides the obvious Just Don't, which is fair as far as it goes but has nothing to do with crypto)

hiisukun• 2 years ago

Just for example: It could be stored encrypted with the user's credentials. So when you want to browse "Recall" it might pop-up requesting a fingerprint, password, token etc.

Then it decrypts, presents to the user, and closes the handles when you're done. In this way, the "key" isn't present -- the user has it!

That's quite a different security situation to it just sitting around in plaintext on disk. Certainly there's a number of Windows and MacOS features sitting behind authentication already, from a UX perspective.

andrewflnr4• 2 years ago

I don't think users would go for that UX, so if you're committed to building this feature, I don't think you can go that route without basically accepting that no one will use it. I suspect MS wants this to be a lot more casual than the other features behind authentication now. (Like I told akkartik, I'm open to the argument that there's no way to implement it securely and with an acceptably easy UX, but I don't see people actually making that argument.) Maybe they'll prove me wrong by doing something like that, though.

akkartik3• 2 years ago

The way I read these arguments is: the lack of security adds to the case against this product. The reviewers aren't saying there's a self-evident solution, just that there are self-evident problems. How did this get through review? So it's not that there are two separate arguments, but that the security issues strengthen the main argument that this is a product that shouldn't exist.

andrewflnr4• 2 years ago

That's a very charitable reading. In my reading, the dismissive tone of the "security" arguments implies there's something obvious that MS should have done better specifically security-wise. If there's no such better implementation, then security has nothing to do with the argument, and they should stop talking about it. If they wanted to make a more nuanced argument about how there's no way to make a secure implementation that provides the smooth UX people will actually use, I would respect that a bit more, but I'm not seeing either that particular claim or the evidence to back it up in the mainstream commentary.

jijji• 2 years ago

it's like a keystroke logger on steroids

localfirst• 2 years ago

Apple is working on something like this too I bet but with Siri + OpenAI

ChrisArchitect• 2 years ago

[dupe]

Actual article: https://doublepulsar.com/recall-stealing-everything-youve-ev...

More discussion: https://news.ycombinator.com/item?id=40540703

elmar• 2 years ago

Could this be the best contender to the stupidest idea on the last 10 years by a big tech corporation?

deely3• 2 years ago

I like the idea of Recall. But its a Microsoft with 'lets hide/remove these settings from user' bullshit, so...

the feature is enabled by default when you set up a new Copilot Plus PC, and there is no option to disable it during the setup process

leotravis10• 2 years ago

It's coming to every new Windows 11 powered PC in the future so this will be harder to ignore.

polotics• 2 years ago

Reading about the details of this, it really looks like a feeble attempt to jump on the AI bandwagon. Such a shoddy design could easily be implemented on top of any linux desktop distro, and much better. Also isn't it amazing that despite being fully in control of the OS, the code of this recall function will burn GPU cycles doing OCR on images of the screen?

woleium• 2 years ago

looks like a coop project, or a friday afternoon tech demo. hard to believe its on by default

smarm52• 2 years ago

PSA for how to disable Co-Pilot (and Recall) on Windows.

https://www.reddit.com/r/Windows11/comments/1bh7n4d/how_do_i...

klipklop• 2 years ago

What’s the point really? Microsoft has a habit of re-enabling all telemetry with Windows update which is really annoying to block or disable. They will do the same with Recall and just say it’s a bug that turned it back on.

DemocracyFTW22• 2 years ago

I haven't used Windows in ages but there used to be popular tools that were used even by not-so-very nerdy, more average folks that would search your PC, indicate all the problems they found with wasted disk space and security, and offer fixes.

It still goes without saying that without a resounding "I understand the risks and still want to use this inherently insecure feature" things like these should not be enabled by the OS manufacturer / vendor.

firtoz• 2 years ago

I love the feature, it finally convinced me to give Lunix another try!

floppiplopp• 2 years ago

Microsoft payed a shitload of money to tech oligarchs for "AI" and is now desperately looking for a use case. And this is the best they could come up with. The future sucks. We gave it up to some scammy tech bros and the tech illiterate management caste.

DemocracyFTW22• 2 years ago

Recall doesn’t perform content moderation, so it won’t hide information like passwords or financial account numbers in its screenshots [which it then stores in an unencrypted SQLite DB]

I predict this thing will be big, like Scripting Host and Outlook were in their day /s

Windows AI feature that screenshots everything labeled a security 'disaster'

Related Stories