Nightshade, a tool allowing artists to 'poison' AI models

https://venturebeat.com/ai/meet-nightshade-the-new-tool-allowing-artists-to-poison-ai-models-with-corrupted-training-data/

by Anon84 • 3 years ago

77 71 3 years ago

venturebeat.com

pixl97• 3 years ago

Previous discussion on topic

https://news.ycombinator.com/item?id=37990750

artninja1988• 3 years ago

I wonder if you can build a classifier to detect those attacks by nightshade. Either way it's clear that these supply chain attacks need to be taken more seriously. I wonder how come humans aren't that prone to this adversarial nosie compared to models

taeric• 3 years ago

What makes you think we aren't as prone to this as the models are? There is a good reason folks rarely notice the many animals that are surrounding them on a walk in the woods.

Even worse, we are prone to being influenced by the decor or surroundings in ways that we know, but still partake in. Mood lighting would be an easy example. And we are very influenced by our other senses at the same time as purely visual. (Though, I'm always amused by how much the VR experience of racing a car hits, all without too much haptic feedback from the steering wheel.)

xyzal2• 3 years ago

Paper> https://arxiv.org/abs/2310.13828

schleck84• 3 years ago

There was an attempt at poisoning training data with such invisible cloaking before and it turned out that this didn't really affect training quality (Glaze was the project name)

Iirc there was also some license violation with the cloaking code ironically, it was like a pillow fight

schleck84• 3 years ago

https://www.reddit.com/r/StableDiffusion/comments/11sqkh9/gl...

Apparently this is the same team and they claim that the previous one was only intended to prevent training on an artist's style.

brucethemoose24• 3 years ago

The ML community is stuffed with license violations. I have seen more than once instance of randomly copy-pasted code by sheer coincidence, recognizing it from older projects.

ML researcher culture is hard to describe... But its something like "Everything is a research prototype, so why does it matter?"

rhaps0dy• 3 years ago

There was an attempt at poisoning training data with such invisible cloaking before and it turned out that this didn't really affect training quality

Source? The authors claim this is not the case, and they have experiments showing it.

Garvi4• 3 years ago

There's too much money at stake for the big players for this not to be labeled illegal any time now.

NoboruWataya2• 3 years ago

Which big players? There are some big players with an interest in facilitating the pervasion of AI, sure, but as far as I can tell there are are least as many big players who stand to lose from the widespread erosion of intellectual property rights that comes with it.

Garvi4• 3 years ago

I had Microsoft, Google and Apple in mind as they are heavily invested into AI. Which big players are you talking about?

jncfhnb2• 3 years ago

Seems incredibly unlikely

Hamuko• 3 years ago

Illegal where? Because I have a hard time believing it'd be illegal in the US due to the First Amendment existing.

Garvi4• 3 years ago

Freedom of speech in the US? I'm gonna laugh all the way into negative internet happy points that are gonna teach me a lesson never to say this again, unless this comment gets removed by the local censor :)

MisterBastahrd6• 3 years ago

The first amendment is going to slap the taste out of their mouths if they try it. If you want to steal my data for your personal use, I am under no obligation to play nice with you.

orbital-decay3• 3 years ago

It's illegal to booby-trap your house with the malicious intent to kill trespassers. With enough lawyers/lobbying, the same argument could work to criminalize the intent to break the product, even if it violates your license (separate offence).

survirtual3• 3 years ago

I don't understand what you're saying here. Why are you upset about creating new tools for current and future humans to be more capable at unleashing their creative visions?

All progress comes at labor from the past. If you didn't utilize past labor, you would not be talking to us now on the internet. You would be banging rocks against some cave wall instead.

If someone is inspired by art and they become an artist creating a similar style, is that stealing from you too? Is using a computer stealing from the engineers that created them? Is sleeping in a house stealing from the architects and builders?

If someone speaks do they need to pay royalties to the creators of the English language?

This technology is the beginning of creating a real life holodeck. Anything someone imagines they can creatively bring to life. How is it possible there are people greedy and self centered enough to not desire that for the future?

It is hard to follow the logic of your stance. I am genuinely curious about your modality of thinking if you would care to elaborate.

itslennysfault• 3 years ago

Nah, they'll use the easiest/cheapest/fastest way to prevent this, and, in this case, I'm pretty sure that is not litigation, but technical. I'm pretty sure they will be able to filter this stuff out or minimize its impact on training.

Garvi4• 3 years ago

That makes a lot of sense. I was going off the assumption the technical way would be impractical and an never ending arms race. We'll see, I guess.

dragonwriter• 3 years ago

This isn't the first such poisoning method published is that they are announced, and then quickly its shown that (1) they aren’t really visually acceptable for most real art when you look at images at a bigger size than what is included in the announcement paper, resulting in noticeable quality degradation and/or (2) they can be easily countered, usually, IIRC, both.

I think it seems very likely that that this will always be the case: if it is visually neutral enough to be acceptable for use, there will be a way to train systems to generate art that visually resembles the original based on the output.

Devasta3• 3 years ago

Delighted to see it. Fuck AI art.

schleck84• 3 years ago

It's the way forward though. AI won't stop like this, psychotherapy will be next in the coming years (look at Pi, it has a therapy mode now even). Just accept that AI will affect all aspects of life in 10 years

timdiggerm• 3 years ago

"forward" towards what?

JumpCrisscross• 3 years ago

AI will affect all aspects of life in 10 years

This is good peak of inflated expectations material [1]. LLMs will wholly transform some industries. Others will be virtually unchanged.

[1] https://en.m.wikipedia.org/wiki/Gartner_hype_cycle

brucethemoose24• 3 years ago

Yeah, well, fuck Photoshop too. And video editors.

Jerks are going to find dumb, lazy, malicious ways to abuse good tools, like they always do, though diffusion does have an conspicuously low effort barrier.

Another HN user summarized my thoughts on this better than I possibly could: https://news.ycombinator.com/item?id=37978805

Devasta3• 3 years ago

Artists are still able to produce art with or without photoshop. Without their LLMs and the stolen art within, the best an AI "artist" can do is to scrawl "Tracer Overwatch big boobs not ugly trending on artstation" into a page.

echelon• 3 years ago

That's like saying "fuck computers" because they automate typists, elevator operators, and Rolodex company salespeople.

To quote an artist that I know, "artists already steal. They're the masters at it."

Stealing inspiration and style is how artists learn. It just takes time. They're angry that the time investment is going to zero. And also that their examples are what are being used, despite the fact that other human artists are doing that already.

I'm not sure we should defend any difficulty or complexity. Even that of our own careers.

We'd still be churning our own butter if we were afraid of automation. And what time would that leave us with?

jwells89• 3 years ago

There might be an ethical space in which AI imagery can exist, but consent is a direly needed component.

The typical argument against this is that human artists and ML models learn the same way, but the problem is that there's a fundamental difference: humans produce art by abstracting the concepts (think directional lighting, the kinematics of skeletal creatures, color theory, or physics of cloth) from the pieces they've studied and combining that with what they've learned from observing the world around them and by producing original art. This is why a human artist can produce competent art after studying only a handful of art pieces and gain most of their skill through practice.

By contrast, ML models do not learn by producing and have no way to directly observe the world. They don't abstract concepts, they "learn" by memorizing huge numbers of pre-existing images, and they produce images by patching together tiny bits and pieces of of those memorized images.

Where the human artist can function on little to no input, an ML model cannot, and so it's only fair that human artists have the choice to not have their works included in training datasets because if there is no feedstock of art there is no AI generated imagery.

jdietrich2• 3 years ago

The middle class are starting to experience what the working class have spent the last two centuries enduring. You're a skilled artisan with years of training and experience? Tough luck, we've got a machine that can do your job better, faster or at least cheaper.

I have a clog you can throw into a loom if it'll make you feel better, but there's no stopping AI art.

epups• 3 years ago

Sounds interesting, but without more details I'm not sure whether it would be so hard to detect and even correct for. Might be defeated even by some pixel averaging.

XorNot• 3 years ago

These are temporary measures though. By most respects, training against data with this type of poisoning is going to improve the overall performance of models because they'll necessarily have to learn to disregard this sort of invisible data (which necessarily it's far a field from human perception).

natch4• 3 years ago

With a mere moment's thought, one thinks of the obvious way to defeat this:

Simply rescale all images in a preprocessing step.

Since this defeat is so easy, and so obvious, either the authors are daft, or they are brilliant and have thought of a way to prevent it.

So, not wanting to waste much time on a paper from daft people, I did a quick skim of the paper. I don't see any mention of this defeat. No prevention of it, no assessment of it, not even any evidence of awareness of it.

Did I miss something?

I'd hate to see technically naive artist types (a mere subset of artist types, I acknowledge) being hoodwinked by such snake oil and paying money for such "tools."

delecti• 3 years ago

If you rescale all your training data, then you're training on deliberately blurry images.

byteknight• 3 years ago

That's not accurate. Downsizing doesn't always blur.

natch4• 3 years ago

Ever so slightly degraded perhaps in some cases but "blurry" is overstating it. And worth the trivial cost if such dumb-as-bricks techniques take hold.

One could also submit each image to an image-to-image model with a blank prompt and the right settings to essentially ask for almost a copy, and come out with washed images. Could even do a sharpening (or upscale and sharpening for that matter) at the same time. It has a small (tiny really) GPU cost per image but it's a one-time cost.

riidom• 3 years ago

You could use it to check if the image is poisoned. Original image is a "cat", downscaled/blurred/posterized or whatever is a "dog", then something is strange and better skip it.

numpad03• 3 years ago

Is it legal to willfully circumvent DRM for the purpose of ML training?

Intralexical• 3 years ago

As with any DRM technology, the most it can hope to accomplish ultimately is to put more barriers and risks in the way. It won't stop the unwanted behavior completely, but provided it does what it says it shouldn't have zero impact either.

valine• 3 years ago

That sounds like a lot of work for something that can be worked around trivially. Add a small amount of random noise to the images and you’re done. Most people are probably doing this already for data augmentation purposes.

NoMoreNicksLeft• 3 years ago

Supposing this works as claimed, I'd be fine with denying copyright protection to any work proven to have the poison.

Most likely though, it will be simple to strip out.

Seriously, there's a "copyright derangement syndrome" going on right now, and these people need to settle the fuck down already.

_chu12• 3 years ago

I always had this idea in my head about feeding AI garbage it cannot regurgitate into anything useful but not a clue on how to create said garbage. Hopefully tools like this can stop image AI dead in it's tracks.

xyzal2• 3 years ago

Not exactly concerning image AI, but regarding LLMs -- in case your site does not depend on indexing -- you can easily generate boatloads of garbage text via Markov chains and invisible-link them.

One cool project for that> https://github.com/jsvine/markovify

(Personally had the most fun outputs when constructing the corpus from technical articles mixed with esoteric babble)

Active defense!

michaelmrose2• 3 years ago

Can you explain why we would want to stop image AI in its tracks? Why do we want to replace taxi drivers and factory workers but not cheap crap commercial artists?

Given that real art is both incredibly cheap to produce and incredibly cheap to distribute I should think those who are so inspired will continue to produce works to inspire us. I mean the advent of printers, cheap prints, and stock images didn't destroy art so why should AI?

_chu12• 3 years ago

Artists and photographers were never asked permission of their photos to be used to develop image AI yet they ended up doing so, it was built off theft and will continue to be build off theft by using user data from social media companies.

Nightshade, a tool allowing artists to 'poison' AI models

Related Stories