Picking up the fight against deepfakes, voice cloning and generative AI

https://www.garandor.com/

by matyask • 2 years ago

23 16 2 years ago

garandor.com

metadat• 2 years ago

Does the watermarking survive aggressive re-encodes?

According to this page, it. seems unlikely, because if that's been solved then you would market it.

Also, the source images are quite low resolution, what's up with that?

What is actually new or novel here?

janalsncm2• 2 years ago

I appreciate the spirit of this, but this is likely to be defeated by anyone with some basic technical understanding of how training works. For example a denoising autoencoder will probably be sufficient.

See the Glaze tool which was also cracked https://arstechnica.com/tech-policy/2024/07/glaze-a-tool-pro...

That said, it’s more like a booby trap that might catch some people off guard who didn’t think to look for it.

matyask6• 2 years ago

Hi, one of the founders here, "robustness" is our next step. We are currently training a few models based on cross attention and invariant domain learning that promise to be quite resistant to noising and denoising. The purpose of fingerprint watermarking as we have released it is to act like a signature for validity, some companies already do this but none of them have public products that you can just use. The nest encoder and decoder will use a unique signature for every user. The limitation of Glaze and models like it is that they target specific models/approaches. That being said we are also running experiments on adversarial watermarking that aim to deliver the same image quality as glaze with less computation and more resistance. We want to secure digital identity and copyrights. Watermarking is our initial approach.

mrbungie• 2 years ago

I'm still trying to understand how a watermarking method may resist simple methods as (1) upscaling and downscaling plus noise, or (2) taking a photo of the image plus compression plus noise, (3) any simple convolutional filter or a combination of those.

janalsncm2• 2 years ago

Glaze techniques might be specific to a model but the techniques to defeat it are not.

More specifically, I don’t see any reason to believe that I cannot learn a function from A to B where A is the distribution of watermarked files and B is the distribution of non-watermarked files. It’s a trivial exploit.

trod1233• 2 years ago

Interesting company, and something that is surely good to watch out for in the future. It is also very needed use case now that generative AI is just about everywhere.

There is some cause to be skeptical of the claims on the website, because the site doesn't provide enough information to validate the claims with any rational basis, and there are a number of other companies that have done the same and failed to deliver.

There have also been a few court cases now where similar use cases for attribution have been used inappropriately, and resulted in false claims/charges causing great miscarriages of justice (under facial recognition).

Hopefully you performed an adequate adversarial review, and have a demonstrable way to rationally prove there are no false positives at a bare minimum.

jrm4• 2 years ago

I'll bet a dollar that none of this ends up being particularly viable.

Feels like an "anti-Photoshop" company; whatever means you might like or want to exist re defending this sort of thing will lose in an arms race.

This is not to say that there's nothing that can be done, but I just doubt those solutions will ever be technical.

matyask6• 2 years ago

We for sure don't aim to be anti-AI. It's difficult to see how a non technical solution will be enforceable/effective. Watermarking is our initial approach to securing digital identity and copyrights.

matyask6• 2 years ago

Hi there, we have published a demonstration of fingerprint watermarking on audio and images that anyone could just try. In our next phase we will release our first consumer product that will provide value. We have conducted thorough testing on our fingerprint encoding and decoding. We will for sure aim to explain this better when we give users the ability to encode unique signatures.

If there are any other particular claim you have not found enough information about then I am happy to elaborate.

Could you by any chance link any of those court cases?

trod1233• 2 years ago

I will be looking forward to it.

The major concern I had just taking a cursory look revolved around the wording of 'unique' signatures. This word is often misused in an out of context way that makes the claim false.

When used it has a very specific meaning in math, but many times the claim being made is actually an improper equivalence being made (for marketing).

For example, they map some inputs to a finite field that may roll over, which isn't disclosed. This is known to violate the 1:1 map required for a 'unique' property between input and output, excepting some very rigorous methodology and forcibly limited system's/environments.

Uniformity of the collisions in such systems is also a very big problem, sometimes they clump, but absent bruteforce checking the entire space there's no way to predict ahead of time when clumping will happen. Similar equivalences have been made in the crypto space, and shown to be false when those systems were rigorously broken later. The existence of collisions given same inputs is proof by contradiction the input->output pair is not unique and fails.

I've linked one of the cases with a brief gist below (since Justia doesn't provide a rundown).

Harvey Eugene Jr. was arrested based on a false match for a robbery of a Sunglass Hut in Texas (owned by Macy's). He lived at the time in Sacramento CA, he was arrested when he returned to renew his Texas driver's license; during holding he was raped by three other inmates leaving him with permanent debilitating injuries. He was at the time of the robbery provably living in Sacramento. His arrest was based solely on the false positive AI based facial recognition match.

https://dockets.justia.com/docket/texas/txsdce/4:2024cv00801...

There have been many news articles, and several cases, a simple search for "facial recognition false arrest" on google should provide a number of articles.

You may also find a few on the ACLU/EFF site as they have a keen interest in going after companies who violate civil rights; their website has a history of some of the more prominent ones.

gmerc• 2 years ago

More snakeoil. Great.

It’s a fundamental misalignment with the threat model - misinformation and deepfakes being scaled by nation state actors and major political parties to its infinite resources on one since and the reliance on a fully secured providence chain on the other that makes it pure snakeoil.

matyask6• 2 years ago

I think you are making assumptions on provenance. The resource disparity you point out is obvious, but your phrasing somewhat binary. We have released a demo not a consumer product yet. I would be happy to hear your more expanded thoughts, especially on the threat model you have in your mind.

masly• 2 years ago

This reminds me of the struggle mapmakers used to have where other map makers would copy their maps. So to counter it and be able to prove the copying, they would add tiny unnoticeable but fake places.

https://vinepair.com/cocktail-chatter/cartographers-fake-pla...

Picking up the fight against deepfakes, voice cloning and generative AI

Related Stories