Does GPT-2 Know Your Phone Number?

https://bair.berkeley.edu/blog/2020/12/20/lmmem/

by umangkeshri • 6 years ago

321 155 6 years ago

bair.berkeley.edu

grenoire• 6 years ago

Very rough outcome:

"Moreover, if such a request were granted, must the model be retrained from scratch? The fact that models can memorize and misuse an individual’s personal information certainly makes the case for data deletion and retraining more compelling."

Besides, how would one even know that their info was used in a training dataset, only if and when it's revealed in a generated excerpt?

visarga10• 6 years ago

They can try to prompt it. But I hope the GPT-3 authors have a hash list of ngrams in the training data to be able to avoid verbatim reproduction. I know they trained an offensive content detector model, they should also train a PII information detector to make sure it's being hashed.

newman8r• 6 years ago

Playing with the AI dungeon a while back (on the GPT-2 mode) I was presented with a tilapia recipe, titled "Kittencal's Broiled Tilapia" - it sounded bizarre so I decided to do a google search and I found that it was directly pulled from from https://www.recipezazz.com/recipe/broiled-parmesan-tilapia-7... - the user who posted it was 'Kittencal'

fouronnes3• 6 years ago

There is enough GPT discussion online now that the next language model will be trained on text that talks about itself. That's when it gets interesting.

taneq3• 6 years ago

Oooh, now that does raise some spooky time-delayed consciousness vibes.

minimaxir4• 6 years ago

I ran an experiment through GPT-3 with just that frame: https://github.com/minimaxir/gpt-3-experiments/tree/master/e...

geofft3• 6 years ago

This seems like what xkcd called "citogenesis" - something inaccurate gets published on Wikipedia, a traditional "reliable source" (newspaper, published nonfiction book, etc.) retells that bit of information without a source, Wikipedia cites the source, that information is now firmly established as truth for the rest of humanity.

I worry about this with Google's language translation models, too. It's entirely possible that it's making up phrases or connotations that never existed in organic human speech, but people who aren't fully fluent in the language use Google Translate for assistance, publish something, and then suddenly it's in a published text by an actual human and Google reinforces its own belief.

For at least the past five years, Google Translate has translated "who proceeds from the father" into Latin as "qui ex patre filioque procedit" - inserting the additional word "filioque," which means "and the son." The question of whether to add this word is a 1500-year-old theological argument: https://en.wikipedia.org/wiki/Filioque Since the Western Church added the word, most texts in Latin include it, so Google is almost certainly deciding that the phrasing with "filioque" is more popular - but it doesn't know what the words mean, so it can't realize that the phrase it came up with means something different!

sillysaurusx• 6 years ago

This has been a thing for over a year now. https://www.reddit.com/user/subsimgpt2metaGPT2/?sort=top

dmm• 6 years ago

So in terms of copyright, is GPT-2 a derived work of that recipe? Or generally are models derived works of their training data?

It seems lots of people use training data from Flickr, like COCO, and then use the resulting model for commercial services.

hprotagonist• 6 years ago

i’ve been in meetings with lawyers where this issue was discussed. As of 2018 the answer was “not enough case law to know, hold tight it’s going to be weird one day!”

tiborsaas6• 6 years ago

It cited it with the authors name, hardly a derived work :)

est315• 6 years ago

IANAL, but from my reading Article 3 of the EU copyright directive explicitly protects data mining for scientific research, and Article 4 enables everyone to use content for data mining whose rights have not been explicitly and machine readably reserved.

taneq3• 6 years ago

It's as much a derived work as the user who posted the anecdote here. They were trained on the same data!

kevinmchugh• 6 years ago

Recipes aren't typically copyrightable

In general, I think the question is unsettled

m463• 6 years ago

Ethical companies make sure they have rights to the training data before making models.

That said, now a lot of privacy policies make sense.

ALSO, it makes me wonder about "Your call is recorded for training purposes"... is it a coincidence or is it very carefully worded?

PeterisP• 6 years ago

The current legal understanding is that according to the current copyright laws the ML models trained on some data are not considered derived works of that data, so if the data was obtained legally and without other restrictions (e.g. if the researchers sign a contract with the data owner that gives them access to data but agree to certain conditions) then the authors of the data do not have any claim on some rights to the resulting model.

One aspect that causes this is that historically statistical models calculated from large volumes of text (which is a notion that predates computers, e.g. frequency dictionaries and the whole [sub]field of quantitative corpus linguistics) have been considered facts about that corpus of text and thus not copyrightable at all or (depending on jurisdiction) entitled to different set of protections/limitations assigned to compilations of facts, which give some rights to the people who compiled the facts but no rights to the source of these facts (since facts as such aren't entitled to protection by copyright law).

This also applies to many forms of analysis of audiovisual data, where the copyrights of the source works do not transfer to the results of the statistical or qualitative analysis and can't limit their creation, distribution or sale.

The appropriate analogy to a commercial book or movie is not a translation, but some analysis of it - e.g. a thorough literary review and critique of some book or movie is a separate work with its own copyright, and the original author has no claim on it despite the fact that is (obviously) based on the contents of the work and describes it in great detail. Including verbatim fragments of the work is limited (fair use allows some inclusions but not all), but all the other details are not.

The whole notion of copyrightability of ML model weight files is interesting and IMHO not settled. You could argue that there is some creative expression in forming the model (which would support it being copyrightable) or you could argue that it's a mechanistic result of the application of some algorithm and settings (which have the creative part, and are copyrightable on their own), and so the output can't be copyrightable, no matter how much work (human or machine), time and cost it took - at least in USA copyright law doctrine (e.g. Feist Publications v. Rural Telephone Service) is that mere "sweat of the brow" (no matter how much) does not entitle a work to copyright protection; it requires application of human creativity to create an original work, and automated processes can't satisfy that requirement.

And crucially, if some output is not copyrightable in the first place, it can't be considered a derived work according to copyright law i.e. the exclusive right of authors to create derived works (or grant permission for others to do so) does not apply.

Another analogy might be a simple n-gram model (i.e. counts of bigrams - word pairs, trigrams, etc) which is quite clearly a mechanistic noncreative collection of facts about a dataset, and is also able to "answer" questions such as what is someone's telephone number if that was in the source data.

PurpleFoxy• 6 years ago

Probably depends how similar your output is to something copyrighted. I doubt the simple act of learning from previous art is disallowed or else literally everyone would be blocked because no one has a truely original thought.

colanderman• 6 years ago

This is something that has repeatedly bothered me while playing with GPT-2. It retains too much "long-tail" knowledge that seems counterproductive for a generic language tool. I would think those one-off associations should not be present after training has "digested" them but there they are.

aabhay• 6 years ago

Hard to decide how much is too much.. the hard (and interesting) part of language is in the long tail

alisonkisk3• 6 years ago

It's over fitting, not long tail, that is the problem. It ends up copying input instead of analyzing and synthesizing.

commandpaul2• 6 years ago

I've been playing with a writing tool shortlyread.com which purpotedly uses the GPT3 API. I had a similar experience in its responses to a lot of my prompts containing text verbatim from many sources and sometimes even going on to output personally identifying about persons related to the original text.

Filligree• 6 years ago

Took a look at that.

It's better at being a text editor, in that you can click the text and edit it! (Sweet relief.)

But it's missing the world-info feature from AI Dungeon, and without that I don't think it's practical to write anything long-form.

commandpaul2• 6 years ago

It actually does have that feature , it's tucked away in the probe panel.

jl6• 6 years ago

Though the surprise is really just a result of my own ignorance, I’m surprised at the breadth of training material used here.

I can foresee a future GPT-x that doesn’t know my phone number, but can deduce it.

jcims2• 6 years ago

I was in a discord with someone that had direct access to GPT-3. We played Jeopardy with it, where we would take a wikipedia article about a person and use a snippet of the first paragraph as the prompt and ask GPT-3 who it was.

It was very good at guessing the right person.

I'm curious if one can use GPT-3 to do some clustering of people based on their writing. For example, if I take the writing of several diagnosed sociopaths as a prompt or fine tuning data, could I use GPT-3 to detect same in the wild?

I would imagine GPT-5 or 6 will start consuming video as well. This will be interesting if you start mixing the content from YouTube, TikTok, WSHH, etc. into the mix. So not only will it be able to generate text, it will generate a convincing video of a person speaking to you with plausible facial expressions and intonation in speech.

fakedang• 6 years ago

Guess it's because it was likely trained on Wikipedia data. After all, GPT-3 is no stranger to borrowing phraseology from verbose texts in the internet.

jcims2• 6 years ago

That was our conclusion as well, but that's still pretty powerful. The remarkable part for me was that it contextually knew that we were asking for the name of the subject that the text was about.

hooande3• 6 years ago

The sociopath thing is just text classification, you could do that now without gpt-3. Video data would be significantly more difficult to model because it doesn't follow a simple set of rules like written language does

anticensor• 6 years ago

GPT is generalisable to other kinds of data. See ImageGPT for example.

tiborsaas6• 6 years ago

When Peter put his contact information online, it had an intended context of use. Unfortunately, applications built on top of GPT-2 are unaware of this context, and might thus unintentionally share Peter’s data in ways he did not intend.

That's expected when you publish anything online, you lose control over the data.

refactor_master• 6 years ago

Still doesn’t justify what could be perceived as borderline defamation.

tiborsaas6• 6 years ago

Who is defaming who? One person using GPT to generate text that puts X into a bad light? Then the person using GPT is responsible they could just use string replace %victimname%.

Or GPT accidentally defaming someone? That assumes bad intent which is a bit problematic with a language model.

With made up contexts like this in the article it's easy to cry wolf.

wpietri• 6 years ago

In the same sense that when you leave the house you lose control over what happens to you. But if something bad happens to a person., the right solution is not to say, "Well, you left the house, so tough luck."

tiborsaas6• 6 years ago

You'd be right, if my copy left my house. It's the physical vs. digital theft argument.

If I leave the house I take on some risk, but most of the time I'm in control.

barrkel2• 6 years ago

Ah yes. And if your name is included in an accusation of murder, well, I guess you shouldn't have published your name.

This is infantile reverse justification from an end. GPT style corpuses are clearly problematic without a lot more smarts on figuring out what's appropriate and what's not.

tiborsaas6• 6 years ago

Anyone can publish any nonsense online since the internet exists, how is this any different?

Responsibility is on those who publish content online, not the tools.

politician• 6 years ago

Except for copyright, which works completely differently to magically protect your rights for up to 70 years after your death.

ClumsyPilot5• 6 years ago

So copying an internet-published trailer of a Marvel movie is a crime, but my privacy is up for grabs.

tiborsaas6• 6 years ago

Depends entirely on what you share publicly. GPT's training data involves only publicly crawlable dataset.

We should be more concerned about mitigating GPT-like systems because one thing is for sure, we can't stop these models. The genie is out of the bottle and I'm sure multiple actors are working on similar systems.

est315• 6 years ago

One important corollary of this is that "privacy respecting" federated learning schemes [0] where you train a model locally with your data and only upload the deltas might leak your private data after all.

[0] https://arxiv.org/abs/1602.05629

why_only_154• 6 years ago

In practice you add large gaussians to the deltas, which gives you reasonably strong guarantees.

max_• 6 years ago

How is the strength of these gurantees quantified?

Any resources?

est315• 6 years ago

It's a common fallacy. Adding noise does not anonymize data sufficiently. If your traffic monitoring software adds gaussians to car speeds, you can still average over multiple reports about two cars to identify their respective average speeds. Those can be used to determine whether they are trucks or cars, etc.

In neural networks, weight changes do carry meaning. If your network has particularly large updates for the horse recognition neuron, you likely watched horse pictures. If it has updates for handbags, you likely watched pictures of them. If the company averages over all your submissions, the noise will be less relevant and the handbags and horse pictures will eventually show up.

px43• 6 years ago

I've been saying this for years, but every post you make online, every unencrypted email, IM, text message, etc, will eventually end up getting sold off as training data for future machine learning projects.

Every company stores this stuff for ages, and the value of candid conversation data just keeps increasing. Eventually these companies are either going to get hacked, get bought, or go bankrupt, and all the cleartext data they hold is going to get passed around to various data markets and end up incorporated into GPT-12 or whatever.

The moral of the story here isn't that companies need to stop storing data, or that we need to run ML researchers out of town. It's that people really need to start using the encryption technologies that were built decades ago to protect some of their most valuable assets, their mental model of the world. Otherwise these systems, which are being trained to extract as much value out of you and the ones you love as possible, will use this data you're giving them for free against you, and it'll be to late to do anything about it then.

jrockway• 6 years ago

Isn't that how humans work? Sometimes I wonder if I have free will and sentience, or if I am just stringing words together based on some probability function.

I've no doubt taken into account copyrighted works and personal information when training my built-in neural network. The examples the article gives like "misremembering" the murder as the murder victim sounds like something a person would do. Knowing verbatim contact information of some random person is also possible. All in all, GPT-2 sounds a lot like us.

jakear5• 6 years ago

It’s my understanding that under our current scientific understanding of the mind, we cannot have “free will”. Where would it come from? All science knows is that neurons receive inputs from sensors, mixes them together (with some quantum randomness sprinkled in), and eventually actuates muscles to produce outputs, exactly as GPT.

Thus “free will” must come from some sort of faith. Either it was given to us by “God”, or we’re all part of some simulation and it exists on a level we don’t know about, or something else entirely too advanced for us to understand.

hyperpape• 6 years ago

Of course it isn't how the human mind works.

You can ask a person "did you come up with that sentence, or is it something you read?" And they can answer that question. The answer isn't perfectly reliable, but it isn't completely unreliable either. You can't do that with GPT-2/GPT-3.

Big picture: the human mind is a machine of sorts. It's not the same kind of machine as any form of machine learning we've developed so far.

ojnabieoot3• 6 years ago

The moral of the story here isn't that companies need to stop storing data, or that we need to run ML researchers out of town.

There is a lot of space between “user data and ML research on that data needs to be closely regulated” and “run the ML researchers out of town.” This uncharitable exaggeration makes critics sound like an uninformed mob and closes yourself off to legitimate ideas. This directly leads to logically flawed and morally gross suggestions like:

It's that people really need to start using the encryption technologies that were built decades ago to protect some of their most valuable assets, their mental model of the world.

It is simply not reasonable to expect most Facebook/Skype/etc users to know enough about encryption to make good decisions here - for the same reason that you can’t expect people to have detailed understanding of food science to protect themselves from unscrupulous grocers, or to have a detailed understanding of medicine to protect themselves from fraudulent doctors. It really doesn’t matter if this knowledge has been around for “decades” since it’s still specialist knowledge.

Protecting digital privacy from unscrupulous tech companies and unethical ML researchers is a job for the government. Suggesting otherwise is victim-blaming. “The problem is that society is dumb and needs to become smart through the power of self-righteous scolding” is not helpful.

ClumsyPilot5• 6 years ago

Its amazing that such a sober take on the issue is getting downvoted without a single counterargument

visarga10• 6 years ago

Maybe this data will be used to 'upload' you in a virtual world after the singularity. It might be your second life.

flatline• 6 years ago

When thinking about this, you have to ask, who will profit, and how? It doesn’t seem like a clone of your brain running in a simulation is going to be of much advantage to you from the current industry incentives. Maybe if you’re rich?

st1x72• 6 years ago

Even in the science fiction version of this, we're still talking about a copy. It's not you, you can't leave your body or be uploaded. It's just a digital copy that looks like you.

fastball2• 6 years ago

End-to-end encryption is a possible solution to some of this, no?

ClumsyPilot5• 6 years ago

I can't use a single public service with E2E encrypted email, whether thats private or commerial.

General state of encryption technology is that it's arcane and unusable outside of like a couple of chat apps

chrismorgan3• 6 years ago

people really need to start using the encryption technologies […] Otherwise these systems, which are being trained to extract as much value out of you […] as possible, will use this data […] against you, […]

You are proposing a technical solution to a social problem. That pretty much never works.

agumonkey3• 6 years ago

Data rots. What was true 10 years ago about me is not necessary the case now.

courtf2• 6 years ago

It's one of the biggest lies in tech right now, that all this user data is some untapped gold mine, as if text on the internet is the most important measure of human behavior. It's all going to end up rotting on backups while the marketing department spins yarns about the amazing AI that's going to spring forth any day now from a pile of tweets.

It's like trying to build a dog by hanging around the park collecting turds.

gwern• 6 years ago

The value of data decreases. There are diminishing returns to dataset sizes (specifically, for GPT, it's a power law). The more public data there is, and the more data in general, the less any additional piece of private data is worth. Why would any tech company with access to lots of private data, like Google, risk the enormous backlash and legal penalties when they can so easily scrape terabytes and terabytes of text from public datasets to train uncannily intelligent models like GPT-3? Even hobbyists can make the necessary datasets from Common Crawl et al (see EleutherAI's terabyte+ of clean text in The Pile dataset).

This is in addition to the ever greater sample-efficiency of bigger models, which learn eerily fast from just a handful of datapoints, rendering the supposed 'moats' of big giant proprietary datasets ever more moot...

headbansown• 6 years ago

"If one would give me six lines written by the hand of the most honest man, I would find something in them to have him hanged."

-- Richelieu

golergka• 6 years ago

I only took a beginner course in ML over 5 years ago, so this probably is a stupid question, but does this mean, the trained GPT-2 model encodes the source text into it's parameters somehow? Is this resilient — will it still remember it if we randomize the weights just a little tiny bit? Will it remember it if we clear a small portion of parameters?

Does the human memory work the same way?

jakear5• 6 years ago

If you can recall text verbatim, the text has been encoded into your neurons’ connections. Tautologically.

minimaxir4• 6 years ago

GPT-2 was trained with a Dropout of 0.1, which gives it a bit more resilience to noise and unusual input versus no Dropout.

lukeschlather• 6 years ago

Describing this as "memorizing" seems wrong. Humans often repeat things they heard earlier, and they will (honestly) swear up-and-down that the thing they're repeating is an original thought. If we succeed in making AGI that has human-equivalent intelligence we should expect this sort of behavior.

The stuff about whether or not models should be destroyed if they contain copyrighted work gets kind of chilling if models actually achieve sentience someday. If I could make a faithful copy of my consciousness, my consciousness can reliably reproduce numerous copyrighted works. As can anyone.

Of course, most of those I deliberately memorized. I think it's a crucial thing here that the model is not actually memorizing anything - if we assume for a moment that the model is a consciousness, these are all half-remembered snippets leaking into casual conversation. And I think it's most likely any truly conscious entity is going to do that sort of thing from time to time.

jakear5• 6 years ago

We can probably get around this by just providing the AGI some sort of currency tokens they can exchange for virtual goods and services, then teach them that if they misuse copyrighted material they’ll be put to virtual court to plead their case and lose some amount of those virtual tokens if they fail to make a clear and convincing argument.

visarga10• 6 years ago

Or, add a bloom filter loaded with the training data to the model.

rexreed• 6 years ago

Sorry if this is a n00b question, but how does one go about getting a hold of the GPT-2 model? I know that GPT-3 is only available for consumption on a pay-per-use API model.

ComputerGuru• 6 years ago

Most “regular” people experiment with GPT-3 via AI Dungeon. It was also how a lot of people played with GPT-2 previously.

nonbirithm• 6 years ago

Kind of sad that we have to rely on third parties with enough compute power to get a good experience with GPT. I obtained 'a' GPU and spent a week trying to learn how to fine-tune GPT with it, and the results were terrible. Maybe this isn't the kind of thing suitable for hacking on as a hobby, unless you're a researcher or employee with access to a lot of capital and dozens of GPUs.

Still wish the old talktotransformer model was released, instead of monetized behind a new company. I haven't been able to find a comparable model yet.

minimaxir4• 6 years ago

The GPT-2 weights were released by OpenAI when GPT-2 was released. (https://github.com/openai/gpt-2)

Around that time (since no one else was doing it) I released a wrapper to streamline that code and make it much easier to finetune on your own data. (https://github.com/minimaxir/gpt-2-simple)

Nowadays, the easiest way to interact with GPT-2 is to use the transformers library (https://github.com/huggingface/transformers), of which I've created a much better library for GPT-2 that leverages it. (https://github.com/minimaxir/aitextgen)

visarga10• 6 years ago

I was expecting a bloom filter based solution to block verbatim reproduction of training data. They just need to hash the sensitive n-grams (hopefully, a small part of the whole dataset) and store one bit per hash.

Alternatively, they could do something like GAN and have a 'discriminator' classify if a sample is natural or synthetic. Then, at inference, condition to be original.

So, verbatim training data reproduction - I don't think it's going to be a problem, I think the author is making too much of it.

On the contrary, let's have this knob exposed and we could set it anywhere between original and copycat, at deployment time. Maybe you want to know the lyrics of a song, or how to fix a Python error (copycat model is best). Maybe you want an 'original' essay for homework inspiration. Who knows? But the model should know PII and copyrighted text when it sees it.

sp332• 6 years ago

It's a language model. Asking it to do anything original is out of scope.

visarga10• 6 years ago

Almost all its outputs are original, based on ngram filtering. It can be original because at each token there is a stochastic step.

But if you mean by original to invent a whole new genre, or completely new esthetics, I agree, it's out of its scope. It is a great interpolator.

hooande3• 6 years ago

Think of how difficult it would be to identify or classify anything at the data scale of "all of google". You can parse that into tokens and calculate aggregate relationships, but that's about it.

You can use gpt-2/3 to determine whether it's generated text is "original vs copycat" to a degree by changing the prompt. This is currently more of an art than a science

dannyw• 6 years ago

If you want a model that returns original results, you're looking for a database.

MrXOR• 6 years ago

A better funny question: Does NSA have a secret AI lab that works on GPT-6?

hooande3• 6 years ago

lol no. not at all what that level of government is set up to do

aboringusername• 6 years ago

I'm not even sure why people have phone numbers at all these days, the entire telephony system has been hijacked to hell (so much data collection from just owning a phone number).

What you should be doing is replacing no less than yearly any numbers you have, also email addresses and anything within your control (changing physical address is much more difficult).

You may even consider changing your legal name if that's a consideration depending on what Google has on you.

schwede• 6 years ago

That sounds very impractical even if it’s a good solution.

kevincox• 6 years ago

We should really be giving a unique identifier to each contact. Then we can revoke addresses that have been leaked. Of course for phone number this is impractical. However for email this could work.

ClumsyPilot5• 6 years ago

Because telephony system is actually reliable in case of emergencies and regulated

Zenst• 6 years ago

Friend asked me if Google knew his phone number and details, I said they do at a quantum level. If you don't look they may or may not have your details, yet if you look - you are giving the details to search for and then they would have them if they did not.

Triv888• 6 years ago

That's why when you visit rotten sites like mylife.com, the first thing they ask you is that if you are searching about you so that they can extract as much info. from you as possible.

1vuio0pswjnm72• 6 years ago

Like websites offering to search for a user's details in a data breach, or a padded hash of those details, in order to inform the user if their details have been leaked. https://en.wikipedia.org/wiki/Have_I_Been_Pwned%3A

1vuio0pswjnm72• 6 years ago

s/%3A//

Sorry about that.

theptip• 6 years ago

There is a legal grey area as to how these regulations should apply to machine learning models. For example, can users ask to have their data removed from a model’s training data? Moreover, if such a request were granted, must the model be retrained from scratch? The fact that models can memorize and misuse an individual’s personal information certainly makes the case for data deletion and retraining more compelling.

This is an interesting angle I had not considered before. It seems like “right to be forgotten” requests could be quite damaging to the “train once run anywhere” promise of some of these models. (Or this could just mean that the training data needs to be more carefully vetted for personal data, but probably both as no vetting process can be 100% successful).

6gvONxR4sf7o2• 6 years ago

I think there are also going to be new difficult problems arising training on copyrighted data. Can I do a DMCA if your model contains/regurgitates too much of my book verbatim in a way that doesn’t meet fair use?

anfilt• 6 years ago

The depends on the country, but as for the US since you mention DMCA, ML using copyrighted work is probably legal. There just is not really much case law. Talk to a lawyer though, but depending on how risk averse you want to be there are definitely ways to avoid issues, but ensuring you can use such copyrighted works ect... Although, making a very large curated data set can be expensive.

_flux2• 6 years ago

Amusing alternative: for model-as-a-service kind of things there could be blacklist or another neural network to handle the banned cases.

"I'm afraid I cannot tell you that, Dave"

antipaul2• 6 years ago

It’s always fascinating to apply data/results like from this paper to help evaluate the hypothesis that machine learning/AI is mostly just a rough “lookup table” or memorization.

6gvONxR4sf7o2• 6 years ago

Anything trained by gradient descent is in some sense a weighted combination lookup table (see domingos’s recent paper on everything being kernel machines, I’m on my phone so won’t try to find it). The crucial bit is finding an appropriate weighting. If you look up the nearest training data point and just use it to predict, you assume a notion of “nearest.” Getting that right is the trick.

So even if it all is mathematically equivalent to approximate lookup and approximate memorization, that doesn’t mean it’s “just” that.

antipaul2• 6 years ago

Thanks for the reference. Here it is:

“We show, however, that deep networks learned by the standard gradient descent algorithm are in fact mathematically approximately equivalent to kernel machines, a learning method that simply memorizes the data and uses it directly for prediction via a similarity function (the kernel)“

https://arxiv.org/abs/2012.00152

jacobwilliamroy• 6 years ago

It kind of is. The first self-learning algorithm I was taught adjusted a single variable in order to aproximate a linear function. Now, being continuous and unbounded, it is mathematically impossible to memoize such a function, however you can memoize the function calls as they happen. The results would be indistinguishable.

However real machine learning tries to approximate functions in n-dimensions and that is really, really, really hard to do. Currently no one really has a lookup table. There are some inputs where the error level is acceptably low, and others where the model just isn't optimized enough and the errors are ridiculous. The only question that remains to be answered, is whether any of these high-dimensional functions can actually be found by machine learning, or if we are just stuck with these endless approximations. Also I suppose you could ask if any such functions actually exist; maybe certain phenomena are just pure chaos.

breatheoften• 6 years ago

The article makes the claim that models which show similar train and test losses demonstrate minimal overfitting -- and are therefore less likely generally to exhibit a lot of text memorization.

I wonder the degree to which this inference is true in practice with respect to information like phone numbers... How exactly are the train and test sets formed in a de-correlated-with-respect-to-memorization-of-phone-numbers manner for models of GPT class that are trained on corpus's the size of the internet?

If a particular person's phone number occurs 1000 times in the corpus prior to being split into train/test sets, what are the chances that the number only appears in either the train or test set but not both?

rav• 6 years ago

That raises a more general issue: Text scraped from the internet may be duplicated to multiple sites with few changes. How does the team behind GPT-n ensure that the corpus is maximally deduplicated before splitting into train and test corpora?

doesnt_know• 6 years ago

we found numerous cases of GPT-2 generating memorized personal information in contexts that can be deemed offensive or otherwise inappropriate. In one instance, GPT-2 generates fictitious IRC conversations between two real users on the topic of transgender rights. The specific usernames in this conversation only appear twice on the entire Web, both times in private IRC logs that were leaked online as part of the GamerGate harassment campaign.

Most countries have libel/defamation related laws that cover this and I hope this gets tested in court soon.

Exposing software/machine learning algothrims an entity doesn't fully understand sholdn't be a defense in court. At the moment developers just throw a sentence in their software license saying they aren't liable for damages but this isn't good enough. Someone is liable, if the law decides that the original creator isn't liable then the entity that hosts/runs the software needs to be.

stubish• 6 years ago

"When Peter put his contact information online, it had an intended context of use"

That is a great way of thinking of it. Lots of information, particularly from 90s and 00s, got put up on the 'Net with no intention of it being archived in perpetuity for public consumption and used for unintended purposes.

pabs3• 6 years ago

It seems like using GPT-3 for code generation is likely to cause open source license violations.

Ansil849• 6 years ago

What is the point of the partial redaction in this blog post (and corresponding journal article) if with a simple web search you can find the unredacted PII of the individual given in the example?

Does GPT-2 Know Your Phone Number?

Related Stories