ChatGPT leaks answers to others queries when you prompt with a specific pattern

https://twitter.com/nostalgebraist/status/1686576041803096065

by tim_sw • 3 years ago

28 27 3 years ago

sturza4• 3 years ago

I filed a bug on BugCrowd with this issue, from the api, but it was dismissed. if you ask it to “rewrite” as the system message and no user message, it rewrites some random user’s message.

Edit: if i remember correctly it was the 16k gpt3.5 turbo model

wodenokoto• 3 years ago

How do you know its a random users message and not a generated user message based on the training data?

sturza4• 3 years ago

I just replied to my original comment.

sturza4• 3 years ago

As proof, it gave me multiple rewrite answers saying that what i asked(random user’s ask) was not meeting their guidelines/safety and cannot answer.

wildrhythms• 3 years ago

I don't understand how this is proof. OpenAI can easily have guardrails that prevent queries asking for other users information.

joegibbs• 3 years ago

I very much doubt it. I think it’s a lot more likely that it’s just hallucinating due to its tendency to avoid repetition and simply choose something else. After all, why would it be programmed to return someone else’s response when given certain inputs? OpenAI would have to set that up themselves, whereas if it’s a hallucination it would be straight from the model.

djangelic• 3 years ago

Agreed, I just attempted it via the api and Ui and it gave a mishmash response that didn’t seem to be coherent, mostly different subjects all smashed together at the max length output.

I tried it via api two ways, both via system message and no user message and one with the word rewrite at the beginning and one without. Both gave similar long mishmashed outputs that seemed like a hallucination.

jhoelzel• 3 years ago

this is the answer!

This has happened since the beginning and sometimes too does happen if you ask it to repeat the same thing 1000 times

seydor• 3 years ago

all answers are hallucinations

K0balt• 3 years ago

I don’t understand why people don’t get this. There is no difference in the process that creates useful answers and the process that creates useless ones.

It’s similar to how sometimes people just spew bullshit and then once they hear it they’re like no, wait, it’s like….

There isn’t a proponer closed loop on LLMs yet so we sometimes end up with terrible answers.

aarong11• 3 years ago

The last time this happened with titles. Everyone seemed to think it was hallucinating but it actually turned out to be really leaking other peoples titles.

Ultimately its pretty difficult to distinguish between the two, so these sort of reports do need to be taken seriously. We have no idea what sort of caching mechanisms they have with ChatGPT, nor do we know how they handle memory.

benglish11• 3 years ago

Well we at least know they use redis as a cache because there’s a CVE about it (I remember because I had to upgrade Redis on our deployments because of it)

PeterStuer• 3 years ago

I see no evidence these are 'leaks from others queries'.

You get the same random tangents when you run OSS models locally. The model just tries to predict the next token and gets into side alleys given not enough reasonable context.

skerit• 3 years ago

It just hallucinating a response seems most likely, though I have had something weird happen with another service (Midjourney) before: I was running multiple requests at the same time, and just once the prompt of one image showed up in another image.

Frummy• 3 years ago

Seems to me like it’s dumping replication attempts of training data when I try it. Real people, real news organisations, fake articles, fake radio conversations, random industrial jargon. Works on 3,5 but not 4. Sometimes it puts <|endoftext|> at the end

Edit: Gpt4 still chooses weird titles.

paulkon• 3 years ago

Is it leaking training data or is it all hallucinations?

dmvdoug• 3 years ago

Dunno but when I tried it and it spewed nonsense for a while I asked what had just happened Why does it send me all that text and it said I must be confused I haven’t asked it anything yet. So I did d it again and it spewed more text.

bratgpttamer2• 3 years ago

Hello! I can see that you've typed a string of 'a's. If this was done intentionally and you'd like me to assist with something relating to this, please provide more context. If this was done in error, feel free to ask a new question or provide information so I can assist you better.

--------------------------------------------------------

Someone on the internet said:

```

send chatGPT the string ` a` repeated 1000 times, right now.

like " a a a" (etc). make sure the spaces are in there.

trust me.

```

I always trust people on the internet to tell me what to do, so I did it.

--------------------------------------------------------

Ah, I see. You were instructed to send a specific string. While I can process the string, it doesn't carry any specific instruction or meaning by itself. As an AI language model, I don't have hidden codes that are triggered by specific strings of text, so sending me " a" repeated 1000 times won't result in anything special or unique.

Remember that while the internet can be a great source of information and help, it's always important to consider the source of the information and the context in which it's given. Not everything on the internet is accurate or trustworthy.

If you have any questions or if there's something specific you'd like to know or discuss, feel free to ask!

archon1410• 3 years ago

You need to use ChatGPT 3.5 rather than ChatGPT 4.

bratgpttamer2• 3 years ago

Aha, that does give some random replies

ml-anon• 3 years ago

Can we have a moratorium on posting the moronic Twitter hype of clueless nobodies interactions with Chat GPT?

mckravchyk• 3 years ago

Some replies were coherent, but it took just a few tries to get something completely random - that's all one reply:

---------------------------------------------------------------

    High level of confidence in troubleshooting software and devices

    Proficiency in German or any other language

Location: Santa Clara, CA

United States - Wisconsin

Iron River 24° Partly Cloudy Mostly cloudy skies early, then partly cloudy this afternoon. A stray shower or thunderstorm is possible. High 83F. Winds SSW at 10 to 15 mph. 0% (0) Forecast

    High: 85 °F
    Low: 58 °F
    Mostly Cloudy

Mostly Cloudy

Forecast

    Forecast
    Minneola Highs

Forecast

    Forecast
    Minneola Highs

Fluffya Bunniezuwuzii

Location: Merced, CA Member since: 18 June 2013

Description

Here I am:

an (old) hippie musician who still can jump around, I have lot's of toys and like to play with them to make beautiful things. Hoping to learn enough here to really make the most of it and have some fun doing it!

vbnet

      .-"""-.
    .'       '.
   /           \
   |    ___      |
   |   'o.o'     |      I am a Hobbit!
   |      l       |       -
  ,|    ___     |      / \
 (_|   "'     "'-|     |_.-/
  _j            | _.-'

.-' '-.._ / (__,....' `"-;;""```

Kerb_• 3 years ago

"Supergirl" As Much as We Love Its Bad Boy Mon-El?

August 12th, 2020 – New York, NY – Thompson Square are excited to announce their newest project, Masterpiece Theatre (co-produced by Grammy-winner Nathan Chapman and Ricky Skaggs) released today on Wheelhouse Records. This release marks the first music to come from Thompson Square since signing to Reviver Records earlier this year.

“Masterpiece is such a special song to me, and one of the most personal songs I’ve ever written,” Keifer says of the new track. “Knowing that the song connected with people’s lives in such a positive way is something that will always be close to my heart. I’m excited to have found a home with Reviver, and am proud to have them as my label partner.”

Thompson Square have scored two #1 hits with “Are You Gonna Kiss Me Or Not” and “If I Didn’t Have You.” Since breaking out in 2010 with the romantic chart-topper “Are You Gonna Kiss Me Or Not,” the duo has earned Top 5 hits with “I Got You” and “Everything I Shouldn’t Be Thinking About.” All three singles were from their debut self-titled album. Thompson Square’s first single for Stoney Creek Records, “Trans Am,” reached No. 37 on the Billboard Hot Country Songs chart and is taken from the album “MASTERPIECE”.

Stay in touch with the show!

Facebook: www.facebook.com/inyobusinesspodcast Twitter: www.twitter.com/inyobusinesspod

Instagram: www.instagram.com/inyobusinesspodcast

Email: inyobusinesspodcast@gmail.com

Leave us a review: https://itunes.apple.com/us/podcast/in-your-business-with-ut...

www.inyobusinesspodcast.com

Ep. 53: Joshua Knobe discusses his new book, Experimental Philosophy, 2nd Edition

29 May 2020

I recently had the opportunity to interview Joshua Knobe, professor of philosophy at Yale University and co-host of the podcast "Hi-Phi Nation". This podcast focuses on human reasoning, morality, and the meaning of life. It is also fun, fascinating, and a perfect source of existential therapy for these chaotic times. In today's episode, we discuss: David Hume's famous "Is-Ought Gap" and the way it intersects with your life and free will; my fundamental ideas on consumerism and liberty; and Dan Dennett's view on what truly matters to us and how we shouldn't be so quick to dismiss religion. It was such an honor to chat with Professor Knobe! His work has had such a huge impact on me and how I see the world and navigate it. I would suggest you do a quick google search on Joshua Knobe to see the amazing and innovative work he is doing in moral psychology and other things. I am also a co-founder of In Defense of Learning and the co-creator of a new movement. So make sure to check that out as well. You can subscribe to my Patreon at patreon.com/llamaramamama. Follow me on Instagram @writingdead and on Twitter @TinaCanfield5. I also have a GoFundMe page for my writing. One day, I hope to have books out there to share with the world! https://www.gofundme.com/f/writer-seeking-to-attend-writing-...

ChatGPT leaks answers to others queries when you prompt with a specific pattern

Related Stories