GPT (text-DaVinci-003) can run hallucinated scripts

https://twitter.com/jamesjyu/status/1600959843669377025

by jamesjyu • 4 years ago

17 2 4 years ago

i had something similar happen but with chatgpt (idk if that uses the same model?)

i tried to get it to write a web scraper but the code wasn't very good. so i fed it the html and told it to write some code that outputs the results in a certain way. the code was still not good. then i input some code that i had already written to do this exact task, including a print statement at the end. then i asked it for the results of the print statement. it printed the right information and the original structure i told it.

it was wild. at first i thought that i had somehow made it run code then i realized that it was able to logically produce the result, kinda like how i can read code and surmise the outcome. maybe? i have no idea.

unfortunately i closed out the session and forgot what exactly i had said to do this, because i tried to keep doing it again but it told me that it couldn't run code.

jamesjyu• 4 years ago

yes exactly! it's able to "guess" the output of the code without running it in the traditional sense. I've even able to get it to do loops correctly. This is kinda the reverse of it producing code based on comments.

But it goes further: now you can make up API function calls that don't exist and compose them with code that does. It feels like the entry to an entirely different space of prompt engineering.

GPT (text-DaVinci-003) can run hallucinated scripts

Related Stories