Ask HN: Anyone else had problems with ChatGPT returning real looking, fake DOIs?

https://twitter.com/stophammotime/status/1629826270338121728

by StopHammoTime • 3 years ago

3 3 3 years ago

I'm trying to use ChatGPT as a search engine to see how effective it is at assisting me with things I'm researching. However, I've noticed that multiple times it just makes up journal article titles and pairs them with metadata from actual journal articles. I haven't actually been able to get a DOI that has linked to an actual article yet. When you tell it, it's done something wrong it just makes up completely random DOIs that you can't even find in a search.

I know it's not meant to be trustworthy as a data source, but this seems like ChatGPT is just outright lying to me?

For context, I asked the following questions: 1/ What problem did tools like flightradar24 solve and can you provide me some academic references outlining these 2/ Can you give me DOI references for those three references please

I actually can't find the articles in the listed journals at those locations either so it's not just a metadata match issue. The articles just do not seem to exist at all.

Linked my twitter post with images of the real articles along with the list that ChatGPT supplied to me.

williamcotton• 3 years ago

I would imagine that those are not actual academic papers and that the ChatGPT consumer web application is not the right tool.

Try using the OpenAI APIs directly, but through few-shot examples train it to return some interpreted language like Python or JS, which has the ability to execute further completions, but with the added context of a search from a web search or better yet the search results from an academic journal search engine.

Here’s the general approach:

https://github.com/williamcotton/empirical-philosophy/blob/m...

uberman• 3 years ago

Nothing chatGPT returns should be taken as absolute truth. That would be like attempting to find the real people at this "party" https://nypost.com/2023/01/16/ai-generated-party-pics-look-e...

Ask HN: Anyone else had problems with ChatGPT returning real looking, fake DOIs?

Related Stories