SafeGPT: New tool to detect LLMs' hallucinations, biases and privacy issues

https://www.giskard.ai/safegpt

by Googleton • 3 years ago

114 82 3 years ago

Is there a way to test it? Curious to see if it's directly embedded in ChatGPT with an add-on on top of it or it's something outside of it?

Googleton2• 3 years ago

The waitlist is here for this, we're still actively working on it !

It is directly embedded in the ChatGPT website when you get the extension. As you ask questions, it will be added on the sidebar of each answer

anonzzzies• 3 years ago

How does this work? Does anyone know?

And for a large swats of things, how can it possibly work? It’s not possible to say if or if not it is hallucinating code for almost all code and apis, for instance. And I see similar issues with many fields outside pure facts. With privacy issues as well.

upghost• 3 years ago

Looking at their “documentation”: https://docs.giskard.ai/start/

It would appear that this is not automated monitoring but more like a second stage of human reinforcement learning or perhaps a classifier. It seems that you create input/output examples and the LLM responses are examined by a secondary system (which I’m guessing is probably NOT an LLM, otherwise it would be vulnerable to attacks) and perhaps force regenerates the LLM response if it doesn’t meet the classification threshold.

At least, that sounds more believable to me than someone claiming they’ve fixed the inherent flaws in LLMs.

Googleton2• 3 years ago

We are a team of engineers & researchers on AI Alignment & Safety, we're investigating multiple methods, including metamorphic testing, human feedback, benchmarks with external data sources, and LLM explainability methods.

Currently, fact checking works on straight facts. It does a Google Search and uses LLMs to shorten it. Once it has the short version, it will compare the short results with the answer provided by ChatGPT itself. Premium tiers would get better fact checking sources than just google. We're investigating various data sources and comparison methods.

Note that fact checking / hallucinations is just one of the types of satety issues we'd like to tackle. Many of these are still open questions in the research community, so we're looking to build and develop the right methods for the rights problems. We also think it's super important to have independent third-party evaluations to make sure these models are safe.

This is a new tool we're building in the open, and we're interested in your feedback to prioritize!

bheadmaster• 3 years ago

Currently, fact checking works on straight facts.

Wow, you guys have a database of all the facts?

It does a Google Search and uses LLMs to shorten it.

Oh...

...actually, this is an empirical fact checker. I wouldn't call it "fact-based", as it's epistemologically an absurd statement, but "empirical fact checking" sounds good and presents an idea that is very close to how humans verify information in the first place - by checking multiple sources and searching for correlation.

For what it's worth, I think your approach makes sense. Good luck.

danShumway2• 3 years ago

Currently, fact checking works on straight facts. It does a Google Search and uses LLMs to shorten it.

So your fact-checking LLM is also vulnerable to injection and unethical prompting then when it ingests website text. And a Google search is far, far away from fact checking, particularly for the subtle errors that GPT-4 is prone to making.

bjt2n39044• 3 years ago

This is like saying, "I've developed a new compass for a deep space probe to help it find North!"

Our society is actively declaring that falsehoods are truth, and should be celebrated. We're hallucinating ourselves. All this software does is make sure LLMs hallucinate with us.

plagiarist• 3 years ago

Such as?

FeepingCreature2• 3 years ago

It's impossible to answer this without getting political. Instead, let's just say every previous generation has been critically wrong about some things. Statistically, we're unlikely to be the outlier.

foobarbecue• 3 years ago

Such as religion, for example.

autophagian• 3 years ago

I think its easy to guess what the commenter is obliquely referring to.

fdschonborn• 3 years ago

People who write with their left hand existing.

becquerel• 3 years ago

Generally when people talk like this nowadays they mean trans people, or the LGBT community in general. Sometimes Jews, though those types don't say that part out loud on HN too often.

aftbit• 3 years ago

There were lots of good examples during COVID. Remember that you don't need to wear masks, because washing your hands is enough (and we need to save the masks for doctors, but we're afraid to say that, because it will cause a run on masks). Remember that staying 6 feet apart is a magic distance over which COVID cannot cross (or maybe 1 meter, truth depends on the country you live in). Remember that you can't eat inside a restaurant, but you can eat outside, and it's okay for the restaurant to build partial walls around their outdoor spaces to make them more comfortable. Remember that COVID definitely could not have come from a lab leak, and it's racist to even suggest it might have happened. Never mind that the scientist who started the anti-lab-leak open letter was himself heavily funded for GoF research, and he refused to sign his own letter for political reasons.

I don't claim to know the answers to all of the questions (and I certainly don't know where COVID came from), but clearly there are plenty of cases where dubious statements were strongly enshrined as "True" in a way that required major online players to suppress alternative beliefs as "False".

Ialdaboth• 3 years ago

I'm pretty interested and fascinated in all the efforts that go into making sure it lies as convincingly as us.

bjt2n39044• 3 years ago

I firmly reject that GPT is sentient, or can even tell truth. You're no better off asking it questions about society than asking a magic eight ball.

And yet there's something sinister about twisting an evolutionary model into a reeducation camp.

LLMs are not human, but strikingly similar. If we have no qualms about how we treat it, then what will we do to real people?

bluescrn• 3 years ago

Before long, we could end up with left-leaning and right-leaning AIs autonomously fighting the 'culture war' over social media, much more advanced than simple bots spamming copy+paste comments.

Combined with ever-improving ways to fake video and voices, things could get even uglier than they've been over the last few years.

bjt2n39044• 3 years ago

Alternatively, we're reaching the point where we're creating a secondary AI to keep the first AI in check, like Wheatley and Glados.

mountainb• 3 years ago

Well I think that its hallucinations are a good demonstration that none of the post-truth subjectivist philosophies were ever things that many people took very seriously. ChatGPT is the real thing as it relates to extreme relativism: it really cannot tell the difference between true and false and doesn't really care, either. By contrast, the apostles of post-truth were really only trolling for a response... no one really lives by its principles because it's so impractical and disastrous. To really believe in post-truth, you must perceive no difference between raisin bran with cyanide and raisin bran.

sebzim45003• 3 years ago

Seems too good to be true, and I don't understand what it even means for an LLM to be unbiased.

hospitalJail• 3 years ago

They could use different models to test it. They could use common biases and test those.

Turing_Machine• 3 years ago

"Unbiased" almost always means "has biases that are similar to mine". I can't think of very many exceptions to that, frankly.

flangola72• 3 years ago

That's obviously untrue after more than three seconds of critical thought

henri183• 3 years ago

There are many bias detectors developed in the research world. To have a deeper look, you can look at this paper: https://arxiv.org/abs/2208.05777

thfuran2• 3 years ago

One with only weights and no biases in the ANN is unbiased.

taneq2• 3 years ago

I feel like this went underappreciated. [b] = [0] indeed.

sebzim45003• 3 years ago

Someone I know thought that LLaMA was unbiased because they 'read' the paper and clearly didn't know what anything meant. A great example of "a little knowledge is a dangerous thing".

taneq2• 3 years ago

It doesn't. Mean anything, I mean. Language isn't well defined, so its accuracy is also undefined and a non-uniform deviation from that accuracy is super undefined.

nickwritesit• 3 years ago

These are the kinds of things I can see taking off, for better or worse. I know Adobe's product is worse than Midjourney's, for example, but once the hype meets reality, companies are going to want to be safe when they start using AI formally.

fatso784• 3 years ago

Looks a bit like snakeoil to me. A lot of companies now spinning up simple demos with opaque backends, making huge claims they’ve solved X hard problem for/with AI, then saying “trust us” and “join our waitlist” without hard details or facts to show for it. If you could detect hallucinations/biases etc that easily, don’t you think OpenAI would’ve worked on something like this?

henri183• 3 years ago

It's good to have third parties (apart from Open AI) that assess the quality of Open AI results. It's the way audits work, it has to be independent... Also, third parties are essential to compare the results from ChatGPT with the results of other LLMs. These are important checks to assess the robustness of OpenAI results!

jsheard• 3 years ago

I can't help but notice your accounts only activity before this post was praising another giskard.ai submission a few months ago. Anything you'd like to disclose?

amitport• 3 years ago

He didn't say it's not important. He is just pointing out that black-box third party verification is not worth much when you can't independently verify the verifiers.

mirker• 3 years ago

The OPs point is that it’s likely impossible to do what is claimed here in general. Imagine the LLM says something like Fermat’s Last Theorem. To verify it, you’d have to either 1) have a proof assistant powerful enough to construct a proof 2) use a second ML model to guess truthfulness. The former is technically challenging and the latter is another model, with its own biases and factual inconsistencies.

MuffinFlavored• 3 years ago

don’t you think OpenAI would’ve worked on something like this?

Along this line of thought: was it a massive oversight for them to not train the model to say "math detected, let me pass that to a solver" instead of trying to guess what token should come next in a math problem?

TazeTSchnitzel• 3 years ago

There's a million categories of problem you could ask an LLM to try to solve. You'd need a million solvers…

kolinko• 3 years ago

They are solving that with plugins now.

roflyear• 3 years ago

This isn't new it's just more obvious with this tech. Every sales team at nearly every company has been performing this dance for like hundreds of years.

danShumway2• 3 years ago

In this area, if there's not a public demo and the results aren't verifiable, then it's not worth paying attention to.

kmod• 3 years ago

I think part of the problem is that it's technically correct to say "my product does X" even if it does X extremely poorly. I'm not sure if this can be changed because any line for "does-X vs does-not-adequately-do-X" is going to necessarily be subjective.

So personally I think the problem is that people see "this product does X" and interpret that to mean that it does X well. I don't think it's necessarily bad that we're seeing an explosion of AI tools that are a bit underwhelming if people understood it as such -- we're on, after all, a site with a heavy startup focus and saying "your product doesn't do everything that I want" is a bit antithetical to that.

But yeah specifically for this one there are arguments that "X is not even possible, especially not with this approach" so it's a bit more egregious.

ulfw• 3 years ago

AI is the new crypto (though with more substance). Attracts many of the same self obsessed snake oil selling characters though.

cudgy• 3 years ago

True. I imagine many crypto startups have desperately pivoted to AI with their last gasp of cash given the recent blowup.

crop_rotation3• 3 years ago

What does it even mean to detect hallucinations. The AI doesn't say something trivially false. While using GPT4 I have observed that it lies on simple things I didn't expect it to, while complex things it does very well on.

TLDR: It lies on fact based information which is mentioned in very very few places on the internet and not repeated too much. Short of having a human with the context, how do you even detect it.

Example: Ask it to describe a "Will and Grace" episode with some guest appearance. It will always make up everything including the episode number and the plot, and the plot seems very believable. If you have not watched and can't find a summary online, it is hard to say that it is a lie.

QuercusMax3• 3 years ago

But isn't that half the reason people are so excited about this stuff - that you can ask it to make up an episode and it does a plausible job.

crop_rotation3• 3 years ago

That is beside the point. My point is that detecting hallucinations seems like a very very hard problem.

The utility of it is there and has nothing to do with making up episodes instead of quoting the current one. Like you can ask it to write new episodes with specific settings and specific constraints. Hallucination is not the value add. Nobody is excited because it hallucinates. People are excited despite it since the other value add is too much.

thfuran2• 3 years ago

But deliberately requesting and receiving content generation is altogether different from requesting a factual answer and receiving plausible-seeming nonsense. Or at least, it's different to the person asking; it's the same thing as far as the model is concerned.

pornel• 3 years ago

If the hallucinations are random, how about generating multiple answers, and then checking whether they all agree on key points?

henri183• 3 years ago

There are many ways to detect hallucinations. Basically, either you have the ground truth answers in external database, in that case you compare to ground truths. Or you don’t have the ground truth. In that case, you need to do metamorphic testing. See this article on it: https://www.giskard.ai/knowledge/how-to-test-ml-models-4-met...

crop_rotation3• 3 years ago

But GPT4 doesn't hallucinate on things which are popular enough to be replicated enough times on the web as knowledge. It hallucinates on things which are very less likely to be repeated many times. That rules out an external database with true answers. Unless the external database is supposed to contain all info queryable in all ways, in which case the database is just a better version of GPT-X.

The metamorphic testing approach is interesting and might work.

krona• 3 years ago

Metamorphic testing seems to try to map an output of a model to a ground truth, which I guess is great if you have a database of all the known truths in the universe.

throwoutway• 3 years ago

This is exciting to see, as I am concerned about the hallucinations, biases, privacy, licensing, etc issues. I imagine the results are minimal at the moment, but perhaps soon they will be useful

Tepix• 3 years ago

This should be interesting: If they don't catch some hallucination they may be liable and can be sued…

rig777• 3 years ago

$10 bucks says this just uses the GPT4 API to go over gpt chat.

mxcrbn• 3 years ago

Good to see initiatives like this one popping up. Congrats on the launch Giskard team!

SafeGPT: New tool to detect LLMs' hallucinations, biases and privacy issues

Related Stories