Revealed: Bias found in AI system used to detect UK benefits fraud

https://www.theguardian.com/society/2024/dec/06/revealed-bias-found-in-ai-system-used-to-detect-uk-benefits

by jimnotgym • 2 years ago

4 6 2 years ago

smidgeon• 2 years ago

Clearly trained on Benefits Britain: Life on the Dole

figassis2• 2 years ago

Government departments, including the Home Office and the DWP have, in recent years, been reluctant to disclose more about their use of AI, citing concerns that to do so could allow bad actors to manipulate systems.

As we speed into a future where AI will likely be everywhere and fully control our lives, being afraid to say whether certain processes use AI because it can be manipulated is the same as having low confidence in the reliability, consistency and fairness of such AI. I’d say every process that uses AI should clearly state which AI is being used, which model/version, what data it was trained on, whether the dataset is approved, etc. I for one think AIs that use prompt engineering and where the user input can directly affect how such prompts are executed are not the AIs to use in any kind of critical work that affects people at scale.

Ok another note, if a group is more likely to commit fraud (maybe due to culture, religion, social status, wealth level, etc), and the system takes these into account so that it disproportionately negatively affects such group, is it being biased? I think not, but it’s is probably being unfair. Being fair in this case might mean being biased, but can a government create/use such a positively biased system? Isn’t that what led to the university admissions lawsuits in the US? So what would the solution be here?

eesmith2• 2 years ago

if a group is more likely to commit fraud (maybe due to culture, religion, social status, wealth level, etc), and the system takes these into account so that it disproportionately negatively affects such group, is it being biased?

Who gets to define what the group is?

If the system sees that members of Reformed Baptist Church of God are more likely to commit fraud, but can't distinguish between the good members who follow the reformation of 1879, while it's the heretics who follow the reformation of 1915 who are committing all the fraud, then yes, it's a problem.

(Examples drawn from https://kozar.stanford.edu/EmoPhillips.html .)

figassis2• 2 years ago

So problem here is no one will define a group, and a system will not automatically say if religion = foo then bar. Things will just statistically start going this way. Might not even be negative bias. If people with college degrees start appearing as less likely to commit fraud, then it might become biased toward degrees, discriminating against those who truly need help.

beardyw• 2 years ago

If it targets groups in proportion to their likelihood to commit fraud then the outcome should itself be in proportion. I think the difficulty comes when people get lazy and only target one group. Or am I wrong?

Revealed: Bias found in AI system used to detect UK benefits fraud

Related Stories