Two women had a business meeting. AI called it childcare

https://medium.com/hold-my-juice/two-women-had-a-business-meeting-ai-called-it-childcare-6b09f5952940

by sophiabk • 8 months ago

23 32 8 months ago

sophiabk2• 8 months ago

We’re building a family AI called Hold My Juice — and last week, our own system mislabeled a recurring meeting between two founders as “childcare.”

Calendar: “Emily / Sophia.” Classification: “childcare.”

It was a perfect snapshot of how bias seeps into everyday AI. Most models still assume women = parents, planning = domestic, logistics = mom.

We’re designing from the opposite premise: AI that learns each family’s actual rhythm, values, and tone — without default stereotypes.

orochimaaru2• 8 months ago

AI is trained off Reddit and other social media. If most portrayal in social media of women and girls is (and men for that matter) is biased towards certain activities - that’s what AI is going to spit out. AI doesn’t think.

Is this right or wrong is the incorrect question - because AI doesn’t understand bias or morality. It needs to be taught and it’s being taught from heavily biased sources.

You should be able to craft prompt and guardrails to not have it do that. Just expecting it to behave that way is naive - if you have ever looked deeper into how AI is trained.

The big question is - what solutions exist to train it differently with a large enough corpus of public or private/paid for data.

Fwiw - I’m the father of two girls whom I have advised to stay off social media completely because it’s unhealthy. So far they have understood why.

daveguy• 8 months ago

The problem is crafted prompts and guardrails don't work very well, because these entire networks are trained on average internet garbage. And guess what's getting worse?

cperciva2• 8 months ago

I run into this sort of bias all the time -- in the real world, not just in AI. I take my daughter to medical appointments, both for scheduling reasons (my wife's schedule is less flexible) and rapport reasons (I'm not that kind of doctor, but I know the terminology and medical professionals treat me far more as a peer), and I routinely get "oh we expected her mother" or "we always phone the mother to schedule followup appointments".

Is it so hard to understand that men can be parents too?

junaru• 8 months ago

Is it hard to understand you are the minority? The world keeps presenting you with data.

cperciva2• 8 months ago

Understand that I'm in the minority? Sure.

But the fact that I'm bringing my daughter to a medical appointment should be a pretty clear indication that, you know, I bring my daughter to medical appointments.

toomuchtodo• 8 months ago

Is it so hard to understand that men can be parents too?

Overton window and cultural norms take time to slide. Might be there after another generation, too early to tell.

0xdeadbeefbabe• 8 months ago

in the real world, not just in AI

The scheduler is trained to give higher weight to those sorts of questions apparently. This begs some questions for GPTs, questions like how are they supposed to model something not implied in the training data?

FloorEgg5• 8 months ago

I have been building applications on LLMs since GPT-3.

Thousands of hours of context engineering has shown me how LLMs will do their best to answer a question with insufficient context and can give all sorts of wrong answers. I've found that the way I prompt it and what information is in the context can heavily bias the way it responds when it doesn't have enough information to respond accurately.

You assume the bias is in the LLM itself, but I am very suspicious that the bias is actually in your system prompt and context engineering.

Are you willing to share the system prompt that led to this result that you're claiming is sexist LLM bias?

Edit: Oidar (child comment to this) did an A/B test with male names and it seems to have proven the bias is indeed in the LLM, and that my suspicion of it coming from the prompt+context was wrong. Kudos and thanks for taking the time.

small_scombrus3• 8 months ago

You assume the bias is in the LLM itself

Common large datasets being inherently biased towards some ideas/concepts and away from others in ways that imply negative things is something that there's a LOT of literature about

johnisgood2• 8 months ago

"imply negative things"? What is "negative" here? I see nothing that is "negative".

FloorEgg5• 8 months ago

That's not a very scientific stance. What would be far more informative is if we looked at the system prompt and confirm whether or not the bias was coming from it. From my experience when responses were exceptionally biased the source of the bias was my own prompts.

The OP is making a claim that an LLM assumes a meeting between two women is childcare. I've worked with LLMs enough to know that current gen LLMs wouldn't make that assumption by default. There is no way that whatever calendar related data that was used to train LLMs would include majority of sole-women 1:1s being childcare focused. That seems extremely unlikely.

callan101• 8 months ago

This feels a tad rigged against the LLM with the meeting being after Kids drop off.

cheald• 8 months ago

Easily half the other events on the calendar are kid-related. Of course it's going to infer that, absent other direction, the most likely overarching theme of the visible events is "child care".

slau2• 8 months ago

Then why doesn’t it infer it when it’s two male names?

snowe2010• 8 months ago

And yet it doesn’t when it’s male names. https://imgur.com/a/9yt5rpA

drivingmenuts2• 8 months ago

Sure, but the LLM needs to prove that it can make inferences as well as or better than a human, in order to be useful. Aside from that, it's not human, so there's no need to be fair - it should do what we tell it, not decide on its own.

broof• 8 months ago

I hate that when I see this many em dashes, as well as statements like “it’s not x, it’s y” multiple times, I have to assume it was written or at least heavily edited by AI.

somewhereoutth• 8 months ago

LLMs: The chemical weapons of public discourse.

The cleanup is going to be a grim task.

drivingmenuts2• 8 months ago

There will be an LLM for that.

God help us all.

oidar• 8 months ago

Here's an A/B

Emily / Sophia vs Bob / John https://imgur.com/a/9yt5rpA

FloorEgg5• 8 months ago

This is really interesting and way more compelling evidence to me of gender bias in the LLM than response bias in the prompt and context.

Thank you for taking the time to approach this scientifically and share the evidence with us. I appreciate knowing the truth of the matter, and it seems my suspicion that the bias was from the prompt was wrong.

I admit I am surprised.

sophiabk2• 8 months ago

Thank you for doing this analysis. It's shocking (if understandable why given the examples it was trained on). What is exciting though is as we're working to train each individual family's AI - understanding roles, jobs, interests etc - it's picked up on things in a much less biased way.

ryandrake2• 8 months ago

I wonder if the users who flagged this could chime in to explain what is rule-breaking about this article?

FloorEgg5• 8 months ago

I was wondering that myself too.

Also, do moderators ever move comments around? I thought one comment was a child to my comment last I looked, but now it's a top level comment to this post. I'm not sure if I am mistaken or a moderator moved things around.

ryandrake2• 8 months ago

This does happen from time to time. A moderator will "detach" a subthread[1] and move it to the top-level (usually also burying it at the bottom of the page, which tends to silence the discussion).

1: https://news.ycombinator.com/item?id=23441803

Two women had a business meeting. AI called it childcare

Related Stories