Amazon scraps secret AI recruiting tool that showed bias against women

https://www.reuters.com/article/us-amazon-com-jobs-automation-insight/amazon-scraps-secret-ai-recruiting-tool-that-showed-bias-against-women-idUSKCN1MK08G

by wyldfire • 8 years ago

317 406 8 years ago

reuters.com

trustmath9• 8 years ago

I hate this industry. Shooting themselves in the foot over and over again because no one can get passed the idea that possibly, women can be just as good at math, logic and computer science - if people would just let them. This never ends. It's just one place after another, when it gets discovered. It never changes.

ry_ry• 8 years ago

Your comment will probably end up buried, but it does raise the question - if they want more female employees, was the issue in the training data, or their recruitment process?

trustmath9• 8 years ago

Buried, yes, I'm sure.

This should be obvious when testing. Whether the algorithm discriminates should be a top priority for designing these algorithms. That's half the damn math of machine learning. If you can construct an AI, you should know how to test it for flaw in reasoning. It's just another layer of ML to do that. Outliers. It's short sighted to push these things out assuming their output is correct just because it looks 'normal'.

exo762• 8 years ago

Why not "women are just not as interested in math, logic and computer science to pursue it AS OFTEN as men"? Why are you not considering this possibility?

trustmath9• 8 years ago

That's always the excuse but an algorithm that shows bias against women has nothing to do with who has what interest unless that's a variable included in the data set initially. You are inferring that relation implicitly due to the result of the algorithm, but it doesn't mean that value is measured in the original set of data. If the algorithm is skewed to imply that, there is at least the possiblity that the algorithm has been trained to yield that result.

I wouldn't know unless I looked at all the data. But I'm not going to default to the popular opinion because that's literally half or more of the problem.

village-idiot2• 8 years ago

They did in the medium past, they didn’t in the recent past, and maybe they will again in the near future.

komali28• 8 years ago

Ah, the Damore argument. Besides the fact that his psuedo science has been summarily handled[0], to consider his argument you then have to equally consider the possibility of sexism in academia pressuring women to not study these subjects and societal pressure their whole lives pressuring them to not persue these career paths.

There's also the idea that lack of women scientist "heroes" can be limiting (lack of role models). Basically the idea that if you stack the cards against a population, you're gonna see population-wide effects.

Given these data points, a biased hiring AI contributes to the problem. Therefore, it should be fixed, along with the above points.

[0]https://www.bbc.co.uk/news/world-40865261

[0]https://www.theguardian.com/technology/2017/aug/13/james-dam...

kaitai4• 8 years ago

If these women are applying for tech jobs at Amazon, they're by definition interested. The uninterested (male or female) are not relevant to this discussion.

I must say, I am frustrated by this being brought up in every discussion of women & STEM. Want to discuss the leaky pipeline from physics PhDs to full professor in physics? "Maybe those women did a PhD in physics despite not being interested, and they just didn't notice before!"

thoughtexplorer5• 8 years ago

Because that's an inconvenient truth that doesn't fit the oppressor vs oppressed narrative.

mcguire4• 8 years ago

Let's say you have two otherwise-identical resumes in front of you.

* One says "executed concentration camp prisoners in Kosovo". (Yeah, ok, I'm kidding. How about "Executed a plan to reduce production costs by 30%"?)

* The other says, "Won the Women's World Chess Championship 3 years in a row".

The first has five stars (thanks to "executed") and the second has three (courtesy of "women's"). Which are you more interested in?

tzs• 8 years ago

I haven't seen anything suggesting women are less interested, but there is some support for the idea that before college girls who are interested in math., etc., are more likely than boys to have other subjects that interest them more.

There was a study published a while back that looked at PISA data and found that girls and boys were pretty evenly represented among the kids who were at the top in STEM [1].

But it also found that for the boys in that group quite often STEM was the only thing they were outstanding at. In other areas they were average to good.

For the girls, on the other hand, they were often excellent at something else in addition to STEM, with them often even being better at that something else than they were in STEM.

People have a tendency to pursue a career in one of the areas they are very good at.

This suggests that boys who are very good in math, etc., are more likely than similarly good girls to pursue it as a career because that is their only choice if they want to go into something they are very good at. The girls are more likely to have math, etc., as one of two or more possible careers in areas they are very good at.

In pop culture terms, STEM boys are more like Martin Prince, and STEM girls are more like Lisa Simpson.

[1] I didn't save the link and have failed to find it with Google. Anyone have it?

UncleMeat• 8 years ago

We did consider this possibility. Then we did research and the experts and they found that this isn't supported by evidence.

komali28• 8 years ago

I understand your frustration, but in my experience recruiting, the primary reason behind there being less women getting hired into engineering roles is almost never raw sexism. Maybe in the 90s, but in the early 10s there was tons of policy around it, bosses were setting the culture, we were doing everything "right." But we were still not hiring that many women, simply because hardly any women ever applied. For chem e, Mech e, EE roles with 100 applicants, usually I'd see at most one female applicant. It was rare to getm but when we did we'd push for the interview and they'd get through with an average success rate (compared to make applicants).

I'm hoping industries that hire young are seeing different numbers than I did, because that should signal a shift in older ones that hire senior discipline engineers after a decade or so.

Edit: that said, companies should continue to do what they can to remediate this, but I am furious that the government has done almost nothing about the issue. The underrepresented remain exactly that.

trustmath9• 8 years ago

Yes, I am female, I am aware of the statistics. It's frustrating because my life literally gets impacted by automated reasoning such as this. It's not frustrating for anyone who says 'there just aren't enough of you'. That's something that is very easy to say by people who never have to experience that sort of discriminatory practice.

The painful stuff is when it's obvious and provable, because it highlights all the times it can be questionable as to whether it occurs.

arbitrage• 8 years ago

but in my experience recruiting, the primary reason behind there being less women getting hired into engineering roles is almost never raw sexism.

In my experience in the industry, this is a laughable statement to make. It's a shame that unless one is a victim of unconscious, systemic bias, one is so much less likely to acknowledge it as a problem, that actually exists, and hurts people all the time.

212• 8 years ago

I worked in a tech company which was heavily biased in favor of hiring woman, there were special hiring tracks for woman where the interviews were easier and the interviewers received special training in unconscious biases, and the managers received bonuses for having a closer to 50/50 ratio.

It was still just a trickle, for the same reason you stated - very very few woman apply to tech positions.

InclinedPlane2• 8 years ago

I find this sort of sentiment almost hilarious in how out of touch it is.

Time and time again people (mostly men of course) keep asking "but why? why aren't there more women in the field?" Time and time again they keep saying "but I don't see any sexism in the workplace, it's nothing like it used to be, it's practically a meritocracy these days!" Yes, indeed, it truly is a giant mystery.

And yet, at the same time there is a constant deluge of stories about rampant sexism in the industry. Of all sorts, at all levels, at almost every company, and often of shockingly regressive character even up through the present time. There are countless stories in the industry of how women in tech are persistently denigrated, how men talk over them in meetings, how their ideas are ignored until they come out of the mouth of a man, how sexual harassment is ubiquitous, how they are routinely excluded from workplace culture through extremely male-centric activities that include things as ridiculous as morale events or even meetings held at strip clubs.

All of this takes a toll, and that toll is ultimately to stunt the careers of women in tech and to push women out of the industry entirely. Working in tech as a woman is climbing a hill with a much steeper slope than it is for guys. Women routinely get passed over for promotions, are routinely underpaid, routinely do not receive credit for their ideas, and routinely experience more hostile working conditions (through bias as well as sexual harassment). So they leave. They find something better to do with their time because they just can't take the stress and harassment anymore or because it just does not provide the same return on investment as it does for guys.

And we know this. We know this from studies and exposes and a torrent of anecdotes from individual women who have been in the field for years or decades. Some people (guys) have a tendency to write off each and every one of these stories and studies as somehow individual aberrations or outliers which don't have any bearing on the fundamental overall character of the industry, but this is a mistake, they are absolutely representative. The problem of over-representation of white men in tech cannot be solved by "fixing the pipeline" in the educational system nor can it be solved by making hiring processes perfectly unbiased (or even biased towards women) because the real problem is much bigger, it's systemic, widespread misogyny throughout the entire industry. That will take a tremendous amount of work to fix, but once the industry stops treating women as second class citizens (or exotic outsiders) and stops pushing them out of the industry through its toxicity then the problem will mostly fix itself.

Skrillex• 8 years ago

Is this not Amazon trying to not discriminate against women?

trustmath9• 8 years ago

Sure. It just sounds like their code and it's results got too complicated to reason about.

thoughtexplorer5• 8 years ago

if people would just let them.

People do let them. You can't force what people are interested in and you can't let in that which does not exist. In fact many places in tech give preference to women applicants, because they don't apply often and the companies want more women. They're just rare to see. :(

There's no grand conspiracy. The truth is much less exciting: Women and men have different preferences, generally speaking.

Women are more interested in people (i.e. healthcare). Men are more interested in things (i.e. engineering).

Most nurses are women. That's not because women are activity trying to keep men out. It's because fewer men are interested or apply!

We also don’t see women in the most dangerous jobs. No one seems to have a problem with that, just as no one has a problem with most healthcare jobs being dominated by women. As they shouldn't, because people should be allowed to pursue and apply to what they want to.

trustmath9• 8 years ago

Please read the article. This article is about automated reasoning that discards resumes that are strongly correlated to resumes of women.

komali28• 8 years ago

There's no actual evidence to suggest preference for career type is inherent to being a woman or man. There's plenty showing that women right now have different interests of men, but the cause can easily be societal conditioning - i.e., something that can be fixed.

ncallaway• 8 years ago

I would guess that the training data for the ML set was the set of all resumes and an indicator of whether the candidate was eventually hired (maybe with supplemental data about how far in the process the candidate got).

Could this be a direct indicator of a powerful subconscious bias in Amazon's existing hiring process?

throwawaymath2• 8 years ago

Could this be a direct indicator of a powerful subconscious bias in Amazon's existing hiring process?

Yes, but only in the obvious sense that we all already knew: tech companies hire more men than women for technology-focused roles. That's not to say it isn't an issue; it is, but it's nothing new, and almost certainly not unique to Amazon.

Without significant oversight and manual tuning, any training dataset based wholly or in part on current employees is going to demonstrate a bias against women, because there are strictly fewer women. Moreover it's likely that (for a variety of reasons, both intrinsic and extrinsic) fewer women succeed in the interview process as a ratio compared to the number who actually apply.

Spooky23• 8 years ago

I spent a number of years working in tech organizations where the male:female ratio was 60:40. Now I work in a similar org with the same rules where it's more like 85:15.

The difference is that old place transitioned administrative staff to IT roles in the 90s/early 2000s when more things were computerized. Those admins, financial analysts, program analysts were more likely to be female and had degrees in liberal arts, accounting, business/finance, etc.

In the newer place, they filtered based on computer-related degree upon hiring. That automatically excludes many women. Once hired, female candidates advance as well or better.

Anecdotally, I've hired interns in recent years with no tech-specific qualifications as an experiment. If you select for "smart and gets things done" I don't see much of a disadvantage for many roles. You get some duds too, but it wasn't as dramatic a difference as I expected.

mcguire4• 8 years ago

"...any training dataset based wholly or in part on current employees is going to demonstrate a bias against women,..."

"Demonstrate" may be the wrong word. How about "encode". And in the future, "enforce".

[Edit] On second thought, it would seem like there would be a way to filter out a "raw proportions bias" (like 80% of the resumes in the data set are male) before training.

elfakyn• 8 years ago

That gives rise to a very interesting concept: ML-based bias assessments. If you take some real-life hiring data (or other applications such as sentencing or generally human behavior data) and train the AI on it, then run it through a bunch of tests to see whether there's bias, that can reveal trends in the underlying training data.

I can't imagine this not already being a thing, but I haven't really heard of people using this method.

bloak• 8 years ago

I don't think you can detect "bias" by running "a bunch of tests". "Bias" is a very slippery concept and is probably essentially subjective. When people say an algorithm is "biased" what they seem to mean is that when the judgements of the algorithm are compared with the judgements of a committee of fair-minded and diligent humans then the number of positive outcomes for members of some fashionable minority that we care about is less that what it was with the human judges. It's hard to automate that. And in any case, if you manipulate the algorithm until it "passes" a test like that then you might not really have improved it: when you turn a measure into a target it ceases to be a good measure.

KevinEldon• 8 years ago

Maybe, but it seems more likely that the model just didn't work.

From the article: "Gender bias was not the only issue. Problems with the data that underpinned the models’ judgments meant that unqualified candidates were often recommended for all manner of jobs... With the technology returning results almost at random"

WalterSear• 8 years ago

Sounds like they've successfully emulated their real world interview process.

throwawaymath2• 8 years ago

Those aren't mutually exclusive. Technically speaking, the model can return results "almost at random" and still demonstrate a bias against any particular attribute if that bias is evident in the underlying training dataset.

If there are strictly fewer women in the underlying training set, the model can still return something resembling a uniform distribution of candidates while exacerbating the diminished representation of women.

To give a concrete example: you have a bag of blue dice and red dice. There is a supermajority of blue dice in the bag. Your algorithm selects a single die out of the bag on every iteration. The output sequence of dice numbers appears uniform, but there are more blue dice than red dice in the output sequence.

michaelt2• 8 years ago

  Could this be a direct indicator of a powerful
  subconscious bias in Amazon's existing hiring
  process?

Maybe - but maybe not.

Imagine a company with 2 men in HR, 2 women in HR, 40 men in engineering, and 10 women in engineering. That's with gender-blind hiring, reflecting only the 4:1 ratio of male to female CS graduates.

If you picked a random male hire, there's a 40/42=95% chance they're an engineer whereas if you picked a random female hire, there's a 10/12=83% chance they're an engineer.

Thus if you look over all hires' CVs, due to Bayes’ law the dataset says being male increases the conditional probability you meet engineering hiring requirements - and the ML system picks up on that.

sanxiyn3• 8 years ago

Why would one train on "all hires CVs"? It'd be "engineering CVs", moreover it'd be "engineering applicants CVs", not "engineering hires CVs".

skohan• 8 years ago

So it depends how you set up the experiment right? I would assume the question you are posing to AI is not how much the resume in question resembles the set of resumes of hired engineers, but rather: given a resume, what is the probability that candidate will eventually be hired?

So the classification function should take into account the resumes of rejected engineers, rather than the pool of resumes of hired employees at Amazon. If someone is seeking a position as an engineer, it is not relevant how much their resume resembles that of HR people, but it is very relevant how much it resembles that of rejected engineering candidates.

If that's the case, then something like having the phrase "women's chess club" in one's resume should not be a meaningful factor for the classifier unless it disproportionately leads to rejection in the current process.

nkassis• 8 years ago

If they naively just feed the entire dataset without some inspection maybe. But I doubt the people working on this model stopped at the most basic level. What you describe is a common class imbalance issue. I would expect that they have accounted and addressed (at a minimum oversampling the less represented class for example) for this issue while working on the model.

So I doubt it's enough to explain their issue here. I agree that we can't really take any conclusion of their broader hiring patterns from this experiment.

TangoTrotFox2• 8 years ago

Step into your engineering or computer science department and walk into any upper division class and count the females and count the males. That some companies have upwards of 20% females is more likely indicative of extreme bias in hiring as you're not going to find even remotely close to 20% females there. Enter in most measures of competence and you'll find the division is no different. E.g. - if females were being disproportionately hired because of disproportionately positive performance then this might not be an issue, but there seems to be no evidence for that whatsoever.

This view that the only thing holding people back is some sort of social or systemic bias seems to be based on nothing except ideology. Incidentally, it's an ideology I also used to hold. Like a good egalitarian I pushed my wife away from sociology and into majoring in CS. She did perfectly well, as did I. More than a decade later she works with people and I work with code. I've no regrets there, but it's not so clear that my persuasion was really the best idea.

Norway is another interesting example here. It is considered by many to be the most gender equal location in the world. Yet you'll still find that nurses are primarily female, doctors are primarily male, and all other 'stereotypical' divisions present in most all developed nations. They tried to change these divisions and with extensive effort were able to effect a roughly constant change in some fields. But again, once that push was relinquished things went just about identically to as they were in very short order. Ultimately we're flexible enough that you can manage to fit a square peg into a round hole at times, but once you stop squeezing that peg goes back to what it wants to be.

mcguire4• 8 years ago

"I've no regrets there, but it's not so clear that my persuasion was really the best idea."

Does she make more money than a similar person with a degree in sociology?

hellogoodbyeeee2• 8 years ago

Right, so the "AI" is not finding the best candidates, but instead finding candidates that succeed in the old hiring process. What I'd like to see (though I'm sure this would get me laughed out of a manager's office) would be a certain fraction of random hires. Then we could train the model on success statistics based on 60,90,120 day performance (or longer!).

At the least, they should interview a certain subset of randomly chosen applicants or else the feedback loop from the interviewing process and the AI is going to grow tighter and tighter.

ummonk2• 8 years ago

Machine learning should not be used in this way on humans, whether for resume screening or even more dystopic, for sentencing.

mcherm• 8 years ago

Why not?

Seriously: let us take as given that the AI models are biased. Will you also admit that the existing processes are biased? If so, then what we need to ask is which is MORE biased. It might be complaining that we shouldn't release self-driving cars because on rare occasions they cause accidents.

There is, however, another criterion besides how biased it is: how biased it will be in the future. Human-driven processes have the opportunity to become less biased in the future (also the chance to become more biased, but overall things tend to improve). AI processes that are opaque might lock in bias in a fashion that is unreviewable. I believe that the solution is to build AI models that are more transparent -- that could be BETTER (in terms of avoiding bias) than the human-driven processes we use today.

ummonk2• 8 years ago

I think we basically have the same view. I don't support black box machine learning models. I do support using automated tests and simple well defined objective criteria though, which is basically a transparent AI model.

It's just that generally what seems to distinguish whether something is called "machine learning" rather than "data science and modelling" is that the former is black box and the latter is not.

Novashi2• 8 years ago

Why not?

It has to be better than humans.

That requires the rare genius to fix the problem, or we have to accidentally invent something better than ourselves.

This is a common defense of autopilot systems in cars: it's not perfect, but it's better than the average.

linuxftw• 8 years ago

Machines are not intuitive or nuanced. They are incapable of learning and formulating abstract though, only identifying patterns and optimizing for some desired outcome.

zdragnar• 8 years ago

I'm reminded of why Watson failed, and the problem with ml and ai in general- you can't peek under the hood to see why something happened, or how to keep it from happening without a lot of time, a lot of hard work, and a whole lot of carefully groomed data.

bglusman• 8 years ago

This is especially memorable/well treated in the Cory Doctorow novella/short story "Human Readable"

https://craphound.com/stories/2005/10/12/human-readable/

strict9• 8 years ago

At best ai amplifies existing patterns and biases when handling repetitive work. Over and over we hear how Facebook, Twitter, Google, and others will solve the problem of problematic content and bad actors through ai and neural networks. It's a fraud and the digital potemkin village of our era.

fx32s• 8 years ago

AI learns from the training data it's given and copies any biases this data exhibits. Pretty much all software today uses ML in some form to improve their services. I feel it's here to stay and not bad by default. We just have to make sure we are aware of its current limitations.

Facebook is already auto-flagging content this way but it's just a very hard problem (even for humans).

mindcrime• 8 years ago

AI learns from the training data it's given and copies any biases this data exhibits

I hate to sound like "that pedantic guy", but I'd argue that the quote above is only partially true. It's the case that some subset of AI techniques "learn from the training data it's given and copies any biases this data exhibits". There are AI techniques that aren't based on supervised learning from a pre-existing training set. That doesn't mean that those techniques can't wind up adopting the biases of their human overlords, but I believe some aspects of AI are less susceptible to this kind of bias, than others.

gfodor2• 8 years ago

This is a direct and clear example of bias which made it easy to flag the ML algorithm. But what about ML algorithms that are inducing benefits to groups in less obvious contexts? What about groups that are not so easily identified as being protected classes by simple, human-understandable model features? What about cases where the features are just merely correlated with a subpopulation of a protected class?

If we're being honest, a system only needs to be in a decision-making capacity for discriminatory behavior to be scrutinized, since in many cases human operators will not be able to identify the specific features being used to make decisions about people -- the features could be highly correlated with some subpopulation of protected class. If you take that to be true, the question reduces onto what decision-making roles ML algorithms have that could be discriminatory, and it's hard to argue this is not a massive part of their current and expected roles.

I think this is going to be a long, winding ethical nightmare that is probably just getting started by human-digestible examples such as these. One can imagine things like this one being looked back on as quaint in the naivety to which we assume we can understand these systems. Where do we draw the line, and how much control do we give up to an optimization function? Surely there is a balance -- how do we categorize and made good decisions around this?

As far as I know, a cohesive ethical framework around this is pretty much non-existent -- the current regime is simply "someone speaks up when something absurdly and overtly bad happens."

abdullahkhalids• 8 years ago

What about cases where the features just merely correlated with a subpopulation of a protected class?

This is just Simpson's paradox [1] which is notoriously hard to identify because you have to compare the overall with the breakdown. As you say, current-AI probably already has such biases.

[1] https://en.wikipedia.org/wiki/Simpson%27s_paradox

jakelazaroff2• 8 years ago

What about cases where the features are just merely correlated with a subpopulation of a protected class?

This question can be rephrased as "is there a difference between de facto and de jure discrimination?"

My answer is no, causality doesn't matter here: if feature A is a good predictor that some person belongs in group B and not group C, then filtering out feature As is effectively the same as filtering out only group Bs.

darawk5• 8 years ago

Ok, so if you're hiring professional arm wrestlers, and your model looks at bicep muscle mass, is that discrimination because it selects against women?

If you're hiring therapists, and your candidates take a personality test, and your ML model weights the 'nurturing' feature highly, is that discrimination because it selects against men?

darawk5• 8 years ago

All of that is true, but I think the most important question is: compared to what? ML is substantially more transparent than human decision makers. Human decision makers will actively lie to you. ML is a major step forward in correcting these sorts of biases, by making interpretable (relative to humans) models in the first place.

Bartweiss3• 8 years ago

the features are just merely correlated with a subpopulation of a protected class

The article notes that Amazon's system rated down grads from two all-women's schools. But it immediately occurs to me to wonder what the algorithm did with candidates from heavily gender-imbalanced schools, which could be much harder to spot.

RPI's Computer Science department is about 85% male, while CMU's is just over 50% male. CMU's CS department is also considered one of the best in the world, and presumably any functional algorithm that cared about alma mater would respond to that. So if the bias ends up being "because of CMU's gender ratio, CMU grads with gender-unclear resumes are advantaged slightly less than otherwise would be", how on earth would someone spot that?

Once you're looking for it, you could potentially retrain with some data set like "RPI resumes, but we adjusted their gendered-words rate" and see if you get a different outcome on your test set. But that's both a labor intensive task, and one that's only approachable once you already know what you're looking for. And even if you do see a change, you'd still have to tease it out from a dozen other hypotheses like "certain schools have more organizations with gendered names, and the algorithm can't tell that those organizations are a proxy for school".

Of course, the counterpoint is that human decisions can't be scrutinized any better, and it's not entirely clear they're less arbitrary or more ethical. At a certain point algorithmic approaches are being scrutinized because they're slightly transparent and testable, so running them on a range of counterfactuals or breaking down their choices is hard rather than impossible. I suspect that's true, but it doesn't really comfort me - humans at least tend to misbehave along certain predictable axes we can try to mitigate, while ML systems can blindside us with all sorts of new and unexpected forms of badness.

grogenaut2• 8 years ago

A subtle point you may have missed, amazon knew about and accounted for the gender bias, the scrapped it because of all of the biases that they couldn't identify and were leery of. Most of your suggestions seem to be solving for the known biases, which I believe they did.

Also knowing some people who worked on this, they were VERY cognizant of re-encoding biases from the start of the project, it was one of the main reasons they thought the project might fail.

kgwgk• 8 years ago

I’m curious: how is it that this submission has not “reused” the submission I made eight hours before? In my experience, sending a link that is already there simply upvotes the existing submission.

adventured• 8 years ago

After a certain amount of time, the same link can be posted by a different user. I'm not sure if there is a point or point vs time qualification on that. You can post old, popular stories again for example.

Your submission was made 11 hours ago. If they merely applied an upvote to that existing story after 10 hours, the point time value rot would be such that it would have greatly reduced impact at ranking the story to the front page (ie it would be nearly useless for discovery purposes).

village-idiot2• 8 years ago

We’re going to keep seeing stuff like this until people finally realize that AI isn’t some magic tool that solves every problem. It still reflects the biases and assumptions of its creators and its training data set.

fuscy2• 8 years ago

The eye opening thing here is not that the AI failed, but why it failed.

At start the AI is like a baby, it doesn't know anything or have any opinions. By teaching it using a set of data, in this case a set of resumes and the outcome then it can form an opinion.

The AI becoming biased tells that the "teacher" was biased also. So actually Amazon's recruiting process seems to be a mess with the technical skills on the resume amounting to zilch, gender and the aggressiveness of the resume's language being the most important (because that's how the human recruiters actually hired people when someone put a resume).

The number of women and men in the data set shouldn't matter (algorithms learn that even if there was 1 woman, if she was hired then it will be positive about future woman candidates). What matters is the rejection rate which it learned from the data.. The hiring process is inherently biased against women.

Technically one could say that the AI was successful because it emulated the current Amazon hiring status.

hkai• 8 years ago

How did you come to the conclusion that gender was being the most important, rather than skills or aggressiveness?

kaitai4• 8 years ago

I don't think that's what the parent was claiming; the parent says "gender and aggressiveness" were most important and skills listed on the resume as providing such an unclear signal for actual hires that they were not picked up by the AI.

lalaland11252• 8 years ago

The number of women and men in the data set shouldn't matter (algorithms learn that even if there was 1 woman, if she was hired then it will be positive about future woman candidates).

This is incorrect. The key thing to keep in mind is that they are not just predicting who is a good candidate, they are also ranking by the certainty of their prediction.

Lower numbers of female candidates could plausibly lead to lower certainty for the prediction model as it would have less data on those people. I've never trained a model on resumes, but I definitely often see this "lower certainty on minorites" thing for models I do train.

The lower certainty would in turn lead to lower rankings for women even without any bias in the data.

Now, I'm not saying that Amazon's data isn't biased. I would not be surprised if it were. I'm just saying we should be careful in understanding what is evidence of bias and what is not.

tomp3• 8 years ago

The lower certainty would in turn lead to lower rankings for women even without any bias in the data.

I don't think that's true. "No bias" means that gender is irrelevant (i.e. its correlation with outcome is 0%). Therefore the system shouldn't even take it into account - it would evaluate both men and women just by other criteria (experience, technical skills, etc), and it would have equal amounts of data for both (because it wouldn't even see them as different).

You need bias to even separate the dataset into distinct categories.

blt• 8 years ago

It's wrong even if their model doesn't output a certainty (not all classifiers do). Almost all ML algorithms optimize the expected classification error under the training distribution. So if the training data contains 90% men, it's better to classify those men at 100% accuracy and women at 0% accuracy, than it is to classify both with 89.9% accuracy. Any unsophisticated model will do this.

gp: "The number of women and men in the data set shouldn't matter (algorithms learn that even if there was 1 woman, if she was hired then it will be positive about future woman candidates)."

This is false for typical models.

bcheung• 8 years ago

How did you control for these things? Wondering what patterns there are that people use to prevent social discrimination.

Seems challenging since much of AI, especially classification, is essentially a discrimination algorithm.

screye2• 8 years ago

The lower certainty would in turn lead to lower rankings for women even without any bias in the data.

This is not true.

Probabilistic-ly speaking, if we are computing P(hiring | gender); Lower certainty means there is a high variance in prior over women. However, over a large dataset, the "score" would almost certainly be equal to the mean of the distribution, and be independent of the variance.

In simpler words, if there was a frequency diagram of scores for each gender (most likely bell curves), then only the peak of the bell curve would matter. The flatness / thinness of the curve would be completely irrelevant to the final score. The peak is the mean, and the flatness is the uncertainty. Only the mean matters.

kareemsabri10• 8 years ago

This doesn’t seem to be a reasonable conclusion. There is no reason to assume the AI’s assessment methods will mirror those of the recruiters. If Amazon did most of it’s hiring when programming was a task primarily performed by men, and so Amazon didn’t receive many female applicants, they could be unbiased while still amassing a data set that skewed heavily male. The machine would then just correctly assess that female resumes don’t match, as closely, the resumes of successful past candidates. Perhaps I’m ignorant about AI, but I don’t see why the number of candidates of each gender shouldn’t increase the strength of the signal. “Aggressiveness” in the resume may be correlated but not causal. If the AI was fed the heights of the candidates, it might reject women for being too short, but that would not indicate height is a criteria of Amazon recruiters hiring.

kaitai4• 8 years ago

The whole aim of the AI was to make decisions like the recruiters did -- that is explicitly what they were aiming to do. It might be worth reading the article as it addresses your two ideas (the aim of the project and the fact that the training set was indeed heavily male).

bilbo0s• 8 years ago

Did you read the article?

(Serious question. Not intended as snark. Genuinely wondering if I'm missing some deeper current in your post?)

fuscy2• 8 years ago

I'm going to make a supposition here but one of the first things I think they did (especially when trying to fix the AI) was to balance and normalize the data so that there would be no skew between men and women number of records in the data set.

If my supposition is correct then the other parameters are at fault here from which gender and language used stick out.

Another supposition I'm going to make is that they even removed the gender from the data set so that AI didn't know it, but cross-referencing still showed "faulty" results due to hidden bias that the AI can pick up, like language used.

BurningFrog• 8 years ago

Control question for if you're making a certain intellectual mistake.

The data set will also have skewed heavily against people named "David". Probably only ~1% of the successful applicants.

Would you also expect the machine to be biased against candidates named David?

zby6• 8 years ago

"they could be unbiased while still amassing a data set that skewed heavily male" - this sounds like a self contradiction

danShumway• 8 years ago

This is a subtle point but worth stating -- AI does not mirror or copy human reasoning.

AI is designed to get the same results as a human. How it gets to those results is often very, very different. I'm having trouble finding it, but there was an article a while back trying to do focus tracking between humans and computers for image recognition. What they found was that even when computers were relatively consistent with humans in results, they often focused on different parts of the image and relied on different correlations.

That doesn't mean that Amazon isn't biased. I mean, let's be honest, it probably is; there's no way a company this large is going to be able to perfectly filter or train every employee and on average tech bias trends against women. BUT, the point is that even if Amazon were to completely eliminate bias from every single hiring decision it used in its training data, an AI still might introduce a racial or gendered bias on its own if the data were skewed or had an unseen correlation that researchers didn't intend.

IshKebab• 8 years ago

They didn't scrap it because of this gender problem. That wasn't why it failed. They scrapped it because it didn't work anyway.

Note the title is "Amazon scraps secret AI recruiting tool that showed bias against women" not "Amazon scraps secret AI recruiting tool because it showed bias against women". But I guess the real title is less clickbaity - "Amazon scraps secret AI recruiting tool because it didn't work".

stcredzero• 8 years ago

The same AI should be applied to hiring nurses and various other fields which show population skews in gender, as well as fields which are not skewed. I'd be curious as to the outcome.

Consultant32452• 8 years ago

Without regard to this particular issue, you also have to concern yourself with the bias of the person determining if the AI has a bias.

louwrentius• 8 years ago

Thanks for spelling this out, I think this is exactly how to look at this.

gambler2• 8 years ago

The article didn't specify how they labeled resumes for training. You're assuming that it was based on whether or not the candidate was hire. Nobody with an iota of experience in machine learning would do something like that. (For obvious reasons: you can't tell from your data whether people you did not hire were truly bad.)

A far more reasonable way would be to take resumes of people who were hired and train the model based on their performance. For example, you could rate resumes of people who promptly quit or got fired as less attractive than resumes of people who stayed with the company for a long time. You could also factor in performance reviews.

It is entirely possible that such model would search for people who aren't usually preferred. E.g. if your recruiters are biased against Ph.D.'s, but you have some Ph.D.'s and they're highly productive, the algorithm could pick this up and rate Ph.D. resumes higher.

Now, you still wouldn't know anything about people whom you didn't hire. This means there is some possibility your employees are not representative of general population and your model would be biased because of that.

Let's say your recruiters are biased against Ph.D.'s and so they undergo extra scrutiny. You only hire candidates with a doctoral degree if they are amazing. This means within your company a doctoral degree is a good predictor of success, but in the world at large it could be a bad criteria to use.

jonny_eh• 8 years ago

Men are promoted quicker, and more often, than women.

noetic_techy2• 8 years ago

I'm not a ML guy, but reading this, it almost sounds like the training data needs to be a fictional, idealized set, and not based on real world data that already has bias slants built in. Possibly composites of real world candidates with idealized characteristics and fictional career trajectories. Basically, what-my-company-looks-like vs what-I-want-it-to-look-like. I'm not sure this is even possible.

Its an interesting questions. On one hand, a practical person could argue: "Well, this is what my company looks like, and these are the types of people who fit with our culture and make it, so be it. Find me these types of candidates."

"I don't like the way may company culture looks, I would rather it was more diverse. This mono-culture is potentially leaving money on the table from not being diverse enough. I'm going to take my current employees, chart their career path, composite them (maybe), tweak some of the ugly race and gender stats for those who were promoted, and feed this to my hiring algorithm."

b_tterc_p• 8 years ago

A reasonable assumption but, in practice, false. Many companies believe (perhaps correctly) that their hiring system is good. Using hiring outcomes would be a reasonable dependent variable, especially if supply is lower than demand, performance is difficult to measure, or there’s a huge surplus of applications which need to be cut down to a smaller number of human assessed resumes.

brown9-22• 8 years ago

This seems to assume that performance evaluation is itself free from bias.

ewjordan3• 8 years ago

You're assuming that it was based on whether or not the candidate was hire. Nobody with an iota of experience in machine learning would do something like that. (For obvious reasons: you can't tell from your data whether people you did not hire were truly bad.)

It's a fine strategy if all you're trying to do is cost-cut and replace the people that currently make these decisions (without changing the decisions).

I agree that most people with ML experience would want to do better, and could think of ways to do so with the right data, but if all the data that's available is "resume + hire/no-hire", then this might be the best they could do (or at least the limit of their assignment).

asaph• 8 years ago

For obvious reasons: you can't tell from your data whether people you did not hire were truly bad.

Many companies are fine with false negatives in their hiring process. Better to pass on a good candidate than hire a bad one.

femidav• 8 years ago

This also means that if you hire unqualified women only because they are women, then your AI will have bias against women.

monochromatic• 8 years ago

The AI becoming biased tells that the "teacher" was biased also.

That doesn’t follow.

s73v3r_6• 8 years ago

Someone had to decide on the training material. Note that saying that they had bias does not mean that they acted with malicious intent; most likely they didn't. That doesn't change the outcome, however.

roenxi2• 8 years ago

Do you have some information not present in the article? There seem to be some assumptions on the training process in your comment that are not sourced in the article.

I'll don my flack jacket for this one, but based on population statistics I believe a statistically significant number of women have children. A plausible hypothesis is that a typical female candidate is at a 9 month disadvantage against male employees and that that is a statistically significant effect detected by this Amazon tool.

Now, the article says that the results of the tool were 'nearly random', so that probably wasn't the issue. But just because the result of a machine learning process is biased does not indicate that the teacher is biased. It indicates that the data is biased, and bias always has a chance to be linked to real-world phenomenon.

brown9-22• 8 years ago

Does Amazon give 9 months of parental leave, or are you saying women employees are disadvantaged for their entire pregnancy?

dheera2• 8 years ago

The term "AI" is over-hyped. What we have now is advanced pattern recognition, not intelligence.

Pattern recognition will learn any biases in your training data. An intelligent enough* being does much more than pattern recognition -- intelligent beings have concepts of ethics, social responsibility, value systems, dreams, ideals, and is able to know what to look for and what to ignore in the process of learning.

A dumb pattern recognition algorithm aims to maximize its correctness. Gradient descent does exactly that. It wants to be correct as much of the time as possible. An intelligent enough being, on the other hand, has at least an idea of de-prioritizing mathematical correctness and putting ethics first.

Deep learning in its current state is emphatically NOT what I would call "intelligence" in that respect.

Google had a big media blooper when their algorithm mistakenly recognized a black person as a gorilla [0]. The fundamental problem here is that state-of-the-art machine learning is not intelligent enough. It sees dark-colored pixels with a face and goes "oh, gorilla". Nothing else. The very fact that people were offended by that is a sign that people are truly intelligent. The fact that the algorithm didn't even know it was offending people is a sign that the algorithm is stupid. Emotions, the ability to be offended, and the ability to understand what offends others, are all products of true intelligence.

If you used today's state-of-the-art machine learning, fed it real data from today's world, and asked it to classify them into [good people, criminals, terrorists], you would result in an algorithm that labels all black people as criminals and all people with black hair and beards as terrorists. The algorithm might even be the most mathematically correct model. The very fact that you (I sincerely hope) cringe at the above is a sign that YOU are intelligent and this algorithm is stupid.

*People are overall intelligent, and some people behave more intelligently than others. There are members of society that do unintelligent things, like stereotyping, over-generalization, and prejudice, and others who don't.

[0] https://www.theverge.com/2018/1/12/16882408/google-racist-go...

platz• 8 years ago

"a worldview built on the important of causation is being challenged by a preponderance of correlations. The possession of knowledge, which once meant an understanding of the past, is coming to mean an ability to predict the future." - Big Data (Schonberger & Cukier)

so, knowledge now is allegedly possession of the future, rather than possession of the past.

This is because the future and past are structurally the same thing in these models. Each could be missing, but re-creatable links.

Also, conflicting correlations can be shown all the time. if almost any correlation can be shown to be real, what's true? How do we deal with conflicting correlations?

GuB-42• 8 years ago

We are pattern recognition machines. If you consider pattern matching unintelligent, then machines are more intelligent that we are since they rely more on logic than pattern matching.

For the black man = gorilla problem, an untaught human, a small child for instance, can easily make the same mistake. Especially if he has seen few black people. And well educated adults can also make the mistake initially, even if they hate to admit it.

However, in the last case, a second pattern recognition happen, one that matches the result of the image classifier with social rules. And it turns out that mixing black men and gorillas is a clear anti-pattern and anything that isn't certain is incorrect.

Unlike us, computer image classifiers typically aren't taught social rules, so like a small child, they will tell things without filter. It will probably change in the future for public facing AIs.

Not stereotyping is not a mark of intelligence, it is a mark of a certain type of education. And I don't see why it couldn't be done with the usual machine learning techniques.

Yoyoyou• 8 years ago

It failed because rationally interpreting gender data leads to politically incorrect conclusions.

Novashi2• 8 years ago

I think there's also a subtle point here that an AI figured out that Amazon hiring was biased more quickly than Amazon itself.

I wouldn't go so far to say that running your history through an AI is necessarily proof of anything though (esp. in court). Imagine using another company's historical data to suggest that they're discriminatory.

patwolf• 8 years ago

The explanation seems overly simplistic. If the difference in volume of male candidates mattered, then I would also expect to see a bias in favor of applicants from larger universities. That seems like too obvious an issue in the way the algorithm was designed.

I see four possibilities here:

1. The algorithm was designed in a completely inept fashion

2. The algorithm design was sound, but ultimately ineffective

3. The algorithm was sound and effective, but results were considered discriminatory.

4. There's something biased about how employees are rated--the data that would feed into the algorithm, which is possibly more of a human element.

Edit: Added fourth possibility

sanxiyn3• 8 years ago

My current hypothesis is that training data was too noisy.

bicubic3• 8 years ago

We're talking about Amazon, one of the biggest powerhouse ML employers. I don't buy that the model was poorly designed or ineffective. They also didn't just scrap the model without understanding how or why it failed to meet its objectives.

And whatever the cause was, it was not the poor quality of the training data. They tried to stop the model from downranking women based on obvious keywords, only to find it learning to downrank them based on more subtle language cues:

Amazon edited the programs to make them neutral to these particular terms. But that was no guarantee that the machines would not devise other ways of sorting candidates that could prove discriminatory, the people said.

So the answer is 3 or 4.

If the answer was 4 then they would have probably mentioned the cause of the bias somewhere in that otherwise detailed article. But they didn't, possibly because the cause is controversial - probably option 3 but possibly still option 4.

And then there's the subtle cop-out:

Gender bias was not the only issue. Problems with the data that underpinned the models’ judgments meant that unqualified candidates were often recommended for all manner of jobs, the people said. With the technology returning results almost at random, Amazon shut down the project, they said.

If the model was actually useless and returning random noise, then there wouldn't be any bias, and the article wouldn't need to talk about discrimination. This paragraph reads to me like they decided to mention long-tail results (that you'd find in any ML model) as supportive 'evidence' that the model was somehow broken rather than producing valid but controversial results.

moate2• 8 years ago

Well it's also what you're training for. If you're building a machine to return "John Doe, The Company's Best AI Dev" then there's a few things that you might get back from a working and effective machine. The problem is that while the machine is doing the best to replicate a John Doe, the humans who designed the machine might realize that there's a lot of variables in what they're looking for that scoping the design is impossibly complex.

Basically, people WANT bias, but they want specific bias. One of the difficulties in training a machine to understand what you find as viable bias vs problematic bias is all the tiny nuances. Yes, you want a great engineer on paper, but you also need to have as diverse a cast as you can in your company (both for optics and creative solutions) AND you need to get people you can afford AND you need someone who's enjoyable to work with etc etc.

Hiring is always going to be part art and part science. There will always be some type of discrimination because of the perceptions of what makes a good qualification for the job. Any hiring group is just going to have their own hierarchy of what they think are the most important skills to have. You can only approach perfection/unbiased hiring, you can never actually achieve it.

sonnyblarney• 8 years ago

"I don't buy that the model was poorly designed or ineffective."

Just because some groups have competencies in this area, doesn't mean that others do. I've worked at big tech companies that couldn't get their HR systems to work properly ... IT was abysmal even though we made 'high tech'. Also, it's an internal project, not a product, so the scope of investment etc. might have been very different than otherwise.

karrotwaltz• 8 years ago

From the article:

Gender bias was not the only issue. Problems with the data that underpinned the models’ judgments meant that unqualified candidates were often recommended for all manner of jobs, the people said. With the technology returning results almost at random, Amazon shut down the project, they said.

It looks like the bias wasn't the only flaw.

electrograv3• 8 years ago

I wish we could move away from resumes for tech role screening anyway, since they convey very little real reliable information. I’ve seen too many great hires from candidates with relatively weak resumes, and failed interviews from candidates with great resumes (and obviously vice versa).

I’m not sure what the best alternative should be, though. I am a fan of open source work as a sort of code portfolio, but it doesn’t work for every kind of engineering/science (edit: and also would introduce bias against professionals too busy for open source.)

Regarding bias — it seems the only way to truly eliminate it (including unconscious bias) is author-blind reviews, i.e. reviewing code written by a candidate without knowing anything about that candidate’s identity. (And the nice thing about code is it usually doesn’t signal any identity traits of the author via side channels.)

fhbdukfrh• 8 years ago

It doesn't work for the vast majority of professional work because it's not open source and not done in the public

kentm2• 8 years ago

Unfortunately using open source work, even if only for programming, introduces all sorts of biases as well. A lot of very competent programmers work at jobs that do not have open source contributions and also have families which limit the time they can spend coding after work.

electrograv3• 8 years ago

Good point! I didn’t mean to propose it as a solution, but rather as an enumeration or one possibility I’ve considered, and ruled out because it doesn’t work for everyone.

stef252• 8 years ago

author-blind reviews

Should I ever be in a position to hire a colleague, I wouldn't ever do so without having a chat with them.

I spend 8hrs a day in an office with my colleagues (sometimes more than with my wife & kid) and the ones I can't stand is about the only thing wrong with my job.

If we can't even see the person's face over some gender bias hysteria then I wonder how the hell we got here.

People should just get over the fact that men and women are different.

moate2• 8 years ago

You'll also get people who (rightfully) point out that "plays nice with others" is a nice to have and not a must have. Certain people have skills so valuable they don't need to be liked (the productivity lost due to internal politics/people complaining is < the productivity gained by having them hired).

It's great when everyone can coexist perfectly. However, that might not be the best business decision. There's no such thing as "objectively best" just a list of pros and cons to any candidate, and a company's internal preferences.

electrograv3• 8 years ago

I’m not saying that all candidates should be interviewed without ever meeting or interviewing the candidate face-to-face; rather, I was speaking specifically of the coding interview portions. In these segments, face-to-face doesn’t really matter IMO, since it’s all about the candidate’s problem-solving ability.

Yes there’s a lot of “bias hysteria” out there, as you put it, but I would dispute that advocating “author-blind meritocracy” falls into that category.

Quite the contrary: An author-blind review process would actually make any bias impossible — either for or against any particular identity group. It seems to me most people should be able to get behind that, but maybe I’m wrong.

In fact, the main opposition to author-blind meritocracy is the “post-meritocracy” movement which is slowly making its way into open-source projects codes of conduct.

cthalupa• 8 years ago

I spend 8hrs a day in an office with my colleagues (sometimes more than with my wife & kid) and the ones I can't stand is about the only thing wrong with my job.

There's plenty of people I would never ever spend time with outside of work, and try to minimize my time with at work.

But that's fine, because 'cthalupa would like to have a beer with you after work' isn't part of the requisites for doing a job on my team. The 'Finding people that fit in with the culture'/'Finding people that I don't mind being around' is how you get monocultures and a lack of diversity in your team.

If we can't even see the person's face over some gender bias hysteria then I wonder how the hell we got here.

Gender bias is a real thing, a big deal, and certainly not hysteria. There's a lot of ways to reduce it. It doesn't necessarily require never seeing someone's face - though I think automating "skills" related interviews could be a good thing - because you can start with having a structured interview program where you have specific questions to ask and a specific rubric to grade against. Making sure you have solid, unbiased questions, and measure the answers evenly against the same rubric solves the majority of the problem.

People should just get over the fact that men and women are different.

Well, of course men and women are different. But even for jobs that involve heavy labor, this isn't actually significant - while the average woman has less physical strength than the average man, the type of woman who applies for that sort of job has self-selected into it, and is almost certainly more capable of doing that sort of work than the average woman - meaning it's still not a good indicator even for areas where the differences are largest.

For a white collar job, like the type we're discussing? It's even less relevant. There are extremely few times where you should ever care about gender when it comes to hiring.

arayh• 8 years ago

I agree that resumes are a poor means to distinguish between good and bad candidates. Humans already struggle with the screening process. There's no way an AI can reveal some kind of hidden secret sauce written into all great candidate resumes. This project was doomed to fail from the beginning, in my opinion.

jkingsbery• 8 years ago

(Disclaimer: I am an Amazon employee sharing his own experience, but do not speak in any official capacity for Amazon. I don't know anything about the system mentioned in this article.)

I am a frequent interviewer for engineering roles at Amazon. As part of the interview training and other forums, we often discuss the importance of removing bias, looking out for unconscious bias, and so on. The recruiters I know at Amazon all take reaching out to historically under-represented groups seriously.

I don't know anything about the system described in the article (even that we had such a system), but if it was introducing bias I'm glad it's being shelved. Hopefully this article doesn't discourage people from applying to work at Amazon - I've found it a good place to work.

To say something about the AI/ML aspect of the article: I think as engineers our instinct is "Here's some data that's been classified for me, I can use ML/AI on it!" without thinking through all that follows, including doing quality assurance. I think a lot of focus in ML (at least in what I've read) has been on generating models, and not nearly enough focus has been on generating models that interpretable (i.e., give a reason along with a classification).

darawk5• 8 years ago

It seems like they did think it through, though? And that's why it's being shelved. I don't really see what the story is here. It seems like the whole process worked exactly as it should - Amazon tried something, it had some unintended consequences, they caught it, and shelved it.

weliketocode2• 8 years ago

Agreed. There is no story here.

nobody271• 8 years ago

This never has been a problem about gender issues. It's about finding bugs in magical machine learning algorithms. Im sure the algorithm suffers from many other deficiencies but gender bias is the one people write articles about.

izzydata2• 8 years ago

It is no secret that more men pursue careers in this field. How can you expect any algorithm to produce an equal number of men and woman applicants? If it did this then that would be actually biased in favor of woman.

If this is what they want then they have to feed their neutral algorithm pre-biased data to get their expected results.

briandear3• 8 years ago

I think it isn’t equality in numbers but in activity screening out women.

So there may still be a 8:1 ratio of men:women but according to the article, it would seem than even that 1 women would have been negatively impacted.

I am not arguing the validity of their conclusions, just saying that I don’t think that it’s about equal numbers: given a male and female applicant, according to the article, women would have had an unequal chance of passing the screening. (Now if that inequality was for valid reasons, that to me, is an open question, but the article indicates that there was unjustifiable bias.)

izzydata2• 8 years ago

Fair enough. I'm still skeptical of their interpretation without more details, but anything is possible.

mankypro2• 8 years ago

Probably being used for age bias as well...

sanitycheck• 8 years ago

Given that Amazon is so far unable to successfully recommend any product which I actually want, even given the vast dataset of my Amazon purchase history, I am not remotely surprised that their engineers can't successfully develop a people recommendation engine either.

maym862• 8 years ago

- If you hired this male candidate you may be interested in this identical male candidate.

LinuxBender2• 8 years ago

What would happen if you removed gender from all HR and recruiting systems and then retrain the AI? Or for that matter, remove ethnicity, age, creed, etc... Is there any reason we need to be more specific?

Race: Human

Gender: Yes

Age of legal contractual consent: Yes

TangoTrotFox2• 8 years ago

The AI did not have access to gender. It was just word weighting and it turned out that words that could be linked to females ended up with applicants that had a negative outcome. Like the article says the AI ended up giving a negative weight to any resume containing the word women's as in women's [---] club, or those that mentioned certain all women's colleges.

LinuxBender2• 8 years ago

It should be fairly easy to filter / replace all of that. The same logic can apply. Can we add some simple filters and re-train it?

vishal_pym• 8 years ago

This is not very surprising - Machine Learning algorithms trained on biased datasets tend to pick up the hidden biases in the training data. It’s important that we be transparent about the training data that we are using, and are looking for hidden biases in it, otherwise we are building biased systems. Fortunately, there are open source tools out there that help audit machine learning models for bias, such as Audit AI, released by pymetrics - https://github.com/pymetrics/audit-ai

mankypro2• 8 years ago

Wouldn't an easy way to eliminate bias be to remove any algorithms that use name recognition and gender? If the Ai doesn't have this data to "reason" from wouldn't it level the playing field?

falcolas• 8 years ago

If, and only if, those are the only differences between candidate resumes. I don't think that's a reasonable assumption to make. Work history differences, sentence structure, word choices - all of these can quietly reflect gender differences.

paulcnichols• 8 years ago

Stupid question probably but why not have two models.

T-hawk• 8 years ago

Because evaluating differently based on gender actually is discriminatory.

commandlinefan5• 8 years ago

Well, that sounds about right, then - every proposed solution to subtle, unconscious bias inevitably turns out to be explicit bias in the other direction.

kaitai4• 8 years ago

I'm doing a bunch of ML on a very different data set -- looking at what people eat (survey data). What's interesting to me is that if you do principal component analysis, for instance, there are some differences between the boys & girls in the sample, but they're not very distinct. If you do clustering or random forests on the dietary intakes of the whole cohort, you get mushy and unclear signals. If you split the survey respondents and bin by age and gender, and run different models for each, suddenly signals jump out of clustering incredibly clearly! What's weirdest is that you get some of the same dietary clusters for the different demographic groups -- but those clusters were not evident when you did clustering across the cohort.

It's surprising to me that Amazon didn't (apparently) try different models for different populations. Sure, it might open you up to criticism, but there are some good data-driven reasons to do so. Women's colleges won't show up with regularity on men's resumes, for instance. Similarly, there are fraternities and sororities around engineering and STEM that may provide different signals, but won't appear equally distributed on men's & women's resumes. Language use on resumes does differ by gender, and using "Captured value of $100 million by..." rather than "Created value of $100 million by..." may describe the same project. (I gotta say, using verbs at all seems silly, since it really is about how well you market, rather than what you did.)

So, curious about the model. Different models for different subsets of the training data can lead to big wins.

rfeather• 8 years ago

Off topic, but I'm looking at similar data. Are you talking about a public source (eg NHANES) or something else?

foolfoolz• 8 years ago

this sounds like a bad ML tool to begin with because the input data sucks. it’s probay judging who can write a “good” resume. plenty of bad candidates write good resumes. and good hires write bad ones

whatgoodisaroad• 8 years ago

I feel like many comments here are taking this story on face value, but the cynic in me reads this as a planned leak to skapegoat a (hitherto unknown) AI system for their existing hiring biases. Public perception of AI is more aware of model biases nowadays, and we seem all to willing to accept this explanation over the simpler explanation that tech hiring at Amazon is broken in the same way it is everywhere.

macinjosh6• 8 years ago

This will be unpopular but I don't care. What is the evidence that the source data for this 'AI' is biased because the men it came from did not want to hire women? Is there a reserve of unemployed non-male engineers out there? If so what evidence is there of that?

Technical talent is both expensive and a rare commodity for tech companies. The non-male engineers I've worked with have always been exceedingly competent, smart, and their differing perspectives invaluable. If there was an untapped market of engineers you'd better believe every tech company would be taking advantage of it.

briandear3• 8 years ago

That is very interesting. What are the stats for unemployed female engineers? Is there a shortage of female representation because female engineers aren’t being hired or is there a shortage because there are actually fewer female engineers?

There is a shortage of male therapists and kindergarten teachers: is that because males aren’t being hired or because there are fewer of them in existence?

mrnobody_67• 8 years ago

Also airline pilots, nurses, michelin-starred chefs...

petsormeat3• 8 years ago

A lot of us drop out mid-career. I'd argue there's more a shortage of female engineering directors, VPs, CTOs.

commandlinefan5• 8 years ago

Yeah - I'm not even expressing an unpopular opinion, just asking a (leading) question: where are all these women who are chomping at the bit to get into _technical_ positions like programming but find themselves being turned away by biased recruiters? I've never even seen somebody _claim_ that they were a woman who couldn't find a tech job, just people wondering where all the women were.

bicubic3• 8 years ago

Last place I worked at had to invest in additional resources to hire several females because the office was mostly male and applicants were overwhelmingly male. We did eventually find a few great female applicants, but it took a lot of work and a lot of time dedicated specifically to that goal.

There does not exist some magical undiscovered pool of talented female engineers that are being turned away by biased recruiters. It's hard enough to find any sort of talented engineers regardless of other factors. Shit, it's not uncommon to recruit from other countries and cover relocation costs these days.

crooked-v3• 8 years ago

They all stopped joining up around 1985. https://www.npr.org/sections/money/2014/10/21/357629765/when...

arandr0x6• 8 years ago

They don't need to be currently unemployed. There are certainly enough women currently employed as programmers in the US to fill all the currently open Amazon positions, it's just it would be positively insane for Amazon to offer twice the salary and work-from-home to female candidates just to have prettier numbers.

Amazon (and a lot of other big non-diverse companies) are therefore hoping what is actually happening is that they are the women's first choice already, but turn them down for some reason, and that they would not have to change a thing about their work process to attract more women, except to start seeing them.

It's obvious why companies like thinking that way, and it's possible to some extent it's true. However, the fact is, if they're playing like this it's a zero-sum game and it is not actually going to improve the diversity numbers.

On my part, I've been wondering. If all these companies want to know where the women who can code are... why not just ask them? Why do you never see a "Female Programmer's Career Survey" with questions such as "On average, over the last ten years, how long have you been looking for a job?" "Would you accept a new job for a $5k raise?" "Have you ever dropped out of a hiring process because of sexism?" Take it out there in the open. Ask the real questions.

Dirlewanger• 8 years ago

It's taking a little longer than it should, but people are finally starting to realize that the actual reason there aren't as many women in software is because they've chosen not to be. They're wired differently and therefore have different interests.

happytoexplain• 8 years ago

I agree with you, but I have to point out, because it's so common: The "this will be unpopular, but I don't care" preface is, I feel, about as damaging to the perception of whatever you're about to say as "I'm not racist, but". To make an effective point, I think you should avoid, as the first thing you say, painting yourself as an underdog brave enough to speak out by preemptively criticising your audience's reactions that haven't even happened yet. That's not to say you should never admit those observations - rather, I would reserve such broad criticism of people's opinions for a separate train of thought or conversation.

crooked-v3• 8 years ago

Also, the "this will be unpopular" seems like it inevitably turns out to be virtue signaling for the "un-PC" people who will then aggressively make the thing popular anyway.

guitarbill3• 8 years ago

What is the evidence that the source data for this 'AI' is biased because the men it came from did not want to hire women?

One issue that keeps happening is an over-emphasis on CS-related questions. There are many great engineers I've worked with who didn't do a CS degree, and even though they are brilliant thinkers and talented engineers, too many times the interview question is "solve this problem using <pet CS 101 lesson, like red-black trees>".

And the number of people who are hired who can barely communicate effectively is still shocking. Very few interview questions focus on communication outside the technical realm.

So you can argue there is a bias in recruiting, simply because different people have different criteria for what the best traits/skills to look for is - even though everybody has the same goal, hiring the "best".

I'd also caution about taking Reuters too seriously though. Seems that they've only focused on the gender issue, but this is the money quote:

With the technology returning results almost at random, Amazon shut down the project, they said.

dudul2• 8 years ago

I don't understand how your example is an evidence that data came from mean who didn't want to hire women.

These interview processes exist sure, and I personally find them idiotic, but is there evidence that they disqualify women more than men?

bicubic3• 8 years ago

One issue that keeps happening is an over-emphasis on CS-related questions

No. If you go down that path, then you are implying that women do in fact perform worse at CS-related questions. That's a much bigger can of worms than the bias being implicated here.

petsormeat3• 8 years ago

Oh, there are a bunch of us, even here in the SF Bay Area. Trouble is, we're older than 35, or don't have degrees from "top" schools, and/or don't have the "passion" for bizarre extended hiring rituals. I could staff an entire dev team with non-male people within a week.

macinjosh6• 8 years ago

I don't have any sort of degree beyond high school and I have been a programmer/engineer for almost 15 years. Do I work at top tech companies? No. But that doesn't mean people like us can't be hired in the industry.

openbasic• 8 years ago

we're older than 35, or don't have degrees from "top" schools, and/or don't have the "passion" for bizarre extended hiring rituals

This hits home so hard.

jhfhhhf3• 8 years ago

So what do you do now? Btw some men in tech are also over 35 and tired of hiring rituals.

hamandcheese4• 8 years ago

I could staff an entire dev team with non-male people within a week.

I think the recruiting team at my company would very much be interested in speaking with you.

blub• 8 years ago

SW engineering can be a pretty brutal job psychologically - there's a reason people are burning out. Interviewing is particularly bad.

I guess most men have thick skin or got lucky, so they don't see that, instead they think it's this dream job and everyone should partake in its wonderfulness.

I believe that the lack of women in tech is explained by societal bias against women and the nature of the job.

zby6• 8 years ago

"The non-male engineers I've worked with have always been exceedingly competent, smart, and their differing perspectives invaluable" - that is really an (anecdotal) evidence that there is a bias indeed. If the recruiting was all unbiased than the quality of existing male and female force would be the same - if female workers are of higher quality than it means that they need to pass higher requirements.

macinjosh6• 8 years ago

I didn't mean to imply the males I worked with weren't equally good. I was simply pointing out that, in my (anecdotal) experience, I couldn't see any reason non-males couldn't do the job well.

gspetr4• 8 years ago

Another corollary should immediately follow: If an average woman in the industry is that much better than an average man, where are all the female-only companies?

Does nobody want money in a capitalist society?

amanaplanacanal• 8 years ago

Two comments:

1. The issue is certainly bigger than hiring. In the many years between birth and looking for a job, there are a lot of societal pressures that will impact what eventual careers people end up in.

2. Hiring managers are people. They are not perfect. They have biases. If someone expects an engineer to look, talk, and act a certain way, that can impact their decision making completely independent of the fact that they want to hire the best people for their company.

Bonus third point: I still see a whole lot of "We want to make sure that the hire fits on the team." This is completely natural, and comes with its own set of built-in biases.

Marazan2• 8 years ago

If there was an untapped market of engineers you'd better believe every tech company would be taking advantage of it.

That's exactly the same argument used to justify every regressive policy. If x was true then rational y action would happen.

But that's the poo tof racism and sexism rational y action doesn't happen due to the -ism.

macinjosh6• 8 years ago

Do you really think white men care so much about keeping down other groups of people that they would prioritize it over making more money and having more access to good workers? That sounds like a conspiracy theory to me.

beat18• 8 years ago

An untapped market of engineers would be tapped...

...if and only if...

...there were no other factors at play that cause that market to remain untapped.

For further rational thinking, consider this. If there's a bias, it doesn't mean women won't get hired. It just means they won't get hired for the best positions. Everyone else gets Amazon's cast-offs.

kat• 8 years ago

"Untapped market of engineers" yes it exists. The majority of my female friends with STEM degrees ended up as high school teachers. I had several older people suggest to me that teaching should be my preferred career choice because it was more flexible than a programming job (wtf...)

"every tech company would be taking advantage of it" - nope, no one is. I don't know why but my guess is its hard to admit you're doing hiring wrong, hard to hire people who think differently than you, etc.

jhfhhhf3• 8 years ago

Maybe being a high school teacher is simply more pleasant than working in tech to some people? I don't think your example shows there is an untapped talent pool.

Of course, in general, you can make a job more attractive (raise salaries, roll out red carpets, install slides...), and you will attract more people. That doesn't prove those people were an untapped talent pool.

Presumably there is a price that would make a high school teacher consider working in tech again. That doesn't imply companies should be willing to pay that price.

totony• 8 years ago

To note, there are more women than men in high school teaching positions, so what you're seeing might not be that STEM has a bias against women but that teaching has a bias for women (or any other out of a very big set of possible conclusions)

see: https://stats.oecd.org/Index.aspx?DataSetCode=EAG_PERS_SHARE...

hamandcheese4• 8 years ago

Why did they choose being a high school teacher?

Are there specifics about hiring processes in tech that bias against or scare away female candidates?

jahewson• 8 years ago

Some reasons why I disagree with this polemic:

1. There's no reason to expect that these women will be unemployed - they just won't be working for Amazon. That's all we know. No point going looking for them.

2. You can't assign intent to hiring decisions made in the training data - there's no reason to believe that men (and why single them out?) "did not want to hire women". Maybe they did. Maybe they have no idea that they're biased - maybe the women making such hiring decisions are just as biased. We have no idea.

3. The evidence that the AI is biased, is that.... the AI is biased. Which means that the training data is biased. Why that is, is a great question - it may reflect unconscious bias in the hiring process, or more obvious old-fashioned biases. It may reflect that the model amplifies some minor bias in the training data and turns it into something much bigger. We don't know.

So yeah, it's biased - the question is why.

s73v3r_6• 8 years ago

"What is the evidence that the source data for this 'AI' is biased because the men it came from did not want to hire women?"

Nobody is saying that the biases was caused because it was created by men who didn't want to hire women. That's a fear-mongering straw man.

What people are saying is that there was bias in the training data selected, and so the algorithm exacerbated that bias. Thus, being a cautionary tale about the training data you feed to these things.

" If there was an untapped market of engineers you'd better believe every tech company would be taking advantage of it."

You're assuming rationality where there really is no cause to do so.

rrcaptain5• 8 years ago

Biases are rarely knowingly done, that's why they're biases. The proof is in the pudding though. The model Amazon came up with was biased against women. That suggests that female candidates in the dataset were discriminated against.

briandear3• 8 years ago

“Instead, the technology favored candidates who described themselves using verbs more commonly found on male engineers’ resumes, such as “executed” and “captured”

This is the most interesting part to me. It’s suggesting that men and women think about things differently. Which, if true, suggests a lot of other possibilities that have relevance when hiring for certain positions.

40acres• 8 years ago

I know diversity in tech is a hot button issue but I think hidden underneath the politics of it all the most important reason to champion more diversity is AI.

I don't think we're anywhere close to general AI so any AI system out there is built on what we feed it. If your music suggestion algorithm has a biased AI that's one thing, but when you're using AI to make critical decisions in society like who gets hired/recurited, medical diagnoses and other things you need to be extremely careful.

When designing AI models the broadest set of viewpoints should be considered. Any piece of AI is simply a reflection of it's creators, we need to make sure that some sort of equitable consensus is reached before deploying AI to critical human scale issues.

patrickg_zill• 8 years ago

What if we trained a bunch of robots with AIs embedded and they turned out to be racist misogynists? There has got to be a science fiction story about that...

castlecrasher2• 8 years ago

You mean like Tay?

daenz2• 8 years ago

It seems that the only socially acceptable output for the AI would have been hiring women 50% or more and hiring minorities at a rate greater than or equal to their representation in the populations. Anything else is clear bias and discrimination.

The project was doomed from the start.

HashHishBang2• 8 years ago

Hold on here. This article seems to have buried a pretty important piece of information wayyy down in the middle of the text.

Gender bias was not the only issue. Problems with the data that underpinned the models’ judgments meant that unqualified candidates were often recommended for all manner of jobs, the people said. With the technology returning results almost at random, Amazon shut down the project, they said.

Granted an article isn't going to get as much attention without an attractive headline but that seems a far more likely reason to have an AI based recruiting recommendation scrapped. The discovery of a negative weight associated with "women's" or graduates of two unnamed women's colleges is notable but if it's tossing out results "almost at random" then...well there seems to be bigger problems?

user812• 8 years ago

The media is no longer reporting things. You can't make money with reporting. The media is actively creating narratives, and one of the narratives that people are fed nowadays is that women are victims.

Men and women are pitted against each other.

Due to the way the media has evolved people consume their own biases and most often just read the headlines.

HashHishBang2• 8 years ago

I mean, yeah you're not wrong. I try not to be too cynical about the whole thing even if I think the narrative is suspect. Yes women and minority representation in tech is a potential issue but I really want to know more about the AI recommendation system for potential hires. Especially if it was giving out spurious recommendations.

It's amazon, I can't imagine how many millions went into something like that. We'll almost certainly not get a postmortem but it's definitely intriguing.

njharman• 8 years ago

No longer? Read some history, start with yellow journalism and muckraking.

Learning history will teach you more about things today than any news source.

rrcaptain5• 8 years ago

The media has always been about narratives.

Faark• 8 years ago

The article leaves a lot open to interpretation, including what was expected of the tools... that could range from providing some beneficial hints to replacing all hiring. As you rightfully remark, having the tool rank candidates for highly specific jobs and its tech requirements would be a great achievement. But is also a big challenge, thus they probably were aiming at something more basic initially. Building models for broad categories like "manager" or "box packer" and hope they will detect soft skills or work ethics seems more achievable. Thus the additional star rating that can be used for hiring and provides some value.

Now having known limited capabilities isn't great. But those can and will be worked on. Unknown / unexpected biases wont, making finding them important.

leeny• 8 years ago

I tried to do something similar a while ago (for eng hiring specifically). It turned out that the number of grammatical errors and typos mattered way more than anything else on a resume.

http://blog.alinelerner.com/lessons-from-a-years-worth-of-hi...

That aside, what sucks is that attempts to automate resume scoring rarely look at harder-to-quantify features and focus on low-hanging fruit like keyword occurrences... though in my experience it's such a low-signal document for engineering hiring that the whole thing is a fool's errand.

lawnchair_larry• 8 years ago

So why is it not ok that women are penalized by being statistically less likely to get an offer yet it’s just fine to penalize men on auto insurance for being statistically more likely to cause an accident?

maym862• 8 years ago

Because men are statistically more likely to cause auto accidents. There is no evidence that women are not statistically more likely to be bad at getting good work done at companies like Amazon. It's pretty simple.

crimsonalucard2• 8 years ago

Do you have a source for your second statement?

zakm• 8 years ago

I'm sure that's true, but do you have any evidence for it?

arandr0x6• 8 years ago

The idea that Amazon is trying to enforce diversity by using an algorithm that is made to detect, and match, patterns boggles the mind. Why, yes, if your recruiting cost function is "is that person just like all the others we hired", you will end-up with a non-diverse workforce, no matter whether the model optimized with this function has 20 layers, 250 hyperparameters, or two legs, two arms and a fast-receding hairline.

You can de-bias by explicitly controlling for gender, but now everyone in your company went to CMU and likes dogs.

The more I see news about what recruiting for ultra-large corporations, the more I think one of two things is true:

* ultra-large corporations are doomed to hire less and less well in a way that is more and more biaised, and we should regulate against such corporations in a way that forces them to redistribute their wealth to SMBs;

* ultra-large corporations need to start exclusively growing through acquisitions, which will have the effect of redistributing their wealth to SMBs, and also of hiring a more diverse base of employees because there is a priori a greater diversity of backgrounds leading to success in the free market than the diversity of backgrounds leading to success in the Amazon interview.

thoughtexplorer5• 8 years ago

Anyone who has worked at tech companies and has been involved in the hiring process knows the following:

The best thing to be right now is a woman engineer. You can easily get hired within the week.

Unfortunately this doesn't seem to be well known outside of those involved in hiring.

fizwhiz3• 8 years ago

Is this anecdotal? Are you insinuating that women have an upper hand now for reasons beyond their skillset?

thoughtexplorer5• 8 years ago

Yes. There is a bonus to being a woman in this situation, all other things being equal.

Silicon Valley did a good bit on this https://www.youtube.com/watch?v=Dek5HtNdIHY

It's funny because it's true.

InclinedPlane2• 8 years ago

This is simply silly. The reason why there's a bias toward hiring women in some roles at some corporations is because they are trying to course correct for the massive amount of systemic bias pushing the other way. For some reason many people (mostly men) seem to easily spot the one type of bias while never being able to see the other.

darawk5• 8 years ago

There's something I don't understand in stories like this. It ought to be relatively straightforward to correct biases like this. All you need to do is train a model to explicitly classify gender from resume's, and then use that model to de-gender resume's before passing them on to the hiring model. Is there some reason people aren't doing this?

xarill• 8 years ago

If the gender was passed as a "feature" in the machine learning model, then the company may have been biased. If not, then the majority of the women's CVs were inferior. This important information is not present in the article.

core-questions• 8 years ago

This entire article is a strong hint that the AI system simply picked the better resumes, where better = more likely to get hired, exactly as it was designed to do. The fact that more men than women satisfy these requirements should come as a surprise to precisely nobody.

Fact is, there are more men than women working in tech, especially seriously hard-core stuff like these big companies need. This is most likely out of their own personal volition, and no matter how much outreach we do, this is likely to stay the case for generations.

Of course, that's also an egregiously wrongthink position to take. Double plus ungood.

beat18• 8 years ago

Garbage in, garbage out.

The system failed because they were trying to solve the wrong problem, or maybe more specifically, didn't solve the problem that led to the problems with the AI. Amazon was treating the hiring problem as an efficiency problem alone, and ignoring the bias problem. So they wound up training the AI to do a shitty job much faster than humans ever could be shitty - and, by analyzing the data in a way the human results weren't analyzed, showed the failings of the human hiring process.

Existing process is sexist. Automate to "improve" it, and you wind up with something even more sexist. What this means is that Amazon needs to go back and revamp their whole hiring process to make it fair, before trying to make it faster.

pcmaffey• 8 years ago

So they wound up training the AI to do a shitty job much faster than humans ever could be shitty

If nothing else, modeling our existing behavior in this way is a great use case for ML. As it allows us to "fast forward" and thus--hopefully!--identify our flaws based on modeled iterations.

fromthestart• 8 years ago

Things are going to get very interesting when AI inevitably reveals consistent differences in performance among genders.

There's only so much you can sweep under a rug.

jiveturkey• 8 years ago

Is it a Microsoft Tay problem?

otakucode• 8 years ago

It seems strange to me that they would train the system on resumes submitted to them over the past 10 years. I presume they judged the ones they hired as 'successes' and those they did not as 'failures'? Or did they judge all submissions as successes, expecting to be testing for 'does this person fit in with the group of people who submitted to us over the past 10 years?'

Neither matters much, neither makes sense. What they should have been doing was training it on the resumes submitted by employees who then went on to be very successful within the company. Those are the successes. Those are what you want more of. And, probably far less likely to be weirdly biased by gender or race or whatnot.

Jyaif• 8 years ago

Every product in development is going to be secret, but Reuters tries to make it sounds as if some sort of conspiracy was going on. What's next? "Jeff Bezos secretly goes to the restroom"?

RangerScience• 8 years ago

One of the first good critiques of the "oncoming AI apocalypse" that I've seen is exactly this: the encoding of existing biases into the resulting AI system.

But, it also means that bias is now measurable...

...and there were more than a few papers at NIPS that were directly dealing with "fairness" in a NN, aimed at addressing and using these issues and effects.

Jtsummers• 8 years ago

Weapons of Math Destruction offers good discussion on this problem, if you want to read more about it.

foxhop• 8 years ago

I'm pretty sure this tool reached out to me at one point. Here is the email excerpt:

http://pad.yohdah.com/638/amazon-ai-recruiting-email

crimsonalucard2• 8 years ago

Once I worked for a startup selling fresh human baby milk to mothers who couldn't produce milk. I was contracted to write a supply side AI that would find people willing to sell the milk they produced to us. The AI had the exact opposite bias as the one in this article... it showed extreme bias against men.

I also wrote a similar AI for finding surrogate pregnancy candidates and it also showed bias against men.

Goes to show how AI can fail and be incredibly sexist.

screye2• 8 years ago

The article never mentions what form of supervision Amazon used for building this model.

I can easily see a bias towards a particular gender arising, even when a team had intentions to do, simply because of how the data is selected.

________

Case 1 : They used similarity statistics between candidates that were hired and applicants.

This is the easy one. No need to label datasets. The approach is semi-supervised. It will also 100% cause a bias towards employees with similar profiles as those already working at Amazon. (ie. men)

________

Case 2 : They manually labelled/ranked a dataset of resumes and assigned them scores. (more likely)

Here the implicit bias of the mechanical turks / rubric would be visible. If higher scores were assigned to traditionally masculine activities, then male resumes would stand out. I doubt this was the case though, as people at Amazon are generally competent enough to not make such a trivial mistake. Also, gender only gets mentioned in non-technical skills (mean's team, women's club, etc), which in general are not the most relevant part of the profile anyways.

________

Speculation:

Instead, the technology favored candidates who described themselves using verbs more commonly found on male engineers’ resumes, such as “executed” and “captured,” one person said.

I wonder if this has anything to do with gender at all. All good resumes that I've read use action words irrespective of gender. Maybe type A personalities use words like "captured" more often than type B ones, and the % of men in type A categories are greater. Would discrimination against women in such a case be unfair ?....Maybe.....Maybe not.

Extracurricular activities are more prominent on weaker resumes than stronger ones. So, the model may be weighing down resumes with too much extracurricular fluff vs technical skills. Men's activities are rarely prefaced with the word "men" in it. (they would just say Football team, Chess team). Women's activities on the other hand, always have the word "women" attached to it. If the extracurricular activities were penalized, then the words inside of them, including "women" would also be penalized. Thus, the model learns a latent gender bias without any bad intentions.

_________

In ML, one of my favorite statements is : "The model is only as good as the data it is trained on." If the data is not sufficient, rich enough or prepared in the correct manner, then unintended consequences are nearly guaranteed.

apercu• 8 years ago

The discussion here (versus other channels) is so much better. Not that I expected any different. I'm just glad no one is using this media report to confirm their own biases.

noetic_techy2• 8 years ago

The problem is you cant feed the ML algorithm training data based on what your company currently looks like, you have to feed it an idealized set of what you want it to look like. It almost needs to be fictitious training data to hide the ugly bias that's already built in.

I don't think this will ever work. There is too much variability in resume wording that correlates to gender and even culture of origin even when you take out names and any other protected class identifying markers. The Dutch tried this and ended up with less diversity.

I'm going to go out on a limb and say you almost want to leave all that identifying data in, but put each candidate into buckets with separate rating algorithms trained against only that "type" of candidate. The top candidates from each culture, and the top candidates from each gender, etc etc, however you want to do it. Feed them into a picking algorithm that builds a composite of what you want your team to look like diversity wise based on the top candidates from each bucket, and go from there.

Don't take my opinion seriously, I'm not an ML guy.

gumby• 8 years ago

I hate the clickbait way in which this story has spread across the net. "secret AI recruiting tool" sounds like Amazon did something nefarious. Instead they built a tool, found out it was broken, and didn't deploy it.

The actual newsworthy part, which is getting slightly stale, is that it was influenced by the data bias.

I am not even a fan of Amazon but I think this is unfair to them. They did the right thing here.

hahan4• 8 years ago

Call me cynical but I found it amusing that no one points out the fact that engineering work is laborious and dry to say the least for most people. That's the reason why there are so few people who have other options, females, upper middle class people, people of means, in the scene. There are so many lowish paid low status unsought for sectors where majority workers are male, say janitors in Universities, why I never see any discussions on that bias ?

anon0007658• 8 years ago

Because, though logical, it's a highly unpopular thing to discuss. Narrative has replaced critical thinking in many areas, this being one.

zaksoup4• 8 years ago

Cynical, or wrong? There's a clear history of women in software engineering when it was considered a low-class low-skill job. The first "programmers" in the modern sense of the word were women at NASA. The lead software developer for Apollo 11 was a woman. There's so much evidence to suggest that there is far more at play here than simply "engineering is laborious and dry [so women must not want to do the work]" that I'm willing to state it falls under wikipedia's citing rules for globally known knowledge: I don't need to cite that the capital of France is Paris any more than I need to cite this.

This is the same line of already-refuted reasoning behind the "I'm just asking questions" in the infamous Google Memo.

To answer my original, rhetorical, question: It's not cynical. It's wrong.

_Tev3• 8 years ago

There's so much evidence to suggest that there is far more at play here than simply "engineering is laborious and dry [so women must not want to do the work]" that I'm willing to state it falls under wikipedia's citing rules for globally known knowledge: I don't need to cite that the capital of France is Paris any more than I need to cite this.

What about citations proving that "software engineering when it was considered a low-class low-skill job" is the same profession as the programming in the past 30years. Or at least that it has the same difficulty / processes.

Btw usage of phrases like "clear history", "so much evidence" (especially when you cite one(!) arguable data point), "already-refuted" does not convince anyone about you being right. It is at best annoying.

hahan4• 8 years ago

While I agree with the early days female programmers history thing, I'd like to point one thing you've neglected, that programming work 60 yrs ago is not comparable or even relevant to that of today. They are of different nature. What's farther from the reality is , back those days programming was not particularly losing paying or low status, I'd say it was elite. After all, most people then had no access to tertiary education, let alone computers.

skywhopper• 8 years ago

You're saying that software engineers are in the role because they have no other option? No upper-middle-class people in the position? What are you talking about?

You're missing the point anyway. The article made it pretty clear that this AI amplified biases humans already have about women applicants to tech positions. Stating your own biases about women make no sense to the topic, or to the argument you seem to be trying to make.

Janitors are worth talking about as well (women in the same job usually have a different title with less authority and less pay), but high-status, highly-paid, highly influential jobs are where it's most important to avoid bias, and so we talk about those more.

hahan4• 8 years ago

You're saying that software engineers are in the role because they have no other option? No upper-middle-class people in the position? What are you talking about?

There are enthusiastic people who start with enthusiasm and can keep that enthusiasm. Most begin with practical concerns. And most of those who don't begin with stay with. he

And yes, rich people, upper echelon, people of means are rare in the industry

hahan4• 8 years ago

And of you think programming is high status, high paying, highly influential, why we import immigrants to do that ? You think the talent pool of some hundred million American people with most entensive education system is insufficient ?

rrcaptain5• 8 years ago

You don't see discussion on that because you're on a tech industry website, not a janitorial forum. And the percentage of women in janitorial work is nearly twice that as the percentage in tech per BLS: https://www.bls.gov/cps/cpsaat11.htm

arountheworld• 8 years ago

Have they not removed protected characteristics from the data? If they had, AI wouldn't know about gender and there would be no bias.

just_myles• 8 years ago

Gender. I am more concerned about race and ethnic background. Why is this never addressed?

chiefalchemist2• 8 years ago

"Gender bias was not the only issue. Problems with the data that underpinned the models’ judgments meant that unqualified candidates were often recommended for all manner of jobs, the people said. With the technology returning results almost at random, Amazon shut down the project, they said."

Doesn't this mean the headline is incorrect, incomplete and/or (intentionally) misleading?

c3534l2• 8 years ago

I'm honestly pretty shocked that Amazon went ahead with this which is such an obvious legal liability in an extremely touchy area of business practice. Everything about this should have been a red flag that meant it never should have gotten off the ground, let alone been in place for years.

megy2• 8 years ago

You don't think they should try new things, even though they might not be successful?

c3534l2• 8 years ago

When it systematically and illegally oppresses a class of people in an obviously ethically dubious manner, I would expect a certain level of caution.

iamleppert• 8 years ago

The thought that you can somehow suss out with any consistent performance who will be good or not at their job based solely on a training set consisting of resumes is so banal and inept that its laughable. Team building has so much more to do with assembling the right kinds of people with the right talent and personalities that no present day machine learning could ever hope to come close in effectiveness to a thoughtfully applied interviewing and hiring process.

It is clear whatever executives in charge of this project haven't the first clue of whatever it is they are doing. It lacks not only any technical deliberateness but also fails the common sense test. I really wouldn't want to be working for such people.

ramblerman2• 8 years ago

How do they measure non-bias? i.e when is it working as intended?

My only worry is that they are aiming for 50/50 which doesn't reflect the underlying gender ratio of the developer pool.

Amazon scraps secret AI recruiting tool that showed bias against women

Related Stories