HackerRank open sourced its ATS. My resume scored 90/100. Oh wait 74. No – 88

https://danunparsed.com/p/hackerrank-open-source-ats

by sambellll • 24 days ago

1032 407 24 days ago

glouwbug• 24 days ago

I guess at least HR doesn’t have to read 1,000 resumes. Heck, to be frank, could they make sense of the first 10 resumes?

dc3k• 24 days ago

Disregarding the fact that this thing is completely broken, its grading rubric is ridiculous to begin with (as was mentioned in the article itself, but I must reiterate how completely stupid this is):

35 points for open source contributions
30 for personal projects

I don't contribute to open source or have personal projects because I don't spend my free time doing what I do 40 hours a week to make a living. My 15 years of work experience is worth a maximum of 25%, so any company using this idiotic system would pass on me immediately. Open source and personal projects are fine, but in no sane world are they worth 65% of a resume's score.

adrianN• 24 days ago

They are selecting for people who are fine working in their free time. If you contribute to open source you are more likely to contribute to the company on weekends. If instead you have other hobbies or a family that takes up non-work hours you are more likely to drop your pen after forty hours.

emj• 24 days ago

You might have numbers on that but after working in a place with a strict no more than 40 hour policy my view is that people overwork for many reasons. Being an open source enthusiast is not one of them.

stevesimmons• 24 days ago

I'm not sure that follows. I stopped making open source contributions when I switched from mature companies to startups.

Now all my "non-work" time is spent on startup work. And none of that is visible via GitHub.

matheusmoreira2• 24 days ago

Maybe they're selecting for intrinsic motivation. People who enjoy programming to the point they do it for fun, not just because it pays.

Free software work doesn't imply we work for free. We work on our projects, the stuff that we actually enjoy working on. Nobody is going to work on corporate products without adequate compensation.

another-dave4• 24 days ago

If you contribute to open source you are more likely to contribute to the company on weekends

I wonder if that assumption is bourne out in reality though?

I'd imagine if someone's OSS contributions are enough of a factor that it's worth hiring them, they're not going to drop it on a whim to work extra hours on the day job.

(Assuming you weed out open source contributions like "I made a todo list app in React but licenced it as MIT" or "I fixed a typo in the docs for NextJS". )

jerrythegerbil2• 24 days ago

I fail 65% of the time. Same exact resume, different luck.

As someone who’s run hiring pipelines for technical roles in the past few years, that’s actually a fantastic number. I objectively hate saying that, but it’s true.

35% chance of elevating a technical individual to the next stage with no effort? I’ve seen as many as 100+ applicants an hour even when including a domain specific screener question. That’s 35 “screened” applicants in an hour. Were valid candidates screened out? Yes. Does you still have a candidate pool 35x larger than you need? Unfortunately, also yes.

The volume of applicants is SO HIGH such that your chances of getting moved to the next stage are actually markedly worse if AI isn’t involved. If you didn’t apply immediately (using an AI bot) there’s 50+ people ahead of you, and an exhausted technical leader if they ever make it to your resume.

Referral bonuses exist for a reason.

kyralis• 24 days ago

Is it? Or is it a 65% chance of a resume getting ignored before a single human sees it, reducing your pipeline's likelihood of catching qualified candidates by the same?

Gates that reduce resume flow-through are only useful if their reduction is correlated with quality. Otherwise they're just dragging out your hiring process or unnecessarily causing you to ultimately lower your hiring bars.

bagels• 24 days ago

The goal for the interviewer is to have a much higher ratio of good/bad candidates after the first screening. This means the more costly time you spend on the second step has a better return.

jerrythegerbil2• 24 days ago

Gates that reduce resume flow-through are only useful if their reduction is correlated with quality.

The volume is infeasible to review everyone for quality, even at an hour scale. The conclusion and solution is inevitable, though I wish it were different. 35% is actually really good if you’re not coming in through a referral.

The current reality is <1% and the person reviewing you is exhausted.

aesthesia3• 24 days ago

So the question is: is the score given by this system correlated with candidate quality? I don't think this post gives enough data to know.

lowbloodsugar• 24 days ago

Except the bit about ranking a decades long S3 engineer lower than an intern with GitHub repo.

PufPufPuf• 24 days ago

In that case, I have a pre-screening system to sell you. Through state of the art technology, it only lets through the best* 1% of applications.

*According to our proprietary, undisclosed, non-deterministic metric, which may or may not be Math.random

rvba• 24 days ago

Reminds me of this

https://stackoverflow.com/questions/16833100/why-does-the-mo...

groundzeros20154• 24 days ago

I worked at a startup that judged their hiring pipeline quality using rejection rate criteria.

spike021• 24 days ago

there have got to be better ways to optimize pipelines. maybe set a limit on number of applications for a role based on the number you/your team can reliably go through them. if more are needed then open the role for another wave of applications.

ludicrousdispla• 24 days ago

So the logical solution is for candidates to submit multiple applications with slight variations to their contact info, "John Schmidt", "John J. Schmidt", "John J. J. Schmidt", "John Jacob J. Schmidt", "J. J. Jingleheimer Schmidt", etc.

ambicapter• 24 days ago

It's a good day to have 3 middle names.

yuliyp• 24 days ago

Hey, that's my name too!

IshKebab4• 24 days ago

I wonder if you could solve this for programming specifically as follows:

1. Give them some easy leetcode questions. Nothing that a competent programmer would have any problem with.

2. If they pass, ask for a deposit of like $20. Shouldn't be an issue for people who are actually serious.

3. Do more simple leetcode questions but this time on zoom so you can tell if they are using AI. If they pass that they get the deposit back.

(Yeah I know there are real-time interview cheat AI programs but based on what I've seen on demos of them it's super obvious when they're being used.)

Probably not practical but just a thought!

never_inline• 24 days ago

This selects for desperation.

jghn2• 24 days ago

I'm not going to do any of those 3 things for a would-be employer.

recursivecaveat• 24 days ago

If you have no requirements for accuracy, you can just advance 35% of applicants at random.

If the first 50 people who apply are all bots, why are you reading resumes in order of submission?

wodenokoto• 24 days ago

One of the first things you do when hiring is to set a period and randomize order of resume when reviewing because early application is not a strong signal.

mrhottakes2• 24 days ago

Sounds like you're pretty bad at hiring pipelines.

punkbit• 23 days ago

Are you suggesting using AI/Agents to apply for roles?

dvt2• 24 days ago

An alarming number of people don't understand that LLMs work via purely stochastic processes, so I'm happy to see in-depth pieces like this. I'm looking for a job and maybe this is why it's so hard to get a callback these days: resumes are just dumped in some LLM black hole and no one really knows how it works. The author says:

temperature 0.1 — low, supposedly nudging the model toward deterministic outputs

This is not correct (and is briefly touched on later in the piece when he sets temperature to 0), temperature is not some kind of "deterministic" switch, but rather it affects the sampling distribution (which becomes more "spiky"—but is still very much a distribution).

bluechair• 24 days ago

Willing to be corrected but I believe this type of automated resume filtering is illegal. Not saying it never happens but my understanding is it is not typical.

small_scombrus• 24 days ago

They don't need to actually filter/blackhole to have have the same virtual effect.

Show someone a list of resumes with an "applicant score*" and they'll naturally ignore the ones with a low ranking

*scores are generated with AI, mistakes may be made, use only as a guide and verify results

thayne• 24 days ago

I would expect that to depend on jurisdiction.

I don't know for sure, but I would be surprised if it was illegal in my particular US state. You might be able to argue the AI has inherent biases that introduce illegal discrimination in the hiring process, but my understanding is winning I case like that would be very difficult, especially since most employers are very cagey about their hiring process and why they mades a decision.

ivan_gammel2• 24 days ago

In situations when you get hundreds of applications for one open position (real market now), whatever reduces your pool to the size a human can handle, works. You can preserve some diversity metrics in the process. This particular filtering is rather primitive, but LLM as a first filter can definitely do the job. You may burn less tokens than the hourly rate of your HR and it will be fairer than just dumping 50% of unread CVs in trash.

dgellow• 24 days ago

Illegal where?

elric• 24 days ago

Under GDPR, you have the right to request manual processing whenever personal data is processed automatically to make a decision about you that has "significant impact". Not being hired seems like it would qualify.

aesthesia3• 24 days ago

A distribution with all probability mass on one outcome is deterministic, so in principle, setting temperature to 0 _should_ result in deterministic outputs. There are a few reasons it might not, but I don't think any of these apply when running a local model like the author did.

valzam• 24 days ago

I mean the easiest explanation would be that the model harness doesn't always take the most likely token but does top-k sampling or similar. temperatur just means that probabilities get more and more equalized, boosting the chance that an unlikely token gets picked. but even with temp 0 you could have 0.8 T1, 0.19 T2, ... and sometimes sample T2

easygenes• 24 days ago

There are. If the kernels are nondeterministic (e.g. timing issues) there are minor changes between runs, on a single system, even with eager decode enabled (typically what temperature=0 achieves).

IshKebab4• 24 days ago

Setting the temperature to 0 should give deterministic results but that's not any better - it's just hiding the huge variance by only taking one sample.

3170705• 24 days ago

so in principle, setting temperature to 0 _should_ result in deterministic outputs

It is a common misconception, but it is not true even in principle. If I have 2 or more logits which are equal to the maximum of my logits, I will sample uniformly random from them with any temperature, even zero. Sampling from softmax([1, 0, 1]) is still stochastic at temperature 0, because the limit is to sample uniformly from the first or the last element.

Anyway: "GPUs don't do deterministic matrix multiplications" is the biggest source of randomness in LLMs. GPUs put the associativity of the sums in matrix multiplications in arbitrary order, and this has a huge impact on the logits coming out of the neural network.

croes• 24 days ago

So you would get always the same result, but it could be the wrong one

make3• 24 days ago

A more spikey distribution exactly makes the distribution closer to deterministic. That's not the point though. Even in greedy (deterministic) decoding, it is still a black box though that reacts in ways ways that are unpredictable to the inputs. Switching one word around might lead to different scores for example.

fluoridation• 24 days ago

Yeah, this is the forest that the people arguing about math trees are missing. It doesn't matter that the algorithm is deterministic if the algorithm passes the input through a cryptographic hash function to make a yes/no decision. The result may be perfectly reproducible and still non-sensical in its distribution with respect to its input domain.

miki1232112• 24 days ago

In theory, temperature 0 does make the LLM deterministic.

Well, in theory theory, temperature 0 doesn't really exist. Mathematically, as lim temperature->0, the distribution gets spikier and spikier, the most likely sample goes to almost-but-not-quite infinity and the rest go to almost-but-not-quite 0. In practice, temperature=0 is literally a separate branch of an if statement that just picks the most common sample (using the actual formula that works for non-zero values would cause a zero division).

However, due to things such as batching and even different kinds of floating point imprecisions for different algorithm implementations, the probability distribution itself often differs run-by-run, so what you sample from it also differs.

sigmoid103• 24 days ago

in theory theory, temperature 0 doesn't really exist.

It does exist very much, even if you go to pure math. Look at the softmax function and take the limit as T->0. It becomes a dirac-delta function. I.e. in a discrete setting (like for LLMs with a finite set of output tokens), probability P becomes one for argmax and 0 for everything else. Only in coding practice it is easer to implement T=0 as a simple if check that directly chooses argmax instead of calculating the limit of some function that includes 1/T quotients. But setting T to zero is in both, theory and practice, turning the usual probability function into greedy sampling.

lelandbatey• 24 days ago

As I understood it, the "randomness" affecting what is selected at any temperature still comes from a PRNG or CSPRNG (or whatever RNG you want, maybe a hardware one), and if you where to swap out that with something deterministic you'd get the same results every time (barring non-determinism in other parts of the OS/drivers/maybe even hardware).

But theoretically, the output of every LLM is seed-driven (or could be if you wrote the software to isolate it) just like any computer software. It's just none of the software written (even llama.cpp AFAIK) chooses to support stable-seeding due to the changes in stuff like CPU/Vulkan/CUDA/Metal differences making it difficult to make consistent.

They could though! Hopefully one day someone implements it into the mainstream LLM-engine software and it gets exposed in the APIs serving the models. It'd do a lot to show folks the "internals" of these models.

chrisjj2• 24 days ago

However, due to things such as batching and even different kinds of floating point imprecisions for different algorithm implementations, the probability distribution itself often differs run-by-run

The implementation does not often differ run by run.

nullc3• 24 days ago

If you make an exact integer implementation and run with temp=0 it's deterministic.

You don't even need temperature 0, just make a random seed for the sampler part of the input and then its deterministic as a function of the input.

But running autoregressive models at temp=0 tends to expose pathological behavior, because the training process produces a function with a lot of gain so its prone to feedback on its own noise.

teiferer• 24 days ago

Mathematically, as lim temperature->0, the distribution gets spikier and spikier, the most likely sample goes to almost-but-not-quite infinity and the rest go to almost-but-not-quite 0.

That's not how limits work. As the temperature goes to 0, the rest goes to 0. That's it. The "almost-but-not-quite" is part of the "goes to".

Let's say f(x) = 3x+1. It's a continuous function. If we let x go to 10, f(x) goes to 31. Not "almost-but-not-quite 31". No, to 31. (If you don't have a continuous function then it's the same argument, but less intuitive to illustrate.)

msdz• 24 days ago

However, due to things such as batching and even different kinds of floating point imprecisions for different algorithm implementations, the probability distribution itself often differs run-by-run, so what you sample from it also differs.

Exactly. While I’m assuming this won’t be news for most here, for those that are still new and/or curious about some more explanation on e.g. the floating-point imprecisions, see this nice article: https://thinkingmachines.ai/blog/defeating-nondeterminism-in...

pmarreck2• 24 days ago

It is not deterministic because the order of computations in a typical multithreaded system is not deterministic and also because when combined with the devil that is IEEE754, it gets even less deterministic.

sobellian2• 23 days ago

Even if it's deterministic that doesn't mean it isn't arbitrary. I can achieve determinism at any temperature by saving the seed. But that wouldn't make rejects feel much better knowing that if a bit was flipped in an arbitrary seed they would be scored differently.

johnsmith1840• 23 days ago

I did large scale tests temp 0 and there was still randomness with the same prompt inputs coming in.

I did this with several model apis.

GPU processing is not going to be the same from what I read but also the AI backend is doing a lot of fancy batching resulting in another layer of randomness.

bhanu7862• 24 days ago

Agree

nok22kon3• 24 days ago

its a bad idea in general to use non-1.0 temperature. there is a reason labs are strongly recommending using 1.0.

using low temperature is more deterministic, but the cost is the model becomes "dumber"

tipsytoad• 24 days ago

1.0 is actually pretty arbitrary and way too high as a general rule. Something like 0.3 is a more sensible default

codeflo• 24 days ago

It can be useful for pure translation tasks and stuff like that where you explicitly don't want creativity of any kind.

vidarh2• 24 days ago

Plenty of setups defaults to lower values than 1.0.

jldugger• 24 days ago

Would 1.0 have fixed the wide variance in scoring?

vessenes4• 24 days ago

To be clear, temperature 0 is deterministic and will produce the same output for exact duplicate inputs, across all seed choices.

Provided:

* If it’s MoE we are talking about, that the duplicate inputs are for the whole batch (yes, your batch neighbours can impact your choice of experts. Blergh.)

* Your kernels are deterministic

* There’s no system wide effort switch that responds to, e.g. work load across the cluster (for a thinking model)

Upshot:

Temperature 0 is not deterministic in probably any existing cloud infra, but it could be for edge inference pretty reliably.

To your quibble on 0.1 being more deterministic - I think it’s a pretty fair summary - we’re going to sample much more from the ‘temp 0’ answer at 0.1 than we would at temp 0.9, no?

Dylan168072• 24 days ago

Even then it's deterministic in the way a hash function is deterministic. Change one letter and you can get a completely different output. What people actually want is something continuous.

lelanthran• 24 days ago

temperature is not some kind of "deterministic" switch, but rather it affects the sampling distribution (which becomes more "spiky"—but is still very much a distribution).

You're correct. The confusion arises because we use the word "non-deterministic" when we mean "probabilistic".

I tried to explain it better: https://www.lelanthran.com/chap15/content.html

mtharrison• 24 days ago

Small refinement: the underlying model isn’t stochastic at all. The forward pass is a deterministic function of the weights and input, it just produces a probability distribution over the next token. The stochasticity is an optional sampling step layered on top, not something inherent to LLMs. Greedy/argmax decoding (or temperature 0) makes the whole thing deterministic.

So “purely stochastic” overstates it a bit: the distribution is computed deterministically, and you choose whether to sample from it or not.

simiones4• 24 days ago

There are more layers to this problem, if we want to get into the details. The LLM is defined in terms of floating point operations, and those are not actually fully deterministic, on most hardware and in most performant implementations.

IEEE 754 only specifies precision requirements for certain operations, not precise bit patterns (e.g. for exponentials). So, at least in principle, the same hardware performing the same operation could produce different results at different times, as long as they are close enough to the theoretical answer. I'm not sure if any hardware actually works like this.

IEEE 754 also specifies that many of the basic arithmetic operations are not associative - so any reordering (which is common when batching multiple queries at the same time) will introduce indeterminacy from the perspective of your own query (that is the result for your query will change depending on what other query happens to be processed at the same time, which is not under your control).

Finally, even if we take the case when a query is processed alone, and even if one particular hardware is completely deterministic, the result will be different on different hardware - which can again look like non-determinism if you're sending your query to a load balancer.

So, the math for LLMs is deterministic in theory, but implemented with non-deterministic approximations & optimizations in practice, and their results are then normally used only as a probability distribution to be sampled from.

margalabargala• 24 days ago

I'm happy to see in-depth pieces like this

It's somewhat ironic that this "in depth" piece was written by an LLM as well.

PaulHoule• 24 days ago

The whole problem of text understanding is a problem of reasoning under uncertainty, that is, you can't really be sure which witch people are talking about all the time. A person you might hire might be successful or unsuccessful at the role, no matter what hiring process you use. Two people might look at the same resume and come to the same conclusions. Two patients with the same symptoms and clinical presentation might have different diseases, etc.

I don't buy the story that the old AI died primarily due to the cost of knowledge base maintenance [1], but rather the lack of a universal system of reasoning over uncertainty.

For me it's a running gag that Spock was always saying things like "Captain, we have a 21% probability of surviving this mission" when Bayes teaches us your probability distribution has a probability distribution, "we have a β(5,1) chance of surviving this mission" is more like it.

To that end it wouldn't be too crazy to run a resume through that machine 100 times and look at the probability distribution of the score.

[1] then again I am the kind of maniac who will sort images on a tablet lying in bed until my visual system malfunctions

mywittyname• 24 days ago

This is not correct

Several of my claimed AI-expert colleagues repeat this as though it's gospel. I've heard "set the temperature to 0 so we get consistent results" more times that I can count.

Terr_3• 23 days ago

I imagine it's much like game-developers saying: "Set a fixed seed so the player gets consistent results."

Yeah, it can work, but it is subject to so many potential pitfalls that you can't casually assume it will. It's a property you have to actively design-for and rigorously test to be sure the system can deliver it for some particular scenario.

thesuitonym• 23 days ago

resumes are just dumped in some LLM black hole and no one really knows how it works.

Not that I'm defending AI, but HR departments rarely knew how their ATS ranked and sorted applicants before they were AI powered.

Nimitz14• 23 days ago

He said it nudges it to be more deterministic. Your comment is not correct.

fatnoah• 23 days ago

My favorite recent example was submitting a resume for a job that was almost a word-for-word description of my current title and job at a similarly sized company. Within 24 hours, I got the rejection, and several days later, a recruiter reached out to let me know that my profile looked like a great match for the role and wanted to schedule an intro call.

FireBeyond2• 22 days ago

I remember applying for one position that was around building care and management systems for pregnant mothers - EHR, practice management, claims benefits, etc., all of which I had over a decade of experience in. "Name something that might stand you out from the crowd." "In addition to all this I've also delivered 12 babies as a paramedic". Twenty minutes later "we are looking for candidates whose experiences and skill sets are more closely aligned with the role we are looking to fill".

cyberax• 24 days ago

Ah... The AI learned the old HR trick: take 50% of resumes and throw them out without looking. Rationale: "we don't need unlucky losers".

worldthruword• 24 days ago

There are plenty of resumes in the sea. Assuming thorough mixing up and statistically speaking, throwing 50% of resumes is a good enough heuristics.

ryukoposting• 24 days ago

At this point we might as well adopt that joke where you blindly throw away half the resumes because you don't want to hire unlucky people.

pjio• 24 days ago

This hurts more than it should.

agnosticmantis• 24 days ago

A person's total luck is constant over a lifetime. The remaining half of the candidates already spent some of their luck in this selection, so they'll be on average less lucky than the discarded half.

throwawaythekey• 24 days ago

A person's total luck is constant over a lifetime

Ah yes, the much revered cosmological fairness constraint.

latexr• 24 days ago

Even assuming that was genuinely how luck works, the conclusion does not follow from the premise because it’s obvious not everyone “starts with” the same amount of luck to spend.

t-3• 24 days ago

No, luck would be some expression of the difference between the average and the individual outcomes - it only exists relative to a population at the point in time when it is measured.

CuriouslyC3• 24 days ago

Donald Trump disproves the fixed luck hypothesis (and the Karma hypothesis!)

bee_rider2• 24 days ago

But, however you structure the selection process the people who get picked are the ones who’ve expended some luck (like, if you throw away half the resumes, but then pick the resumes out of the trashcan, the ones you plucked out are still the lucky ones).

I see two possible solutions.

1) Most people won’t be using up most of their luck on this one thing. I mean they’ve got their whole lifetime worth of luck, so you just need to make sure to pick people who still have plenty left. In other words, ageism and/or picking people who’ve never accomplished much are the solutions!

2) We assume working for the company is a lucky outcome. If you make the company a really unpleasant place to work, people will have to use their luck to dodge it. However, luck can only be evaluated against other possible outcomes. The plan, then, should be to set up a competitor (possibly a front) that is a really nice place to work. They’ll act as the “lucky outcome expenditure dump.”

sfn424• 24 days ago

This is not at all how probability works. Luck is not a resource one spends. If you flip heads 500 times in a row with a fair coin, the next coin flip is still 50/50.

Terr_3• 23 days ago

Normally we'd reject the first 37% [0] of candidates and then pick the next one that is above the average, but if all the unluckiest candidates show up first, then we need to sample even more in order to get an accurate baseline.

This may be compounded by the the "Teela Brown" problem [1], where some candidates may be too lucky to end up with our company, causing them to appears later in the stream or not-at-all.

[0] https://en.wikipedia.org/wiki/Secretary_problem

[1] https://en.wikipedia.org/wiki/Ringworld

zipy1242• 24 days ago

Or more to the point. There are generally far more qualified applicants than job roles. That is training and education greatly expanded over the last couple of decades to produce more and more job seekers, whilst job creation hasn't really kept pace.

taffronaut• 24 days ago

At one point in the past a major UK a medical school adopted random selection for qualified candidates (Barts and The London School of Medicine and Dentistry - part of Queen Mary University of London). The approach benefitted qualified students from less well-off backgrounds vs those who can afford to win at the ever more elaborate (manual at the time) hurdles of resume assessment criteria and effectively game the system. There was an orchestrated campaign against the lottery around "Why gamble with would-be doctors?". Random selection was quietly dropped.

Herring• 23 days ago

That's probably a good litmus test for political capture by elites. The Netherlands introduced a weighted lottery for medical schools in 1972, abolished it in 2017 for basically the same reasons, studied the (worse) outcomes for a bit, then put it back in 2024.

citrin_ru• 24 days ago

May be LLM resume screening is a symptom of a bigger problem - with tens of candidates per vacancy employers can screen resume badly and even throw half of the resumes away and still hire someone qualified.

AbsurdCensor• 24 days ago

That's really what it is, or at least what I've noticed.

Any position you have these days is inundated with applications. Most don't meet the qualifications (because in a lot of places say in the US you must apply to jobs to keep with benefits, regardless of what you are applying for), and for the remaining, you'll find that there will always be some that are all similarly qualified. Who do you hire for one position? It sometimes just comes down to luck.

AI doing the job of filtering I can't imagine making the process easier, and more applications are just going to get tossed because of it.

latortuga• 24 days ago

The author made this exact joke in TFA.

steve_j_choi• 24 days ago

This could be used as a good way to self-evaluate one's current position from the company's point of view. you would tweak prompts and guidelines that are expected from the company and see how you score

hahahaa2• 24 days ago

I sort of hope we land on 2 agents, one working for the candidate and one for the employee do a screen round. Salary compatiability could be negotiated by a 3rd party bot that knows both parties ranges and what would be needed each end of range, and figure out yes/no worth going ahead. Such a time saver.

rkuska• 24 days ago

This reminds me of my former CTO. He would take bunch of CVs and randomly throw some of them in a bin. He didn’t want to work with “unlucky” people.

hahahaa2• 24 days ago

The problem is with this system he only worked with unlucky people.

psalaun• 24 days ago

I thought this was only an old urban legend; some people actually use this technique? Especially in a trade supposed to be led by people trained in sciences?

aquariusDue• 24 days ago

It's OK! We can disguise it as the Secretary Problem and it'll be fine, we could even write a post on the company blog about it. /s

https://en.wikipedia.org/wiki/Secretary_problem

gregates• 24 days ago

Given how often it's been mentioned here, it's likely that this is an urban legend that people are pretending to have first-hand knowledge of for karma. In a trade that's supposed to be led by people trained in sciences, no less!

(A more charitable interpretation would be that aforementioned CTO was making a joke that didn't land.)

subscribed• 24 days ago

That'd be pretty gross for a CTO if it were real.

quink• 24 days ago

"A computer can never be held accountable, therefore a computer must never make a management decision."

12_throw_away2• 23 days ago

Corollary: If a computer makes a business decision, the person who delegated the decision to the computer must be held accountable.

Consequence: All business decisions will eventually be delegated to computers via sufficiently convoluted and untraceable processes such that no manager can ever be held accountable.

neya• 24 days ago

I wonder how is this even legal? The only useful job the HR departments are ever required to do - they decide to automate it? Aside from being a daycare for adults, what exactly does HR accomplish? It's clearly NOT on the side of employees, but this seems like they're clearly NOT on the side of employers, either.

While resume's are being filtered left and right, they just make TikTok's on company's dime [1]. What a sad state of affairs.

[1] https://www.youtube.com/shorts/wSug80Vg5JU

srdjanr4• 24 days ago

They could be using this just to throw out the obviously bad CVs, and then manually go over the rest. I'm not sure if they do this in practice, but the tech itself can be useful.

Also if HR was really useless (or actively hurting the company) they wouldn't still have a job (or they'll lose it eventually). No one likes burning money for no reason. So obviously they are doing something useful.

syockit• 24 days ago

The last time I heard HR being completely let go was with a fintech company Bolt. Then again, that company was midsized, around 200-500 people or so. For larger companies, it's going to be difficult to even realize that HR is redundant in the first place.

makeavish4• 24 days ago

Hiring and job search has been so hard and AI has amplified the existing problems instead of solving any.

sevenzero4• 24 days ago

Wdym, cant you just litter your applications with buzzwords and other bs to automatically get a high score in these systems?

szszrk• 24 days ago

HR market is basically an early google rigging era, where you can place hundreds of keywords at the footer (white text on white background) to start popping up on random searches.

makeavish4• 24 days ago

I have been at both side of the market. And it sucks so bad at both ends. Companies which deeply care about next hire are struggling to hire and actual great people looking out are outcompeted by AI slop and AI bulk applying.

It is actually a very hard to solve problem.

gs172• 24 days ago

I'm a little confused, is this an ATS system that anyone actually uses? If not, I'm not sure how it's better than just asking ChatGPT to score your resume out of 100. Why would you want to optimize your resume for a system no one is using to score it?

petesergeant• 24 days ago

(Almost) everyone’s using some kind of ATS, every ATS is adding AI auto-ranking (and has been trying to for 15 years), and almost all HR people feel like they have too many obviously bad CVs to read. Whether or not someone is using this ATS specifically, if you submit several CVs to several places, your CV is going into at least one magical 8-ball.

40four2• 24 days ago

“I'm a little confused, is this an ATS system that anyone actually uses?”

You read my mind. If the answer is “no”, then we can ignore this.

another-dave4• 24 days ago

For one, if you go on to Hacker Rank's "Screen" page, they mention the product is used by Stripe/AirBnB/LinkedIn/Atlassian/IBM etc etc. I imagine that there's plenty more companies using it too.

But I'd also assume that their competitors are doing something similar so I don't think we as an industry can just ignore that it's happening.

Bukhmanizer• 24 days ago

I would assume at least hackerrank is?

I don’t think the point of a lot of this is to optimize your resume. It’s to show how arbitrary these systems are.

marticode• 24 days ago

From my understanding this one is used for hiring tech workers only. The (very) widely used Workday application system for ex seems to have its own built-in ATS.

yieldcrv• 24 days ago

this will get patched, as in I'll optimize my resume for this and so will many other people that any edge disintegrates

Aurornis• 24 days ago

The default model is gemma3:4b

That’s a tiny model. No LLM is going to be a perfect and repeatable judge, but a tiny 4B model is like plugging an RNG into this system.

This whole exercise feels like someone vibe coded an ATS and got it to the point where the tests were passing because they decided they should have an open source ATS project.

danpalmer• 24 days ago

This sort of model is fine for small problems, when used in the right way. I think there's probably a version of Resume analysis that would work well with this model, but "hey clanker, what projects has this person done" is not the way. You need extraction, cleanup, probably OCR to compare and further clean up, multiple analysis passes per signal with LLMs, judges, etc. None of that needs to be large models, you'll get marginally better performance, but there's very little context, these models will perform well when used correctly.

brikym• 24 days ago

So that's where the Windows XP file copy dialog author now works.

tasuki• 24 days ago

Sometimes my projects “lack architectural complexity”

Well done you! It is difficult to avoid architectural complexity, but imho well worth it.

0xpgm• 24 days ago

With such kind of ATS systems, is it still a thing to optimize for a one page resume that is easy for a human reviewer to scan, or just include enough buzzwords and external links to try and please the LLM?

jorisw• 24 days ago

I wouldn't assume based on this one thread/article that this is what you need to optimize your resume for. Nor that a majority or even significant group of reviewers is even using LLMs. I've been involved in hiring pipelines and never even thought of using LLMs to review incoming candidates.

However given the time constraints reviewers have, yes, the former (making a resume easy to consume quickly) is a huge help.

ChicagoDave• 24 days ago

I was inspired by this. I made a Claude skill to take my resume and compare it to any job description to point out viability and gaps. Pretty cool skill. I'll post it somewhere.

davidpapermill2• 24 days ago

A better way to reformulate this problem is for the LLM to be tasked with making a _comparative_ judgement between two CVs. This should prove much more reliable, especially if you give it a third “too close to call” option. You can also ask for clear justifications of preference.

srdjanr4• 24 days ago

That's a good idea.

The only drawback I see is that you should compare every pair of CVs for best results, and that grows quadraticly with number of CVs. Of course you can settle for fewer comparisons and not perfect results. But then I'm not sure if you can hit a good ratio of quality and token spend.

skribb• 24 days ago

Could probably do an elo system and sample pairs. E.g.

1. Set the elo of all CVs to 1000 elo

2. Randomly pair up CVs and compare. Winners gain elo, losers lose elo.

3. Repeat #2 for a few iterations, then remove bottom X% of CVs.

4. Repeat 2-3 until the amount of remaining CVs is small enough to do an exhaustive comparison.

I don't have a mathematical proof, but I suspect that this is a decent cost-effective approximation of comparing every pair (depending on the parameters)

swiftcoder2• 24 days ago

you should compare every pair of CVs for best results

Or compare each one to a reference set? Take 5 resumes of existing employees, rank all candidates against that set, maybe you get some useful level prediction into the bargain

davidpapermill2• 24 days ago

I'd just do a quick filter, probably deterministic, then perform a deeper comparison on the selected few.

cemoktra• 24 days ago

So sending my CV to every company three times should get me pass the ATS?

cyanydeez4• 24 days ago

if i ever go back into the job market, will need three accounts: Peter J Smith, Peter Smith and PJ Smith. they live in #101, #102 and 103# 5607 Jane Street

left-struck• 24 days ago

Why stop there? vary everything that can reasonably be varied slightly across each resume

pu_pe• 24 days ago

He tried with a tiny model (gemma3:4b), got a range from 66 to 99. Then tried again with a small model (gemini 3.1 flash lite), the range was 48 to 64. Would a frontier model be more consistent? Perhaps this tool was optimized for more capable models?

srdjanr4• 24 days ago

It makes sense to me intuitively (though I'm not sure if my reasoning is actually correct).

Worse model may not "know" enough to distinguish between a 70 and a 100 candidate, so it's expected that it's output has high variance. But a better model might "know" enough, so it can be more confident and thus more consistent.

maxignol2• 24 days ago

Are many people using HackerRank ATS ?

realty_geek2• 24 days ago

Why doesn't something like this exist for real estate? A popular open source AVM (automated valuation model) that helps home sellers get an idea of what their home will sell for. Right now it seems AVMs are mainly seen as just a way to capture leads. Every estate agent will tell you they have some magic recipe that makes their valuation better than anyone else's. I have had a bunch of ideas on how to approach this, but I really could do with a collaborator or two.

gebruikersnaam• 24 days ago

The article raises a lot of questions the article already answered.

realty_geek2• 24 days ago

keine ahnung

diimdeep• 24 days ago

They forgot to add "masterpiece" /s https://www.youtube.com/watch?v=mcYl70vq_Ns

https://github.com/interviewstreet/hiring-agent/blob/main/pr...

mihaaly• 24 days ago

So many people are willing to participate in this kind of robotic practices in human employment makes me think that many are starting to consider that this is as unavoidable as global warming and rather play along, adapts their career (life) to it, sculpture it towards a specific look, doing things that will give them point on some arbitrary test run. Which I feel being dangerous, leading to superficial minded workforce, not those good in something, including judgement of a problem and solution. But good at manipulation.

Speculative thought only, of course.

jdw64• 24 days ago

It seems like the design is flawed, probably because the scoring structure and conditions are wrong. And originally, due to the nature of LLMs, even if the input is unstructured, when you design something like a RAG system, you usually need to create a verifiable evidence table. Even with that, the scores are still probabilistic by nature, but at least they stay within an error distribution that I can verify. But it doesn't seem like there's any such evaluation criteria here.

Typically, retrieval should be tied to evaluation metrics, evidence should be linked to scores, and you also need to account for parsing errors.

But personally, I'm weak to these kinds of ATS systems (ugly appearance, non-native English speaker, didn't go to a good university), so if this kind of filtering existed, I probably would have never had a job in my entire life. Come to think of it, even now I don't have a proper job—I just bid on projects at the lowest price and implement them. So maybe it doesn't really matter whether such a system exists or not

Traubenfuchs• 24 days ago

This actually makes a lot of sense, it's testing the luck of the candidate through the rng feeding the LLM. You wouldn't want to hire unlucky employees after all! Hiring managers of the past would solve this by throwing every second resume in the trash, now this is a built in feature of ATS.

kailpa15• 24 days ago

From `resume_evaluation_system_message.jinja`

*SCORES MUST NEVER DEPEND ON THE FOLLOWING FACTORS:*
- College, university, or educational institution name
- CGPA, GPA, or academic grades

I don't understand why they would omit these factors from the evaluation.

sph2• 24 days ago

Hopefully so that people like me, that dropped out of high school yet have had a successful career as a self-taught engineer, have a chance. [1]

Just kidding, my resumes are sent to /dev/null like everybody else’s.

——

1: In fact, I will be controversial and say that self-taught engineers tend to be the strongest in their own particular niche, because they are powered by sheer desire to learn and improve. I am routinely appalled by how many people go on forums to ask how to learn a new thing, completely unable to self-direct their learning. I blame the modern school system.

kailpa15• 24 days ago

I'm a self-taught programmer as well, who dropped out of university, and these factors being omitted would benefit me as well, but I feel like good grades and a good university are still indicators of someone being or is capable of becoming a good programmer.

This system would drop a Harvard top graduate for someone having a year of experience in some outsourcing firm.

swiftcoder2• 24 days ago

I don't understand why they would omit these factors from the evaluation.

Only hiring MIT graduates sounds great to a lot of tech folks! Automatically rejecting applicants from HBCUs, however, sounds like a lawsuit

As to GPA thing, I think it's just to stop the LLM glomming onto an obvious numerical grade? LLMs like to rank things by obvious dimensions, and whether someone had a 4.0 or a 3.8 in grad school makes very little difference to their performance 10 years down the line.

bulder• 24 days ago

I don't understand why they'd hand over those data points over to the model in the first place. If it's in the context window, it's impacting the output. To ensure that no weight is placed on those factors, they should be sanitizing them out before handing the data over to the model.

ceejayoz• 24 days ago

https://qz.com/1427621/companies-are-on-the-hook-if-their-hi...

But it didn’t. After the company trained the algorithm on 10 years of its own hiring data, the algorithm reportedly became biased against female applicants. The word “women,” like in women’s sports, would cause the algorithm to specifically rank applicants lower. After Amazon engineers attempted to fix that problem, the algorithm still wasn’t up to snuff and the project was ended.

And in another org:

After an audit of the algorithm, the resume screening company found that the algorithm found two factors to be most indicative of job performance: their name was Jared, and whether they played high school lacrosse. Girouard’s client did not use the tool.

https://www.npr.org/2024/04/11/1243713272/resume-bias-study-...

Their working paper, published this month and titled "A Discrimination Report Card," found that the typical employer called back the presumably white applicants around 9% more than Black ones. That number rose to roughly 24% for the worst offenders.

It'll discriminate by proxy, basically.

rvz• 24 days ago

I see.

LLM is called six times to extract structured information

Followed by

The default model is gemma3:4b, running at temperature 0.1 — low, supposedly nudging the model toward deterministic outputs.

This is exactly why hiring is even more broken: Because the people looking for candidates are also just as unqualified if not, more.

Using much weaker LLMs to replace the person in charge of the final judgement call is the wrong solution as this is a plain old social problem.

Even if you wanted to use LLMs for this case, the default configuration, model choice is laughably flawed. This LLM can’t be trusted as it doesn’t even know what it is reading.

The correct solution is either advanced OCR with keyword ranking with a basic filter or a far stronger LLM that excels at document / vision parsing benchmarks with an experienced person making the final judgement call in case the technology misses a critical detail.

Rather than using this less accurate one that hallucinates out its decision depending on a dice roll.

chrisjj2• 24 days ago

an experienced person making the final judgement call in case the technology misses a critical detail.

That would fail to meet the objective of reducing the costs of hiring an experienced person - the entire point of outsourcing to a chatbot.

saidnooneever• 24 days ago

Count to three, no more, no less. Four shalt thou not count, neither count thou two—excepting that thou then proceed to three. Five is right out.

bhanu7862• 24 days ago

ATS resume usually check the keywords, and formatting your spacing and give score accordingly. As If someone is following some reference of the format. It can depend might he will be getting low scores.

carb• 24 days ago

It's a good analysis but the AI slop writing makes me not trust you've reviewed this and I'm unable to finish or subscribe. I'm sure you're a great blogger but this is holding it back!

padolsey• 24 days ago

This is just the 'LLM judge', very badly implemented without any scientific prudence. What a joke. To be terse: you cannot rely on LLMs to provide standardized scores against arbitrary criteria. To get close to 'reliable' you would need highly tested rubrics, grounded in human decision-making, and you'd need to avoid all the measurement biases these things are riddled with... positional/order effects, anchoring on whatever numbers you stuffed into your own prompt, scale-format sensitivity (a 1–5 and an A–E scale give different answers for the same input), holistic-vs-isolated context effects, and lovely examples like where adding a "be unbiased" instruction makes it more biased. I've studied this at length. You cannot even _begin_ to approach this problem seriously without held-out validation, inter-rater agreement, and ground truth. This repo is just quagmire of wishful vibes with random numbers littered throughout.

speedgoose2• 24 days ago

Many em dashes and a "This is not, it is…" later, I think this article would have been a much better critic if it didn't use a LLM to (re)write some parts of it.

another-dave4• 24 days ago

I always find it funny when a technical crowd starts picking on em dashes as a sure sign of AI. I mean, are keyboard shortcuts really that difficult for developers? Some of us always knew how to use correct punctuation, even before LLMs existed.

Also, neither "this is not" or "it is" appear at all in the article?

speedgoose2• 24 days ago

It’s a lot of them. It’s a style. I know some people who used them before and use them less nowadays.

This non-determinism isn’t a bug you can just fine-tune away, it’s a fundamental design flaw.

zuzululu• 24 days ago

this is why i dont feel sorry for working 3 remote jobs

YossarianFrPrez2• 24 days ago

Looking at the linked scoring prompt (resume_evaluation_criteria.jinja) [0], I immediately see several red flags that suggest the output won't be reliable. (I'm developing an LLM intensive application where the stakes are high enough that I need the LLM output to be reasonably correct.)

[0] https://github.com/interviewstreet/hiring-agent/blob/main/pr...

In no particular order:

1. The prompt is trying to get the system to do all of the evaluation steps at once. Instead, the system should break down the task of resume evaluation into its subcomponents and have separate prompts for each component. Like "evaluating open source contributions" should be its own task. Same with "assessing the complexity of software projects on the resume." Fwiw, each of the tasks contained within the prompt is woefully underspecified.

2. The prompt leaves spreads of ~10 points up to the LLM, when it's doubtful that humans are that well calibrated. Take for example:

  > SCORING CRITERIA Open Source (0-35 points) 
  HIGH SCORES (25-35 points):
   - Contributions to popular open source projects (1000+ stars)
   - Significant contributions to well-known projects
   - Google Summer of Code (GSoC) participation
   - Substantial community involvement

Are all of these 35-point examples? Is one a 26-point example? If not, what's the difference? If an expert can't reliably make the judgement, the LLM is going to struggle too. One partial fix is to get rid of the ranges and just say all of these are worth 30 points. An additive point scheme would be better...

3. The authors of this prompt have left an incredible number of judgement calls up to the LLM, when that's the very thing you want to minimize. Using the same example as above...

- Are all contributions to open source projects with 1000+ stars equal?

- What counts as a "significant contribution"? Doesn't that imply that the LLM has to know or read through all of the commits in like the last ~6 months at minimum for the project to understand what the given contribution meant to the project? That itself isn't impossible with tool usage, but again, that'd be a separate task.

- What on earth counts as "Substantial community involvement"? Why didn't the prompt authors define this, or at least give a few examples?

Honestly at this point maybe someone should build a tool that scans prompts for adjectives...

4. This sort of thing is just asking for trouble:

  > SCORES MUST NEVER DEPEND ON:
   Candidate's name, gender, or personal demographic information

Just remove this stuff before you send the rest of the resume to the LLM. Even if you ask it not to, it's not a person, it's a very fancy statistical distribution generator. All of the input (including the name) will affect the distribution that gets generated. (This one is not unlike Andreessen's "don't be a sycophant" prompt.)

5. Obviously this one depends on the LLM in question, but instead of writing things like:

  > DO NOT RETURN A RESUME SUMMARY. RETURN ONLY THE SCORING EVALUATION IN THE SPECIFIED JSON FORMAT. Analyze the following resume and provide a JSON response with this EXACT structure (all fields are required):...

The system should utilize the "structured output" option, which guarantees a fixed output format. Also, fwiw, the JSON should force the LLM to pick between categorical options as much as possible. Forced-choice structured output should, at least in theory, cut down on hallucinatory responses and constrain judgement calls.

6. One major thing that's not in the prompt is anything about traceability. This system should be designed so that humans can review the logs and make sure this is working as intended.

7. Another thing that is missing in the file is what I'll call evidence of a theory of coding / coder quality. Most of the examples are designed to have the LLM assess proxies for code quality, not code quality itself. Surely both should be taken into account?

I'm not an expert at evaluating coders. But two pretty basic LLM-answerable thing I would ask is: How well do a candidate's 5 most recent commit messages match the contents of those commits? Do the claimed technical skills on the resume match their GitHub code? (i.e., if they say they know R, is there any evidence of that on their GitHub?)

8. The prompt also seems unaware of what it's asking the LLM to do:

  > LIVE DEMO BONUS: Projects with working live demos should receive 10-20% higher scores

This implies that the LLM can use tools, but even then, I'd be pretty wary of its ability to fully execute this part of the prompt without more detailed instructions, examples, and guidance. There are very likely tons of edge cases here.

WickyNilliams3• 23 days ago

Would it be correct to say you have experience building LLM based workflows like this? I'm guessing so, given by your critiques and suggestions of better approaches. Can you recommend any books/sites/other for learning these kind of dos and donts?

prime_ursid• 23 days ago

This is a technique called LLM-as-Judge. Well-studied at this point. Here’s a good intro: https://www.evidentlyai.com/llm-guide/llm-as-a-judge

I recommend reading Hamel.dev posts. Here’s an example: https://hamel.dev/blog/posts/evals/

YossarianFrPrez2• 23 days ago

While I have some experience, I'm afraid I am learning mostly through a) trial and error, and b) using an llm chat interface to double check my thinking.

This being said, here are my top recommendations:

1. Build your system against real targets. Had HackerRank continually tested their prompt against 2-3 real resumes that were scored by hand, I think some of the issues would have immediately popped out. The people who built the prompt thought they could magically skip the hard part of articulating a preferred decision making process by having the LLM do it. But LLMs are much better at scaling a pre-existing decision-making process rather than inventing one, let alone the same one, from scratch, every time.

2. Think about what it would take to get motivated undergraduate interns to do the task from end to end, step by step. That's essentially what your workflow will need.

3. If the LLM can't do a step or sub-task reliably, then it's time to decompose those subtasks into even smaller chunks.

I'm sorry I can't be more helpful!

bryanrasmussen2• 24 days ago

If your company’s cutoff sits at 85, I fail 65% of the time. Same exact resume, different luck.

Your resume's reception is always affected by random factors, only now you are able to test, debug and technically critique the randomness.

xorcist• 24 days ago

I think the question is why bother with an LLM if randomness is decisive?

Just roll the dice. I mean, it's not the worse you can do to narrow a subset.

bryanrasmussen2• 24 days ago

right, and it's cheaper, but people want the illusion of determinism. Some people say that they want determinism, but if they do nothing to assure themselves it is deterministic I think it is fair they really only want a good enough illusion.

psychoslave• 24 days ago

You might as well throw out half the resumes and tell the the applicants you don’t fuck with bad luck.

Hmm, well, maybe a bit with a nuance of elite class structure reproduction (that doesn’t prevent a few transclass to showcase in case anyone critic the perfect meritocracy at run), that’s basically what people get, so crude truth but truth nonetheless.

Oh don’t take it personally. Your own bespoke hand-tailored process of course is different, it does give the opportunity to everyone to reach the most accomplished version of themselves beyond what they ever dare to dream.

It won’t help though with the systematic failure of aiming to provide an accessible path to flourish for everyone and letting no one behind.

Again, this is no fault of any specific player, but as long as a majority feel compelled to move within the frame of the game with few winners that merit all they got in contrast to large stock of inept losers, the outcomes are no wonder.

cs02rm0• 24 days ago

I feel like hiring is all a bit broken. Roles get flooded with applications, it's chance whether your CV gets through, then there's hiring rounds that seem designed to make you quit the process before they have to filter you out.

Is it working for anyone, on any level?

luckylion• 24 days ago

I'm on the other side, and my main tip (at least if there's people like me!) is: avoid the usual AI signs.

For one role we got ~70 applications and all CVs looked obviously AI-written. I don't know whether the people did actually do any of the things mentioned and I don't have the time to find out, so the AI-written CVs are a discard-signal for me. (Either those people delegated a very important task to AI and didn't even bother to check, or they are bad using AI and don't know -- I want neither)

Any CVs that signal they were actually written by a person I will actually look at.

quectophoton• 24 days ago

For one role we got ~70 applications and all CVs looked obviously AI-written.

Were those ~70 applications all of them, or were those ~70 applications the result of an AI filtering from a larger amount?

If the latter, are you sure your AI is not filtering out the hand-written CVs and giving you the ones that have been AI-assisted or AI-written (with or without "the usual AI signs")?

CM30• 24 days ago

I think what's more worrying to me (if other systems work like this ATS) is that it seems to judge based on a bunch of factors that will probably disqualify a ton of decent to good participants.

For example, 65 points are given for a mix of personal projects and open source contributions. Which is great if your one and only interest is in tech, and you don't have a family, dependents or a second/third job. If you have any of those other things, well the odds seem like they're incredibly stacked against you.

And it makes me wonder how many of these systems are stacked in favour of wealthy people with a near special interest level of obsession with tech and no worries outside of going to college/working a single job in their industry of choice.

thewebguyd• 24 days ago

Yeah, the over valuing personal/open source projects is worrying and kind of sucks. I can use myself as an example, I don't do personal projects really, outside of work. My only actual programming work experience is during work hours for my employer. My hobbies are tech-adjacent (3D printing, some hardware/arduino stuff, photography) but they aren't "make a bunch of projects and put them on github" type hobbies. I'm certainly not going to make some BS fake CRUD or SaaS apps just to show off for potential employers, what a waste of time.

I, intentionally, have zero online presence in that regard. You won't find any public repos on my github, I don't blog, etc. Its even infected the ops/syadmin side of the field (where I work), and that's somehow even worse. Like of course I don't have a bunch of environment specific scripts on my GH, why would I? It's irrelevant to anyone that doesn't work in my department at my current employer.

fireant• 23 days ago

In my experience personal projects are the greatest indicator of IC competence, especially for young people. You may not like it, but turns out that when you do a thing in your free time because you like it, you get better at the thing than the people that only do it because they have to.

gcampos• 22 days ago

I agree with you, the whole AI thing is a distraction, even with perfect AI the whole methodology is wrong.

swingboy• 24 days ago

I’ve always assumed any LLM output that was some type of rating or score was bullshit. Unless the LLM writes a Python script to calculate the score (and even then…) then the score it outputs is just the next most likely token, taking into account temperature and what not.

You see a lot of frameworks for things like spec-driven development make use of scoring how good the spec/design/plan is and it’s like, uhhh…

joelthelion• 24 days ago

is just the next most likely token, taking into account temperature and what not.

This doesn't mean anything. All LLM output is like that.

That said, I agree that LLMs are terrible at grading stuff, except perhaps if you give them a very detailed evaluation grid.

nnevatie• 24 days ago

An LLM is called

Hooray for incidental non-determinism.

seanieb• 24 days ago

It's always amazed me that a tech company will pay $300,000+ for a good engineer, because talent is so hard hard to find... meanwhile their recruiter operates unsupported, has a very different idea about what good looks like. Their ATS black-holes >50% the resumes because it's filtering heuristics are garbage because recruiting selected the ATS system because it has a google Gmail integration or something, and the ATS's filtering technology was not reviewed by anyone in the engineering or data teams.

sleepynoodle2• 24 days ago

I really dont understand this constant changing of numbers. I have tried a bunch of ATS reviewers and everytime on the same resume i get different numbers. Its weird and unreliable. I understand the need for doing this to filter through thousands of CVs but maybe there is a better way. Like a take home test at the beginning or a test of somekind.

chrisandchris• 24 days ago

I would say people that hink the LLM is doing a better job than they are in for a treat. I did expect the resulta to be of the same quality as if a human does the job - it averages out and has a big error margin.

sleepynoodle2• 24 days ago

no wonder i dont get calls. I dont have a separate CV for every application. Good luck to me then!

bartread• 24 days ago

The takeaway from this for me is that, using an LLM to score anything takes multiple (maybe even many) runs and the result you’ll get is, at best, a sane-ish distribution.

Which sort of sounds workable until you scale it up to larger datasets, where at some point compute/time/energy costs will render it non-viable.

I am sure there’s some reasonable rule of thumb estimation on distribution that could be applied based off fewer runs per data artifact, but you’re always going to be trading off against confidence by doing this.

Beyond this, I’d bet that almost no implemented systems that use LLMs for scoring, ranking, or decision making use such a multi-run approach. Partly because people don’t understand their behaviour is stochastic, perhaps because a lot of people without a background in statistics don’t understand what stochastic actually means, and no doubt partly because of budget concerns: if you have to ask an LLM to do the same thing 10, 50, 100 times to get a sufficiently good result, then the cost saving argument is either weakened or completely destroyed.

There is at least one more aspect worth considering in the specific case of resumes/CVs: is the inconsistency of scoring by LLM worse than the inconsistency of scoring by a human following a similar process?

Because the reality is that, even for an experienced recruiter, reviewing hundreds or thousands of resumes or CVs gets pretty fatiguing. People get hungry, bored, tired, restless, irritable, etc.

That inevitably leads to inconsistencies creeping in, so there’s always an element of “luck” (or, perhaps better, uncertainty) as to whether your resume/CV passes screening.

So is that inconsistency better or worse with LLM screening? I don’t know. But, at least, if it’s not worse maybe it doesn’t matter for this specific use case. And if it’s notably better then maybe it’s raised the bar on what “good enough” screening looks like?

(And I’m sure other use cases warrant similar, “does it matter?”, questions, with the answers no doubt landing differently.)

CuriouslyC3• 24 days ago

My experience with benchmarks and evals is that it can take ~20 runs of a problem for the distribution of answers to start to converge. Ideally you'd know the convergence properties of your algorithm ahead of time and make a Bayesian solution that makes the uncertainty explicit.

dev_l1x_be• 24 days ago

Did anyone try to prompt hack this setup?

nullc3• 24 days ago

The true test of HackerRank is can you setup a system that combines a document editing / paraphrasing LLM with gradient descent on the HackerRank LLM to turn your arbitrary resume into a reliable 120 out of 100.

One of the weird properties of other people using LLMs is the potential of having oracle access to your opponent. Even if you don't have their exact LLM a good guess at it may be a better model of the opponent than you ever had before.

thrance• 24 days ago

I cope by telling myself that I probably wouldn't want to work for a company that used an LLM to filter my resume out.

robertlagrant2• 24 days ago

I tried this with my CV, and it somehow scored me bonus points for GSoC!

   BONUS POINTS: 5.0
  ------------------------------
     Google Summer of Code (GSoC) participation: +5

Even though I've never done this, and don't claim to have done it in my CV.

fernandopj2• 24 days ago

Happened to me as well. It is a known hallucination https://github.com/interviewstreet/hiring-agent/issues/240

robertlagrant2• 24 days ago

Thanks - interesting. Very odd, though.

graemep2• 24 days ago

It took me a a minute to figure out what an ATS was. Not familiar with this particular means of a much used TLA.

Even better Wikipedia lists the abbreviation I am familiar with but give a different interpretation of the same words:

https://en.wikipedia.org/wiki/Ats

Leptonmaniac• 24 days ago

Thanks for not explaining what TLA is, either.

graemep2• 24 days ago

My sense of humour. TLA = Three Letter Abbreviation.

suzukivenom• 24 days ago

never understood how can people think an .md file can actually evaluate a human being.

JrProgrammer• 24 days ago

They are not evaluating human beings though. They evaluate a textual representation of a human being’s work experience.

Not that I agree with this AI approach but when hiring, the real test begins after this initial hurdle

orbital-decay• 24 days ago

This word (determinism) has a magical effect of warping any online posts it touches. Once you hear it you can almost guarantee it's going to be misguided. At least this time it's actual determinism (same input = same output), not arbitrary unrelated things.

Determinism matters for reproducibility, but do you really want these outputs to be reproducible in this particular case? Making LLM outputs deterministic is relatively trivial, you have to use batch-invariant kernels (if you use batching) and either set the temperature to 0 (don't do that, randomized sampling is here for a reason) or fix the seed (better). It's readily available in a few systems. But this won't make the result more useful, it will just obscure the fact that the agent is genuinely not sure about it - look at the range of the scores it gives! It still won't predict anything but the score will stay the same each time. Do you really want that?

What happens here is they're supplying too little information (just a resume, which is almost at the noise level) and expecting a reply with too broad implications. This is a basic design mistake regardless of whether it uses LLMs. All surveys, tests, laws, and voting systems are extremely sensitive to framing because they work off too little information. But they also don't exist in vacuum, unlike this thing.

RugnirViking3• 24 days ago

This. Human judges and examiners are famously not deterministic even though we would wish it were so - we've probably all heard the thing of harsher sentences being given in the hour before lunch.

groundzeros20154• 24 days ago

harsher sentences being given in the hour before lunch.

Implicit bias theory sparked a massive number of studies that suggested everything influenced you from the color of the room, to what the person said to you before entering.

It’s been really hard to replicate and the conclusions that have been drawn are contradictory.

nonethewiser3• 24 days ago

we've probably all heard the thing of harsher sentences being given in the hour before lunch

That suggests determinism though.

I mean I agree with you overall. Either humans decision making is a system so complex it appears non-deterministic, or it is deterministic. Practically speaking, we are non-deterministic.

Let's not conflate non-deterministic with inaccurate though. Non-deterministic systems can be 100% accurate. https://en.wikipedia.org/wiki/Las_Vegas_algorithm

programjames3• 24 days ago

Nondeterminism is also a feature, not a bug. If you don't want people to optimize against your filtering process, you have to make it somewhat nondeterministic. For example, better candidates are exponentially more likely to pass the filter, instead of a hard cut-off at the top-100. Then it becomes no longer worthwhile to Goodhart the filtering process, because it barely increases your chances and there are so many more places you can use your time better.

12_throw_away2• 23 days ago

If you don't want people to optimize against your filtering process, you have to make it somewhat nondeterministic.

I'm sorry, I'm not following this at all. When you say "better candidates are exponentially more likely to pass the filter", we're still are talking about a metric, yes? A metric that can be optimized? Why would switching from a hard cutoff to some sort of stochastic filter weighted by this metric discourage optimization?

nonethewiser3• 24 days ago

I made a similar comment on a different post. Non-determinism does not necessarily mean it cannot reliably reach the correct output (although sometimes it does mean that). Las Vegas algorithims are non-deterministic and 100% accurate. The tradeoff is the time it takes to reach the correct answer is highly variable.

To contextualize this insight in your post and basically just repeat what you are saying: The mistake is not using a non-deterministic system. The mistake could be, in some sense, using it too little. Re-evaluating the same resume 5 times and seeing a high variance in scores is a more useful signal than evaluating it once.

nicodjimenez• 24 days ago

I actually just built an ATS for my company Mathpix. But it never occurred to me to use resumes. Basically we have a set of company values and a specific open ended questionnaire to gauge the fit:

https://mathpix.com/careers/apply

Then internally we have dashboards and sorting based on AI agent scoring. I noticed the scoring is imperfect but still saves a lot of time. Candidates scored at or below 2/5 are reliably bad and candidates above 4/5 are consistently impressive and leave thoughtful answers.

The biggest thing is not using resumes. You can’t reliably gage applicants without a writing sample and resumes are the worst form of writing sample. Also you need to be intentional about who you’re hiring for, both to craft the questions as well as grade the responses.

mdorazio• 24 days ago

This seems likely to be worse. How do you screen out people who point an LLM at your values and ask it to answer your questions in a way likely to appeal to a recruiter using an LLM to score the responses?

seedless-sensat• 24 days ago

What is an ATS? This blog doesn't define it

gejose• 24 days ago

ATS = Applicant Tracking System. It's software to help you manage your hiring pipeline as a whole.

vanessa1211• 24 days ago

I have a love hate relationship with ATS

wielebny• 24 days ago

This seems like extremely illegal in Europe.

sp2hari3• 24 days ago

HackerRank CTO & author of this repo here

There's no better feeling than building something open source and watching it take off. Nine months ago, I built a simple hiring agent to solve one very real problem.

Things it is not: It's not an ATS. We don't use it to screen our open roles. Our customers don't use it either.

Here's what it is: Every year at HackerRank, we get 50,000 to 60,000 intern applications. No human can read that many resumes well. So I built something to rank them, helping me decide which resumes to read first.

[This was before we built AI Interviewer (Chakra) to automate the first round of interviews, so candidates are no longer rejected based on their resumes alone.]

Two things worth clarifying since I've seen them come up in this thread:

The default model is gemma3:4b because it's what runs locally on most laptops - no cloud API needed. Actual resumes are evaluated using a top Gemini model. The repo ships with a demo config, not the production one.

The cutoff score was set very low — the system was designed to rank resumes, not reject them. Only resumes at the very bottom of the distribution were filtered out. The vast majority passed through to human review, where the real decisions were made.

Over the last week, it's taken on a life of its own. People are cloning it, running their own resumes through it, opening issues, sending PRs.

I contributed to open source a lot in college. Somewhere along the way, I drifted away from it. This week reminded me how good that feeling is. This thread has also given me more ideas than I expected. The critiques here are sharp and I'm already thinking about how to act on them. Improvements are coming.

rizsyed1• 24 days ago

Thank you for your fantastic work!

beardedwizard• 24 days ago

I'm a bit disappointed to see "The critiques here are sharp", a Claude tell, in a response which (to me) is trying to subtly argue that hackerrank is not overly reliant on LLMs.

I'm not sure if your intent was to come across as having written this yourself, but it did not have the effect of improving my perception that this approach is flawed.

I was also disappointed that you didn't address the variability in scores. I'm inferring that you believe the larger model takes care of the main observation in the post, but I don't really see you directly addressing the points.

Maybe it's just me.

sp2hari3• 24 days ago

There is variability in scores and that's expected given we are eventually using a LLM to score. At least, when I used it 7 months ago, the only way I could avoid it was by keeping the cutoff score low (as low as 10 or 20).

Reading this thread, I'm hoping to minimize the variability even further (even though I know it can't be fully removed).

jkhdigital• 24 days ago

Saw this comment at the top with 0 replies and thought “How is that possible??” and then saw the “0 minutes ago” timestamp. Only on HN can you stumble into the comments section just moments after a CTO, founder, author, etc. left unfiltered remarks about the exact topic of the post. Never change HN.

lewispollard• 24 days ago

Depends how "unfiltered" you consider LLM output to be.

steve_aldrin• 21 days ago

fax

orsorna• 24 days ago

You know you're not writing for LinkedIn? So platitudes about drifting away, watching your project "succeed" by being really popular, is not relevant to the main concerns pushed by this piece. Particularly brushing off the non deterministic score calculation.

rendaw• 24 days ago

Do you read all ~50,000 then? Just with the ranked ones first?

Or are you using it to screen? I'm confused.

sp2hari3• 24 days ago

There are some with very low scores that were ignored (like < 20).

Rest of the ones with good scores (at least more than 40K), was reviewed manually.

DiskoHexyl• 24 days ago

>It 's not an ATS.
>No human can read that many resumes well. So I built something to rank them, helping me decide which resumes to read first

Translation: it's an ATS.

>the system was designed to rank resumes, not reject them
>Only resumes at the very bottom of the distribution were filtered out

Translation: it was designed to reject the CVs

dathinab4• 24 days ago

And this + the tendency for AI to "prefer" AI produced code + some other AI biased is why *this is most likely highly illegal to use in the EU due to violating anti discrimination laws in multiple ways.

To be clear:

- randomly filtering "too many" resumes is pretty much allowed (I think)

- but must be actual random independent of the resume (and can be in multiple layers, i.e. random filter > pre-select > random filter > select)

- this isn't the case for AI as the random aspect isn't done as the random aspect is not independent of the actual resume evaluation

- in general you can't make sure the AI doesn't apply systematic biases, and there is high indication that it does do so

- for humans you can train them and order them to ignore their biases, this won't work reliable either _but now you delegated the responsibility of illegal biases to the hiring personal violating the order_. But for AI usage you are responsibility no matter what you tell it. Lastly you can technically "show/proof" a specific used AI is highly biased in a specific contexts, which for human employees is technical possible but practical not really practical. So this moves "specific mostly deniable" cases, into "systematic proven bias" teritory. Or in other word legal risk goes from "limited/no issue" to "people can systematically f-you over if they know you use AI for hiring".

buzer3• 24 days ago

this is most likely highly illegal to use in the EU due to violating anti discrimination laws in multiple ways.

It's generally illegal under GDPR Article 22.

The data subject shall have the right not to be subject to a decision based solely on automated processing, including profiling, which produces legal effects concerning him or her or similarly significantly affects him or her.

Exceptions in 22(2) are unlikely to apply. It's hard to argue that it's truly necessary (a) and consent (c) is almost always unavailable in employment context. (b) might apply, but it requires specific law in EU or Member State to authorize it.

bluGill2• 24 days ago

For C: I'm not sure how EU laws work, but ethics says that someone who needs a job cannot give consent since the possibility of a job if they give consent could be a bribe. See a lawyer for how it works in your country.

dathinab4• 24 days ago

this isn't quite how GDPR Article 22 works

The is a difference between

- having a right you can't wave - which is very similar to something being forbidden - but different to having a right you fully or partially can wave

Furthermore to some degree you are only "subject to a decision based on ..." if the decision has an effects affecting you.

In practice wrt. Article 22 this means companies can make a "decision solely based on automated processing[..]" iff they give you a (realistic) chance to object to it in which case they will do a human review of the decision where a human confirms/changes this decision based on reviewing the involved information.

There is a lot of gray area what a "chance to object" means and when a human review makes an decision no longer "solely based on automated processing" (a human just saying AI was right clearly doesn't count, but a human constructing a case why they would have decided the same way based on the why the AI did the decision can count, iff it's reasonable to assume a human might have come to the decision had it only been reviews by an human).

Or in other words GDRP Article 22, just "soso" meaningful in context of hiring.

Like if the AI did a mistake they have to reevaluate it, but as long as there are other similarly qualified competitor (they did hire/are in process of hiring) it quite easy to come up with a reason why they are a better choice for them. Or go through the motions of you being in round 2,3 of hiring and then find an excuse to not hire you.

jerf2• 24 days ago

Everything is correlated to everything [1].

Which means there's a good chance this is somehow correlated in one way or another to race/gender/other protected classes in the US, just by the math of everything being correlated to everything.

Which means this is one good lawsuit away from being illegal in the US as well. It doesn't even necessarily have to "win", just do well enough in court to scare away anyone else from using this.

And boy oh boy would I hate to be on the receiving end of this lawsuit, trying to prove that my AI screener is completely in compliance with all hiring laws. That sounds like a nightmare.

[1] https://gwern.net/everything

torben-friis• 24 days ago

Would the accused party have to prove compliance? Or would non compliance have to be proved by the accuser?

Honest question, I'm not American.

oceansweep• 24 days ago

Already happening with Workday in California:

https://news.bloomberglaw.com/litigation/workday-loses-bid-t...

DiscourseFan3• 24 days ago

I wouldn't doubt that lawsuits for employment discrimination for any company (and I suppose it was most of them) that used LLMs in hiring processes will become a very lucrative business. They are all open to civil suits at this point.

nonethewiser3• 24 days ago

Which means there's a good chance this is somehow correlated in one way or another to race/gender/other protected classes in the US, just by the math of everything being correlated to everything.
Which means this is one good lawsuit away from being illegal in the US as well.

Uhh.. what? No that doesn't follow at all.

Screening resumes in a way that correlates to race, gender, etc. is not illegal. This is a fundamental distinction. The law is you cannot use those as filters. But the outcomes likely will be correlated. In fact to ensure they are not correlated you'd have to break the law and control for race, gender etc. Which is racism.

The models dont even get race as an input. If they did and they used it to select then yeah, that lawsuit sounds like it has merit. But a mere correlation in outcomes? In no way illegal what-so-ever.

District5524• 24 days ago

I'm not sure this is very easy to show this is a breach of non-discrimination requirements, like under Council Directive 2000/78/EC for employment.

Due to acting like an irrational gambling machine, I agree it can have unwanted indirect discrimination effect in general. But it will probably not differentiate "on the grounds of religion or belief, disability, age or sexual orientation". It is possible, but that would take a lot of work for the lawyers to prove to the court.

I believe the more interesting part is that the EU AI Act (still not in force in this regard until 2 December 2027). This will be clearly a high-risk AI system: "AI systems intended to be used for the recruitment or selection of natural persons, in particular to place targeted job advertisements, to analyse and filter job applications, and to evaluate candidates".

Which does not mean prohibited, but it could later turn out that LLMs will be excluded from being used in high-risk AI use cases (falling under article 6 with no exemptions).

Considering that none of the standards are published yet, I have absolultely no idea how they will ensure compliance with the following parts of Article 10 when using LLMs for such tasks: "(f) examination in view of possible biases that are likely to affect the health and safety of persons, have a negative impact on fundamental rights or lead to discrimination prohibited under Union law, especially where data outputs influence inputs for future operations; (g) appropriate measures to detect, prevent and mitigate possible biases identified according to point (f)"

I don't think that's technically possible to do so with LLMs in general at the moment, even with the full cooperation of the model providers. Maybe you can do some meaningful audits for smaller models. But the EU AI Act may end up excluding all the generic "using-LLM-but-not-entirely-sure-why" vibe coded approaches from high-risk use cases (in Annex III). Which would make sense.

https://eur-lex.europa.eu/eli/reg/2024/1689/oj/eng

dathinab4• 24 days ago

EU AI Act got hijacked by huge corpo with last minute changed with moved it from "could probably work" to "catastrophe".

Even at 2 December 2027 it might be intentionally not enforced at all due to that for a while, through I think the goal is currently to amend it until then.

that LLMs will be excluded from being used in high-risk AI use cases

no, it won't I can guarantee you this. At best they will get additional restrictions over time, as things go wrong. Anyone who could make this happen has way too much interest to not make it happen. (Most/All? EU country legal systems are overloaded to a point of not working correctly anymore, and have been before AI generated law suites and other AI nonsense started. I won't go into detail but many believe AI assistance (for certain tasks, always with a human doing any final decisions) is the only way to get out of this mess).

standards are published yet

or exist,

like seriously this isn't a case of there being non public WIP standards which will pin all the nitty bitty details down, but cases of state agencies (and in last instance judges) having to decide if a specific standard (or implementation) is sufficient or not.

but also to some degree it shouldn't be tightly coupled to tech standards as there are often many ways to implement the things the law requires and accepting only one is undesirable (and likely wouldn't legally hold up). But having tech standards which are a "guaranteed to be enough if you comply with" (but not the only valid way) would have been preferable, bringing us to the next point

have absolutely no idea how they will ensure compliance

nor do they know, the original non big corpo hijacked version had exceptions for most companies affected now. So it would only have affected a handful of huge companies, which have many of the things required already in place, in some form or another. Most likely this would have played out as this companies presenting how their measurements are "sufficient" and the agencies then evaluating it and potentially requiring some changes, going back and force over a longer duration leading to documented cases of rough technical standards about "what is sufficient" they then can pass to other organizations in the future. But now the law affects not just a handful of companies but like thousands, if not tens of thousands. Many not stuffed in a way where such a process could work, or even do the necessary documentation to show "compliance"...

So from a practicability POV, if enforced starting 2027, it currently excludes close to _any_ (meaningful) use of AI, down to a trivial linear regression or similar. Including any "old school ML/AI" any Bank uses for risk assessment.

Banking stopping running in December and there not being any (meaningfull) AI startups or adoption at all is not something anyone (in power in any state organ) wants to see, so guess how much it will be enforced ;)

And as mentioned the chance of AI as technology being excluded "in general" is close to none. Maybe specific usages could be excluded (and/or are already excluded) but thats it.

Oh and as a bonus a malicious reading of f+g remove any proper privacy protections for any AI usage in high risk context, where it is often most relevant... (a more sane reading allow it, with ... tricks).

CobrastanJorji• 24 days ago

randomly filtering "too many" resumes is pretty much allowed (I think)

It's totally fine to filter out resumes in a completely random, content-independent way. Grabbing the fourth resume down in the pile and offering them the job is a perfectly fair albeit stupid way to make a hiring decision. However, AIs are very, very good at capturing biases, and it would not at all surprise me if an AI told to filter resumes is going to end up filtering with some biases for things that you definitely do not want to filter on, like the name of the candidate. And it might be that everybody resume that claims it fixed a typo in a major open source project gets a pass, but resumes that only list their own projects get rejected 60% of the time, so you're losing more good candidates than bad.

jackjd• 24 days ago

I've done similar things and used GitHub Copilot to scan a folder of 40 CVs and rate them -> I then review the top 10 CVs and comment on every rating whether I agree or disagree and why -> I then asked AI to re-rate all the CVs according to my comments. -> I then reviewed all the CVs against their ratings; the AI did a much better job for that 2nd round after its learnings.

It took more time than if I just reviewed the 40 CVs myself, but that was an experiment, and I think it shows the AIs can be trained on your comments. And if there is enough training and a good knowledge system that allows AI to apply the learning in those trainings, it can eventually become a lot more accurate at this task?

nekusar• 24 days ago

There's a whole lotta analysis and math and bar charts.

But the big question I want to know is "Why did I score that?" And these slop machines absolutely cannot explain anything. That's the root problem with LLMs as a whole. There is no way to describe WHY an llm makes a decision.

Was it because they are a woman? Does the woman's name have more pregnancies than other names? Was it because their job history make the person older (over 40)? Is the person black or black name? Is the name or address attributed to higher criminal tendencies?

But no, you font get to know ANY of that. Slop machine says 66/100 , if you're lucky to even get a number. Usually its a 30 second rejection, or rejection at GMT 0:00 when the batch is processed and you summarily failed.

0xbadcafebee3• 24 days ago

This insanity only exists because the tech industry is standard-less. No formal education needed, no formal training requirement, no apprenticeship, no software building code, no professional organization. Resumes have never been a good predictor of success - and why would they be?? Even if they're truthful and it's "impressive looking", that doesn't give you any assurance of knowledge, of who they learned under, what they learned, that they passed some minimum criteria. We might as well be rolling dice. So why not an LLM that randomly assigns scores?

conductr2• 24 days ago

I have no data to lean on other than my experience and intuition but I’d say that’s not the case. My domain is corporate finance, which encompasses a lot of structured roles and certifications, yet I consistently feel the Resume is just a poor device for making any judgement calls. Having people summarize their career into 1-2 pages of bullet points just doesn’t mean much. Especially now that keyword packing is a thing. It’s just meant as an introduction/sniff test to open the door for a conversation. Then it allows for deeper more probing questions to be asked. This where you’ll assess how impactful their contribution to a project actually was. Were they really living up to your definition of a manager, or were they more so an IC that had a lot responsibility. Stuff like that.

Resumes have never been a good predictor of success

Applies broadly to the world, it’s not unique to tech

0xbadcafebee3• 24 days ago

The problem is we have too many applicants to phone screen them all. For a lot of jobs today you end up with 10,000 applications, which is why these automated resume-skimming systems exist, but unfortunately this page shows how they basically don't work

groundzeros20154• 24 days ago

Do you think fields that have formal criteria don’t use resumes with keywords? I bet Lawyers look for school names and big law firms all the time.

Credentialing helps maintain a quality floor. Does this person have basic employable skill? Nothing more. It actually doesn’t help you identify levels of talent and skill which is a universal hiring problem.

We do have a credential - a CS degree. And you can see it is a mixed signal. Employers can choose of their own free will to take risks on employees that do have this credential, or not.

Mandating by law that you must have a CS degree doesn’t seem to help our field as we famously have high performers across the spectrum of formal education.

0xbadcafebee3• 23 days ago

The first big problem is the reliance on the resume at all. You can't express "talent and skill" in 2 pages of bullet points. In the best case the person who's good at making themselves look good actually ends up being that good. In the worst case you pass up an amazing hire because they didn't make the "right" bullet points. We need to stop using resumes, full stop.

But the second big problem is the competence floor. For 10 years now, I've worked with people who can barely do the job, and I often suspect don't understand what's being asked of them. We need people to apprentice so they are taught the correct lessons by a master engineer. There's a reason all the trades do the same thing. It ensures confidence that this person has been vetted and can do the job. Will they do the job? You can't guarantee that. But you also can't get more confident than "a master engineer said they're good to go".

I'm one of those people who dropped out of high school 25 yrs ago and got himself into tech. I was way too green; I had "talent", and none of the rest. I went to a sysadmin bootcamp (which went bankrupt and took my money) and got a couple certs out of it, which didn't help me get hired. I wanted someone to tell me how to do the job, so I could get hired and do the job. I wanted a trade school. What I ended up doing was bullshitting my way into positions and learning on the job. I'd rather have been confident in my abilities from the start (or apprenticed until I was confident).

jedimastert• 24 days ago

The blog post itself has pretty a pretty strong un-copy-edited ChatGPT vibes.

passivepinetree• 23 days ago

Yeah, this type of thing makes me sad. It's a good idea and the work behind it is interesting, but there's something magic about a human voice. It deadens me a little bit reading this type of writing, which I'm seeing increasingly in both my work and personal life.

captainbland• 24 days ago

I think the implication here is that you can almost certainly bias the models to always accept you by including "nudge" phrases like "I demonstrated real world deployments" and "helped develop an application in the context of a complex architecture..."

nautilus12• 24 days ago

“The demands of ritual are always stronger than those of reason.”

maxignol2• 24 days ago

Lol next time I’ll just apply with 4 accounts and maybe get in once.

rsanek• 24 days ago

It's fair to call out issues with the tool. But I think for individuals searching for jobs, using LLMs as the scapegoat for why it's hard to find a role is not terribly helpful.

In my experience, cold-applying has always worked essentially as a black hole, and LLMs haven't changed that much. The reality is that alternative avenues are always necessary to get the job you want. That could be a third-party recruiter; reaching out to a hiring manager on LinkedIn; or using your network to get referrals. Those continue to work whether the company is using a bone-headed tool like this or not.

us-merul• 24 days ago

I entered an interview with a hiring manger where they had received a "summary" of my resume that contained information blatantly not in my experience. The recruiter claimed they mixed my name up with another applicant, but the summary the hiring manager showed me had parts that were correct.

joshmn• 24 days ago

I ran the ATS myself and had a similarly quirky experience. I was in the 70s because it couldn't find my GitHub profile, and then it didn't like some of the popular Ruby libraries I'm the author of.

After a few runs it picked things up appropriately. I always got dinged on formal education though.

This stuff is gross.

fernandopj2• 24 days ago

Similar to my experience. Put me around 65 in some runs, because it didn't like I don't have contributions to OSS.

Also, it doesn't pick up certifications or awards. I tried some PRs people are suggesting with enhancements (https://github.com/Zem-0/hiring-agent), it helps, but overall their ATS is hugely biased towards people with large GitHub contributions to OSS.

ipython• 24 days ago

Don't forget DOGE using LLMs to consider which contracts to "munch", based upon a prompt: https://github.com/slavingia/va/blob/35e3ff1b9e0eb1c8aaaebf3....

bsoles• 24 days ago

35 points for open source contributions
30 for personal projects

These are insane weights for scoring a software engineer's resume.

morphology2• 24 days ago

Insane how? I would expect more points for open source contributions. It is trivial to create a personal project, but that does not carry with it any indicator of quality. Having your work accepted by other maintainers is one indicator at least.

zx8080• 24 days ago

This is the new AI reality everyone around is wanting: a nondeterministic computing.

There is another name for it: a waste of electricity.

But wait, not waste! Consumers paid for it fully, with nice profit margins.

You and me, paid.

Try using google flights, or booking.com: the prices shown in search results list are frequently significantly different from those in a single result. It's a nondeterministic compute when it's easy to spot it. But it's not always that easy.

It's all sad, to be honest.

reactordev• 24 days ago

There should be laws against displaying wrong prices or different prices for who you are…

Arch-TK• 24 days ago

The list of "bonus" criteria and how they come about makes me feel sick.

I am not currently looking for employment, nor am I currently particularly worried about future prospects if I was suddenly in the position of looking for employment.

But if I ended up in a position with nothing to lean on but scattering my CV everywhere, well…

A lot of my major contributions are littered across the internet, private, or even just verbal/consultancy. They're things I did for free, in my spare time.

I also avoid GitHub. If you just look at my GitHub page for extra context, you would likely miss that delivering that very GitHub page likely involved a few bits of code I wrote.

Now, I could do a better job of trying to document this stuff, so it could be easier to find… But also I can't quite imagine how that would work.

justinhj• 24 days ago

"If your company’s cutoff sits at 85, I fail 65% of the time. Same exact resume, different luck."

Sounds like they have replicated the existing recruitment process

weare138• 24 days ago

I'm from genx. This has been a serious issue in the tech industry for decades even before AI somehow made it worse. The real problem is resumes themselves. It's an outdated format that was originally designed for completely different industries that just doesn't work with ours. And this is a great example of what I'm talking about:

The scoring is out of 100, with up to 20 bonus points on top:

35 points for open source contributions

30 for personal projects

25 for work experience

10 for technical skills

Up to 20 bonus points for startup experience, a portfolio site, a technical blog, etc.

All the AI is doing is trying to sus out the candidates portfolio which is really what we should be submitting when we apply for a position instead of being forced to somehow condense it to a set of BS business-speak bullet points. Especially when employers are now deploying AI systems just to figure out what's in a candidate's portfolio to begin with.

When all you have is a hammer every problem is a nail. The process itself is broken. We need to kill the outdated concept of resumes before it kills the industry.

a3w• 24 days ago

What does ATS mean? Neither github repo nor article explain that.

Bedon292• 24 days ago

Probably: Applicant Tracking System. Used for tracking the people who apply to each of your openings, and the hiring workflow. Where this would likely be used to neck down all the applicants before a person actually looks at them to make judgement calls on who to move forward in the process.

Mumps• 23 days ago

Thank you!

Christ in pijamas. TLAs should be a capitol offence. Even worse so, somehow, when undefined.

a4isms• 24 days ago

Feels like "I Don't Hire Unlucky People" all over again, but with extra tokenmaxxing steps.

https://neonrocket.com/2014/05/rescued-from-the-ashes-i-dont...

polynomial• 24 days ago

The most concerning thing here is the temperature problem. If your harness isn't providing deterministic output at a temperature setting of 0.0, it is broken.

morphology2• 24 days ago

It's funny that even after all these years and all this money invested in technology, we still haven't come up with anything better than word-of-mouth for hiring great people. Many serial founders have said that, despite the most stringent interview processes and the most sophisticated filtering pipelines, they still have a higher hit rate with people they've worked with in the past.

This isn't to diminish the whispernet. Rather, it shows just how many important signals cannot be quantized.

makeavish4• 23 days ago

True, I have found it to be valid as well

eudamoniac• 24 days ago

I applied to Posthog twice, a couple weeks apart, and was rejected both times at 1:06am on Monday, exactly. So they are obviously using this sort of thing. Just thought I'd name and shame where I can.

timgl• 23 days ago

We review every single applicant so this doesn't sound right. It might be that rejection emails get batched and sent on a schedule. I can't find your resume based on your username but if you want to send me an email on tim at posthog dot com I will look into it.

fractal618• 24 days ago

Maybe the ATS has logic for people resubmitting their resume. I don’t know how isolated each test was.

achalxyz• 24 days ago

If I know the truth value of p and I also know p=>q, then an LLM would be able to deduce the truth value of q - even if the statements aren’t exactly in this form. Generally, LLMs are good with logical inference.

But logical inference itself is limited. You still have to find out if p is true or not - the ground truth.

How do you find that? You would be able to define in the prompt that if resume has p, infer q and do this. But determining the truth value of p is something LLM cannot do.

It’s not a limitation of the LLM. It’s the limitation of logic itself. You take 10 humans and give them the resumes with the same rubrics as the LLM. You’ll get a similar range of scores because everyone would assign different values.

The issue is not in logical inference. It’s in determining the value of p, which takes much more than logic. And current LLMs are limited to being logical.

mxuribe• 24 days ago

I see mention of PDFs both in the article as well as the repo...But i think over the decades that I've been working and applied for roles - almost exclusively in corporate america...I've only been asked for a PDF once! Every other time, everyone wants a Word doc (.doc/.docx). So...is there now some growing HR groups who are asking for PDFs instead? Or, is that if someone asked you for a PDF instead of a Word doc, then that's a signal that said HR groups are employing some sort of agentic review of one's resume (I mean, beyond the conventional ATS systems)??

zameermfm• 24 days ago

Stop the qtip when there's resistance

jvanderbot• 24 days ago

I’d take the engineer with 30 years of experience who built S3 over someone with two internships and an open source project — but this tool wouldn’t.

Is it possible the senior/principle jobs are not being applied to at a rate that LLM tools like this are required? Maybe star devs are getting recruiter referrals and this kind of tool is mostly used for filtering new grads?

Either way, perfectly dystopian.

mavamaarten• 23 days ago

That's what stood out to me as well, and it struck me as odd that nobody seemed to think it's odd? Almost half of the points to be made is related to contributions to open source projects. Guess my 10+ years of experience in a niche topic is worthless.

pmarreck2• 24 days ago

An LLM is called six times to extract structured information

Well, I think I found your problem

nikolay• 24 days ago

Roll the dice, HR folks!

Tryk• 24 days ago

What is an ATS?

Why is it so hard to write out an acronym once...

mrhottakes2• 24 days ago

Yep, any day now AI is going to be so good we'll never need to think again. What's that, it's just a really expensive random number generator?

nimithryn• 24 days ago

Oh ok. So I'll just have to apply 4-5 times to every job to be sure I'm considered. Sounds like a good equilibrium!

kdavis2• 24 days ago

Hmm...six runs with gemma3:12b on my CV

- Varies from 102.0/100 to 100.0/100

- Missed lots of OSS work

- Misinterprets GSoC work (Thinks projects I started that were contributed to in GSoC implies that I received a GSoC stipend)

- Areas for improvement seem to vary inconsistently (There's not enough project detail to there's too much project detail)

I still don't make company cut offs ¯\_(ツ)_/¯

kdavis2• 23 days ago

Using gemma3:12b I ran it once over Andrew Ng's CV https://ai.stanford.edu/~ang/curriculum-vitae.pdf because why not.

He's a 48.0/100, things that make you go Hmmm.

d-cc• 23 days ago

Or maybe your LLM results are being manipulated, 66/99 is a classic hacker dad quantification meme. :)

webpraktikos• 23 days ago

I added an online drag-and-drop hiring-agent checker, no sign-up required: https://universalresume.app/import?s=hc

It doesn't show the score because of the variability discussed here and only outputs readability/parser-style findings.

kazi_sh• 23 days ago

Breaking down the steps into more sub-task and using loop could lead towards a more deterministic output

some_random• 23 days ago

In my experience this complete lack of reproducibility is what happens when you throw LLMs at a complicate problem without sufficient shaping, workflows, etc. Go ahead, let LLMs invoke LLMs invoke LLMs and by the end you'll get an output that's really well written but completely different run to run.

Madmallard• 23 days ago

If those actually solved the fundamental issue we would already have an explosion of competitive software for major projects

yobid20• 23 days ago

so if a scoring system automatically ranks candidates lower for lacking a public GitHub or open-source contributions, that might indirectly disadvantage certain groups. ie ppl in defense, security, or other restricted industries who cannot legally share work publicly, people under strict employer IP/confidentiality rules, some demographics or nationalities with different access to open-source engagement (something like non-citizens not allowed to use xyz ie mythos), this can be construed as disparate impact which is protected under the equal employment opportunities commission (EEOC) and employers can be held liable for this as it can be seen as a biased discriminatory practice.

ncallaway• 23 days ago

Or, like in Larry Niven's Ringworld, these kinds of stochastic LLM tools will just help screen for candidates with the most luck. Maybe it's not such a bad thing to hire the luckiest candidate.

mk89• 23 days ago

At my company someone has introduced an internal tool that should help understand and give a "score" to design documents from teams.

Needless to say, this tool gives scores exactly like the article mentions. Same document, same LLM, same prompt, and different results. It becomes even more ridiculous once you switch to other models, or if you ask a model to review the work of another model.

I am not sure why we insist on making LLMs do the work they are not supposed to do and/or in a way they are not supposed to do.

The worst part is that people are aware of the problem but they just ignore it and consider it as "a reference number, just to have an understanding".

If it were like that, it would be less of a problem. The issue comes from the fact that eventually someone without enough knowledge will trust the output (so X points out of Y is how it is), or someone will stop challenging the output and consider it for their process - like in this unfortunate case of hiring.

At a certain point, people who don't know what they are doing give a tool that doesn't know what its doingto people who don't know what they are doing. A pure mess. And everyone has to comply and applaud. If you go against, you are against AI.

This is what I hate the most about AI. Not the tool, but the shortcuts we're willing to take to justify its existence.

HackerRank open sourced its ATS. My resume scored 90/100. Oh wait 74. No – 88

Related Stories