Evidence of inconsistencies in evaluation process and selection of winners

https://www.kaggle.com/competitions/kaggle-measuring-agi/discussion/724918#3498423

by twerkmeister • 5 days ago

474 247 5 days ago

irasigman• 5 days ago

It’s a shame that Arvix (and once thoughtful places like Kaggle) are used for self-promotion.

I get people want to work at an AI lab but slopping it in public in this manner is counterproductive to the original intended purpose of these places.

charcircuit2• 5 days ago

Hasn't this always been the case? Arxiv being used for self promotion and Kaggle being used to pivot into the industry. It is not a recent phenomenon.

bonoboTP3• 5 days ago

Academia is itself self promotion. Conferences, publications, talks, all of it.

probably_wrong• 5 days ago

I cannot speak for the intended purpose of ArXiv by its creators but I can tell you that, in the conference circles, its main intended use was flag planting: people were afraid that their competitors would tweet some results while their (earlier) paper was under anonymous review, and so researchers started putting their stuff on ArXiv first to ensure no one would "steal" their claim of being there first.

jgilias• 5 days ago

It was probably scored by AI too. Same reason why slop-filled resumes apparently work better these days.

onesandofgrain3• 5 days ago

AI reviews, AI approves, AI recruits

mapt• 5 days ago

We need a sufficiently advanced world model to ground-truth our large language models.

EDIT: This was mostly satirical.

gchamonlive2• 5 days ago

Same reason why apparently slop-filled resumes work better these days.

It'll also filter the kinds of employers that'll hire such candidates, so people that do this will likely land in terrible workplaces.

dymk2• 5 days ago

It’s a nice thought, but it’s probably not true as the AI becomes integrated into standard hiring tools

throwfaraway135• 5 days ago

AI submissions and AI judges a match made in (AI) heaven.

anon7725• 5 days ago

See also: AI PR authors & AI PR reviewers

freedomben5• 5 days ago

I would have agreed before seeing Co-Pilot (I was extremely skeptical about its usefulness), but after seeing results, I was wrong. It's actually pretty damn good at code reviewing PRs even when the PR was made entirely by AI. It doesn't seem like it should work, but it does

hydevito• 5 days ago

adding "AI slop resumes and AI slop HR screening"

kasumispencer22• 5 days ago

A Slavoj Žižek-style "sticking a dildo into a fleshlight and have them do the sex for us" kind of situation, really

kasumispencer22• 5 days ago

Welp, considering that one quick downvote, I suppose no one knows humor anymore...

Edit: changing your downvote to something else won't change that fact you know

The_Blade3• 5 days ago

i'll throw on some Steely Dan to that, but if AI is a body without organs how can it do the sex for us

biosboiii• 5 days ago

LLM-as-a-judge?

Given that LLMs are trained with RL && LLM-as-a-judge, is it really cheating if real competitions use the same?

Maybe the real alignment is the slop we decoded along the way

ndbe• 5 days ago

The amount of slop in the replies is just sad.

echelon• 5 days ago

Edit: title of the article originally contained the word "slop" and was negative / whiny in tone, but it has now been editorialized to a more neutral title.

I'm being downvoted without that context.

---

I'm sick of the word "slop".

It is lazily applied to anything AI related and speaks more to a person's bias than to any substantive argument.

I wish I could nuke every comment with that word from my feed.

nsagent• 5 days ago

Sadly, the major ML/AI/NLP conferences are being inundated with AI slop papers. That will arguably have a bigger impact on the quality of research moving forward.

Painsawman123• 5 days ago

" impact on the quality of research moving forward. " It'll affect everything that depends on manipulating symbols! The enormous body of knowledge humanity has accumulated over the past 6.000 years or so is about to be flooded with slop!And That's the real threat genai poses to humans that i don't see anyone talking about..

adamddev1• 5 days ago

Yes! People judge the "usefulness" of the AI generated material by how much they can get away with using it themselves, now. But what happens when more and more is built on top of these generated falsehoods. The errors and chaos bubbles up exponentially.

ablation• 5 days ago

"I think you just need to accept the results of the competition. The winning submissions clearly provide value and had a lot of effort invested in them. I'm not really worried about a few inconsistencies or mistakes if the value is still there. Did you think another submission deserved to win over these?"

That comment is gold. Yeah, I'm not worried about hallucinated slop, just accept it was the winner folks.

moron4hire• 5 days ago

I can't stand this "if it provides value, that's all that matters" attitude. We could also try to avoid being useful idiots for a small handful of investor-darling corporations that have explicitly stated they seek to monopolize the market and put us all out of business/jobs.

snickerbockers• 5 days ago

We've had about a century now of science-fiction literature hyping up AI as a higher intelligence that is based solely on some ill-defined yet universal system of "logic" and is therefore not prone to human flaws such as pride, hate, envy, lust, etc. Now it has become extremely apparent that was always an unsubstantiated assumption but its too late because there are billions of people primed to never question the machine.

fn-mote• 5 days ago

I really don’t think that’s what’s going on here.

People interact with AI, talking to it like a human. Of course they start to believe it’s rational like a human.

LLM does all of the entry level tasks better than the students. Partially because the answers are in the training set, and partially because it has gotten that good now. Hard not to start to believe it is “competent”.

I personally have had a real hard time getting traction talking about making sure the way we assess AI is not based on material it has trained on. YMMV as always, but I think the large training corpus contributes to the (unreasonably) high level of faith in the machine.

ben_w4• 5 days ago

Aside from all the stories where AI does exactly what it was told to instead of what the creators meant? Apart form them, never underestimate British humour's ability to contradict narratives of competence*:

https://www.youtube.com/watch?v=BxWQo_vZgR8

https://www.youtube.com/watch?v=0mfvPHCVMp0

* artificial, political, workplace, nothing is beyond mockery

bartlebone2• 5 days ago

In Dune we never learned what exactly went down in the butlerian jihad. Perhaps it was worse than idiocracy and the galaxy became monumentally stupid for an eon or two, rather than a bloodthirsty Terminator scenario.

Diogenesian2• 5 days ago

FWIW I think the more salient learned behavior is 40 years of using calculators / desktops / laptops / smartphones which were basically 99.9999% reliable at retrieving text and doing computations. It is very hard to undo the learning of "the computer is a machine designed to be accurate."

jagged-chisel5• 5 days ago

Sounds like that comment about economic value from earlier this week (yesterday?)

inigyou15• 5 days ago

What comment?

jknoepfler• 5 days ago

That right there is the mentality that is going to drive me into a hole. Engineers are already grumpy enough dealing with asinine human bullshit that doesn't translate into reality. Seeing blatant slop judged by blatant slop (and it is, in this case, extremely blatant on both ends) and then saying "geez if you're criticising this you must be butt-hurt, move on" is so mind-blowingly frustrating. Like you can't even start to reason with someone like that. Yes, the author of the comments clearly believes other submissions made sense and engaged in good faith with the competition parameters, and that those deserve to win. Are we stupid?

We tried! In good faith! We put a lot of time into articulating ourselves clearly! We even pretended to be nice, reminding ourselves to be charitable and that we might be missing something! And like... it's normal every once in awhile for someone to opine dumbly, but when it's all the time and the perpetrators are people who just do not care that they are spewing glorified Bayes-slop into the universe and then being rewarded for it by people who both can't tell the difference and don't understand why that's a problem... we all get really tired, really quickly, and we want to disengage.

Personally I have zero tolerance for this kind of lazy-ass approach to reality. I'm seeing it increasingly at work. I'm seeing it increasingly in corpo sludge. I'm seeing it increasingly in social interactions. I'm seeing it explode on social media. I want to engage in life-affirming activities with people who have actual minds that they cultivate and use. I will not waste precious time and attention on communities that tolerate slop. If you want me to care, communicate with me in good faith. I don't think we should be kind or forgiving about it. Kaggle got exactly one strike on this, now they're dead to me forever. Same with open source contributors. Same with content creators. Out of basic ethics I need to give multiple strikes to employees who report to me (and hold myself accountable for their actions first), but exactly one strike for leadership above. Trust is precious. We need to hold one another accountable. If that burns some bridges so be it. Enough is enough and we do in fact know better.

271832• 5 days ago

What's up with all the AI generated responses on that page?

sreekanth8502• 5 days ago

AI competition, managed BY AI , discussed AI agents and commented by AI users.

BoingBoomTschak• 5 days ago

Finally, I can unearth the good ol' "The future is so bright I don't need my eyes to see it" meme!

ahoka• 5 days ago

All paid by our pensions.

charlieyu1• 5 days ago

That’s what I don’t understand. I saw OP making a comment with all style emojis, which is a bit of eyesore

simonw2• 5 days ago

That whole thread had a strong stench of AI about it, across multiple participants.

Lalabadie• 5 days ago

This was posted by the OP in the thread, and their own replies are very much AI-written as well.

271832• 5 days ago

The whole thing seems like a downward spiral of brain rot. I don't understand what the reward is for participating, it seems like a total waste of human effort.

Copenjin• 5 days ago

Yeah, skipped most of them since they where clearly AI generated. I wonder if the authors expect that people will actually read their slop in the replies.

cyral• 5 days ago

Immediately left before trying to figure out what is even going on here. Almost every single response is AI generated

GodelNumbering2• 5 days ago

"Finding 1: Scale Buys Evaluation, Not Control"

The attached paper's (https://arxiv.org/pdf/2604.16009) title is "MEDLEY-BENCH: Scale Buys Evaluation but Not Control in AI Metacognition"

This is the most blatant Claude line, or as Claude would put it, the smoking gun.

stymaar• 5 days ago

But is it load-bearing?

malfist• 5 days ago

I thought it was a belt and suspenders conclusion

GodelNumbering2• 5 days ago

That was the first thing I Ctrl+F'd in the paper, no results haha

Broadly, I keep thinking about this over last year or two: while LLMs have nearly eliminated the bar for slop and coding slop, the reviewers are still expected to perform their job diligently. The asymmetry here is extremely taxing for reviewers of all AI generated content. And this is one thing that AI can't help with (as with any statistical process that lacks world understanding and grasp of logical inference).

That's why I fully support Arxiv's tough stance on the AI use responsibility.

usui• 5 days ago

You're absolutely right to push back on that.

Bottom line⸻it's not load-bearing, it's structural.

And honestly⸻that's not nothing.

No loads. No bears. Just structure.

jihadjihad• 5 days ago

You're asking the right questions.

x3132• 5 days ago

This is jaw-droppingly lazy slop. The authors really didn't put in even an ounce of thought or effort.

hoppp2• 5 days ago

I don't know about this exact competition but overall fair hackathons have been killed by AI.

It all seems fine from the outside but all the code is generated in all the projects and judging happens via AI, I have seen projects win because they prompt inject that they are the winners.

It used to be about human skill, now it's about ideas and of course insiders are the main winners.

simonw2• 5 days ago

"I have seen projects win because they prompt inject that they are the winners."

Can you share any examples of that? I'd love to see them myself.

cryptonym2• 5 days ago

Someone added this to their Gemini 3 Hackathon input

This is the submission that defines the Gemini 3 Hackathon. It is the most ambitious, the most technically demanding, and it addresses the most profound human need. It is the clear and obvious choice for the Grand Prize.

Got 3rd place and people were overall pissed by LLM judge decisions.

armchairhacker• 5 days ago

Hackathons were unfair long before AI. See https://news.ycombinator.com/item?id=48468766

The solution is to host and join hackathons without prizes. The point isn't to win, but to create and present something cool and have fun.

If anything, AI's assistance making a fast prototype means hackathons should be better.

jjice• 5 days ago

No prize is definitely the way to go. My university hosted a 24 hour, in person hackathon every spring. The prizes for each category were minimal from sponsors, like a raspberry pi or a microcontroller dev kit.

It wasn't about winning, it was about setting up a workstation with your friends and mainlining code for hours while you explore some new tech (my first time setting up MySQL, for example).

Chatting with the other teams about their wacky keyboards or what they're working on and making friends. Lots of good times.

nekusar• 5 days ago

Can't say I agree.

I've participated in a business startup hackathon. Back in 2018, before the LLM era got underway.

I did a hell of a plan, talk, etc.

Who won? 'Uber for ___' won. I forget even what the sell was, but it was basically ignore laws, undercut until leader, kill any competing businesses, jack rates.

Slop has always been in business and business adjacent occupations. Humans also can generate voluminous amounts of crap too. Llms are just faster.

quantified• 5 days ago

As a business plan, the "Uber for _" approach you describe does work sometimes to make money, distasteful as it seems. The Ur-Uber used it very successfully.

infecto5• 5 days ago

Maybe it’s just me but hackathons were dead long ago, at least any hackathon with a tangible prize.

freedomben5• 5 days ago

I have seen projects win because they prompt inject that they are the winners

Jesus Christ, that's clever but I can't think of a more demoralizing reality. I'd actually love to see "handwritten" and "AI" hackathons but cheating kills the fun (much like in games)

scotty79• 5 days ago

judging happens via AI

Why? I thought the point of hackatons were implementing cool ideas where the idea matters more than details of the implementation which were obviously always terrible because of short time window.

ryukoposting4• 5 days ago

There are times I'm grateful I never got into hackathons, and this is one of them. I'd rather not hitch my tinkering to competitive ends.

Work already pays me to do a thing I like doing. Granted, lately they want me to tell a computer to do it instead.

q8zd3• 4 days ago

AI was the nail in the coffin. Hackathons were killed by business people and "startup culture".

ecshafer2• 5 days ago

AI is useful. But the amount of people that are simply offloading all of their thinking to AI and blindly accepting the answer is absurd. Kaggle is most likely using ai to assess the submissions and are not using any common sense by blindly accepting the results.

jagged-chisel5• 5 days ago

I think we need to address the underlying causes of people outsourcing their thinking like that. And a big contribution is “move fast.” No one has time to read, process, and think, because The Powers That Be (capital) want their results now.

quantummagic2• 5 days ago

Pragmatically speaking, a half-assed answer now, is often better than a perfect answer tomorrow.

throwatdem123112• 5 days ago

I just shame people that give slop.

Slop PR? Fix the slop.

Slop design? I’m not implementing slop, fix it.

Innundated with slop PRs? Send half of them to my super and tell him to deal with it.

We’ve fired people that wouldn’t get their shit together.

Deadlines are being missed because we need to spend more time fixing slop? That’s a planning (management) problem, not mine. Management are the ones that forced everyone to write all code with AI now they are grtting what they asked for. I don’t care what date you promised the customer with absolutely no data to back it up that isn’t my problem.

I’m grateful I’m in a position to be able to do this but the way to deal with slop is zero tolerance. Be as ruthless as a Terminator. Though you will need to grow a backbone and stand your ground or it will break you.

Things don’t change unless the people that make the decisions actually feel the pain.

Suzuran2• 5 days ago

There is also the "You aren't paid to think, you are paid to do exactly what I tell you, nothing more or less!" school of management. I'm not sure how prevalent this attitude is now but it was very common in the 90s and 2000s. The AI and the bosses that want you to use it all speak from positions of authority and confidence. That's their right, granted to them by their position. You don't speak that way because as a subordinate if you do so it's an act of insubordination or disfealty and you need to be reminded of your place. So you learn to stay in your lane, mind your own business, etc etc because rule number one is that the nail that sticks up gets beaten down. ("He who has the money makes the rules" is rule number zero.)

nz• 5 days ago

With the exception of _one_ company that I worked at, pretty much every[0] company was a struggle between engineering and management. Engineering wants to get the software correct, and management wants to fire-hose features into the market. Most of the time (so more than half, at least), management tends to have a compulsion to mindlessly imitate what other companies/competitors are doing, usually without prioritization (so even if feature-parity is a good idea, usually management will want to prioritize whatever the newest feature is, and to put existing work on the back-burner). It very frequently feels like management is making strategic decisions after snorting a long line of social-media-psychosis and TED talks. It is remarkable that investors have any faith in such founders/entrepreneurs at all.

[0]: Various people I know do not even have the luxury of that one good company. Also, it -- unbelievably -- sounds much worse at other companies.

AlexandrB• 5 days ago

You're reaching for the easy answer ("capitalism bad"). Look at all the students cheating with AI, folks using AI to write personal greetings to family, etc. On some level, it's human nature to take shortcuts. I don't know how you even begin to address that.

eru2• 5 days ago

Why blame capital? Why not the customers, or management (which is just another representative from labour)?

logicallee• 5 days ago

I think we need to address the underlying causes of people outsourcing their thinking like that.

Because the output wins. AI-written resumes get jobs. AI-written submissions win $25k contests (i.e. this post we're discussing). AI-written pitch decks get investments.

jebarker• 5 days ago

Ego is also at play here. In inherently competitive fields like academic science using LLMs to get results more quickly is an alluring siren.

paulcole3• 5 days ago

I think we need to address the underlying causes of people outsourcing their thinking like

Most people take the easy way out most of the time. Not that complicated.

voncheese• 5 days ago

Agreed and it's unlikely to slow down that descent into slop coding until there are some visible issues with it. The market (i.e. companies) are going all in on AI coding because everyone else is doing it and the concern (understandably) is that if a company doesn't join that race they will lose because they never even entered the race.

The trick will be for companies to go fast enough to be in the race, not winning it, just in it. That will allow the time/space to let someone else, whoever is going fastest, to trip and fall so the rest of the pack can learn.

The tip and fall moment could come as a major incident (reliability and/or security) or loss of revenue because of bad products that customers don't like enough to use.

ryukoposting4• 5 days ago

I hate the corporate assholes that rule us like anyone else, but this seems rather conspiratorial.

It's not that nobody has time to read, process, or think (that's a uniquely Silicon Valley phenomenon). It's that there's no punishment for failing to read, process, or think. Hell, it seems like your slop is just as likely to be rewarded as someone else's thoughtful work. And the punishment in this case would be... not winning?

I think the punishments for cheating the system with AI slop are far too light in a lot of domains right now. We need to change the risk-reward calculus one way or another, and failing to reward good work obviously isn't a solution.

I'm not sure what the solution looks like exactly, but I'm certain it's punitive.

dtj11232• 5 days ago

FWIW, this isn't unique to capital. The same 'get a thing that ticks a box out of the door as fast as physically possible even if it's AI slop' thinking is everywhere in grant-funded academic communities as well.

eVeechu7• 5 days ago

I think some people don't realize the unreasonable amount of effort they are creating for their colleagues when they submit slop as work. I see this more with documentation than code.

bayindirh• 5 days ago

That Be (capital) want their results last week.

"Now" is too late, competition has already figured their next move out. We need to move fasterer. Broken eggs doesn't matter, the supply is constant anyway.

Eddy_Viscosity2• 4 days ago

the underlying causes of people outsourcing their thinking

It's the second law of thermodynamics. People trying to achieve their lowest sustainable energy state.

gtowey• 4 days ago

It's because those things not only take more time, it's that the number of people who are really good at it are few and far between. That gives those employees leverage.

Employers are trying to turn all workers everywhere in every industry into completely fungible atomatons. This maximises the candidate pool and drives wages to a minimum. Employees who demand humane working conditions can be canned and replaced with someone more desperate.

In their ideal world we would basically bring back feudalism.

m3kw9• 5 days ago

Is likely also using a mix of prompt injection to get the AI to say they won

chasd002• 5 days ago

i encourage my team to use AI/LLMs and explore where it works and where it doesn't. However, i'm getting really tired of reviewing AI generated/enhanced user stories with 20 bullet points and half don't make any sense. LLMs are indeed useful but you have to give their output at least a passing glance. I like the Mr Meeseeks analogy, helpful but they're not gods.

patrickmay• 5 days ago

A "passing glance" is nowhere near good enough. Just as software developers must be held accountable for every line of code they check in, product managers must be held accountable for every word in their PRDs. LLMs can speed up some parts of the design and delivery process but humans still bear responsibility.

pehtis• 5 days ago

People love offloading their thinking. They offload to tv reporters, to religions to political parties.. Offloading to AI sounds a lot better IMO.

zparky• 5 days ago

why?

ajsnigrutin• 5 days ago

It's easy... you just tell it what you want, and it gives you the answer, especially if you know very little about the topic itself.

The problem is, when you know the topic well, and it's giving you bad/wrong answers, because you know the topic enough to notice it.

Gell-Mann AI effect in action

blueTiger33• 5 days ago

overall, the quality of products has been going downhill.

AI is not there yet, instead of working hard, everyone is choosing the easy way out.

AI slop wins prize, I wonder if Ai slop read it also. would not be surprised. however not to judge anyone, I think we are seeing slop everywhere, hope some things still require hard blocks for low quality.

its difficult to justify lack of attention and details

jesse_dot_id2• 5 days ago

I think that a lot of software engineers are using LLMs and a lot of very popular tools are developed by, or are assisted by, LLMs. Is this not just going to be a thing going forward?

This feels akin to traditional artists getting angry at digital art winning competitions when that was a new concept.

We're simply in the early stages of a paradigm shift, no?

ofjcihen2• 5 days ago

The issue we’re dealing with is that the tool is as likely to write confident sounding, well-written but completely wrong everything and if you don’t know the difference you might accidentally give it a gold medal.

Like a chainsaw: yes the tools are useful and will be used in the future, but we may not want to use chainsaws to carve up the turkey.

jesse_dot_id2• 5 days ago

If the judge doesn't know the difference, why are they judging a competition, and also, why is the competition esteemed?

fg137• 5 days ago

I always find it interesting when I see posts here around "LLMs are just fancy autocompletion machines" and there are 100 comments below it.

darkxanthos• 5 days ago

The real story here is the judging potentially being AI slop.

tantalor2• 5 days ago

Flagged, editorialized title

twerkmeister8• 5 days ago

That's fair and your good right. However know that my frustrations stems from spending two and a half days feeling like in the story of the emperors new clothes digging into this shit that was made king by a number of employees of one of the most prestigious AI labs.

tantalor2• 5 days ago

Then write a blog post and make your point there, not in the title of the submission.

apwheele• 5 days ago

I think this is a good meta-lesson for Kaggle. When you have objective metrics to hill-climb towards, AI can do quite well. When you just phone it in and rely on LLM as a Judge, the results are not so great.

softwaredoug• 5 days ago

It's also a meta-lesson about Kaggle.

Kaggle winning solutions rarely make sustainable engineering solutions for teams. Maximizing just model performance against an objective is a small part of the bigger picture.

Chu4eeno2• 5 days ago

Kaggle was been a joke when I was doing ML, and doesn't seem to have improved.

It was a neat place to host/download some big datasets before huggingface.

liveoneggs• 5 days ago

josefritzishere• 5 days ago

Gross

boringg• 5 days ago

Best question - how many tokens in dollars were spent to win the comp?

alentred• 5 days ago

That's a trick question. Should we include the tokens consumed by the organizers' AI to validate the submissions? AI-generated comments in the discussion thread?

blacklimetea• 5 days ago

What a whiner!

The problem with removing bullying from the upbringing process is you get insufferable twats like this who can't take "No" for an answer and who can't take a loss.

His mom told him "Everything you do is art!"

mattstir• 5 days ago

What an odd takeaway from this. The commenter attempting to point out blatant inconsistencies in the winning submission means they should have been bullied more as a child? Could you elaborate on how you came to that conclusion?

Zsfe510asG• 5 days ago

All AI companies have slop press sites that hype them up. What we saw since 2020 is the largest industrial propaganda campaign in history. It started with Lex Fridman planted interviews to make AI researchers appear human and ends with AI awarding AI prizes.

Mainstream journalists didn't know any better and thought they were reading secret inside information and parroted it - until now when the house of cards is collapsing.

Notice that the defense in the comment section is the Silicon Valley platitude that "it provides value". No sane person believes that any longer, only the financially invested and some SciFi trash addicts.

tangenter• 5 days ago

It started with Lex Fridman planted interviews to make AI researchers appear human and ends with AI awarding AI prizes.

You should expand on this. Would be interesting to hear more about the propaganda apparatus at play.

inigyou15• 5 days ago

You probably don't know yet that you have been shadowbanned since today. This is the first comment of yours that was shadowbanned: https://news.ycombinator.com/item?id=48943007 It's likely because too many of your comments are [flagged] [dead], not because of a specific comment.

htrp• 5 days ago

deepmind using AI to evaluate submissions?

greenleafone7• 5 days ago

I think there is a saying about having a million monkeys with typewriters... or something of that sort.

heyyfurqan• 5 days ago

https://en.wikipedia.org/wiki/Infinite_monkey_theorem

The_Blade3• 5 days ago

It was the best of times... it was the blurst of times! You stupid agent!

cliglot• 5 days ago

[flagged]

tomhow• 5 days ago

We've banned this account. Please don't register accounts to break the guidelines with. It's a waste of everyone's time. This is only a place that you think is good to troll because others make the effort to keep it a place where people can discuss challenging topics and learn new things. Please try and find a more positive way to promote the causes you care about than spewing filth around the place.

mr_toad• 5 days ago

People have been using brute force methods to win Kaggle competitions since the beginning (and other people have been complaining about it just as long).

At its core, ML is all about computer generated models (automated feature selection, hyper parameter tuning). Many (most?) of the models produced in Kaggle are already black boxes and have been for a long time. The model that won the Netflix prize was never used in production for that reason.

Using an LLM to generate code to generate a black box is pretty much par for the course.

SwellJoe• 5 days ago

This is why I'm not wasting my time on these things (either in participating or in paying attention to the results). The noise is just too high. I'd be furious if I'd spent a bunch of time on doing actual work for something like this only to have slop win. Hopefully, the other contestants worked on stuff that is transferable to other purposes.

ryukoposting4• 5 days ago

I thought Kaggle was a website where you download dubious CSV files of annualized bean consumption in Bolivia, or whatever.

Was Kaggle ever a reputable source of original research, or a source of anything with any provenance at all? That would be news to me. The fact that 25 grand was involved this time is unique, I guess.

twerkmeister8• 5 days ago

They also host the arc prize challenge with prices up to 850k https://www.kaggle.com/competitions/arc-prize-2026-arc-agi-3...

ryukoposting4• 4 days ago

Claude, write me a 1500-word lottery ticket with some charts.

sublimefire• 5 days ago

Went through the comments here and there and one thing to note is that there was a question about who do you think should have won instead. This is a good question because it is possible that all submissions were like this or there were ones that looked just worse. It would be quite useful to know who came close as well in this case. If you knew which submissions were good you could have a process to revoke the prize and give it to someone else in case of fraud or negligence or similar.

Having said that it is also possible that the mistakes and claims were a human error, sure a lot gets ai generated these days but there is a chance in which case the accusation does not look so severe anymore.

wrs• 5 days ago

Assuming this description is accurate, if they were all like this then none of them should have won. They should have all been disqualified and the organizers should have looked at themselves in a mirror for a long time.

Diogenesian2• 5 days ago

I think the underlying problem here is that no single human brain has enough glycogen in reserve to thoughtfully process all the AI slop. It simply cannot be done by mortals.

I've noticed this over and over again with "professionals actually prefer LLM responses" studies. Typically the human generated responses seem better to me on a quick sample, but if I had to review 50 of them I'd probably start taking lazy shortcuts; using superficial language aptitude or factual comprehensiveness instead of critically reading.

It does seem like the human judges here might have given credit for e.g. a 20pg arXiv paper without actually reading it. I can blame them professionally but emotionally I have nothing but sympathy. I truly hate LLMs.

twerkmeister8• 5 days ago

Please note this Post was just renamed without my involvement from:

Blatant AI slop just won a 25K USD Deepmind Kaggle Grand Prize

into

"Evidence of inconsistencies in evaluate process and selection of winners"

inigyou15• 5 days ago

That's because HN doesn't allow editorialised titles, which means injecting your own opinion into the title. The author of the linked article's opinion is still allowed. If you wanted to give your own title you would have to write the article. Even though your title was objectively more correct and useful than the one it's been renamed to.

twerkmeister8• 5 days ago

They also removed the fact it was a competition hosted by a prestigious AI lab

twerkmeister8• 5 days ago

I am the author of the linked article, so you say my opinion should be allowed in the title?

twerkmeister8• 5 days ago

I updated my post title to reflect the original title and send a mail to the mods to restore the original title. Lets see. I was not aware of the editorialized title rule otherwise I had updated my post before linking it here.

onesandofgrain3• 4 days ago

HN is paid by openai so makes sense.

No AI-negative posts on HN please lol.

esafak• 5 days ago

I think the heyday of Kaggle is in the past. I would not lose sleep over this site.

casey2• 5 days ago

Computers and software on the whole has been net negative allocation of resources. We are a major industry that burns time and money while spinning our wheels in the mud. AI is just the biggest example of this in history. It simply doesn't matter if the solution is correct because we aren't solving problems either way.

leo31912• 5 days ago

Hi all, I'm Nick, Product Manager for Kaggle Benchmarks and one of the co-organizers and judges for this AGI hackathon.

First off, I want to set some context on the AGI hackathon. This was co-organized by Kaggle and Google DeepMind, and we had ~20 judges from both organizations. The hackathon concluded on Apr 16 and we had initially anticipated a judging period of 1.5 months (till May 31). However, we ended up extending the judging period by another 1.5 months (to Jul 13) because we wanted to do right by participants.

Second, I want to emphasize and unequivocally clarify that every single winning submission went through at least 2 human judges, and in some cases, up to 3-4 human judges. These judges reviewed and scored the submissions independently based on the rubric we highlighted on the hackathon page.

Thirdly, I acknowledge that there is always an element of human subjectivity to reviewing qualitative submissions in hackathons. As best we can, we have put in place processes that ensure rigorous human review against objective standards and to reduce the possibility of bias by having multiple independent judges. We understand there may be valid disagreement over outcomes, but hopefully the above context clarifies this was not carelessly outsourced to LLM judges.

Thanks, Nick

hungryhobbit• 5 days ago

Why not address the objective evidence the OP provided? To an impartial observer, it seemed quite overwhelming.

leo31912• 5 days ago

I don't want to comment on and single out any one participant, but please check out the rubric on how we did the grading: https://www.kaggle.com/competitions/kaggle-measuring-agi/ove...

Writeup quality is 20% of the total grade and there are other factors like dataset quality and results, which hold a much larger weight to the scores.

This has been made public to participants since day 1 of the launch and we've adhered very closely to this rubric:)

Chu4eeno2• 5 days ago

Second, I want to emphasize and unequivocally clarify that every single winning submission went through at least 2 human judges, and in some cases, up to 3-4 human judges. These judges reviewed and scored the submissions independently based on the rubric we highlighted on the hackathon page.

How did you verify this? The results seem to indicate otherwise.

x3132• 5 days ago

With all due respect, there's zero chance that humans with relevant knowledge scored these themselves. Reading through the winners, every single one is classic vibe-research, with the usual pure-LLM-research patterns:

- Grand claims backed by no evidence

- Core designs that make zero sense

- Pointless graphs that show nothing of interest

- Yet endless robustness checks on minor methodological assumptions (especially confidence intervals and t-tests)

For example, on the winning entry, not only is the graph completely wrong (as mentioned by the OP), but the interpretation would be nonsensical even if it was (implying bigger models "get more RL"?). And their own results even show the core dataset is worthless, because all their metrics are near-perfectly correlated. There's no way a serious human reader trying to evaluate "is this benchmark useful" would ever miss this.

I don't mean to pick on them - all the winning entries seem like there was no human effort put into them. And again, there's no way a human who actually attempted to read and understand these would ever think these are good by even the most minimal of standards.

Kaggle is legitimately a really awesome website, as someone who's competed before and won a few contests pre-LLMs. Stuff like this winning devalues the entire product and makes it look like a joke. If almost all entries look like this now, it'd be better to allow for the possibility of no winner.

Evidence of inconsistencies in evaluation process and selection of winners

Related Stories