Top AI models fail at >96% of tasks

https://www.zdnet.com/article/ai-failed-test-on-remote-freelance-jobs/

by codexon • 6 months ago

24 10 6 months ago

codexon2• 6 months ago

This paper creates a new benchmark comprised of real remote work tasks sourced from the remote working website Upwork. The best commercial LLMs like Opus, GPT, Gemini, and Grok were tested.

Models released a few days ago, Opus 4.6 and GPT 5.3, haven't been tested yet, but given the performance on other micro-benchmarks, they will probably not be much different on this benchmark.

kolinko• 6 months ago

They didn't test Opus at all, only Sonnet.

One of the tasks was "Build an interactive dashboard for exploring data from the World Happiness Report." -- I can't imagine how Opus4.5 could've failed that.

codexon2• 5 months ago

Check the link to the study. It has been updated for Opus 4.5.

Venn1• 6 months ago

ChatGPT: when you want spellcheck to argue with you.

zb3• 6 months ago

You think they don't? You think AI can replace programmers, today?

Then go ahead and use AI to fix this: https://gitlab.gnome.org/GNOME/mutter/-/issues/4051

stoneforger• 5 months ago

Rewrite it in react it will.

tessitore• 5 months ago

This post really should be edited to say 96% of tasks posted on Upwork. Since we would all expect that to happen.

scotty79• 5 months ago

Kinda sus that least known model did best and none of the more recent models were tested. Capabilities grow very fast. So things that now routinely succeed rarely ever succeeded even half a year ago.

rsynnott• 5 months ago

I mean performance is so bad across the board that this is likely essentially random. Monkeys accidentally doing a bit of Shakespeare.

ben_w• 5 months ago

That's wildly overestimating what monkeys can do on a typewriter.

It takes a lot to just be mediocre. Which, don't get me wrong, I'll agree current ML is, it's just that "mediocre" is an incomprehensibly huge step up from "random".

Top AI models fail at >96% of tasks

Related Stories