Amazon scraps AI leaderboard to stop workers chasing usage scores

But more seriously, while this is worded as a result of abuse, it had been up for more than a year at this point as a way to push people to use Kiro (you get badges for each "level" basically). Once you reach a point where everyone is using those tools, it makes no sense to keep it around.

Also it was not related to any performance metric, it was a pure vanity thing of getting virtual awards to display.

pylua• 2 months ago

Do you think ai usage will decrease due to this change ?

SpicyLemonZest• 2 months ago

Yes. I've seen the AI usage patterns from people at the top of these kind of leaderboards, they understand that they are not doing things that they expect to produce commensurate value. At best they've inferred (sometimes correctly!) that the existence of the leaderboard means leadership has decided it's OK to burn prodigous amounts of token on any experimental thing one can imagine that might be useful.

belval2• 2 months ago

Do you think ai usage will decrease due to this change ?

Frivolous usage sure, internally at Amazon there is a subculture (if you can call it that) of award chasers. Using Kiro for mundane task to burn tokens does not sound that far-fetched.

Overall usage though no I don't think so, these tools have some pretty wide adoption at this point and not by people chasing awards.

smrtinsert• 2 months ago

How many employees were hatetokenmaxxing I wonder

david_shi• 2 months ago

Incredible how often perverse incentives are implemented, it's like there's no foresight or understanding of behavior at all.

novia2• 2 months ago

I made a high score list for my team to get people to care about catching bugs during code review.

The gamifying started immediately.

If you ever make anything competitive there will be that one guy trying to cheat his way to the top.

The high score list was supposed to be for finding logic errors. This guy was trying to pass off finding typos in text strings as equivalent to finding something that would blow up prod if pushed up.

I really... hate... reality

hnfong• 2 months ago

If you implement a game, people would play the game. (which is better than if they ignored it)

I would have thought that software engineers, of all people, would understand that one's intention and what they actually implemented in a system can be two different things, and careful effort must be expended to properly implement it, and that "good intentions" alone is not sufficient.

Doxin• 2 months ago

There's playing the game as intended and playing to the rules. People who insist on doing the latter even after being asked not to are infuriating. I've got one friend who got himself permanently uninvited from board game night for these sorts of shenanigans.

remarkEon• 2 months ago

This isn't always true, if it makes you feel better.

In order for a game like this (what you described is a type of game) to work, all players need to have the same level of trust vectors toward the other players. This type of situation only emerges in very localized instances. The military, for example.

brador• 2 months ago

1. Don’t make the game unless you can enforce the rules.

Yizahi• 2 months ago

Did you try to analyze afterwards why your subordinates tried to do extra work, in that particular case? Maybe they thought that their performance (aka livelihood) would be tied to that leaderboard? And why would they have such a thought? I have some suspicions about the root cause of the issue :)

novia2• 2 months ago

I was an IC procrastinating on my real work, not a manager. I think I implemented this in Github during my first sprint on the team. Managers are probably wise enough to the way of humans to not attempt something like this (Goodhart's Law and all that). I know as a fact my peers started pointing to the high score list I made as a way to start justifying asking for raises and things like that, to demonstrate that they were pulling their weight on the team harder than others.

arjie• 2 months ago

Seems reasonable. The intention of these things has always been to use a blunt tool to force adoption. Once you have tool penetration you don't need a reward mechanism. The question (which Amazon has no interest in answering for us, I suppose) is whether this tool worked.

Amazon scraps AI leaderboard to stop workers chasing usage scores

Related Stories