Detecting potential cheaters in Advent of Code Leaderboards

https://adamfallon.com/aoc/advent-of-code/cheating/2022/12/23/detecting-potential-cheaters-in-private-advent-of-code-leaderboard.html

by AJRF • 4 years ago

18 24 4 years ago

adamfallon.com

sokoloff2• 4 years ago

My day 6 part 2 time was 46s, in a language fairly new to me and entirely manually (no automation at all). I just happened to have a solution that could quickly be changed to partition by 14 instead of 4

I’m not sure that just fast Part2 is that strong a signal. (I’m also not even sure that I think GPT is cheating any more than having pre-written frameworks, ready Dykstra, A*, and min-max solvers, automated input downloaders, scripted submission functions, etc. I don’t think any of those are cheating. GPT is more a grey area, I guess.)

Waterluvian• 4 years ago

It sounds like AOC is very formulaic if there’s a set of things to come prepared with.

Is it basically the same kinds of CS exam puzzles each year?

sokoloff2• 4 years ago

It’s going to have graph traversals, path optimization, input parsing, various flavors of map-reduce, and recursing every year.

In that sense, it has a definite pattern, but I wouldn’t say I experience it as formulaic.

seba_dos1• 4 years ago

I haven't been doing it this year, but looking at my answers from last year it looks like I have "cheated" day 24 part 2 and answered in 17 seconds (it was just a matter of adjusting a single range). There were also several "close calls" where it took me just slightly above one minute (day 2, 4 and 13).

tmtvl2• 4 years ago

Oh, it's about people who use AI, I thought they were going after the real cheaters: the people who use a prefab computer to solve the problems rather than building their own from scratch.

nikanj• 4 years ago

And describe tasks to the compiler, instead of assigning registers manually

jacobmischka• 4 years ago

This is an extremely bad faith reduction. It is a series of programming puzzles, not invent-the-universe problems, or paraphrase-problems-to-AI problems.

wetmore• 4 years ago

It's a joke

tmtvl2• 4 years ago

Programming is telling the computer to do something, using the AI to generate code is just the next step up from a compiler.

tomxor• 4 years ago

for a programmer looking up something like “network request python” things like ChatGPT or Github CoPilot become much easier to use, as they have IDE integration (Why didn’t Stack Overflow make IDE plugins for this stuff years ago?

No? isn't that just like Google "i'm feeling lucky" but without even bothering to look up who said what and why.

GPT strips things of context, and this makes it difficult to know how much to trust it, and value it in comparison to alternatives.

I mean if you don't mind quake3 code turning up in your work, and don't mind even knowing that's what it is and where it came from then i guess that level of don't give a crap matches what you are doing - seriously if it's just a hack i have nothing against this, but it doesn't feel right for any long living code.

joenot443• 4 years ago

Maybe someone more knowledgeable can explain - is there a reason people would be concerned about cheaters in AoC? Is it just like for speedrunning, where the repercussions of cheating are largely about your perception in the community itself?

I guess what I’m trying to figure out is whether anyone’s going to try and get a job using their Top 10 Advent of Code medal.

ellisv2• 4 years ago

There’s no real concern because AoC is just for fun. Cheating has always been easy but AoC isn’t a serious competition.

No one is going to get a job because of their AoC score, they’ll get a job because they have the knowledge and skill to complete the problems though.

Mattwmaster58• 4 years ago

I know a guy in the top 100 and a HFT recruiter reached out to him, mentioning his AoC score. Of course, his resume is already impressive as is

lolsowrong2• 4 years ago

Seemingly unpopular opinion: large language models are just a new higher level of abstraction we can use to describe what we want the computer to do.

Can I claim this guy is a fraud for using python instead of typing opcodes into a hex editor? He doesn’t even know what registers hold his data!

bmitc2• 4 years ago

It’s yet another needless and error-prone abstraction that only works if it’s able to read human written code in the first place. I’d say it’s a pretty uninteresting, possibly dangerous, and ultimately distracting abstraction. My experience with ChatGPT was that it couldn’t ever tell me something that was actually correct, and I was asking it straightforward questions.

lolsowrong2• 4 years ago

Large language models aren’t magic, they are tools. Most tools are potentially harmful in the hands of a novice and potentially powerful in the hands of an expert.

ellisv2• 4 years ago

The “fraud detection“ method used here was underwhelming. I was hoping for a statistical learning approach rather than a blunt heuristic. More justification for the heuristic would have been interesting.

1f60c• 4 years ago

Why?

The leaderboard fills up in anywhere from 10 to 15 minutes. I bet it would take people at least as long to come up with a suitable prompt (just copying the text on the site doesn’t work), refine the response, and implement the solution.

cjbprime• 4 years ago

Using this API and a Python script, it is possible to mark any user who solves the second part less than a minute after the first as suspicious

Looks like for Day 25, this would have marked the entire leaderboard (top 100) as suspicious. (No, they didn't use ChatGPT.)

Part of the meta-game while solving part one is to predict what kind of parameterization part two might depend on and make a flexible solution, and the best solvers are also the best at doing that.

Detecting potential cheaters in Advent of Code Leaderboards

Related Stories