A Large-Scale Empirical Study of AI-Generated Code in Real-World Repositories

by softwaredoug • 9 days ago

4 1 9 days ago

fwlr• 9 days ago

Near the end of their methods:

    For each likely AI-generated target, we search human controls from the same repository and same programming language, controlling for project-level practices … We also apply size matching: a control must be within 20% lines of code of the AI target

And then from the start of their results and analysis:

    Comparing the lines of code (LOC) between AI-generated and human-written files, we observed no statistically significant differences

Here I would expect some sort of caveat that this result is a verification that their size-matching was effective in removing differences in lines of code between the target and control. However, there is no such thing, indicating to me that the authors are unaware that what they are detecting are the effects of their own methods. This leads me to strongly discount their stated insight regarding structural differences, and frankly the rest of their paper too.

A Large-Scale Empirical Study of AI-Generated Code in Real-World Repositories

Related Stories