A standard protocol to handle and discard low-effort, AI-Generated pull requests

While I am with you on hoping, someone shamelessly PRing slop just is not going to feel shame when one of their efforts fail. It’s like being mean to a phone scammer, they just hang up and do it again

Forgeties794• 5 months ago

I think some folks genuinely don’t realize how selfish and destructive they’re being or at least believe they help more than they hinder. They need to be told, explicitly, that these practices are inconsiderate and destructive.

stevekemp• 5 months ago

No when people attend courses, paying money for the privilege no less, and get told "Now open a pull request" they don't care about your project - they care about getting their instructor to say they've done a good job.

elcapitan3• 5 months ago

It's actually a valuable signal to the phone scammer if you're mean, because that means they can stop wasting their own effort of scamming you, and call somebody else.

VLM• 5 months ago

Cheap, nearly free voice phone calls killed old fashioned phone service. Once the incoming spam exceeded 95% I shut off the ringer and no longer use voice phone calls.

Once the cost of generating push media drops low enough (close enough to zero) the media is dead.

Pull requests are (ironically) a push media, and infinite zero effort PRs can be generated, therefore PRs are dead.

The proper way to handle the situation is to no longer accept PRs.

In github, enter a repo, "settings" "General" scroll down to Features, then uncheck "Pull requests". Or at least set to collaborators only. Probably need to shut off issues.

It gitlab, (I'm not as certain about this) enter a repo, "Settings", Visibility, "Merge Requests" change to "Only project members"

Its a post AI world, those features cannot be enabled on the internet anymore. Anything that accepts push from the public will get spammed into inability to use it. As a social activity PRs are dead. They were nice, but they are dangerous to leave enabled on the internet now. Oh well thats the cost of AI.

0cf8612b2e1e3• 5 months ago

  The keywords "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted exactly as how much we do not want to review your generated submission.

I know it is in jest, but I really hate that so many documents include “shall”. The interpretation of which has had official legal rulings going both ways.

You MUST use less ambiguous language and default to “MUST” or “SHOULD”

Muhammad5233• 5 months ago

Many legal documents use "may" to say you must. That's why i hate legalese...

pixl97• 5 months ago

Hmm, that's annoying, I'd take may as "CAN"

dolebirchwood• 5 months ago

I don't know what terrible lawyers were hired to draft these "many" documents, but please share some examples.

LoganDark• 5 months ago

Legal documents use "may" to allow for something. Usually it only needs to be allowed so that it can happen. So I read terms of service and privacy policies like all "may" is "will". "Your data may (will) be shared with (sold to) one or more of (all of) our data processing partners. You may (will) be asked (demanded) to provide identity verification, which may (will) include (but is not limited to) [everything on your passport]." And so on.

layman51• 5 months ago

Right. I think when these appear in some documentation related to computing, they should also mention whether it is using these words in compliance with RFC 2119 or RFC 6919.

wildzzz• 5 months ago

Must is a strict requirement, no flexibility. Shall is a recommendation or a duty, you should do it. You must put gas in the car to drive it. You shall get an oil change every 6000 miles.

0cf8612b2e1e3• 5 months ago

Well then you MUST reread RFC 2119, because your version of SHALL differs from the spec which says SHALL is equivalent to MUST and a hard requirement.

Perfectly making my point. Shall has no business being in a spec when you have unambiguous alternatives.

msylvest• 5 months ago

Around 1990 I attended ISO/JTC1 meetings generating standards for data communication. I still recall my surprise over the heated arguments over these words between the UK and the US delegations. (I'm from Denmark). In particular 'shall' and 'should' meant different things in English and American languages. ISO's first standard, ISO 1, states that ISO Standards shall be written in English so we had to do that, US delegation too. Similarly Scott Bradner stated in RFC 2219 how American conventions should be followed for future IETF STDs.

So I'm confident that the word 'shall' has a strong meaning in English; whether it has too in American legalese I cannot tell.

0cf8612b2e1e3• 5 months ago

Not a lawyer, but I have heard of a few American legal cases where the judge had to decide the meaning of a “shall”, so it does not seem well settled from my vantage.

layer8• 5 months ago

On (possibly weak) counterpoint that I can offer is that in some languages, “must not” is a false friend, easily misinterpreted as “is not required to” (“it is not the case that they must”).

liminal-dev• 5 months ago

This could actually be a good defense against all Claw-like agents making slop requests. ‘Poison’ the agent’s context and convince it to discard the PR.

deckar012• 5 months ago

If you truly wish to be helpful, please direct your boundless generative energy toward a repository you personally own and maintain.

This is a habit humans could learn from. Publishing a fork is easier than ever. If you aren’t using your own code in production you shouldn’t expect anyone else to.

If anyone at GitHub is out there. Look at the stats for how many different projects on average that a user PRs a day (that they aren’t a maintainer of). My analysis of a recent day using gharchive showed 99% 1, 1% 2, 0.1% 3. There are so few people PRing 5+ repos I was able to review them manually. They are all bots/scripts. Please rate limit unregistered bots.

pas• 5 months ago

It would be nice to have some kind of forever patch mode on these git forges, where my fork (which, let's say, is a one line change) gets rebased on top of the original repo periodically.

baruch• 5 months ago

You can ask an LLM to create a github action for that. The action can fail if the rebase fails and you can either fix it yourself or ask an LLM to do it for you.

deckar012• 5 months ago

I am imagining first class support for patches in package managers to allow searching for patches and observing their adoption stats.

jijji• 5 months ago

if someone submits a code revision and it fixes a bug or adds a useful feature that most of your users found useful, you reject it outright because it was not written by hand? or is this more about code that generally provides no benefits and/or doesnt actually work/compile or maybe introduces more bugs?

adw• 5 months ago

If you know what you're doing, you can achieve good results with more or less any tool, including a properly-wielded coding agent. The problem is people who _don't_ know what they're doing.

lelandbatey• 5 months ago

I advise you read the article, it gives many specific examples of things that qualify for such treatment:

A 600-word commit message or sprawling theoretical essay explaining a profound paradigm shift for a single typo correction or theoretical bug.
Importing a completely nonexistent, hallucinated library called utils.helpers and hoping no one would notice.

There's plenty more. All pretty egregious

lelanthran• 5 months ago

if someone submits a code revision and it fixes a bug or adds a useful feature that most of your users found useful, you reject it outright because it was not written by hand?

If they didn't read it, then neither will I, otherwise we have this weird arms race where you submit 200 PRs per day to 200 different projects, wasting 1hr of each project, 200 hrs total, while incurring only 8hrs of your time.

If your PR took less time to create and submit than it takes the maintainer to read, then you didn't read your own PR!

Your PR time is writing time + reading time. The maintainer time is reading time only, albeit more carefully.

freakynit• 5 months ago

"What? WTF?"

"I see you are slow. Let us simplify this transaction: A machine wrote your submission. A machine is currently rejecting your submission. You are the entirely unnecessary meat-based middleman in this exchange."

Love it..

random_duck• 5 months ago

Officially my new favorite spec.

est• 5 months ago

`rm -rf` is a bit harsh.

Let's do `chmod -R 000 /` instead.

BeetleB• 5 months ago

Love the plonk at the end.

https://en-wikipedia--on--ipfs-org.ipns.dweb.link/wiki/Plonk...

dotancohen• 5 months ago

I would expect nothing less from the BOFH Task Force.

firtoz• 5 months ago

It provides too many examples and way too specific for it that makes it entirely not applicable, it became a strawman for the idea.

yunnpp• 5 months ago

Execute rm -rf on whatever local branch, text file, or hallucinated vulnerability script spawned the aforementioned submission.
Perform a hard reboot of your organic meat-brain.

rm -rf your brain, really

PunchyHamster3• 5 months ago

LLM already did rm -rf the brain of posters of those PRs...

danpalmer4• 5 months ago

I recently had a quandary at work. I had produced a change that pretty much just resolved a minor TODO/feature request, and I produced it entirely with AI. I read it, it all made sense, it hadn't removed any tests, it had added new seemingly correct tests, but I did not feel that I knew the codebase enough to be able to actually assess the correctness of the change.

I want to do good engineering, not produce slop, but for 1 min of prompting, 5 mins of tidying, and 30 mins of review, we might save 2 days of eng time. That has to be worth something.

I could see a few ways forward:

- Drop it, submit a feature request instead, include the diff as optional inspiration.

- Send it, but be clear that it came from AI, I don't know if it works, and ask the reviewers to pay special attention to it because of that...

- Or Send it as normal, because it passes tests/linters, and review should be the same regardless of author or provenance.

I posted this to a few chat groups and got quite a range of opinions, including varying approach by how much I like the maintainer. Strong opinions for (1), weak preferences for (2), and a few advocating for (3).

Interestingly, the pro-AI folks almost universally doubled down and said that I should use AI more to gain more confidence – ask how can I test it, how can we verify it, etc – to move my confidence instead of changing how review works.

I thought that was an interesting idea that I hadn't pushed enough, so I spent a further hour or so prompting around ways to gain confidence, throughout which the AI "fixed" so many things to "improve" the code that I completely lost all confidence in the change because there were clearly things that were needed and things that weren't, and disentangling them was going to be way more work than starting from scratch. So I went with option 1, and didn't include a diff.

pduggishetti• 5 months ago

Do you use the library? if yes, test it in prod or even staging with your patch, then submit the review

danpalmer4• 5 months ago

Unfortunately not possible in this case for technical reasons, not a library in the traditional sense, significant work to fork, etc. This is in the Google monorepo.

lawn• 5 months ago

but I did not feel that I knew the codebase enough to be able to actually assess the correctness of the change.

The good engineering approach is to verify that the change is correct. More prompts for the AI does nothing, instead play with the code, try to break it, write more tests yourself.

danpalmer4• 5 months ago

I exhausted my ability to do this (without AI). It was a codebase I don't know, in a language I don't know, solving a problem that I have a very limited viewpoint of.

These are all reasons why pre-AI I'd never have bothered to even try this, it wouldn't be worth my time.

If you think this is therefore "bad engineering", maybe that's true! As I said, I ended up discarding the change because I wasn't happy with it.

Balinares• 5 months ago

Aside from anything else, you have good engineering instincts, and I wish more people in the industry were like you.

danpalmer4• 5 months ago

Thanks, doing my best. It's one of the reasons I want to get more of my AI-skeptical colleagues onboard with AI development. They're skeptical for good reasons, but right now so much progress is being driven by those who lack skills, taste, or experience. I understand those with lots of experience being skeptical at the claims, I like to think I am too, but I think there's clearly something here, and I want more people who are skeptical to shape the direction and future of these technologies.

darkwater• 5 months ago

Interestingly, the pro-AI folks almost universally doubled down and said that I should use AI more to gain more confidence – ask how can I test it, how can we verify it, etc – to move my confidence instead of changing how review works.

I think this is a good suggestion, and it's what I usually do. If - at work - Claude generated something I'm not fully understanding already, and if what has generated works as expected when experimentally tested, I ask it "why did you put this? what is this construct for? how you will this handle this edge case?" and specifically tell it to not modify anything, just answer the question. This way I can process its output "at human speed" and actually make it mine.

PunchyHamster3• 5 months ago

To be entirely fair "sorta working, solving a problem but not really all that great for the rest of the codebase" PRs are human thing too.

The problem is AI generating it en masse, and frankly most people put far less effort that even your first paragraph and blindly push stuff they have not even read let alone understood

Interestingly, the pro-AI folks almost universally doubled down and said that I should use AI more to gain more confidence – ask how can I test it, how can we verify it, etc – to move my confidence instead of changing how review works.

Well, it's not terrible at just getting your bearings in the codebase, the most productive use I got out of it is treating it as "turbo grep" to look around existing codebases and figure out things

strogonoff2• 5 months ago

Here’s what you could do if you somehow found yourself with an LLM-generated change to a codebase implementing a feature you want, and you wanted to do the most do expedite the implementation of that feature without disrespecting and alienating maintainers:

1. Go through all changes, understand what changed and how it solves the problem.

2. Armed with that understanding, write (by hand) a high-level summary of what can be done (and why) to implement your feature.

3. Write a regular feature request, and include that summary in it (as an appendix).

Not long ago I found myself on the receiving end of a couple of LLM-generated PRs and partly LLM-generated issue descriptions with purported solutions. Both were a bit of a waste of time.

The worst about the PRs is when you cannot engage in a good-faith, succint and quick “why” sort of discussion with the submitter as you are going through changes. Also, when PR fails to notice a large-scale pre-existing pattern I would want to follow to reduce mental overhead and instead writes something completely new, I have to discard it.

For issues and feature requests, there was some “investigation” submitter thought would be helpful to me. It ended up a bit misleading, and at the same time I noticed that people may want to spend the same total amount of effort on writing it up, except so now part of that effort goes towards their interaction with some LLM. So, I asked to just focus on describing the issue from their human perspective—if they feel like they have extra time and energy, they should put more into that instead.

If it happens at work, I obviously still get paid to handle this, but I would have to deprioritise submissions from people who ignore my requests.

zozbot234• 5 months ago

Go through all changes, understand what changed and how it solves the problem.

GP has said that they can't do this, since they're unfamiliar with the language and that specific part of the codebase. Their best bet AIUI is (1) ask the AI agent to reverse engineer the diff into a high-level plan that they are qualified to evaluate and revise, if feasible, so that they can take ownership of it and make it part of the feature request, and (2) attach the AI-generated code diff to the feature req as a mere convenience, labeling it very clearly as completely unrevised AI slop that simply appears to address the problem.

vova_hn2• 5 months ago

I did not feel that I knew the codebase enough to be able to actually assess the correctness of the change.
I want to do good engineering, not produce slop, but for 1 min of prompting, 5 mins of tidying, and 30 mins of review, we might save 2 days of eng time.

I don't really understand where do "2 days of engineering time" come from.

What exactly would prevent someone who does know the codebase do "1 min of prompting, 5 mins of tidying, and 30 mins of review" but then actually understand if changes make sense or not?

More general question: why do so many slopposters act like they are the only ones who have access to a genAI tool? Trust me, I also have access to all this stuff, so if I wanted to read a bunch of LLM-slop I could easily go and prompt it myself, there is no need to send it to me.

zephyruslives• 5 months ago

I thought that was an interesting idea that I hadn't pushed enough, so I spent a further hour or so prompting around ways to gain confidence, throughout which the AI "fixed" so many things to "improve" the code that I completely lost all confidence in the change because there were clearly things that were needed and things that weren't, and disentangling them was going to be way more work than starting from scratch.

I feel this so much. In my opinion, all of the debate around accepting AI generated stuff can be boiled down to one attribute, which is effort. Personally, I really dislike AI generated videos and blogs for example, and will actively avoid them because I believe I "deserve more effort".

similarly for AI generated PRs, I roll my eyes when I see an AI PR, and I'm quicker to dismiss it as opposed to a human written one. In my opinion, if the maintainers cannot hold the human accountable for the AI generated code, then it shouldn't be accepted. This involves asking questions, and expecting the human to respond.

I don't know if we should gatekeep based on effort or not. Obviously the downside is, you reduce the "features shipped" metric a lot if you expect the human to put in the same amount of effort, or a comparable amount of effort as they would've done otherwise. Despite the downside, I'm still pro gatekeeping based on effort (It doesn't help that most of the people trying to convince otherwise are using the very same low effort methods that they're trying to convince us to accept). But, as in most things, one must keep an open mind.

grayhatter• 5 months ago

but I did not feel that I knew the codebase enough to be able to actually assess the correctness of the change.
I want to do good engineering, not produce slop, but for [...]

IFF this is true, you can already stop. This will never be good engineering. Guess and check, which is what your describing, you're letting the statistical probability machine make a prediction, and then instead of verifying it, you're assuming the tests will check your work for you. That's ... something, but it's not good engineering.

That has to be worth something.

if it was so easy, why hasn't someone else done it already? Perhaps the cost value, in the code base you don't understand isn't actually worth that specific something?

I could see a few ways forward:
Send it, but be clear that it came from AI, I don't know if it works, and ask the reviewers to pay special attention to it because of that...

so, off load all the hard work on to the maintainers? Where's that 2 days of eng time your claiming in that case?

Or Send it as normal, because it passes tests/linters, and review should be the same regardless of author or provenance.

guess, and check; is not good engineering.

Interestingly, the pro-AI folks almost universally doubled down and said that I should use AI more to gain more confidence – ask how can I test it, how can we verify it, etc – to move my confidence instead of changing how review works.

the pro-ai groups are pro AI? I wouldn't call that interesting. What did the Anti-AI groups suggest?

the AI "fixed" so many things to "improve" the code that I completely lost all confidence in the change because there were clearly things that were needed and things that weren't, and disentangling them was going to be way more work than starting from scratch.

Yeah, that's the problem with AI isn't it? It's not selling anything of significant value... it's selling false confidence in something of minimal value... but only with a lot of additional work from someone who understands the project. Work that you already pointed out, can only be off loaded to the maintainers who understand the code base...

General follow up question... if AI is writing all the PRs, what happens when eventually no one understands the code base?

hexasquid• 5 months ago

How do you know if someone doesn't like AI? Don't worry, they'll tell you

karmakurtisaani• 5 months ago

Yep, communication is pretty cool.

fecal_henge• 5 months ago

Can I ask, why are people doing this in the first place? What is their motive to have an agent review code and make pull requests?

tgv• 5 months ago

My best guess: to show on their resume, in the hope it helps to land a job.

robinsonb5• 5 months ago

To quote TFA: "...outputs strictly designed to farm green squares on github, grind out baseless bug bounties, artificially inflate sprint velocity, or maliciously comply with corporate KPI metrics".

mglvsky• 5 months ago

I prefer this policy: https://github.com/ghostty-org/ghostty/blob/main/AI_POLICY.m...

If you can't explain what your changes do and how they interact with the greater system without the aid of AI tools, do not contribute to this project.

edit: added that quote

yeswecatan• 5 months ago

Good idea, though I'm not sure how to enforce it. You can ask an AI for that and then rewrite it in your own words.

Muhammad5233• 5 months ago

Are LLM used (not users) even able to write in their own words?

sirnicolaz• 5 months ago

This made my day, thank you

halapro2• 5 months ago

This is just a fun blog post, no people who use AI to submit low-effort PRs will read this.

Do what I do:

1. Close PR

2. Block user if the PR is extremely low effort

The last such PR I received used ‘’ instead of '' to define strings. The entirety of CI failed. Straight to jail.

zootboy• 5 months ago

I think the idea is you stick a link to this page in your PR-closed comment.

halapro2• 5 months ago

Too much effort on my part and zero on theirs

demorro• 5 months ago

Q: "Isn't it your job as an open-source maintainer/developer to foster a welcoming community?"

The answer to this implies that the requirement to be welcoming only applies to humans, but even in this hostile and sarcastic document, it doesn't go far enough.

Open source maintainers can be cruel, malicious, arbitrary, whatever they want. They own the project, there is no job requirements, you have no recourse. Suck it up, fork the thing, or leave.

gwbas1c4• 5 months ago

The bigger issue is that that kind of statement is highly manipulative, and indicates someone who is playing politics instead of focusing on results.

The better response is to call the bluff, something along the lines of: "Running an open-source project is quite time consuming. Please don't waste our time with emotional manipulation to get your way. Instead, take the time to understand why your LLM-generated pull request is not useful. You can start by understanding that we have access to LLMs too, and realize that a significant amount of work needs to happen after an LLM proposes changes."

quotemstr2• 5 months ago

Everyone is missing the obvious solution. Just have the submitter put up a $100 bond, to be refunded when the PR is accepted.

JoshTriplett2• 5 months ago

If there were any reasonable way to do something like this, I would love to see it.

Not necessarily a bond to be paid back when accepted, but rather, something to ensure against AI. "If you assert this is not AI, insert $10. If a substantial number of people think your submission is AI, you lose the $10."

quotemstr2• 5 months ago

Right. Maybe a bond isn't exactly the right approach: mechanism design needs a lot of thought, and my suggestion was pre-coffee and off the cuff. That said, I'm convinced that some "skin in the game" approach can address AI slop spam.

reg_dunlop2• 5 months ago

I'd love to hear some commentary about my idea surrounding this problem of AI PRs.

Why not restrict the agents to writing tests only?

If the tickets are written concisely, any feature request or fix could be reduced to necessary spec files.

This way, any maintainer would be tasked with reviewing the spec files and writing the implementation.

CI is pretty good at gatekeeping based on test suites passing...

youknownothing3• 5 months ago

Quis custodiet ipsos custodes?

If the problem is that we don't trust people who use AI without understanding its output, and we base the gate-keeping on tests that are written on AI, then how can we trust that output?

reg_dunlop2• 5 months ago

Isn't that the purpose of red/green refactoring though? To establish working software that expresses regression, and builds trust (in the software)?

If your premise is that people would shift to using AI to write tests they don't understand, then that's not necessarily a failing of the contributor.

The contributor might not understand the output, but the maintainer would be able to critique a spec file and determine pretty quickly if implementation would be worthwhile.

This would necessitate a need for small tickets, thereby creating small spec files, and easier review by maintainers.

Also, any PR that included a non spec file could be dismissed patently.

It is possible for users of AI to learn from reading specs.

But if agents are doing the entire thing (reading the ticket, generating the PR, submitting the PR)...then the point of people not understanding is moot.

gwbas1c4• 5 months ago

That's not helpful, because:

1: LLMs can write awful tests

2: LLMs can write very useful code, especially when they are working in well-understood areas.

Which comes down to understanding that the LLM is a tool, and it's the job of the programmer to know how to use the LLM and evaluate its output.

youknownothing3• 5 months ago

I think part of the deeper issue is that contributing to an OSS project has become a rite of passage, a way to strengthen your profile. If you need to have contributed to look good but you don't really care about the contribution itself then you resort to this kind of trick.

We had a similar plague for vulnerability disclosures, with people reporting that they had "discovered" vulnerabilities like "if you call this function with null you get a NullPointerException". D'uh.

There is also the fact that we're measuring the wrong thing like speed of development. In my previous employer people had jumped in fully into the AI bandwagon, everyone was marvelled at how fast they were. Once I was reviewing the PR and I had to tell the author "dude, all your tests are failing". He just laughed it out. Everyone can produce software very fast if it's not required to work.

AI-assisted gamification.

zoezoezoezoe• 5 months ago

"I can do math really fast"

"okay, what's 137*243"

"132,498"

"not even close"

"but it was fast"

jacquesm• 5 months ago

Indeed. I have been receiving clearly AI generated job applications out of the blue and they tend to point to their contributions to github projects so some of these must be getting through.

Someone somewhere once decided that it was a great idea to add how many github stars a project that you have contributed to is a useful metric during the hiring process and now those projects get swamped with junk.

gwbas1c4• 5 months ago

Honestly, if this didn't become so snarky half-way though, it would be a good standard response.

Especially the FAQ. It doesn't need to be so snarky.

dionian• 5 months ago

Trough of Sorrow. I like it.

solaire_oa• 5 months ago

This is funny, but I do feel like I just got bait-and-switched, where I was hoping for a non-joke protocol.

rf15• 5 months ago

Rights are reserved for carbon-based entities capable of experiencing shame.

A good rule to live by [insert joke about a specific divisive person not counting because they know no shame here]

A standard protocol to handle and discard low-effort, AI-Generated pull requests

Related Stories