PR-Agent — extension that adds AI chat to code reviews on GitHub

https://chromewebstore.google.com/detail/pr-agent-ai-powered-code/ephlnjeghhogofkifjloamocljapahnl

by marsh_mellow • 2 years ago

24 15 2 years ago

chromewebstore.google.com

lvncelot• 2 years ago

I'm sorry if this makes me sound like a cranky old man but I find the idea of AI-generated PR description and reviews entirely misses the mark of what they're for.

The PR description should contain the context that made a change necessary, which is something that can not be contained in the change itself and thus can't reliably be inferred from it.

And if I request a PR review, I trust you, as my peer, to understand the problem my change is trying to solve and to judge whether it succeeds in doing so. If I wanted AI suggestions for it I can skip the middleman.

I guess the "chat with the codebase" feature could be useful, but the auto-review feature just gives me very uncanny vibes. If someone would "review" my code by blasting it through an LLM I'd be ticked off.

nine_zeros• 2 years ago

The PR description should contain the context that made a change necessary, which is something that can not be contained in the change itself and thus can't reliably be inferred from it.
And if I request a PR review, I trust you, as my peer, to understand the problem my change is trying to solve and to judge whether it succeeds in doing so. If I wanted AI suggestions for it I can skip the middleman.

Are you talking about keeping meaningful processes around? Because companies are trying to get rid of humans and people altogether. They don't really know why PR reviews exist - only that reviews can take time - and they don't want to spend time. But can't remove reviews anyway - so just paper over it.

joshgachnang• 2 years ago

We use this feature and it's actually been pretty handy. I still require my devs to write their own PR description (because of course they should), but it does give a few suggestions for things to look into as they review. Occasionally it's pointed out a few things I might have missed. But at least half of the suggestions are obviously off base, so I can't imagine trusting it to actually do the review.

boredtofears• 2 years ago

I find AI summaries to be useful when writing PR's as a starting point. Having it generate a bulleted list summarizing the changes is a good way to make sure you've covered your entire PR (and it's easy to trim down unnecessary things). You can add context as needed.

I'd never trust an unrevised version though.

johnnyanmac• 2 years ago

Yeah, thars the big issue. Really only takes one unengaged or lazy reviewer to not take the time to revise that AI output and it will ruin the entire Org. Honestly it's a theme for all LLMs: they are great as a guide but people inevitably use them as if they are a replacement. The companies marking it as such don't help either.

As is, getting in revised LLMs responses seems like it'll be Gen Alpha's "let me Google that for you" swipe. Except at least Googling means you still need to put in leg work.

mistrial9• 2 years ago

your statement is clear without the first half sentence .. another way to characterize this is discussing the role of human people as team members versus robot support services.. whose role? whose benefit? from the human side.. what accuracy ? what costs (both hidden and overt) ? from the toolset side

Groxx• 2 years ago

Having seen / used / etc similar things: yes I fully agree.

For people who already know the code well, it can catch things you didn't think about, and that's kinda nice. I suspect it has cost me/my team more time on wild goose chases than it has saved on real findings, but it might still be worth it.

For others, I've seen it constantly mislead people, and convince them to make changes that look reasonable but lead to the PR being incorrect. Over and over and over. And the "look reasonable" part is particularly dangerous during code reviews - reviewers are less careful about things that look reasonable, that's a relatively normal and unavoidable thing for humans even though it's incorrect.

When it's exposed to everyone, there's a pressure to follow and trust it, like existing linters. Private use is less bad, but still risky due to the plausible-change-review-negligence issue.

---

Also PR descriptions in particular are borderline useless when written from an LLM, because they just summarize the changes (or mislead summary-readers about the change, when incorrect). So you're just littering your commit history with the equivalent of auto-generated blogspam. No thanks.

talrid• 2 years ago

Hi @lvncelot

I am Tal, one of the developers of this extension.

See our FAQ regarding your concerns about the PR description, and the review process: https://pr-agent-docs.codium.ai/faq/#answer1

In general, PR-Agent is designed to assist, not replace, human reviewers.

Many of the flows and processes we deployed are aimed at meeting this goal. We try not to overclaim, or overpromise. PR-Agent should not be the sole reviewer of a PR. We believe it can definitely enhance and improve the reviewing process.

teqsun• 2 years ago

Oh, Codium?

My last company had codium on our internal codebases for PR review/summary. I found it neat, but I never saw it make any suggestions that surpassed linting. But as a lazy dev who didn't want to rewrite all the details I put in my commit messages when making a PR, I felt it did a pretty good job summarizing.

Funny story: One time I jokingly asked it to rate the skill level of the programmer who wrote a certain PR (which was created by me). It was very complimentary of my skills, which I shared in the humor chat about how it was a very smart AI. The very next day I was laid off (unrelated, it was along with most of my team).

marsh_mellow• 2 years ago

There's an open source version of this as well: https://github.com/Codium-ai/pr-agent

lucabetelci• 2 years ago

Most of the reviews for this extension look like they were generated with LLM.

aravindputrevu• 2 years ago

Looks interesting at the same moment, I don't know why it should be a chrome extension and not a hook into my workflow like Coderabbit dot ai.

What I love with these tools, I'm getting productive. Although, I'd still like to do the review myself.

gengstrand• 2 years ago

Has anyone here tried out https://github.com/freeedcom/ai-codereviewer which I suspect is something similar only in GHA form?

mass_and_energy• 2 years ago

Wow there really is nothing that people won't shoehorn AI into these days eh? What is the point of this? What is the net business benefit, other than "but, but it's got AI"?

nextworddev• 2 years ago

This looks like a security nightmare

PR-Agent — extension that adds AI chat to code reviews on GitHub

Related Stories