dakshgupta

Apart from the job-related stuff others have already said, there is a bit of novelty/bragging rights in landing a PR into a major open source project.

Not yet - but we want to do this. Similarly true for the ephemeral unit tests that greptile writes.

The signal-to-noise ratio problem is unexpectedly difficult.

We wrote about our approach to it some time ago here - https://www.greptile.com/blog/make-llms-shut-up

Much has changed on our approach since then, so we'll probably write a a new blog post.

The tl;dr of what makes it hard is - different people have different ideas of what a nitpick is - it's not a spectrum, the differences are qualitative - LLMs are reluctant to risk downplaying the severity of an issue and therefore are unable to usefully filter out nits. - theory: they are paid by the token and so they say more stuff

Thanks! We go over that on many other pages. Here are some:

https://www.greptile.com/benchmarks https://www.greptile.com/greptile-vs-coderabbit https://www.greptile.com/greptile-vs-bugbot

I agree that none perform _super_ well.

I would argue they go far beyond linters now, which was perhaps not true even nine months ago.

To the degree you consider this to be evidence, in the last 7 days, the authors of a PR has replied to a Greptile comment with "great catch", "good catch", etc. 9,078 times.

2. There is plenty of evidence for this elsewhere on the site, and we do encourage people to try it because like with a lot of AI tools, YMMV.

You're totally right that PR reviews go a lot farther than catching issues and enforcing standard. Knowledge sharing is a very important part of it. However, there are processes you can create to enable better knowledge sharing and let AI handle the issue-catching (maybe not fully yet, but in time). Blocking code from merging because knowledge isn't shared yet seems unnecessary.

Independence

It is, but when a model/harness/tools/system prompts are the same/similar in the generator and reviewer fail in similar ways. Question: Would you trust a Cursor review of Claude-written code more, less, or the same as a Cursor review of Cursor-written code?

Autonomy

Plenty of tools have invested heavily in AI-assisted review - creating great UIs to help human reviewers understand and check diffs. Our view is that code validation will be completely autonomous in the medium term, and so our system is designed to make all human intervention optional. This is possibly a unpopular opinion, and we respect the camp that might say people will always review AI-generated code. It's just not the future we want for this profession, nor the one we predict.

Loops

You can invest in UX and tooling that makes this easier or harder. Our first step towards making this easier is a native Claude Code plugin in the `/plugins` command that let's Claude code do a plan, write, commit, get review comments, plan, write loop.

We have ways to approximate our impact on code quality, because we track:

- Change in number of revisions made between open and merge before vs. after greptile

- Percentage of greptile's PR comments that cause the developer to change the flagged lines

Assuming the author is will only change their PR for the better, this tells us if we're impacting quality.

We haven't yet found a way to measure absolute quality, beyond that.

Apologies, that is poor wording on our part. It's internal data from engineers that use Greptile, which are tens of thousands of people from a variety of industries. As opposed to external, public data, which is where some of the charts are from.

Most of our customers are enterprises, so I feel relatively comfortable assuming they have some decent testing and QA in place. Perhaps I am too optimistic?

Thanks! The first 4 charts as well as Chart 2.3 are all from our data!

This is a good one, wish we had included it. I'd run some analysis on this a while ago and it was pretty interesting.

An interesting subtrend is that Devin and other full async agents write the highest proportion of code at the largest companies. Ticket-to-PR hasn't worked nearly as well for startups as it has for the F500.

How would you measure code quality? Would persistence be a good measure?

This is a great suggestion. I'll note it down for next years. Curious, do you think this would be a good proxy for code quality?

This is per month, I see now that's not super clear on the chart!

We're careful not to draw any conclusions from LoC. The fact is LoCs are higher, which by itself is interesting. This could be a good or bad thing depending on code quality, which itself varied wildly person-to-person and agent-to-agent.

We weren’t able to find a good quality measure. LLM-as-judge dint feel right. You’re correct that without that the data is interesting but not particular insightful.

We weren’t able to agree on a good way to measure this. Curious - what’s your opinion on code churn as a metric? If code simply persists over some number of months, is that indication it’s good quality code?

We expressly did not conclude that more lines = better. You could easily argue more lines = worse. All we wanted to show is that there are more lines.

We were trying not to insinuate that, because we don’t have a good way to measure quality, without which velocity is useless.

Hi, I'm Daksh, a co-founder of Greptile. We're an AI code review agent used by 2,000 companies from startups like PostHog, Brex, and Partiful, to F500s and F10s.

About a billion lines of code go through Greptile every month, and we're able to do a lot of interesting analysis on that data.

We decided to compile some of the most interesting findings into a report. This is the first time we've done this, so any feedback would be great, especially around what analytics we should include next time.

Greptile | Software Engineer (junior, senior, staff)| San Francisco ONSITE | https://greptile.com

Greptile is building AI agents that catch bugs in pull requests. Over 2,000 teams including Brex, Whoop, and Substack use Greptile to review nearly 1B lines of code every month.

We're a team of ~20 in San Francisco, working on things like better agent evals and sandbox execution environments.

We've raised $30M to date, including our recent Series A led by Benchmark.

Stack: Typescript

Open roles: greptile.com/careers

Salary ranges: $140k-270k base (depending on seniority) + $40-100k/yr equity

Greptile | Software Engineer | ONSITE San Francisco (SF) | https://greptile.com

Greptile is working on AI agents that catch bugs and enforce standards in pull requests. Reviewing nearly 1B lines of code a month for 1000+ companies including Brex, Substack, Whoop, as well as multiple F100s.

<20 people, raised ~$30M from Benchmark, YC, Paul Graham, SV Angel and others.

To apply, email daksh [at] greptile.com with subject line "Engineering at Greptile". Include most recent role and company and links to your LinkedIn and GitHub.

That used to be how we did it, but this method performed better on super large codebases. One of the reasons is that grepping is a highly effective way to trace function calls to understand the full impact of a change. It's also great for finding other examples of similar code (for example the same library being used) to ensure consistency of standards.

Respectfully, did you read the post?

Reading that post sent me down the path to this one. This stack order makes total sense, although in practice it's possible 1-2 merge into a single product with two distinct steps.

The 3. is interesting too - my suspicion is that ~70% of PRs are too minor to need human review as the models get better, but the top 30% will because there will be opinion on what is and isn't the right way to do that complex change.

Of the things we had built it was the most useful for us, and the early users found it very useful too

Co-founder of Greptile - that was a bad feature that we since deprecated to focus entirely on AI code reviews

Surely a well-functioning team is high trust enough that the reviewer should shads some of the blame? Of course, the majority might go to the author.

This is surprisingly hard to mitigate with system prompts because not being opinionated is ingrained so deeply in (presumably) post-training