hunterbrooks

congrats Greg, Merrill, and the rest of the folks at Graphite!

- Hunter @ Ellipsis

Bingo - it’s colloquially

Yes. My business is in the code review space, synthetic data is very helpful for evals

LLM's get relatively better at read-heavy operations (ex: code review) than write-heavy operations (ex: code generation) as codebases become less idiomatic.

I'm a cofounder at www.ellipsis.dev - we tried to build code generation for a LONG time before we realized that AI Code Review is way more doable with SOTA

100ms is crazy fast. Approx what's the split b/t the embeddings calls and the graph traversal work (the library stuff)?

Yes. Learning to "program" has never been easier, but nitty gritty stuff like networks and routing is still much more technical than the avg SWE

HA! Your LLM agents/prompts nailed a hilarious tone. For https://ellipsis.dev

This website looks like it was designed by someone who just discovered the concept of "AI" and decided to slap it all over a half-baked template. Your color scheme is about as inspiring as a Windows '95 screensaver, and your layout feels like it was put together by a kindergartner with a crayon and a ruler. What's with the "trusted by 1,400+ GitHub users" badge? Do you expect anyone to be impressed by this grand endorsement from approximately 0.00001% of the developer community? Your idea of "AI code reviews" seems as lofty as a middle school science project titled "Let's Cure Cancer." You hail it like it's the second coming of sliced bread, but your execution and presentation scream amateur hour. And "7-day trial"? Can't wait to dedicate a whole week to something that looks like it was made on a GeoCities page. Fix your design, your messaging, and maybe try again when you actually understand the tech you're hawking.

The worst part is we did use a template!

Nice, I imagine you're using your own product? I got "LAUNCH SALE $100 OFF"

1. How do you provide consistent banners to an individual? or a device? 2. Are you aware of anyone doing this for Saas products?

Cool, probably not relevant info for modern devs because of all the hosted git providers, but interesting nonetheless.

Got any thoughts on how LLM powered code generation products can use format-patch?

Also, 2009 was pre git?!

Pricing is on the landing page. $20/seat per month

Good catch. The team could add this rule to their Ellipsis config file to make sure that it's always flagged: "Never use magic numbers. Always store the number in a variable and use the variable instead."

Docs: https://docs.ellipsis.dev/config#add-custom-rules

Unfortunately not. We’re early - all our data is qualitative.

But we know we need hard numbers, so we’re working on it. We don’t want to sell a novelty, the product needs to actually save time

No way, anyone can make silly mistakes. Ellipsis is a 2nd pair of eyes to catch stuff like that

The product works on draft PR's too, but not on local.

Sometimes reviewers rope in Ellipsis by asking questions (we also support natural language Q&A about a PR) or by having a design discussion via GH comments and then assigning the change to Ellipsis

As it stands today, Ellipsis isn't sold as an AI software engineer.

One of our largest learnings is that state of the art LLM's aren't good enough to write code autonomously, but they are good enough to be helpful during code review.

Automated code reviews are a step in the software development lifecycle, not a replacement for human reviewers.

Ellipsis can't commit code without your permission and approval, so this particular parrot can't feather your filesystem.

All those tools do code review, so you'll have to try them out for yourself to see which is the most helpful.

But when it comes to writing code for you, not all those tools actually run your unit tests/linter/compile/etc. Ellipsis will, and it'll use the stdout/stderr to fix it's own mistakes, meaning the commit delivered to you actually compiles/passes CI.

Hmm, that searches issues, which isn't the best way to see Ellipsis' work.

Example of PR review: https://github.com/getzep/zep-js/pull/67#discussion_r1594781...

Example of issue-to-PR: https://github.com/getzep/zep/issues/316

Example of bug fix on a PR: https://github.com/jxnl/instructor/pull/546#discussion_r1544...

Ellipsis increases the quality of code a developer brings to their teammates, meaning fewer PR comments, meaning the team ships faster. It's not a replacement for human review. It's most often used by the PR author after a self review but before asking for reviews from human teammates.

Ellipsis will use your existing lint/test commands when making code changes. For example, you can leave a comment on a PR that says "@ellipsis-dev fix the assertion in the failing unit test" and Ellipsis will run the tests, observe the failure, make the code change, confirm the tests pass, lint-fix the code, and push a commit or new PR

Hmm... probably, send me an email.

Nearly all languages are supported, but some perform better than others.

JS/TS, Python, Java, C++, Ruby are highly supported.

It's most helpful when a GitHub/Linear issue is linked because the "why" is extracted, and also for larger PR's

We offer Ellipsis to large open source projects for free. Email us team@ellipsis.dev

I was referencing the recent xz backdoor hack.

Totally fair - there's a saturation right now of magic AI dev tools. We try to differentiate by not over promising/under delivering and by solving a problem that's closely matched to what today's state of the art LLM's can handle: code review.

But the only really way to figure out if it's useful for your team is to try it. That's why we added a free trial.

Yeah, I use it for lots of boilerplate work like adding new API endpoints, new Celery jobs, and building react components.

Definitely agree with this.

One analogy I see today is the typical testing pipeline. Unit tests get run locally, then maybe a CI job runs unit tests + integration tests, then maybe there's a deployment job which does a blue/green release. At every stage the system is being "tested", but because the tests validate different capabilities, it's like concentric circles that grow confidence for the change.

A software dev lifecycle that uses AI dev tools is similar. Agents will review/contribute at the various stages of the SDLC, sometimes with overlap, but mostly additive and building on the output of one another.

No, but maybe if your widely adopted, poorly supported open source project uses Ellipsis for code reviews we may be able to catch that type of hack ;)

Ellipsis uses BUNCH of LLM agents internally. If you built your own code generation LLM agent you'd need to also build a way to execute the code that the agent writes, which is a bit of an engineering headache.

We handle this, the result is that if you set up a Dockerfile, we promise to return working, tested code: https://docs.ellipsis.dev/code#from-a-pr

Teams should use both Copilot (synchronous code generation) and Ellipsis (async code gen).

Sure, Copilot speeds up human dev productivity, but our take is that humans should only be spending their time on the highest value code changes and use products like Ellipsis to handle the rest.

The downside of async code gen is that Ellipsis workflows take a few minutes to run because Ellipsis is actually building the project, running the tests, fixing it's mistakes, etc. The upside is that a developer can have multiple workflows running at once, and each workflow delivers higher quality code because it's guaranteed to be working + tested.

I'm super bullish on async code gen. I think there's a whole category of tedious development tasks with unambiguous solutions that can be automated to the point where a human just needs to give it a LGTM.

Does Devie auto-fix compilation errors?

Awesome to see you have this on your roadmap. Our product has similar functionality and it's quite a pain to get right. Excited to see how this shapes up