Chunking 2M files a day for code search using syntax trees

https://docs.sweep.dev/blogs/chunking-2m-files

by kevinlu1248 • 3 years ago

168 63 3 years ago

docs.sweep.dev

yding2• 3 years ago

This is really cool and a much needed contribution to helping LLMs run better on large code bases.

kevinlu124829• 3 years ago

Thanks! Would love to see this algorithm in LlamaIndex.

thomasfromcdnjs• 3 years ago

Also thank you! I had an idea of how this all worked but this definitely concreted my thoughts.

sandGorgon• 3 years ago

oh really ? Thats awfully kind. I'll take that in for EdgeChains as well.

https://github.com/arakoodev/EdgeChains/issues/172

kevinlu124829• 3 years ago

For others in this thread, this algo is now publicly accessible in LlamaIndex at https://github.com/jerryjliu/llama_index/blob/e567e6a20cf89b...

intalentive• 3 years ago

Next step is to train models directly on syntax trees. Higher probability of correct output.

kevinlu124829• 3 years ago

That's interesting, I've seen a few papers about this. I'm personally curious about editing syntax trees using language models, since it would prevent syntax errors altogether.

vladf• 3 years ago

Which papers?

madeofpalk• 3 years ago

In my limited use, I've never seen these models (chatgpt, and github copilot) generate invalid syntax. I don't see much to improve there

I do see them generate code that fails the type checker though.

Zambyte• 3 years ago

John McCarthy was right

kevinlu124829• 3 years ago

This is interesting. I'm taking a read on this.

eldenring• 3 years ago

I'd guess these model's understand works more closely to people so encoding in text is more token efficient and things like comments help.

Also syntax seems a lot easier to understand for them than semantics/logic. If you've used GPT-4 it almost never makes syntax errors. Logical errors on the other hand...

kevinlu124829• 3 years ago

From my experience, GPT-4 never makes syntax errors directly but when making edits to existing code it's harder to prevent these syntax errors from appearing. We used to add a second pass to check for these syntax errors.

It also frequently makes undefined variables and the like, however.

karmasimida• 3 years ago

Programming languages are artificial languages. LLM are able to synthesize human languages with almost perfect grammatical quality, they are in fact, very unlikely to make obvious syntactic errors on programming languages.

Also, syntax level information are local or short sighted, it is called context-free grammar for a reason. My own observation with playing with those coding LLMs all day, is that they most likely had acquired the grammar themselves implicitly. Providing explicit regularization by enforcing grammar, is going to provide at best modest benefits, and that is dependent on good that parser is written, in many cases, it is not a given.

kevinlu124829• 3 years ago

Ya I think forcing correct syntax at the generation level likely will not be extremely beneficial. At Sweep, we iterate the language models on linters and type-checkers using GitHub Actions and it yields better results.

woadwarrior012• 3 years ago

Indeed, it's intuitively more efficient for LLMs to operate on ASTs instead of raw source code. I came across a recent paper[1] that takes this approach.

[1] https://arxiv.org/abs/2305.00909

kevinlu124829• 3 years ago

This is interesting, I'll take a look. My main concern with running this in production is that there is more text data in the world than code. Further, a pure tree-manipulation model has less explainabilities since you can always ask GPT-4 what it's thinking.

awwaiid• 3 years ago

Also git diffs and execution traces.

sdesol3• 3 years ago

Congrats! Your project is off to a very good start as shown at https://devboard.gitsense.com/sweepai

What is interesting to me is the sharp increase in forks, which is a good indicator that others will contribute code in the near future.

Full Disclosure: This is my tool

kevinlu124829• 3 years ago

Hey thanks for showing this dashboard. There's some crazy analytics in here, the tool looks awesome!

sdesol3• 3 years ago

Thanks. For privacy reasons, I'm not showing a lot, but in the future when auth is in place, I can show deeper insights for repo members.

alj032• 3 years ago

I just wanted to say the site looks a little odd on mobile

Edit: I guess it is just the app I am using, it looks fine on my mobile browser but odd in the app

kevinlu124829• 3 years ago

Which app? We did notice a few visual errors a while back.

kartoolOz• 3 years ago

Would see great improvments in retrieval accuracy by finetuning e5-base-v2 or the newer leaders on mteb benchmark.

kevinlu124829• 3 years ago

Definitely. I prefer the sentence-transformers ones since they have been fine-tuned on codesearchnet. I'm also really excited about the latest gte models by Alibaba, their smallest model is the size of MiniLM L6 but beats MPNet.

alchemist1e94• 3 years ago

Awesome first step. Next is to figure out how to apply syntax trees for diffs and then train the LLM on code and diffs but all in syntax trees somehow.

kevinlu124829• 3 years ago

Yup, saw a few papers about this over the past two years, using graph neural networks for code generation. There's also another thread below on this topic.

Edit: Here's some of the papers: https://arxiv.org/abs/1911.09983 and https://aclanthology.org/2021.findings-acl.384.pdf

alchemist1e94• 3 years ago

Very interesting! Thank you.

I’ll try to explain something I’m thinking, it comes down to a type of agglutination.

https://en.wikipedia.org/wiki/Agglutination

The ASTs need to become sequences of tokens for LLMs to work well. Also the embedding space is related, this should all be specifically optimized for code, not based on general human language .

Of course the description of the code would be english. But my point is that a subspace of the encoding should be specifically designed for an agglutination based sequential expression of ASTs.

Not sure any of that makes sense to someone with more expertise in this space than me.

wanderingmind2• 3 years ago

Tangent, are there any other similar alternatives to sweep that is not restricted to github but can be installed in other places like gitlab, bitbucket or even self hosted.

kevinlu124829• 3 years ago

Not a great answer but we are open-source so forking us is an option.

wanderingmind2• 3 years ago

Yes I had that thought, but your license looked custom so was not sure if that was allowed.

kevinlu124829• 3 years ago

Update: this algo is now publicly accessible in LlamaIndex at https://github.com/jerryjliu/llama_index/blob/e567e6a20cf89b...

d4rkp4ttern• 3 years ago

curious why it is under "langchain_helpers"... I assume there is nothing specific to langchain here?

yding2• 3 years ago

Currently all of LlamaIndex's text splitters were/are Langchain compatible.

mellosouls2• 3 years ago

OT: I'm not clear on whether or not Sweep is fully open source - I mean, can you run it fully self-hosted (apart from the GPT4 engine obv), or is the repo essentially a client to a Sweep API/binary?

Cool project btw!

kevinlu124829• 3 years ago

Thanks! The repo is just the backend that runs the GitHub webhooks. We used to have a "chat with your code" client but stopped supporting it. Now it's only the GitHub interface with creating tickets and comments.

mellosouls2• 3 years ago

Thanks for the clarification.

Chunking 2M files a day for code search using syntax trees

Related Stories