Enhancing Code Completion for Rust in Cody

https://sourcegraph.com/blog/enhancing-code-completion-for-rust-in-cody

by imnot404 • 2 years ago

82 26 2 years ago

shahahmed• 2 years ago

arguably you can reduce even more latency by keeping the model on-device as well, but that would mean revealing the weights of the fine-tuned model.

If the user preferred reduced latency and had the RAM, is that an option?

s1mplicissimus• 2 years ago

the model is probably most of the "secret sauce" of cody, so if they gave that away people could just copy it around like mp3s. my guess

morgante• 2 years ago

Completely incorrect, as Sourcegraph has not historically trained models and Cody swaps between many open source and 3rd party models.

rdedev• 2 years ago

Looking at their GitHub page or seems like they are using existing LLM services. It should be possible to modify cody to make it work with a local llm

ado__dev• 2 years ago

You don’t have to modify anything. We support local LLMs for chat and completions with Ollama.

https://sourcegraph.com/blog/local-code-completion-with-olla...

daemonologist• 2 years ago

This is true, but only if you have a GPU (/accelerator) comparable in performance to the one backing the service, or at least comparable after accounting for the local benefit. This is an expensive proposition because it will be sitting idle between completions and when you're not coding.

sqs6• 2 years ago

Not for this fine-tuned model yet, but Cody supports local models: https://sourcegraph.com/docs/cody/clients/install-vscode#sup....

I just used Cody with Ollama for local inference on a flight where the wifi was broken, and it never fails to blow my mind: https://x.com/sqs/status/1803269013310759236.

sqs6• 2 years ago

Awesome to see our Rust fine-tune here on HN. We (Sourcegraph team) are here if anyone has questions or ideas for us!

BTW, Cody is open source (Apache 2): https://github.com/sourcegraph/cody.

IshKebab• 2 years ago

Your Cody page doesn't answer a very obvious question: does the LLM run locally or is this going to send all my code to Sourcegraph? I assume that is a deliberate omission and the answer is the latter.

jdorfman• 2 years ago

The default LLM is Claude 3 Sonnet hosted by Anthropic. However you can run local models via Ollama: https://sourcegraph.com/docs/cody/clients/install-vscode#sup...

tamimio• 2 years ago

Thank you, cody.dev kept giving me a white blank page.

mutematrix• 2 years ago

Sorry about that! I just looked into this (also on the Sourcegraph team). If you get a chance, could you check if it's working now?

riquito• 2 years ago

Just wanted to thank you, is a great product. I find it to have a better context and awareness of my code compared to copilot

ekidd2• 2 years ago

From the article:

Although most LLMs are trained on corpora that include several programming languages, we often observe differential performance across languages, especially languages like Rust that are not well represented in popular training datasets.

Interesting! My experience with Rust and Copilot is that Rust is actually a relatively strong language for Copilot, in many ways. I suspect that this might be a mix of different things:

- Is the training set code for Rust relatively high-quality?

- Do strongly-typed languages have a better chance of catching incorrect completions quickly?

cageface2• 2 years ago

My very limited experience with GPT and Rust is that the code it generates rarely satisfies the borrow checker out of the box. I’ve had much better luck with garbage collected languages.

ekidd2• 2 years ago

Interesting. I find that Copilot is quite good with the borrow checker, at least if it can "see" the signature of the function that it's calling. Sometimes it inserts an extra "&" where it really shouldn't. That's a no-op, clippy catches it, and I take it out.

But I tend to write my code in a style that easily passes the borrow checker 99% of the time. I keep mutation very local, and rely on a more "functional" architecture. So Copilot is naturally going to write new code in the same borrow-checker-friendly style, which may keep it out of trouble.

When using ChatGPT, I've actually had more luck asking it to write Python, and then to translate the Python into Rust. ChatGPT seems to be better at algorithms in Python, and it's surprisingly good (in my personal experience) at inserting the fine details of Rust types and references when translating.

MarkMarine2• 2 years ago

I’ve been using Cody and liking it, but one thing I noticed is there is no loop that even takes the data from the IDE and incorporates the IDE’s feedback into the code it generates. It’ll happily generate code the IDE draws red wiggle lines under and just continue on with its day. Would be nice to see a tighter integration

sqs6• 2 years ago

Yeah, that is needed to make everything more accurate. We are working on 3 related things to make Cody better understand the structure of your code and take feedback from the compiler (and other tools).

All are experimental and not enabled by default, but you can peek at them if you're interested.

1. (experimental) LSP-based context — https://github.com/sourcegraph/cody/pull/3493

2. (experimental) TypeScript tsc context — https://github.com/sourcegraph/cody/pull/3843

3. (experimental) OpenCtx, which can pull in diagnostics from tools outside your editor — https://openctx.org/

And then of course the agent to actually iterate on the generated code based on this feedback.

We're really excited about this stuff, and it'll arrive in Cody when we can get it to a high quality bar and prove it helps.

MarkMarine2• 2 years ago

Awesome! I’ll take a look

esafak3• 2 years ago

Cody is decent. It's my copilot of choice at the moment. It indexes your code base so it can give contextual responses.

sqs6• 2 years ago

What would make it better than decent for you?

esafak3• 2 years ago

1. Write the commit messages too. 2. Allow the user to refactor across selected portions of the entire project, not just one file.

Enhancing Code Completion for Rust in Cody

Related Stories