The VSCode GitLab extension now supports getting code completions from FauxPilot

https://twitter.com/moyix/status/1581069716231049216

by moyix • 4 years ago

190 65 4 years ago

twitter.com

cercatrova2• 4 years ago

Good to see the OSSification (no pun intended) of prorpietary VSCode parts such as Copilot and their extension server.

Havoc2• 4 years ago

Extension server has a oss now?

commoner• 4 years ago

Yes, it's called Open VSX and it's developed by the Eclipse Foundation.

- Open VSX: https://open-vsx.org

- Source: https://github.com/eclipse/openvsx

VSCodium and Eclipse Theia use Open VSX by default.

- VSCodium: https://github.com/VSCodium/vscodium#extensions-and-the-mark...

- Eclipse: https://www.eclipse.org/community/eclipse_newsletter/2020/ma...

syntaxing• 4 years ago

This is super awesome, too bad a A6000 cost $4500 or I would try this out myself.

moyix9• 4 years ago

The smaller models run on smaller GPUs too! :) You can see how much VRAM is required for various models in the Documentation:

https://github.com/moyix/fauxpilot/blob/main/documentation/s...

And we have an initial GPU support matrix page here:

https://github.com/moyix/fauxpilot/wiki/GPU-Support-Matrix

A planned feature is to implement INT8 and INT4 support for the CodeGen models, which would let the models run with much less VRAM (~2x smaller for INT8 and ~4x for INT4) :)

Namidairo• 4 years ago

A planned feature is to implement INT8 and INT4 support for the CodeGen models, which would let the models run with much less VRAM (~2x smaller for INT8 and ~4x for INT4) :)

That would certainly put the larger models within reach of most users. Part of my reluctance to deploy this myself was I'd have to either redline the VRAM on a 8GB card or choose the smallest model.

Would this require a retrain of the models?

daviding3• 4 years ago

The 7GB runs great on a 3080Ti. I am getting a lot of 'ValueError: Max tokens + prompt length' errors with larger files. Can this Gitlab client also replace the vocab.bpe and tokenizer.json config like Copilots? Thanks for your work on Fauxpilot, really enjoying playing with it.

KaoruAoiShiho3• 4 years ago

Ahh, 3090/4090 conspicuously missing, those affordable 24gbs are very common in the ML community, please add them to the matrix.

teruakohatu• 4 years ago

Two 3090 GPUs could probably handle it, making the cost slightly cheaper.

KaoruAoiShiho3• 4 years ago

Is it possible to do 2x 4090s or is 3090s the only option putting 2 together?

john_cogs2• 4 years ago

GitLab team member here.

This work was produced by our AI Assist SEG (Single-Engineer Group)[0]. The engineer behind this feature recently uploaded a quick update about this work and other things they are working on to YouTube[1].

[0] https://about.gitlab.com/handbook/engineering/incubation/ai-...

[1] https://www.youtube.com/watch?v=VYl0dg8xyeE

remram5• 4 years ago

Why do you call this a group? Why don't you say "one of our engineers did this"? I read the linked article [1] and that seems to be the accurate situation: not a group of people including one engineer, not one engineer at a time on a rotation, but literally one person. In what way is that a group?

[1] https://about.gitlab.com/company/team/structure/#single-engi...

john_cogs2• 4 years ago

Great question. I'm not entirely clear on the origin of the name and it would probably be hard for me to find the folks behind this decision on Friday evening/Saturday so I'll share my interpretation.

At GitLab, we have a clearly defined organizational structure. Within that stucture, we have Product Groups[0] which are groups of people aligned around a specific category. The name "Single-Engineer Groups" reflects that this single engineer owns the category which they're focusing on.

I'll be sure to surface your question to the leader of our Incubation Engineering org. Thanks.

[0] https://about.gitlab.com/company/team/structure/#product-gro...

lbotos• 4 years ago

John nailed it. A single engineer group is something being explored by one person, but could eventually represent a "product group" in the future. See https://about.gitlab.com/direction/#single-engineer-groups for more things that could be explored by such a person.

If you think there is a better name to capture that I'm sure it would be considered.

sidlls• 4 years ago

I’m always suspicious of “one engineer did this” projects. There are probably quite a few engineers, product people and other employees who are simply unattributed for this.

wharfjumper• 4 years ago

Classic off by one error.

RobotToaster• 4 years ago

"one of our engineers did this" isn't a technical sounding acronym.

abirch• 4 years ago

This is so awesome!!! Your team rocks. Too bad it wasn't named after your cat: FelinePilot or CatPilot.

arvonle• 4 years ago

The Gitlab mascot is a raccoon dog, closely related to the true fox. FauxPilot.

moyix9• 4 years ago

I will inform Andrei and Katya that they have a fan :) Although they already have a pretty high opinion of themselves!

sva_2• 4 years ago

From https://github.com/moyix/fauxpilot

if you have two NVIDIA RTX 3080 GPUs, you should be able to run the 6B model by putting half on each GPU.

Thats quite excessive, but nice that it can be ran locally.

Note that the VRAM requirements listed by setup.sh are total -- if you have multiple GPUs, you can split the model across them.

This might not work with the upcoming RTX 40x0 GPUs unless they have enough memory, as they abolished NVLink in that series.

Edit: I'm now seeing there are smaller models too. Going to give it a try tomorrow.

woodson• 4 years ago

You don’t need NVlink for that to work.

sva_2• 4 years ago

Apologies for my ignorance, but don't you need NVLink to pool the memory of the cards together?

doctoboggan• 4 years ago

I know there seem to be a few different models to choose from, does anyone have a resource that collects them and shows the strengths and weaknesses of the various options? I’d be particularly interested in models optimized for apple silicon.

Also, how do they compare to copilot?

Vetch• 4 years ago

Unfortunately, Copilot is a lot more capable. Most important is that it works with many more languages out of the box, is continuously updated, has more mathematical plus scientific knowledge and is better at understanding your comments.

As for which to use, you'd want to take the largest model that can you can fine-tune (not necessarily on your machine) and fits comfortably in your machine. The small models aren't as flexible and are basically just a cleverer autocomplete. Copilot is a lot smarter than that and smarter than all models I've tested up to 6B. It can often cleverly work out a lot of local patterns for what you're doing and this is true surprisingly often for novel languages, math, understanding graph notation, categorization policies and much, much more.

I'd gone in hoping I could remove my dependence on Copilot but left disappointed. The small models up to 6B just don't compare. I don't have the resources to run the 13B model which is probably behind anyways due to not having trained as long on as much as copilot (which is continuously updated). I also suspect Copilot has a great deal of hand-coded hacks and caching tricks to improve the user experience.

moyix9• 4 years ago

Yep, I definitely agree that the 6B and below models are worse than Copilot. The 16B ones are pretty good! But IMO still undertrained compared to Copilot, and of course much less accessible (though see elsewhere in this comment section; I think INT8 and even INT4 should be doable – this won't help much with inference latency but it should help most people fit the 16B model locally).

I have high hopes for the BigCode project too; they will get to take advantage of a lot of things that have been learned about how to train code models effectively.

One last note – I think that many of the improvements to Copilot since its release are attributable to "prompt engineering" and client-side smarts about what context should be included; I'm not certain, but I can believe the underlying Codex model used hasn't changed much (if at all) since code-davinci-002.

croes• 4 years ago

I'd gone in hoping I could remove my dependence on Copilot

You already depend on Copilot?

dang• 4 years ago

FauxPilot – an attempt to build a locally hosted version of GitHub Copilot - https://news.ycombinator.com/item?id=32327711 - Aug 2022 (79 comments)

daviding3• 4 years ago

This seems to work ok on a 3080Ti, even under WSL2. Nice! For larger files I'm getting errors for maximum token + prompt length - is there an easy way to tweak these limits for the client? I think the Copilot client needed some overrides for Fauxpilot for this, so hoping Gitlab client has that too.

speedgoose• 4 years ago

Shouldn’t this be in a separate extension?

avip3d• 4 years ago

That would make sense... but them most people wouldn't install the base gitlab extension.

eddyg• 4 years ago

Came here to say this.

I’d much rather see effort being put into other areas of the GitLab Workflow extension (like being able to approve / view approvals of MRs).

ddevault• 4 years ago

You'd think that GitLab would pause from chasing after GitHub's tail for at least long enough to question whether or not these features are ethical.

The VSCode GitLab extension now supports getting code completions from FauxPilot

Related Stories