BrandonJung

One of the team here at Tabnine. Thank you for choosing us and please let us know if we can help.

interesting point

thank you you are right, I did comment in the above threads I am with Tabnine but can always do that better.

I am biased but the update Tabnine made in the last 6 months are very interesting.

Tabnine can train on your codebase for you AND your team. It is secure then to only you and the team you add.

Tabnine has been doing this for years longer than copilot with more users and better ratings. Not a monopoly (except that MSFT has so much marketing clout). The LLM space that we build on has gotten much more diverse after MSFT bought OpenAI and closed collaboration for many industry participants including ourselves. (We used to run on GPT2 but changed to an agnostic strategy that can use any LLM when MSFT would not let us use GPT3) Take a look at the new innovation with T5 from SFDC, AI21, Google, Meta all building powerful models. Transparently we need your help in ensuring that this is not a monopoly.

Yes we at Tabnine have been at this for 7+ years and we believe that this is the current best solution. Only train on fully permissive code and then specialize those models privately for each team/org/org.

Does this change if the AI only uses fully permissive code? We took that approach at Tabnine and would be interested if the communities primary concern is disregard of licensing or the use of AI at all.

It looks like you used Tabnine local only model in the past (awesome for security, connectivity) if you used it in "cloud only" mode you would see much lower CPU and get longer snippets and completions. If you are with a company we are also happy to build and run all of this with custom models and in your own VPC or datacenter. Please contact us if you are interested and happy if you are also just an IC to get you Tabnine Pro. Disclosure: I am from Tabnine.

love the speed you are moving!

Clarification - a user can run Tabnine on their laptop only OR as a part of Tabnine Enterprise we can run the large cloud models in your VPC (on GPU's) for your entire team of developers. Your code is your code and you can run it anywhere.

If you would like a custom model on your code (or language) this is one of the unique capabilities of Tabnine. We have done this on lots of code already and do it all the time for companies across the globe. Your code is your code. Transparently I am with Tabnine.

Good feedback and thank you for taking the time to share

Ballpark guess is that to build CODEX it would be ~$10M.

A model is only as god as the code that it trains on and the people it learns from. In your example above we can build a custom model on "best practices" code and not the companies codebase. Our work here has focused on working with ISV's like AWS, GCP, etc as they will have best practices for their tech. We then pull that together for a team in a custom model.

The code-search application is completely distinct from the IDE-assistant application. They do not share any code, have completely different pipelines for training the backend and completely different datasets. The specific source you mentioned will not be part of the training dataset for Tabnine's IDE assistant. Hope this clarifies.

it is likely that for the stuff you are paid for a custom model might be a better fit.

it is a question we are always asking but not sure what people would do if they were open-sourced. Did you have a specific use case in mind?

Cost wise the biggest spend is building the large model (CODEX for Copilot). While CODEX is close sourced there are a number of other good open source transformer models from SFDC (T5), AI21, Meta that are the foundational models we use. Going forward we will continue to partner with the community to enable these large models to reach developers in code-time.

please take a look at inline suggestions that will be in grey and not in the regular autocomplete box. It is now the default setup.

Would you be willing to share your concerns about the EULA? The pricing question is something we are currently discussing as you can imagine. It is surprising that our EULA was a problem but GitHub's was ok given they collect your information for training.

my guess given that I do not have significant karma points is that this is a subject that millions of devs are trying to understand more deeply (a million plus for both Tabnine and Copilot). There are two competitors with significantly different takes on how we solve this problem both technically and securely.

no need to worry we love good feedback here at Tabnine ;-) Your observations are appreciated. In this case we are using general code and not a custom model but if you would like a custom model then I would wager you would get better variable names... maybe even "awesome". Our goal with our customers is to build models based on the best custom code and input from the best developers so that your new devs can learn from YOUR CODE and with YOUR BEST PRACTICES.

feel free to turn off local completely. The bursts you are seeing is building a local transformer model on your computer. For enterprises and developers that dont want their data to go anywhere this is critical but does come with tradeoffs. We continue to make this better or feel free to use with GitPod (https://www.gitpod.io/blog/tabnine)

Appreciate the feedback and looks like you are clearly in the developer group that wants very fast small useful completions. Can you share the modifications that you made and perhaps a few more details on what they were?

Jacob Jackson is awesome and we loved the time we got to work with him. Brilliant dude

You are not the only one that has said this but would love a link to a video on how you like to work with both together.

Copilot uses a huge model and a ton of GPU's (and an unlimited budget) to focus on really long completions. This is a very specific choice to go for long completions.

We have worked hard to solve this problem differently and with the new generation you will see longer multi-line completions while also giving developers the options to use shorter suggestions (perhaps together with Copilot)

For this specific comparison, it’s essential to start from the technology, as many of the product differences stem from the differences in approach, architecture, and technology choices. Microsoft and OpenAI view AI for software development almost as just another use case for GPT-3, the behemoth language model. Code is text, so they took their language model, fine-tuned it on code, and called the gargantuan 12-billion parameter AI model, Codex.

Copilot’s architecture is monolithic: “one model to rule them all.” It is also completely centralized - only Microsoft can train the model, and only Microsoft can host the model due to the enormous amount of computing resources required for training and inference.

Tabnine, after comprehensively evaluating models of different sizes, favors individualized language models working in concert. Why? Because code prediction is, in fact, a set of distinct sub-problems which doesn't lend itself to the monolithic model approach. For instance: generating the full code of a function in Python based on name and generating the suffix of a line of code in Rust are two problems Tabnine solves well, but the AI model that best fits every such task is different. We found that a combination of specialized models dramatically increases the precision and length of suggestions for our 1M+ users.

A big advantage of Tabnine’s approach is that it can use the right tool for any code prediction task, and for most purposes, our smaller models give great predictions quickly and efficiently. Better yet, most of our models can be run with inexpensive hardware.

Now that we understand the principal difference between Microsoft’s huge monolith and Tabnine’s multitude of smaller models, we can explore the differences between the products:

First, kind of code suggestions. Copilot queries the model relatively infrequently and suggests a snippet or a full line of code. Copilot does not suggest code in the middle of the line, as its AI model is not best suited for this purpose. Similarly, Tabnine Pro also suggests full snippets or lines of code, but since Tabnine also uses smaller and highly efficient AI models, it queries the model while typing. As a user, it means the AI flows with you, even when you deviate from the code it originally suggested.

Second, ability to train the model. Copilot uses one universal AI model, which means that every user is getting the same generic assistance based on an “average of GitHub”, regardless of the project they're working on. Tabnine can train a private AI model on the specific code from customers’ GitLab/GitHub/BitBucket repositories and thus adjust the suggestions to the project-specific code and infrastructure. Training on customer code is possible because Tabnine is modular, enabling the creation of private customized copies.

Third, Code security and privacy. There are a few aspects of this. Users cannot train or run the Copilot model. The single model is always hosted by Microsoft. Every Copilot user is sending their code to Microsoft; not some of the code, and not obfuscated - all of it. With Tabnine, users can choose where to run the model: on the Tabnine cloud, locally on the developer machine, or on a self-hosted server. This is possible because Tabnine has AI models that can run efficiently with moderate hardware requirements.

In addition, Tabnine makes a firm and unambiguous commitment that no code the user writes is used to train our model. We don’t send to our servers any information about the code that the user writes and the suggestions they’re receiving or accepting.

Fourth, commercial terms. Microsoft currently offers Copilot only as a commercial product for developers, without a free plan (beyond a free trial) or organizational purchase. Tabnine has a great free plan and charges for premium features such as longer code completions and private models trained on customers’ code.

https://tabnine.com/tabnine-vs-github-copilot

to be clear they actually took much of what Tabnine was doing and declined to let us use GPT3 (MSFT closed Open AI). It turned out to be good because there are lots of other great models out there from T5 (SFDC), Meta, Google, etc that will continue to move forward faster.

it is possible that on a long snippet that you could be copying code from non-permissive code. In addition GitHub themselves have said that this is not resolved and will likely need to be adjudicated in the courts. The only way to avoid this is to train on only fully permissive code. We know here at Tabnine having done this for 5+ years.