HN user
If you're interested in these kinds of things, many years ago we created TerpreT (https://arxiv.org/pdf/1608.04428.pdf and https://github.com/51alg/TerpreT) to look into generic program synthesis problems, using a set of very different techniques (gradient descent, ILP, SMT) on different problem settings (turing machines, boolean circuits, LLVM IR-style basic blocks, and straight assembly).
There's been a long, long tradition of software verification at Microsoft; look for SLAM (the original research backing the Static Driver Verifier), Yogi (SDV backend V2), Slayer (generic memory safety) and Terminator (termination of system code). All of the research is out in the open, often with reference implementations.
The problem with all of these things (as with FramaC) is that they only work if you restrict the fragment of programs that you're looking at. Kernel modules work out surprisingly well, because they are not allowed to do a whole lot by convention.
That's an interesting question which we haven't studied so far. There is a good argument to be made that ML models trained on public data do not work well on private data on the interesting queries, where I consider those queries interesting that only make sense in the specific project. The core issue there is around the specialised vocabulary that you would only see in the (private) project, which the model would be unfamiliar with. This could be mitigated by using absurd amounts of data, models that can generalise more easily (character-level/BPE/subtokens/...), and finetuning on project-specific data, but it's an open question nonetheless.
It's also easy to test without access to private repos, by just splitting some repos off as a separate test set, so should hopefully be something that other people can also make progress on!
This is a release of a dataset, to help people train machine learning models. That's why the blog post says "We’re also releasing a large dataset to help data scientists build models for this task."
We also provide a way to evaluate how well your machine learning model works. That's why the blog post says "We’re announcing the CodeSearchNet Challenge evaluation environment."
The released data and code (and hence, the announcement) are meant for data scientists and ML researchers who want to work on this problem, and the rest of the world does not need to care. There are no products or applications of this work at this time. If the terms in the blog post don't mean anything to you, then you are not in the target audience for this announcement, and that's OK.
You would need to extend the data-processing pipeline for the new language, which in the best case only requires to adapt the standard wrapper around the Tree Sitter parser. The wrapper needs to take care of language-specific details (e.g., where to find the documentation for a function, what methods should be filtered out, etc.) but can be fairly small. See https://github.com/github/CodeSearchNet/blob/master/function... for examples.
We only used code from repos that github has marked as using a non-copyleft open source license (i.e., we had an explicit license whitelist and used only repos matching that).
[I'm one of the Microsoft Research people who worked on this]
There are many interesting ideas that you could build on top of this kind of data, and we only scratched the surface so far.
For example, the simple "search" technique we are using as a baseline is based on the idea of joint embeddings: We learn functions f_query and f_js/f_python/... to map from the inputs into some vector space such that, for example, for a (python method, docstring) pair, f_query(docstring) is near to f_python(method). To search given a query, we just do f_query(query) and look for nearest neighbours in all the code we indexed before.
Now, we could also just do f_python(def bubblesort(...): ...) and look for the nearest neighbour that is in C#, and should get out a C# implementation of bubblesort. Similarly, we could apply all kinds of filters on the results (code from highly-starred repos, code that uses framework X, ...) to do more interesting thing.
[I'm one of the Microsoft Research people who worked on this]
There wasn't a technical reason (unless you count lazyness as a technical reason) -- we simply had infrastructure for Python specifically lying around from past research projects, which we initially reused.
After we got Nat's feedback, we redid our data processing pipeline completely to be based TreeSitter (which wasn't around when we started thinking about parsing Python), which makes it much easier to scale to the number of programming languages on GitHub.
[I'm one of the Microsoft Research people who worked on this]
We used the repo-level license information from github to filter to free non-copyleft licenses, and the license files are actually stored next to the extracted data corpus (see https://github.com/github/CodeSearchNet#Licenses).
[I'm one of the Microsoft Research people who worked on this]
All code is from public repositories. We only used publicly available APIs to obtain the data, so that others can reproduce the results / build on top of the data extraction pipeline we built: https://github.com/github/CodeSearchNet/tree/master/function...