mallamanis

[I'm one of the Microsoft Research people who worked on this]

Thanks for your questions! We have thought of many heuristics but we didn't want to constrain the dataset release on some heuristic that we picked, possibly ruining the dataset. Participants in the challenge should feel free to apply additional filters as they see fit. For example, this [1] work could be useful as a filtering method.

Unfortunately, we do not have the budget to provide any compute resources to help with running the models at this time. Note that any techniques developed with this dataset will be owned by those who develop them and it's up to them how/if they will make them available/open-source.

[1] https://arxiv.org/abs/1806.04616

[I'm one of the Microsoft Research people who worked on this]

(thanks Nick. Here are the links)

Generative Code Modeling with Graphs: https://arxiv.org/abs/1805.08490 Learning to Represent Programs with Graphs: https://arxiv.org/abs/1711.00740

[I'm one of the Microsoft Research people who worked on this]

That's certainly true for simple use cases. Our goal here is to eventually also capture the long-tail of queries about a codebase. Often, within the domain of a project there is a set of natural language terms/jargon that describe complex tasks specific to the domain. Imagine for example a developer joining a mid-sized project and trying to find how to achieve some simple but project/domain-specific task.

[I'm one of the Microsoft Research people who worked on this]

We did consider adding StackOverflow questions. Some of our queries in the CodeSearchNet challenge do actually come from StackOverflow (via StaQC [1]). It's certainly interesting to see how all other SO data can be useful for this task. Thanks for the suggestion!

The reason we didn't try this at this point:

Many people in research have tried working with SO data. In my experience I have observed an interesting problem with the data: it's deduplicated! This is great for users but bad for machine learning, since the data looks "sparse" (roughly, each concept appears once). Sparsity is an obstacle, since it's hard for most existing machine learning methods to generalize from sparse data. In contrast, in natural language there are (e.g.) multiple articles describing the same event more or less.

[1] https://ml4code.github.io/publications/yao2018staqc/

Using machine learning like this https://arxiv.org/pdf/1506.05869.pdf would be fun for this and avoid(?) canned responses...

An upcoming NIPS paper seems to be a followup

"Hidden Technical Debt in Machine Learning Systems" D Sculley*, Google Research; Gary Holt, ; Daniel Golovin, Google, Inc.; Eugene Davydov, Google, Inc.; Todd Phillips, Google, Inc.; Dietmar Ebner, ; Vinay Chaudhary, Google, Inc.; Michael Young, Google, Inc.; Jean-Francois Crespo, Google, Inc.; Dan Dennison, Google, Inc.

(see https://nips.cc/Conferences/2015/AcceptedPapers )

Unfortunately, I haven't been able to find the paper yet.

http://googleresearch.blogspot.co.uk/2015/08/the-reusable-ho... this seems relevant to the article.

http://research.microsoft.com/en-us/um/people/marron/selectp... has some interesting technical information about time-travel debugging.

I use https://github.com/magicmonty/bash-git-prompt which I also like. It seems to present less information than this one though

There is this https://code.google.com/p/gittorrent/ although I've never tried it...

edit: a more recent (?) link https://github.com/cjb/gittorrent

Well, they should now ask for this link to also be removed...

So, is this essay also considered harmful by self-reference?

Some of the advice seems to be oriented for a US-like culture (e.g. be direct). Other cultures might have different conventions, I think.

I do not expect them to decrease military expenditures. Their coalition partners (the far-right "Independent Greeks" that will be "holding" the ministry of defense) will not allow that easily.

There is some recent research has shown that programming languages present some "naturalness":

Hindle, Abram, et al. "On the naturalness of software." Software Engineering (ICSE), 2012 34th International Conference on. IEEE, 2012. http://macbeth.cs.ucdavis.edu/natural.pdf

Although it is interesting, I think that the data should be normalized for life expectancy. A country with a lower life expectancy will necessarily have a younger population. So instead of 14 years old, it would make sense to show the age of the (e.g.) first percentile of the population.

From a machine learning perspective (i.e. theoretically according to my view) this is useful because source code is highly structured with very complex constraints and tons of data available (e.g. every project in GitHub). This means that machine-learning methods for handling such problems need to be developed. Such methods may eventually be useful in other application.

Now on the applied side (software engineering, programming languages), such methods (probabilistic machine learning and probabilistic/statistical models) can handle uncertainty with a principled way and provide useful tools to software engineers that exploit the amounts of data that is available both in internal and external codebases. This is something that is not 100% possible with formal tools, that usually require some form of human knowledge to be embedded. For example, you will see on the list above, tools that can do autocompletion, others that suggest "reasonable" renamings or others that help migration of source code between languages and all thanks to data.

Hopefully, at some point these methods will be so advanced that they will be able to learn (i.e. trained) from every piece of code that is available online and (e.g.) spot bugs in your code, semi-automatically refactor your code etc.

This is a new and very interesting area in machine learning and software engineering. Anyone interested might also find the following papers interesting too

Hindle, Abram, et al. "On the naturalness of software." Software Engineering (ICSE), 2012 34th International Conference on. IEEE, 2012.

Tu, Zhaopeng, Zhendong Su, and Prem Devanbu. "On the Localness of Software."

Nguyen, Tung Thanh, et al. "A statistical semantic language model for source code." Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering. ACM, 2013.

Campbell, Joshua Charles, Abram Hindle, and José Nelson Amaral. "Syntax errors just aren't natural: improving error reporting with language models." Proceedings of the 11th Working Conference on Mining Software Repositories. ACM, 2014.

Allamanis, Miltiadis, and Charles Sutton. "Mining source code repositories at massive scale using language modeling." Mining Software Repositories (MSR), 2013 10th IEEE Working Conference on. IEEE, 2013.

Movshovitz-Attias, Dana, and William W. Cohen. "Natural Language Models for Predicting Programming Comments." ACL (2). 2013.

Allamanis, Miltiadis, Earl T. Barr, and Charles Sutton. "Learning Natural Coding Conventions." arXiv preprint arXiv:1402.4182 (2014).

Allamanis, Miltiadis, and Charles Sutton. "Mining Idioms from Source Code." arXiv preprint arXiv:1404.0417 (2014).

Naturalize ( http://groups.inf.ed.ac.uk/naturalize/ ) is another related machine learning based tool that suggests appropriate identifier names for Java. Details on the implementation and an evaluation may be found here http://arxiv.org/abs/1402.4182

This is a stopper. I don't want to pollute the filenames with this information. Using a sqlite db or something even simpler would have been more preferable imho

It is indeed a hard problem, but there is some research going on in this area http://people.csail.mit.edu/nkushman/papers/naacl2013.pdf

and the application is definitely worth it, because it lowers the barrier for doing "complicated" stuff for people that are not very computer-savvy

I totally agree with you. todo lists are necessary to reduce cognitive burden. It is absolutely necessary due to the non-linear way that tasks appear in your linear timeline of life..

But maybe Jeff Atwood in his position doesn't need a todo list. He can probably afford to not have a todo list. Or more specifically to have only 2-3 things to do for the next day.

Probably, less "successful" people than him have to do daily a more diverse things, from grocery shopping to coding.

But maybe I am just wrong, tied in my todo-list-mania...

Wow, and we needed research to get to that conclusion! As if someone would have any doubt that this is the case...

I've been using n-grams heavily in my PhD. I like a lot what you are describing. Also a simple extension is to use "cache" n-gram models, that can naively simulate longer-term "memory".

What's fascinating is that in the entropy/information-theoretic sense, instead of using wpm, one could aim for "constant" information rate of reading (as in bits/token). That's really cool as a concept. I don't know if it would work, obviously.

I'd definitely be up to for hacking this up, but currently a bit too busy. Will ping you at some point though!

I'd really like to see this model with an adaptive speed. Same speed for all the words isn't always the best.

Maybe using an n-gram model to predict how probable the next word is, could pass high probability phrases at a faster rate, while slowing down for "harder" words

Comparing WhatsApp to space exploration is unfair. Arguably both of them are equally useful/useless (depends on who you ask). But what about if all (or even half) that money had been spent to (i.e.) cancer research or something that would radically improve people's lives? I think that's the "utility trade-off" question one should ask...

"Programming by example" is becoming an interesting trend these days. VS seems to be the first to implement widely such a feature. This of course, suggests that this small-scale reuse is very common. But could it be a "problem" of modern languages like Java and C#?

I've been lucky enough to attend one of Peyton Jone's talks in person. He is not suggesting to "write a formal spec" before you do research. In contrast, his attempt is to be agile: keep going back and forth in the paper writing as you do research. Only then you will see what's missing.

Essentially, what he is suggesting is actually "agile academic research", compared to the "waterfall academic research" (do stuff, then write-up a paper with what you've got...)