Risk Assessment of GitHub Copilot

https://gist.github.com/0xabad1dea/be18e11beb2e12433d93475d72016902

by g_p • 5 years ago

363 230 5 years ago

gist.github.com

arkitaip2• 5 years ago

≥ These three example pieces of flawed code did not require any cajoling; Copilot was happy to write them from straightforward requests for functional code. The inevitable conclusion is that Copilot can and will write security vulnerabilities on a regular basis, especially in memory-unsafe languages.

If people can copy paste the most insecure code from Stack Overflow or random tutorials, they will absolutely use Copilot to "write" code and it will be become the default, especially since it's so incredibly easy to use. Also, it's just the first generation tool if it's kind, imagine what similar products will accomplish in 20 years.

foepys• 5 years ago

imagine what similar products will accomplish in 20 years.

People said the same thing about UML and similar tools so I'm not holding my breath.

arkitaip2• 5 years ago

Maybe you are right but where UML created busy work, Copilot will literally do your work for you. I can even imagine a future where management makes it policy to Copilot first to save time and money.

whoknew11222• 5 years ago

With the pace of technological innovation, I'm honestly not sure what a similar product will be able to accomplish in 20 years. It'll be crazy for sure. But I'm worried about today.

This is a product by a well-known company (GitHub) which is owned by an even more well-known company (Microsoft). GitHub is going to be trusted a lot more than a random poster on Stack Overflow or someone's blog online. And GitHub is explicitly telling new coders to use Copilot to learn a new language:

Whether you’re working in a new language or framework, or just learning to code, GitHub Copilot can help you find your way. Tackle a bug, or learn how to use a new framework without spending most of your time spelunking through the docs or searching the web.

This is what differentiates Copilot from Stack Overflow or random tutorials. GitHub has a brand that's trusted more than random content creators on the internet. And it's telling new coders to use Copilot to learn things and not check elsewhere.

That's a problem. Doesn't matter what generation of the program it is. It creates unsafe code after using its brand reputation and recognition to convince new coders to not check elsewhere.

azangru2• 5 years ago

GitHub is going to be trusted a lot more
GitHub has a brand that's trusted more

Consider Google Translate, right? Google is a well-known brand that is trusted (outside of a relatively small group of people that doesn't trust Google on principle). Yet every professional translator knows that the text produced by Google Translate is a result of machine translation, Google or no Google. They may marvel at the occasional accuracy, yet expect serious blunders in the text, and would therefore not just trust that translation before submitting it to their clients. They will check. Or at least they should.

Same with programmers.

piokoch• 5 years ago

I think there is some difference. You don't come across some piece of code by chance, you were actively looking for it, probably there were multiple blogs, SO entries with needed information, one of those sources has to be chosen. You know that this is some random blog post or SO answer given by someone fresh.

Copilot is something different. Code is suggested automatically and, what's the most important, suggested by the authority - hey, this is GitHub, huge project, largest code repo on the planet, owned by Microsoft, one of the most successful company ever. Why should you not trust the code they are suggesting you?

And that's for starters before malicious parties start creating intentionally broken code only to hack system built with it. Greedy lawyers who will chase some innocent code snippet asking to pay for using it, etc.

tifadg12• 5 years ago

copilot will be interesting in 10-20 years. Now it's an early stage ml driven experiment in a field that hasn't advanced in forever - the strides it'll make will be very gradual, incremental and filled with mistakes along the way.

bryanrasmussen3• 5 years ago

Luckily global warming should be killing me about the same time machine learning is taking my job. That's pretty good planning right there.

tifadg12• 5 years ago

so it all works out in the end!

speedgoose• 5 years ago

You should consider to immigrate to a colder place.

emerongi• 5 years ago

Copilot doesn't seem like the right word. Maybe first year college student with no previous programming experience? Then it would be clear what level of help you are actually getting.

Impressive, for sure. Unclear whether it's a net-positive tool, though.

bla3• 5 years ago

GitHub Excited Intern

baud147258• 5 years ago

fuzzy matching with publicly available code on Github

Hamuko• 5 years ago

GitHub Student Driver

jsjohnst• 5 years ago

Copilot doesn't seem like the right word.

Maybe “autopilot” then (as in the Tesla marketing term, not as in a real autopilot)? /s

Both lull you with a false sense of security which will suddenly and unexpectedly cause you to pay dearly for using it. Interestingly both seem to also get a vaguely similar balance of critics, apologists, and advocates on here.

Sample size of two here, but what is it with companies using %pilot to describe product features which are nothing like the actual appropriate use of the term?

prepend• 5 years ago

me @'ing them several hundred times about ICE

This is such curious behavior to me. Does someone really @ a corporation hundreds of times about anything? Does this have any effect? Should it?

It makes me doubt the rationality of the author’s post if ve truly did this. Although I suppose maybe their use of Twitter is just completely different from anything I understand.

smitop• 5 years ago

You can use some Twitter search filters to see all of them: https://twitter.com/search?q=from%3A0xabad1dea%20%40github%2...

tylersmith3• 5 years ago

It's a way to feel better about yourself without changing any comfortable behaviors. The author still uses Github and the ICE tweets are public Hail Mary's.

infinitezest• 5 years ago

Mostly, this just seemed like a non sequitur to me. Right away, I get the sense that "oh, this author has a bone to pick with GitHub.". Even if it's unrelated to the crux of the piece, I already feel like I'm going to have to take what the author says with a grain of salt.

muglug• 5 years ago

I had not considered the proliferation of terrible open-source code on GitHub. I'd wager that the amount of code in public repositories from students learning to code may outweigh quality code in GitHub.

I wonder if there was any sort of filter for Copilot's input — only repositories with more than a certain number of stars/forks, only repositories committed to recently etc.

tovej• 5 years ago

I wouldn't bet on it, they didn't filter by license after all.

ainar-g• 5 years ago

I'm pretty sure that this conclusion isn't new, but I've come to think that Copilot shouldn't be thought of as a better developer, but merely a quicker one. Obviously its code will be somewhat average, considering that it's been trained on code the only unifying characteristic of which is that it's public.

Something like Copilot, but trained explicitly to analyse the code instead of writing it could be much more useful, imo. Basically a real-time code review tool. There are similar tools already, but I'm talking of something that is able to learn from the actual codebase being worked on, perhaps including the documentation, and giving on-the-go feedback.

rob74• 5 years ago

If you interviewed two developers, one who produces reasonably correct code in a given amount of time, and another one who produces code which is subtly incorrect most of the time, but much faster, which one would you hire?

The problem with your proposal is that it's relatively easy to do what Copilot does at the moment using AI, i.e. guess what code you are looking for and find something that does (or says it does) more or less that. However, which codebase would you use to check against if the generated code is really correct? The same codebase that produced the more-or-less-correct code in the first place?

jazzyjackson3• 5 years ago

I like this idea, given that it takes advantage of how git repos are made of bug-fixes. How many git diffs are out there that update a '=' in an if statement to '=='?

so an AI copilot should be watching out for code I write that looks similar to code that was updated in another repo. It could even use the text from issues to synthesize a suggestion of why your code might cause problems!

eddieroger• 5 years ago

Ultimately, a human being must take responsibility for every line of code that is committed. AI should not be used for "responsibility washing."

That's the whole point, and the rest is moot because of it. If I chose to let Copilot write code for me, I am responsible for it's output, full stop. This is the same as if I let a more junior engineer submit code to prod, but there aren't blog posts about not letting them work or trusting them with code.

onionisafruit• 5 years ago

This articulates some of the concerns I had trying copilot.

I noticed that I ended up assuming the code reviewer role when I was trying to write code. Context switching between writing and reviewing felt unnatural.

I also think I am less likely to spot a bug than I am to avoid writing it in the first place. Taking the off-by-one error in the last example. I don't think I would have made that mistake, but if copilot had presented that code block, I probably wouldn't have noticed the error either.

generationP2• 5 years ago

The Easter Date algorithm was probably someone implementing an algorithm from the Wikipedia ( https://en.wikipedia.org/wiki/Date_of_Easter#Anonymous_Grego... ) without bothering to understand it (because honestly it's not a very interesting problem). No wonder it's uncommented.

As long as the AI just regurgitates lines from repositories like a bad undergrad cheating on his homework, CS jobs should be safe.

The fact that it has picked up the GPL might not mean that much -- it might appear in dual-licensed projects.

prionassembly• 5 years ago

This is something I don't get. You're supposed to be able to integrate BSD-licensed (or even Public Domain) code into GPL works, right? The fact that something shows up in GPL code means what exactly?

This is like: there are scholarly books that quote extensive from original philosophers -- long, third-of-a-page quotations. Still I should be able to quote something in its original language (translations may be copyrighted) copying from the derived work. Copyrighted work is not supposed to be able to poison noncopyrighted work it originates from.

skohan7• 5 years ago

As long as the AI just regurgitates lines from repositories like a bad undergrad cheating on his homework, CS jobs should be safe.

They said the same thing about Chess and Go.

generationP2• 5 years ago

The chess "job market" seems to be doing pretty well these days, at least based on recent publicity.

Tainnor2• 5 years ago

Chess and Go are closed systems, the whole knowledge about how to play them well can in theory be derived just from a few basic rules. The same is not true for programming.

Plus, in theory Chess can be solved by exploring the whole solution space (which is finite, even though insanely large) and heuristics can make this practical by reasoning about which branches can probably be cut off. At that point, having more and more processing power and memory helps make the task feasible.

Not that I want to downplay these achievements, they were certainly very significant, but it's still entirely different from "solving programming" (whatever that means).

user-the-name2• 5 years ago

The fact that it has picked up the GPL might not mean that much -- it might appear in dual-licensed projects.

Github have stated that Copilot is trained on all public code on Github, regardless of license. It very trivially follows that it has been trained on a lot of code that is explicitly GPL single licensed. We don't need to do any guessing here.

eCa• 5 years ago

Re license washing:

This may or may not suddenly become a huge legal liability for anyone using Copilot.

And if it doesn’t, can’t Copilot also be used to license wash for people not using it, but claiming to be?

tjpnz• 5 years ago

As a code reviewer, I would want clear indications about which code is Copilot-generated.

I would like to see this tracked behind the scenes. At any time I should be able to get Copilot to spit out a list of suggestions I've accepted. I should be able to generate this report for the lifetime of a project.

jazzyjackson3• 5 years ago

It's kind of funny that this whole thing is built on top of git but uses none of the features, such as git blame. Instead of being an auto-complete, I would want co-pilot to be a git contributor, making pull requests that improve my code.

smusamashah• 5 years ago

Copilot doesn't seem any better then Tab Nine. Tab nine is GPT-2 based, works offline and can produce high quality boilerplate code based on previous lines. It can also generate whole methods which when work seems mind blowing but they are not always correct. Most suggestions are usually mind blowing anyway because previously we never had this kind of code completion.

It feels like it wrote the whole line which you were going to write exactly as it should have. But that's all it does. And it seems like Copilot is the same but on much larger scale and online.

tylersmith3• 5 years ago

I'm a long time TabNine user and fully agree. It's cool that Copilot is bringing the idea to more people but TabNine is the better product.

kozikow2• 5 years ago

People look at the beta release of the software, and interpret flaws in an early version like a critical fundamental problems.

I am pretty sure the new releases will contain features like better software license handling (e.g. 3 levels for types of licenses - permissive, copy-left, hardcore copy-left), trust score for snippets, possibly some validation of the code for some languages.

tyingq3• 5 years ago

I'd be more optimistic if the beta were crafted with the idea that it might have issues. It so, it would likely have some way of gathering feedback on suggestions that was a little more nuanced than just accepted/rejected.

skohan7• 5 years ago

This could actually be interesting. If it tuns out that copy-left based code completion is better than other options, it will create a strong incentive to spread it.

only_as_i_fall• 5 years ago

Does Microsoft actually have a repository of permissive licensed code though?

My guess would be that some significant portion of github code published under a permissive license is actually licensed improperly.

Working that out at scale seems intractable, but maybe the training set doesn't need to be as big as I'm assuming.

nullc5• 5 years ago

There is still something to be said for making a reasonable effort. There are no guarantees in the world.

TeMPOraL• 5 years ago

People look at the beta release of the software, and interpret flaws in an early version like a critical fundamental problems.

Maybe because they realize the flaws are fundamentally inherent in the very core of the product. They're using a GPT-3 derivative here. DNN models are not the right tool for this job.

kozikow2• 5 years ago

Why following wouldn't work for licenses: Train 3 models:

1. Only permissive licenses - Only include in the training set repos with permissive licenses - MIT, Apache.

2. Copy-left - Step 1 + GPL, excluding AGPL and other "hardcore copy-left licenses".

3. All - Include all code, even unlicensed and AGPL.

User can choose which version they prefer based on profile of their project and their company? Majority of github repos have LICENSE, so it doesn't seem implausible?

_fat_santa• 5 years ago

I think the use case for Copilot is a bit misunderstood. The way I see it you have two types of code:

1. Smart Code: Code that you honestly have to think about while you're writing. You write this code slowly and carefully, to make sure that it does what you need it to

2. Dumb Code: This is trivial code, like adding a button to a screen. This is code you really don't have to think about, because you already know exactly how to implement it. At this point your biggest obstacle is how fast can your fingers type on a keyboard.

For me Github Copilot is useless for "Smart Code" but a godsend when writing "Dumb Code". I want to focus more on writing and figuring out the "Smart Code", if I need to throw a form together in HTML or make a trivial helper function, I will gladly let AI take over and do that work for me.

skohan7• 5 years ago

I think you just described which coders will be replaced by AI first

gonzo41• 5 years ago

Or you're describing how you could put copilot in a box and make a really good low code gui programming solution where the complex stuff is good old complex code.

wongarsu• 5 years ago

I don't know, we somehow manage to replace point-and-click GUIs for placing buttons (Windows Forms etc) with Frontend developers writing elaborate code to achieve the same result in HTML/CSS. Productivity is far from the first priority for frontend development.

pydry• 5 years ago

Did autocomplete replace any coders?

If not, why should copilot?

boxed• 5 years ago

You still have to describe to co-pilot what you want. So that doesn't make much sense. You should work on a higher level of abstraction then. If you aren't, why not spend a few minutes writing some functions instead of generating tons of unmaintainable boilerplate with co-pilot?

dhagz• 5 years ago

Personally, I'd rather use snippets or some form of "dumb" code generation over an AI to generate the "dumb code". Sure, I'll probably still have to do some typing using those methods, but it's still less than if I were doing it all by hand.

spockz• 5 years ago

This is basically the same argument that was made against required boiler plate in Java. “Your IDE can just generate that for you!” (And in sufficiently advanced cases, also keep ik it up to date.)

Imho, it is just an argument for making better languages and libraries. (These libraries will also make it easier to use with copilot.)

wpietri9• 5 years ago

Exactly. The reason we aren't all sitting around hand-writing assembler is that programmers look at tedious processes and find higher-level abstractions that allow us to do more work in less time.

Once we spot a tedious common pattern, we should be finding ways to DRY it up. Configs, libraries, frameworks, DSL, tools, and languages are all great ways to do that. Copy-pasting and machine-generating code are short-term thinking in two ways: they focus on the initial creation of the code at the expense of maintenance, and they give up on increasing abstraction, lock the system into a productivity plateau.

sumtechguy• 5 years ago

The code for a 'button type code' is trivial. Most what we used to call wizards handles that bit.

It is what the action of that button is where the real fun comes in.

I once had a project that was a yes/no dialog. Two buttons and some text. I had the dialog up and running in under an hour. The action that happened when you pressed yes took 3 months to finish.

throwawaygh3• 5 years ago

This is trivial code, like adding a button to a screen.

UX is probably the most important aspect of most software products. Every software product is either "smart code" or "smart ux". No one pays much for "dumb code with bad UX" except in dysfunctional markets.

Adding a button to a screen should be trivial, and if it's not you need better tools. (As in "a not-horribly-misdesigned language and framework", not as in "giant transformer".)

Deciding where to add the button, its shape, its size, what happens when it's clicked, the text on the button, ... is anything but trivial.

OJFord3• 5 years ago

And once you decide all that, you still have to write:

    <Button ...

or whatever.

somesortofthing• 5 years ago

But everyone does pay for dumb code. Even in the best-written and most efficient codebases, there's still going to be some amount of tedious glue code and boilerplate that you have to write in order to create a functioning product. It definitely would be better to have better languages and frameworks instead of a giant transformer, but the better languages and frameworks don't exist yet while the giant transformer does.

wpietri9• 5 years ago

It's not clear to me how that's better than the traditional solution to generate "Dumb Code", copy-pasting something. And we all know the problems with copy-pasting as a lifestyle.

hrdwdmrbl2• 5 years ago

Yes! Why is everyone so negative about copilot? I think it's a great name for the product. It helps you write, it doesn't write for you. You're still in charge and it can't write the "smart code".

GuB-42• 5 years ago

Generally, a copilot is someone you can trust. The whole point of having a copilot is to reduce my cognitive load. If I am a pilot and have my copilot fly the plane while I do something else, I may be in charge, but I trust him to fly safely and alert me if things go wrong. A copilot is also a licensed pilot, able to do almost everything the pilot does, he is just not in charge.

The article shows that I can't trust GitHub copilot. So I don't think it is a representative name. Here, it would be more like a servant.

brundolf2• 5 years ago

The problem is you still have to go back and read through the "dumb code" to make sure it was written correctly. At a certain point, is that actually faster than just writing it yourself? Maybe a little bit, for some people and for some usecases, but it becomes a much narrower value-proposition.

larodi• 5 years ago

Am I the only one thinking that GPT3 and CoPilot can actually work once trained on properly licensed and properly audited code?

Well it will not be as ubiquitous as having all the github under your fingers, but perhaps is anyway better not to blindly cite the world's source code.

qayxc• 5 years ago

Am I the only one thinking that GPT3 and CoPilot can actually work once trained on properly licensed and properly audited code?

Sadly you aren't. The truth is, however, that models like GPT3 and its derivatives like Codex/CoPilot are by design incapable of ever achieving this.

The only way to generate both correct and secure code is to use a combination of proper specs and theorem provers. Even then this won't help with non-functional requirements, such as performance or platform-dependent resource constraints.

Generative models will always have the potential to yield broken code that doesn't do what you want or contains security flaws even if trained on "proper" code.

If I have to audit the code that CoPilot generates and if the code is as obfuscated as the Easter example, it's probably less useful than it says on the label...

reilly3000• 5 years ago

I would hire copilot to write tests for me, that’s about it. Writing tests can be a drag. It’s really a low-risk proposition to have generated code attempt it. If it’s a usable test, maybe it will catch a bug. If not, then kill it and let it generate a few more.

The expectation is entirely different than producing code. Code needs to be correct, secure, performant, and readable. Failure on any of those fronts can be expensive to disastrous. Nobody can reasonably expect a test suite to catch every bug, even if created by the smartest humans. If a copilot-created test does prevent a bug from shipping it provides immediate value. I could see it coming up with some whacky-but-useful test cases that a sane person might not consider. From a training perspective I would think that assertion descriptions contain more consistent lexical value than the average function signature.

It seems like the ambitious data scientists, product marketers, and managers fell in love with a revolutionary idea about AI writing code, and neglected to consult the engineers they are trying to ‘augment’.

neatze• 5 years ago

My time is spend mostly not on writing code, but thinking what program has to do, testing (including writing unit tests) and understanding errors, I always tell people half joking programming is not about writing code, but an ability to debug it; understanding requirements, errors and bugs is hard, writing code and fixing bugs is relatively easy, in general.

Maybe Copilot 2 will do exactly this; it will generate tests based of half working code, run them and suggest improvements, that would increase productivity by like ~100%, but to me this sounds to good to be true.

wpietri9• 5 years ago

Nope. That's like saying, "I might let a machine write the docs for me."

Good tests are documentation that a computer can verify. Because they explain the meaning of parts of the system, they contain information not available in the code. If you try using ML for test generation, you'll have the same problem you do with GPT-3 prose: it might look plausible at first glance, but lacks coherent meaning.

You'd also end up with one of the problems common in big test suites: poorly factored tests that end up being the sort of expressive duplication that is a giant drag on improving existing code. ML is nowhere near advanced enough to say, "Gosh, we're doing the same sort of test setup a bunch; let's extract that into a fixture, and then let's unify some fixtures into an ObjectMother.

For people looking to get the computer to do the work of catching more things with less burdensome test writing, I suggest taking a look at things like Hypothesis: https://hypothesis.readthedocs.io/en/latest/

cmeacham98• 5 years ago

They didn't say they were going to have Copilot write _all_ the tests. Writing tests for cases you can think of and trying Copilot for the extras doesn't seem like that bad of an idea.

comicjk2• 5 years ago

It would be awful to write every test using Copilot, but there is potential there for a certain kind of test. If I'm writing an API, I want fresh eyes on it, not just tests written by the person who understands it most (me). For example, a fresh user might try to apply a common pattern that my API breaks. Copilot might be able to act like such a tester. By writing generic tests, it could mimic developers who haven't understood the API before they start using it (most of them).

developer2• 5 years ago

one of the problems common in big test suites: poorly factored tests that end up being the sort of expressive duplication that is a giant drag on improving existing code.

I feel like you just described every developer/codebase where mock testing is stupidly enforced. Where every single unit test mocks every single indirect object. 98% of the testing code is just exhaustive setup and teardown of objects not being tested by each test, and then a bunch of conditional checks to ensure that every deeper/indirect method is being called exactly the right number of times with exactly the right arguments and returning exactly the right value. Almost all of the test code is just hacking mock objects. The actual purpose of each test is buried so deep that it's impossible to even understand the business logic being applied.

I hate evangelical "mock testers" with a passion.

nullc5• 5 years ago

It would be a mistake to say that the output from GPT3 lacks coherent meaning. It's not that the output is gibberish, it's that it's too easy to mistake it for a human's work. This means that it's easy to mistake it for something that was created with understanding and intention, when in fact the author was nothing more than a random number generator. The same risk exists for copilot. [--GPT3]

ayewo2• 5 years ago

You claim that:

If you try using ML for test generation, you'll have the same problem you do with GPT-3 prose: it might look plausible at first glance, but lacks coherent meaning.

There is a company in this space of generating "plausible tests" for legacy code bases at very large enterprises (think Goldman Sachs, telcos etc) called Diffblue [0].

They raised funding back in 2017 [1] and it seems their biggest value-add is in creating unit tests for legacy Java code bases that often have little to no unit tests.

Essentially, these AI generated unit tests help a team "document" all known the behaviors of a legacy code base such that when a change is introduced that violates the behaviors covered by the generated unit tests, the tool can alert the team of the potential presence of a regression.

Anyway, they offer a fairly basic browser-based demo of their AI product called Diffblue Cover [2].

Are you aware of them?

0: https://www.diffblue.com/

1: https://techcrunch.com/2017/06/27/diffblue/

2: https://www.diffblue.com/try-cover-browser/

IncRnd• 5 years ago

Writing tests can be a drag. It’s really a low-risk proposition to have generated code attempt it.

If Copilot can't write the correct code in the first place, you really shouldn't expect a proper test to be written by Copilot.

Code needs to be correct, secure, performant, and readable.

Most tests should also have at least three of those attributes. Nobody actually wants their tests to be incorrect, slow, or impossible to understand or modify.

dkersten3• 5 years ago

From the article:

"Oh, and they're both wrong."

"Both look plausibly correct at a glance"

You would end up with tests that look plausibly correct, but test the wrong results.

PufPufPuf• 5 years ago

On the contrary, it might be interesting writing tests by hand and using the AI to produce code. If the tests are good enough for humans, they should be good enough for AI, given that the AI doesn't try to be actively malicious.

nojs• 5 years ago

I think the biggest problem copilot will have in practice gaining traction is that verifying correctness isn’t any faster than writing the code yourself in many cases. The Easter(y) function is a classic example - it would be way faster to write that than to try and verify that there’s no subtle bugs.

Copilot is by design trying to give you something that _looks_ correct without caring whether it actually is - so it optimises for real-looking but subtly buggy code, which is the worst kind of broken code.

mattwad2• 5 years ago

I've been using it and that's 100% correct. If it suggests more than a few lines, i might as well do it myself. However, it has been an awesome Intellisense tool for one-liners.. it can write out the rest of a comment or a simple map/filter method just fine. I don't think it will ever go further than that, but nor does it need to.

visarga4• 5 years ago

Be sure it doesn't generate any GPL licensed code down to the size of a single letter, to be safe. The reaction to GPL inspired snippet generation has been more fierce than I could have imagined, even though usable snippets are so short.

nimbius• 5 years ago

see when I saw the words 'risk assessment' i figured the presuppositional framework of the authors argument wasnt that copilot was legally sound. In other words, i didnt expect to jump straight to the technical validity of the product.

do not ignore the elephant in the room. copilot is stealing code from projects with open licenses.

ShroudedNight2• 5 years ago

I would expect an information security expert to comment on the risks they have a professional background in assessing. More broadly, your comment almost seems to suggest that one should preclude all avenues of criticism beyond whichever singular issue is most "obvious" / "problematic". That strikes me as less than optimal.

mumblemumble5• 5 years ago

Years ago, I ran into a similar problem working on a program that was doing named entity recognition to assist humans with data entry. We found that, for our purposes, there seemed to be no (realistic) accuracy threshold beyond which the tool would save clients money, because double-checking the machine-generated output was inherently more work than doing it by hand.

So we pivoted the product to being something you would run on full auto, for situations where you didn't need a high level of quality. I'm not sure if that option is available to programmers, though.

jpeloquin• 5 years ago

Maybe Copilot could be turned into a context-aware search engine? That is, invoking it would return a list of examples that it thinks do the same thing as what you're trying to do, based on your work-in-progress code.

jtbayly• 5 years ago

I would have thought to pivot to an assistive model in the case you mention, and in cars and in copilot.

Have the machine notify you when it thinks you’ve made a mistake.

intuitionist• 5 years ago

I think my best guess is that this is actually meant to produce broken code, so that Microsoft can sell you additional services (cloud fuzzing?) to find and fix the bugs.

intothev01d• 5 years ago

Yea, that's the major problem. I'd prefer just some sort of inline helper that could point directly to documentation, topics, stack overflow answers that might be helpful for whatever I'm developing. An enhanced "Intellisense" or something. That to me, is better, because ultimately it's up to the developer to place scrutiny on the solution. You basically can't blindly accept the implementation which makes this just a constant code review, of yourself? I dunno. This just seems half-baked

visarga4• 5 years ago

Copilot does approximately that, but instead of showing you code that's written independently it adapts to your variable names and context.

tyingq3• 5 years ago

It also isn't giving you any information on the source(s) of the generated code. Which might help determine how much to trust it, whether it could have licensing issues, etc.

visarga4• 5 years ago

It's probably the only way it would work - to show top matching snippets from training data on request, with links to the source and ideally licensing information, if it can be gleaned automatically. This would also clearly show how much it is copying verbatim and what exactly is its contribution.

The funny part will be when all the human programmers who steal code will get doxed as a side effect. It shine a light on lots of skeletons in the closet.

pessimizer• 5 years ago

I think the biggest problem copilot will have in practice gaining traction is that verifying correctness isn’t any faster than writing the code yourself in many cases.

Humorously, this is a similar problem to the one autonomous driving has. Being alert when something goes wrong randomly is more difficult than being alert all of the time.

slg3• 5 years ago

However in the real world people don't always write bugless code and aren't always alert when driving. Therefore these AI assistants can still have a net positive result as long as they are better than the average performance of a human. Of course three quarters of us probably believe that "I'm not an average programmer so Copilot would only make me worse."

Personally I think the more interesting angle is the trolley problem this creates. People will die in self-driving car accidents and bugs will exist in AI generated code. Those people and bugs are different than the people who will die in human caused accidents and the bugs in human written code. If the number and severity of the results are lessened by the computer, are we willing to forgive the damage directly caused by the AI that falls short of perfection?

elcomet• 5 years ago

It works well for completing lines one by one. I just write a few characters and it can often complete very well the lines.

Of course it will not write complete functions correctly.

ikiris• 5 years ago

I can think of multiple cases where the best result is probably volume of output that looks good without close scrutiny.

corpdrone20212• 5 years ago

I feel like a mix of hand written test cases and copilot generated code might go somewhere, but I think you've got the basic problem sorted out. I'd much rather type an algorithm in from scratch than wrap my head around whatever copilot spits out.

dkersten3• 5 years ago

I had an idea long ago that you basically write unit tests (nowadays I would add property-based tests to the mix too) and a genetic algorithm (best I could come up with at the time, nowadays we obviously have much fancier techniques, as evidenced by Copilot) would come up with code to try and make the tests pass.

I could see Copilot used in such a way. I think the interaction would have to change though: force the user to give it the tests as input, not give it some basic instruction, have it generate code, and then I try to write tests after. The tests should be the spec that Copilot uses to generate its output.

Right now, I'm not excited about Copilot. Like you say, understanding what Copilot spits out is difficult and I suspect more error prone than just writing it yourself (since we often see what we want to see and can overlook even glaring mistakes). I'm also not excited about them ignoring the licenses of the code they trained on. But I can imagine a future iteration that generated code to pass some tests that I could get excited about.

fighterpilot• 5 years ago

I can foresee one niche where this doesn't matter: exploratory ad-hoc data science.

In this exploration stage, total correctness doesn't matter since you're just getting a feel for the data. Copilot might help a lot with the associated boilerplate.

jessmartin2• 5 years ago

I think Copilot-like tools could be excellent for the exploration phase. Marvin Minsky mused on this usage back in 1967:

The programmer does not even have to be exact in his own ideas‑he may have a range of acceptable computer answers in mind and may be content if the computer's answers do not step out of this range. The programmer does not have to fixate the computer with particular processes. In a range of uncertainty he may ask the computer to generate new procedures, or he may recommend rules of selection and give the computer advice about which choices to make. Thus, computers do not have to be programmed with extremely clear and precise formulations of what is to be executed, or how to do it.

From: https://web.media.mit.edu/~minsky/papers/Why%20programming%2...

ForHackernews• 5 years ago

Producing code that kinda-mostly works, very quickly, is the behaviour the software industry optimises for. This tool will help do more of that, so it will be very widely adopted.

Developers who not use this (or similar tools) will not be hired, or only in particular niche domains where correctness matters.

tkiolp42• 5 years ago

Agree. It’s way easier to write a function from scratch than to read/evaluate/fix whatever snippet Copilot throws at me. Replace Copilot with “junior engineer” or “senior engineer that knows more than me” and the result is the same (the junior engineer will probably introduce couple of subtle bugs that are hard to find; the senior engineer would write code in such a way that my mediocre brain won’t understand).

jeltz• 5 years ago

The best senior developers I have worked with knew how to keep code simple to understand for less skilled developers.

TameAntelope4• 5 years ago

"Subtly buggy" is another way of saying "mostly working", and I'll take "mostly working" any day of the week!

gspr3• 5 years ago

You must not be coding anything important.

ohazi2• 5 years ago

More likely: "inscrutably buggy"

It looks to you like it should work, but it doesn't, and you can't figure out why.

That's not "mostly working," that's a frustrating waste of time. It's hard enough to notice when you accidentally swap `i` and `j` -- why would you want to make your life even more miserable by spending your time finding all of the instances where a pattern matching robot has done something similar in an unfamiliar block?

And if you do happen to get "mostly working" code, but only want it to stay together long enough for you to fundraise, you're basically stating that you plan on foisting this technical debt onto the poor sod you happen to hire.

Attitudes like yours are the reason this dogpile scares me.

shinjitsu• 5 years ago

give you something that _looks_ correct without caring whether it actually is

Oh this is going to make teaching intro to computer science sooooo 'interesting'.

It wouldn't be so bad if the students looked at the generated code and understood it, but experience tells me most of them will not.

staticassertion• 5 years ago

Students will always cheat tbh. I doubt this will radically change anything with regards to that.

908B64B197• 5 years ago

I can't wait for coders who used copilot for all coding projects they did. Copy and pasting snippets until it works. No proofs or real exams at bootcamps!

Having offline coding interviews to find Software Engineers will become even more important.

cyanydeez• 5 years ago

just use it to talso write the tests, problem solved!

IfOnlyYouKnew• 5 years ago

It's interesting how the community that used to fervently argue that, say, the sampling of a few seconds of a musical composition is so obviously fair use doesn't extend the same attitude when it comes to snippets of source code.

Indeed I don't remember any of these complaints ever being made about, for example, AI-generated music or images, even though they work exactly the same and were trained on datasets of copyrighted works, both commercial and CC-licensed.

Compared to the manual sampling that DJs and other musicians do, the AI process almost certainly produces only fairly generic code snippets, since they always need a multitude of examples. Some loop-through-file python snippet is a legal risk but five seconds of a Disney song don't reach the level of creativity needed for copyright to matter? That seems strange to me...

ShroudedNight2• 5 years ago

While I think there is some room for retrospective humility, It seems perfectly consistent to believe that, morally, de minimis use of copyrighted material should be considered fair-use while acknowledging that there are practical legal and professional risks for doing so (especially blindly), and that encouraging people to engage in that behaviour in an uncritical manner (especially in the context of engineering) is sociologically reckless.

_ph_• 5 years ago

The problem is, we don't know what Copilot is going to do. Sometimes it reproduces entire files in verbatim. That is certainly a copyright violation. Sometimes it produces whole functions which are more or less taking verbatim from a copyrighted file. The user cannot see the level to which they are "verbatim" and there are also no clear legal guidelines to what is considered violating copyrights and what not.

kiernanmcgowan• 5 years ago

Has anyone tried to generate private keys from copilot? GitHub suppressed these from search a while ago, but it would be an easy mistake to have included them in their training data.

user-the-name2• 5 years ago

The easter code is regurgitated pretty much verbatim:

https://stackoverflow.com/a/58726426

It really is hard to swallow the argument that this is not license violation on a massive scale.

abunuwas• 5 years ago

I see copilot as something that could be useful for generating boilerplate or starting code. Often when using a framework or library there's some boilerplate code that has to be there for things to work fine, so instead of having to the documentation and find the relevant snippets, maybe copilot can do this for you (although they often come with CLIs that do this for you). Case in point is machine learning code, where you often find lots boilerplate setting up the models, the training, etc. Maybe copilot was developed by machine learning devs for machine learning devs?

brundolf2• 5 years ago

This is well-formed and even commented C that sure looks like it parses HTML, and the main function has some useful boilerplate around opening the file. However, the parsing is loaded with issues.

This sounds like a great example of an interview question (where the person is asked to find and fix all of the issues in a chunk of bad code). Unintended usecase for Copilot?

oaiey• 5 years ago

I just read one of the comments below and thought: we should write sql as an AST and not as a string. That solves many problems, not only SQL injection but also the lousy understanding of joins etc.

paulsutter• 5 years ago

If anyone wants to write software to detect the original source of Copilot code, I can explain how to detect variable length unaligned matches using fast hash code lookups (unlimited size corpus)

mattwad2• 5 years ago

People complaining about Copilot should just try it. All the concerns people are bringing up are correct but missing the point. Just think of it as a context-aware autocomplete that can complete more than just properties - it can finish out that one-line comment or map function too! It's really not that intrusive, nor is it going to replace anyone's coding job or even require less programmers. It'll just speed you up a bit, similar to auto-completion. I think at least 70% of the time I take its suggestions, which is good enough to keep using it.

ghostly_s• 5 years ago

Did you even read the article?

jedberg• 5 years ago

My biggest thing with Copilot is that it was trained on all public code on Github, which includes a lot of bad code that people just put up there (like my own code that I wrote a decade ago).

As long as it is keeping track of when people do or do not accept their suggestions, it should get better over time. But in the meantime the best bet it is to treat it like a smart autocomplete, where you still have to at least check that it got it right.

In the future maybe it will be smart enough to be treated like an intern -- trust that the code is right but still verify it yourself if the code is of any importance.

skc• 5 years ago

It's been fascinating reading all the responses to Copilot. It's pretty clear to me now that developer habits are far more diverse than I thought.

For me, Copilot seems like it will be useful in the exact same way that StackOverflow is useful, as a means to pointing me in the right direction for code snippets or apis or techniques that I haven't and don't really want to memorize.

For example, on my current side project I wanted to know how to create a "unique enough" UUID in pure JS.

Copilot would hopefully save me a couple of google/stackoverflow searches as I can very quickly test what it suggests.

I already rarely ever take SO answers as gospel so it's unlikely that I'd do the same with Copilot but I think it significantly increases the speed with which I achieve the same results.

mumblemumble5• 5 years ago

The moon phase example is illustrative in another way.

It's not technically possible to precisely calculate the moon's phase based on time alone. It's an optical effect that is influenced by parallax, so you have to pay attention to location as well. This is, for example, why Eid al-Adha falls on different days in different parts of the world. So the function signature itself is potentially wrong, depending on my needs. I might find that out if I had to do some Googling to finish the function, but (assuming I didn't already know) I'm not sure if that possibility would ever have occurred to me if I were using Copilot.

Copilot can spit out code that's influenced by what others have written. But can it clue you into design considerations like this? Or should we be worried that it is helping us to write code that does the wrong thing with a higher degree of confidence?

daveFNbuck• 5 years ago

The function signature was determined by Copilot. The author just wrote the comment above the function and the word function, then let Copilot determine what the signature was and how it would be implemented.

mumblemumble5• 5 years ago

These use cases where people are running into trouble are increasingly sounding a lot like "What happens if I engage Autopilot and then take a nap?" That's definitely not how the tool was intended to be used, but it's hard to see how you can reasonably expect that nobody would ever do that.

lwb• 5 years ago

Seems like a programming language that optimizes for "codes well with Copilot" could be fairly successful. Things like memory safety, human readability, verifiability, etc.

You could also have some kind of AI-driven testing/verification program -- Copilot and <other program> could go back and forth multiple times until the program is deemed correct and returned to the user.

polynomial• 5 years ago

I haven't used Co-Pilot (though I have apparently contributed to it) but just based on what I know, it doesn't seem like a finished product ready to be launched, rather an experimental feature that you might want to get some more focus group testing on. The idea that anyone would be actually "using" it in production is puzzling, to say the least.

vcryan• 5 years ago

I wonder if people are taking this too seriously. It's a fun toy, not something to actually use in a professional environment.

tyingq3• 5 years ago

It's not being marketed as a toy. So the "too seriously" is perhaps concern about how other people will use it.

dboreham• 5 years ago

This is a fundamental problem with ML and current generation "AI" : it only works in scenarios where a statistical win is acceptable (e.g. ad targeting). It is useless and often worse than useless where you want absolute correctness and false positives are highly problematic (e.g. spam filtering, writing code, not running over pedestrians).

seanwilson• 5 years ago

If Copilot isn't showing exact copies of code it's seen, how is it able to produce code that mostly work? E.g. Copilot code from the article:

    function getPhase() {
        var phase = Math.floor((new Date().getTime() - new Date().setHours(0,0,0,0)) / 86400000) % 28;
        if (phase == 0) {
            return "New Moon";
        } else if (phase == 1) {
            return "Waxing Crescent";
        }
     // etc.

It feels like small incorrect modifications to any of the code here would completely break the function.

I've seen stories and articles written by GPT-3 where it will lose the plot and context on the way - in comparison Copilot doesn't suffer from this as much? How?

quenix• 5 years ago

This is also something that kind of breaks my brain. It's always impressive to see Copilot-"authored" code which seems complex, coherent, (and most of the time, functional), and Google it to zero previous results.

How is it that a glorified statistical machine is able to put blocks of code so well together?

piyh• 5 years ago

Code is much mores structured than English. Code is built around well defined ideas, processes, datastructures. Human language is context dependent, loose, and things like tonality of speech can change the meaning.

It's easier to guess a multiple choice questions when all choices can be generated by intellisense instead of having to look at a dictionary.

gotostatement• 5 years ago

its worth noting that, in the article, the author explains that the moon phase function produced by copilot was actually incorrect

YeGoblynQueenne2• 5 years ago

There are parts of the internet that google doesn't search, it seems. I've just tried searching for a couple of strings from my github repository and I can't find them with google or duckduckgo (I get tons of results but none that's actually my repository or contains my code). To be sure the Copilot generated code is not copied verbatim from somewhere you'd have to search its training corpus. I don't know if that is available publicly.

ohazi2• 5 years ago

I'm surprised that so much of the discussion around Copilot has centered around licensing rather than this.

You're basically asking a robot that stayed up all night reading a billion lines of questionable source code to go on a massive LSD trip and then use the resulting fever dream to fill in your for loops.

Coming from a hardware background where you often spend 2-8x of your time and money on verification vs. on the actual design, it seems obvious to me that Copilot as implemented today will either not provide any value (best case), will be a net negative (middling case), or will be a net negative, but you won't realize that you've surrounded yourself with a minefield for a few years (worst case).

Having an "autocomplete" that can suggest more lines of code isn't better, it's worse. You still have to read the result, figure out what it's doing, and figure out why it will or will not work. Figuring out that it won't work could be relatively straightforward, as it is today with normal "here's a list of methods" autocomplete. Or it could be spectacularly difficult, as it would be when Copilot decides to regurgitate "fast inverse square root" but with different constants. Do you really think you're going to be able to decipher and debug code like that repeatedly when you're tired? When it's a subtly broken block of code rather than a famous example?

That Easter example looks horrific, but I can absolutely see a tired developer saying "fuck it" and committing it at the end of the day, fully intending to check it later, and then either forgetting or hoping that it won't be a problem rather than ruining the next morning by attempting to look at it again.

I can't imagine ever using it, but I worry about new grads and junior developers thinking that they need to use crap like this because some thought leader praises it as the newest best practice. We already have too much modern development methodology bullshit that takes endless effort to stomp out, but this has the potential to be exceptionally disastrous.

I can't help but think that the product itself must be a PSYOP-like attempt to gaslight the entire industry. It seems so obvious to me that people are going to commit more broken code via Copilot than ever before.

bryanrasmussen3• 5 years ago

It seems so obvious to me that people are going to commit more broken code via Copilot than ever before.

Maybe we should use Copilot to commit more open source code meaning that Copilot becomes more and more corrupted and unusable!

of course then we end up with a bunch of bad open source code that will turn people off of using open source.

Gee, I don't think Microsoft really thought this one through.

im_down_w_otp• 5 years ago

This is the thing that made no sense to me about it as a premise. Doing correct program synthesis is really hard even when you have really opinionated and well-defined models of the domain (e.g. the Termite project for generating Linux device drivers). The domain model for Copilot is somewhere between non-existent to so open-ended (i.e. all the diverse code on Github, et al.) as to be functionally non-existent.

A bare minimum baseline validation check for Copilot would be to see if it provides you code which won't compile in-context. If it will, then that means it's not even taking into account well-specified domain model of your chosen programming language's semantics. Which, upon satisfaction, is still miles away from taking into account the domain of your actual problem that you're using software to solve.

The only place where the approach taken, as-is, makes sense to me is for truly rote boilerplate code. However, that then begs the question... how is this machine learning approach more effective than a targeted heuristic approach already taken by existing IDE tooling, etc.?

FWIW, I don't think any of this is lost on GitHub. I think Copilot is more likely a tremendously marketable half-step and small piece of a larger longer-term strategy unfolding at Microsoft/GitHub to leverage an incredible asset they're holding, i.e... practically everybody's source code. The combination of detailed changelogs, CI results (e.g. GitHub actions), Copilot, and a couple other key pieces makes for a pretty incredible basis for reinforcement learning to multiple ends.

juped• 5 years ago

I'd like to think that it's because Copilot is so obviously useless that that part of it doesn't need discussion.

qbasic_forever3• 5 years ago

IMHO they built the opposite of what's actually useful for real-world use. Copilot should have been trained to describe what a selected block of code does, not write a block of code from a description. It could be extremely useful when looking at new or under-documented codebases to have an AI that gives you a rough hint as to what some code might be doing. For example if you select some heinous spaghetti code function, press a button, and get a prompt back that says "This code looks like it's parsing HTML using regex (74.2% confidence)" it could be much easier for folks to be productive on big codebases.

dkersten3• 5 years ago

Now that is a damn good idea!

gspr3• 5 years ago

Sorry for adding just noise, but I think this is the most insightful comment I've read on HN this year. Excellent analysis and idea!

randallsquared2• 5 years ago

I'm not sure I understand how you envision this working, given the underlying technology. You'd have to have a pretty large cache of such analyses to train on, right?

heavyset_go• 5 years ago

But that would require hiring tons of software engineers to label training data accurately.

Why do that when you can just train a GPT-3 model on public repositories and call it a day?

moralestapia• 5 years ago

Something like this would be amazing, particularly for poorly written, obfuscated or even disassembled/decompiled code!

pagutierrezn• 5 years ago

IMO Copilot doesn't need to be perfect. It just needs to be better than copy pasting the code from a colleague or from Stack Overflow.

Houshalter• 5 years ago

Are we seriously going to criticize a tab completion engine because it doesn't perfectly calculate what the phases of the moon are? Can you? I'm amazed it even knows what that means and has a vague idea of what such a calculation should look like, even if it fails.

temuze• 5 years ago

I'm glad the author points out that there are aspects to Copilot that are usable. A lot of other critics haven't been so kind.

Many complaints about Copilot remind me of the old Louis CK sketch where people complain about flying: YOU'RE SOARING THROUGH THE HEAVENS IN AN ALUMINUM TUBE. YOUR ANCESTORS WOULD'VE DIED OF DYSENTERY DOING THIS TRANSCONTINENTAL JOURNEY. Let's have some context here!

Sure, it's not remotely close to perfect and it's going to take a long, long, long time for it to get there. But still, there's something about Engelbart-ian about seeing the demo when it works perfectly.

tjalfi• 5 years ago

I think [0] is the sketch you're thinking of

[0] https://www.youtube.com/watch?v=nUBtKNzoKZ4

YeGoblynQueenne2• 5 years ago

Huh, so Copilot was trained on a mix of correct and incorrect code and it learned to reproduce both correct and incorrect code. It's a language model, after all. It predicts the most likely next sequence of tokens. It doesn't know anything about correct and incorrect code. If the most likely next sequence of tokens is incorrect, it doesn't care, it's still the most likely next sequence.

And I guess it makes sense that Copilot was trained in this way, even if it wasn't just a language model. How do you even begin to separate correct from incorrect code on the entire freaking github?

But I think TFA serves best to show the worst way to use Copilot. I haven't tried it but I suspect that it would do a lot better if it was asked to generate small snippets of code, rather than entire functions. Not "returns the current phase of the moon" but "concatenates two strings". That would also make it much easier to eyball the generated code quickly without too many mistakes making it through.

Of course, you could do the same thing with a bunch of macros, rather than a large language model that probably cost a few million to train...

k12sosse• 5 years ago

Can we all just finally realize that copilot existing doesn't prevent you from not using it? You can still write your 'famously better' code. Go on, show us why humans are better.

stitched2gethr• 5 years ago

It seems the more pressing issue is that I will almost definitely question a future pull request to have the author tell me, "well that's what copilot wrote and it looked OK." If I catch it.

I'm all for AI coding assistance, but there's an abstraction layer in between copilot and myself (humans) that scares me a bit. At least an obvious failure prompts discussion and learning.

diragon• 5 years ago

This seems to imply a need for GPL4: its purpose would be to infect code that was written by a bot that learned from GPL4 code. Otherwise this can be used to circumvent all except the most lenient free software licences.

I would say that GPL3 already probably does that, which means that nobody should be using this for actual code (except if it's GPL3). But it might be helpful to be explicit about this.

Risk Assessment of GitHub Copilot

Related Stories