GitHub Copilot: First Impressions

https://vladiliescu.net/github-copilot-first-impressions/

by vladiliescu • 5 years ago

125 98 5 years ago

lma21• 5 years ago

I love the fact that it can help write tedious, repetive, and simple logic code blocks! It'll save me time googling basic stuff that I tend to forget.

Will it also learn (i.e. feed GPT) from the code we're writing which it is also helping to write? How do you think it'll learn to deprecate bad practices or evolutions observed in a language (think writing concurrency code in Java 5 vs. Java 9, or any other relevant Programming Language evolutions)?

klohto• 5 years ago

Lol yea, feed it even more licensed code. That’s exactly what we need even more of now.

stared• 5 years ago

Will it also learn (i.e. feed GPT) from the code we're writing which it is also helping to write?

Most likely not. And at least right now Tabnine's selling point is that you do inference locally (so no need to send code elsewhere) and can be trained on such (vide https://www.tabnine.com/tabnine-vs-github-copilot).

dathinab3• 5 years ago

The problem is it produce "more tedious/repetitive but maybe subtitle slightly of code" where we should try to have "less and higher quality code".

lttlrck• 5 years ago

What kind of code? I'm having trouble thinking of anything for the kind of work I do, that isn't catered for by "intellisense". Anything beyond that seems to be impinging on why I actually enjoy programming.

obviyus• 5 years ago

I very recently got access to Copilot, while I was in the middle of learning and playing around with Clojure.

It’s surprisingly useful when you’re not sure about how you want to proceed. E.g. While I was trying to make a simple function for printing all palindromic numbers under 10,000, Copilot inferred from the function name what I was trying and suggested a function using threading macros (something I hadn’t yet come across in Clojure). The result was a much neater affair than what I came up with on my own. I feel it could be a fantastic way to build familiarity with a new language.

Waterluvian2• 5 years ago

I hear you. But I see this as a horrible thing. Now people are going to be further absolved of responsibility of thinking through the problem first and then implementing it after.

Won’t this result in more junk code?

judofyr• 5 years ago

Shouldn’t the question be whether it’s better or worse than copy-pasting random functions from StackOverflow you found when googling?

bsenftner2• 5 years ago

This is the issue I see. Developers will employ logic they don't understand, create systems they don't really understand, and large investments in time and finances will realize absolute failure. The risk of junk code and misunderstood logic grows exponentially with this type of tool. In the hands of a skilled Computer Scientist, it is a godsend, in the hands of the majority of us developers, it will create unmanageable complexity, stress and failure.

playcache3• 5 years ago

This will then fold in on itself. The AI will start learning upon its own generated code, as there is no differentiator from what it wrote before. It has got a leg up from training on lots of human code, some brilliant written, some not, but eventually it will start dog fooding its own creations.

I am honestly pleased I am not using a Microsoft IDE and won't be part of all this crap. I have already started creating new projects on GitLab from now on.

vladiliescu10• 5 years ago

Not necessarily horrible but still, a bit frightening :).

I fear hordes of inexperienced developers, mindlessly clicking accept on the first suggestion, and then on the second, third, and so on, until one seems to work. And the more advanced tools like this get, the harder it'll be to review the code and make sure nothing's messed up.

I guess we'll have to think long and hard about what safeguards to build against these risks.

6gvONxR4sf7o3• 5 years ago

If anything it’ll make hiring harder. We’ll all get accused of gatekeeping for requiring candidates to really understand their output, because they don’t need to know all that junk! Just use copilot and tweak it until it seems to work!

(this what hiring in ML is like)

bastardoperator• 5 years ago

I see co-pilot more as an intelligent suggestion engine or tab complete on steroids versus something solving all of my problems or absolving one of thought. Does pair programming result in junk code?

imdsm• 5 years ago

20 years ago I would download as many projects as I could fit onto the floppy disks I had on me within the time I had access to the internet. I'd scour planet source code, and all the others, and get all those zips. All I'd have is the title to pick from. Then I'd go home, open them up, and pour over the code, understanding it, ripping it apart, re-using it. I learnt so much.

As an educational tool, being able to generate code and then pick it apart is as important as learning to put code together from nothing.

claviska• 5 years ago

GitHub Copilot is a tool that helps you write better, faster, and most importantly, more code.

I’ll agree with faster and more code, but from the many examples I’ve seen, it’s not better.

dkersten3• 5 years ago

Its not even clear to me that writing faster and more code is necessarily a good thing, or, at least, that beneficial. I've said it in a previous Copilot discussion and I'll say it again, but actually writing code is a small part of what I do as a programmer.

I spend much more time figuring out what the requirements even are, refining them, figuring out what that even means in terms of code, figuring out the overall architecture, how it fits in with other systems or existing code, what data formats it uses, how it handles faults, persistence, scale, security. How it interfaces with the outside world (UI or API). Besides the code itself, I also spend a lot of my time on writing tests (which I wouldn't want to pawn off on an AI outside of fuzzing or generating data for property-based tests; unit tests should mirror what the spec dictates and needs to test the correct things) and on writing documentation.

Yes, the code does take up a good chunk of time, but really, its the easy part of my day!

Also, speeding through the code means I'm not thinking about it very deeply. That's when I introduce the most bugs, design flaws or shortcomings that bite me later. I wonder if we'll end up with a situation like the old quote about code reviews: a ten line code review gets a hundred comments/suggestions/questions, a thousand line code review gets a ship it. If much of the code is written for us, will we have the attention span to scrutinize it and understand it deeply? Or will our eyes eventually just glaze over as we go yeah its probably fine, ship it.

b9a2cab5• 5 years ago

This is exactly why I think Copilot (and other "AI writes code for you" solutions) are going to fail. It's harder to read code than it is to write it. That's the opposite of how English works.

the_lonely_road• 5 years ago

CoPilot for a few years while we train it and then in 2037 introducing Microsoft Pilot.

esperent• 5 years ago

Yes but 2027. If that.

ekster3• 5 years ago

For me the AI part of this is all the bad parts. What would really be cool is better search, templating, and best practices at your fingertips in the IDE.

Exact same problem to solve, because it is an important an interesting one, but a totally different approach. Maybe AI assisted somehow, but human curated.

Otherwise, imagine how bad legacy codebases of the future will be when they are full of autocomplete code that nobody understands or cared enough to think through even originally.

bsenftner2• 5 years ago

We should have AI tools to aid software engineers understanding of logic chains, and assorted visualizations like CAD, but for logic and the code creating that logic. And not UML nonsense, but some type of AI tool that begins where Doxygen ends and simply keeps going with various means of aiding the developer's understanding as they construct hierarchical logic systems.

ekster3• 5 years ago

Totally agree. Something that could interrogate and answer arbitrary questions about complex or new codebases would be incredible.

vladiliescu10• 5 years ago

I think we'll have smarter AI shifting through those legacy codebases of the future, not sure if that'll be a big problem.

The way I see this, GitHub Copilot and the like are true next-gen compilers, translating English into code. They'll only get better with time.

softwaredoug• 5 years ago

We’ll also code reviews. Lots and lots of code reviews. Like, all the time. The algorithm will have to be kept in check.

This is a repeated theme of the article. I think it’d be simpler to write the non boilerplaty code, no? Plus who’s going to be excited to have a job as “AI code reviewer”

vladiliescu10• 5 years ago

You know that gif where Gary Oldman yells 'EVERYONE'? :P

I've got a sneaking suspicion that this is what we'll be doing in the future, I see Copilot as a taste of what future compilers will be like.

In a few years I expect to have most of an application built by an AI, with me developing the business/core logic by hand.

dathinab3• 5 years ago

In a few years I expect to have most of an application built by an AI, with me developing the business/core logic by hand.

I can see that too be preferably not with the approach done by GitHub.

dathinab3• 5 years ago

It's worse then that, because the AI is likely to reproduce the kinds of bugs which where overlooked by the source it learned it from.

At the same time code reviews are hard, much harder then writing code.

So making it faster to write boiler code at the cost of harder code reviews seems to not be a good trade of for me.

Bitwit• 5 years ago

What if I slip in various 0-days? If crafted carefully, the sky is the limit. I'll try it and see what happens. I just have to know...

This worries me.

qayxc• 5 years ago

This would be even more difficult to achieve than previous attempts (e.g. in the Linux kernel [0]) due to the fact that an attacker needs to corrupt thousands of repositories that are guaranteed to be part of the training set.

Potential attackers would have two problems: 1) getting malicious checked into many repos and 2) making sure that these repos find their way into future deployed versions of GPT-3/Codex/CoPilot.

CoPilot generates enough vulnerable code as-is [1], so the extra effort isn't even required.

[0] https://www.bleepingcomputer.com/news/security/linux-bans-un...

[1] https://cyber-reports.com/2021/07/14/devsecai-github-copilot...

UncleMeat2• 5 years ago

The number of people who never write a vuln normally but would write a vuln if they were using machine synthesized code has got to be fewer than ten people on the planet.

remram• 5 years ago

Crafting might not be necessary. You might find a vulnerability in a commonly copiloted piece of code, and now you can exploit it in many projects. Better yet, those snippets cannot be updated even if Copilot improves, and there is nothing to file a CVE against either.

playcache3• 5 years ago

GitHub Copilot is a tool that helps you write better, faster, and most importantly, more code.

I don't have a lot of faith in the author's code, if that is there opening statement ("better")

vladiliescu10• 5 years ago

That makes two of us, I don't have a lot of faith in my code either.

blunte5• 5 years ago

The AI coding approach is solving the wrong problem. The problem isn't with the low level detailed work. That problem should be solved by building composable, tested, audited libraries. Legos, if you will.

If the problem is trivial enough that you can completely trust the AI coded solution, then you could have either done it yourself very easily or used a premade solution from a good library or toolkit.

If the problem is not trivial, then you have the outsourcing challenges (which apply to a lot more scenarios than using AI to help you code).

If you are not personally capable of judging the outsourced work, then whether you use AI or type it yourself, you will end up with errors or misfeatures.

If you are capable of judging, then you must pay attention and read/review. So your job shifts from defining the problem and programming a solution to defining the problem and reviewing potential solution(s). Either way, you must focus and think. But again, perhaps you would be better off building a solution composed of known good blocks. <- This should be the future of software development...

Sadly, I think that open source and freedom to (re)invent has worked against us in the long run. If instead of each of us going off and thinking, "I can make a better language/framework", we had built on existing technologies, I daresay we would be further along. To be fair of course, some level of dissatisfaction and divergence would be necessary or we would still be using assembly.

Github does have one thing right though (from a business perspective) - they are making a remedy to a symptom, and in that they can expect longer term revenue than if they actually solved the core problem.

sktrdie• 5 years ago

I think you’re missing what the problem they’re trying to solve is. They’re solving the problem of “writing code faster”

With that in mind I think this problem is totally real and worth solving. If copilot can save me those mundane moments during my day where I have to figure out “how to do this common thing that I already did 100 times” then it’s a win for everyone.

It’s not trying to solve the whole of programming. It’s just a nice tool to let you actually concentrate on the non-automated parts of coding such as: actually translating requirements into code.

zabzonk• 5 years ago

how to do this common thing that I already did 100 times

You don't do it 100 times - you do it once, test it thoroughly, and put it in a library.

jameshart• 5 years ago

Well there’s your problem.

We don’t need more code. In general, code is a liability. A tool that helps create more configurations of the same terrible boilerplate incantations that we already have to maintain in a million code based is just adding to the problem.

donmcronald• 5 years ago

If you've done something 100 times, wouldn't it be faster to copy / paste the implementation you wrote (and know is good) than to rely on something like copilot and have to review / check the solution every single time?

I think what's actually going to happen is that copilot will be successful, but will be a much worse version of poor quality outsourcing. There are going to be people "writing code" that don't even have the ability to evaluate the implementation.

You see the same thing in some industries where the institutional knowledge of the baby boomers is disappearing. No one stays at the same job for more than a few years and you can see people making mistakes with things that are out by a factor of 10x or 100x sometimes. Very often people don't have the ability to grasp the basic concepts of the work they're doing. I think the same thing will bleed over to software development a lot over the next decade.

I also have a huge objection to having any code I write used to develop an "AI" for the benefit of a huge corporation. Do I get a cut? I doubt it. That alone should be enough for people to quit using GitHub. They're training they're own replacement and are too dumb to see it IMO.

citrin_ru• 5 years ago

If it is not something temporary a line of code is written once and then read many time by many people (for different reasons). Over the long run speed of reading more important than speed of writing. If copilot doesn’t help to write easy to read code it would make the problem worse.

bitwize2• 5 years ago

The thing is, there's already a powerful AI technology we can use to solve the "boilerplate code" problem.

It's called Lisp. You know, Lisp, the language for AI written way back when brand-new Cadillac Eldorados with tailfins were still on the road that is self-reflective enough to make writing "programs that write programs" an absolute doddle? :)

Good luck getting bigco to sign off on a Lisp project, though. Even if it is of demonstrably profound practical utility.

You're right -- we have a huge problem with trying to trowel the new hotness (currently, "AI/ML" aka statistics) instead of taking advantage of tools that are highly suited to purpose. Instead of letting us use those tools, bigcos instead subscribe to the "programmer-clerk" myth in which programming is a menial task of mostly rote coding undertaken by minions in legion strength. And this affects not only the tools we use but our processes and professional values.

blunte5• 5 years ago

The underlying language doesn't actually matter as long as you can deliver the toolkit. I do agree that Lisp(y) language(s) may be very well suited, but I don't care.

I just want us to stop reinventing the wheel hundreds of times each year. At least paper shuffling and fax sending took long enough that you could get a coffee and have a chat while it was happening. Now instead we toil over configuration files (which ironically Rails, my daily toolbox, aimed to solve a decade ago).

I should be able to define a few data models and relationships, processes, and some business rules. The rest absolutely should be generated for me. If cars worked like this, we would still be custom building wooden wheels.

drevil-v2• 5 years ago

code reviews code reviews code reviews

So basically shift the cognitive burden on to your coworkers? And what if they are also using copilot hoping that you will sanity check it’s output for them? Tit for tat prisoner’s dilemma and no one even realises they are playing the game..

frabcus• 5 years ago

Presumably copilot will easily do (bad!) code reviews as well, trained on all the world’s PR comments…

ptx3• 5 years ago

Maybe a future version could even write the requirements for the application. And when the users can't figure out how to use whatever comes out, we can have an AI do that part too.

pdelgallego• 5 years ago

That is what I like (at least in theory) of AWS CodeGuru. It helps you to detect bugs and common bad practices automatically in the code review.

kristiandupont• 5 years ago

So far, I am feeling positive about Copilot. I've used it for about a week and it has been useful at times. Like the author, I've mostly found it useful in situations where I am doing something repetitive. I definitely need to look over suggestions carefully though.

I don't think it will go much further than that and I don't know what that would even look like anyway, unless I could actually start discussing architectural decisions with it like I do with a human pair programmer. I guess you could say that this is what the comments are for, so who knows.

losvedir• 5 years ago

I understand some of the legal implications to be regurgitating licensed code verbatim. But, what about this: what if the current Copilot is working out the kinks, and the real product is per-organization models with transfer learning using their repos' code?

At work, we store all our code in our GitHub repo, some public, some private. As-is, I think there's a lot of legal ambiguity around using Copilot, but if all that code just served to teach the model structure of programs and common syntactical constructs, but then it had another layer with our code and its idioms, modules, names, then maybe it would regurgitate our code in a way that's useful and doesn't run afoul of licenses.

I'm thinking of a fast.ai course I did where I took a base model trained on generic image data, and then did transfer learning on top where I fed it labeled images of Go games and Chess games, and with only maybe 100 of those images it learned to distinguish the two with shocking accuracy. As I understand it, the base model taught it how to look for things like lines, corners, contrast, etc, and then it could be easily specialized. Could something similar be the case here?

vladiliescu10• 5 years ago

Yes, I think so too. Only not in the near future, as we don't really have enough computing power to make that feasible. I've written a few thoughts about this here https://vladiliescu.net/github-copilot-first-impressions/#po...

zaptheimpaler• 5 years ago

Microsoft just stole all the code on github to do this. Regardless of what the minutiae of the law say, no one really expected their work to be used this way. Open source code powers a huge chunk of the industry while capturing little value for the maintainers already. Github even explicitly supports a standard format for declaring the license of a repo, which was cleverly ignored.

Here is the relevant section from Githubs privacy policy [1]

6. Contributions Under Repository License
Whenever you add Content to a repository containing notice of a license, you license that Content under the same terms, and you agree that you have the right to license that Content under those terms. If you have a separate agreement to license that Content under different terms, such as a contributor license agreement, that agreement will supersede.

From GPLv2, "When distributing derived works, the source code of the work must be made available under the same license."

------

This is not about technology, it is a legal endrun around using open source code without open sourcing derived work. It is using AI as a form of "license laundering".

"OpenAI" is not open at all. Truly open AI means the code, the data and the model are all open. OpenAI sold the source to GPT-3 to Microsoft, received $1 billion from them in 2019 and does not make most of their work available except behind a highly exclusive, paid API - https://beta.openai.com/pricing/. Its a joke to call that "open". I urge you to read up on OpenAI and look at what the have actually done.

Their plan in the future is to sell access to Copilot, directly monetizing work they stole from others for free:

According to GitHub, “If the technical preview is successful, our plan is to build a commercial version of GitHub Copilot in the future.”

I've deleted all my code from github and hope others do the same. Maybe if some bigger profile project starts doing this, we can start to organize around opposing Pilot and OpenAI.

Others have also pointed out similar concerns - see https://news.ycombinator.com/item?id=27687450 for example.

[1] https://docs.github.com/en/github/site-policy/github-terms-o...

[2] https://beta.openai.com/pricing/

6gvONxR4sf7o3• 5 years ago

It’s a shame that copilot would not be possible without all the zillions of hours of work that went into writing that code, while the authors of that training data get zero compensation for their contribution to copilot (and zero ability to opt out).

deep_etcetera2• 5 years ago

I'm guessing that since there are hundreds of millions of repositories the typical marginal value of someone's contributions would optimistically be on the order of a few dollars. But since the consensus on HN is that they spend very little time actually coding and there is no use-case for copilot, perhaps it worth a lot less.

ahofmann• 5 years ago

It is kind of impressive, that Copilot understood the misspelled "einz" and translated it as "one" ("eins" would have been correct).

vladiliescu10• 5 years ago

Ouch, thanks for pointing this out, my German has a long way to go.

ipaddr3• 5 years ago

This might work for some domains but I'm trying to write less code and abstract where possible.

This writes more code and doesn't help design the application. It's a function autocompleter but it doesn't take my abstractions into account.

IfOnlyYouKnew• 5 years ago

It's good to read a less dramatic take on Copilot. The initial echo chamber of outrage felt rather strange, especially considering the usual attitude when people point out risks in AI systems. I guess everyone else's reading-file-line-by-line loops in python are the epitome of creativity, and being inspired by them is its own category of crime compared to the exact same thing happening in uncreative professions such as photography, music, or writing? And not being hired because the algorithm prefers people who played lacrosse in school is, like, your problem, because waiting to release models until they do not harm anyone would seriously mess with our agile process.

As an aside, I really enjoyed the writing style. The subtle humour is better at signalling competence and friendliness than any CV ever could.

vladiliescu10• 5 years ago

Thank you :)

ptx3• 5 years ago

I guess everyone else's reading-file-line-by-line loops in python are the epitome of creativity

I would certainly hope not, since there is barely any code to write:

  for line in my_file: ...

For even more convenience in common cases, there is the fileinput module[1] in the standard library.

I can see how a boilerplate-generating AI could be helpful in a more boilerplate-heavy language like Java, but a better solution is to use a language that better suits your usecase and lets you express it without the boilerplate.

[1] https://docs.python.org/3/library/fileinput.html

vladiliescu10• 5 years ago

I'd say using a better language is an easy decision to make when you're in a team of one and not depending on any framework-specific features and somewhat harder when you're in a team of more, depending on some framework-specific features, or both.

DantesKite• 5 years ago

Not a single positive comment eh?

Github Copilot and its iterations are the future.

You can complain and whine about what problems are being solved, how it'll affect human developers (making them weaker instead of stronger over time). And to some extent, that's probably true.

But it's still the future and it's coming. It's already here.

solipsism• 5 years ago

Is this your idea of a positive comment? "It's happening no matter what!"

alphachloride• 5 years ago

I think he means a comment in support of Copilot's advancements and not one criticizing its kinks.

leksak• 5 years ago

One thing not mentioned in this article but that's on Github Copilots page is that they imagine it'll be useful for learning how to code.

... or just learning to code

I've done some teaching and my mental model for what is necessary to learn a language and, more generally, learning how to program and my view is that the rudimentary boilerplate-y type of stuff that this tool seems to excel at are mindless to most are essential for beginners as part of their learning.

Any educators here with different ideas or thoughts?

arduinomancer• 5 years ago

To me it seems like copilot only helps with local code, which is absolutely not the bottleneck (it’s maybe 20% of my time)

I spend much much more time figuring out how to convert requirements to code and how to structure it non-locally (as in how it fits into the whole codebase)

Also if you’re concerned with writing clean code, copy-pasting boilerplate everywhere is not the right approach, you have to actually think about interfaces and abstractions

ellen3642• 5 years ago

It’s early days for CoPilot, but I find myself wondering if it will eventually reduce idiomatic use of languages and increase the stickiness of old practices.

In the article, the author included this example:

  alternate_word_mapping = {words[i]: words_in_english[i] for i in range(len(words))}

That line is probably more readable than the Pythonic enumerate:

  alternate_word_mapping = {word: words_in_english[i] for i, word in enumerate(words)}

But it superficially reminded me of a style that I see on Leetcode, which rarely uses Python features like enumerate or dict.items.

CoPilot was trained on GitHub repos and many repos are mini-projects for learning a new language. Does that mean CoPilot will tend to suggest more generic, less language specific, implementations? If it does, will that change perceptions of what’s idiomatic? And will the volume of old code on GitHub influence CoPilot’s suggestions, making us slow to adopt new language features?

falcor84• 5 years ago

I agree in general, but just wanted to add that the pythonic approach would be to avoid the explicit index entirely, with something like:

alternate_word_mapping = dict(zip(words, words_in_english))

vladiliescu10• 5 years ago

Thumbs up, I totally missed this. Didn’t encounter this approach in the suggestions either.

That’s actually one reason why I don’t think it’s current incarnation is a good fit for learning new apis, it’s been trained on a lot of code, not all of it good.

Seems like something a future version might be able to fix, perhaps by training a new layer using just demonstrably ‘good’ code?

ellen3642• 5 years ago

Agreed and an excellent point. Ironically, my example was not nearly as Pythonic as it could have been!

(V late reply as still learning how to keep track of replies on HN. But perhaps you’ll see it anyway.)

ipaddr3• 5 years ago

Can I use this during my leetcode tests. I feel like it would be the perfect use-case

orange_puff• 5 years ago

I’m not a denier. I believe AI will vastly alter the way we write code. But, to be honest, it’s very depressing to me. I think I am someone who selfishly enjoys writing code, not necessarily getting software built. If my job became designing something at a very fine grained level, feeding it to an AI, having the AI write it and then code reviewing the AI, I’d just switch careers. Unfortunately for me, I’m super early in my career so I hope I can make enough money in the next 20 or so years such that I can retire young.

GitHub Copilot: First Impressions

Related Stories