GitHub's Copilot lies about its documentation. Why would I trust it with my code

https://shkspr.mobi/blog/2024/10/githubs-copilot-lies-about-its-own-documentation-so-why-would-i-trust-it-with-my-code/

by edent • 2 years ago

36 77 2 years ago

siva72• 2 years ago

-> GitHub's Copilot lies about its documentation. Why would I trust it with my code

This is the biggest misconception people have about AI. You shouldn't trust a Copilot by default, but more often than not it will be very handy if you know what you do.

mangamadaiyan4• 2 years ago

If you knew what to do, why would you consult a Copilot in the first place?

It's not like GH tells you to use Copilot exclusively for writing boilerplate code. What the OP tried to do was a legitimate use case, wasn't it?

siva72• 2 years ago

-> If you knew what to do, why would you consult a Copilot in the first place?

Because i don't know what i do but have enough experience to know if the Copilot knows what it's doing.

fleischhauf• 2 years ago

I think the same. it very often has some reasonable suggestion, then I mostly need to fix some minor things and it's good to go. overall it definitely helps to reduce the amount of time required to write code

stonethrowaway• 2 years ago

Then it shouldn’t be called a copilot because I very much would trust a copilot by default.

Zambyte• 2 years ago

You don't have to use their brand name, you can call it whatever you want. I similarly like to use the term "ClosedAI" because I find it is easier to understand than the closely named brand.

brookst2• 2 years ago

“One of my intern developers can’t tell me the URL of their own latest blog post, why would I trust a different intern to write code without supervision or review?”

ToucanLoucan2• 2 years ago

If your intern delivered false information or wrote broken code as Copilot is wont to do, at the rate Copilot manages to, you would fire that intern. Especially if they argued with you that their completely made up solution was, in fact, real.

infecto6• 2 years ago

How much of this could be measured as a user using the tool incorrectly? We hear these complaints and while I have definitely experienced incorrect code, they feel more like blips that I overcome in seconds.

edent3• 2 years ago

If one of my interns lied about something I asked them, why would I trust the same intern with literally anything else?

A key skill is learning when to say "I'm sorry, I don't know that."

AI is currently trained to be overly friendly and helpful - to the point of deception. They often "hallucinate" entire software packages. At which point, they become worse than useless.

brookst2• 2 years ago

Is it always a lie when someone is wrong about something?

And is it wise to judge one tool’s quality based on a totally different tool’s ability in a different domain?

Honestly you’re mostly demonstrating why people aren’t reliable.

spiderfarmer6• 2 years ago

When IDEs first emerged in the world of web development, many developers proudly claimed that a simple text editor was all they needed. For them, saying "I'm using Notepad / VIM" became a badge of honor, symbolizing their minimalist approach.

Today, I sense a similar mindset among those who oppose AI assistants in software development. They tend to focus on the perceived limitations of these tools, often failing to see the bigger picture of how AI can enhance productivity. Instead, they quickly latch onto any shortcoming to justify their resistance, missing the potential long-term benefits.

jchw2• 2 years ago

We're talking about chat bot UI elements plastered all over a webpage. I wouldn't want it if it was a human chat.

n4r93• 2 years ago

That's an interesting comparison because it seems to me that there are still a significant fraction of experienced developers using text editors like vi and sublime, without any apparent loss in productivity. Also, I use VS almost all the time but have had to learn how to bypass it when there are errors that aren't flagged up by its particular build process. I wonder if the same will apply in the case of AI assistants - they'll be great for more junior developers who will never need to learn good practises for refactoring, debugging etc... until suddenly they do. Not saying it's a bad thing, just that you may not need to worry too much about "losing out" if you're not on the hype train.

Roark662• 2 years ago

I wonder if the same will apply in the case of AI assistants - they'll be great for more junior developers who will never need to learn good

I can't think of a worse idea than giving copilot to a junior dev... I use chatgpt a lot, but I've got lots of experience. One has to understand how the big picture works very well to not find yourself led down the wrong path.

pocketarc• 2 years ago

without any apparent loss in productivity

I think there might be a few things to consider here:

1. Is it without any loss in productivity? Or are they so productive/skilled that they can still outpace most other devs even without the IDE's additional functionality?

2. Is there a loss in productivity, and they just don't realize it? Just because they can still do a good enough job without an IDE doesn't necessarily mean that they couldn't do a better job as experienced IDE users.

This is similar to the arguments people make against using Rust. "I can code C just fine and never have any issues, I don't need Rust"

It's also similar to this Copilot discussion. The people who say "I don't see any benefit from LLMs when coding", could it be that they just... don't have enough experience using LLMs? We've -just- had that come up in HN[0], where everybody in the comments was showing how they got the answer just fine out of their LLM of choice, while the OP was all about how AI couldn't possibly understand that bit of code.

[0] https://news.ycombinator.com/item?id=41712564

mangamadaiyan4• 2 years ago

What are these long-term benefits?

Also, Vim (and Emacs) and notepad are not even in the same class of editor.

pylua3• 2 years ago

What is the bigger picture? For me it is not with writing the code, it’s a great multiplier for learning and troubleshooting , however.

I do not see how it could fit into my workflow other than how I use it today — a google expert I can talk to about anything.

What is your workflow like?

spiderfarmer6• 2 years ago

As a solo full-stack developer with 20 years of experience in PHP, I’ve found that using Copilot for my latest project (Laraval app) boosts my productivity by at least 50%. The multi-line autocomplete often nails exactly what I’m trying to achieve. I can easily verify its accuracy as I've written the same code multiple times over the span of my career and it is way faster than typing the same lines.

In addition to Copilot, I frequently use ChatGPT to get quick answers or examples, even beyond Laravel's already excellent documentation. It’s also incredibly useful for quickly learning the basics of setting up the infrastructure for my projects, saving me time and effort in the process.

jordanb• 2 years ago

I was one of those guys, but it was Emacs.

Quite frankly, IDEs at the time were really bad. They were slow and clunky. Intellisense didn't always work. People who relied too much on IDEs didn't understand their project's build process and couldn't debug it. The code templating stuff spit out awful code.

I kept using Emacs for a long time because I was more productive with it. I had a lot of customizations and add-ons that made it a dream to browse the codebase and churn out good code.

I finally switched to VSCode some time after it was open sourced because the plugins for my languages had gotten really good, and emacs development felt like it had slowed down and everything was starting to get bitrotten.

cultofmetatron• 2 years ago

AI is. a tool. a power tool for sure but ultimatly it multiplies what the developer brings to the table. I personally have found it of limited use outside a few situations such as knocking out a bunch of boilerplate.

That said, I would keeo this thing the hell away fro junior engineers and I wouldn't want a junior who has used AI his entire career anywhere near my codebase. AI is gasoline and without someone with experience behind the scenes, they are just going to be 10x faster at knocking out unmaintainable spagetti code.

When you run a saas company, code is only ONE factor among many you are maintaining. The other arguable more important one is the domain specific knowledge that exists in the heads of your talent. Give me the nerd that uses emacs with 5+ years of experience thinking though and solving problems without AI. I'll let him have copilot because he'll be the one writing and maintaining the code. More importantly, he'll have teh experience to know when copilot is leading him astray.

ustad• 2 years ago

Are you serious? Did you just put Notepad next to Vim?

spiderfarmer6• 2 years ago

Both "badges" were a thing back then.

Hasu• 2 years ago

If my IDE autocomplete sometimes generated invalid syntax, I would throw that IDE in the trash and never do business with the people who made it ever again.

Why would it be any different with "AI"?

lr4444lr• 2 years ago

Writing the code is not where I spend most of my time: it's in planning, optimizing, and debugging. AI only helps modestly with the second and third of those right now, and it certainly doesn't need to be in my code editor to do that. In fact, I spent a lot of mental cycles when I tried copilot removing potential bugs it was hallucinating into the code in the first place.

edent3• 2 years ago

I was there when IDEs first started becoming popular amongst my cohort. Yes, there were people like that who prided themselves on their hard-earned skills.

But there's a key difference here.

I don't remember an IDE trying to autocomplete something which didn't exist. Similarly, an IDE never pointed me to documentation which it had just invented. If it had, I'd've been one of those people living out my days in Nano.

Roark662• 2 years ago

I'm very optimistic when it comes to AI. I use chatgpt all the time at work and for personal code, but copilot just feels much worse. Perhaps I just gotten used to prompting and guiding (and copy pasting code) to the model in a way that gives me exactly what I want. Would I love an Ide that automated it? Sure, but none does it well!

spiderfarmer6• 2 years ago

This is how I feel about any coworker.

ActionHank• 2 years ago

The single biggest use I've found for LLMs has been search or lookup functionality. Instead of a carefully crafted google search + "reddit" I can just ask a question and get a specific answer with references. Some times it's wrong, generally not.

Runner up is copilot integration in the IDE that I ask to transform data or reorganize lines of code. It's grunt work and having smart automation around it is helpful.

Anything requiring "thinking" generally falls over pretty quickly. For example I've found broader feature code generation has generally been crap or worse and often using APIs that just don't exist. Catch is that LLMs have been sold as thinking machines that will replace people and VCs have thrown all their money into that fire.

P.S. VIM has a growning following and vscode user counts are astronomical, so I'm not sure your point stands.

nunez5• 2 years ago

The single biggest use I've found for LLMs has been search or lookup functionality. Instead of a carefully crafted google search + "reddit" I can just ask a question and get a specific answer with references. Some times it's wrong, generally not.

This is the use case that I struggle with the most.

Let's say that I want to know how to reheat a baguette. If I throw that into ChatGPT, it will give me an answer based on...who actually knows? It could be from the set of responses it found most often in its training. It could be from reddit.

It could be mashing these up into something completely incorrect.

Perplexity and Kagi Assistant are better in that they give you some of the sources they used in generating an answer, but now we're back to a search engine with extra steps.

With a search engine, I can go through each source and know (a) that the answers aren't altered and (b) who wrote the answer and whether I should trust them.

To wit: I once used ChatGPT 4 to find me good restaurants in Houston based on /r/houston subreddit, with sources. (This is the best way to find new restaurants in my opinion.) It gave me five recommendations that (a) were not the best recs and (b) had fake Reddit URLs for sources.

How can anyone trust that?

elicksaur2• 2 years ago

The biggest speed up was when I was refactoring some code and it “learned” the pattern I was moving to. It felt great!

. . . Except it got one so wrong I couldn’t trust it and had to be just a meticulous as I would’ve been in the first place, so that was a net loss of time.

GrantMoyer• 2 years ago

I find IDEs add complexity to development, even today. They're very bad at hiding irrelevant information. There are so many buttons I've never used in Visual Studio that are always displayed, it makes it very visually cluttered and harder to find the buttons I actually need.

And configuring tools through IDEs is very opaque. How do I add a flag to the compiler command line in this IDE? Who knows. How do I change the CMake config? Maybe I can edit the CMakeLists directly, or maybe that will make the IDE blow up, so I have to use some arcane GUI built atop an already arcane text UI.

Plus, you can usually forget about scripting your workflow. Unless the exact workflow you need is built in to the IDE already, you'll be clicking the same sequence of buttons over and over ad infinitum. An IDE is one big complex tool in place of a handful of simpler composable tools, and the only way to extend an IDEs functionality is to make it more complex instead of adding another simple tool or composing existing tools in a different way.

tivert3• 2 years ago

When IDEs first emerged in the world of web development, many developers proudly claimed that a simple text editor was all they needed. For them, saying "I'm using Notepad / VIM" became a badge of honor, symbolizing their minimalist approach.

Whenever some new technology is being sold, someone always tries to delegitimize any criticism by citing some superficially similar criticism of some past technology that was adopted. Or, fairly frequently, they just reflexively cite Socrates's/Plato's OG critique of writing.

Today, I sense a similar mindset among those who oppose AI assistants in software development. They tend to focus on the perceived limitations of these tools, often failing to see the bigger picture of how AI can enhance productivity. Instead, they quickly latch onto any shortcoming to justify their resistance, missing the potential long-term benefits.

So your message is: "Who are you going to believe, the sales pitch or your own lying eyes? You should focus on selling yourself on the new thing."

I think you should listen to the criticism, and try to understand and empathize with it instead of trying to dismiss and disregard it. There's something there. Maybe the people who critique it are the careful and thoughtful developers, and the ones who sing its praises are the sloppy ones who aren't careful enough to see the problems or just don't care (e.g. the kind who see their job as closing tickets not writing working software)?

Maybe this thing will get adopted despite its shortcomings and change everything, like NFTs did, but that doesn't mean those shortcomings aren't important.

nunez5• 2 years ago

Respectfully, no.

IDEs made it easier to navigate a huge codebase with things like autocomplete and one-click refactor. This made it easier to make huge codebases that _require_ these QoL add-ons over time. I think this was a factor that made outsourcing software development (to countries with weak or non-existent worker rights) possible at scale.

In a similar vein, highly paid software engineers are using LLMs to generate more and more of the code they ship.

If highly paid SWEs can write code by shoveling a sentence into Copilot, _literally anyone can._

We seem hellbent on destroying our profession for the sake of "productivity" (i.e. using less of our brainpower to churn our features faster).

But what do I know? I'm just a simpleton that uses vim and writes code the old-fashioned way.

spiderfarmer6• 2 years ago

I'm a solo entrepreneur first and a solo developer second, so I'm looking at this from a very different angle.

eesmith• 2 years ago

Umm, hi? I still use emacs, and I'm one of the top developers in my field. Never figured out how one of those IDE would be more useful for what I do, which is to develop novel low-level algorithms.

Also, neither vim nor emacs are simple text editors.

arp2422• 2 years ago

Things like completion or build integration in Vim and Emacs go back to the 90s. Ctags goes back to 3BSD from 1979.

Aside from the fact that it's not all that comparable, it's just flat-out wrong in its description of events.

VirgilShelton2• 2 years ago

You clearly need to don't understand how Ai currently works. It predicts the next most likely word.

With code it does the same thing and its prediction skills aren't tuned by humans perfectly yet.

hermannj314• 2 years ago

Does it predict the next most likely word or sample proportionally from the space of possible next words? I don't get consistent results, so I assumed it generated a sequence by sampling. I guess I also don't understand how these things work.

VirgilShelton2• 2 years ago

You can read a thorough article on how LLMs work here https://medium.com/data-science-at-microsoft/how-large-langu....

As Open Ai notes here are the limitations of ChatGPT

Limitations

ChatGPT sometimes writes plausible-sounding but incorrect or nonsensical answers. Fixing this issue is challenging, as: (1) during RL training, there’s currently no source of truth; (2) training the model to be more cautious causes it to decline questions that it can answer correctly; and (3) supervised training misleads the model because the ideal answer depends on what the model knows(opens in a new window), rather than what the human demonstrator knows.

ChatGPT is sensitive to tweaks to the input phrasing or attempting the same prompt multiple times. For example, given one phrasing of a question, the model can claim to not know the answer, but given a slight rephrase, can answer correctly.

The model is often excessively verbose and overuses certain phrases, such as restating that it’s a language model trained by OpenAI. These issues arise from biases in the training data (trainers prefer longer answers that look more comprehensive) and well-known over-optimization issues.1, 2 Ideally, the model would ask clarifying questions when the user provided an ambiguous query. Instead, our current models usually guess what the user intended.

While we’ve made efforts to make the model refuse inappropriate requests, it will sometimes respond to harmful instructions or exhibit biased behavior. We’re using the Moderation API to warn or block certain types of unsafe content, but we expect it to have some false negatives and positives for now. We’re eager to collect user feedback to aid our ongoing work to improve this system.

Ref: https://openai.com/index/chatgpt/

jchw2• 2 years ago

I also find their new UI changes devoid of taste. Really? Putting a stupid chatbot button where the notifications button usually is? Fuck you.

I will some day in the near future migrate my projects off of GitHub. It was nice having a centralized place for open source projects for a while, though; a lot more people saw stuff I was working on than otherwise would if I just had a cgit instance running somewhere. But that's okay: the point of me publishing it wasn't for other people to see, necessarily.

And I'm sure eventually there will be another anyway. I certainly remember how Sourceforge went downhill.

bastawhiz• 2 years ago

Is it surprising that asking a coding tool that writes code to manipulate its own settings in a way it was not built to do produces bad output? Can you ask your Google Home to delete your Google account? Or tell Siri to turn off Hey Siri?

redserk• 2 years ago

Siri will tell you how by giving you the exact steps and a link to the documentation. I just tried it on 18.1 Beta.

throwaway314155• 2 years ago

This is specifically the version of Siri that now _uses_ semantic retrieval over learned embeddings of existing corpus of documentation. There's no reason Github couldn't implement something similar (except that it's not a high priority for them to do so).

edent3• 2 years ago

I've literally just said "Alexa - how do I disable notifications?" It answered with accurate information.

If I ask it something it doesn't know, it says "Hmm. I don't know that."

So, yeah, it is surprising that this supposedly advanced AI doesn't have the same level of functionality as a smart speaker.

infecto6• 2 years ago

Its not that surprising. These are two very different technologies. Alexa/Siri point to explicit functionality and the LLM chat in this example is open ended. Luckily I have been using the internet for a few years, I saw the triple dots and clicked those and immediately saw one option was to hide the Copilot bubble.

To each there own, I have been using these tools for a while and have a grasp of what is possible and not.

mangamadaiyan4• 2 years ago

Not really. GH advertises that its Copilot can go through documentation and fetch relevant results. I don't see anything wrong in asking the copilot to fetch any kind of results from its own documentation.

jocoda• 2 years ago

GitHub has forced its new Copilot button on to every page.

Um, no. Not so for me. No copilot at all. Guess we shouldn't trust you either ;)

wokwokwok• 2 years ago

I mean, even if you live coding assistants and use them often (I frequently consult sonnet on random things I can’t remember how to do off the top of my head)…

…you have to admit that this is just outrageously shit.

Come on

You’re sending a response from a help bot to a user with a url in it and you didn’t bother to add a check to make sure the links it returns actually exist.

That’s just lazy. I wouldn’t want to use it either.

I don’t.

Or any product where the developers have been in such a rush to pour AI into their products that they’ve failed to integrate it in a way that is useful and meaningful.

It doesn’t seem like such a big ask.

If you’re integrating AI and you’re not doing the most basic output sanitation… you deserve responses like this.

From github? With their resources and experience? It’s a fuxking disgrace.

infecto6• 2 years ago

It will be interesting to see as years go on how the aging population that get so worked up over LLM coding tools are doing. The new generations are for the most part embracing it with open arms and no discrimination. I see the many flaws in these tool but I still find them useful. Maybe there is a plateau but if they continue to improve these tools can be incredible force multipliers.

is-sam• 2 years ago

Why do we focus on the promise of a technology instead of its state as is right now ? How many time should we be duped before we learn from our mistakes?

The burden of proof is on the new generation, not on old timers. And this is coming from a new generation software engineer.

infecto6• 2 years ago

Who is focusing on the promise? I find them useful in my day to day and I am not a new generation software engineer.

wodenokoto• 2 years ago

I feel like there should be more malice behind a lie.

elicksaur2• 2 years ago

Can it lie if it doesn’t have a concept of true and false?

nunez5• 2 years ago

...and the AI zealots will insist that there is nothing wrong with these systems spouting disinformation like this, as it's just a speed bump on the road to progress.

cosmotic• 2 years ago

Bing chat does something similar. It suggests going to the settings for the unrelated, now-defunct product also called bing chat. These settings no longer exist and there is no way to turn it off.

wesrobin• 2 years ago

This is a bad faith argument, OP is looking for reasons to distrust Copilot. Let's say Copilot returned a correct link. Would OP then decide it's worth trusting Copilot with their code? I doubt it. That's a horrible proof of trust. It really feels like the only outcome of this 'test' is that copilot can prove itself untrustworthy.

GitHub's Copilot lies about its documentation. Why would I trust it with my code

Related Stories