GitHub Copilot loses an average of $20 per user per month

https://www.thurrott.com/cloud/290661/report-github-copilot-loses-an-average-of-20-per-user-per-month

by georgehill • 3 years ago

161 208 3 years ago

skilled• 3 years ago

From yesterday,

GitHub Copilot loses an average of $20 a month per user (https://news.ycombinator.com/item?id=37821756) (45 points | 16 comments)

empath-nirvana• 3 years ago

I would, without hesitation, pay $30/mo for co-pilot. I would even consider paying $100/mo for it.

arriu• 3 years ago

At that rate I’d run my own model

hesdeadjim• 3 years ago

The moment I can train Copilot on my codebase and improve its predictions, I’d gladly pay more than $20. Even now, I find that I can’t live without it simply as a slightly more intelligent auto-complete.

bloopernova3• 3 years ago

Yeah, I really, really want to be able to configure copilot to explicitly include directories to scan in addition to its built-in corpus of knowledge. That would increase its worth to me a lot.

visarga4• 3 years ago

That's what I expected Copilot 2.0 to be, what a bummer.

grogenaut• 3 years ago

I've used some prototypes that tried it. One of the things it does it regurgitate the old patterns you want to stop using and not the new stuff you do. Basically it's a copy pasta tech debt generator.

js82• 3 years ago

That's a problem with all LLMs. They will average what they are trained on, without any reflection what is good and what is not. They shouldn't be called intelligent (AI), they're information meatgrinders.

falcor84• 3 years ago

I've actually had the opposite happen to me several times - of trying to get Copilot to help me fix something I wrote that isn't quite working, and instead of giving me an alternative solution, it just returns my bad code.

lvncelot• 3 years ago

I would love for Copilot to receive lsp information together with the raw code, I think that could already provide it with enough context, especially in strongly typed codebases.

benterix• 3 years ago

I have mixed feeling on this. On the one hand, sometimes it's able to guess correctly and save me a couple of seconds, up to half a minute I think. On the other hand, very often it's completely wrong, so I needed to constantly verify the suggestions and I found out it interrupted my flow and in the end I'd code faster without it.

visarga4• 3 years ago

Maybe you just need shorter predictions, like one single line or 3-4 words

_fat_santa• 3 years ago

I tried Copilot in the early days but stopped using it pretty quickly as it just wasn't that useful in an already established codebase. Often times I'm not writing completely new code but rather implementing a pattern that I've previously defined somewhere else. In this case Copilot just completely falls over because it doesn't know about that existing code. My other gripe is language specific but in VSCode it breaks Typescript hints, often times I'm writing something like `MyCustomObject.` and expect Typescript to help me autofill the field, this breaks with Copilot as it always tries to suggest some other code.

The second issue I could work around but combined with the fact that it doesn't have context of my current codebase, my impression was that it would be a fantastic tool for someone going to school for CS but practically useless in a professional codebase.

stephendause• 3 years ago

It's very interesting to me that you say that because it hasn't been my experience. I am working in a ~20k-50k SLOC codebase with a few common patterns, and it seems to have picked up on most of them. It's not always right, but even when it isn't, I can write a comment above an empty line to nudge it in the right direction.

simonw3• 3 years ago

Copilot pulls in similar code from your current project as part of its prompt.

If you understand how it works you can sometimes lay out your code in a way that makes it more likely to include the relevant examples to get the effect that you want.

There is no guidance or documentation for this at all!

I've been running a hack to help me see the prompts in order to better understand what's going on: https://gist.github.com/simonw/30fec0ac20244b4b921532b63b8b0...

GrinningFool• 3 years ago

Could it be the language and/or the editor's integration?

In neovim w/ a medium-sized go project, I"m finding that it's really good at providing meaningful and at least directionally-accurate contextual suggestions. It seems that the plugin is providing a good bit of context, but maybe the vscode one doesn't?

wrs• 3 years ago

I have found it surprisingly good at some pretty sophisticated “grunt work”. For example, I’m converting some code to TypeScript. Looking at a long and twisty “getSomething” JavaScript function that builds up an object one field at a time, it can complete “interface Something {“ with all those fields, with types! It might duplicate or miss a field sometimes but it makes fewer mistakes than I would.

I’m pretty sure it pulls context from at least the other open files in VS Code, if not the ones on disk. Hard to tell for sure.

It’s kind of like having an eager, well-read, but naive and over-caffeinated intern as a pair programmer.

esalman• 3 years ago

It's amazing and all, but can be so subtly wrong. I let it autocomplete a function for generating time zone offset. It sneaked a minus sign in there so central US offset would be +0600 instead of -0600. My manager caught it during code review. We both are proponents of copilot but it was still embarrassing for me.

gwern• 3 years ago

Sounds like something a few tests would have caught. I've found my GPT-4 coding use to be much more satisfactory when I have it generate a few dozen test cases after finishing (either my code or its code), and then auto-repair based on any errors.

reustle• 3 years ago

Check out https://cursor.so - It does this pretty well without requiring much setup.

linuxftw2• 3 years ago

Copilot is useless compared to chat-gpt4. There's really no comparison. As far as the costs, it seems like transmitting things repeatedly would be more expensive than the chat model where you just transmit things when you want them.

tedivm• 3 years ago

Copilot is chat gpt4, and has been since roughly April. It's built on top of it.

tmikaeld• 3 years ago

To be fair, before they switched to GPT-4 it was pretty bad..

TheRoque• 3 years ago

Wow, I didn't know there, I must have missed the news. I swear I have seen everywhere that it was using this "Codex" model

linuxftw2• 3 years ago

The underlying model might be as good, but how you interface with it seems to deliver wildly different results. I can easily get chat-gpt to give me entire blocks of code. Copilot just trickles out small bits. Copilot is also very distracting because I find myself constantly waiting to see if what it's going to do next is helpful or not, whereas just typing into chat gpt I know I'm going to get something in the direction of what I want.

bhouston6• 3 years ago

I suspect the cost problem arose specifically upgrading to ChatGPT 4 over ChatGPT 3 - right there their costs increased more than 5x. So before it was likely under $5/month on average, but once you changed to ChatGPT 4, it jumped to like $20/month on average.

volkk• 3 years ago

Is that actually true? I remember Copilot-X being containing the GPT4 version and I'm pretty sure that wasn't out back in April. I don't even think it was out this summer

ruffrey• 3 years ago

It’s built on it, and pretrained on most of the open source code on GitHub.

sebzim4500• 3 years ago

I don't see how this can be true. It generates code so much faster than GPT-4 does when you call it through chatgpt.

siva73• 3 years ago

It's not. Copilot is based on GPT-3

aryadee2• 3 years ago

For code completion, it uses an improved version of Codex[1], and for chat beta (part of Copilot X), it uses 3.5-turbo[2].

[1] https://github.blog/2023-07-28-smarter-more-efficient-coding...

[2] https://github.com/orgs/community/discussions/56975#discussi...

simonw3• 3 years ago

It's not GPT-4, it's gpt-3.5-turbo (or a variant there-of). Source: I'm sitting in the audience of a talk about it at AI Engineering summit right now, the speaker confirmed it as gpt-3.5-turbo, switched from Codex.

elil17• 3 years ago

Constant transmission is what makes it so useful, though

zarzavat2• 3 years ago

I disagree. I have both but I use Copilot a lot more. Editing the results of ChatGPT takes as long as it would take for me to write the code myself. Whereas Copilot is autocomplete, you can choose whether to accept it or not line by line.

prattcmp2• 3 years ago

Are you using ChatGPT 4 or 3.5? GPT 4 is phenomenal.

swexbe• 3 years ago

They're trying to corner the market and betting on the running costs of these models going down substantially in the future.

rgbrgb• 3 years ago

I think it’s a good bet based on watching inference speed of llama.cpp consistently improving and model ability / size on a similar trajectory. I expect there’s similar room for optimization (probably more) with hosted models. If you don’t care about code/model privacy (I do but it seems like most don’t yet at least) there are even more batching/caching tricks to engineer.

cabirum4• 3 years ago

How came that usually privacy-minded hn crowd uses run-by-someone-else models so cheerfully and carelessly?

You do realize any data sent to MS/openai/anyone will be stored forever and probably be used against you in the (near) future?

thsksbd• 3 years ago

Todays HN is as interesting as it was back in 2011. But it is not privacy minded like it once was (was it ever?) at all.

yjftsjthsd-h2• 3 years ago

Are you sure it's the same people? That said, even if it is, there's something to be said for intentionally sharing a constrained set of code vs using something that spies on you in real life.

cabirum4• 3 years ago

Could be different sets of people actually!

However. It will eventually, chunk by chunk, upload the full source code of the app you are writing. Complete with all typos and mis-pastes. Like youtube blocks videos with just a hint of copyrighted music, they may be able to detect use of "patented" algos or something like that and block you/ send a lawyer/etc.

jasonlotito• 3 years ago

How came that usually privacy-minded hn crowd

HN has never been a privacy-minded crowd. It's been a collection of random people, and depending on the threads of topic, you'll get more people speaking out than others. See the Dropbox post from 2007. Nary a privacy concern. HN isn't a single voice.

Why do you think it's a privacy-minded crowd? Because you probably react and respond to similar posts, and you see people with a like-mind there.

How came that usually privacy-minded hn crowd uses run-by-someone-else models so cheerfully and carelessly?

Let's answer this question: they probably aren't doing it carelessly. Consider that Copilot is run by Microsoft, who owns GitHub and your code already. So, they already have the code. I'm not sending them something they don't already have.

Anyways, when I see something I didn't expect to see on HN, my initial thought isn't how it's wrong, but rather, what am I missing? And rather than post an antagonistic question, I instead try to research more to get a better understanding of what I'm missing.

cabirum4• 3 years ago

My own code is on my own gitea instance in my home, exposed via a cheap jump server. My companys code is stored on-premise. Nothing touches github and no, ms does not already have my code.

pjc503• 3 years ago

Indeed. Our company looked at the IP issues and issued a very strong "do not put any company IP in an AI" edict.

Have you double-checked the terms of service? Are you absolutely certain that there's no risk of your copilot use impairing your intellectual property rights in the future?

Given what happened to Unity, how comfortable are you with becoming dependent on copilot and then having the rules change underneath you?

rsynnott• 3 years ago

HN's privacy-minded crowd and HN's AI-obsessed crowd don't have much crossover, I don't think.

esafak2• 3 years ago

I bet there is an age gap; the former are older than the latter.

bloopernova3• 3 years ago

The article says Microsoft wants computers to one day have Neural Processing Units (NPU) like most have GPUs. If that led to faster suggestions for copilot, I'd be all for it.

Right now, a GPU is an NPU. Does anyone how an NPU would differ from a graphics card?

dragontamer2• 3 years ago

GPUs are a SIMD architecture designed for 4x4 float32 matrix multiplications found in video games.

But NPUs (like Googles NPU) are systolic arrays designed for 16x16 or even 256x256 float16 or even int8 matrix multiplications instead.

-------

NVidia builds larger matrix multiplications out of the 4x4 float16 base that a SM is designed for. After all, a 8x8 matrix multiplication is just four of the smaller 4x4 matrix multiplications.

dmurray• 3 years ago

After all, a 8x8 matrix multiplication is just four of the smaller 4x4 matrix multiplications

Isn't it 8 of them? That's why you can't go the other way around and use your 8x8 multiplier to efficiently multiply 4x4s in parallel.

Squarex• 3 years ago

isn't it what neural engine on apple silicon is?

hospitalJail• 3 years ago

Integrated vs Discrete.

0x0082• 3 years ago

I think Meta (facebook) is already working with android phone manufacturers to include chips with specific instruction sets to support the llama architecture.

probably intel is working on the same. I assume the next generation of mobile chips (laptop or phone) will all include special LLM processors.

WithinReason• 3 years ago

If you already have a GPU, adding tensor cores to it is more optimal (less area=cheaper) than adding a separate NPU.

visarga4• 3 years ago

Does anyone how an NPU would differ from a graphics card?

Compare NVIDIA to Groq TSP (Tensor Streaming Processor)

This is how an AI chip should be designed, you write the neural net compiler first, then design the chip. These guys have combined a number of smart ideas in their product: synchronous operation makes memory and network operations simpler to plan and optimise; all software defined memory access, no caches; a simple set of primitive operations that can express every model architecture - so they can simply make it pytorch compatible, no need to write kernels, no need to have 100 versions of 2D conv, for all sizes and shapes.

https://groq.com/groq-tensor-streaming-processor-architectur...

Groq recently demonstrated 240T/s for LLaMA-2 70B. Probably world record, the next best I know is 40T/s at labs.perplexity.ai

https://youtu.be/hk4QGpQAvSY?t=57

bloopernova3• 3 years ago

Very cool stuff, thank you for the new information

tandav• 3 years ago

Does anyone know vscode extension that can use OpenAI API to perform code completions like Copilot?

There are several ChatGPT-like UIs that you can self host and pay only for API and not for ChatGPT plus. For example, I'm using https://github.com/Yidadaa/ChatGPT-Next-Web. It would be nice to use Copilot in the same way.

buremba• 3 years ago

In my own testing, Copilot has much better suggestions compared to GPT-4

0x0082• 3 years ago

continue.dev can complete code as well, but it is not a mere code completion plugin like copilot, it is more like an interactive chatbot where you can edit and talk about parts of the code which you have selected.

for mere code completion there is this: https://github.com/FarisHijazi/PrivateGitHubCopilot

but I haven't tested it.

ntoshev• 3 years ago

You can modify this one to do that - or write your own using the same approach:

https://github.com/danielgross/localpilot

noodlesUK• 3 years ago

I feel like the centralisation of server resources for something like copilot doesn’t really make sense. Many (most) professional developers are working on beefy laptops. If ever there were a case to run these models client side, a software developer’s laptop is probably the ideal place. What are the specs that are needed to run inference on these models?

civilized• 3 years ago

There's a huge range of CPU and memory requirements to run these models. The lower end of the range can be run locally but performance is noticeably worse. I'm guessing this will change as more people are looking at efficiency now. Specialized hardware like Apple's Neural Engine on the MacBook Pro may also help.

KaoruAoiShiho• 3 years ago

A lot higher than the beefy laptop... Even a 4090 can't run it well.

insanitybit3• 3 years ago

I don't think most dev laptops have 8GB+ of GPU memory, which (based on ollama requirements) seems like it's on the low-mid end of the requirements. I've tried experimenting with some local models and:

a) They're much slower on my 6GB laptop GPU

b) The seem to not be as good, functionally

b) I can't make use of larger models

I haven't done more than just some experimentation but I can see how this would make sense to put on a server.

baq5• 3 years ago

MacBooks especially pros are in a very privileged position due to their memory architecture. With some dedicated hardware it may be reasonable to run, perhaps even train, useful LLMs on device.

I guess we’ll see in a year or two. This must be on everyone’s radar now, Apple won’t be the odd man out.

Saline95153• 3 years ago

At 600$+/y subscription fees, it makes sense to own a beefy NUC with 32Gb and a speedy SSD to run the model locally.

pjc503• 3 years ago

I feel like the centralisation of server resources for something like copilot doesn’t really make sense.

For whom? For those selling them it absolutely does, because that's how they get to charge a huge ongoing markup on hardware and electricity costs. That's why everything is a service now.

lathiat• 3 years ago

The secondary problem to the high resource requirements, is that I think once you ship the model people can just basically pirate it right?

I’d be surprised to see the models go the ground breaking commercial AIs shipping but then I’m surprised at how much we’ve gotten open source anyway.

LeicaLatte2• 3 years ago

GitHub’s CoPilot is the most disappointing of AI products out there.

bhouston6• 3 years ago

For some developers for sure, but not all. I am a senior developer with +20 years of experience and I find Github Copilot amazing. It significantly boosts my productivity. I do not expect the world from it, and I do run into limitations, but it helps me tremendously to be productive in languages and tools which I have low/no familiarity.

I think Github Copilot paired with ChatGPT fails the most in idiosyncratic code bases where you are doing maintenance or complex refactors. I think Github Copilot with ChatGPT shines in writing green field code in areas where you are will to use the best practices and the open sources libraries that Chat GPT recommends. Basically, if you align with where Chat GPT recommends to go, then it will help you a lot more than if you try to fight it all the way.

uxp8u61q2• 3 years ago

It's also amazing to write documentation or unit tests. Or repetitive editing tasks that are hard to automate (e.g. "invert this if condition").

pjc503• 3 years ago

Which language? New or maintenance code?

__mharrison__• 3 years ago

If you are an expert (or very good) with your tools, these AI products can save tons of time.

Due to hallucinations, beginners can get knee deep very quickly, but experts can see through this and use these tools will skill.

I don't write as much code as I used to, but I still write code daily and appreciate how they can turn what was a 10-60 minute task into 1 minute of prompting plus 30 seconds of generation/reading to make sure it is valid.

infecto• 3 years ago

Agree with you there. I find it pretty useful especially for the price and where it might evolve in the future.

I am not sure why so many people have hard time getting value out of it.

LeicaLatte2• 3 years ago

To clarify, I have been paying the 10$ since it was released last year. I used it in neovim, vscode and found it helpful.

Cut to few months later with OpenAI API release, so many code completers released in many useful form factors willing to go wherever I code even if its Android Studio or Xcode. The product has stagnated completely.

I do expect it to change significantly in the future though, might go back then.

tkellogg2• 3 years ago

imo it's the form factor that works. ChatGPT is good for doing standalone things like generating a script. CoPilot is good for making incremental changes to an existing code base. Honestly, its REALLY good at it. One of my favorite parts is that it mimics the style of the surrounding code.

nmfisher• 3 years ago

Yeah, I was very disappointed. I cancelled my free trial three days in, it was clearly inferior to the alternative (copy+paste into Claude).

bhouston6• 3 years ago

Thus, the price of Github Copilot is going to go up, 2x to 5x.

For all of the criticism of Github Copilot, for a lot of developers (but not all), the value Github Copilot is incredibly high, much more than $20/month.

The current rock bottom pricing is low compared to the value provided for those users.

As such there is a big opportunity to multiply the price here being charged. Probably an increase between 3x to 5x.

The excuse is because they are losing money, but the underlying reason is that the value it is providing is so high in terms of developer productivity.

insanitybit3• 3 years ago

I'd probably pay ~2-3x, idk about 5x. I'd consider it though. Hopefully they can cut the costs and meet in the middle.

bhouston6• 3 years ago

At some point, the open source models you can run locally are going to start to get competitive. I suspect the cost problem came from upgrading to ChatGPT 4 over ChatGPT 3 - right there their costs more than 5x.

DoesntMatter22• 3 years ago

100 bucks a month I'll gladly pay. It stops me from having to worry about so many annoying coding tasks. I'd maybe consider 2 or 300 a month

baq5• 3 years ago

I’m ok to pay for usage privately and my company is ok to pay $100/mo for unlimited (probably, haven’t asked)

bosie2• 3 years ago

100 per developer?

dist-epoch2• 3 years ago

Mark my words, developers will soon pay $5k per month for access to GPT-7 Copilot.

AnimalMuppet• 3 years ago

Marked. Mark my reply: No, they won't.

For score keeping, let's define "soon". Does "by the end of 2025" sound reasonable?

antoineMoPa• 3 years ago

I feel like part of the productivity is just moved to reviewers. Every once in a while, I have to review large PRs and find out which comment is AI-generated nonsense.

LLMs are really good at fooling us into thinking that code comments make sense.

That's why I prefer if devs don't use copilot, but instead cherry pick good output from ChatGPT.

_cenw2• 3 years ago

This is why I'm concerned about AI. It enables people who don't value their colleagues time to value it even less. Be that code or contributing to a discussion with unedited and false LLM output "as a starting point".

jacurtis• 3 years ago

Earlier this year, we had this one developer who has worked with us for a while (8 months or so). He was never a top developer, but he was average.

Anyway, I have to review all of his code, and I've been doing this the whole time he worked for us. But, starting around the beginning of the year, I started to notice problem after problem coming through his merge requests (MRs). At first I gave the guy the benefit of the doubt, like I said this was unusual for him. He always needed moderate coaching, but the mistake ratio was way higher than usual.

Anyway, I did some basic coaching and whatever. I talked to him about the increase in errors and he said he was stressed with moving to a new house at the time. It made sense, so I decided not to even record the first performance review with HR.

But the problems kept coming and getting worse. It was getting to the point that he had so many problems, I wouldn't even address them all. I would only kick back the top dozen or so. I was noticing that his code had entirely different styling and voice across methods like it wasn't written by him. I also started to see weird lines that were trying to catch unusual edge cases that would never even apply in our scenario. There was one time where there was this insanely complex filter that fed into an insane regex that I couldn't decipher. I asked him about what it is even trying to do and he had no clue, he couldn't even explain the purpose of the line, let alone how it worked. I pushed him on where it came from and he said "StackOverflow". But I reversed searched it and couldn't find it.

We ended up doing performance reviews almost weekly for a while, and now I was formally writing them up. I told him that he was more of a burden on the company at this point. He was offering a negative value. I was getting stressed out because I was spending so long fixing his code and coaching him, that I was working longer hours because of him. Furthermore, I could have fired him and not even replaced him, just taken over his job myself and it would have been less work than this back and forth of trying to repair his work.

Eventually I did fire him. After he left and I talked to the team about him being let go, one of his friends told me that this guy had been using ChatGPT and Copilot for all of his work. It was a secret because our company doesn't allow AI tools in our codebase right now because of compliance reasons. Which is why he would rather say his mistakes were "copypasta from StackOverflow" as he always put it, than admit it was copilot.

That's not to say that CoPilot is bad. But its not ready to replace developers just yet. Even junior developers still need to wrangle and check the work coming out of the AI. And yes, people do notice. Even after this employee polishing the work coming out of the AI model, it still wouldn't work in our codebase.

And to the original point. Yes, these AI tools really are just passing increased burden onto the experienced developers. The juniors are using these tools as crutches to speed things up so they can watch more YouTube videos. But the code gatekeepers who have to defend the application are the ones with increased burden and workloads of fixing the mistakes the AI is generating. I've been saying for a long time that AI will simply separate the good from the bad in software development. For the past 15 years you could make $150k or more a year as a developer that is only capable of producing sub-par code. But those days are gone. The sub-par code is being produced by AI now. So you need to be at least mediocre now.

sureglymop• 3 years ago

I don't think so. It helps increase productivity but it's just too inaccurate and generates too much stupid output. Not that I could do better but it's just not good enough yet.

marcosdumay2• 3 years ago

for a lot of developers (but not all), the value Github Copilot is incredibly high, much more than $20/month.

I really doubt this.

IMO, the jury is still out deciding if the value is above zero for enough people to matter.

simonw3• 3 years ago

From that article: 1.5m users and up to 50% of their code is now being produced by Copilot.

Sounds to me like the verdict on its value is in already.

JohnBooty2• 3 years ago

Are we talking about Copilot in particular or AI code assistants in general?

The value, for me, is extremely high.

My teammates feel the same. Our shared opinion/experience is that ChatGPT 4 is better than Copilot in general but Copilot shines in-editor because it's aware of your project. So we use both in tandem. They mostly use Chat GPT and I split about 50%/50%. (Note: I'm using the Copilot X beta which I believe uses GPT-4)

People say they're "only good for boilerplate code" but well, that's the vast majority of what anybody is writing IMO.

If I need to traverse a tree or list or something, I'm letting AI write that code. Could I write it myself faster? No, and it's going to have an off-by-one error some non-zero portion of the time if I write it. I also find it's superior to e.g. memorizing all 10,000 CSS properties along with all the classes that pertain to Boostrap or Tailwind or whatever.

I see the AI code assistant hate here and it just baffles me. It's so obviously useful to me, and I really can't imagine I'm that atypical.

Edit 1: AI help is especially pertinent if you are a "full stack coder" who is working on everything from database to frontend. Since frontends really multiplied in complexity about 15 years ago, I have not met a single "full stack engineer" who is truly fluent and expert in the entire db->app->frontend stack, because complexity and choice has proliferated at each of those levels.

Edit 2: While most of us are (hopefully) not literally writing tree or list traversals by hand in our actual daily programming lives, I hope my meaning is still clear -- I'm talking about that mundane sort of code, iterating over things, etc.

Klaus23• 3 years ago

I doubt the price will go up that much, if at all. Maybe there will be more expensive tiers, but $20 will probably stay.

Between better quantization, pruning, smaller models through better training/distilling/architecture, hardware price drops and purpose-built hardware, it will probably be much cheaper to run such a model in the future.

tristanb2• 3 years ago

Im not getting $20 of value as is.

thiht• 3 years ago

I happily pay it for $10 a month, but I won’t pay $20. My company could pay it for me though.

bryanlarsen2• 3 years ago

AI is such a huge expectations dichotomy. For those of us used to the continual disappointment that pre-LLM AI was, the current crop of LLM's are amazing, mind blowing things. We start raving about them, so other people take a look expecting that modern LLM's are the greatest thing since sliced bread. They're not quite that, so we get HN comments complaining that AI sucks.

_5ygc• 3 years ago

I think the problem is that they aren’t mind-blowing, but an improvement.

I resent some LLM implementations on principle, but decided to give these code helpers a try. What I found was they’re reasonably bad, and I kept telling them the solution doesn’t work, only to be presented with a little tweak.

So I don’t see the point of outsourcing my thinking, I’d rather remain intelligent and do the search/try/tweak on my own, instead of pretending a half-assed LLM is genius.

That doesn’t mean they don’t have good use cases, or aren’t an improvement on previous tech. But we definitely should stop calling them mind-blowing. Jaron Lanier had long ago predicted we’d willingly downplay human intelligence to pretend AI was… I.

ryanklee• 3 years ago

I think the problem is that they aren’t mind-blowing, but an improvement.

You seem to be doing what GP is pointing out.

GP's claim is that the relative improvement itself is mind-blowing, not that the tech is mind-blowing in an absolute sense.

I tend to agree: much of the detraction hangs on current-state rather than a probable potential-state informed by recent relative advancements.

In other words, many proponents are chuffed because of that potential; not because actuality. Likewise, skeptics are reserved because of the actuality, and not because of potential, for whatever reason (there are at least a few main ones, I think).

finnthehuman• 3 years ago

We start raving about them, so other people take a look expecting…

If that’s what you want to call it.

I see hypemen overpromising and product underdelivering. And when pressed about specifics, attempts to drown queries in jargon or an ass-covering retreat to treating it like it’s just a tech demo not intended to be used for anything ever.

lukeschlather• 3 years ago

It looks like it's capable of anything, so it's easy to point out that it's pretty bad at certain things, but that's an easy way to miss the things it's actually very good at. Also even if it's bad at most things it's still often good enough to be useful.

And it seems likely that with an order of magnitude better hardware it might be good at some things that it seems really bad at now. So yes, it's a tech demo for a lot of things that aren't quite ready, and it's also very useful as it is.

cm2187• 3 years ago

And management consultants solving every problem in the world by adding a “AI” box in their powerpoint flow diagram.

baz002• 3 years ago

They do suck unconditionally. They're like having a mediocre person working with you that you have to verify and validate all the time that sucks up a ton of money and electricity and generally does bad things to the environment.

Edit: I haven't even started on the social and political impact this has either.

IshKebab2• 3 years ago

You missed the point. And you're also completely wrong. Copilot is extremely useful and in general a huge timesaver.

Yes if you tell it to write an entire program it will get it wrong and you'll spend some time verifying things. But that's not a sane way to use it. As a very clever auto-complete it's fantastic. It's also pretty great at getting past "blank page syndrome". Even if what it spits out is wrong it's still helpful to get you started.

YeGoblynQueenne• 3 years ago

> For those of us used to the continual disappointment that pre-LLM AI was, the current crop of LLM's are amazing, mind blowing things.

Aren't you overgeneralising a bit? Not even I would say that CNNs for image classification, or Deep-RL for board game-playing are a "continual disappointment" and they certainly predate LLMs. Are you talking about NLP? Even Neural Turing Machines were quite capable in language pairs with large parallel corpora (and similar linguistic structure).

Basically, what do you mean by "continual disappointment"? What I'm aware of is an incessant hype crescendo that crashing over everything like a relentless wave.

nvm0n24• 3 years ago

Pre-2022 most AI was disappointing not because it was unimpressive but because it near exclusively took the form of blog posts or university papers that announced something interesting but unusable. Either because it was merely an academic curiosity (AlphaGo) or because the tech wasn't being released for public usage at all, and wasn't really easy to replicate externally either, or because there was no obvious way to apply it to normal problems. So we all got used to this constant year-in-year-out stream of "amazing" "breakthroughs" that ended up being a thats-cool followed by a shrug.

RLHF trained GPT3 and then DALLE/Midjourney/Stable Diffusion changed all that. Suddenly AI not only got good, but the field broke loose of the inane and insane obsession with pseudo-safety that had been holding it back. Now the rest of us can use it without dropping $5M on a GPU cluster and hiring a dozen researchers first. AI is no longer a disappointment.

bdavbdav• 3 years ago

I’m not sure about this pre-LLM AI being disappointing take. What are we considering as AI? AI encompasses all manner of fields. The face detection in my iPhone, nest doorbell works fantastically. The DSG in my car learns how I shift, fraud detection in finance is heavily driven by ML and typically very effective. It may not be helpful to you, but product recommendations and targeted advertising are ML driven and incredibly effective. They may not be exciting uses, but they’re incredibly helpful.

fennecfoxy• 3 years ago

Eh I think people's expectations for things they don't understand are always overblown.

One thing I always find funny is the general expectation that machine learning models are both incredibly generalised and designed based on the way biological systems work, but should also be 100% perfect and never be wrong just like a machine and NOT like a biological system, those things are mutually exclusive; even the best, smartest most physically capable humans will still sometimes spill their coffee, yet we expect coffee-bot 2024 not to do this.

Certainly machines can be much better at a task than humans are, but if that tasks requires generalisation then it's still gonna fuck up from time to time.

Machine learning programs may think a dog is actually a cat sometimes, but afaik they ain't ever called their teacher "Mum" yet.

habosa• 3 years ago

So the service costs MSFT an average of $30/user/month. So if they want to make a business with decent margins out of this, they're going to need to charge $60/user/month or more.

That's not too much if it's adding a lot of value but it's also definitely a lot of money. It would be the most expensive SaaS my team pays for (and we have a lot of subscriptions).

jetrink• 3 years ago

Also, that is assuming that the group willing to pay $60/month includes many who use the service less than average. It's possible that only heavy users would stick around after a 6x price increase.

yeck• 3 years ago

As users' codebases grown over time, the average usage of users will probably continue to grow. By a static price increase alone won't be a great model. They need to make their models and infrastructure more efficient first.

Or create a pricing model where the monthly sub covers a certain amount of tokens and then beyond that is pay-as-you-go. Sucks for users, but a scalable model.

baby• 3 years ago

What’s remarkable to me is that when copilot launched there were a lot of people on HN saying it was not good. Now it seems like everybody has adopted it and can’t live without it.

ackbar03• 3 years ago

I have a small cloud based Ai image processing service running that's still running (we did it before it was cool). The processing is currently done locally on a computer in my partners closet, but scaling it to that point, whilst ensuring it runs smoothly for multiple users with reasonable processing times was no joke.

ChrisArchitect• 3 years ago

[dupe]

throwaway4good• 3 years ago

Maybe the guys doing DDoS attack on Google should give MS/Github a go?

Gijs4g• 3 years ago

Interesting. What does this say about GPT-4? How much do they lose per User per Month?

monocasa2• 3 years ago

Doesn't copilot slurp up oodles of the code in your workspace for non corporate accounts? Is that data acquisition maybe worth the cost to Microsoft?

I know they have github, but tiny scripting glue tasks like copilot excels at aren't the best represented on GitHub.

jrockway2• 3 years ago

I use Copilot in Emacs and it doesn't send Github any data other than my prompt.

daft_pink2• 3 years ago

Do you specifically activate it?

The main functionality I get out of copilot in vscode is autocomplete on steroids, so it's getting all code as I write it. I don't use the question based generator most of the time.

monocasa2• 3 years ago

The vast majority of people using copilot aren't using it that way.

siva73• 3 years ago

I've done my own startup calculations for this. You shouldn't provide a heavily used openai wrapper below 30$/month/user. That will be likely a loss leader. At 46$/month it starts getting profitable for me.

Saline95153• 3 years ago

This is not surprising, the chat function is mind-blowing. It knows about your code, it is helpful, explains concepts, and is rarely wrong. It is the first time I'm genuinely impressed about an AI tool, and makes learning new frameworks very easy.

SadWebDeveloper3• 3 years ago

I would gladly pay for copilot 5 USD a month (its an specific use case IA), but the current price point 20 month is incredible expensive it only add costs if the developer you are hiring requires it to properly function... thats the only value it adds.

IMHO If you are a code monkey copilot makes you a 5x developer, 10x if you have 5+ years of experience on the business

Maybe selling tokens/search could be benefitial for copilot, like only pay for what i use will be an interesting approach.

WillPostForFood2• 3 years ago

$20 a month is incredibly expensive? If it saves 15 minutes over a month, it pays for itself (assuming your work is ~$100/hr). If it can provide any positive value, it is going to be worth more the $5 a month.

SadWebDeveloper3• 3 years ago

Maybe if you live around the bay area where you get paid absurd money by the hour for a job that can be done overseas for 10 bucks, outside or the rest of the world that 100 usd is per week tops, thats way too much i would pay for service that only barely helps my in my day to day...

If you need to constantly use chatgpt/copilot to do your job properly, maybe its time to learn something else because your job will be automated sooner or later, what will be more cheaper a 100/hr or the 20/month service.

zackmorris• 3 years ago

Capitalism cracks me up. We live under this fantasy that everything we do can be turned into a profitable business. Even the government should be run like a business, right? Like somehow educating a child for 12+ years should be a profitable venture. Or housing the elderly. Or providing a basic service like water, sanitation or electricity that everyone uses at about the same rate. Surely we can stand on each other to turn a buck.

I think that AI will be the biggest loss leader ever invented. We'll buy a robot one time for $10,000 that will perform $1 million of mental and physical labor over its lifetime. Except nobody will pay for that $990,000. There will just be this expectation of something for nothing, so the anticipated payment never comes. And it will happen so quickly, so completely, that we'll wonder how it was ever possible to pay people to do all this free stuff.

No, the GitHub Copilot model of charging for AI services isn't going to scale. People are going to open source AI and own it themselves and stop paying for goods and services. Why would they when the AI can provide everything they need? AI is the beginning of the end of money. The headline foretells the end of the era of artificial scarcity, because capitalism can't compete with free.

esafak2• 3 years ago

It will not be free; payment will simply shift to the hardware and data used to train the model, etc.

hm-nah• 3 years ago

I recently cancelled my GitHub CoPilot license. I prefer learning and co-developing with GPT-4.

aryadee2• 3 years ago

GitHub Copilot is not using GPT-4 (yet)! I'm seeing a lot of people under the wrong impression.

For code completions it uses an improved version of Codex[1], and for chat beta (part of Copilot X), it uses 3.5-Turbo[2].

However, they're claiming to have "early adoption of OpenAI's GPT-4" for Copilot X on their marketing page[3], which is confusing/deceptive. To be fair, it's still in beta, but they should state outright what model you currently get if you sign up for it.

[1] https://github.blog/2023-07-28-smarter-more-efficient-coding...

[2] https://github.com/orgs/community/discussions/56975#discussi...

[3] https://github.com/features/preview/copilot-x

_cenw2• 3 years ago

If Copilot loses this much, just imagine how much OpenAI is losing on non-paying users using ChatGPT.

Who's gonna pay for this as soon as the VC money runs out?

tkellogg2• 3 years ago

the environment

kyriakos• 3 years ago

Maybe by then compute will become cheaper. I am sure they have some plan for the future regardless if it pans out in the end.

ComputerGuru• 3 years ago

Unless even their paid api usage is subsidized (and it very well could be for all I know) they can’t be losing that much.

But it doesn’t matter, I’m they never would have gotten the valuations they did if it weren’t for the insane hype around the (free) ChatGPT offering. It more than paid for itself just in that regard.

treis2• 3 years ago

They just need to keep it going until GPUs get cheaper. The point is to build/protect the monopoly until that happens. Then costs come down and they start printing an absurd amount of money.

VirusNewbie• 3 years ago

well, and straight caching. If they know that 80% of people ask the same 10,000 questions without a back and forth dialogue, it's not hard to just write a front end for that.

outside1234• 3 years ago

The problem is that they don't have a monopoly. Not hard to see how CodeLlama and other future open source models can't stand-in here for GPT-4 at dramatically lower cost.

toomuchtodo• 3 years ago

https://www.reuters.com/technology/chatgpt-owner-openai-is-e... (ChatGPT-owner OpenAI is exploring making its own AI chips)

They are proving the market to generate demand to invest to vertically integrate, which then drives down costs while revenue remains flat or (hopefully) increases.

adam_arthur• 3 years ago

I suspect many would pay far higher than $20/month to use an intelligent LLM. Just that too many players are subsidizing free use so its non-competitive to charge that much.

Eventually reality will force cost to the consumer towards cost of serving. Cost of serving LLMs will decline alongside this, so maybe the magic number stays at $20

soultrees• 3 years ago

OpenAI is amassing an ungodly amount of though, there’s efficiency efforts that have been made for sure but scooping all that data up is what they need to train for gpt5. Those chat logs are worth their proverbial weight in gold.

tristanb2• 3 years ago

I use Copilot... Or rather I pay for it. I pretty much always end up using ChatGPT4. I honestly haven't had a very good experience with Copilot. It never "learns" anything. It doesn't remember where my imports are, it gives me the wrong paths all the time. Sure occasionally it will write a nice comment for me when I make a new prop, but other than that it's not as good for writing code as GPT4, and its auto suggestions are pretty weak. Wondering if others have different opinions.

jpallen• 3 years ago

A lot of people are commenting about how gpt4 is better than copilot, both in terms of getting it to do exactly what you want and in the quality of response. I've also found this and so I wrote a simple VS Code coding assistant that uses gpt4 and let's you be direct in asking what you want it to do: https://github.com/jpallen/biggles. Maybe it can be useful to some of you too.

It also supports voice, which is a fun way to code!

petercooper2• 3 years ago

FWIW, a GitHub employee just said on stage at the AI Engineer Summit that this claim is "not true" and that it's a roughly $100m ARR product (first keynote speaker in this stream: https://www.youtube.com/watch?v=qw4PrtyvJI0)

prattcmp2• 3 years ago

Revenue != profit

petercooper2• 3 years ago

Correct, but he also explicitly said the reports that it was losing money were "not true". Given that he's a VP at GitHub, I would consider him to be more authoritative on the matter than the WSJ's "multiple sources."

bluelightning2k• 3 years ago

Please don't take my Copilot away!

vinni2• 3 years ago

And it’s free if you’re an educator or a student.

tortoise_in• 3 years ago

Microsoft should only focus on windows at the end of day and keeps things as they are.

ojkelly• 3 years ago

When I first used copilot a few months after it came out, I thought it was wildly undervalued—I’d pay $100/m for the value and time saved.

It’s become such an integral part of my workflow, that until Zed got Copilot integration I couldn’t switch to it.

Of course I’d prefer to continue paying $10 a month.

archibaldJ• 3 years ago

hmm intriguing; I wonder how is Cursor/Tabnine doing in this aspect

or are they suffering similiar lose?

nymalt• 3 years ago

Just one thing I hate about Copilot is how it needs to be updated two times a day.

namelosw• 3 years ago

The problem is that the industry has't figured out how to properly pricing LLM applications yet.

Before LLM, it's perfectly possible to spin up a SaaS on 5$ Digital Ocean VM and charge $4.99 per seat monthly. If you're using low overhead techs like Go and SQLite you might get away pretty far with a decent user base.

But LLM is inheirently costly compared to those traditional apps. No matter if you're calling OpenAI or DIY your own GPU cluster it's gonna be way more expensive. Spin your own GPU might ended to be more expensive because utilization problems and upfront costs.

The subscription model was kind of the silver-bullet for SaaS but it's probably not going to work well in the AI era.

OpenAI, Elevenlabs, Runway, and Midjourney: they have subscription model but the quota is strict and tight. The "unlimited" plan is simply pay-as-you-go.

Early wave of LLM products with unlimited subscription models like Github Copilot and Notion AI are probably pricing way too low. $7 or $10 is way too low to support heavy usage.

But charging $50 might scare most user away because it exceeded people's expectation for SaaS. And probably still end up losing money. And hobby users may ended up paying too much for the core users - that will lead us back to sophisticated pricing tiers like Elevenlabs and Runway.

Are there alternatives? I dunno. Maybe implement bring-your-own-key properly? Like OAuth but for LLMs? It's definitely interesting to see how things will turn out eventually.

GitHub Copilot loses an average of $20 per user per month

Related Stories