How much would it have cost if GPT-4 had written your code

by yodon • 3 years ago

65 89 3 years ago

ldoughty• 3 years ago

Isn't this akin to: "The cost of an iphone is $500" ? Sure, it's $500 in parts if you just quantify the final working result... but it ignores all prototyping, overhead, and manual work to link together code (e.g. assembling the phone)...

Maybe I'm misunderstanding just how good GPT-4 is at making entire repos of code... or exactly how this tool functions?

starbugs6• 3 years ago

You aren't misunderstanding it. This is not a valid way to calculate the cost of your code.

SketchySeaBeast• 3 years ago

Well, that's a fun exercise, but it's not the volume of code that has value, it's how it's put together.

taneq3• 3 years ago

What, you don’t pay your devs per kilogram of code? ;)

krick• 3 years ago

Volume is measured in liters or m³.

magic_hamster• 3 years ago

It's how much it would cost to produce the text files in your code. I think that most software engineers will agree that writing the code is the easiest part of the job. The hard parts are figuring out the problem, figuring out your data modeling, what is going to work and what it would cost, considering tradeoffs in speed, scalability, maintainability, etc. And, you know, getting the thing to actually run.

There's no doubt that AI is going to be a huge factor in coding. But we will have to change a lot of things for it to outright replace humans. I'm sure some people are already working on those changes...

mackopes2• 3 years ago

I could imagine an LLM that brainstorms all the edge cases, data modelling, algorithm, etc... The final decision will still have to be done by a human, but having a companion LLM that gives you ideas, plays devils advocate and looks for problems would definitely speed up the decision process.

starbugs6• 3 years ago

Unless it leads you astray and that costs more time than coming up with the code on your own in the first place.

chaxor4• 3 years ago

A 'Tree of thoughts' if you will...

alfalfasprout2• 3 years ago

Yeah, but then you need the knowledge, experience, and decision making to act on that feedback. LLMs are also notoriously bad at things like edge cases, etc. and more training data won't necessarily help.

0xr0kk3r2• 3 years ago

The hardest part is supporting the platform for 7~10 years. With those lifetimes, writing code is like 5% of the effort, the rest is supporting it for years and handling all the nasty bugs that show up when thousands of engineers pound on it 24/7. They don't teach you about that in college, and I wish they did, but I doubt anyone would take a class on how to support software even though it's the most important part.

midasuni2• 3 years ago

I’ve got code barely touched in 15 years still in day to day production use.

layer8• 3 years ago

I’ll add that 7-10 years is rather on the short side compared to the systems I’ve been involved with in my career. 20 years isn’t that rare, and the more interesting challenges tend to happen after the 10-year mark.

bradlys• 3 years ago

It’s really only important to the companies that can afford to teach it. If you’ve got a product that’s lasting to stick around for 10 years - it’s probably a cash cow. You can afford to teach engineers at that point.

For a lot of the companies out there that are just trying to get off the ground - all this talk of supporting the system long term is irrelevant outside of major system crashes or whatnot.

BrissyCoder• 3 years ago

The hard part is figuring out other peoples code.

webnrrd2k• 3 years ago

Even all that, difficult though it is, doesn't include the cost if understanding the actual problem, knowing the field well enough to propose a good solution, and coordinating and communicating with everyone the whole way through.

"Every problem is a people problem", and people-solutions are something that AI still doesn't solve.

I'm not saying that AI doesn't have it's uses, it certainly does, but I'm trying to highlight that purely technical solutions very often aren't good solutions to a lot if real-world problems.

worldsayshi2• 3 years ago

When AI truly becomes helpful in the workplace developer jobs will probably change a lot but I think the biggest change will be in the size of the business scope. Most projects I've worked in could easily grow ten times if developers were ten times as efficient.

At the same time I've always wondered why IT problems haven't all been solved comprehensively already because so many problems seem so similar.

I can't really fit those two narratives together.

lumost• 3 years ago

This… is not what people would have said 10 years ago. Remember the idea of the 10x developer? Even at a design heavy firm, 80% of engineers despise writing the design docs.

kaba04• 3 years ago

But we will have to change a lot of things for it to outright replace humans

Like having AI in the first place. I still haven’t seen any example generated by GPT that is not a naive translation (perhaps between languages) of some code already in its training set in one way or another. Which is a cool thing, we get a much more intelligent (though factually less punctual) search engine for code! But we are very far from it replacing any programmer worth their weight.

worrycue• 3 years ago

Everyone is freaking out about AI writing “common code” yet we have reusable code in the form of libraries for decades - and now frameworks. You will never have to write code that does what the library does ever again.

alfalfasprout2• 3 years ago

The hard part of software engineering should be figuring out the problem, tradeoffs, etc.

But the reality is this field has multiple completely different groups of people calling themselves software engineers.

There's those who basically work at consultancies/shops that get paid to push out tons of client projects using a standardized approach and their job is to write a bunch of code. Or those that work on in-house systems that get paid to largely implement pre-scoped work.

Then there's those that are actually designing systems, improving them, iterating on them, etc. You know, actually "engineering".

The fact that coding itself is fairly easy and the proliferation of bootcamps that teach coding (and now ChatGPT) has flooded the market with individuals that know nothing of good software engineering practices let alone CS and systems fundamentals.

I expect that's the group most exposed to AI risk. Pumping out lots of shitty self-contained code is something LLMs excel at.

jbverschoor• 3 years ago

Let's just all print $100 bills.. it can be done for under $5

starbugs6• 3 years ago

This seems to assume that GPT4 can actually generate the code of your repo. Good luck with that assumption.

mackopes2• 3 years ago

It seems that the package calculates the lower bound of how much would it cost to write the same code with GPT-4. It does not take into consideration that even GPT-4 cannot write code at god mode and write it correctly on the first try.

However, it gets interesting when you realise that if writing the code with an LLM is dirt cheap, then 1000 iterations of writing the same code with the guidance of a skilled software engineer would still be cheap and probably faster. I can imagine a world where whole engineering teams are replaced by just one engineer with a code-generating LLM.

wild_pointer• 3 years ago

It's even cheaper to get a monkey to type that code for you. Will probably cost a banana. Chances it will type the code you expect? Similar to GPT-4 for anything remotely non-trivial.

planetpluta• 3 years ago

If a banana costs ~60¢ (depending where you get it!) and GPT-4 is 6¢ per 1k tokens… you’re better off using a monkey for everything > 10k tokens!

KMnO42• 3 years ago

What’s the purpose of putting this on the package index instead of just a GitHub repository?

elbigbad• 3 years ago

Better discoverability and so people could very easily install and run. Git adds more overhead.

KMnO42• 3 years ago

Pip supports installing directly from git anyway.

pip install git+https://...

pmontra• 3 years ago

A lot of conversation is needed to create that code. I guesstimate that the actual tokens are at least 100 times than the ones in the repository, even if one can convince ChatGPT not to explain every choice it makes.

starbugs6• 3 years ago

convince ChatGPT not to explain every choice it makes

I assume this is a major issue, because it tends to be more talky than my ex which is a real feat.

BeefySwain• 3 years ago

Come on everyone, we all know that "most of the work that goes into software engineering isn't writing code", that "it's impossible to create a working system in one shot", etc etc etc, and therefore this tool will not give a realistic idea of what it would cost to build a system with nothing but GPT-4.

No one here (including the author) is under the impression that you can just ask ChatGPT to "write a Unix-like kernel" and get Linux spit out the other side.

That said, I am really curious what it would cost, as a theoretical lower bound, to pay OpenAI to write the Linux Kernel. 10 grand? 100 grand? I have no clue, but I'm glad someone wrote a tool to make it easier to find out!

londons_explore3• 3 years ago

No one here (including the author) is under the impression that you can just ask Chat GPT to "write a Unix-like kernel" and get Linux spit out the other side.

I really don't see such a thing being far off. GPT-4 is already pretty good at writing small modules ("write a disk IO queue for my custom kernel in C"). With a little more work in allowing GPT-4 to test out code it has written and iteratively make changes, allowing it to use debuggers, benchmarks, and sanitizers, allowing it to write its own tools, and then put modules together, I think we could very soon be asking it to "write me a Unix-like kernel".

gwoolhurme8• 3 years ago

It’s that good in your experience? Enough that it is close to writing something as nice as the Linux Kernel we have now? How close is close? Reading comments like these makes me feel like I’m swallowing crazy pills. It legitimately makes me feel like I’m using it wrong. I do Android and iOS native development and some IoT and it gives me really wrong code very often. To the point that I don’t see much difference between chatgpt and gpt4. But you say it’s close to just “write me a unix-like kernel”

kaba04• 3 years ago

allowing it to use debuggers, benchmarks, and sanitizers, allowing it to write its own tools, and then put modules together

That’s a huuuge if whether it can meaningfully reason about such. GPT-4 has improved a lot over 3, but I really wouldn’t call its capabilities reasoning at all. We often under-appreciate human intelligence

lamp987• 3 years ago

as a hobby osdever, this:

I really don't see such a thing being far off. GPT-4 is already pretty good at writing small modules ("write a disk IO queue for my custom kernel in C").

is completely delusional.

chaxor4• 3 years ago

I tend to agree, though with perhaps a much larger tolerance in the number of years "not far off" is. What may be interesting is seeing the emergence of hardware specific bootloader's, kernels and OS, rather than using boot/Linux/etc with it's support for all different SoCs, etc.

Rust embedded code for specific hardware seems like a regression, but it would make things remarkably smaller codebase for specific app deployment.

gwoolhurme8• 3 years ago

Is this that hard of a tool? Why is it a package? Seems like something someone with a few lines of code could do. OpenAI is transparent and fairly open about pricing per token… it seems like it’s just adding bloat.

pixl972• 3 years ago

I am really curious what it would cost, as a theoretical lower bound, to pay OpenAI to write the Linux Kernel

Billions of dollars.

Eh, what? Why in the living hell would it cost that much!

Because the actual Linux kernel cost that much or more. Of course most of these costs were volunteer labor or born by someone testing out a failure on their own companies time. To greenfield an OS kernel as comprehensive as Linux and with as much hardware support as linux would be one of the more expensive human endeavors ever.

But wait, why doesn't Windows cost this much?

Oh, but it did. Of course Microsoft has thrown a huge amount of the costs of on to users and hardware development companies too.

dathinab• 3 years ago

all your mony, as you never would have reached a complient program

maybe with GPT-6 or so, we will see, but not with GPT-4

gwoolhurme8• 3 years ago

The entire premise of this thing is wrong too because what if you need to regenerate something? Which even with the mighty gpt4? Still happens a lot… It’s almost akin to me asking a co-worker how fast he types and calculating how much of his salary it is for him to generate code based on that without any other factors. Also pointed out but why is this even a package? It seems like the definition of bloat when this should have just been a blog post tool or something. It’s all completely unnecessary and seems to try and cash in on hype.

tr33house• 3 years ago

now, was this written ... by GPT-4?

Mistletoe3• 3 years ago

Can anyone give me some estimates instead of defending your jobs as coders?

gwoolhurme8• 3 years ago

What? Nobody is defending their jobs here. It’s explaining how non-trivial this is. What is the point of your comment? If you wanted pure token to money amount and are too lazy to calculate it your self use the tool in the link. That’s not how the real world even with an LLM will work. Let’s assume the LLM writes everything, even gpt4 won’t get everything in 1 go now. Requirements of software are hard to put into pure English… why write this snarky message. Adds nothing to the conversation.

Mistletoe3• 3 years ago

These comments remind me of newspaper articles saying the internet will fail and horse breeders saying cars will fail. All want to deflect to how wrong this is when it should be like the Drake Equation and can still give you data and ranges. With time and AI evolution, the estimates will get more close to this.

“It is difficult to get a man to understand something, when his salary depends on his not understanding it.” -Upton Sinclair

swyx2• 3 years ago

its obviously an impossible question to answer with confidence but i attempted to put some fermi estimates here https://twitter.com/swyx/status/1657603239251181570

and based on kiinda conservative assumptions, ai coders start being competitive in 2028

gwoolhurme8• 3 years ago

This seems sloppy too. Is this based on anything scientifically rigorous? Where I live even startups will take on less profit for a quarter to train new graduates who have never touched an IDE on purpose. The point of training a junior isn’t to be immediately competitive. It’s to take over your job as you move on to other roles.

Mistletoe3• 3 years ago

Thank you!

sph• 3 years ago

Nice guesstimate, but I have an issue with this: "One junior dev will improve to senior dev in 5 years"

That's true only for the HR department. Because code and experience wise, someone that has been programming for 5 years isn't that much experienced than someone fresh from college. In 5 years you barely have time to get to know your editor inside out, and start to have a grasp on your first tech stack.

I consider someone to be an experienced developer one that has seen a couple frameworks and hype cycles come and go. Let's say 10 years of experience.

Also, AI has 0% productivity. Being able to churn out code without being able to reason about it in the grand scheme of things makes it worse than even a non-programmer in problem solving ability. AI has no problem-solving ability. Ignoring this issue makes the whole thing moot, because it assumes that GPT is actually AGI.

ipaddr• 3 years ago

It's like asking what the squareroot of yellow is.

How cheap could it cost would be calculated based on the smallest cost of input parameters required with the smallest code output while minimizing how many time something needs to be reprompted.

No one knows the minimum prompting required.

It's like asking how much does it cost to recreate google (a popular upwork fake project). However much money you want to pour into it but you are still not getting google.

Jeff_Brown• 3 years ago

How much money could a restaurant save by replacing the food with dirt.

jumploops• 3 years ago

It cost me $7.61 to build this library[0] using GPT-4, much more than the direct cost of tokens required to reimplement the library.

The majority of the cost was the large context windows being proxied back and forth to OpenAI from my local machine, rather than just the additional code with each new message. Also, I had no idea how to do what I was doing, so even if I could tell GPT-4 exactly what to build, I wasn't even sure it was possible!

[0] https://github.com/jumploops/magic

furyofantares• 3 years ago

Nice - now please share the prompt to get it to write my code.

tpoacher• 3 years ago

You mean before, or after the bankruptcy? /s

zubairq• 3 years ago

As a developer I “wish” chatgpt could take my job. Really hoping it happens sooner than later so I can get my product out into the world faster. Have tried many chatgpt and other A.I. tools to do this but all of these AI coding tools seem to have no understanding of the code for maintenance when dealing with edge cases and things that are hard to explain to A.I. tools. I wish we had these tools tomorrow but it looks like it is 50 years off . In fact using chatgpt makes a lot of products take longer to build i suspect as 99.999 % of the work is in understanding the code for small upgrades and bug fixes and these tools seem to produce alot of complex and unmaintainable code

How much would it have cost if GPT-4 had written your code

Related Stories