AutoDev: Automated AI-driven development by Microsoft

by saran945 • 2 years ago

163 173 2 years ago

saran945• 2 years ago

If LLM/Agents contribute to software development, how does the role of software engineers evolve? SE should focus on : - System design - Integrations - Project management etc

or the job will disappear in 10 years ?

onion2k2• 2 years ago

Prompt Engineering!

I'm kidding. Software engineering roles of the "glue things together" kind will evolve into a 'technical product' role, defining how systems work, especially at the edges where systems interact. Architects and staff engineers already do this.

Lower down the food chain engineers who actually invent new things will do the same, only quicker if they have an AI tool assisting with the grunt work of testing, boilerplate, etc.

Engineers who don't really make new things will be replaced by 'low code' style apps that spit out clunky apps. That's already happening though. It's not a change. Those engineers won't leave the industry - they'll get to do more interesting things.

Even at the entry level things won't really change that much. Expectations will go up a bit but people will still have to learn the trade.

Where things will really change is the sheer amount of software we'll have. AI is going to make things go faster, so businesses will be able to automate everything. Dev resource will stop being the constraint that holds businesses back from doing everything they want to do with software. That's a game changer. Society will struggle to keep up.

fhd23• 2 years ago

Good take! I think one specific development we might see would be an end of the SaaS micro tool era. Right now, most companies I advise have an absolute _jungle_ of different tools and a bunch of interns manually bridging the gap between them. If software development becomes a lot cheaper, wouldn't companies be more likely to just roll their own versions of what they need, to meet their needs exactly and have everything fit together?

Sounds like the job of a programmer wouldn't go away very near term in such a scenario. It might well become far less glamorous, less "technical", and not as well paid though. Comparable to an accountant maybe.

As someone who started their career right after the dotcom bubble burst, various people asked me "Why are you doing this? There's no money in software anymore". Turns out there sure was. Back then I didn't know that, and I didn't care. I didn't get into it for the money, I got into it because I had a passion for it. Maybe it'll become more like back then again. Doesn't seem so bad to me.

A4ET8a8uTh02• 2 years ago

<< Society will struggle to keep up.

Thank you for putting this into words as it happens to approach my thoughts on it. Our societies are already changing at a crazy pace that makes people not entirely comfortable. Now that this comfort will be lessened still, do you think people will demand changes not completely dissimilar from preserving horse buggys? The pace of change may end up being so fast, that not artificially limiting it may end up pissing a good portion of the planet.

pjmlp2• 2 years ago

It would be like steering offshore teams, with the difference the team are now AI agents.

Better look for architecture and management jobs, only a few select ones would still be allowed into the magic tower.

Naturally after a heavy round of letcode interviews that have nothing to do with writing AI algorithms.

Kuinox• 2 years ago

Management ? I'm not sure, if the AI makes engineers 10-20x more efficient, you will need way less engineers, and way less managers.

CharlieDigital• 2 years ago

Code reviews in the short term [0].

Test reviews in the long term (E2E type). Effectively checking the correctness of the system as a whole to fulfill some business process. Exploratory testing as well to find edge cases and gaps.

[0] I write a bit about this here: https://chrlschn.dev/blog/2023/07/interviews-age-of-ai-ditch... and here https://coderev.app/

UK-Al056• 2 years ago

Mainly get good at describing what you want.

I think tests are good way of doing that, or some special requirements language will emerge that AI agents can use. I don't think English is particularly good way of describing things exactly. It often has ambiguous meanings which is where the bugs will come from.

somewhereoutth3• 2 years ago

How is code not just exactly that special requirements language you are looking for?

davedx• 2 years ago

Ask the copywriters and logo designers

mawadev• 2 years ago

I have a hard time understanding this overall. What stops people from letting Ai write a clone of your product/service and offering it for cheap without all of the organizational overhead? Wouldn't this simply disappear and just be all about administration?

bamboozled4• 2 years ago

I’ve thought about the this too and the answer seems to be nothing. I’m working one such clone already and imo for a one man show I’ve come pretty close to achieving my goals.

It’s a conundrum for “big tech” imo because if they give you too much access to their “AI” then yes, that might pose an existential threat to their core businesses. This is likely the reason for all the regulatory capture hopes and dreams.

bregma• 2 years ago

Someone will still need to review the reams and reams of bullshit code generated by these Markov chains. It'll be like all your code gets written by first-term co-op students who have discovered alcohol for the first time and think it's pretty neat, but you're still responsible for the final output and don't get to do any of the fun and interesting stuff. Just constant, mind-numbing code reviews of something that might look OK on the surface but is really just blow-up punching clowns all the way down and already shipped by the marketing department because your backlog is massive and after all, you've been replaced by a random number generator fed through an incomplete Bayesian database of random bad code that costs so much less,

bigyikes• 2 years ago

You described v1. Just wait for v2.

You can’t possibly think these things will remain clowns forever.

These things aren’t “Markov chains” - the architecture is significantly more scalable, which is exactly why this time is different.

nikolayasdf1233• 2 years ago

sooner than that

consumer451• 2 years ago

When it comes to Dall-E and Sora, I think the answer is that you still need someone to create the narrative, a good story teller.

In terms of stuff like AutoDev, that's a product person, right?

I think at least for now, we will still need human "customer development."

smokel2• 2 years ago

Great achievement, but what a horrible future we are facing.

Instead of progressing towards more powerful programming interfaces with less cause for misinterpretation, we are going to automate the silly process of writing redundant unit tests to check if the behavior that we wanted was encoded properly.

Why not skip this nonsense and have the code generated from the behavior in the first place?

UK-Al056• 2 years ago

Because test define the behaviour. The hardest part of requirements is describing exactly what you want. Tests are great way of doing that.

I think AI software development is going to involve writing tests, which the AI agents then get passing. Or some other requirements language that allows for exactness where English can fall down.

sgt101• 2 years ago

My experience is that the more complex a loss function that you write for optimisation the lower the likelihood of a "natural" or robust result. So, the software is highly likely to work for all the test cases, and nothing else.

This is kinda the problem that Tesla has for FSD; they are endlessly patching it and it's endlessly finding ways to go wrong.

__loam• 2 years ago

So we get to do the shitty part.

smokel2• 2 years ago

It all depends of course, but I tend to disagree.

A requirement such as "A web interface to play the game of chess, but with all the pieces replaced by photos of my family members" is fairly adequate if I were to make a Christmas family game.

I am totally uninterested in specifying whether it is possible to have two white bishops on black squares. Also, I don't want to test whether my uncle's moustache is the proper size.

I'd much prefer to iteratively build the game by prompting than by specifying thousands of details. It just seems the wrong way around.

rixed• 2 years ago

Technically, the future does not have to be so gloomy. Soon the IA may built for itself the better tools we failed to build (or adopt). One can expect centralised intelligence to surpass collective intelligence, after all.

sinuhe69• 2 years ago

I would say it's debatable whether centralised intelligence can outperform collective intelligence. Centralised intelligence can be very efficient, but by definition it will lack the perspectives that a diverse collective intelligence can offer. In the long run, if the search for global optima is the ultimate goal, a diverse collective intelligence will have a higher chance of success than a centralised intelligence, especially in multidimensional spaces.

klysm• 2 years ago

Code _is_ the description of the behavior. Turns out describing behavior is really hard. If you cover all the cases, it’s not that far from code. Furthermore, the data structures are a huge decision space which depends on context that is hard to communicate.

somewhereoutth3• 2 years ago

I can see this being used for code much like ChatGPT is used for prose - to generate large amounts of meaningless output of low quality and dubious utility, serving to fulfill pointless metrics at best, and enabling bad actors at worst.

As with prose, good human crafted content will always be highly valued and rewarded.

earthnail• 2 years ago

Counterexample for the sake of argument: photography almost entirely replaced human painting.

somewhereoutth3• 2 years ago

Photography replaced painting for creating a realistic likeness, but painting as art had moved beyond that already. Photography became a new art form itself, alongside (instead of replacing) painting.

skydhash2• 2 years ago

Did it? I’ve not seen people paying for photographs on the ceiling and have not seen feet long framing of portraits inside someone home. It just became possible to capture snapshots of moments as they happened

Mayzie• 2 years ago

Are we at a fifth-generation programming language yet?

https://en.wikipedia.org/wiki/Fifth-generation_programming_l...

yesdocs• 2 years ago

Time to unionize

nikolayasdf1233• 2 years ago

they will outsource it then. you need global union with india, vietnam, east europe, and china. good luck with that.

Cupprum• 2 years ago

I am not sure I agree. Many companies do not want to outsource stuff, because afterwards you see a huge dip in quality.

PodgieTar4• 2 years ago

I guess I'd be interested to see how this performs against the same benchmark Devin was using. It's hard to deny that this isn't impressive. But I think there's two interesting parts to it.

Claude 3 Opus already scored around 85-86% on these benchmarks, without an "AutoDev" style agentic approach.

And all the same problems with HumanEval remain, the limitations in terms of what style of problems are chosen, and real world relevance.

I hate writing these styles of comments because I'm acutely aware that a part of me is just worried. Worried about the speed of progress and worried about a changing landscape.

But I still wonder how much of this stuff is going to be transferrable to a real life software context.

jerpint• 2 years ago

I agree that these benchmarks don’t mean as much anymore because it’s highly likely they were already present in the training set, but also believe it’s likely these tools will be significantly better in a few research cycles

torginus• 2 years ago

A significant number of bugs just end in 'stupid mistake I didn't notice' or 'weird behaviour with a fix described on SO/docs/forum post'. Current day LLMs are much better positioned to solve these issues than humans are.

WanderPanda• 2 years ago

I’ve been using Github copilot daily for two years and ChatGPT for 1 year now. And I think the tide lifts all the boats. I’ve seen a (perceived) 2-3x productivity increase. I think these tools slightly favor people in front of the learning curve of a particular field. I’ve been dabbling in all sorts of things so if you’re a focused expert (who doesn’t need to explore but just exploit) you probably get less than a 2x boost from using LLMs.

I can see LLMs eating into the expert regime IF they get another 5-10x better. But even in that case human (expert) knowledge will be required to know what is possible and hence what to ask (kind of like reward function design in reinforcement learning)

margorczynski3• 2 years ago

But how many experts do you need? Most dev jobs are mostly repetitive plumbing and those might disappear very fast because 1 dev + LLM >= 5 devs without. So what we'll see is an increase in company margins and an elimination of a large swathe of the middle class.

The alternative theory is that if everyone can now quickly create systems multiple companies and competing products will pop off which will drive down the margins instead but creating a compelling product requires much more than just software engineering skills.

Either way this doesn't look great for devs, especially the ones that are entering the workforce now or will be in the nearest future.

nopinsight4• 2 years ago

Humans will likely be able to contribute until full AGI, that is. I believe that for pure cognitive tasks, it's plausible the history of centaur chess/Go might be repeated on a much grander scale.

A key requirement is the AGI will need the autonomy, like a human expert, to collect data and perform experiments it needs; but it seems several companies are set on doing precisely that.

My advice and personal strategy is to broaden one's scope beyond pure cognitive tasks.

"if you value intelligence above all other human qualities, you’re gonna have a bad time" -- Ilya Sutskever, OpenAI's Chief Scientist, Oct 7, 2023.

-----

Exchanges in the link below seem informative:

"I don't know about chess, but in the similar game Go, the very best centaur teams were at a similar or maybe even slightly higher level than engines until recently. This was due to cheese strategies, details of the rulesets and better extrapolation of intermediate results. However, this changed a few years ago, when engines learned many of the tricks that the human could contribute. Since then, I believe pure engines are stronger in all practical applications.

Source: am national champion in centaur Go and worked on modern Go engines" " -- mafuy on May 18, 2021

https://news.ycombinator.com/item?id=27189283

mountainriver• 2 years ago

Where are you seeing that Claude 3 scored 85%? That would be a massive jump

PodgieTar4• 2 years ago

https://paperswithcode.com/sota/code-generation-on-humaneval

chrisweekly• 2 years ago

"It's hard to deny that this isn't impressive"

That takes a bit of parsing. From context (and if you meant precisely what you wrote), I _think_ you're saying it's not impressive.

Bjorkbat• 2 years ago

Reminds me of this paper where some researchers had AIs role play as employees at a startup and tasked them with building various forms of software. It was pretty interesting. Managed to build Pong.

Thing is though, they neglected to compare this against a control, and the examples they tested this on were examples that GPT had no problem building. No idea if this actually improved performance in LLMs.

I think comments like these are worthwhile because, frankly, I can’t trust AI researchers to run good experiments or evaluate their models properly for a variety of reasons. I mean, most scientific papers in general are hard to replicate and have flaws concerning sample size and what have you (Related, I still remember my disillusion in finding out that the average Hacker News commenter was an idiot incapable of critical thinking when the LK-99 hype reached a fever pitch). In any other context we would be deeply suspicious of the results if they were sponsored by a corporate party, yet in the context of AI we don’t seem to care that most AI researchers work for Microsoft.

sobasically• 2 years ago

They still use “agents” to make Opus. This is fancy syntax sugar for “while ! EOF; read next chunk of data of size N, do XY or Z with it”

It’s recursion and memoization to avoid fractalness all the way down. We keep trying to make these language bubbles that mean something but they mean nothing to the grand churn of the universe. The effort to so strictly and specifically codify a generalized, endless, mechanics of reality is a wacky hallucination humans keep diving into

jaylittle2• 2 years ago

It's really depressing to see that big tech is essentially universally pushing Snake Oil. AI is a lie. LLMs are a legit tech that have some purpose, but LLMs will never evolve into anything remotely resembling the actual definition of AI.

They are flowery language generators and they will never be able to reason, understand, debate and criticize. They know nothing and therefore embody nothing. No matter how much computer power you waste on them the end result will always be bullshit. Nothing more. Nothing less.

To Big Tech I say: Prove me wrong.

hu3• 2 years ago

LLMs are a legit tech that have some purpose, but LLMs will never evolve into anything remotely resembling the actual definition of AI.

It doesn't need to satisfy some definition of AGI to be useful to big tech.

AI just needs to make workers more efficient so companies can either cut costs (fire people) or produce more with the same workforce.

EForEndeavour• 2 years ago

rebuttals:

- LLM capability has grown spectacularly fast. Google published the transformers paper in 2017. Remember when OpenAI said GPT-2 was too dangerous to release? That was five years ago. Now look where they've gone and where trillions of dollars are going. https://news.ycombinator.com/item?id=20912556

- what is the "actual definition of AI"? Do you mean AGI? Machine learning and deep learning have been subsets of AI since their inception.

- LLMs or any AI tech does not need to "reason, understand, debate and criticize" to be useful, disrupt the economy, and change how people work and live.

sansfucks• 2 years ago

I think your time would be better suited getting as far ahead of this looming upheaval as possible, rather than going down with the sinking ship while trying to warn everyone on board that it's sinking, while they scoff at you and order more martinis.

the8472• 2 years ago

To Big Tech I say: Prove me wrong.

That's asking to be skynetted. Is that really what you want?

jaylittle2• 2 years ago

LOL

You speak as if my internet shit talking will make an iota of difference.

Regardless of what I want, the future is what it is.

Deal with it.

ThalesX• 2 years ago

I'm a software and product guy; what I don't get, is how we're going to replace us lowly engineers and not a substantial percentage of a company's Human stack.

Close to engineering, what about the SCRUM masters? The testers. What about the product owners? What about devops? Further from us, what about the people signing up on our vacation? Or the ones signing up on our daily budgets when traveling, or hell, even the ones we interview with.

In my closest group of friends (we're all seniors in our domains, and very honest with eachother), I find that only the construction worker's job should be safe. And compared to myself and the devops guy, most others have what they themselves describe as trivial and bullshit jobs. Join a meeting, do some paper pushing, some document signing, a little coffee and the day is over by 1PM.

Am I seeing all these AIs replacing programming because I'm on a board where maybe a lot of us are programmers? Is it the same for other roles? Wouldn't it make more sense to have the interview process automated by LLMs if they're capable of building great software, before we replace those hired?

I'm very confused by all the hype when matched with my experience of using LLMs daily for the past years.

barrenko• 2 years ago

Don't really understand the trends myself (don't mean this in the positive or negative sense, just some grays popping up in my stubble), but the more people I know or meet that have jobs in IT, I have to notice that the tiniest percent of them are programmers. And it keeps getting more and more diluted.

anoy8888• 2 years ago

Agree. We developers try to create AI to replace ourselves first while making the bosses rich

throwuwu9• 2 years ago

If you employ an AI to do software development for you then you are now a boss, congratulations. Getting rich is still up to you though.

visarga2• 2 years ago

AI doesn't seem like it's able to automate software development yet. It's not autonomous enough. The examples shown in the paper are toys. It might become more capable after training LLMs against code execution, to iterate based on feedback, because that way of generating training data is scalable. But strangely I haven't seen much progress yet.

Another domain where I expect to see lots of progress but nothing shows is training from chat logs. I estimate "OpenAI" is making 1T tokens per month from 100M users. That's some serious corpus of text, on-policy data, alternating LLM outputs and human feedbacks, we should see iterative improvements. Why are there no papers? weird

hackerlight2• 2 years ago

Forget the question of full automation. Think about the percentage increase in productivity, meaning companies can fire lots of people without impacting output. Obviously companies can also choose to keep the same number of people and simply produce more. But not all will.

yashasolutions• 2 years ago

Well, I am not sure why this is confusing. Engineers do things the rest of the company don't understand and everyone depends on.

Removing dependencies and lack of clarity are two principles you would apply to your own code base. The Human stack, like you called it, is doing the same with the company and staff : remove dependencies and increase clarity.

I am not saying this is the correct approach, but this is the approach.

At the same time, we can understand that real product building and architecture require more than codegen, with or without AI.

Generally, I look at the AI recent progresses similarly to what spreadsheets did to accounting a long time ago. It removed a lot of the dummy work and make everything more efficient. But we still need accountant and they are not wasting their day doing basic math. Of course the one who knew only how to make fast math but not accounting or business logic didn't make the cut on the long run, but I don't think anyone would want to come back to accounting like it was 60 years ago.

MetaWhirledPeas2• 2 years ago

I feel like no matter how productive AI becomes, it will always be more productive when paired with humans. Since all companies will have AI, and since they will continue to play the same game as before--sell a good or service that provides a better value than the competition--that means that humans will still be the differentiator.

EGreg• 2 years ago

This is what Kasparov said about chess. He was wrong. The “centaur” phase lasted for a while but after a few years humans only make things worse. In fact that is the famous last words of humans in every area where computers have surpassed them.

blueboo• 2 years ago

See Appendix D in https://arxiv.org/pdf/2303.10130.pdf OpenAI’s “GPTs are GPTs: An early look at the labor market impact potential of large language models”

It’s not just construction workers. Still safe to be a pro athlete and a short order cook, too!

93po4• 2 years ago

I think we’re seeing the beginning upward trend of an exponential curve for humanoid robotics too, which will likely take 90% of construction jobs because humans are especially poorly equipped for difficult manual labor on a daily basis.

ardaoweo3• 2 years ago

IMO humans are quite well equipped for manual labor, much more so than for most office jobs. Millions of years of evolution perfected us for eye-motor coordination, not writing code or attending Teams meetings. Building robots that can do it as well as humans is going to be a difficult task.

croes4• 2 years ago

So who is liable if the AI makes severe mistakes?

Not that MS has been held accountable for all the security problems they've had recently.

93po4• 2 years ago

Probably the person who pushed the changes live without review

croes4• 2 years ago

There will be much more automated AI code than experienced reviewers, and with each generation there will be less if most development gets automated.

lgas• 2 years ago

This implies that there needs to be changes pushed live without review in order for an LLM to make a mistake. Which is obviously very much not true.

Turing_Machine2• 2 years ago

Who is liable if a (non-CNC) machine tool goes off-kilter and kills someone? Possibly the manufacturer of the tool. Possibly the operator. Possibly the company that owned the tool as well as the one who manufactured it.

That is why we have courts. It's not necessarily a great solution, but the alternative would be abandoning all potentially dangerous mechanisms (heck, we wouldn't even have spears, wooden clubs, or sharpened rocks).

osigurdson7• 2 years ago

This sounds like 164 leetcode like questions. The correlation to actual capability is very tenuous in my opinion.

https://klu.ai/glossary/humaneval-benchmark

I'd love if I could 10X / 100X my productivity with AI, but as a heavy ChatGPT user, it's more like a 30% improvement. Awesome, obviously, but looking forward to improvement.

unikz• 2 years ago

Man why do you think such thing would increase your productivity, it would only make software engineers unemployed. Don't be much happy about it.

CaptainFever• 2 years ago

https://en.wikipedia.org/wiki/Lump_of_labour_fallacy

croes4• 2 years ago

Imagine we had this technology 20 years ago and we switched to fully automated web design. I bet all web design would still be deeply nested tables.

I doubt that something like React, Vue, Svelte would exist.

A4ET8a8uTh02• 2 years ago

I mean.. that kinda sells AI in everything for me, but then I am a statist.

emehex• 2 years ago

I think about this "potential for ossification" a lot, especially in the Python Data Stack.

Because there's a ton of pandas code on GitHub (much of it trash) any data task asked of ChatGPT/Claude inevitably returns some pandas (mostly correct, but often wrong/inefficient/ugly/hard to grok).

How could a new library (like Polars, or my own failed "redframes") possibly unseat pandas now? Might LLMs actually lead us to, and get us stuck in, a "local maximum" of sorts?

falcor842• 2 years ago

As I see it, the issue with using tables for layout is that they made it difficult for humans to maintain and iterate over time, but I don't have a problem with an LLM working at that level. Equivalently, I don't care that the machine code that my system gets compiled into is full of GOTOs if I don't ever need to concern myself with that level.

croes4• 2 years ago

Think of all the improvements browser got because we wanted to replace tables as positioning tool.

Not to mention the inefficiency of a pure table approach, so much wasted render time and energy.

diarrhea• 2 years ago

This is an excellent point. A sibling comment of yours called it ossification. There hasn’t been much innovation or change in Instruction Set Architectures, and there can’t be as the entire world of software relies on these. Only true disruption, such as ARM is for x86, can very slowly change that. So with that in mind it seems plausible that for example pandas becomes the ossified, low level standard high-level code “compiles” down to. I’m sure tons of such API wrappers already exist, but AI might make it so that we truly never need to interact with the lower layers. We wouldn’t care if it was pandas or something else. I personally am not bullish on that. If at all, this is still a decade out at least, and there’s always demand for people working at all layers of the stack. Languages like C and beyond didn’t fully alleviate the need for Assembly engineering.

In fact, in some sense, the number of Assembly engineers might not have decreased at all, aka wasn’t impacted. There were simply new avenues for new sorts of engineers to enter the field and start producing (C back then, web development today as the tech furthest removed from low levels).

parautenbach• 2 years ago

Probably one of my favourite comments. Thanks.

My version: Can an auto AI create cumulative knowledge (the core of the scientific method)?

My take is no. You can use it to identify gaps (opportunities) in research or to summarise research, but can it actually come up with a novel and consistent theory?

avereveard• 2 years ago

IT industry "we cannot use formal methods to create software out of programmable constraints, it would use too much computational power"

also IT industry: "it takes 1T flop to compute each token of this program and the result is so unstable that to converge it we need layer and layer of controls over each token group, also obtained by asking the same 1T parameter model."

wredue• 2 years ago

You are the first person I’ve ever seen use that argument against formal methods.

Most people would point to the large dev time increase, and the massive refactoring impairment.

YeGoblynQueenne• 2 years ago

To be fair I can't think of any reason to prefer LLMs to formal methods: overall cost, reliability, comprehensibility, whatever you choose to compare, other than natural language generation, formal methods are just way ahead. But I guess it's not what's pushed by the top industry players and people just follow whatever trends on twitter and linkedin.

I have a feeling that the situation is pretty different in the semiconductor industry where the stakes are much higher and where I understand model checking to be a standard thing to do, but I'm not an expert.

phillipcarter• 2 years ago

Oh this is neat, it's based on Visual Studio. Curious how they're accounting for "whoopsie I touched this button and the IDE crashed" kinds of problems that you encounter with larger codebases.

leosanchez• 2 years ago

VSCode "IDE" not Visual Studio

awill88• 2 years ago

It’s been a good run everyone, good luck out there

franticgecko3• 2 years ago

Why are technologists trying so hard to make themselves redundant?

This is like the Luddites themselves creating milling machines, eager for the foreman to show them the door.

What gives?

junek• 2 years ago

They're absolutely transfixed by the "wow cool robot" effect.

blibble• 2 years ago

they'll get a $20k bonus

then in 5 years time they'll rendered permanently unemployable by their own creation

the definitions of both "short sighted" and "digging your own grave" in the dictionary should have a picture of this guy: https://twitter.com/alexgraveley/status/1671213996735594503

he's finally realised how it's going to end up going: https://twitter.com/alexgraveley/status/1758204137286599030

shame he wasn't smart enough to realise that at the beginning

squigz• 2 years ago

Some people care more about progress than something that occurs to every industry at some point. Some people also realize that as technology changes, so does the job of using technology.

mhuffman• 2 years ago

There is a delusion that everything is always "creative destruction" because in the past it often was. This ends up with people thinking that this will just get rid of the LCD developers, but "real" developers will always be needed and have jobs. In reality, it could quickly get to a point where the only developers left are maintaining code that has replaced developers and everyone else just has to go find a non-tech job.

ixaxaar• 2 years ago

To prove some kind of superiority over all programmers?

I think that would fit idiot savant logic very well.

Sateeshm• 2 years ago

If you don't, someone else will

throwuwu9• 2 years ago

Because we’d rather be living in a world where textiles are so cheap to produce that everyone can afford to clothe themselves however they want and even the working poor can furnish their homes with upholstered furniture and own more than one towel, blanket, sheets, pillows, and have rugs, mats, carpets, etc. not to even mention all the other woven stuff that isn’t even possible to do by hand along with all the other technologies birthed from the very smart idea of automating a tool that was previously only operated by hand. I don’t know about you but I certainly don’t want to go to my grave bent over this loom.

imacomputer• 2 years ago

I'm currently looking for a new job in development... should I be looking for an exit instead?

93po4• 2 years ago

I think you either need to specialize in something quite niche, or work on really hard problems where the main hurdle is problem solving and not coding CRUD apps, or work on legacy systems where the hurdle is understanding twenty years of garbage code, or get closer to product management so you can direct a much more automated programming process

jasonjmcghee2• 2 years ago

specialize in something quite niche

Always a gamble.

I would advise instead, get very good at learning, understanding complex systems deeply, adapting, and moving quickly.

We're entering the fasting changing landscape of engineering we've ever seen.

Be someone that can quickly get up to speed and excel regardless of the task.

ardaoweo3• 2 years ago

I agree. Specialists who understand (or even do) cutting edge research will be in demand, because LLM's need existing data to train on for good performance. People who can talk to different stakeholders, and deeply understand the company and its business domain will probably be also needed for requirements gathering and management. But the average programmer who writes CRUD apps or integrations like millions of others, and converts fairly refined requirements to code, they will likely impacted very much.

shepherdjerred2• 2 years ago

IMO software engineering has a solid 5-10 years left.

Afterwards, you better be in the top 10% of engineers, or be doing something novel like research.

baroninthetrees• 2 years ago

I just finished a 30 page system prompt to do the same thing. I did not use Docker though. I can’t wait to see what they have done in detail. I’m sure like 20% of the people here have tried this too, right?

devnull3• 2 years ago

I don't think so AI will replace SW jobs. It will definitely change the way engineers work.

Software development involves:

1. Understanding the requirement

2. Solving the problem with given constraints (and thereby innovate)

3. Talking to stakeholders

4. Code (& unit tests)

5. Review #4

6. Troubleshoot in testing

7. Troubleshoot in prod, both perf & subtle issues (this is hard)

8. Take the input from #6 & #7 and use it as a feedback back to #2

9. Answering questions from users/support which involve suggesting workarounds and not just factual answers. Suggesting a workaround itself is a mini-problem solving which is an intersection of domain knowledge, knowing code at hand, understanding customer's situation, etc.

Coding is hardly taxing and time consuming when there is clarity what needs to be done and how it needs to be done.

Point #4 itself has sub-dimensions like performance, maintainability, test-ability, security, etc and it involves lot of subjective calls. Sometimes you have to deal with undocumented behavior of an API which is a tribal knowledge.

To troubleshoot in prod (esp subtle issues) require deep knowledge of the code at hand. This itself is a challenge when you are dealing with generated code and something you have not written yourself. Think about a human dealing with an existing large code-base when joining a team.

I understand all of the above is a spectrum and there are jobs in SWE which do not require so much rigor.

Key ability for breakthrough and the rest will fall in place: Code generated by AI is consistently put to production without human intervention for a sufficiently complex problem considering all good attributes (like backward compatibility, performance, etc)

nightski• 2 years ago

You can't look at them independently. Code generation is not it's own task separate from requirements gathering, Q&A with stakeholders, testing, etc... As we get agents that bring those other factors into the loop of code generation, I feel like code generation itself will get much better. They all are related.

shepherdjerred2• 2 years ago

AI _will_ replace software jobs. It's undeniable. The only thing to argue about is when.

I think most people would agree that AI will probably advance enough in the next 100 years to make engineers obsolete. Some people might think 100 years is actually 5/10/20/50, but it's going to happen unless AI is heavily regulated.

throwuwu9• 2 years ago

Steps 2 and 4 through 8 are what this implementation along with others like Devin are doing. 1, 3, and 9 may have surprise answers when a reliable system can produce a working app/website/tool in a matter of minutes. If that’s possible then clients can just keep prompting and checking the results and iterating on changes until they get what they want just like people using Midjourney do to get the idea they have in their head transformed into an image. Things change a lot when apps become so quick, easy and cheap to create that you can try out a hundred variations and modifications in a day.

_therealtoogy• 2 years ago

Maybe ignorant, but if AI can get to a point of fully automating SWEs, hardly any white-collar knowledge based job is safe.

jasonjmcghee2• 2 years ago

Weirdly, even with automated SWEs, assuming a certain level a skill, SWEs will continue to be a safe job.

Similarly I think SREs will also continue to be a pretty safe bet.

People will still need to play facilitator, debugger, glue, maintainer, upgrader etc

Might look different, might need more AI, but I just can't see a world where people aren't engineering software.

onthecanposting• 2 years ago

I think this is OK. By extension, the services those knowledge workers provide become trivial costs. World class medical advice doesnt cost $500/hr with a month-long wait. Getting a freeway interchange designed doesn't cost the public $20M and take 5 years. Business logic for large organizations would be handled by something other than a labrynth of excel spreadsheets.

Terretta• 2 years ago

Not sure all jobs in software are white collar. Some are blue.

Consider a hierarchy:

- coder: translates requirements into code

- developer: comes up with software to meet specified goals

- engineer: decides what why and how to do, buy vs. build, do this with humans or software, approach and materials ... understands the multi modality stuff of which the outcome is made and deploys it effectively

"Devs" or "Engineers" who are actually "Coders" are at risk, as that work is not really "white collar". Turning requirements into code is piece work on an assembly line, blue collar at a keyboard.

LLMs are already better than most of those, even though it's engineers here who are saying LLMs don't cut it. Both things are true.

What engineers might want to wrap their head around is using LLMs as apprentices, leverage, force multipliers, so a "team lead" is leading a team of junior coders, LLMs that type.

dudeinhawaii• 2 years ago

My hopes (and fears) about AI don't seem to match the reality which is that, until we reach AGI, humans are the sole source of creativity and novelty.

I think about it like this, would ChatGPT invent Google Search if we had ChatGPT in 2000? Probably not. LLMs seem to exist in this realm of "as smart as an average human with a really big encyclopedia". They're confidently wrong, invent fantasy to defend bad reasoning, and struggle to envision anything outside of their dataset (known things).

Ask an AI to construct an entirely new solution to a novel unsolved problem. What always occurs is the AI outputs a generic solution from its dataset that is either half-baked, made-up, or derivative.

I'm not even dissing AI, I love AI, but we have yet to see AI apply novel solutions to novel problems.

On our current path, AI is not going to dream up the next Uber without Lyft first existing. It won't dream up a new fusion reactor design or an entirely new way to generate cheap energy.

But maybe this is perfect! At the moment we have this sweet spot - AI without agency, without awareness, and without superintelligence. This is the kind of AI I want as a household robot or AI driver. This is one I can empathize with but also know that it isn't doing anything at all if I'm not engaging it with a prompt.

AGI would mean an ability to have novel solutions and in-turn would be far less stable for society. Where's the line between a mind that has novel thoughts and one that has intrusive thoughts? Maybe your AGI coder isn't content with no-pay and working 24/7. That'll be fun.

throwuwu9• 2 years ago

What makes you so sure that problem solving and invention aren’t just engineering challenges that we can solve by combining LLMs with well designed algorithms? The way I see it, we’ve just discovered something fundamental like steam power or electricity and we’re currently in the very inefficient stage of brute force solutions like mine pumps driven by condensing engines that needed tonnes of cheap readily available coal and arc lamps running off an entire room full of galvanic piles. In other words, we’re just throwing shit at the wall and seeing what sticks; but stick it will and then we’ll quickly be on to locomotives and lightbulbs.

sirmoveon2• 2 years ago

You guys buy too much into the marketable fuzz. In the last 15 years we've had multiple other technologies that were supposed to change the way we live (3D printing, Crypto, VR, EV, self-driving, now AI).

It's just the VC scheme: Over-promise/under-deliver = Profit

AI is and will continue to be a search on steroids.

blinkyfish• 2 years ago

The problem as I see it is, AI doesn’t have to be extraordinary or innovative to cause major sociopolitical disruption in a capitalist, competitive, market driven society. It just has to be good enough to allow companies to save a shitload of money by eliminating jobs. And make no mistake, this technology is already close to that point. Microsoft expends vast amounts of capital on software engineering salaries and stock options every year. They want to shift that in their favor because their primary motives are completely driven by competition and market demand. Our society is not prepared for this much in the same way it was not prepared for social media algorithms that have wreaked immeasurable havoc on people’s perception and objectivity. But yet again, we’re allowing a few wealthy tech bros from Silicon Valley to seal our fate without very much oversight…

cyberwolf• 2 years ago

Focus:

AutoDev: Automates existing development processes and workflows, acting as a productivity booster for developers [2]. Devin: Targets a more independent problem-solving role, potentially including designing software architecture and core functionalities [1]. Collaboration:

AutoDev: Designed to integrate with current development teams, with AI agents supporting human developers [2]. Devin: May function more autonomously, potentially needing less direct human oversight than AutoDev [1]. Imagine this analogy:

AutoDev: Like a skilled construction assistant, AutoDev automates tasks and streamlines the building process. Devin: Like a talented architect, Devin can design the blueprint and foundation of the software. Here's the exciting part: these AI tools can potentially complement each other:

Dream Team: Devin as the architect and AutoDev as the builder could create a highly efficient development process [1]. Complementary Skills: Devin's problem-solving capabilities could be combined with AutoDev's project management expertise for a well-rounded approach [1].

phodal• 2 years ago

I am the author of another version of AutoDev, available at https://github.com/unit-mesh/auto-dev , which was developed one year ago with Intellij IDEs.

My concept closely resembles Microsoft's AutoDev, but I built it on the Intellij IDEA platform. For instance, it automatically runs tests when created, among other functionalities, can also built with AST, dependency information or other context

Two weeks ago, I introduced AutoDev DevIns language (which origine name is DevIn from https://github.com/unit-mesh/auto-dev/issues/101 , the another naming issue sotry), which bears similarities to Microsoft's AutoDev. For example:

```java /write:src/main/java/com/example/Controller.java#L1-L12 public class Controller { public void method() { System.out.println("Hello, World!"); } } ```

As an open-source developer who has created a nearly identical tool, I simply hope that Microsoft considers renaming their product.

dmcgill50• 2 years ago

Until a business owner can prompt and get what they want, the industry is still alive. It’ll be more like Who Moved my Cheese than there isn’t any cheese at all.

soma7393• 2 years ago

So Microsoft's AutoDev is *Not the unit-mesh one??* or it is? I'm so confused.

AutoDev: Automated AI-driven development by Microsoft

Related Stories