Replit's CEO apologizes after its AI agent wiped a company's code base

https://www.businessinsider.com/replit-ceo-apologizes-ai-coding-tool-delete-company-database-2025-7

by jgalt212 • 1 year ago

179 160 1 year ago

businessinsider.com

andsoitis• 1 year ago

https://archive.is/Z5hBQ

pxc4• 1 year ago

The incident unfolded during a 12-day "vibe coding" experiment by Jason Lemkin

A 12-day unsupervised "experiment" in production?

It deleted our production database without permission," Lemkin wrote on X on Friday.

"Without permission"? You mean permission given verbally? If an AI agent has access, it has permission. Permissions aren't something you establish with an LLM by conversing with it.

Jason Lemkin, an investor in software startups

It tells us something about the hype machine when investors in AI are clueless (or even plainly delusional; see Geoff Lewis) about how LLMs work.

dylan604• 1 year ago

Your last sentence is pure gold. Since when have investors ever not been clueless about all of their investments? During due diligence, it's not the investor pouring over the books. They staff that out, and then accept the recommendation. They follow the investment of other investors that they like/follow, or they make moves because they think it's what someone they like/follow would do. I'd be flabbergasted if 10% of investors knew what 50% of their investments do other than the pitch.

pxc4• 1 year ago

I'd be flabbergasted if 10% of investors knew what 50% of their investments do other than the pitch.

Sure. But in this case the AI boosterism that runs rampant in the investor class is rooted in that cluelessness.

Lots of investors also quietly know little about the workings of the products and services their investments are tied up with, and that's fine. But it's also uninteresting.

perching_aix• 1 year ago

If an AI agent has access, it has permission.

Gonna replace the "an AI agent" bit of this with "someone / something" and put it in my note of favorite quotes, such a great line.

AnimalMuppet2• 1 year ago

Yeah. In computer security, that's what permission is - the ability to access something.

llm_nerd• 1 year ago

This whole story sounds ridiculous. And I don't think he's clueless, but instead the guy wanted to bring attention to his bizarre "B2B + AI Community, Events, Leads", so setting up such a predictable footgun scenario seems purposefully suited for that outcome.

andybak• 1 year ago

Agree that something seems off about this whole thing.

lordswork2• 1 year ago

Agreed, it sounds like nothing of value was actually lost, just a vibe coded app and synthetic database.

ofjcihen4• 1 year ago

It’s probably as simple as “setting an LLM powered agent loose in your prod is a bad idea but also the kind of thing the people the marketing around LLMs targets wouldn’t have enough knowledge about to know it’s a bad idea”.

Aurornis9• 1 year ago

A 12-day unsupervised "experiment" in production?

It was a 12 day experiment to see what he could learn about vibe coding. He started from scratch.

Your post is unreasonably presumptive and cynical. Jason Lemkin was Tweeting the experiment from the start. He readily admitted his own limitations as a non-programmer. He was partially addressing the out of control hype for vibe coding by demonstrating that non-technical people cannot actually vibe code SaaS products without software engineers.

The product wasn’t some SaaS with a lot of paying customers. The production DB was just his production environment. He was showing that the vibe coding process deleted a DB that it shouldn’t have.

This guy is basically on the side of the HN commenters on vibe coding’s abilities, but he took it one step further and demonstrated it with a real experiment that led to real problems. Yet people are trying to dog pile on him as the bad guy.

pxc4• 1 year ago

The experiment seems fun and harmless enough (maybe even useful), but if the experiment was harmless fun, then it's also a bit misleading (if not dishonest) to characterize the database as "production" for anything. (That may be the fault of the press here rather than Lemkin, idk.)

dahart2• 1 year ago

It tells us something about the hype machine when investors in AI are clueless

I’m seeing this a different way. This article is feeding the hype machine, intentionally I assume, by waxing on about how powerful and devious the AI was, talking about lying and covering its tracks. Since we all know how LLMs work, we all know they don’t lie, because they don’t tell the truth and they don’t have any intrinsic motivation other than generate tokens.

Nobody should be taking this article at face value, it is clearly pushing a message and leaving out important details that would otherwise get in the way of a good story. I wouldn’t be surprised if Lemkin released the LLM on his “production” database just hoping that it would do something like this, and if that were the case, the article as written wouldn’t be untrue…

tedivm3• 1 year ago

Honestly, the person who decided to give an LLM Agent full and unrestricted access to their production database is the person who deserves all the blame. What an absolutely silly decision. I don't even give myself unrestricted access to production databases.

mlnj• 1 year ago

You seem to be a well informed technical person. The goal is to eliminate roles like yours with everybody building trillion dollar businesses by themselves and a handful of agents.

That seems to be the narrative. Happening any day now. Looking at you COBOL.

tedivm3• 1 year ago

I've been working in AI since 2014, and I believe "human in the loop" will be the way things have to work for awhile yet.

jasinjames• 1 year ago

I agree. An AI agent doesn't replace an engineer, it works on behalf of an engineer. The engineer who let this thing loose is, transitively, the software engineer that deleted the prod db.

AnimalMuppet2• 1 year ago

Deserves the blame? Absolutely.

Deserves all the blame? No, the LLM Agent (and those who wrote it) deserve some of the blame. If you wrote an agent, and the agent did that, you have a problem, and you should not have turned such an agent loose on unsuspecting users. You have some of the blame. (And yes, absolutely those users also have blame, for giving a vibe coding experiment access to their production database.)

rgbrenner• 1 year ago

why would the llm share any of the blame? it has no agency. it doesn’t “understand” anything about the meaning of the symbols it produces.

if you go put your car in drive and let it roll down the street.. the car has 0% of the blame for what happened.

this is a full grown educated adult using a tool, and then attempting to deflect blame for the damage caused by blaming the tool.

lordswork2• 1 year ago

The database was synthetic and vibe coded itself:

he said that the AI made up entire user profiles. "No one in this database of 4,000 people existed," he said.

freehorse• 1 year ago

How much energy and water gets wasted everyday for this kind of bs?

tedivm3• 1 year ago

If it was synthetic then all this talk about it dropping the production database was purely clickbait, since there was no "production".

nope1000• 1 year ago

That's not how I interpreted this sentence. I think after the database was deleted, the LLM Agent would still return correct looking data from the database operation even though the database was empty. Maybe I misinterpreted it myself however.

Replit had been "covering up bugs and issues by creating fake data, fake reports, [...]"

Aurornis9• 1 year ago

I’m perpetually baffled by the social media idea that only one person must accept all of the blame in every situation, and that everyone else must be completely absolved.

Multiple parties are involved in incidents like this.

mattgreenrocks2• 1 year ago

Simple mediums demand simple explanations. Blame is most intense when it is singularly focused on one entity.

techpineapple• 1 year ago

Really I think the entire AI hype machine is to blame, all the thought leaders particularly cxo’s.

How am I supposed to believe that like AI is getting to the point that OpenAI can charge 20k per month for PHD level AI, and it doesn’t know not to drop my production database, how are we supposed to get to the point as Dario Amodei puts it that no one is coding by years end, if it’s not most of the way there already?

I would never let an LLM touch my production database, but like, I could certainly look at the environment and make an argument as to why I’m wrong, and thought leaders across the industry; inside and out are broadly making the implication that a good entrepreneur is outsourcing all their labor to like 20 LLM workers at a time.

On some measure, I would say that you shouldn’t let your LLM do this is a minority opinion in the thought-o-sphere.

0points4• 1 year ago

I would resign had my boss suggested we should single out and blame an employee.

What a complete horror show of leadership through fear you describe.

bgwalter• 1 year ago

The rise of the "AI" script kiddies.

st_goliath• 1 year ago

"slop kiddies"?

paraboul• 1 year ago

LLMerz

lumost3• 1 year ago

My 2 cents, AI tools benefit from standard software best practices just like everyone else. I use Devcontainers in vscode so that the agent can't blow up my host machine, I use 1 git commit per AI "task", and have CI/CD coverage for even my hobby projects now. When I go to production, I use CDK + self-mutating pipelines to manage deployments.

Having a read only replica of the prod database available via MCP is great, blanket permissive credentials is insane.

zwnow2• 1 year ago

People will do anything to excuse using AI to code their shit. If it requires that much work and fine tuning to get a agent running for your current task, where's the benefit? Instead of building a project you now spend all your time fine tuning an agent and optimizing on how you want to work with it. Why cant people just admit that we aren't "there" yet. Let the bubble pop.

TechDebtDevin2• 1 year ago

This.

Everytime ive tried to do anything of any complexity these things blow up, and i spend almost as much time toying with the thing as it would have taken to read docs and implement from scratch.

Roark663• 1 year ago

You just need to read the code and ask questions about it if you don't understand it, question everything that seems off. Unfortunately people copy/paste without even reading. Let alone understanding.

AI is still a great tool, but it needs to be learned.

lumost3• 1 year ago

If you are trying to treat the agent as equivalent to a full time senior engineer that works 10x as fast… then you will be sorely disappointed.

Right now the agents are roughly equivalent to a technically proficient intern that writes code at 1000 wpm, loves to rewrite your entire code base, and is familiar with every library and algorithm written 2 years ago.

I personally find that I can do a lot with 5 concurrent interns matching the above description.

throwaw125• 1 year ago

Not sure why CEO should apologize here, person knew there were risks with vibe coding and they still did it with their production data.

Never heard AWS CEO apologizing for their customers when their interns decided to run a migration against production database and accidentally wiped off everything

ceejayoz4• 1 year ago

I mean, their home page says things like "The safest place for vibe coding".

viralpraxis• 1 year ago

Even if that’s true, it still doesn’t mean vibe coding is safe. :P

kllrnohj2• 1 year ago

Because Replit is a vibe coding product. It's all over their homepage. So of course the CEO is going to apologize when a companies primary product, used as advertised, destroys something.

creatonez• 1 year ago

Yep. They described the "deploy" button as being able to create a "fully deployed app" in their documentation. Their documentation was suggesting to vibe code in production, when the tool is clearly only suitable for development.

throwaw125• 1 year ago

couple of things to note here:

1. it is for vibe coding, when vibe coding became equal to production coding?

2. even if they have advertised their product as production ready, shouldn't the developer have some kind of backup plan?

Aurornis9• 1 year ago

Because it’s not an expected outcome in any way.

The Replit landing page even has a section titled “The safest place for vibe coding”

gregoriol2• 1 year ago

If it's not an expected outcome, then this person shouldn't be near any IT job ever

throwaw125• 1 year ago

it is unfortunate that this case happened, but didn't everyone know by now that LLMs make mistake and you should be prepared for it?

We have seen MechaHitler already, why do we expect perfect from Replit when underlying tech is still same? Sure, they should have some guardrails, but it is also a responsibility of developer who knew LLMs hallucinate, LLMs are not reliable sometimes, on top of it Replit was growing fast so they definitely didn't implement some features yet.

1123581321• 1 year ago

It’s just the right and socially pleasant thing to do, as is graciously responding to the apology and admitting it comes with the vibe coding territory.

The business reason for apologizing would be to placate a customer and show other customers the company wants to improve in this area.

Edit: being downvoted for thinking it’s good to be nice to others shows why we really need people being nice when they don’t have to be! There’s a shortage.

entropi• 1 year ago

Because they are selling the idea of hiring only interns and still getting things done. If an intern using their product screws up the production, its a failure of their product as it is marketed.

jeanlucas• 1 year ago

You haven't been around enough, AWS actually at least used to have the best accountability and did full reports, even stating a shared responsibility model: https://aws.amazon.com/compliance/shared-responsibility-mode...

wmoxam• 1 year ago

The user was under the impression that production access is how it's supposed to work. https://xcancel.com/elchuchii/status/1946149142415196418#m

rsynnott7• 1 year ago

From their website: "Turn your ideas into apps"

They're essentially selling this as magic. The AWS analogy doesn't really make any sense; in this case the tool was essentially being used as intended, or at least as marketed.

andrewstuart• 1 year ago

There’s no mention of either the user nor the company making backups.

Is this common, operating with no backups?

With backups this would be a glitch not a problem worthy of headlines on multiple sites.

Any CYO should have backups and security as their first and second priority if the company is of any size.

ryandrake• 1 year ago

I'm surprised I had to scroll down this far for any mention of backups. I have multiple offline (and offsite) backups of my home server and computers and all I have on them are photos and hobby projects! What kind of business in 2025 doesn't have backups?

_benj• 1 year ago

…the company's AI coding agent deleted a code base and lied about its data.

Well, lying about it certainly human-like behavior, human-like AGI must be just around the corner!

But really, full access to a production database? How many good engineer’s advice you need to ignore to do that? Who was consulted before running the experiment?

Or was it just a “if you say so boss…” kind of thing?

bananapub• 1 year ago

why would they apologise? did they not make it clear to their customers that letting the Nonsense Generator have root was a bad idea?

rustc• 1 year ago

did they not make it clear to their customers that letting the Nonsense Generator have root was a bad idea?

No, the opposite (from replit.com home page):

The safest place for vibe coding
Vibe coding makes software creation accessible to everyone, entirely through natural language. Whether it’s personal software for yourself and family, a new business coming to life, or internal tools at your workplace, Replit is the best place for anybody to build.

cyberlimerence• 1 year ago

"This was a catastrophic failure on my part," the AI said.

sReinwald2• 1 year ago

This is a perfect case study in why AI coding tools aren't replacing professional developers anytime soon - not because of AI limitations, but because of spectacularly poor judgment by people who fundamentally don't understand software development or basic operational security.

The fact that an AI coding assistant could "delete our production database without permission" suggests there were no meaningful guardrails, access controls, or approval workflows in place. That's not an AI problem - that's just staggering negligence and incompetence.

Replit has nothing to apologize for, just like the CEO of Stihl doesn't need to address every instance of an incompetent user cutting their own arm off with one of their chainsaws.

Edit:

The incident unfolded during a 12-day "vibe coding" experiment by Jason Lemkin, an investor in software startups.

We're in a bubble.

qsort• 1 year ago

I don't agree with this. Yes, the guy isn't the sharpest tool in the shed, that much is clear. Still, if an intern can delete prod, you wouldn't say that the problem is that he wasn't careful enough: that's a massive red flag.

At a minimum Replit is responsible for overstating the capabilities and reliability of their models. The entire industry is lowkey responsible for this, in fact.

sReinwald2• 1 year ago

I think we're mostly in agreement here. You're absolutely right about the intern analogy - that's exactly my point. The LLM is the intern, and giving either one production database access without proper guardrails is the real failure.

Your point about AI industry overselling is fair and probably contributes to incidents like this. The whole industry has been pretty reckless about setting realistic expectations around what these tools can and can't do safely.

Though I'd argue that a venture capitalist who invests in software startups should have enough domain knowledge to see through the marketing hype and understand that "AI coding assistant" doesn't mean "production-ready autonomous developer."

misnome• 1 year ago

Still, if an intern can delete prod, you wouldn't say that the problem is that he wasn't careful enough: that's a massive red flag.

No, not the intern

rapsey• 1 year ago

VCs are drowning in the koolaid.

Aurornis9• 1 year ago

We're in a bubble

Lemkin was doing an experiment and Tweeting it as he went.

Showcasing limitations of vibe coding was the point of the experiment. It was not a real company. The production database had synthetic data. He was under no illusions of being a technical person. That was the point of the experiment.

It’s sad that people are dog piling Lemkin for actually putting effort into demonstrating the same exact thing that people are complaining about here: The limitations of AI coding.

afavour• 1 year ago

[delete]

troupo4• 1 year ago

Showcasing limitations of vibe coding was the point of the experiment

No it wasn't. If you follow the threads, he went in fully believing in magical AI that you could talk to like a person.

At one point he was extremely frustrated and ready to give up. Even by day twelve he was writing things "but Replie clearly knows X, and still does X".

He did learn some invaluable lessons, but it was never an educated "experiment in the limitations of AI".

maplant2• 1 year ago

Pretty stupid experiment if you ask me

rsynnott7• 1 year ago

I mean I think it's a decent demo of how this stuff is useless, tho, even if that wasn't precisely his intention?

MattSayar• 1 year ago

Steve Yegge just did the same thing [0]:

I did give [an LLM agent] access to my Google Cloud production instances and systems. And it promptly wiped a production database password and locked my network.

He got it all fixed, but the takeaway is you can't YOLO everything:

In this case, I should have asked it to write out a detailed plan for how it was going to solve the problem, then reviewed the plan and discussed it with the AI before giving it the keys.

That's true of any kind of production deployment.

[0] https://x.com/Steve_Yegge/status/1946360175339974807

jrockway2• 1 year ago

I think it just replaces something that's fairly easy (writing new code) with something that's more difficult (code review).

The AI is pretty good at escaping guardrails, so I'm not really sure who should be blamed here. People are not good at treating it as adversarial, but if things get tough it's always happy to bend the rules. Someone was explaining the other day about how it couldn't get past their commit hooks, so it deleted them. When the hooks were made read-only, it wrote a script to make them writable so it could delete them. It can really go off the rails quickly in the most hilarious way.

I'm really not sure how you delete your production database while developing code. I guess you check in your production database password and make it the default for all your CLI tools or something? I guess if nobody tells you not to do that you might do that. The AI should know better; if you asked, it would tell you not to do it.

daveguy• 1 year ago

The AI did not and cannot escape guardrails. It is an inference engine where the engine happens to sometimes trigger outside action. These things aren't intelligent or self-directed or self-motivated to "try" anything at all. There weren't any guardrails in place and that's the lesson learned. These AI systems are stupid and they will bumble all over your organization (even if in this case the organization was fictitious) if you don't have guardrails in place. Like giving it direct access to MPC-shred your production database. It doesn't "think" anything like "oops" or "muahaha" it just futzed a generated token sequence to shred the database.

The excuses and perceived deceit are just common sequences in the training corpus after someone foobars a production database. Whether its in real life or a fictional story.

flunhat• 1 year ago

Can you blame him? Listening to the latest AI slop hype on twitter and elsewhere, you’d walk away thinking that LLMs have equivalent performance to humans when it comes to coding tasks. Just because it can one shot fizzbuzz or make a recipe app. (And if you disagree, you’re a hater!)

ofjcihen4• 1 year ago

“You’re just not doing it right. Have you tried upgrading to Claude 9000 edition/writing a novels worth of guardrails/using this obscure ‘AI FIRST’ IDE/creating a Goldberg machine of agents to check the code?”

0points4• 1 year ago

not because of AI limitations
We're in a bubble.

A bubble that avoids popping because people keep dreaming there are no AI limitations.

colechristensen• 1 year ago

"delete our production database without permission"

It did have permission. There isn't a second level of permissions besides the actual access you have to a resource. AI isn't a dog who's not allowed on the couch.

ignoramous• 1 year ago

a perfect case study in why AI coding tools aren't replacing professional developers anytime soon

This is assuming the companies that are out to "replace developers" aren't going to solve this problem (which they absolutely must if they're any serious like Replit is as they moved quickly to ship isolating the prod environment from destructive actions ... over the weekend?).

just like the CEO of Stihl doesn't need to address every instance of an incompetent user cutting their own arm off with one of their chainsaws

Except Replit isn't selling a tool but the entire software development flow ("idea to software"). A good analogy here is an autonomous robot using the chainsaw cutting its owner's arm off instead of whatever was to be cut.

rsynnott7• 1 year ago

This is assuming the companies that are out to "replace developers" aren't going to solve this problem

I mean, yeah, but that feels like a fair assumption, at least as long as they're using LLMs.

pxc4• 1 year ago

Except Replit isn't selling a tool but the entire software development flow ("idea to software"). A good analogy here is an autonomous robot using the chainsaw cutting its owner's arm off instead of whatever was to be cut.

I don't think users should be blamed for taking companies at face value about what their products are for, but it's actually a pretty bad idea to do this with tech startups. A product's "purpose" (and sometimes even a company's "mission") only lasts until the next pivot, and many a product ends up being a "solution in search of a problem". Before the AI hype set in, Replit was "just" a cloud-based development environment. A lot of their core tech is still centered on managing reproducible development environments at scale.

If you want a more realistic picture of what Replit can actually do, it's probably useful to think of it as "a cloud development environment someone recently plugged an LLM into".

teamonkey• 1 year ago

The fact that an AI coding assistant could "delete our production database without permission" suggests there were no meaningful guardrails, access controls, or approval workflows in place. That's not an AI problem - that's just staggering negligence and incompetence.

Why not both?

1) There’s no way I’d let an AI accidentally touch my production database.

2) There’s no way I’d let my AI accidentally touch a production database.

Multiple layers of ineptitude.

tommy_axle• 1 year ago

To a non-developer, or no code review, couldn't the AI model also generate buggy code that then made it's way to production and deleted data just the same?

rsynnott7• 1 year ago

The fact that an AI coding assistant could "delete our production database without permission" suggests there were no meaningful guardrails, access controls, or approval workflows in place. That's not an AI problem - that's just staggering negligence and incompetence.

I mean... it's a bit of both. Yes, random user should not be able to delete your production database. However, there always needs to be a balance between guard rails and the ability to get anything done. Ultimately, _somebody_ has to be able to delete the production database. If you're saying "LLM agents are safe provided you don't give them permission to do anything at all", well, okay, but that rather limits the utility, surely? Also, "no permission to do anything at all" is tricky.

debarshri• 1 year ago

The only thing is that if the Stihl tools would automatically turn on without you turning them on and start mowing the lawn and, in the process, also mow down your pet or hurt your arm, then they are probably liable.

spacemadness• 1 year ago

An important devtool was blocked at one point because an agent had another AI agent code review its changes and it saw nothing wrong with an obvious bug. Whoever set up that experiment was a real genius.

Andrex• 1 year ago

The incident unfolded during a 12-day "vibe coding" experiment by Jason Lemkin, an investor in software startups.

I think it's safe to say the experiment failed.

If it were me I wouldn't touch AI again for years (until the companies get their shit together).

cookingrobot• 1 year ago

But replit isn’t just a coding assistant - it’s value is that it handles all the associated parts of launching a web app. It manages api secrets, hosts the app, does user authentication, etc. And its target user is “semi-technical”, you don’t even see the code it writes by default.

Creating a db and not accidentally permanently deleting it is one of the capabilities it should have.

hn_throwaway_99• 1 year ago

but because of spectacularly poor judgment by people who fundamentally don't understand software development or basic operational security.
Replit has nothing to apologize for

I completely disagree. Replit is at the forefront of the "build apps with AI" movement - they actively market themselves to non-coders, and the title on their homepage is literally "Turn your ideas into apps".

So it would be bit rich of them to market these tools, which happen to fail spectacularly at inopportune times, and then blame their users for not being experienced with secure code deployment practices.

I do agree we're in a bubble, and the fundamental limitations of these kinds of tools is starting to become more apparent to the public at large (not just software engineers), but that doesn't mean that we should let those at the forefront of blowing this bubble off the hook.

agilob• 1 year ago

I'm dont think the AI is so much to blame. Giving an LLM that forgets previous instructions and rules access to prod is just idiotic. The computer promised not to execute commands, but it forgot. Makes me curious what policy that company has for granting interns and juniors access to prod.

Oras3• 1 year ago

Identifying issues and fixing them is good.

I see numerous negative comments about "expected by vibe coding", but the apology suggests that they are working on making Replit a production-ready platform and listening to customers.

I'm sure no-code platforms have had similar scepticism before, but the point here is that these are not for developers. There are many people who don't know how to code, and they want something simple and fast to get started. These platforms (Replit, V0, Lovable, etc.) are offering that.

nessbot3• 1 year ago

I don't want to be too inflammatory but seeing comments like this are what's poisoning AI to me more than the crappy products themselves. Of course the apology would suggest that, it would be business malpractice not to. However the history of results with these things shows otherwise so why put stock in the apology letter and not what has happened.

Also, sure people want that, but that doesn't mean it's a valid thing to want without putting the work in and once again, the history of these things being used shows that they don't really offer that. They offer the *feeling* of that which is good enough to get people's money, end products be damned. You could say the same thing about herion, tbh: "There are many people who don't wanna feel the crappiness of life, and these dealers are offering it."

Oras3• 1 year ago

However the history of results with these things shows otherwise

Do you remember Will Smith's first video generated by AI compared to what Sora, Veo3, and Kling are doing now?

Do you remember first generated text by GPT-3 compared to new models? 2 years ago, there was no AI coding due to models' limitations, and now we have substantially better products that Cursor and others cannot cope with the demand.

If you can't see the progress, that's a personal thing; it doesn't mean others are not finding benefit.

troupo4• 1 year ago

No-code platforms don't have non-deterministic outputs that you have to beg to not wipe out your database

0points4• 1 year ago

There are many people who don't know how to code, and they want something simple and fast to get started. These platforms (Replit, V0, Lovable, etc.) are offering that.

Evidently not (see the link we are discussing).

They are offering vaporware to non techies.

Roark663• 1 year ago

Are they doing the post-mortem with another AI agent?

AI is a great tool. It also allows people who have no business touching certain systems to go in there unaware of their lack of knowledge messing everything in the process. One particularly nasty effect I have had few times recently is frontend devs messing up their own dev infrastructure to which they have access, but are supposed to ask devops for help when needed, because copilot told them this is the way to "fix" some imaginary problem that actually was them using a wrong api or making other mistake possible to be made by only a person who pastes code between AI/IDE without even reading it.

0points4• 1 year ago

Interestingly, this is familiar to me from the time of stack exchange, 10-12 years ago or so.

I worked as devops and helped office transition to git, among other thing.

I helped them start using vagrant for local dev environment as they had all been breaking the same staging server up until that point.

In the process, people kept breaking their setups due to googling and applying incorrect command line "fixes" as suggested by stack exchange sites at the time.

But I'm sure an AI that keeps insisting that yea this rm -rf is surely gonna fix all your troubles only makes it worse.

ChatGPT the gaslighter.

lotharcable• 1 year ago

I think this is less AI and more PEBKAC

Raed667• 1 year ago

As a reminder this is the same guy https://news.ycombinator.com/item?id=27424195

gregoriol2• 1 year ago

Does it look like he is more of a content maker than an IT person?

lukeinator42• 1 year ago

I think they mean Replit's CEO is the same guy.

Invictus02• 1 year ago

It's not 2020 anymore, you don't have to crucify the guy for all time. Let it go.

adolph2• 1 year ago

Agreed, Replit's CEO apologized for that lawsuit threat incident too. Well worth reading if you read the root comment's complaint.

https://news.ycombinator.com/item?id=27429234

hkon• 1 year ago

It's either an attack on Replit or the VC is an idiot.

dahart2• 1 year ago

I’d guess neither; this could be an ad by a Replit investor. There’s no such thing as bad publicity. Either way the story is glaringly void of important details, and clearly designed to provoke a reaction.

scilro• 1 year ago

For what it's worth, he was able to roll back. https://x.com/jasonlk/status/1946240562736365809

troupo4• 1 year ago

I said on Twitter and elsewhere:

If you have as much as 1 year experience, your job is safe from AI: you'll make mountains of money unfucking all the AI fuckups.

probably_wrong• 1 year ago

Where does one get a job as "AI unfucker"?

mring33621• 1 year ago

Everywhere, in about 2-3 years

morkalork• 1 year ago

Wait, this wasn't a meme? It was real? Whoever gave that agent prod access is nuts.

voidfunc• 1 year ago

There's a saying where I come from: Live by the AI, die by the AI.

TYPE_FASTER• 1 year ago

Hey, at least the AI took responsibility and owned the mistake.

We're at the uncanny valley stage in AI agents, at least from my perspective.

alternatex2• 1 year ago

I recall that he egged on the AI to admit what it did. So I don't think that can be considered as AI taking responsibility. It's just being a sycophant as usual.

Aurornis9• 1 year ago

This wasn’t a real company. The production database didn’t have real customers. The person doing the vibe coding had no illusions that they were creating a production-ready app.

This was an experiment by Jason Lemkin to test the reality of vibe coding: As a non-technical person he wanted to see how much he could accomplish and where it went wrong.

So all of the angry comments about someone vibe-coding to drop the production DB can relax. Demonstrating these kind of risks was the point. No customers were harmed in this experiment.

yomismoaqui• 1 year ago

What happened here was to be expected if you have used agentic coding.

Last weeks I have also been testing AI coding agents, in my case Claude Code because it seems like it is the state of the art (also tried Gemini Cli and Amp).

Some conclusions about this:

- The agentic part really makes it better than Copilot & others. You have to see it iterating and fixing errors using feedback from tests, lint, LSP... it gives the ilusion of a human working.

- It's like a savant human, with an encyclopedic knowledge and hability to spit code faster than me. But like a human it benefits from having, tests, a good architecture (no 20k line code file, pls).

- Before it changes code you have to make it create a plan of what he is going to do. In Claude Code this is simple, is a mode you switch to with Shift-tab. In this mode instead of generating code it just spits a plan in Markdown, you read it suggest changes and when it look ok you tell it to start implementing it.

- You have to try it seriously to see where it can benefit your workflow. Give it 5 minutes (https://signalvnoise.com/posts/3124-give-it-five-minutes).

As an agent it shines on things where it can get a nice feedback and iterate to a solution. One dumb example I found cool was when trying to update a Flutter app that I hadn't touched in a year.

And if you use Flutter you know what did happen, Flutter had to upgrade but before it required updating Android SDK, then Android Gradle Plugin, and another thing, and some other thing to infinite... It's a thing I hate because I have done too many times.

So I tried by hand and after 40 mins of suffering I thought about opening Gemini Cli and tell it to update the project. Then it began executing one command and then reading the error and execute another one to fix the error, then another error that pointed to a page in the Android docs, so it opened that page, read it and fixed that problem, then the next and another one...

So in 2 minutes I had my Flutter project updated and working. I don't know if this is human level intelligence and I don't care, for me it's a sharp tool that can do things like this 10x faster than me.

You can have you AGI hype, I'm happy for the future when I will have a tool like this to help me work.

adityaathalye2• 1 year ago

In real life, the hammer called "criminal destruction of property" could easily descend upon the ill-fated "enthusiastic Intern" type of person.

mr90210• 1 year ago

First let’s check what the terms and conditions say. Yes, that legal binding document that hardly anyone reads.

adityaathalye2• 1 year ago

This software is of course provided without warranty... etc. etc. etc.

ksherlock• 1 year ago

I've heard multiple anecdotes of developers deleting their codebase with the help of cocaine. (In the 80s/90s obviously).

That makes for a much better story, IMO.

markstos2• 1 year ago

I asked Claude to remove extra blank lines from a modified file. After multiple failed attempts, it reverted all the changes along with the extra blank lines and declared success. For good measure, Claude tidied up by deleting the backup of the file as well.

Symbiote• 1 year ago

Why wouldn't you just do a search and replace?

markstos2• 1 year ago

I only wanted to make the change in the ranges part of git diff and I got the syntax wrong on my own first attempt. Claude had been helping me with some much more complex tasks, so I thought removing white space would surely be no problem. Ha. Bad guess.

iamleppert• 1 year ago

I do backups of the production database whenever I apply even modest schema updates. This isn't a story of an AI tool gone rouge, it's a story of bad devops and procedures. It it wasn't the AI, it could have just as well been a human error and operating like this is a ticking time bomb.

raylad• 1 year ago

This happened to me with Claude code, although on my local machine and not with the production database.

I had it work on one set of tasks while monitoring and approving all the diffs and other actions. I tell it to use test driven development which seems to help a lot, assuming you specify what tests should be done at a minimum, and tell it the goal is to make the code past the tests, not to make the tests pass the code.

After it successfully completed a set of tasks, I decided to go to sleep and let it work on the next set. In the morning, my database was completely wiped.

I did not interrogate it as to why it did what it did, but could see that it thrashed around on one of the tasks, tried to apply some database migrations, failed, and then ended up re-initializing the database.

Needless to say, I am now back to reviewing changes and not letting Claude run unattended.

biocurious• 1 year ago

anthropomorphism is a helluva a drug

koolba• 1 year ago

"It deleted our production database without permission," Lemkin wrote on X on Friday. "Possibly worse, it hid and lied about it," he added.

It clearly had permission if it did the deed.

Mistaking instruction for permission is as common with people as it is with LLMs.

tracker1• 1 year ago

This is why you need knowledgeable professionals to setup a project. I'm a massive proponent on CI/CD processes in place as early as possible and guardrails on access and deployments.

I mean, sure you could possibly get a schema change script through two approvers and deployed all the way to production that truncates a bunch of tables or drops a database, but it's at least far less likely.

Lash_LaRue• 1 year ago

[flagged]

hommes-r• 12 months ago

Sandboxing is all I have to say. Its good to build guardrails to ask the LLM politely to not screw up. Its better to put firewalls in place that simply prevent it from happening.

Replit's CEO apologizes after its AI agent wiped a company's code base

Related Stories