Using AI to secure AI

https://mattsayar.com/letting-inmates-run-the-asylum-using-ai-to-secure-ai/

by MattSayar • 11 months ago

101 30 11 months ago

malfist2• 11 months ago

According to my company's senior leadership there's nothing the magic dust of AI can't solve. Even problems with AI can be solved by more AI

jimt1234• 11 months ago

We must work at the same company. LOL

nicce• 11 months ago

This reminds me about "The Emperor's New Clothes" way too much.

malfist2• 11 months ago

Oh, you can't see the emperor's new clothes? Well let's have him wear more of it.

kelseyfrog• 11 months ago

This is where it gets fun.

We're on the precipice of being able to install AI into positions of business critical processes. Hiring, billing, sales, and compliance. It's going to be great watching c-suite and VPs who are drunk on the sauce accept AI in these positions and get golden parachutes when the business ends up facing a massive external audit, fraud, and the possibility of bankruptcy.

bongodongobob• 11 months ago

Pfft. The hammer will come down IT leadership, not execs.

TZubiri• 11 months ago

Thinking about pivoting to pentesting

at-fates-hands• 11 months ago

> nWe're on the precipice of being able to install AI into positions of business critical processes. Hiring, billing, sales, and compliance

We're already there. Have been for several years now. I was doing RPA (robotic process automation) for about 4 years in a corporate environment. It went from, "Lets automate these mundane tasks" to "How can we create a billing platform that can be totally automated?". This was back in 2021, just for reference.

> It's going to be great watching c-suite and VPs who are drunk on the sauce

Hopefully this will be a cautionary tale of what happens when they do?

https://www.reuters.com/legal/lawsuit-claims-unitedhealth-ai...

UnitedHealth Group Inc (UNH.N), uses an artificial intelligence algorithm that systematically denies elderly patients' claims for extended care such as nursing facility stays, according to a proposed class action lawsuit, filed on Tuesday.

Family members of two now-deceased UnitedHealth beneficiaries sued the insurer in federal court in Minnesota, saying they were forced to pay out of pocket for care that doctors said was medically necessary.

aurumque• 11 months ago

And yet when I recommend that replacing senior leadership is one of highest ROI potentials for AI they immediately shut down the conversation.

crinkly2• 11 months ago

Yeah. Our senior leadership just ask ChatGPT what to do. Might as well skip the middle man.

crinkly2• 11 months ago

Going through that now. I’m part of the Chernobyl clean up team already. Mostly because my part of the org is the only one with a positive ROI and isn’t a fucked up mess still so I’ve got plenty of time to deal with everyone else’s problems.

How did we get in this situation? Avoid all the fucking fads.

mmsc• 11 months ago

Currently living through a great litmus test of competency versus luck by company leaders

bink• 11 months ago

I think it's funny that I don't see any findings from either Claude or DataDog that couldn't be detected using static analysis. They're pretty simple code bases and maybe that's why.

I'll pay more attention when they start finding vulnerabilities in commonly used, more complex applications.

boston_clone• 11 months ago

Ah, then you’ll want to check out xbow -

https://xbow.com/blog

I believe some folks here (moyix) are active with the project.

ofjcihen• 11 months ago

This has already been leading to some incredible profits for security companies like mine.

So please, don’t be too loud about how terrible it is :)

johntiger1• 11 months ago

who watches the watch man?

gbrindisi• 11 months ago

We’ve kinda solved the detection of issues. what we still lack is understanding what’s important.

I think an underappreciated use case for LLMs is to contextualize security issues.

Rather than asking Claude to detect problems, I think it’s more useful to let it figure out the context around vulnerabilities and help triage them.

(for better or worse, I am knee-deep in this stuff)

ryao• 11 months ago

The quotation is more impactful in the original Latin: Quis custodiet ipsos custodes?

themanmaran• 11 months ago

custodes[.]ai would be a great startup name

bee_rider• 11 months ago

Actually, Custodes would have nothing to do with abominable intelligence </warhammer 40k>

scarlettadham• 11 months ago

is this the Blackwall from Cyberpunk, kinda reminds me of that.

amelius• 11 months ago

Next: using AI to sue AI.

qingcharles• 11 months ago

I saw in Trump v. Murdoch case that an AI company has filed a motion to be allowed to analyze the filings and report back to the court on anything that looks shady. It's only a small step from there to having AI bulk sue people on behalf of a law firm.

rdegges• 11 months ago

Here's a better option -- what we've been working on at Snyk.

- Take something like Cursor and plug the Snyk MCP server into it: https://docs.snyk.io/integrations/developer-guardrails-for-a... (it has a one-click install) - Then, either within your project or via global settings, create some human-language rules for your AI code editor to use (this works basically the same between all editors: Claude Code, Cursor, Windsurf, etc...)

For example, a rule might state:

"If you add or change any code, run a Snyk Code scan on the modified files then fix the detected vulnerabilities. When you're done fixing them, perform another scan to ensure they're fixed, and if not, keep iterating until the code is secure."

Obviously, there are other rules you can use here, such as using Snyk's open source dependency testing to identify vulns in third-party dependencies and handle package updates/rewrites/etc., but you get the idea.

This works insanely well -- I've been playing around with it for a while now and we're getting close to rolling this out to all of our users in a major way =)

The best part about it is that you can just "vibe code" whatever you want, and you get really accurate static analysis security testing incorporated by default automagically.

I recorded a little video here that walks through this in-depth (https://www.youtube.com/watch?v=hQtgR1lTPYI), if you want to see the part I'm referencing, jump to 20:09 =)

Using AI to secure AI

Related Stories