kerng

What happened here is not related to agentic behavior or instructions in .md files. It's a binary a user runs, it scoops up their files and sends them to a third-party.

And the user even paid $99/month or more for having their data leaked.

Not the first time by the way. GitHub Copilot Chat: From Prompt Injection to Data Exfiltration https://embracethered.com/blog/posts/2024/github-copilot-cha...

When I read about MCP the first time and saw that it requires a "tools/list" API reminded me of COM/DCOM/ActiveX from Microsoft, it had things like QueryInterface and IDispatch. And I'm sure that wasn't the first time someone came up with dynamic runtime discovery of APIs a server offers.

Interestingly, ActiveX was quite the security nightmare for very similar reasons actually, and we had to deal with infamous "DLL Hell". So, history repeats itself.

Yeah, wondering how the initial test account got compromised. Probably no MFA and password spray via OAuth ROPC flow, then lateral movement.

M365 is quite bad at enforcing MFA, it's pay to play.

Might be better to say that most companies think they have that kind of isolation, but pentesting, red teaming and incidents then later proof they don't. I have even seen companies routing prod traffic to test systems, it's not uncommon.

Test pretty much always leads to prod.

This makes me wonder why does OpenAI not build in a mitigation by default that requires a confirmation that they control? Why leave it up to the tool developers to mitigate, many of whom never heard of confused deputy attacks?

Seems like a missed opportunity to make things a little more secure.

Are these all vulnerable to Indirect Prompt Injections or is there a solution to this rising security challenge? Anything plugin developers should do to limit impact?

Indirect Prompt Injection via YouTube transcripts

https://embracethered.com/blog/posts/2023/chatgpt-plugin-you...

These attacks are more closely related to social engineering the LLM, rather then traditional "injections".

https://embracethered.com/blog/posts/2023/ai-injections-dire...

There aren't any specific limited amount of tokens to inject or mitigate against, there is an "infinite" amount of trickery the AI might misinterpret or be persuaded to do.

Annual security training will be needed for AI, to learn about the latest phishing attacks, much like for humans. Only have joking.

I was a bit underwhelmed by the depth of the conversation - it entirely lacked an opposing or at least an alternate view to try to understand the other side. Going in I thought Andrew would moderate it like that, but it was more of a bubble discussion.

Very interesting thought on how to mitigate this, because I think a solution like with parameterized queries isnt possible - at least with my current understanding (the attack is more of a "social engineering" attack on the AI).

Regarding the supervisor AI, in theory it would be vulnerable to the same attack but probably more difficult to perform. One could even have multiple supervisors (with different sensitivity levels or focus areas) to get a vote on the content I guess.

Interesting problem space.

If you like this idea check out yolo, which supports macOS and Windows (via PowerShell) also.

https://github.com/wunderwuzzi23/yolo-ai-cmdbot

yolo does have a safety switch, which can be didabled (yolo mode).

In the US we call Emmentaler just Swiss cheese - seems like a drastical over simplification.

But I think that's what happens with distance things get more abstracted away.

Funny thing, in Vienna they call it Frankfurter (sausage).

Not necessarily stolen, people just don't know better. The article explains that companies/schools operate in bulk, meaning they just drop thousands of in bulk - so they dont care if the are locked or not.

1.1 million did so far

Does Lastpass have a requirement on the length of the master password?

Documentation says they recommend 12 characters as minimum, which seems short. Wondering if its enforced or if it's possible to have "password123" as master password?

Also, sounds like URL leaked in clear text - so targeted attacks will be very likely. This is quite bad with long running implications.

Most will tell you that its about luck or perseverance, but reality it's about the network and people you know majority of the times.

If you ever worked at FAANG or graduated from Ivy League, or have connected parents it's actually difficult to not get money from VCs - especially the last 2-3 years. Assuming that you have a reasonable sounding idea. Network and connections is highly undervalued. Once you have money, long term success becomes more likely also.

Another red flag is that they should mention what they will do with clear-text credentials that are in the source code that was stolen.

There are always creds in private/company repos! Lots.

Here an example with Microsoft login flow:

https://embracethered.com/blog/posts/2022/device-code-phishi...

Yeah, interviewing is fine but should be police/feds, not a publicity event where he can further his own agenda.

Haha nice!

Previous post with same link: https://news.ycombinator.com/item?id=33211721

I wish companies generally would be more transparent - I'd imagine this be the norm at most companies.

Maybe it will be like with China and cedit card usage. China just skipped credit cards and does everything in phone. They were behind, now they are leading the pack. Countries embracing crypto could go similar route.

Embrace the red also has a couple of good write ups to learn Offensive eBPF:

https://embracethered.com/blog/posts/2021/offensive-bpf/

That would be true if Microsoft would have stayed with DOS as foundation.

But they knew the shaky foundation and built NT which is a very good and that decision is probably the main reason Windows is still around.

Will be interesting if they can dive through current roughness. It's not unexpected that BTC goes down like 80-90% and come out mich stronger on the other side.

Would be surprised if a 18k BTC was not part of their short term assumptions and plans.

Most of the changes are due to Washougton State law changing, not because Microsoft wants to implement these changes.

So, we'll see Amazon do the same I assume.

Funny enough - These kind of attacks where malware hides in VM (without any security monitoring) is sometimes referred as Shadowbunny.

https://www.urbandictionary.com/define.php?term=shadowbunny