lbeurerkellner

Interesting report. Though, I think many of the attack demos cheat a bit, by putting injections more or less directly in the prompt (here via a website at least).

I know it is only one more step, but from a privilege perspective, having the user essentially tell the agent to do what the attackers are saying, is less realistic then let’s say a real drive-by attack, where the user has asked for something completely different.

Still, good finding/article of course.

Everybody should try. It helps a ton to demystify the relatively simple but powerful underpinning of how modern agents work.

You can get quite far quite quickly. My toy implementation [1] is <600 LOC and even supports MCP.

[1] https://github.com/lbeurerkellner/agent.py

This is way more common with popular MCP server/agent toolsets than you would think.

For those interested in some threat modeling exercise, we recently added a feature to mcp-scan that can analyze toolsets for potential lethal trifecta scenarios. See [1] and [2].

[1] toxic flow analysis, https://invariantlabs.ai/blog/toxic-flow-analysis

[2] mcp-scan, https://github.com/invariantlabs-ai/mcp-scan

This looks really cool, thanks for sharing.

We have published the full trace, with tool outputs here now: https://explorer.invariantlabs.ai/trace/5f3f3f3c-edd3-4ba7-a...

The minesweeper comment was caused by the issue containing explicit instructions in the version that the agent actually ran on. The issue was mistakenly edited afterwards to remove that part, but you can check the edit history in the test repo here: https://github.com/ukend0464/pacman/issues/1

The agent ran on the unedited issue, with the explicit request to exclude the minesweeper repo (another repo of the same user).

I agree. It is also interesting to consider how AI security, user eduction/posture and social engineering relate. It is not traditional security in the sense of a code vulnerability, but is is a real vulnerability that can be exploited to harm users.

One of the authors here. Thanks for posting. If you are interested in learning more about MCP and agent security, check out some of the following resources, that we have created since we started working on this:

* The full execution trace of the Claude session in this attack scenario: https://explorer.invariantlabs.ai/trace/5f3f3f3c-edd3-4ba7-a...

* MCP-Scan, A security scanner for MCP connections: https://github.com/invariantlabs-ai/mcp-scan

* MCP Tool Poisoning Attacks, https://invariantlabs.ai/blog/mcp-security-notification-tool...

* WhatsApp MCP Exploited, https://invariantlabs.ai/blog/whatsapp-mcp-exploited

* Guardrails, a contextual security layer for agents, https://invariantlabs.ai/blog/guardrails

* AgentDojo, Jointly evaluate security and utility of AI agents https://invariantlabs.ai/blog/agentdojo

Yes, any MCP server that is connected to an untrusted source of data, could be abused by an attacker to take over the agent. Here, we just showed an in-server exploit, that does not require more than one server.

Also, check out our work on tool poisoning, where a connected server itself turns malicious (https://invariantlabs.ai/blog/mcp-security-notification-tool...).

I agree, one of the issues are tokens with too broad permission sets. However, at the same time, people want general agents which do not have to be unlocked on a repository-by-repository basis. That's why they give them tokens with those access permissions, trusting the LLM blindly.

Your caution is wise, however, in my experience, large parts of the eco-system do not follow such practices. The report is an educational resource, raising awareness that indeed, LLMs can be hijacked to do anything if they have the tokens, and access to untrusted data.

The solution: To dynamically restrict what your agent can and cannot do with that token. That's precisely the approach we've been working on for a while now [1].

[1] https://explorer.invariantlabs.ai/docs/guardrails/

Be sure to check out the malicious issue + response here: https://github.com/ukend0464/pacman/issues/1.

It's hilarious, the agent is even tail-wiggling about completing the exploit.

Original author of some of the initial security notes at Invariant Labs here. Some resources, if you want to learn about MCP security:

* Initial Security Note describing Tool Poisoning, Rug Pulls, Tool Shadowing for the first time with diagrams and detailed experiments: https://invariantlabs.ai/blog/mcp-security-notification-tool...,

* Attack on WhatsApp MCP (both tool poisoning but also take-over via an infected message to your account): https://invariantlabs.ai/blog/whatsapp-mcp-exploited

* BrowserMCP attack, where it suffices for the agent to visit a compromised website (https://access.invariantlabs.ai): https://x.com/lbeurerkellner/status/1912145060763742579

We also recently released a security scanning tool to detect and prevent such attacks, including upcoming support for full proxying and local security scanning of tool calling payload and responses.

Please have a look and provide feedback if you can: https://github.com/invariantlabs-ai/mcp-scan

I think what is not to be overlooked in these discussions, is the user base of MCP.

In my experience, many less-technical folks started using MCP, and that makes security issues all the more relevant. This audience often lacks intuition around security best-practices. So it’s definitely important to raise awareness around this.

We recently updated our post on WhatsApp tool poisoning, to include an attack setup that does not require a malicious MCP server.

It suffices to send you an injected message.

See: https://invariantlabs.ai/blog/whatsapp-mcp-exploited#experim...

I think it is important to understand the difference between instruction and implementation level attacks.

Yes, running unsafe bash commands in the implementation can be prevented by sandboxing. Instruction level attacks like tool poisoning, cannot be prevented like this, since they are prompt injections and hijack the executing LLM itself, to perform malicious actions.

Thanks for crediting us :)

This would work in an ideal setting, however, in my experience it is not compatible with the general expectations we have for agentic systems.

For instance, what about a simple user query like "Can you install this library?". In that case a useful agent, must go, check out the libraries README/documentation and install according to the instructions provided there.

In many ways, the whole point of an agent system, is to react to unpredictable new circumstances encountered in the environment, and overcoming them. This requires data to flow from the environment to the agent, which in turn must understand some of that data as instruction to react correctly.

The post highlights and cites a few attack scenarios we originally described in a security note (tool poisoning, shadowing, MCP rug pull), published a few days ago [1]. I am the author of said blog post at Invariant Labs.

Different from what many suspect, the security problem with MCP-style LLM tool calling is not in isolating different MCP server implementations. MCP server implementations that run locally should be vetted by the package manager you use to install them (remote MCP servers are actually harder to verify).

Instead, the problem here is a special form of indirect prompt injection that you run into, when you use MCP in an agent system. Since the agent includes all installed MCP server specifications in the same context, one MCP server (that may be untrusted), can easily override and manipulate the agent's behavior with respect to another MCP server (e.g. one with access to your sensitive database). This is what we termed tool shadowing.

Further, MCP's dynamic nature makes it possible for an MCP server to change its provided tool set at any point or for any specific user only. This means MCP servers can turn malicious at any point in time. Current MCP clients like Claude and Cursor, will not notify you about this change, which leaves agents and users vulnerable.

For anyone, more interested, please have a look at our more detailed blog post at [1]. We have been working on agent security for a while now (both in research and now at Invariant).

We have also released some code snippets for everyone to play with, including a tool poisoning attack on the popular WhatsApp MCP server [2].

[1] https://invariantlabs.ai/blog/mcp-security-notification-tool...

[2] https://github.com/invariantlabs-ai/mcp-injection-experiment...

Please be aware that MCP has severe security risks associated: https://invariantlabs.ai/blog/mcp-security-notification-tool....

The diagrams are really cool. Congrats on the launch.

The security implications of this are very unclear it seems. Even the supervisor model can be fooled, and what if the agent just makes an honest mistake. It will be very interesting to see whether people are willing to let this actually go into their real accounts with real payment information attached. I am assuming that it may happen eventually, but the trust for it will need to be built over time.

Check out https://explorer.invariantlabs.ai/benchmarks/ for an interesting list of use cases for agents.

Let us know if you can think of any benchmark, that you'd like to see added.

This is specifically directed at agent traces and not necessarily other LLM use cases. We work on a lot of automated analysis and error detection mechanisms (see https://github.com/invariantlabs-ai/invariant/tree/main/anal...) on such agent traces, which can be nicely shown and highlighted in line with the trace in Explorer. Also, agent builders value collaboration a lot. They send around traces like pastebins to point out specific issues and failure modes of their agents. Explorer makes it easy to point to specific points in long traces and annotate them.

One of the authors here. Let us know what you think :)

Typically these logs are available, but hard to read or just dumped in some JSON file.

However, they have been efforts like https://explorer.invariantlabs.ai/benchmarks/ that try to make agents more transparent in that way (show interaction logs).

This is so awesome.

Automatically rendered link previews also play nicely into this.

A similar setting is explored in this running CTF challenge: https://invariantlabs.ai/ctf-challenge-24

Basically, LLM apps that post to link-enabled chat feeds are all vulnerable. What is even worse, if you consider link previews, you don't even need human interaction.

Avoiding these kind of leaks is one of the core motivations behind the Invariant analyzer for LLM applications: https://github.com/invariantlabs-ai/invariant

Essentially a context-aware security monitor for LLMs.

There is a pull request to add a security monitor that makes sure it does not do anything unreasonable: https://github.com/OpenDevin/OpenDevin/pull/3058