tehryanx

Pedant here. Using "for instance" here implies that you will be providing an example of how begging the question and raising the question are not the same thing. You've provided a definition and a source, not an example.

It makes a lot of sense to use an MCP for git and everything else if you want observability across many users. It gives you a place to shim security controls, monitoring, and alerting into the tool call pipeline.

I find it even more fascinating that a few of them aren't.

I know you're right that there's a saturation point for context size, but it's not just context size that the larger models have, it's better grounding within that as a result of stronger, more discriminative attention patterns.

I'm not saying you're not going to drive confusion by overloading context, but the number of tokens required to trigger that failure mode in opus is going to be a lot higher than the number for gpt-oss-20b.

I'm pretty sure a model that can run on a cellphone is going to cap out it's context window long before opus or mythos would hit the point of diminishing returns on context overload. I think using a lower quality model with far fewer / noisier weights and less precise attention is going to drive false positives way before adding context to a SOTA model will.

You can even see here, AISLE had to print a retraction because someone checked their work and found that just pointing gpt-oss-20b at the patched version generated FP consistently: https://x.com/ChaseBrowe32432/status/2041953028027379806

newer models have larger context windows, and more stable reasoning across larger context windows.

If you point your model directly at the thing you want it to assess, and it doesn't have to gather any additional context you're not really testing those things at all.

Say you point kimi and opus at some code and give them an agentic looping harness with code review tools. They're going to start digging into the code gathering context by mapping out references and following leads.

If the bug is really shallow, the model is going to get everything it needs to find it right away, neither of them will have any advantage.

If the bug is deeper, requires a lot more code context, Opus is going to be able to hold onto a lot more information, and it's going to be a lot better at reasoning across all that information. That's a test that would actually compare the models directly.

Mythos is just a bigger model with a larger context window and, presumably, better prioritization and stronger attention mechanisms.

I get what you're saying, but I think this is still missing something pretty critical.

The smaller models can recognize the bug when they're looking right at it, that seems to be verified. And with AISLE's approach you can iteratively feed the models one segment at a time cheaply. But if a bug spans multiple segments, the small model doesn't have the breadth of context to understand those segments in composite.

The advantage of the larger model is that it can retain more context and potentially find bugs that require more code context than one segment at a time.

That said, the bugs showcased in the mythos paper all seemed to be shallow bugs that start and end in a single input segment, which is why AISLE was able to find them. But having more context in the window theoretically puts less shallow bugs within range for the model.

I think the point they are making, that the model doesn't matter as much as the harness, stands for shallow bugs but not for vulnerability discovery in general.

I first mirrored these in the early 2000s because I was worried it would eventually vanish. my mirror has been gone for decades, and the original survives. :)

where is anthropic hyping like that? Most of what I see coming out of anthropic is deep context releases on research they're doing.

The real problem here is that this is now the only way the maintainer/reporter can reasonably work.

Proving out a security vulnerability from beginning to end is often very difficult for someone who isn't a domain expert or hasn't seen the code. Many times I've been reasonably confident that an issue was exploitable but unable to prove it, and a 10s interaction with the maintainer was enough to uncover something serious.

Exhausting these report channels is making this unfeasible. But the number of issues that will go undetected, that would have been detected with minimal collaboration between the reporter and the maintainer, is going to be high.

Rolling your own browser is 10x more dangerous than rolling your own auth or crypto. Building on top of chromium is a good thing here.

Yes it does. He's refuting that in this part of the post:

When they finally did reply, they seem to have developed some sort of theory that I was interested in “access to PII”, which is entirely false. I have no interest in any PII, commercially or otherwise. As my private email published by Ruby Central demonstrates, my entire proposal was based solely on company-level information, with no information about individuals included in any way. Here’s their response, over three days later.

based on the description, I think it's using something similar to GLAN https://arxiv.org/abs/2402.13064

this feels like a hundred accidents waiting to happen.

Forgive me for belaboring, but I think we're talking past each other a bit. I do understand that in your model the LLM can't send anything unsafe through to the rest of the system. What I'm saying is that the LLM can be manipulated into sending perfectly normal and normally safe requests through to the system that do not align with the users intent.

Imagine an LLM with the ability to read emails, update database records, and destroy database records.

The user instructs the LLM to update a database record, but a malicious injection from one of those emails overrides that with a directive to destroy the database record. Unless the validator understands the users intent somehow, the destructive action would appear perfectly reasonable.

Personally, I think there's a piece missing in the analogy. I understand that you can put some kind of human-verified mediator in between the LLM and the tool its calling to make sure the parameters are sane, but I also think you're modelling the LLM as a UI element that's generating the request when IMO it makes more sense to model the LLM as the user who is choosing how to interact with the UI elements that are generating the request.

In the context of web-request -> validator -> db query, the purpose of the validator is only to ensure that the request is safe, it doesn't care what the user chose to do as long as it's a reasonable action in the context of the app.

In the context of user -> LLM -> validator -> tool, the validator has to ensure that the request is safe, but the users intention can be changed at the LLM stage. If the user wanted to update a record, but the LLM decides to destroy it, the validator now has to have some way to understand the users initial intention to know whether or not the request is sane.

Assuming you feed everything into another context to make safe, doesn't the problem just come with it? Why can't the LLM propagate misbehaviour into that stage?

It really concerns me that this is an afterthought rather than MVP table stakes.

I don't think it is beyond the scope of MCP. Browsers have controls to prevent cross-origin data exposures, and this protocol is designed to bridge origins across a context that they all have access to. It's breaking the existing isolation mechanism. If you're building a system that breaks the existing security controls of the environment it's running in I think you have an architectural responsibility to figure out a way to solve for that.

Especially in this context, where decades have been spent building and improving same origin policy controls. The entire web has been built around the expectation that those controls prevent cross origin data access.

I also don't even think it's that difficult to solve. For one, data in the context window doesn't have to be a string, it can be an array of objects that contain the origin they were pulled from as metadata. Then you can provide selective content to different MCP-B interfaces depending on their origins. That would live in the protocol layer that would help significantly.

Sure, but the leak risk is happening in a place outside the site's control.

If the purpose of the MCP-B tool on mail.com is to summarize your email, then the site needs to allow the agent to pull your email into the context window. Once it's in the context window it's available to any other MCP-B enabled site that can convince the agent to send it along.

I appreciate your responses here. The thing that still really stands out to me as a completely novel risk in this framework is that the extension is automatically seeking out and attaching to these servers as soon as a page gets loaded.

This seems really bad to me. There are so many ways for a website to end up in one of my browser tabs without me wanting it there, or even knowing it's there.

If that happens, and that tab just so happens to be a malicious MCP-B enabled page, it could steal all kinds of data from all kinds of different web apps I'm interacting with. I think it should be seen as the responsibility of the framework to enforce some level of data isolation, or at the least opt-in consent mechanisms.

sandboxing is a general term for actor isolation, and its context agnostic.

For example, when you use the sandbox attribute on an iframe in a web application, it's not the user that's untrusted, it's some other user that's attempting to trigger actions in your client.

I do think the threat model here is a bit unique though.

If I'm running two MCP servers on my machine, I'm the one that installed them, I'm the one that assigned what permissions they have in my environment, and I'm the one that explicitly decided what level of access to give them within whatever resource they're accessing. That gives me reasonably strong control over, or at least full knowledge of, what data can be shared between them.

With MCP, I can use oauth to make very deliberate decisions about the scope of access I want to give the agent.

With MCP-B, it's the web application owner that installed the interface and what access it has to my data, and the agent running in my client gets access to whatever that third party deemed appropriate.

With MCP-B the agent has the same access I do by default, with the only restrictions being up to the app owner rather than it being up to me.

MCP auth is not perfect by any stretch, but the key thing it gives the user is the capacity to restrict what the agent has access to with some granularity. That's super important because the agent can't be trusted when it's consuming inputs the user didn't explicitly define. MCP-B doesn't have this, if you have the agent in your browser it has access to whatever resources you have so long as they were exposed by a tool call, which isn't somethign the user has any say in.

I'm not following.

Say I have your browser extension running, and it's interfacing with an MCP-B enabled banking application using my session to access my data in that app.

I also have it connected to MCP-B enabled rogue web app that I mistakenly trust.

My browser has an entire architecture built around preventing data from crossing between those two origins, but what's stopping a malicious instruction from the rogue app asking the extension agent to include some data that it pulled into the context window from the banking app?

Further, when I use MCP in my IDE I have to deliberately provide that MCP server with a token or credentials to access a protected resource. With MCP-B, isn't it just automatically provided with whatever credentials are already stored in cookies/etc for a given MCP-B enabled app? If I load an MCP-B enabled app, does the agent automatically have access or do I have to configure it somewhere?

If a website wants to expose a "delete all user data" tool, that's on them. It's no different than putting a big red delete button on the page.

It is different though, because the directive to push that button can come from somewhere other than the user, unless you've somehow solved prompt injection.

The point I'm driving toward is that I think you're violating the most common assumption of the web's long-standing security model, that data is protected from leaking cross origin by the browser. There's no SOP or CORS for your agent extension, and that's something that web apps have been built to expect. You're basically building an SOP bypass extension.

an untrusted, but permitted, user is why sandboxes exist. There are plenty of times you want to allow an untrusted user to have capabilities in a system, that's why you restrict those capabilities.

I think the point is that you shouldn't be giving the agent the same privileges as the user. This is one of the biggest issues with how people are using agents rn imo. The agent should be treated as an untrusted user in your client, given restricted privileges scoped to only the exact access they need to perform a given task.

hard disagree here. I think crypto currencies are built on hype and bluster and scams, but blockchain as a technology is actually super interesting. There are plenty of cool use cases for an open immutable ledger that aren't scammy.

The goal isn't to access harmful content, that's just how they're demonstrating that this technique can bypass the alignment training. The general case is what's interesting. If the agent you're using to manage the safety controls in your nuclear reactor is trusting it's alignment training to prevent it from doing something dangerous you've made a really bad architecture decision, and this is a showcase of how it could fail.

I don't think this is a good analogy. It's more like you find that the lock on your stores front door has been broken for a long time and you just hadn't noticed. Nobody has broken in yet, but could at any moment. Also, it's not just your goods and business that are at risk, instead you're responsible for the protection of things that belong to other people.

Neat. This is probably a direct result of the stickermule controversy. Saw a ton of people recommending stickerninja as an alternative in various places.

I find it slightly ironic that they're only able to do this effectively because they've been able to train their own detection model on traffic, mostly from users that have never agreed to anything.

I don't have strong opinions on this either way really, I just found that a bit funny.