eric-burel

Infrastructure provisioning is a key ingredient of agentic AI viruses: https://www.ericburel.tech/blog/ai-virus-agent This may be the first steps of the worst wave of spamming campaign ever seen on the Internet. We'll need to reinvent how we connect and communicate via computers.

They love their dictator until it backfires, that's a quite old story.

I've been working on making the "lethal trifecta" concept more popular in France. We should dedicate a statue to Simon Wilinson: this security vulnerability is kinda obvious if you know a bit about AI agents but actually naming it is incredibly helpful for spreading knowledge. Reading the sentence "// indirect prompt injection via email" makes me so happy here, people may finally get it for good.

Can it be used to sandbox an AI agent, like replacing eg Cursor or Openclaw sandboxing system?

The thing is running it onto your machine is kinda the point. These agents are meant to operate at the same level - and perhaps replace - your mail agent and file navigator. So if we sandbox too much we make it useless. The compromise being having separate folders for AI, a bit like having a Dropbox folder on your machine with some subfolders being personal, shared, readonly etc. Running terminal commands is usually just a bad idea though in this case, you'd want to disable that and instead fine tune a very well configured MCP server that runs the commands with a minimal blast radius.

Before using make sure you read this entirely and understand it: https://docs.openclaw.ai/gateway/security Most important sentence: "Note: sandboxing is opt-in. If sandbox mode is off" Don't do that, turn sandbox on immediately. Otherwise you are just installing an LLM controlled RCE.

There are still improvements to be made to the security aspects yet BIG KUDOS for working so hard on it at this stage and documenting it extensively!! I've explored Cursor security docs (with a big s cause it's so scattered) and it was nothing as good.

Related: https://news.ycombinator.com/item?id=46223311 Switching to paid version was already problematic as it uses GCP account system which doesn't have a spending limit, and API keys do not have an expiration date. So the free offer was great for freelancers and SMEs and yet the paid version was the worst possible scenario you can imagine for same freelancers and SMEs. OpenRouter + free models and the increased rate limit you get after buying 10 credits (10€) is my current favourite choice for learning/teaching.

Microsoft is using the deep penetration of SharePoint in companies to sell Copilot license. At least in France it's well and alive and I see much more Copilot licenses than actual OpenAI uses.

Yeah I mean you can replace sandboxing buy other safe alternatives but the idea is the same, the generated code has to be considered as 100% untrusted. Supply chain attacks are especially nasty.

Talking about the debt of a system prompt feels really weird. A system prompt tied to an LLM is the equivalent of crafting a new model in the pre-LLM era. You measure their success using various quality metrics. And you improve the system prompt progressively to raise these metrics. So it feels like bandaid but that's actually how it's supposed to work and totally equivalent to "fixing" a machine learning model by improving the dataset.

In an agentic setup you are still dependent on having relatively smart models that's true.

You can have an agent that focuses on studying the interactions. What you're saying is that an AI cannot find every security issue but neither do humans otherwise we wouldn't have security breaches in the first place. You are describing a relatively basic agentic setup mostly using your AI-assisted text editor but a commercial security bot is a much more complex beast hopefully. You replace context by memory and synthesis for instance, the same way our brain works.

Sounds a lot like "nationalism in a single country"

You may want to read about agentic AI, you can for instance call an LLM multiple times with different security consideration everytime.

Smart, thanks for the tip

AI is good for discovery but not validation, I wanted experienced human feedback here

"You can investigate this yourself by putting a logging proxy between the claude code CLI and the Anthropic API using ANTHROPIC_BASE_URL" I'd be eager to read a tutorial about that I never know which tool to favour for doing that when you're not a system or network expert.

I call that self-destructive prompting in the sense that you use AI to output programs that replace calling the AI in the future. The paper seems to indicate that this also brings much better results. However it's subject to attacks as running generated code is usually unsafe. A sandbox has to be used, major agentic AI players are providing some solutions, like Langchain sandbox released earlier this year.

I am pretty sure you can figure massive loopholes like how it's legal to train the model on stolen data but not to steal data etc. For instance advertisers can push model benchmarks that favours some opinions, based on a biased selection of research papers. I think we've only seen the beginnings of what intricate business models can be figured for an AI company, it's much more convoluted than a search engine or even a social network.

"that could redefine the web economy" I don't think that ads in ChatGPT are that disruptive, it's just another channel. I think ChatGPT apps are an order magnitude more game changing, as they are not a new markting channel but a new distribution channel for software. Your next ad will still be an ad, but your next SaaS might be a ChatGPT App.

You don't let customer to decide if they love pesticide or not, their are basic functions even to a minimal state and environment and health protection is among them.

If you are an European, regulation also has the benefit to induce soft protectionism from countries that are less keen on consumer and environment protection. This is the heart of the debate about Mercorsur, as it creates an unfair competition by lowering regulation (in theory european regulation applies but in practice it's harder to verify), and also an internal debate in France related to some pesticide that other European countries can use. Some argue that we should allow the pesticide, some that we should stop importing products that are exposed to it.

That's a shortcut, llm providers are very short sighted but not to that extreme, alive websites are needed to produce new data for future trainings. Edit: damn I've seen this movie before

Yeah but can't we expect bureaucratic companies to adopt such a methodology exactly like that: write a spec for years, run the LLM agent every 6 months, blame techs for the bad result, iterate, and also forbid coding outside of the spec.

I would simply replace LLM by agent in your reasoning, in the sense that you'll need a strong preprocessing step and multiple iterations to exploit such complete specs.

When you take the highway you are expected to respect a few rules but the highway builder is not supposed to tell you how to dress in your car. Theses platforms are the closest that can be to public spaces and they are managed as totally autonomous blackbox, I don't like that.

If you want to sell anything as an SME almost anywhere in the world, you have to play by Google's obscure SEO rules. It doesn't make it right. I feel the same vibe with this kind of rules. Are thise guidelines negotiated with unions or federation that includes the content producers, actors, authors? If yes, is decision power equally balanced? Who writes the rules?

Makes sense, sorry. Was still making a sarcastic point that it's objectively difficult to be strongly against regulating things with basically no upside, though I understand that you can be against regulations in general.

I am thinking of building an association of AI consumers so we can organize to praise or boycott whatever we collectevily find acceptable or not. I'll spend some time reading this in details later on, but whatever it states or imply, positive or negative, it's not for businesses to set the rules as if they owned the place. Consumer associations are powerful and can't be fired when striking, since the customer is always right.

The keywords seems to bring an anomalous amount of trolls in the comments. But maybe I am in the wrong, and there's a positive side into leaking plastic pellets in the ocean. Like turtles can now build their own platic industry and we'll make business with them soon or something.