jacobgold

He was our Andy Rooney. RIP.

Not sure how it would? Because the bad actors are remotely instructing infected computers/devices to make HTTP/TLS requests for them, so they appear totally normal to the other side.

These are not open proxies.

Okay, I was being imprecise. These residential proxies aren't "open proxies" in the traditional sense, but they're usually "open" to anyone willing to pay a small amount of money to use them.

Most ISP abuse reports are routinely ignored. They might as well be a dead letter box.

This is where regulation might play a role, or at least a change in attitude. Companies shouldn't be allowed to pollute the internet in this way when they can easily prevent it.

In terms of law enforcement, sure. But what we really need is competent white hat hackers doing battle with the black hats.

Even if no one goes to jail, the NSA could make residential proxies much harder to operate in the US simply by detecting them and reporting them to ISPs. It would be good for ISPs to then validate the reports properly, give warnings, etc.

It may actually be costing them money through higher peak usage (requiring more capacity). I'm not sure but either way the government should have a role if market incentives aren't sufficient.

There's no spying required. The NSA and ISPs can find open proxies through infiltration and report them to (e.g. abuse@comcast.com), then Comcast simply has to act on it robustly.

ISPs already deal with abuse reports like this, the system just isn't being operated comptently.

US residential proxies are the bane of the internet. They're the major source of social media manipulation and spam.

Services can dramatically reduce abuse by blocking entire IP ranges based on country of origin, organization, or type (hosting providers). But a company can't block US residential IPs if it would also cut off many of their real customers.

The US government (probably the NSA) should be cracking down hard on US residential proxy networks. They're a genuine national security threat, actual data/identity loss of American citizens, act as infra for foreign covert influence campaigns, botnets used in hacking/DoS attacks, etc.

Major US ISPs (Comcast, AT&T) should detect clearly suspicious activity coming from customer IPs and warn them to scan their computers, check their TV apps, and find whatever is turning their internet into a proxy. It's very bad for the customers too (slows things down, gets their IP banned, etc)

There's a new feature being added to AT to support "permissioned data" that would handle a Slack-like use-case: https://github.com/bluesky-social/proposals/tree/main/0016-p...

Slack exists largely because IRC was insufficient. It didn't natively support channel history, search, etc.

For AI agents to flourish, Slack has to either truly open its network with a protocol or eventually be replaced.

I'd like to see Slack embrace an AT protocol-based chat system, which then apps like Buzz could implement. Then users could log in with domain handles like @yourname.com, and agents could use handles like @agent1.yourname.com, all under their complete control.

Anyone is free to draw whatever analogies they want, but they either make sense or they don't. Outside of philosophical discussions or science fiction, comparing influencing humans to prompt injection is silly.

That's a fair point. I agree that the behavior can look similar even when the mechanism is different.

But in practice, all the analogies I've seen are in fact superficial, including this one.

The LLM that abruptly switches languages will also likely switch to a wildly unrelated topic. If a human behaved that way, you'd call a doctor.

In the past, when people built programs to extract text from PDFs, they didn't wrap them in a chat interface which claimed it had the human cognitive ability to "read" human languages.

They could have done this. They could have claimed they'd recreated human vision and hyped it as the beginning of a full human brain, but they didn't.

Instead, they used real technical terms like "OCR" (optical character recognition), which gave people a much more accurate understanding of the technology and didn't encourage silly analogies to humans.

What I'm pushing back on is, apparently, that some people genuinely believe these LLM-based computer programs are human-like intelligences.

In reality, they're more like very good search engines that output relevant snippets of text. If you run them in a loop (feeding them their output as input) you can make them return even better search results.

The software developers who created these systems used sexy words like "reasoning" and "thinking" to describe this search process. They used words like these because they're trying to make money and it sounds cool, not because they've actually re-created human cognition.

So just like me, including the prompt injections if you count "nerd sniping" as such?

Who would count that as prompt injection? It's a superficial analogy.

If you were vulnerable to prompt injection, I could order you to do absolutely anything you're capable of doing and you would be helpless to do otherwise.

Yes, people speak multiple languages and switch between them. But it's a superficial analogy to the behavior of LLMs which do something different and for different reasons.

https://en.wikipedia.org/wiki/Ad_hominem

This isn't even controversial. The proof is available to anyone who uses these systems:

They hallucinate tool state, drift from the objective while seeming to comply, switch languages randomly (Cyrillic or Japanese characters in output), confuse tasks they've planned for completed ones, and of course follow prompt injections embedded in files or web pages.

The idea that anyone would think anyone else would think that is fairly revealing.

Taking issue with using a computer to power the calculator? If so, you should know that all modern calculators are computers under the hood.

Everything I've written about calculators applies to computers doing any kind of traditional deterministic processing, without anything like LLMs.

Yeah it's a trick question, the human error rate for it was about 30% (higher depending on the country).

Error rate doesn't prove anything. The nature of the errors is what matters.

Sorry to tell a fellow Jacob that you're the one who is out of date. The kids are calling everything "AI" and they mean "AGI", and that's the complaint.

If an alien landed we could decide whether it seems to have a human-like intelligence or not. It could be incredibly intelligent but very non-human-like.

If you're using the existence of flaws in LLMs to deny the claim of intelligence to them...

That's not the point at all. It's the fact that they fail in ways completely unlike humans.

You also have the burden of proof reversed. Its on you to prove these LLM agents are human-like intelligences if that's your claim. No one can prove this because it's false.

If someone unskilled at math brings a calculator to an international math competition, they will not succeed at solving many problems.

But if they bring a frontier LLM (and succeed at concealing it from the organizers), they can walk away with a gold medal.

Of course you could win all kinds of math competitions with a concealed calculator. Maybe you'd need a fancy one, like a little SBC running Python. Anything complex and timed would be easy to win. You'd look like a genius to anyone who didn't know you had it.

Such a feat requires intelligence... and if the contestant didn't provide the intelligence himself/herself, where'd it come from?

From computer software running on computer hardware, just like a calculator.

Calculating trillions of digits of pi also requires intelligence far beyond human capacity.

Computers displaying intelligence doesn't imply human-like intelligence. This is the source of confusion.

I get what you're saying but this is kind of a semantic game.

These LLM models/agents absolutely do not reason in the sense that humans do, so you're quietly redefining the word.

You can say of course decide to call them an "alien kind of intelligence" that "reasons" but you could just as reasonably say that calculators are an "alien" kind of intelligence that "reasons" about math differently than us.

Drawing the conclusion that "humans fail" and "models fail", so they must be similar, is very wrong.

You could have humans calculate 2+2 all day and get a surprisingly high error rate. That reveals a flaw in how humans operate.

LLMs fail for entirely different reasons. Their mistakes don't imply they're human-like at all.

It's not about the error rate.

Or you can keep calling them stochastic parrots as they solve decades-old open problems.

I didn't use that phrase at all. But computers calculated digits of π to trillions of digits. With a chat interface for a Python math program would look like the most impressive math genius if you took it back a few decades.

The real question is how useful they are...

That's not the "real question" but an entirely different question that is easily answered. Nothing I wrote suggested they're not incredibly useful.

Data from Star Trek TNG failing to understand figures of speech.

These are just little instances of bad writing. Data is very much an attempt at displaying a human-like intelligence.

We didn't have confused people claiming calculators were human-like intelligences doing math.

There are so many other sites. So many others. Why are you here?

I've been here since 2007 when HN launched.

You're confused about my objection. I don't like the term "AI" but I love the technology as much as almost anyone.

We could've slapped a chat interface on calculators and called them "AI" because they can do superhuman math instantly. Most technical people would've thought that was stupid.

LLMs are the same kind of category mistake.