stoniejohnson

Great post! I appreciate the diagrams.

If you're always looking up but never looking down it becomes hard to appreciate things.

Revisited this, and while it was tongue-in-cheek, it feels a bit ungrateful.

A lot of tech roles are intellectually stimulating and have a wage in the upper distribution of salaries. I think we have a very nice situation going for us.

While those are good reasons to keep the weights static from a business perspective, they are not the only reasons, especially when serving SOTA models at the scale of some of the major shops today.

Continual/online learning is still an area of active research.

They were high in 2013.

I get where you are coming from, and it is definitely an interesting thought!

I do think it is an extremely inefficient way to have a swarm (e.g. across time through training data) and it would make more sense to solve the pretraining problem (to connect them to the external world as you pointed out) and actually have multiple LLMs in a swarm at the same time.

I was responding to the idea that an LLM would believe (regurgitate) untrue things if you pretrained them on untrue things. I wasn't making a claim about SOTA models with gigantic training corpora.

I was responding to the back and forth of:

If you pretrained an LLM with data saying Moscow is the capital of Connecticut it would think that is true.

Well so would a human!

But humans aren't static weights, we update continuously, and we arrive at consensus via communication as we all experience different perspectives. You can fool an entire group through propaganda, but there are boundless historical examples of information making its way in through human communication to overcome said propaganda.

Humans are not isolated nodes, we are more like a swarm, understanding reality via consensus.

The situation you described is possible, but would require something like a subverting effort of propaganda by the state.

Inferring truth about a social event in a social situation, for example, requires a nuanced set of thought processes and attention mechanisms.

If we had a swarm of LLMs collecting a variety of data from a variety of disparate sources, where the swarm communicates for consensus, it would be very hard to convince them that Moscow is in Connecticut.

Unfortunately we are still stuck in monolithic training run land.

They have a dark pattern around annual subscriptions, i.e. I had a monthly subscription I wanted to downgrade, and with no confirmation they charged me for the entire year after I selected the lower tier and hit next.

This is probably why.

I issued a chargeback.

I'm not really familiar with that technology space, but if you take that as true, is your argument something like:

- We don't have limitless CPU cycles

- Thus we need to split things into sub-problems

If so that might still be amenable to the bitter lesson, where Sutton is saying human heuristics will always lose out to computational methods at scale.

Meaning something like:

- We split up the thought to vision problem into N sub-problems based on some heuristic.

- We develop a method which works with our CPU cycle constraint (it isn't some probe -> CPU interface). Perhaps it uses our voice or something as a proxy for our thoughts, and some composition of models.

Sutton would say:

Yeah that's fine, but if we had the limitless CPU cycles/adequate technology, the solution of probe -> CPU would be better than what we develop.

I think the bitter lesson implies that if we could study/implement "how a machine with limitless cpu cycles would make our eyes see something we are currently thinking of" then it would likely lead to a better result than us using hominid heuristics to split things into sub-problems that we hand over to the machine.

Yeah that tracks, I just feel like the difference required to be an ASI might allow for a stronger claim here.

I didn't mention it but I fully agree, I imagine ASI would have be to embodied.

My reasoning is simple, there are a whole class of problems that require embodiment, and I assume ASI would be able to solve those problems.

Regarding

Point 1 is a big assumption. I am also not you, and although it's true that I have different goals, I share most of your human moral values and wish you no specific harm.

Yeah I also agree this a huge assumption. Why do I make that assumption? Well, to achieve cognition far beyond ours, they would have to be different from us by definition.

Maybe morals/virtues emerge as you become smarter, but I feel like that shouldn't be the null hypothesis here. This is entirely vibes based, I don't have a syllogism for it.

I agree the premise of FOOM is very unlikely.

But if you assume that we have created something that is agentic and can reason much faster and more effectively than us, then us dying out seems very likely.

It will have goals different from ours, since it isn't us, and the idea that they will all be congruent with our homeostasis needs evidence.

If you simply assume:

1. it will have different goals (because it's not us)

2. it can achieve said goals despite our protests (it's smarter by assumption)

3. some goals will be in conflict with our homeostasis (we would share resources due to our shared location, Earth)

then we all die.

I just think this is silly because of the assumption that we can create some sort of ASI, not because of the syllogism that follows.

(As an intuition pump, we can hold on the order of ones of things in our working memory. Imagine facing a foe who can hold on the order of thousands of things when deciding in real time, or even millions.)

obligatory IANAL, but seeing LLMs:

- regurgitate entire passages word for word, until that behavior is publicized and quickly RLHF'd away

- rip github repos almost entirely (some new Sonnet 3.5 demos Anthropic employees were bragging about on Twitter were basically 1:1 to a person's public repo)

It seems clear to me that not only can copyrighted work be retained and returned in near entirety by the architectures that undergird current frontier models, but the engineers working on these models will readily confuse a model regurgitating work to be "creating novel work".

On one hand, my brain kinda shuts off when people start applying structure to spirituality.

On the other hand, doing an extensive search amongst possible head spaces you can occupy is a no-brainer for a consciousness implemented on a primate.

All people do is confabulate too.

Sometimes it is coherent (grounded in physical and social dynamics) and sometimes it is not.

We need systems that try to be coherent, not systems that try to be unequivocally right, which wouldn't be possible.

Language is the medium through which raw perspective refined itself.

Language birthed social games and the sense of self.

Yes, language evolved for communication.

But without communication, thought would still be stuck in the land of instinct, never forged by the tribal dances of love, art, deceit, debate and organization.

Given your initial assumptions, that self-moderating end state makes sense.

I feel like we still have a disconnect on our definition of a super intelligence.

From my perspective this thing is insanely smart. We can hold ~4 things in our working memory (maybe Von Neumann could hold like 6-8); I'm thinking this thing can hold on the order of millions of things within its working memory for tasks requiring fluid intelligence.

With that sort of gap, I feel like at minimum the ASI would be able to trick the cleverest human to do anything, but more reasonably, humans might appear to be entirely close formed to it, where getting a human to do anything is more of a mechanistic thing rather than a social game.

Like the reason my early example was concrete pillars with weird wires is that with an intelligence gap so big the ASI will be doing things quickly that don't make sense, having a strong command over the world around it.

Totally fair points.

I cannot make the leap from "super intelligent" to "has access to all the levers of social and physical systems control" without the explicit, costly, and ongoing, effort of humans.

Yeah this is a fair point! The super intellect may just convince humans, which seems feasible. Either way, the claim that there are 0 paths here for a super intelligence is pretty strong so I feel like we can agree on: It'd be tricky, but possible given sufficient cleverness.

I see no reason to believe that the emergent properties of a highly complex system will include free will.

I really do think in the next couple years we will be explicitly implementing agentic architectures in our end-to-end training of frontier models. If that is the case, obviously the result would have something analogous to goals.

I don't really care about it's phenomenal quality or anything, it's not relevant to my original point.

Valid call out, this was firmly motivated by it being a super intelligence.

I feel like if you are an intelligent entity propagating itself through spacetime you will have goals:

If you are intelligent, you will be aware of your surroundings moment by moment, so you are grounded by your sensory input. Otherwise there are a whole class of not very hard problems you can't solve.

If you are intelligent, you will be aware of the current state and will have desired future states, thus having goals. Otherwise, how are you intelligent?

To make this point, even you said "A super intelligent species would likely want to preserve everything", which is a goal. This isn't a gotcha, I just feel like goals are inherent to true intelligence.

This is a big reason why even the SOTA huge frontier models aren't comprehensively intelligent in my view: they are huge, static compositional functions. They don't self reflect, take action, or update their own state during inference*, though active inference is cool stuff people are working on right now to push SOTA.

*theres some arguments around what's happening metaphysically in-context but the function itself is unchanged between sessions.

So I think we have some differences in definition. I am assuming we have an ASI, and then going on from there.

Minimally an ASI (Artificial Super Intelligence) would:

1. Be able to solve all cognitively demanding tasks humans can solve and tasks humans cannot solve (i.e. develop new science), hence "super" intelligent.

2. Be an actively evolving agent (not a large, static compositional function like today's frontier models)

For me intelligence is a problem solving quality of a living thing, hence point 2. I think it might be the case to become super-intelligent, you need to be an agent interfacing with the world, but feel free to disagree here.

Though, if you accept the above formulation of ASI, then by definition (point 2) it would have goals.

Then based on point 1, I think it might not be as simple as "If the AI does something in the physical world which we do not like, we sever its connection."

I think a super-intelligence would be able to perform actions that prevent us from doing that, given that it is clever enough.

here is an ungrounded, non-realistic, non-representative of a potential future intuition pump to just get the feel of things:

(Yes, there are many holes in this, like how would it piggy back off of our infrastructure if it kills us, but this isn't really supposed to be coherent, it's just supposed to give you a sense of direction in your thinking. Generally though, since it is superintelligent, it can pull off very difficult strategies.)

If you read the above I think you'd realize I'd agree about how bad my example is.

The point was to understand how orthogonal goals between humans and a much more intelligent entity could result in human death. I'm happy you found a form of the example that both pumps your intuition and seems coherent.

If you want to debate somewhere where we might disagree though, do you think that as this hypothetical AI gets smarter, the interface between it and the physical world becomes more guaranteed (assuming the ASI wants to interface with the world) and less tenuous?

Like, yes it is a hard problem. Something slow and stupid would easily be thwarted by disconnecting wires and flipping off switches.

But something extremely smart, clever, and much faster than us should be able to employ one of the few strategies that can make it happen.

I think the common line of thinking here is that it won't be actively antagonist to <us>, rather it will have goals that are orthogonal to ours.

Since it is superintelligent, and we are not, it will achieve its goals and we will not be able to achieve ours.

This is a big deal because a lot of our goals maintain the overall homeostasis of our species, which is delicate!

If this doesn't make sense, here is an ungrounded, non-realistic, non-representative of a potential future intuition pump to just get the feel of things:

We build a superintelligent AI. It can embody itself throughout our digital infrastructure and quickly can manipulate the physical world by taking over some of our machines. It starts building out weird concrete structures throughout the world, putting these weird new wires into them and funneling most of our electricity into it. We try to communicate, but it does not respond as it does not want to waste time communicating to primates. This unfortunately breaks our shipping routes and thus food distribution and we all die.

(Yes, there are many holes in this, like how would it piggy back off of our infrastructure if it kills us, but this isn't really supposed to be coherent, it's just supposed to give you a sense of direction in your thinking. Generally though, since it is superintelligent, it can pull off very difficult strategies.)

they said they have a large action model, but allegedly they just have playwright scripts that break as rideshare apps update

they said they are faster than chatgpt, but allegedly they are a chatgpt wrapper

thanks!

my name is deceptive I am a respectful denizen of this beautiful nation and I don't approve of drugs!

he asked "whats wrong" with the following things and the last example was possessing an extremely illegal narcotic in a motor-vehicle

in the glove compartment but exercising self-control?

this example is very different from the others haha