markburns

What explains the emergent abilities of generative pre-trained transformers at massive-scale?

I don't see why the abilities couldn't be an encoded modelling of enough of the world to produce those abilities. It seems like a simple enough explanation. Less data, less room to build a model of how things work. More data, sufficient room to build a model.

Conway's Game of Life is then not conscious in and of itself, because there's not enough in its encoded data to result in emergent behaviour beyond what we see.

If we expand it to also include a vast amount of data such as a Turing machine running an LLM then we can reasonably say we are closer to saying that that configuration of it is conscious.

It's not the firing-of-neurons mechanism and its relevant complexity or simplicity that make us conscious or not.

It's not the GoL algorithm that would make the machine conscious either.

It's the emergent behaviour of a sufficiently complex system.

The system _including_ its data.

To the first questions. No and no. But potentially where consciousness lives is emergent behaviour in systems with iterative feedback loops.

https://en.wikipedia.org/wiki/I_Am_a_Strange_Loop

I personally think we'll need a few more feedback loops before you have more human-like intelligence. For example, a flock of LLM agent loops coming to consensus using short-term and long-term memory, and controlling realtime mechanical, visual and audio feedback systems, and potentially many other systems that don't mimic biological systems.

I also think people will still be debating this way beyond the singularity and never conceding special status to intelligence outside the animal kingdom or biological life.

It's quite a push for many people to even concede animals have intelligence.

For the extraordinary claims/evidence, it's also the case that almost any statement about what consciousness is in terms of biological intelligence is an extraordinary claim that goes beyond any evidence. All evidence comes from within the conscious experience of the individual themselves.

We can't know beyond our own senses whether perception exists outside of our own subjective experience. We cannot truly prove we are not a brain in a jar or a simulation. Anything beyond assertions about the present moment and the senses that the individual experiences are just pure leaps of faith based on the persistent illusion, or perceived persistent illusion of reality (or not).

We know really nothing of our own consciousness and it is by definition impossible to prove anything outside of it, from inside the framework of consciousness.

If we can somehow find a means to break outside of the pure speculation bubble of thoughts and sensations and somehow prove what human experience is, then we may be in a position to make assertions about missing evidence for other forms of intelligence or experience.

But until then definitions of both human and artificial intelligence remain an exercise for the reader.

People often point to the relative simplicity of the architecture and code as proof that the system can’t be doing whatever it is that consciousness does, but in doing so they ignore the vast size of the data those simple structures are operating over. Nobody can actually say whether consciousness is just emergent behaviour of a sufficiently complex system, and knowing how a system is built tells you nothing about whether it clears the bar for that kind of emergence. Architectural simplicity and total system complexity aren’t the same thing.

Ie the intelligence sits in the weights and may sit there in the synapses in our brains too.

When we talk about machines being simple mimicking entities we pay no attention to whether or not we are also simple mimicking entities.

Most other assertions in this topic regarding what consciousness truly is tend to be stated without evidence and exceedingly anthropocentric whilst requiring a higher and higher bar for anything that is not human and no justification for what human intelligence really entails.

I don't think "Sitting in an office you sit in every day" or "Sitting in your living room" are the same amount of bandwidth/storage as "Travelling around the moon". I'm sure we have compression algorithms for this stuff and it's somewhat related to novelty.

I'm aware of an association between perception of time to number of photons received in the eyes.

These relate to both how much time the events appear to take subjectively as well as how well remembered they are or how long they feel retrospectively. As in there is an actual physiological explanation for "time flies when you're having fun".

There probably is something to also be said for attention too. Increased awareness and attention will undoubtedly use up more 'bandwidth' or 'storage' too.

I saw a fancy HTML table generator that had so many parameters and flags and bells and whistles that it took IIRC hundreds of lines of code to save writing a similar amount of HTML in a handful of different places.

Yes the initial HTML looked similar in these few places, and the resultant usage of the abstraction did not look similar.

But it took a very long time reading each place a table existed and quite a bit longer working out how to get it to generate the small amount of HTML you wanted to generate for a new case.

Definitely would have opted for repetition in this particular scenario.

https://link.springer.com/article/10.1007/BF00273442

I haven’t read it but I did find this

That seems to me like it just shifts the problem one level. Why are K's and Kikis spiky and why are B's and Boubas round. Why is it universal too across people with different writing systems and languages.

Reflect on the structure of your own comment

Could you clarify, are you comparing the parent comment to the article?

I think in this example those prep work items _are_ doing the thing.

But then telling people about a new product could also be doing the thing.

There’s definitely something to be said for defining what the thing really is being an important part of doing it, but that can also spiral out of control into not doing the thing.

I think thingness is more of a variable property of the current thing you are doing. Than a binary is or isn’t the thing.

All we can really do is regularly check how much the thingness of the current thing is aligned with the main thing’s thingness.

1-3-5-7-2-4

I would imagine that would be red(1), orange(3), yellow(5), green(7), blue(2), purple(4)

But it seems you have white(1), green(3), grey(5), yellow(7), red(2), purple(4)

I can't quite see what going over the rainbow in thirds here means, but I can see why a fifth would be neutral. Could you expand upon this?

Can anyone explain the kind of dense cloud in the middle? Is that down to human perception? We don't give names to things we can't perceive uniquely?

I love the irony in the pitting of the US vs China, Iran & Russia, whilst talking about stoking division.

Don't corollaries to your comments also apply at a higher level globally, or is there something special about considering countries as a grouping vs political parties?

Surely they're all just games we play in our minds and people kind of arbitrarily just agree that countries most definitely exist and this is my in-group, whereas others are enemies.

I'd love to know why this happens so much. There's enough people in both groups that do spot it and don't spot it. I don't think I've ever felt the need for a sarcasm marker when I've seen one. Yet without it, it seems there will always be people taking things literally.

It doesn't feel like something where people gradually pick up on it either over the years, it just feels like sarcasm is either redundantly pointed out for those who get it or it is guaranteed to get a literal interpretation response.

Maybe it's because the literal interpretation of sarcasm is almost always so wrong that it inspires people to comment much more. So we just can't get away from this inefficient encoding/communication pattern.

But then again, maybe I'm just often assuming people mean things that sound so wrong to me as sarcasm, so perhaps there are a lot of people out there honestly saying the opposite to what I think they are saying as a joke.

what set off the spinning?

So these are the really loud behaviours, and not necessarily what I was thinking of.

https://en.wikipedia.org/wiki/Civil_inattention

So there's a thing which I can't find the name of but it's something like a civil inattention sniff. It's a brief sniff at the moment someone walks past you. I'm not saying this one is cultural or geographical. It's possibly universal, I'm making a reference to the signal vs noise ratio.

If you've never noticed the civil inattention sniff, you may start noticing it now, or noticing yourself do it. I was decades on this earth before I picked up on it. It's a similar amount of signal to feeling the weather change in your knees.

A tiny signal in background noise of input of many physical senses.

I believe and suggest that it's true that there are many of these micro behaviours in many different regional cultures, but I'm not stating as fact or backing that up with science here. This is just an exploratory idea for me that I enjoy speculating on.

I had a realisation recently that we’re pretty comfortable with regional dialect borders being an entrenched and normal thing that reach back in history a thousand years or more and that something as specific as how we move our mouths and tongues is strongly correlated geographically.

But we don’t often pay attention to other types of physical and behavioural culture being as geographically entrenched as they sometimes seem to be.

Accents hold some special place in being so recognisable but I think there’s no obvious reason we wouldn’t have many other layers of physical culture like this.

The signal is a bit harder to pick up but I’m sure it’s there.

I’m not trying to make any particular point for or against damaging stereotypes here.

People obeying the speed limit encounter fewer people obeying the speed limit than people speeding and vice versa.

My theory: I think this explains why so many drivers hate other drivers and think they are bad drivers.

People that love to speed think they are good because they can drive faster and react quickly (presumably). They inadvertently see more people that don’t drive in this style.

People who drive carefully encounter more reckless drivers.

a refreshment, please

food or drink?

why do I need to specify. I've been out all day in the heat. Bring me a few refreshments.

That seems an uncharitable view of the reply.

The search space is huge, we sometimes find needles in haystacks by accident, isn’t it exciting that we have tools now that can systematically check every piece of hay?

The punchline is inside the article:

We at Johnny’s Software Lab LLC are experts in performance.

Comment not intended to come across as snidey, just tickled me a bit after reading yours and clicking the archive link.

This is true of ChatGPT 4 with the default prompt maybe but that’s just the way it responds after being given its specific corporate friendly disclaimer heavy instructions. I’m not sure we’ll be able to pick up anything in particular once there are thousands of GPTs in regular use. Which could be already.

But I agree we will probably very often recognise 2023 GPT4 defaults.

It's possible I'm too optimistic on this point. I feel I have a good grasp of how bitcoin works, but when it gets to layer 2 it gets fuzzier. The initial discussions sounded really promising, as they seemed to mirror HTTP on top of TCP/IP. And we all know how much of a slow burning revolution that turned out to be.

But http seems simpler than what I've read about lightning more recently.

Total wild speculation follows:

As for the channels problem, it feels to me like you have something like ISPs or AOL being the thing that's needed to kickstart it. And perhaps the ISPs or AOL in this world are the employers and lightning banks. So yes you need bitcoin to open it, but with an employer you have the potential to have an entity with bitcoin resources and with a reason to send bitcoin to an individual.

Perhaps HD wallets with lightning kickstarter funds apportioned to new employees. This feels similar to the needing a bank account or national insurance number in the UK (social security number? in the US) to start getting paid, or similar to needing an internet connection. Yes you could choose to get paid in bitcoin and have the fees come out of your wage, and this would be similar to getting paid in physical cash today. Or you could get over the slight roadbump to get you onto the lightning network.

What would a real layer 2 look like in your opinion?

There's not really a problematic upper bound for global transaction volume on transactions per second with the lightning network. There are other problems in terms of adoption and infrastructure etc, but the actual potential for humankind to have a single currency global low-barrier payment system is there. But this is more like the early days of the internet. The internet seemed pointless to most people even as late as the 90's.

A future global system could have settlement on the blockchain. Like the current system does at far less regular intervals than the processing power of non-settled transactions.

YouTube is one of those things that consumes more energy than bitcoin.

That is bad, but it doesn't seem part of the public narrative that YouTube is bad because of global warming. Also the general narrative posits that electric cars are good. They also use a lot of energy. There's nothing inherent about EVs versus bitcoin that suggests one uses clean energy versus another. They both use a lot of electricity.

But so does the banking sector. And entertainment, farming, travel and a lot of the other things 8 billion people choose to do or depend on.

Any usage of fossil fuels should rightfully be a target and held to the same standards.

There is another narrative that we need continual growth and that inflation aids this. And deflationary currencies are bad because they don't help growth. This seems so ingrained in modern politics and economics that I slightly hesitate to question it in public.

Well maybe unbounded growth and exponential usage of resources is the actual problem.

I hope we reach the social limit on unbounded growth before it destroys everything we have. It's literally in physical terms unsustainable. My only hope here is that nuclear fusion gives us a respite for a hundred years or so. But we'd still have the same problem at a different scale if we used that to grow to a trillion people.

I can see that a deflationary global currency could be part of the solution to the problem. And I can see why suggesting deflation is good can be seen as a threatening thing to those that have resources.

So I feel in that light, the bitcoin = global warming narrative is really convenient, even if a little ironic.

Not read past the paywall. Could anyone let me know if this statement is clarified?

no politicians likely to trumpet it

Irrespective of whether or not it's outsourcing to China and so not quite the truth, I can't see why politicians wouldn't lay claim to it. Isn't it their modus operandi to find positive news they could attribute to their own efforts?

Needs to support DMCA takedown notices

I think they are saying they manually tested what they thought the automated process was doing, then found out their assumption about what it was doing was wrong. So the manual tests were moot.

sounds like that's an obvious loophole to get around the protection then if they did allow same domain only

edit: no I'm wrong, it's early here and I've not had coffee

Memory is the current spacetime location's fuzzy view when looking towards the past. Imagination is the fuzzy view when looking forwards to the future or with the many-worlds view, imagination is the looking out(?) towards multiple outcomes.

Consciousness is the step towards the next point in the many world multiverse.

Without the ability to be blind to the future, life as we know it wouldn't exist. Nothing conscious would take risks and so complexity couldn't evolve.

If you could glance to the future and see all of the laid out spacetime, there wouldn't be any interesting exploration of it.

So we have to be blinkered just to exist with this level of insight into the complexities of the universe.

Surely that's what it is though. As well as being literal.