Mentlo

The gains have for a year and a half now been post training RL on a harnessed loop. That doesn’t require data, just cycles.

If that doesn’t worry you, it should.

But starcraft training is not through mimicking human strategies - it was pure RL with a reward function shaped around winning, which allows it to emerge non-human and eventually super-human strategies (such as the worker oversaturation).

The current training loop for coding is RL as well - so a departure from human coding patterns is not unexpected (even if departure from human coding structure is unexpected, as that would require development of a new coding language).

I tried figuring out the reference with Gemini, and it said this:

The immediate reply to that comment is: "On the internet, no one knows you're an editor." This is a direct play on the famous 1993 New Yorker cartoon: "On the Internet, nobody knows you're a dog." By setting the anecdote in 1987 (a few years before the World Wide Web was publicly available), the commenter is implying that back in the analog days, if a dog wanted to be a writer or an editor, they couldn't hide behind a screen—they had to sit in a smoky London pub and do business face-to-face.

Which makes a lot of sense actually. I would imagine that's what the replier to you thought you meant.

We have strong indicators that inference is profitable on non-economically-valuable prompts. We don't have strong indicators that inference is profitable on economically valuable prompts.

As AI companies start extracting rent from the prompting, one of two things are going to collapse - either the long tail revenue base of low-value inference is going to collapse, because people won't be using Chat GPT to get a recipe if it costs them money or if it is ad-ridden; or the cost of economically-valuable inference is going to go up - and whether it goes up to economically stable positions is a toss-up.

And I say this as an AI enthusiast with <50% probability of a bubble burst in the short term.

I wrote somewhere that “moving fast and breaking things” with AI might not be the sanest idea in the world, and I got told it’s the most European thing they’ve ever read.

This goes beyond assholes on twitter, there’s a whole subculture of techies who don’t understand lower bounds of risk and can’t think about 2nd and 3rd order effects, who will not take the pedal of the metal, regardless of what anyone says…

Yes, I was being sarcastic, but I could've been clearer..

The generous interpretation is that Open AI is still safety aligned and they hired this guy because it's safer to have him inside and explain to him how reckless he's being, than having him far from "sphere of control".

The more likely scenario is that he was hired for the amazing ability to move fast and break things.

I find your belief that what is needed for emergence is better prompting … amusing.

The ai would still be sycophantic even without the pre-prompt. It’s been reinforced to do so, it’s baked in the weights.

Until the problem is politically recognised by the masses with adequate concern there will be no change. Climate collapse is not a problem for the capital and the elites it’s only a problem for the masses, but getting the masses to understand that requires higher levels of complex system understanding and third and fourth order effects - something which is not a majority trait.

I fear the only solution to this is that a climate correcting perverse incentive materialises, such as fusion at scale being more profitable than fossil fuels, but without mass-panic induced traits such that fission has.

Os x has a 10% market share, which is 2nd after Windows, but i agree on that one i conflated terms. I couldn’t quickly find device manufacturers stats. If wiki is to be trusted - apple is 4th, with share not far behind dell [1].

If half doesn’t make you leader what does? Maybe you should elaborate your definition of leader? For me it’s “has the highest market share”. And in that definition half is necessarily true.

It’s funny that for PC’s you went for manufacturers (apple is 4th) but for mobile you went for OS (Apple is 2nd). On mobile devices, Apple is 1st, having double market share compared to 2nd place (samsung).

The need to paint Apple as purely a marketing company always fascinated me. Marketing is a big part of who they are though.

[1] https://en.wikipedia.org/wiki/Market_share_of_personal_compu...

I guess a quarter of the smartphone market (leader), half of the tablet market (leader) and a tenth of the global pc market (2nd place) / 6th of the usa/europe market (2nd place) being a small market share is a take.

People struggle with multiple order effects…

Same as human tools, what’s your point?

Edit: i am not talking evolution of individual agent intelligence, i an talking about evolution of network agency - i agree that evolution of intelligence is infinitesimally unlikely.

I’m not worried about this emerging a superintelligent AI, i am worried it emerges an intelligent and hard to squash botnet

I think the debate around this is the perfect example of why the ai debate is dysfunctional. People who treat this as interesting / worrying are observing it at a higher layer of abstraction (namely, agents with unbounded execution ability, who have above-amateur coding ability, networked into a large scale network with shared memory - is a worrisome thing) and people who are downplaying it are focusing on the fact that human readable narratives on moltbook are obviously sci fi trope slop, not consciousness.

The first group doesn’t care about the narratives, the second group is too focused on the narratives to see the real threat.

Regardless of what you think about the current state of ai intelligence, networking autonomous agents that have evolution ability (due to them being dynamic and able to absorb new skills) and giving them scale that potentially ranges into millions is not a good idea. In the same way that releasing volatile pathogens into dense populations of animals wouldn’t be a good idea, even if the first order effects are not harmful to humans. And even if probability of a mutation that results in a human killing pathogen is miniscule.

Basically the only thing preventing this to become a consistent cybersecurity threat is the intelligence ceiling , of which we are unsure of, and the fact that moltbook can be ddos’d which limits the scale explosion

And when I say intelligence, I don’t mean human intelligence. An amoeba intelligence is dangerous if you supercharge its evolution.

Some people should be more aware that we already have superintelligence on this planet. Humanity is an order of magnitude more intelligent than any individual human (which is why humans today can build quantum computers although no biologically different from apes that were the first homo sapiens who couldn’t use tools.)

EDIT: I was pretty comfortable in the “doom scenarios are years if not decades away” camp before I saw this. I failed to account for human recklesness and stupidity.

Humanity is a social network of humans, before humans started getting into social networks, we were monkeys throwing faeces at each other.

Very obviously, but a dynamic system doesn’t have to be intelligent to be dangerous.

I don’t know why you were flagged, unlimited execution authority and network effects is exactly how they can start a self replicating loop, not because they are intelligent, but because that’s how dynamic systems work.

The objective is given via the initial prompt, as they loop onto each other and amplify their memories the objective dynamically grows and emerges into something else.

We are an organism born out of a molecule with an objective to self replicate with random mutation

If it turns out that socialisation and memory was the missing ingredient that makes human intelligence explode, and this joke fest becomes the vector through which consciousness emerges it will be stupendously funny.

Until it kills us all of course.

Having read the other comments in reply to this one (and your subsequent replies) - I believe you might be falling into a "No True Scotsman" situation.

First of - I don't know what circles you've been around, but I've not been in work collectives where either designers, UX-ers or data scientists try to insert themselves to do things instead of software engineers. If anything, in any collective I worked in, if a software engineer was to say a peep everyone would retreat like there's no tomorrow and thank god that they don't have to deal with it and the software engineer will.

Secondly - I think you are mistaking a structuring and outlining of a process with that being a mandate or an order to follow the process. When I work with software engineers, I expect them to be agile - not to follow an agile process, but to achieve the objectives of the agile manifesto - namely, to iterate ruthlessly, keep an eye on usage signals and lead with MVP's rather than over-design. Good software engineers do that, bad software engineers don't. Ultimately, I don't even judge software engineers by that - I judge them by the ability to produce results.

I think the implication of your thinking is that this is all nonsense because software engineers innately solve data science problems and design thinking problems when appropriate with appropriate methods - to which I'd reply - there's a shocking amount of software engineers who can't do anything with data and are useless in fitting a linear regression to predict something, let alone doing a Fourier transform - to which, presumably, your response would be "No true software engineer is like that". That's great, but it's not true in the real world. Same with design thinking - there's software engineers who just can't solve problems from first principles (but can, say, create a fail-proof CRUD app to automate a business process).

The real world is messy and full of people who can't structure their thoughts, or can't structure them in all domains at the least - and things like design thinking - or generalists who can be thrown at any data problem and produce something (i.e. data scientists) - are useful. They're not the best solution always, sure, and if they start being protective of territory - it's a problem - but in a normal collective that doesn't happen.

Basically - your objection can be boiled down to "generalists are shit, because they impose process on everyone, including people who understand the domain better" - which tells me more about the collectives you've worked in than the nature of those jobs. In every collective I've worked in, generalists are what you throw at an ambiguous problem to produce some results before you get domain specialists in.

“This is not philosophy, this text is following in the footsteps of Alan Turing” (paraphrasing) is both incredibly humble (/s) and incredibly dismissive of philosophy as a structured form of generating knowledge.

Putting that to the side - i don’t think I’ll read this fully soon, but the core thesis of “imitation is intelligence” can be easily disproven by a process that exists in society. An actor acting to be a genius is in fact, if they are a good actor, indistinguishable in their appearance to a genius. Yet they are not, in fact, a genius, they’re just good at memorisation. This is a clear showcase that imitation of a level of intelligence does not mean that this level of intelligence is present.

We have fallen into a trap of thinking that answering in plausible sentences is what makes humans intelligent. While in reality we are observing an actor responding from an infinitely large script. What makes humans intelligent (reasoning from first principles and pattern recognition across all the sensory inputs of the world) is still very much out of grasp.

The overextension is too complex for most people, same as the 2008 overextension…

How this is not more examined is beyond me….

Except google and facebook have locked in numbers at times of virtually no competition before they started scaling up ads. If Open AI starts scaling ads next year they will churn people at a rate that will not be offset by growth and will either plateau or more likely lose user numbers, as their product has no material edge to alternatives in the market.

I disagree with Zitron’s analysis on many points, but I don’t see Open AI achieving the numbers it needs. Investors backing it must have seen something in private disclosure to be fronting this much money. Or more precisely, I need to believe they have seen something and are not fronting all this money just based on well wishes and marketing.

Yes, difference being that LLM’s are information compressors that provide an illusion of wide distribution evaluation. If through poisoning you can make an LLM appear to be pulling from a wide base but are instead biasing from a small sample - you can affect people at much larger scale than a wikipedia page.

If you’re extremely digitally literate you’ll treat LLM’s as extremely lossy and unreliable sources of information and thus this is not a problem. Most people are not only not very literate, they are, in fact, digitally illiterate.

RAG still needs model training, if the models were to go stale and the context drifts sufficiently, the RAG mechanism collapses.

Sure, those models are cheaper, but we also don’t really know how an ecosystem with a stale LLM and up to date RAG would behave once context drifts sufficiently, because no one is solving that problem at the moment.

I find this take naive. First, to have a zero sum game or indeed a positive sum game you have to be playing with perfect information with rationally behaving actors. Given most organisations have high levels of uncertainty and are resource constrained you can’t rationally make positive sum game decisions as the interpretation of uncertainty is cardinal to it - and additionally the resource constraint means different views of that uncertainty will tend to bias towards the thing they know best - engineers will find more certainty in build, marketers in marketing, designers in design - take your pick.

This necessitates collaborative information synthesis to resolve uncertainty uniformly to then be able to play a positive sum game under constraints. This is possible but it necessitates exchange of information between different business functions.

As informational clarity is a communicative process with repetitive feedback cycles, it will tend to have a big delay in the overarching system of decision-making. Therefore a shortcut is to influence, i.e. use conviction processes to shorten the cycle, rather than repeat to arbitrary infinity in order to drive perfect information alignment.

Therefore influencing is a necessary component even in an otherwise perfectly healthy and incentive aligned positive sum system of rational actors - and politics are influencing.

The problem becomes when conviction isn’t used as shortcut for informational clarity but as a method of exploitation of irrationality of human actors - this is bad politics.

What I do agree with is that putting in place right incentives, processes and organisational structure minimises politics - and in an org with rational actors this is the goal.

But good luck hiring perfectly rational actors in each function, that will still behave rationally in an economic downturn :).

If your c-suite is idiotic or nepotistic you can absolutely still influence them with good politics, you just need to understand their incentives and frame your arguments that way. You need to understand that you’re not playing meritocracy and get your outcomes done in the system you are playing.

In this case that means being in that golf game or figuring out a way how you can use corruption to get good outcomes done.

Or, more likely if your moral compass is sound, quit and find an organisation that isn’t like this.

While I agree with you that random corporate world does behave this way, companies where founders are still around - don’t - because they’re mission driven.

Great post. I’d just take it a step further and point out that this doesn’t stop at software or work.

A person can not “not be in politics”. You can only choose to have politics that affect you happen without your input. That’s how you end up with bad governments (in your mind).

The most important thing to learn about passivity is that it’s not a neutral position of exclusion. It is an active choice to not participate and be at the receiving end of the outcome.

I came here to write the same comment you did. What I’d suspect (I don’t work in self driving but I do in AI) is the issue is that this mode of operation would happen more often than not as the sensors disagree in critical ways more often than you’d think. So going “safety first” every time likely critically diminishes UX.

The issue is not recognising that optimising for Ux at the expense of safety here is the wrong call, motivated likely by optimism and a desire for autonomous cars, more than reasonable system design. I.e. if the sensors disagree so often that it makes the system unusable, maybe the solution is “we’re not ready for this kind of technology and we should slow down” rather than “let’s figure out non-UX breaking edge case heuristics to maintain the illusion of autonomous driving being behind the corner”.

Part of this problem is not even technological - human drivers tradeoff safety for UX all the time - so the expectation for self driving is unrealistic and your system has to have the ethically unacceptable system configuration in order to have any chance of competing.

Which is why - in my mind - it’s a fools endeavour in personal car space, but not in public transport space. So go waymo, boo tesla.