goolulusaurs

The reality is that o1 is a step away from general intelligence and back towards narrow ai. It is great for solving the kinds of math, coding and logic puzzles it has been designed for, but for many kinds of tasks, including chat and creative writing, it is actually worse than 4o. It is good at the specific kinds of reasoning tasks that it was built for, much like alpha-go is great at playing go, but that does not actually mean it is more generally intelligent.

In my younger years, particularly during my schooling, I held a deep resentment towards the educational system. It felt overtly clear to me, as a student, that schools failed to effectively foster learning and growth. However, my perspective has evolved over time. I've come to understand that the issues I observed are not unique to the school system but rather characteristic of large institutions as a whole.

The pervasive failure of these institutions to meet their stated objectives isn't an isolated phenomenon. It's symptomatic of a larger, systemic problem – the widespread presence of perverse and misaligned incentives at all levels within large organizations.

Unless we find a way to counteract this, attempts at reform will merely catalyze further expansion and complexity. The uncomfortable truth is, once an organization surpasses a certain size, it seems to take on a 'life of its own', gradually sacrificing its original mission to prioritize self-preservation and expansion. Who has ever seen an organization like this voluntarily reform itself? I certainly haven't.

It depends on what you mean by "this discussion", but I don't think that follows.

If for example, we were in scenario 2 and it was still the case that a large number of people thought AI doomsday was a serious risk, then that would be a much stronger argument for taking the idea of AI doomsday seriously. If on the other hand we are in scenario 1, where there is a long history of people falling prey to apocalypticism, then that means any new doomsday claims are also more likely to be a result of apocalypticism.

I agree that is is likely that humans will go extinct eventually, but I am talking specifically about AI doomsday in this discussion.

It was clear that nukes were a risk before they were used; that is why there was a race to create them.

Yes, because there were other kinds of bombs before then that could already kill many people, just at a smaller scale. There was a lot of evidence that bombs could kill people, so the idea that a more powerful bomb could kill even more people was pretty well justified.

if AGI is to become a thing, what does the moment look like where we can see it is coming and still have time to respond?

I think this implicitly assumes that if AGI comes into existence we will have to have some kind of response in order to prevent it killing everyone, which is exactly the point I am saying in my original argument isn't justified.

Personally I believe that GPT-4, and even GPT-3, are non-superintelligent AGI already, and as far as I know they haven't killed anyone at all.

So far, 100% of those which are as intelligent as humans have dominated the planet.

This is a much more subjective claim than whether or not the world has ended. By count and biomass there are far more insects and bacteria than there are humans. It's a false equivalence, and you are trying to make my argument look wrong by comparing it to an incorrect argument that is superficially similar.

Consider two different scenarios:

1) Throughout history many people have predicted the world would soon end, and the world did not in fact end.

2) Throughout history no one predicted the world would soon end, and the world did not in fact end.

The fact that the real world is aligned with scenario 1 is more an indication that there exists a pervasive human cognitive bias to think that the world is going to end, which occasionally manifests itself in the right circumstances (apocalypticism).

Well, for example I believe that nukes represent an existential risk, because they have already been used to kill thousands of people in a short period of time. What you are saying doesn't really counter my point at all though, it is another vague theoretical argument.

Throughout history there have been hundreds, if not thousands of examples of people and groups who thought the end of the world was imminent. So far, 100% of those people have been wrong. The prior should be that the people who believe in AI doomsday scenarios are wrong also, unless and until there is very strong evidence to the contrary. Vague theoretical arguments are not sufficient, as there are many organizations throughout history who have made similar vague theoretical arguments that the world would end and they were all wrong.

https://en.wikipedia.org/wiki/Category:Apocalyptic_groups

Sure, but don't they have a vast network of contact tracing? Presumably if a bunch of cases showed up that weren't connected to any other cases because of the lockdowns then that would be a really good indication of where the animal reservoir was located. But they haven't found the animal reservoir yet either, even with all of the lockdowns.

Well its been years, why haven't they found the reservoir yet then? They would obviously want to since it would prove that it wasn't a lab leak, yet as far as I know they haven't claimed to have found it yet.

IMO, it seems obvious from the behavior of China's government that they know it is a lab leak. If it wasn't a lab leak, then presumably there is an animal reservoir of the virus somewhere in China, but as far as I know they haven't claimed to have found it. But if there is an animal reservoir of the virus in China, then how could the Chinese government ever expect a lockdown to work? A lockdown on travel would only really prevent the virus spreading from people bringing it into the country but obviously wild animals would still be spreading it. Yet the Chinese government claimed that their lockdowns did work. How is that at all compatible with the virus being from wild animals and not being a lab leak? It doesn't make any sense.

There is a lot of negativity towards the idea of AI in this thread, and I feel like someone has to say it: it is quite likely that in the near future computers will be better than almost all humans at almost all cognitive tasks.

If you have a task or are trying to accomplish something, and the way you do it is by moving a mouse around or typing on a keyboard then it is very likely that an AI will be able to do that task. Doing so is a more or less straightforward extension of existing techniques in AI. All that is necessary is to record you performing the task and then an AI will be able to imitate your behavior. GPT3 can already do this for text, and doing it instead with trajectories of screen, mouse and keyboard is not fundamentally different.

So yes, it is true that there is a lot of hype right now, but I suspect it is a small fraction of what we will see in the near future. I also expect there will be an enormous backlash at some point.

"assuming the conclusion" is a much better way to refer to the original concept, IMO.

I don't get why everyone assumes that Havana syndrome is fake. I mean it could be fake but it doesn't seem implausible that it is real either. I seems like everyone jumps to the conclusion it isn't real though.

It's a cybernetic feedback system. Dalle is used to create new images, the images that people find most interesting and noteworthy get shared online, and reincorporated into the training data, but now filtered through human desire.

Sure, maybe it is possible if the residents are okay with the CCP taking over, but on the margin everyone being armed makes them taking action much more costly. Guerrillas were able win against the US in Vietnam, and Mao used guerrilla warfare himself and won against a better equiped military. https://en.wikipedia.org/wiki/On_Guerrilla_Warfare

Yes, I think if literally every single person in Hong Kong had a gun that would be enough to stop the CCP. The entire active military of China is about 2 million people, less than a third of the approx. 7.5 million people in HK. Maybe the CCP could bomb the city into dust, but taking it over intact would be a practical impossiblity.

"Political power grows out of the barrel of a gun." The thing that could have saved HK was legalization and widespread private gun ownership. Yes, there would have been more gun crime, but the victims of all of the random gun crime in the world is a tiny fraction compared to the victims of communism.

Thanks for the recommendation. I am a big fan of Weber, not familiar with Deming but his work sounds very relevant. In general I tend to agree that beyond a certain size organization these problems seem unavoidable. I've read Systemantics/The Systems Bible and it seems to come to a similar conclusion.

When I was younger I was aware of the idea of perverse and misaligned incentives, but I never would have expected the extent to which they pervade practically every human institution.

Today everyone is like "remember when we didn't spend all of our time looking at our smartphones?"

15 years from now everyone will say "remember when we didn't have these computers literally strapped to our face all the time?"

It is the obvious extension of existing trends in technology.

I read recently for the first time about what happened to the Maya at the hands of the Spanish and I was shocked at how bad it was. Even just in the context of the destruction of knowledge it was terrible. There were thousands of books recording centuries of precontact American history that were all destroyed by the Spanish. Only four examples out of the many thousands of piece of Maya literature have survived to the present. A few quotes from https://en.wikipedia.org/wiki/Maya_codices:

"Our knowledge of ancient Maya thought must represent only a tiny fraction of the whole picture, for of the thousands of books in which the full extent of their learning and ritual was recorded, only four have survived to modern times (as though all that posterity knew of ourselves were to be based upon three prayer books and Pilgrim's Progress)." — Michael D. Coe

"We found a large number of books in these characters and, as they contained nothing in which were not to be seen as superstition and lies of the devil, we burned them all, which they regretted to an amazing degree, and which caused them much affliction." — Bishop Diego de Landa

Even if the price is the lives of millions and the whole world in lockdown? We might be better off if you don't take the job then. Also you ignored the rest of the sentence you are quoting and my examples of highly skilled scientist who are willing to make that sacrifice in Antartica and the ISS.

Sure that makes sense, but you could crank their pay way up to compensate, build places for their family to stay if they desired like the families of the scientist working on the Manhattan project, present at conferences through teleconferencing. There are scientist who stay in Antarctica for months and the international space station. Yes it would be inconvenient, but look at what (maybe) happens if you don't.

Why the hell would you put a virology lab in the middle of a huge city with millions of people? I am not a virologist, but to me it seems like if covid was a lab-leak then it could have been avoided very easily just by having the lab in the middle of nowhere, like in the desert or the artic. Instead of the whole world having to go into lock down, with millions dying, just put the lab in a remote location, and make it so that before anyone leaves they have to quarantine for 2 weeks. This seems obvious, why isn't it standard protocol?

Without knowing much about the topic, when I hear about code being proven to be correct it makes me think of the Curry-Howard correspondence, which states that proofs and programs are isomorphic to each other. Is this related at all? If programs and proofs are the same thing and you have a proof that a program is correct, is that like having a proof that a proof is correct? In which case it seems like you are getting into the domain of meta-logic.

Frankly, I don't give a shit about the national interest of the US or the national interest of China. As a user of tiktok, I enjoy the content and I will be saddened if it shuts down. We like to think of our country as our team, and to root for it in international struggles, but the reality is that it's the average consumer that gets harmed by these power struggles between nations. These economic proxy wars are just as much of a racket as conventional wars, and the sooner we stop allowing the few to rule the many, and play out their egoic power struggles with our lives, the better.

For some time I have been thinking about why it seems like scientific progress has slowed down compared to a century ago, and then one day I came across a clip of an interview with Professor Russell where he was saying that if technological progress continued at the current rate, then very soon humanity would drive itself extinct from the creation and use of weapons that were too powerful and too easily produced. I wonder if the slowing of scientific progress might be in some ways a saving grace, where if too advanced of technology was developed too quickly, then society would be unable to evolve in time to handle it responsibly without major potential for calamity.

Any advice on how to get better at making predictions?

Reminds me of the opening paragraphs from SICP (https://mitpress.mit.edu/sites/default/files/sicp/full-text/...)

"We are about to study the idea of a computational process. Computational processes are abstract beings that inhabit computers. As they evolve, processes manipulate other abstract things called data. The evolution of a process is directed by a pattern of rules called a program. People create programs to direct processes. In effect, we conjure the spirits of the computer with our spells.

A computational process is indeed much like a sorcerer's idea of a spirit. It cannot be seen or touched. It is not composed of matter at all. However, it is very real. It can perform intellectual work. It can answer questions. It can affect the world by disbursing money at a bank or by controlling a robot arm in a factory. The programs we use to conjure processes are like a sorcerer's spells. They are carefully composed from symbolic expressions in arcane and esoteric programming languages that prescribe the tasks we want our processes to perform.

A computational process, in a correctly working computer, executes programs precisely and accurately. Thus, like the sorcerer's apprentice, novice programmers must learn to understand and to anticipate the consequences of their conjuring. Even small errors (usually called bugs or glitches) in programs can have complex and unanticipated consequences.

Fortunately, learning to program is considerably less dangerous than learning sorcery, because the spirits we deal with are conveniently contained in a secure way. Real-world programming, however, requires care, expertise, and wisdom. A small bug in a computer-aided design program, for example, can lead to the catastrophic collapse of an airplane or a dam or the self-destruction of an industrial robot."