c1ccccc1

they things they believe in are unlikely to happen.

Sorry, but most times I've talked to someone who says this (about AI completely replacing humanity), they don't have anything to say about why it's unlikely to happen other than:

- "well, it's just such an extreme outcome, it must be improbable"

- "humanity has survived near scrapes with extinction before"/"all the previous doomsday predictions have been wrong"

There is a cliche that all teenagers deep down believe that they are invincible. It seems to me that humanity is still a teenager in this respect: We don't take seriously the possibility of our own extinction. While one might think that the invention of nuclear weapons would serve as a wake-up call, if anything it has done the opposite.

I'm willing to hear arguments besides the two above, if you have them. (And to be clear, being replaced by AI doesn't necessarily mean being replaced by LLMs in particular. They are a relatively new development.)

Just a terminology note: Alignment does not mean the AI will help its owner kill people. (Indeed, an AI aligned to value human life would generally try to prevent murders.) The word for an AI that follows all instructions of its owner, as that owner intended them to be understood, is "corrigible" or "controllable".

Or the AI companies could filter it from their training data. That would be another, probably easier, option.

Yes. I was not re-doing the original post, just describing an interesting consequence of the kinetic energy formula.

A stationary but hot object has kinetic energy due the the motion of the individual atoms that make it up, even though its overall momentum is 0. I.e.

∑ⱼ mⱼ v⃗ⱼ = 0⃗

where the mⱼ are the masses of the parts of the object and the v⃗ⱼ are the velocities of those parts.

If the object initially has 0 velocity, its kinetic energy is:

T = ½∑ⱼ mⱼ v⃗ⱼ²

Now we give the object a kick (or just switch reference frames) to change its velocity by Δv⃗. The new kinetic energy is:

T' = ½∑ⱼ mⱼ (v⃗ⱼ + Δv⃗)²

T' = ½∑ⱼ mⱼ (v⃗ⱼ² + 2v⃗ⱼ⋅Δv⃗ + Δv⃗²)

T' = ½(∑ⱼ mⱼ v⃗ⱼ²) + Δv⃗⋅(∑ⱼ mⱼ v⃗ⱼ) + ½Δv⃗²(∑ⱼ mⱼ)

If M is the total mass of the object, then we can substitute this into the sum in the last term. And we already saw that the sum in the middle term was 0. So:

T' = ½(∑ⱼ mⱼ v⃗ⱼ²) + Δv⃗⋅0⃗ + ½Δv⃗² M

T' = ½∑ⱼ mⱼ v⃗ⱼ² + ½MΔv⃗²

So in terms of the original kinetic energy T, which was purely thermal energy, we get:

T' = T + ½MΔv⃗²

In other words, because of the quadratic kinetic energy formula, we can see that the total kinetic energy T' of a hot object is just its thermal kinetic energy T plus the usual mechanical kinetic energy ½MΔv⃗².

If you use the right formula for calculating it (which approximates p=mv at low speeds), momentum is actually conserved in special relativity, and so is energy.

However: Energy and momentum are not invariant under changes of reference frame, though the magnitude of the energy-momentum 4-vector is invariant between frames.

Energy is conserved in Galilean relativity. The thing you're trying to say is that it's not invariant across reference frames.

The answer linked above actually takes advantage of the fact that energy is not the same in different reference frames in order to make the argument work.

I think you are overthinking the heat thing. If you have a train car full of hot water and you slow the train down (extracting kinetic energy from it) until it stops, the water in the train car does not change temperature at all, other than a bit of sloshing around and loss of heat to the surroundings.

This looks like good work. Unfortunately, this kind of thing always seems to attract midwits on social media who then exclaim "oh, the people worried about AI alignment have caused the very alignment issues they feared? How ironic!"

In reality, it is (as mentioned in TFA) very possible to filter the training data and remove documents that contain discussions of AI misalignment. If an AI lab isn't doing this, it's simply because they don't consider the problem important enough to be worth the expense and development effort.

If you have a limited budget of tokens as a defender, maybe the best thing to spend them on is not red teaming, but formalizing proofs of your code's security. Then the number of tokens required roughly scales with the amount and complexity of your code, instead of scaling with the number of tokens an attacker is willing to spend.

(It's true that formalization can still have bugs in the definition of "secure" and doesn't work for everything, which means defenders will still probably have to allocate some of their token budget to red teaming.)

Kudos for giving a concrete example, but the square-cube law means that scaling area A results in A^(3/2) scaling for the mass of material used and also launch costs. If you make the pyramid hollow to avoid this, you're back to having to worry about heat conduction. You assumed an infinite thermal conductivity for your pyramid material, a good approximation if it's solid aluminum, but that's going to be very expensive (mainly in launch costs).

In reality, probably radiator designs would rely on fluid cooling to move heat all the way along the radiator, rather than thermal conduction. This prevents the above problem. The issue there is that we now need to design this system with its pipes and pumps in such a way that it can run reliably for years with zero maintenance. Doable? Yes. Easy or cheap? No. The reason cooling on Earth is easier is that we can transfer heat to air / water instead of having to radiate it away ourselves. Doing this basically allows us to use the entire surface of the planet as our radiator. But this is not an option in space, where we need to supply the radiator ourselves.

In terms of scaling by instead making many very small sats, I agree that this will scale well from a cooling perspective as long as you keep them far enough apart from each other. This is not as great from the perspective of many things we actually want to use a compute cluster for, which require high-bandwidth communication between GPUs.

In any case, another very big problem is the fact that space has a lot of ionizing radiation in it, which means we also have to add a lot of radiation shielding too.

Keep in mind that the on-the-ground alternative that all this extra fooling around has to compete with is just using more solar panels and making some batteries.

Radiators can shadow each other, so that puts some kind of limit on the size of the individual satellite (which limits the size of training run it can be used for, but I guess the goal for these is mostly inference anyway). More seriously, heat conduction is an issue: If the radiator is too long, heat won't get from its base to its tip fast enough. Using fluid is possible, but adds another system that can fail. If nothing else, increasing the size of the radiator means more mass that needs to be launched into space.

You're right, thanks!

̶I̶ ̶t̶h̶i̶n̶k̶ ̶h̶e̶ ̶a̶c̶t̶u̶a̶l̶l̶y̶ ̶g̶e̶n̶e̶t̶i̶c̶a̶l̶l̶y̶ ̶e̶n̶g̶i̶n̶e̶e̶r̶e̶d̶ ̶h̶i̶s̶ ̶g̶u̶t̶ ̶b̶a̶c̶t̶e̶r̶i̶a̶ ̶r̶a̶t̶h̶e̶r̶ ̶t̶h̶a̶n̶ ̶a̶n̶y̶ ̶o̶f̶ ̶h̶i̶s̶ ̶o̶w̶n̶ ̶c̶e̶l̶l̶s̶ ̶t̶h̶e̶r̶e̶,̶ ̶r̶i̶g̶h̶t̶?̶

EDIT: Above is false. Went back and checked and I had mis-remembered the video.

Name some of the contradictory possibilities you have in mind?

Also, do you actually think the core idea is wrong, or is this more of a complaint about how it was presented? Say we do an experiment where we train an alpha-zero-style RL agent in an environment where it can take actions that replace it with an agent that pursues a different goal. Do you actually expect to find that the original agent won't learn not to let this happen, and even pay some costs to prevent it?

Oh, so just a probability density thing where we sample q and check if it's p^n (retrying if not) rather than sampling p and n separately and computing q=p^n? I guess that's probably what the they were going for, yeah.

Why is that? My guess would be that you could adjoin an i all the time to the p^n field and get the p^2n field, as long as you had p = 4k + 3. But that's admittedly based on approximately zero thinking.

EDIT: Looking things up indicates that if n is even, there's already a square root of -1 in the field, so we can't add another. So now I believe the 1/4 of the time thing you mentioned, and can't see how that's wrong.

Yes, there is: https://manifold.markets/

People's bets are publicly viewable. The website is very popular with these "rationality-ists" you refer to.

I wasn't in fact arguing that giving a prediction should make people more trustworthy, please explain how you got that from my comment? I said that the main benefit to making such predictions is as practice for the predictor themselves. If there's a benefit for readers, it is just that they could come along and say "eh, I think the chance is higher than that". Then they also get practice and can compare how they did when the outcome is known.

I mean, sure people can use this to fool themselves. I think usually the cause of someone fooling themselves is "the will to be fooled", and not so much that fact that they used precise numbers in the their internal monologue as opposed to verbal buckets like "pretty likely", "very unlikely". But if you estimate 56% it sometimes actually makes a difference, then who am I to argue? Sounds super accurate to me. :)

In all seriousness, I do agree it's a bit harmful for people to use this kind of reasoning, but only practice it on things like AGI that will not be resolved for years and years (and maybe we'll all be dead when it does get resolved). Like ideally you'd be doing hand-wavy reasoning with precise probabilities about whether you should bring an umbrella on a trip, or applying for that job, etc. Then you get to practice with actual feedback and learn how not to make dumb mistakes while reasoning in that style.

And do we also pull this one out of thin air?

That's what we do when training ML models sometimes. We'll have the model make a Gaussian distribution by supplying both a mean and a variance. (Pulled out of thin air, so to speak.) It has to give its best guess of the mean, and if the variance it reports is too small, it gets penalized accordingly. Having the model somehow supply an entire probability distribution is even more flexible (and even less communicable by mere rounding). Of course, as mentioned by commenter danlitt, this isn't relevant to binary outcomes anyways, since the whole distribution is described by a single number.

I meant in a general sense that it's better when reporting measurements/estimates of real numbers to report the uncertainty of the estimate alongside the estimate, instead of using some kind of janky rounding procedure to try and communicate that information.

You're absolutely right that if you have a binary random variable like "IMO gold by 2026", then the only thing you can report about its distribution is the probability of each outcome. This only makes it even more unreasonable to try and communicate some kind of "uncertainty" with sig-figs, as the person I was replying to suggested doing!

(To be fair, in many cases you could introduce a latent variable that takes on continuous values and is closely linked to the outcome of the binary variable. Eg: "Chance of solving a random IMO problem for the very best model in 2025". Then that distribution would have both a mean and a variance (and skew, etc), and it could map to a "distribution over probabilities".)

You should basically assume they are pulled from thin air. (Or more precisely, from the brain and world model of the people making the prediction.)

The point of giving such estimates is mostly an exercise in getting better at understanding the world, and a way to keep yourself honest by making predictions in advance. If someone else consistently gives higher probabilities to events that ended up happening than you did, then that's an indication that there's space for you to improve your prediction ability. (The quantitative way to compare these things is to see who has lower log loss [1].)

[1] https://en.wikipedia.org/wiki/Cross-entropy

Would you also get triggered if you saw people make a bet at, say, $24 : $87 odds? Would you shout: "No! That's too precise, you should bet $20 : $90!"? For that matter, should all prices in the stock market be multiples of $1, (since, after all, fluctuations of greater than $1 are very common)?

If the variance (uncertainty) in a number is large, correct thing to do is to just also report the variance, not to round the mean to a whole number.

Also, in log odds, the difference between 5% and 10% is about the same as the difference between 40% and 60%. So using an intermediate value like 8% is less crazy than you'd think.

People writing comments in their own little forum where they happen not to use sig-figs to communicate uncertainty is probably not a sinister attempt to convince "everyone" that their predictions are somehow scientific. For one thing, I doubt most people are dumb enough to be convinced by that, even if it were the goal. For another, the expected audience for these comments was not "everyone", it was specifically people who are likely to interpret those probabilities in a Bayesian way (i.e. as subjective probabilities).

Cryptographers are clever and have figured out a way to let you know your vote was counted without being able to prove it to a third party!

https://www.youtube.com/watch?v=BYRTvoZ3Rho

This comment is interesting in that it itself exemplifies principle #5.

(And also principles #6, #7, #9 to some extent.)

"Below" in this context means "directly below and connected by a line" (the lines are the edges of the cube). So you can have a blue vertex that is vertically below a white vertex, so long as they are not connected by an edge. The first time this can happen is for a 3 dimensional cube. You can have blue at the top, then 2 blue and 1 white below that, and then 1 blue (under and between the 2 blues in the layer above) and 2 white in the layer below that, and then white for the bottom vertex. This configuration can be rotated 3 ways and this takes us from 17 to 20.

The post assumes a 2% annual rate of growth in energy consumption. So, due to the nature of exponential functions, most of the energy loss would concentrated towards the end of the 1000 years, as the energy consumption approaches 400 million times present day energy usage. The first two centuries of use would not have a noticeable impact.

So I'm not a moral relativist, like, at all. But in this case, it seems like we westerners have constructed one particular set of norms for encouraging innovation, where we decide that it's possible for ideas to be owned. It's not like there's anything intrinsically wrong with copy-pasting code, it's just that we have a legal framework where we've traded away the right to freely copy-paste code so that we can grant a temporary monopoly to its author. We do this in the hope that more useful code will be written than otherwise. But if the people of China decide that that's not a trade-off they want to make, then I don't think we westerners get to say that they've committed a moral wrong in making that decision. It's just that they have a different way of doing things.

Like I said, I'm not a moral relativist at all. Murder is still wrong in China, imprisoning people not convicted of any crime is still wrong in China, lying is still wrong in China. But I just don't see how copyright infringement is universally an immoral act.

Hinton and other billionaires are making sensational headlines predicting all sorts of science fiction.

Geoff Hinton is not a billionaire! And the field of AI is much wider than LLMs, despite what it may seem like from news headlines. Eg. the sub-field of reinforcement learning focuses on building agents, which are capable of acting autonomously.

Pretty sure the angst is about the AGI killing everyone. What's the connection between not killing people and enslavement? I don't kill people, yet I don't consider myself enslaved. The entire point of worrying about this at all is that a sufficiently smart AI is going to be free to do whatever it wants, so we had better design it so it wants a future where people are still around. Like, the idea is: enslavement, besides being hugely immoral, obviously isn't going to work on this thing, so we'd better figure out how to make it intrinsically good!

Weird. Seems like this is a plagiarism of this article:

https://www.strangeloopcanon.com/p/agi-strange-equation

(Or maybe it's an independent reinvention of the same idea, and something's just in the water.)

It's much easier than that! Living cells already have ribosomes that construct proteins and all the other molecular machinery needed to go from DNA sequence to assembled protein. You can order a DNA sequence online and put it into e-coli or yeast cells and those cells will make that protein for you.