cjcole

"So I'd like to engage AI risk from both these perspectives. I think the arguments for superintelligence are somewhat silly, and full of unwarranted assumptions.

But even if you find them persuasive, there is something unpleasant about AI alarmism as a cultural phenomenon that should make us hesitate to take it seriously.

First, let me engage the substance. Here are the arguments I have against Bostrom-style superintelligence as a risk to humanity"

--

The framing here seems to me to equate "AI risk" and "AI alarmism" with buying in to belief in "Bostom-style superintellgence".

I'm not sure if the author meant to put anyone who is alarmed by developments in what we're calling "AI" into the same bucket as "AI obsessives want to make it into a programming problem, by designing a God-like machine", but I think this conflation is unfair and, frankly, dangerous.

I don't know what superintelligence is. I don't even know what intelligence is. And I don't really know what either "artificial" or "general" mean either when talking about "AGI".

You can believe, as I do, that these things can be, and will inevitably will be if we don't radically correct course, used to do very bad things independent and short of being "God-like". When you have systems which can hypothesize, synthesize, and test thousands if not millions of potential infectious agents in bulk [0], and can then order the ingredients for you from dodgy websites via some "claw", and then when you put these systems under the unsupervised control of millions of people with varying levels of stability and altruism, something extremely bad is exceedingly likely to happen.

I understand that 2016 is ages ago and things change, but I came away from the article with the impression that if I'm worried about AI risk then I'm a clown like the three pictured in the "Outside Argument" section (you're a Google-Glass-wearing cringe nerd if you're alarmed). Maybe that's my fault and I'm not smart enough to understand the actual point of the article. If I have misinterpreted, I welcome the correction.

[0] https://www.nature.com/articles/s41467-024-53759-4

"John wore a T-shirt that featured a smiley face with a bullet hole in the forehead from which trickled a few drops of blood"

Sounds like a Watchmen Comedian logo t-shirt. It could be construed as a bold choice but was probably just what was on the top of his t-shirt stack that day.

"but I can't help but see parallels between today and the Industrial Revolution"

You're not the only one.

The current Pope Leo XIV explicitly named himself after the the previous Leo, Pope Leo XIII, who was pope during the Industrial Revolution (1878-1903) and issued the influential Encyclical Rerum novarum (Rights and Duties of Capital and Labor) in response to the upheaval.

“Pope Leo XIII, with the historic Encyclical Rerum novarum, addressed the social question in the context of the first great industrial revolution,” Pope Leo recalled. “Today, the Church offers to all her treasure of social teaching in response to another industrial revolution and the developments of artificial intelligence.” A name, then, not only rooted in tradition, but one that looks firmly ahead to the challenges of a rapidly changing world and the perennial call to protect those most vulnerable within it.”

https://www.vatican.va/content/leo-xiii/en/encyclicals/docum...

https://www.vaticannews.va/en/pope/news/2025-05/pope-leo-xiv...

I'm not an enthusiast. I'm a Butlerian.

Imagine hearing pre-attention-is-all-you-need that "AI" could do something that Donald Knuth could not (quickly solve the stated problem in collaboration with his friend).

The idea that this (Putnam perfect, IMO gold, etc) is all just "statistical parrot" stuff is wearing a little thin.

Maybe.

“The laws of nature should be expressed in beautiful equations.”

- Paul Dirac

“It is, indeed, an incredible fact that what the human mind, at its deepest and most profound, perceives as beautiful finds its realisation in external nature. What is intelligible is also beautiful. We may well ask: how does it happen that beauty in the exact sciences becomes recognizable even before it is understood in detail and before it can be rationally demonstrated? In what does this power of illumination consist?”

- Subrahmanyan Chandrasekhar

“I often follow Plato’s strategy, proposing objects of mathematical beauty as models for Nature.”

“It was beauty and symmetry that guided Maxwell and his followers.”

- Frank Wilczek

“Beauty, is bound up with symmetry.”

- Herman Weyl

"Still twice in the history of exact natural science has this shining-up of the great interconnection become the decisive signal for significant progress. I am thinking here of two events in the physics of our century: the rise of the theory of relativity and that of the quantum theory. In both cases, after yearlong unsuccessful striving for understanding, a bewildering abundance of details was almost suddenly ordered. This took place when an interconnection emerged which, thought largely unvisualizable, was finally simple in its substance. It convinced through its compactness and abstract beauty – it convinced all those who can understand and speak such an abstract language."

- Werner Heisenberg

Maybe (just maybe) these things (whatever you want to call them) will (somehow) gain access to some "compact", beautiful, "largely unvisualizable" "interconnection" which will be the self-evident solution. And if they do, many will be sure to label it a statistical accident from a stochastic parrot. And they'll right, for some definitions of "statistical", "accident", "stochastic", and "parrot".

"humans"

Donald Knuth is an extremal outlier human and the problem is squarely in his field of expertise.

Claude, guided by Filip Stappers, a friend of Knuth, solved a problem that Knuth and Stappers had been working on for several weeks. Unfortunately, it doesn't seem (from my quick scan) to have been stated how long (or how many tokens or $) it took for Claude + Stappers to complete the proof.

In response, Knuth said: "It seems that I’ll have to revise my opinions about “generative AI” one of these days."

Seems like good advice. From reading elsewhere in this comment section, the goalposts seem to be approaching the infrared and will soon disappear from the extreme redshift due to rate at which they are receding with each new achievement.

Nov 16

Update:

Rebecca Heineman - Organizer

It’s time. According to my doctors. All further treatments are pointless. So, please donate so my kids can create a funeral worthy of my keyboard, Pixelbreaker! So I can make a worthy entrance for reuniting with my one true love, Jennell Jaquays.

My daughter Cynthia Elizabeth Heineman, will be making the arrangements

> IQ is compute speed, not storage.

Says who?

https://en.wikipedia.org/wiki/John_von_Neumann#Mathematical_...

Von Neumann's mathematical fluency, calculation speed, and general problem-solving ability were widely noted by his peers. Paul Halmos called his speed "awe-inspiring." Lothar Wolfgang Nordheim described him as the "fastest mind I ever met". Enrico Fermi told physicist Herbert L. Anderson: "You know, Herb, Johnny can do calculations in his head ten times as fast as I can! And I can do them ten times as fast as you can, Herb, so you can see how impressive Johnny is!" Edward Teller admitted that he "never could keep up with him", and Israel Halperin described trying to keep up as like riding a "tricycle chasing a racing car."

He had an unusual ability to solve novel problems quickly. George Pólya, whose lectures at ETH Zürich von Neumann attended as a student, said, "Johnny was the only student I was ever afraid of. If in the course of a lecture I stated an unsolved problem, the chances were he'd come to me at the end of the lecture with the complete solution scribbled on a slip of paper." When George Dantzig brought von Neumann an unsolved problem in linear programming "as I would to an ordinary mortal", on which there had been no published literature, he was astonished when von Neumann said "Oh, that!", before offhandedly giving a lecture of over an hour, explaining how to solve the problem using the hitherto unconceived theory of duality.

A story about von Neumann's encounter with the famous fly puzzle has entered mathematical folklore. In this puzzle, two bicycles begin 20 miles apart, and each travels toward the other at 10 miles per hour until they collide; meanwhile, a fly travels continuously back and forth between the bicycles at 15 miles per hour until it is squashed in the collision. The questioner asks how far the fly traveled in total; the "trick" for a quick answer is to realize that the fly's individual transits do not matter, only that it has been traveling at 15 miles per hour for one hour. As Eugene Wigner tells it, Max Born posed the riddle to von Neumann. The other scientists to whom he had posed it had laboriously computed the distance, so when von Neumann was immediately ready with the correct answer of 15 miles, Born observed that he must have guessed the trick. "What trick?" von Neumann replied. "All I did was sum the geometric series."

Thanks!

(My son loves to play.)

BTW, the old Windows installer runs great under Crossover on Apple Silicon.

I'm going to try running this native. I'll file issues for what I can't readily fix.

“Our clients understand that a two- to three-minute ad load is more valuable than a nine-minute ad load,” says Mark Read, head of WPP, the world’s largest ad company and Groupm’s parent firm.

It's appropriate that even the chief of the purveyors of advertisements uses the same language ("load") that clinicians use for viral infections.

Then it's a shame that society has largely decided to discard as outdated longstanding philosophies whose tenets do teach the intrinsic value of each human independent of their economic output.

https://www.pbs.org/faithandreason/theogloss/imago-body.html

But pushing aside those outmoded ways of thinking did make it easier to sell people useless goods and services which can't fill the hole where dignity and self-respect could instead reside.

And on top we've built the artifice of "social" media (the most shameless oxymoron imaginable) producing sufficient anxiety ensuring that the hole remains unfilled despite all effort.

This is a frightening confluence.

The intersection of "AI", "I won't harm you if you don't harm me", and "I get to unilaterally decide what harm is" is going to be wild.

A fair summary (based on information from Open Secrets) would be that the vast preponderance of contributions were to Democratic and/or progressive candidates or funds ($36,846,356), plus relatively insignificant ($240,200) contributions to Republicans who were (in one way or another) anti-Trump.

https://www.axios.com/2022/11/15/ftz-crypto-bankman-fried-de...

"Last spring, Bankman-Fried pledged to spend upwards of $1 billion on the 2024 election, particularly if Donald Trump were to run."

https://www.opensecrets.org/elections-overview/biggest-donor...

Sam Bankman-Fried

  "Solidly Democrat/Liberal"
  Rank 6
  Contributor FTX.US, Washington, DC
  Total Contributions $39,884,256
  Total Hard Money $1,047,256
  Total Outside Money $38,837,000
  To Democrats $36,846,356
  To Republicans $240,200

https://www.foxbusiness.com/politics/ftx-founder-sam-bankman...

"In addition, Bankman-Fried wired maximum individual donations to Boozman and incumbent Republican Sens. Susan Collins of Maine, Richard Burr of North Carolina, Bill Cassidy of Louisiana, John Hoeven of North Dakota and Lisa Murkowski of Alaska throughout the election cycle."

Collins, Burr, Cassidy, and Murkowski are strongly aligned against Trump.

Boozman received his contribution(s) during his primary against Jake Bequette, a strongly pro-Trump Republican.

Hoeven likewise during his primary race against Rick Becker, another strongly pro-Trump Republican.

Also Boozman:

https://www.benzinga.com/markets/cryptocurrency/22/11/296766...

"Stabenow noted that she is working with her Republican counterpart on the committee, Sen. John Boozman (R-Ark), and regulators to finalize the Digital Commodities Consumer Protection Act (DCCPA) bill in preparation for a committee vote. DCCPA gives power to the Commodity Futures Trading Commission (CFTC) to regulate the trading of digital commodities. This bill was backed by Sam Bankman-Fried, co-founder of the crypto exchange FTX."

Yes, please. Thank you. I'm not sure which is the better argument: that they are a safety menace due to distraction from driving (video billboards?) or that they are profoundly ugly (including but not limited to aesthetically). Either would suffice for me. The quicker the better.

Diplomacy is a species of high-stakes asking.

'Mr Herapath, I understand that you wish to visit the whaler, and you have my permission to go. You are no doubt aware that there is a great deal of ill-feeling between the United States and England, and that most unhappily the Leopard was the cause of some of it: that is why I thought best to forbid the usual ship-visiting, to prevent quarrelling of any kind. You also know the Leopard's condition: one day's use of a forge and the proper tools would enable her to put to sea rather than winter here. The whaler certainly possesses a forge, but as a gentleman you will understand that I am extremely reluctant to ask a favour of the American skipper, extremely reluctant to expose the service or myself to a rebuff. I may add that he is equally reluctant to come a-begging to me, and I honour him for it. However, on reflection he may feel inclined to exchange the use of his forge for our medical services. You may give him a view of the situation, but without committing us to any specific request - harkee, Mr Herapath, don't you expose us to an affront, whatever you do. And if it should turn out that he would like the exchange, why, I should be very much obliged to you. Very much obliged indeed, for I should be even more reluctant to use force.'

First of all, I wish you luck with your effort to have children.

It's almost a cliche at this point, but the prefrontal cortex isn't mature until between 25 and 30 on average.

"One key part of that trajectory is the development of the prefrontal cortex, a significant part of the brain, in terms of social interactions, that affects how we regulate emotions, control impulsive behavior, assess risk and make long-term plans. Also important are the brain’s reward systems, which are especially excitable during adolescence. But these parts of the brain don’t stop growing at age 18. In fact, research shows that it can take more than 25 years for them to reach maturity."

So, yes, teach your children how to avoid predators. That is excellent. But this is the last line of defense. Since children have major impulse control and emotional regulation deficits and the predators have a major asymmetrical advantage in behavioral engineering, it is overwhelmingly the job of the parents to the extent possible to just keep the predators away.

"Internet is scary"

Damn right it is. Children are uniquely impressionable and imprintable for a long time. Seeing or being forced to do gnarly stuff at the wrong time is permanently disfiguring.

Kids will find a way to hang out with their friends.

Yes, the traditional way that would happen is at someone's house. Together. In person. Which provides some level of protection against predation and a fuller/richer/healthier social experience. Where the venue is virtual those protections are lost and more vigilance is required.

If you get in the way of it, you'll quickly find yourself on the losing end of a years-long battle.

There are wolves in the world. There always have been and always will be (as you say). It's a never ending and virtually thankless job (in fact, you will regularly be abused for doing it), but keeping the wolves at bay is parenting job #1. Get them to maturity whole, healthy, intact, and self-sufficient.

I'm not going to share experiences to the extent of the OP, but I have kids and I've met some wolves.

Their business model is literally profiting off of poor customer service (as a service).

Get your credit card from a local credit union with at least one physical presence near you.

Do your own backups or contract to an independent company whose primary occupation is doing backups.

Manage your own email either directly or through an independent email provider. Buy a domain and use that for your email address to increase flexibility to move from one provider to another.

Diversify.

Anything else increases your risk of getting caught in a personal and financial wood chipper.

"Sci-Fi Author: In my book I invented the Torment Nexus as a cautionary tale

Tech Company: At long last, we have created the Torment Nexus from classic sci-fi novel Don't Create The Torment Nexus"

https://www.msn.com/en-us/news/world/uk-to-offer-booster-sho...

Three months, available to all adults.

"All adults in the United Kingdom will be able to get their booster Covid-19 vaccine doses three months after their second shot, the government indicated on Monday, in a dramatic acceleration of the country's inoculation drive that comes amid fears over the Omicron variant."

It seems apparent that the US government is pursuing a novel strategy to route around the Bill of Rights:

Steps:

* Allow (and encourage) private sector monopolies to form

* Use its leverage and influence to "encourage" those monopolies to do things which the US government is not allowed to do (examples: censorship, data collection)

The list of First Amendment precedents, for example, with respect to the government is long and detailed. On the other hand, private companies operate under fewer restrictions and can act much more freely.

So if we have a situation where there are a small number of monopolies and they, through personal relationships (marriage), lobbying, and the revolving door have a cozy relationship with the US government, then the government can effectively censor (for example) by more or less declaring "will no one rid me of these turbulent tweets" and have the monopolies jump to comply (or face increased risk of antitrust action for example).

This is a win-win for both: the monopolies avoid antitrust and other unwanted regulation by playing ball, and the government gets to ignore the Bill of Rights by "outsourcing" actions to what are in effect de facto quasi-governmental entities. It's only a loss for the people for whom the Bill of Rights was written to protect.

That "just" is doing a lot of work.

It seems to me that many of these "rationalize all the things" schemes tend to bottom out at and build onto an "incorruptible kernel of truth" which, if corrupted, causes the whole thing to collapse (or worse: have a veneer of impartial truth while being secretly corrupt). It's a sort of microkernel approach to government.

With Hanson's futarchy, it's the hypothetical mechanism for "voting on values" in order to choose what does and doesn't go in the almighty bundle of metrics (and who gets to measure the result: they have the real power). If that can be corrupted then the whole thing falls apart. Yarvin's "neocameralism" has a "cryptographic decision and command chain" which everything else rides on (https://www.unqualified-reservations.org/2008/05/ol6-lost-th...).

This "all the eggs, one basket" approach seems fragile and ripe for subversion, especially when having fallible humans administer it. It seems more resilient to avoid concentrations of this kind of power.

Hanson says regarding national metrics: 'Well, now we’re going to authorize this same sort of agency to estimate number like GDP except we’re going to tell them to put more things in the number. Bills before Congress would say, “Count more trees, and count leisure, and count international reputation.” They would just make a bigger formula that included all the stuff they cared about in this measure of national welfare.'

Here's the tricky part. It seems to me that the metrics which you don't choose to bundle into the aggregate measurement will get annihilated (optimized out) at the expense of those which you do choose to bundle. And the decisions as to which things you do bundle and with what weight represent ethical, moral, spiritual, aesthetic, and otherwise intangible judgments which are (unsurprisingly) difficult to quantify or even come to basic agreement on. He weasels a bit by using relatively uncontroversial examples ("more trees", "leisure") and by handwaving ("authorize": how?, "sort of agency": what sort?), but it's trivial to imagine metrics which produce wild disagreement regarding the magnitude or even sign of the weight to be applied.

(Shadows of paperclip maximization loom.)

At that point, I'm not sure where we've significantly improved things since we're still left with the problem of how to choose the people who decide which metrics go into the aggregate measure and at what weight. Can we make a market on that, too? At that point we're in some crazy recursion and I get lost. I'm deeply skeptical.

What allows this tactic to work is the size asymmetry between very large companies and smaller companies. It is much less effective against similarly sized companies. Very large companies have much more buffer (reserves of capital, talent, etc.) to draw from. The very large company can absorb layoffs, cancelations, closures, and mindshare losses longer than a much smaller and very likely leaner company can.

Defining what does and does not constitute predatory behavior is difficult to define clearly and concisely.

One potential mitigation, given the above, would be to routinely break up companies above a certain size. To make it perfectly typical, ordinary, and automatic to do so. To shift the burden of proof from the government needing to establish monopoly status to the giant company needing to demonstrate that the consumer benefits due to economies of scale derived from its size outweigh the negative externalities due to its size and subsequent disproportionate power and influence.

Economies of scale leading to lower consumer prices are a positive good. The negative externalities of huge companies due to their size and outsized power and influence aren't given as much consideration as they are less quantifiable, but they are real and should be taken into account.

This is true. We are being psychologically manipulated by extraordinarily skilled operators to redirect our attention from our family, friends, and local community toward these non-local (more lucrative) concerns. This makes the manipulators a vast amount of money.

Maybe we should think about the associated negative externalities and act accordingly.

Ok, this makes sense, thanks.

Still, this is a horrible message to show a user who is in that situation and who did not place a sell order themselves (message wording from the article):

“We’ve received your order to sell [#] shares of [stock] at the best available price.”

This is poor communication from RH. A preventable own goal.

I was once directed to appear at a Las Vegas hotel lobby at 7 AM sharp. I was to drive various poker celebrities from shoot to shoot around the city for a movie. Included was a detailed itinerary. I recognized the names from various TV poker shows.

I wonder what happened when I didn't show up (I did let them know that I wouldn't be there), and I wonder if it was ever finished and released.

Yes, I also get the bills (mostly from electricians in Australia for some reason) the porn/game signups, the notes from Grampa, etc.

It's likely a paraphrase of the Jean Rostand quote: "Kill one man, and you are a murderer. Kill millions of men, and you are a conqueror. Kill them all, and you are a God."

It's in the same category as the ancient "history is written by the victors."

So either cynical or realistic (depending on your perspective).

Sorry for my pun. Thanks for the link.

I'm struggling to see the connection.

That might be wise. There seem to be an infinitude of ways this could go off the rails very badly.