HN user

hollerith

8,087 karma

Richard Hollerith, San Francisco Bay Area. hruvulum@gmail.com

No one has any plausible plan or halfway-decent plan for how to maintain control over an AI that has become super-humanly capable. Essentially all of the major AI labs are trying to create a super-humanly capable AI, and eventually one of them will probably succeed—which would be very bad (i.e., probably fatal) for humanity. So, clearly the AI labs must be shut down and in fact would have been shut down already if humanity were sufficiently competent. The AI labs must stay shut down until someone comes up with a good plan for controlling (or aligning) AIs, which will probably take at least 3 or 4 decades. We know that because people have been conducting the intellectual search for a plan for more than 2 decades as their full-time job, and those people report that the search is very difficult.

A satisfactory alternative might be to develop a method for determining whether a novel AI design can acquire a dangerous level of capabilities along with some way of ensuring that no lab or group goes ahead with an AI that can. This might be satisfactory if the determination can be made before giving the AI access to people or the internet. But I know of no competent researcher that has ever worked on or made any progress on this problem whereas at least the control problem has received a decent amount of attention from researchers and funding institutions.

The AIs that might prove fatal to humanity will be significantly different in design from the AIs that have been already widely deployed: for one thing, they will constantly learn (like a person does) as opposed to already-deployed AIs in which the vast majority of the AI's learning happens during a training phase that ends before any widespread deployment of the AI. Also, they will be much better than current AIs at working towards a long-term goal. I say this because I don't want to be misunderstood as believing that Google Gemini 2.5 or ChatGPT 5.0 might take over the world: I understand that those AIs are incapable of a such a thing. The worry is the AIs that are still on the drawing board or that will appear on a drawing board 5 or 10 years from now. There is no need to ban Gemini 2.5 and ChatGPT 5. Since some AI researchers pursue AI "progress" for ideological reasons and will tend to persist stubbornly even after AI research is banned, the best time to ban AI frontier research is now, so that these stubborn ideologues whose research will have been driven underground (because of the ban) but are capable of making a little more "progress" on AI will be unlikely to be able to make enough "progress" to end the world.

Again, as soon as anyone comes up with a solid plan for controlling (or aligning) an AI even if the AI turns out to be more capable than us, the ban on frontier AI research can be lifted as long as the majority of AI experts and AI researchers are in agreement that the plan is solid. No one can say how long this search for a solid plan will take, but IMHO it will probably take at least 3 or 4 decades.

More at

https://intelligence.org/the-problem/

Posts41
Comments6,048
View on HN
news.ycombinator.com 1y ago

Why aren't more people here worried about AI's exceeding us capabilities?

hollerith
5pts5
dontsedateme.org 1y ago

Try to persuade an super-intelligence not to sedate you

hollerith
5pts2
www.greaterwrong.com 2y ago

OpenAI: Facts from a Weekend

hollerith
1pts0
techcrunch.com 2y ago

Russian zero-day seller offers $20M for hacking Android and iPhones

hollerith
6pts0
www.theverge.com 2y ago

iPhone and iPad apps will be available in the Vision Pro App Store by default

hollerith
2pts4
ubiquity.acm.org 4y ago

Cybersecurity Is Not Important

hollerith
19pts8
news.ycombinator.com 5y ago

Where the Web Is Heading

hollerith
10pts2
www.theregister.com 5y ago

Microsoft Will Adopt Google Chrome's Controversial Manifest V3 in Edge

hollerith
16pts4
hruvulum.github.io 6y ago

A suggestion for improving the comments on Hacker News

hollerith
3pts2
www.youtube.com 6y ago

I Changed My Opinion on China

hollerith
2pts0
pastebin.com 6y ago

A scheme for discouraging the formation flash mobs on the internet

hollerith
2pts0
www.nytimes.com 7y ago

5G Phone Won’t Hurt You. But Russia Wants You to Think Otherwise

hollerith
2pts0
arstechnica.com 8y ago

Adware vendors buy Chrome Extensions to send ad- and malware-filled updates

hollerith
99pts43
arstechnica.com 10y ago

Craig Federighi talks open source Swift and what’s coming in version 3.0

hollerith
1pts0
aeon.co 10y ago

If the internet is addictive, why don’t we regulate

hollerith
2pts2
www.macrumors.com 10y ago

Apple Patents Switch-Less Force Touch Keyboard

hollerith
2pts0
mentalfloss.com 12y ago

Russian Travel Tips for Visiting America

hollerith
1pts0
www.npr.org 12y ago

New York City Extends Smoking Ban To E-Cigarettes

hollerith
1pts0
www.theatlanticcities.com 12y ago

A Tour of a Whole Country That Has 'Transformed Itself for Cycling'

hollerith
43pts46
blakemasters.com 12y ago

Judicata: The Path of the Law (2012)

hollerith
1pts0
mjg59.dreamwidth.org 13y ago

How XMir and Mir fit together

hollerith
1pts0
www.youtube.com 13y ago

Managing the Mighty Mississippi (2011)

hollerith
2pts0
en.wikipedia.org 13y ago

IBM Port-A-Punch

hollerith
1pts0
www.theage.com.au 13y ago

Bad reputation: blackmail, corruption plague online reviews

hollerith
1pts0
twitter.com 13y ago

Streaming Curator (streamingc) on Twitter

hollerith
2pts0
www.bbc.co.uk 13y ago

Is this interactive avatar the face of the future?

hollerith
4pts0
arstechnica.com 13y ago

Genius or insanity? Exec did all his work on Galaxy Note for a year

hollerith
9pts0
stevecorona.com 13y ago

College was my biggest mistake (2012)

hollerith
189pts173
techland.time.com 13y ago

Facebook Asks Staffers to Dump Their iPhones and Use Android | TIME.com

hollerith
1pts0
robocall.challenge.gov 13y ago

FAQ | FTC Robocall Challenge

hollerith
1pts0

There was certainly a vast gold rush around the internet with massive attention from almost everybody in journalism or in business and almost every retail investor -- at least in the US (where I was) -- but it differed from the AI hype in that there was no vision of utopia or at least not any that resonated with more than a very small fraction of the population.

The closest the internet got to becoming the focus of hopes for a great transformation of society (or the focus of an ideology or comprehensive narrative) that I saw was a conversation started by John Perry Barlow's "A Declaration of the Independence of Cyberspace" (published Feb 1996) and the crypto enthusiasts. Both conversations consisted mostly of libertarians (with the crypto conversation also having anarchists), which as everyone knows only ever made up 4 or 5% of the US population (but a much larger fraction of internet users in the 1990s).

Also, Teilhard de Chardin's books (written in the 1950s and 1960s) inspired people with a grand vision of the human future and in particular predicted the creation of a "noösphere" which was much like the global internet, so Teilhardists wrote some about that. But the Teilhardists were never more a tiny tiny group, much smaller even than the aforementioned crypto enthusiasts. Almost not worth mentioning.

Was there a general sense that something singularity-like was approaching?

No. Although millions of words were written during the 1990s about the singularity (referring to it using that word specifically IIRC), even then it was always AI that was expected to bring about a utopia, never the internet. And the discussion was restricted to a few mailing lists (the extropians list, Yudkowsky's SL4 list and ISTR a transhumanism list), one or two books (e.g., Great Mambo Chicken and the Transhuman Condition), Omni magazine and maybe Wired magazine.

appears to actually make people more susceptible to nonsense

I think it is more that teenagers and very young adults and people high in the trait of "openness to new experiences" are more susceptible to nonsense and more attracted to new intellectual movements, of which the rationalist community is one.

Also, I expect a rationalist to be lower in conscientiousness than average (because the conscientious people are too busy to devote a lot of time to reading long intellectual texts) and low conscientiousness makes violence and other transgressive behavior more likely.

That's the model they mean

You have incorrectly summarized the position of Rayiner and me.

Let us oversimplify and assume all spending by individuals is either consumption or investment.

Rayiner and I advocated for taxing consumption, but leaving investment untaxed. (And then I suggested that a tax on consumption could be implemented as a tax on income where any spending on any investment is fully deductible.)

You seem to be advocating something different.

For the sake of completeness, we should include a third category of spending, namely, spending necessary to earn money. If we don't leave that untaxed, then an individual can get himself in a situation in which he owes more taxes than he actually earned, which is an undesirable property for any tax system.

But there is a lot of spending, particularly by high-income people, that is not investment and is not spending necessary to earn money. Examples: expensive country-club memberships, yachts, $5000 dresses. Rayiner and I are calling that consumption.

Some spending is necessary just for the average person (i.e., almost everyone who doesn't have rich parents or a rich spouse) to maintain life. The tax system should refrain from taxing this kind of spending (and the US system approximately does in that individuals earning less than about 40K per year almost always owe zero federal income tax).

401Ks would be yet another category of spending that should probably remain untaxed / deductible.

spends zero words to explain how a progressive consumption tax might possibly be implemented.

The implementation could consist of a tax on personal income, but any re-investment of that income is deductible, plus a tax on luxury items and products whose main purpose is status-signaling.

AI 2040: Plan A 12 days ago

"Asian people have pretty much not done that except for two teeny tiny indiscretions that each killed more civilians than all of Europe's and America's colonial incursions combined."

AI 2040: Plan A 12 days ago

China is currently occupying Tibet, which never consented to the occupation and has invaded Vietnam 30 times. It ruled Vietnam for about 2 centuries starting about 600. China eventually had to leave Vietnam, but many other groups ceased to exist as a consequence of Chinese expansion. Here are some:

the Baiyue were a vast umbrella of diverse, non-Sinitic indigenous coastal tribes who inhabited Southern China and northern Vietnam.

The Xianbei were an ancient nomadic Proto-Mongolic people from the northern steppes.

The Di and Jie were two of the ancient "Five Barbarian" (Wu Hu) nomadic tribes of northern and western China during the Han and Jin periods.

The Dian Kingdom were an ancient, sophisticated indigenous southwest culture located in modern-day Yunnan province.

The Tujia were an indigenous group of the Hunan-Hubei region. Centuries of inward Han migration and intermarriage have resulted in the Tujia becoming culturally and structurally indistinguishable from their Han neighbors.

In the US, the state government has absolute power over the municipal and county governments in its borders, which is a very simple legal structure. For example, the NY state legislature can override any law made by the NY city government and can even replace the NY city government with whatever entity it wishes.

Maybe, but I wish people in these conversations would sometimes point out that China is also dependent on imports to continue to make the stuff it makes.

China is a heavy importer of petroleum, food and fertilizer. A modern economy needs petroleum to grow food and to deliver food to consumers. Although China is wisely trying to replace that dependence on petroleum with electric vehicles, it has not yet succeeded. If there's ever a sustained shortage of food, Chinese young men are likely to start revolting and rioting like they've done dozens of times in Chinese history in response to food shortages. China relies on foreign nations for approximately 75% to 80% of its total iron ore consumption, making it the world's largest importer. Despite having had semiconductor manufacturing self-sufficiency as a top national priority for many years, it is still the case that Chinese smartphone manufacturers rely heavily on Samsung and TSMC for the SOCs that go into those smartphones -- and MediaTek (Taiwan) and Qualcomm (US), not Chinese firms, design most of those SOCs.

Geopolitical strategist Peter Zeihan argues that the United States is the only major global power fully capable of national self-sufficiency. I.e., if global trade is ever disrupted by world war, the US would be able replace its missing imports much faster than China would be able to.

AI 2040: Plan A 13 days ago

Yes, there are examples of where we have collective decided not to pursue a particular technology tree.

For one, Japan banned guns for a few centuries. (Its warrior class was politically powerful and judged that guns would disrupt class relations too much.)

And there have been successful world-wide bans.

For example, following the invention of recombinant DNA technology, scientists convened the Asilomar Conference in 1975. They established a voluntary self-moratorium on certain types of genetic engineering until strict laboratory containment protocols were created.

In the 1980s, bioethicists, theologians, and researchers established a hard ethical line between somatic editing (treating an existing patient's non-reproductive cells) and germline editing (altering future generations).

No one has performed the latter form of genetic engineering except for Chinese scientist He Jiankui in 2018. (Chinese society used to be more ambivalent about the technology than the West is.) In response, Beijing heavily tightened its laws, classifying heritable gene editing as a high-risk medical technology subject to the penal code, and He Jiankui was sentenced to three years in prison.

CEO of Mercedes was born in Sweden to Swedish parents; has blue eyes. CEO of Adidas has a Norwegian father and grew up in Norway; has pale skin, reddish-blonde hair and blue eyes. CEO of Bayer was born and raised in the US. Last name is "Anderson". Has pale skin and blue eyes. Two of them could pass as German (at least until they started talking): most Norwegians don't really look like Germans and the CEO of Adidas is no exception.

Now let's do the top 5 companies in the US (by market cap). CEO of NVidia was born in Taiwan to Chinese parents and immigrated to the United States as a child. CEO of Google was born in India to Indian parents, came to the US after college. ditto CEO of Microsoft. CEOs of Apple and Amazon are a white men who grew up in the US.

Resetting Xbox 15 days ago

We should try our best to improve our society. Sometimes that will mean doing things that are economically inefficient. Your example of Stanford professor Nir Eyal is an excellent illustration: by advising corporate managers and product designers how to fulfill their role in our economic system more efficiently, Nir Eyal made our society worse.

We should expect more from our elite professors. Nir Eyal should've known better than to make a career out of studying how deliberately to addict users. But we should also expect our professors, politicians and policymakers to understand the basics of our economic system and to understand when a proposed change to our society sounds good or feels good, but has severe adverse effects on economic efficiency that outweigh the societal benefits. Those aren't the only proposed changes we should avoid, but they form an important class of them.

Resetting Xbox 16 days ago

Our society has a finite capacity to do projects and to run industrial processes. We want that finite capacity to provide as much quality of life as possible (and as much expansion of our capacity to do projects and run processes as possible), and achieving that is a thorny intellectual problem, which for many centuries in our society has been solved mostly by lawyers and judges knowledgeable about corporate law, stock markets, corporate managers, accountants and investors. This entire ecosystem is predicated on the assumption that investor will try to maximize their return of investment, which is strongly correlated with profit margin.

Resetting Xbox 16 days ago

Our society uses return on investment (which is correlated with profit margin) to decide what projects and what processes our society's workers will focus on.

So for example, if I can make twice as much money as a software developer as I can as a musician, that is strong evidence that my doing the former kind of work will benefit society about twice as much as my doing the latter kind of work.

Resetting Xbox 16 days ago

Still, there is a difference of degree. The engineering cost of making one city's electrical grid compatible with the consumers of a different city's grid were much lower (even if one grid was AC and the other DC) than the engineering cost of moving from AWS to a competitor because electricity is not that complicated.

Also, I doubt anyone was referring to electricity as a commodity in the early days before the industry developed standards.

The UK for one. By the time it became a democracy, it was already stable, peaceful and the wealthiest country per capita that had ever existed.

AOSP is just as open-source as desktop Linux (and desktop Linux subsystems reconfigured for use on a phone) is, and has vastly better security.

People sometimes say, "maybe desktop Linux lacks in security, but its privacy is first-rate," which IMHO is silly because if your OS can be completely and totally pwned by your visiting a web page, your opening image file or your using some IDE to inspect some source-code repo, how much privacy do you really have?

GrapheneOS is just as free from data collection by megacorporations as desktop Linux is and has about 1000 times better security.

The police erased fireworks explosions in my neighborhood in the Bay Area.

Two years ago, I heard thousands of explosions in the month of July, same as every year. Then last year, the police posted signs reminding people that fireworks are illegal and that they would begin issuing fines to violators. The number of explosions was about 30% of usual (i.e., still many hundreds of explosions). This year I've heard exactly one explosion (on July 03).

I unambiguously welcome this change. You mention safety, but fireworks are also hard on neighbors with PTSD.

Is littering also in your opinion "an important component of the American zeitgeist"?

Bicycle theft?