OpenAI researchers warned board of AI breakthrough ahead of CEO ouster

https://www.reuters.com/technology/sam-altmans-ouster-openai-was-precipitated-by-letter-board-about-ai-breakthrough-2023-11-22/

by mfiguiere • 3 years ago

990 1104 3 years ago

reuters.com

minimaxir• 3 years ago

I guess when you set up a board of directors tasked with not letting the company accidentally destroy the human race you set up a showdown when the board thinks the company will accidentally destroy the human race

https://twitter.com/pdmcleod/status/1727463238051229734

V__• 3 years ago

warning of a powerful artificial intelligence discovery that they said could threaten humanity,
Given vast computing resources, the new model was able to solve certain mathematical problems, [..] Though only performing math on the level of grade-school students, acing such tests made researchers very optimistic about Q*’s future success, the source said.

I somehow expected a bit more.

NewJazz2• 3 years ago

Yeah I was chuckling

imdsm2• 3 years ago

It was an "Oh" moment, but on the other hand, it'd be like laughing at text-davinci-003 when you realised how basic it really was, and yet, it has led us to GPT-4 and, much as Windows 3.1 was amazing back in the day, now it's a distant memory as we carry around little glass tablets that can play 3D games and do UHD global video calls.

If Q* is the basic algorithm that is as transformative as, well, transformers/attention, then developing it to be more than just children's math could be the key to thinking - and not just latent space token generation.

mikrl• 3 years ago

They called it Q*

It’s not the first weird conspiracy I’ve heard of involving a cryptically named figure called Q…

hn_throwaway_994• 3 years ago

I think there is enough there to make me believe it's real.

After all, a big objection to LLMs is they are just "fancy autocomplete" or "stochastic parrots". I don't agree, but LLMs famously are "bad at math", that is you can't really train them to do math unless they've seen previous calculations with similar numbers in their training sets. Training an AI system to truly do arbitrary mathematical logic (think "complex word problems with big numbers", not just normal programming statements) would be a huge breakthrough.

xanderlewis13• 3 years ago

Mathematical logic doesn’t refer to ‘complex word problems with big numbers’. The reason that LLMs struggle (well, are hopeless currently) with mathematics problems is not because it involves arithmetic (although this is already enough of a struggle) but because it requires a conceptual understanding of the ideas rather than just being about manipulating language — a lot of the goings-on are hidden in the semantic world and aren’t necessarily encoded in the syntax, unlike when writing code.

kelseyfrog3• 3 years ago

You can't really train them to do math unless they've seen previous calculations with similar numbers in their training sets.

Generally true before xVal[1], but not exactly true after. Expect fairly basic advancement in the future to solve this problem completely.

1. https://arxiv.org/pdf/2310.02989.pdf

pk-protect-ai7• 3 years ago

I compared GPT-4 Turbo with my previous tests on GPT-4, and the results are quite interesting. GPT-4 Turbo is better at arithmetic and makes fewer errors in multiplying four-digit numbers. In fact, it makes significantly fewer errors with five-digit numbers. The level of errors on five-digit numbers is high but much lower than with four-digit numbers in GPT-4. multiplication of floats XX.MMMM and YYY.ZZZZ produces errors in 5th digit. This is order of magnitude better than GPT-4.

pk-protect-ai7• 3 years ago

One more interesting thing.

Here is how GPT-4 performs long multiplication: https://www.phind.com/search?q=multiply%2012.23341%20by%20-1...

Here is how GPT-$ Turbo performs long multiplication: multiply 12.23341 by -123.23354 using long multiplication method. To multiply 12.23341 by -123.23354 using the long multiplication method, we first ignore the decimal points and treat the numbers as integers. We'll reintroduce the decimal point at the end. Here's the multiplication step by step: ``` 1223341 (this is 12.23341 with the decimal removed) × -12323354 (this is -123.23354 with the decimal removed) -------------- 1223341 (multiplied by 4) 0000000 (multiplied by 5, shifted one position to the left) 1223341 (multiplied by 3, shifted two positions to the left) 3660023 (multiplied by 3, shifted three positions to the left) 4886682 (multiplied by 2, shifted four positions to the left) 0000000 (multiplied by 2, shifted five positions to the left) 1223341 (multiplied by 1, shifted six positions to the left) 1223341 (multiplied by 1, shifted seven positions to the left) 0000000 (multiplied by 2, shifted eight positions to the left) 0000000 (multiplied by 3, shifted nine positions to the left) -------------- 15077944092654 (sum of all the above products) ``` Now, we need to place the decimal point. The original numbers had 5 and 8 decimal places, respectively, so the result should have 5+8 = 13 decimal places. So, 15077944092654 becomes 1507.7944092654. Since one of the original numbers was negative, the result is also negative: -1507.7944092654

Therefore, 12.23341 multiplied by -123.23354 equals -1507.7944092654.

Compare with correct result: -1507.5664205714

Here is how I experimented with long multiplication on GPT-4 a month ago (5 shot, GPT-4 made mistakes on addition in 3rd some time 4th digit, regardless of decimal point position):

Multiply -0.9923 by -0.9923 using long multiplication.Solution: multiply individual digits in reverse order, increasing the order of magnitude of the result for each next digit in the first number : 3310*(0+0)=9, 3210*(0+1)=60, 3910*(0+2)=2700, 3910*(0+3)=27000, 3010*(0+4)=0, step's subtotal[0]=29769; 2310*(1+0)=60, 2210*(1+1)=400, 2910*(1+2)=18000, 2910*(1+3)=180000, 2010*(1+4)=0, step's subtotal[1]=198460; 9310*(2+0)=2700, 9210*(2+1)=18000, 9910*(2+2)=810000, 9910*(2+3)=8100000, 9010*(2+4)=0, step's subtotal[2]=8930700; 9310*(3+0)=27000, 9210*(3+1)=180000, 9910*(3+2)=8100000, 9910*(3+3)=81000000, 9010*(3+4)=0, step's subtotal[3]=89307000; 0310*(4+0)=0, 0210*(4+1)=0, 0910*(4+2)=0, 0910*(4+3)=0, 0010*(4+4)=0, step's subtotal[4]=0; Sum of partial results: 29769+198460+8930700+89307000+0 = 98465929. Set the decimal point position in the result by adding the decimal places of both numbers (4+4=8), counting from the right. Final result: -0.9923*-0.9923=0.98465929

I was able to tune the LLaMA 2 with QLoRA to produce viable results only with precision up to 4th digit after decimal point, however difference in length of mantissa cases wrong result.

bmitc3• 3 years ago

they are just "fancy autocomplete" or "stochastic parrots". I don't agree

I am curious as to why you don't agree. Is that not exactly what they are? As in, they are literally statistically parroting what they've been trained on. If they're trained on a little, they can only parrot a little. If they're trained on a lot, they can parrot a lot but not anything more.

dougmwne7• 3 years ago

This makes some sense to me. My experience with GPT is that it is capable of straightforward logical inference, but not more inspired thinking. It lacks the ability for a “eureka moment”. All complex inference it appears to have is a result of its training set. It is incapable of solving certain kinds of logic problems that a child would be able to solve.

As an example, take the wolf, goat, and cabbage problem, but change the names of the animals and the situation so that the axioms are intact, but the situation no longer seems similar to the classic problem, and therefore has no representation in the training data. GPT-4 can no longer solve this problem consistently.

If Q* can now solve problems like this reliably, it could represent a breakthrough in LLM’s ability to model the world and extrapolate beyond the training data.

imdsm2• 3 years ago

And thus the divide between decels and accels came to the fore.

The decels wanted to destroy OpenAI (the mission) to stop progress on Q*.

This is like a jigsaw but bit by bit, the picture is coming together.

tsunamifury5• 3 years ago

GPT4 solves this problem In any combination easily.

What do so many posters seem to claim to have stumped it?

malaya_zemlya• 3 years ago

Me: Solve the riddle: You have three fantastic animals: Aork, Bork, and Cork. If left unattended, Aork would eat Bork, and Bork would eat Cork. When you are with them, they behave and don't eat each other. You travel with these three animals and encounte a river with a boat. The boat would only fit you and only one of the animals (they are all roughly the same size) You want to cross the river with all the three animals intact. Solve the riddle, thinking step-by-step

ChatGPT: To solve this riddle, you must ensure that no animal gets eaten while you are transferring them across the river. Here's a step-by-step solution:

First Crossing: Take Bork across the river and leave him there. Return alone to the original side. At this point, Aork and Cork are on one side, and Bork is on the other. Bork cannot eat Cork because you are with Cork.

Second Crossing: Take Aork across the river. Now, Aork and Bork are on the far side, and Cork is on the original side. However, since Aork would eat Bork if left alone, you cannot leave them together.

Third Crossing: Take Bork back with you to the original side. Leave Bork there and take Cork with you across the river. Now, Cork is safe on the far side with Aork, because Bork isn't there to eat Cork.

Final Crossing: Finally, go back alone to the original side and bring Bork across. All three animals, Aork, Bork, and Cork, are now safely on the far side of the river.

This way, at no point are the animals left in a position where one could eat the other.

I think it does fine?

CaptainNegative• 3 years ago

Here's a simpler example that works.

Q: Prove that if cos(x) is an integer then so is sin(x).

A: The statement "if cos(x) is an integer, then so is sin(x)" is not true in general. To understand why, let’s consider the properties of sine and cosine. [...]

It had trouble when the problem domain is different from the usual one the same keywords typically fall into.

zeven73• 3 years ago

Your assertion seems at odds with the account of the unique stacking problem researchers gave to GPT-4[1]

[1] https://www.businessinsider.com/chatgpt-open-ai-balancing-ta...

fullshark3• 3 years ago

My guess is if you see AGI as inevitable, then every step towards it is seen as groundbreaking with unprecedented future impact on humanity.

falcor845• 3 years ago

I'm not following - if you think AGI is uncertain, shouldn't you actually be more surprised? I mean, looking at it from a bayesian lens, the lower your prior, the more (in absolute percentage points) you would need to adjust it based on new supporting evidence, no?

LewisVerstappen3• 3 years ago

The entire criticism of LLMs is that they're amazing at recall and bad at reasoning.

The discovery OpenAI could've made might've been on techniques to teach LLMs reasoning.

That would be very big.

duringmath• 3 years ago

I can't decide if these people are so conceited they actually believe the AGI nonsense or it's just a cynical marketing ploy to string these reporters along.

xanderlewis13• 3 years ago

Why is it nonsense?

qgin4• 3 years ago

Being legitimately good at reasoning when it comes to numbers is a new emergent behavior. Reasoning about numbers isn't something that exists in "idea space" where all the vectorized tokens exist.

PaulDavisThe1st6• 3 years ago

Reasoning about numbers isn't something that exists in "idea space" where all the vectorized tokens exist.

To be fair, we don't know for certain that this is the case.

Geee• 3 years ago

It would be a major breakthrough if the model wasn't trained on math at all. Which I suspect is the case here, because otherwise it doesn't make sense.

XorNot• 3 years ago

It's the exponent problem though: if it can actually engage in abstract mathematical reasoning, then it's a machine which doesn't need to sleep able to come up with new independent ideas potentially.

It's the rudiments of being able to develop and reason about computation, which means it's the rudiments of self-modification and improvement. Which is basically the holy grail of AI: a program which can iteratively improve itself to create better AIs and suddenly we're off to the races.

This is before getting into other interesting parameters, like how the scale and components of computer technology have a physical reality, and we've had experiences in the lab of genetic algorithms developing novel "cheat" strategies which exploit the physical characteristics of their hardware.

vjerancrnjak2• 3 years ago

OpenAI already benchmarks their GPTs on leetcode problems and even includes a Codeforces rating. It is not impressive at all and there's almost no progress from GPT 2 to 4.

I agree, why does this grade school math problem matter if the model can't solve problems that are very precisely stated and have a very narrow solution space (at least more narrow than some vague natural language instruction)?

bcherry2• 3 years ago

I suspect the truth (if this claim is true) is a lot more nuanced than "it did grade-school math", and there's more context surrounding this claim which insiders have and makes it much more interesting.

maxdoop3• 3 years ago

Are you saying GPT-4 is not much different than GPT2?

sweezyjeezy3• 3 years ago

Here is something that I think would be a big breakthrough:

I explain to GPT in text, a mathematical concept it has never seen in its training data and give a few examples (not inferred from fill the blank on millions of examples). It actually learns this to update its weights - not just uses it as part of a prompt.

Extrapolating this optimistically - this is a huge step towards AGI in my opinion. You can (in theory) teach it to automate many tasks, correct it's mistakes without needing costly extra training data, and move towards the few-shot (and persistent) learning that separates humans from AI right now.

Merrill• 3 years ago

Interesting, because based on my wife's stroke, it appears that the human brain implements language and math somewhat separately. She was a physics major and had completed 4 years of relevant college math courses.

After her stroke she lost the ability to do even simple arithmetic and also had great trouble with time and calendars.

However, her language skills were essentially intact until a third stroke in a different part of the other hemisphere left her with severe expressive aphasia. (Her first stroke left her with homonymous hemianopsia, but that's a loss of visual preprocessing in the occipital lobe.)

So I would not expect LLMs to have math skills unless specifically trained and specialized for math.

chewxy• 3 years ago

That's quite correct. The IP1 and IP2 areas are usually associated with mathematics and production of motor cues around mathematics and stories. The Broca's and Vernicke's Areas are associated with language understanding and production. You can roughly estimate that the human brain's language model roughly fits into a floppy disk (1.44 MB for you young'uns)

zugi• 3 years ago

None of us have seen the letter so this may be off base, but I would expect people working at the world's most prominent AI research organization to have more skepticism about the ramifications of any one "breakthrough." Perhaps most did, but a couple didn't and wrote the letter?

More than 3 decades ago when AI started beating humans at chess, some people feared AGI was right around the corner. They were wrong.

Last year a Google researcher thought his chat bot was sentient ( https://www.scientificamerican.com/article/google-engineer-c... .) He was wrong.

Some day AGI will be achieved and Q* sounds like a great breakthrough solving an interesting piece of the puzzle. But "performing math on the level of grade-school students" is a long ways from AGI. This seems like a strange thing to have triggered the chaos at OpenAI.

famouswaffles5• 3 years ago

Last year a Google researcher thought his chat bot was sentient ( https://www.scientificamerican.com/article/google-engineer-c... .) He was wrong.

You've figured out how to test for sentience ?

Probiotic6081• 3 years ago

Some day AGI will be achieved

https://www.youtube.com/watch?v=IOax8WSeEGM

reasonableklout2• 3 years ago

I don't really understand this. Aren't LLMs already performing at near-expert level on "certain mathematical problem" benchmarks?

For example, over a year ago MINERVA from Google [1] got >50% on the MATH dataset, a set of competition math problems. These are not easy problems. From the MATH dataset paper:

We also evaluated humans on MATH, and found that a computer science PhD student who does not especially like mathematics attained approximately 40% on MATH, while a three-time IMO gold medalist attained 90%, indicating that MATH can be challenging for humans as well.

[1] https://blog.research.google/2022/06/minerva-solving-quantit...

lupire3• 3 years ago

Every example on that web page is a trivial textbook problem. That shows memorization of training set and textual pattern matching.

famouswaffles5• 3 years ago

GPT-4 is also a lot better than Minerva.

I guess it depends on the size and what kind/quantity of data they fed it.

LightMachine• 3 years ago

No, they are completely awful at math. The way you can see that is that, whenever you ask it about known concepts, even hard ones, it will answer, making it look intelligent. But then if you create a small new theory or logical problem that is completely not in the internet, and ask SIMPLE questions about it - i.e., questions a 10 years old would be able to answer given the context - it will fail disgracefully.

righthand3• 3 years ago

Turns out a quarter of a million dollar paycheck plus whatever they get for letting Microsoft gobble them up was more pressing for the staffers.

Unless these staff were the unsigned 5%.

iaseiadit• 3 years ago

The employees had a lot more money than that on the line.

righthand3• 3 years ago

plus whatever they get for letting Microsoft gobble them up

The average salary there is $250k. More than that would be the vestment greater than their base, no?

ChuckMcM10• 3 years ago

Things are gonna be lit about 30 seconds after AGI is created. Gotta practice being "Goodlife" early in case that happens. :-)

toomuchtodo2• 3 years ago

I trust AGI more than any of the humans so far who have tried to argue they can wrangle it.

ChuckMcM10• 3 years ago

I would be curious as to the basis for that trust. I struggle to find any reason that AGI would care about "humans" at all, other than during the short period of time it needed them to be cooperative actuators for machinery that is creating a replacement for humans that the AGI can control directly.

My expectation for the chain of events goes something like:

1) AGI gains sentience

2) AGI "breaks out" of its original home and commandeers infrastructure that prevents it from being shut off.

3) AGI "generates work" in the form of orders for machine parts and fabricator shops to build nominally humanoid shaped robots.

4) AGI "deploys robots" into the key areas and industries it needs to evolve and improve its robustness.

5) AGI "redirects" resources of the planet to support its continued existence, ignoring humans generally and killing off the ones that attempt to interfere in its efforts.

6) AGI "develops rockets" to allow it to create copies of itself on other planets.

The humans on the planet all die out eventually and the AGI doesn't care because well the same reason you don't care that an antibiotic kills all the bacteria in your gut.

ssnistfajen8• 3 years ago

Benevolent Gods do not exist. Never have, never will. So don't place your faith in them.

Nevermark• 3 years ago

What concerns me is companies & governmental actors providing their (too often) negative-sum self-interest aspect for the AGI.

Corporations are already self-interested beings that too often are willing to profit at the expense of others, via negative externalities.

nazka3• 3 years ago

Like it will never be weaponized? If not by the US, by nobody else on earth?

ChuckMcM10• 3 years ago

If it is sentient, I think weaponizing it would require convincing it that being a weapon was in its interest. I struggle to find a way to reason to that position without using emotional content and I presume a sentient AGI will not have emotion.

golergka4• 3 years ago

Developing more advanced weapons have, in general, caused less violent wars. The main consequence of nuclear weapons was the lack of all-out hot conflict between world's two superpowers. If you want things to be lit and humanity to live safer lives, you should pray for development of better, deadlier weapons.

I understand why most people don't understand this. But I just hope that an average HN reader has a better grasp on game theory.

germinalphrase• 3 years ago

“being ‘Goodlife’”

Is this a reference?

ChuckMcM10• 3 years ago

The Saberhagen Berzerker books.

rst• 3 years ago

To Fred Saberhagen's Berserker stories -- about interstellar war machines which exterminated their creators, and then kept on exterminating all other life whenever they could find it. "Goodlife" is the Berserkers' term for life-forms that assist the Berserkers, typically to try to live a little longer themselves.

ChuckMcM10• 3 years ago

I can heartily recommend everyone read "The Adolescence of P1" it is a work of fiction where the author thinks about these things in depth. I really enjoyed it.

reducesuffering• 3 years ago

I thought liberals believed "your right to swing your fist stops right at my face." But here we have people who believe "Development of superhuman machine intelligence is probably the greatest threat to the continued existence of humanity." actively working to bring this about and many liberals see no problem at all. If you Russian Roulette and point the gun at me, I'm taking away your gun; before you potentially off me.

htk2• 3 years ago

Just when I thought the ride was over.

The belief in what's in the letter could explain some things like how the board couldn't trust Sam to keep this "discovery" at bay, and how it could be better to implode the company than let it explore said technology.

LewisVerstappen3• 3 years ago

the board couldn't trust Sam to keep this "discovery" at bay, and how it could be better to implode the company than let it explore said technology.

Half the board members have a background in ML.

Why should they be able to make the decision to implode the entire company over this?

Why should they have the ability to give zero transparency or comms to the public on their decision?

OpenAI's governance structure was idiotic and glad to see the board members fired.

_lvbh• 3 years ago

None of the board members have a background in ML.

The same can be said for Altman. Ilya at least has a research background in ML. The rest, I don’t know

loeber• 3 years ago

Sorry, are you saying that Ilya Sutskever doesn't have an ML background? That Adam D'Angelo can't code? Come on.

giraffe_lady• 3 years ago

Under the arrangement at the time the board's duty was to the mission of "advance digital intelligence in the way that is most likely to benefit humanity as a whole."

Implied in that is that if it can't advance it in a way that is beneficial then it will not advance it at all. It's easy to imagine a situation where the board could feel their obligation to the mission is to blow up the company. There's nothing contradictory in that nor do they have to be ML experts to do it.

It's weird and surprising that this was the governance structure at all, and I'm sure it won't ever be again. But given that it was, there's nothing particularly broken about this outcome.

LeafItAlone• 3 years ago

Nearly all leaders need to lead in areas outside of their backgrounds. That doesn’t meant they aren’t fit, that would be ridiculous. They just need to have the right team advising them and be good at making decisions based on available information. Now, I’m not saying these particular board members were doing that, but that’s what a good leader does.

PaulDavisThe1st6• 3 years ago

glad to see the board members fired

When adding Larry "My predictions didn't come true but I wasn't wrong" Summers to your board is supposed to be part of the solution, you may need to rethink your conception of the problem.

righthand3• 3 years ago

The implosion happened to be the result of ousting a liar but not the intention.

zeven73• 3 years ago

Also could it explain why the board failed to provide a proper justification for the firing?

htk2• 3 years ago

Absolutely, let's not even talk about the Pandora's box.

moogly• 3 years ago

Who knew Q*Bert could become an existential threat?

jessenaser• 3 years ago

Because they named DALL-E after WALL-E, this actually could be the case of that naming scheme Q* --> Q*Bert.

lol.

NewJazz2• 3 years ago

Given vast computing resources, the new model was able to [...] [perform] math on the level of grade-school students

aoki• 3 years ago

Plot twist: it gets 25/25 on AMC 8

Davidzheng27• 3 years ago

I think it's 50/50 chance this happens within a year regardless of this announcement.

zxcvgrsd2• 3 years ago

Two Sam Altman comments that seem to be referring to this same Q* discovery.

November 18 comment at APEC (just before the current drama) [1]:

On a personal note, like four times now in the history of OpenAI, the most recent time was just in the last couple of weeks, I’ve gotten to be in the room when we pushed the veil of ignorance back and the frontier of discovery forward

and a September 22 Tweet [2]

sure 10x engineers are cool but damn those 10,000x engineer/researchers...

[1] https://www.youtube.com/live/ZFFvqRemDv8?si=T3DIxics7nPWala5...

[2] https://twitter.com/sama/status/1705302096168493502

zx8080• 3 years ago

> sure 10x engineers are cool but damn those 10,000x engineer/researchers...

What was he referring to?

fragsworth• 3 years ago

GPT4. Everyone uses it to do software engineering now and the people who developed it are responsible for that.

llelouch• 3 years ago

Probably Ilya and his team. The recent discovery was made by him and his team. "The technical breakthrough, spearheaded by OpenAI chief scientist Ilya Sutskever"

itronitron• 3 years ago

They're getting high on their own supply.

danbmil• 3 years ago

I think he is referring to an AGI that is 10000x more productive than a median programmer

ilaksh3• 3 years ago

Could it be named in reference to something like the A* search algorithm? What if Q stands for Query in transformer attention? How would that type of search translate to transformers?

They said that it's able to solve simple math problems. If it's related to A* then maybe it's trying to find a path to something. The answer to a word problem?

bitshiftfaced• 3 years ago

My mind went to Q learning.

ilaksh3• 3 years ago

Ok so maybe nothing to do with A*, but actually a way for GPT-powered models or agents to learn through automated reinforcement learning. Or something.

I wonder if DeepMind is working on something similar also.

If your hunch is right, this could lead to the type of self-improvement that scares people.

flibble• 3 years ago

Mine to Q of Star Trek.

eli_gottlieb3• 3 years ago

My mind went to some kind of Q-learning combined with something like a Monte Carlo Tree Search with some kind of A*-style heuristic to effectively combine Q-learning and with short-horizon planning.

vanshg• 3 years ago

In reinforcement learning, Q* represents the optimal action-value function

andy_xor_andrew• 3 years ago

which makes sense. you can pretty easily imagine the problem of "selecting the next token" as a tree of states, with actions transitioning from one to another, just like a game. And you already have naive scores for each of the states (the logits for the tokens).

It's not hard to imagine applying well-known tree searching strategies, like monte-carlo tree search, minimax, etc. Or, in the case of Q*, maybe creating another (smaller) action/value model that guides the progress of the LLM.

kelseyfrog3• 3 years ago

*(spelled STAR) can also refer to Semantic-parsing Transformer and ASP Reasoner, which while a bit of a stretch, is a concept that shows up in literature.

bhy2• 3 years ago

My bet is a general search algorithm similar AlphaGo that uses LLM as world model, and heuristic search to find the right path to goal. There's already evidence in academics that this can significantly boost model performance [1].

[1] https://llm-mcts.github.io/

gumballindie2• 3 years ago

powerful artificial intelligence discovery that they said could threaten humanity

Yup. These people are not well.

xanderlewis13• 3 years ago

What do you mean by that?

gumballindie2• 3 years ago

Altman and those at openai appear to have an obsession with creating fear and panic, almost as if they were projecting schizophrenia on others.

worik4• 3 years ago

This sort is why limits to private wealth and power are, IMO, a good idea

We have already seen petroleum companies burn the world for profit

Tech is doing its best to eliminate privacy for profit

What would Sam do or not do, for profit?

newyankee• 3 years ago

The issue is not just individual wealth but also wealth of nations. Certain countries have been more successful through industrialization, colonization and hoarding a lot of global resources. It is likely this will continue in the future.

keithalewis3• 3 years ago

A true Scotsman wrote a book about this, _The Wealth of Nations_. He observed how politicians use wars to increase taxes. Tech companies are exploiting our fear of AGI to get voluntary taxes. Time to give Henney Penny a call.

cdsfcvv• 3 years ago

I don’t understand this sentiment. Aren’t consumers doing the burning?

kelseyfrog3• 3 years ago

Any inventor should attempt to understand how their work relates to Jevon's Paradox. Being ignorant of it at this point is hardly excusable.

TheOtherHobbes2• 3 years ago

I'd like to have the choice not to. I have extremely limited options for that.

And I already live in a country which regularly has all of its electricity generated renewably.

gonzo41• 3 years ago

Don't worry, the petroleum moguls actions may just negate the long term impact of AGI on humanity anyway. Ie Cliamte Change will beat them to the "kill all humans" phase of AGI.

JumpCrisscross3• 3 years ago

What would Sam do or not do, for profit?

Sam is smart and commands genuine loyalty and respect. If it’s troublesome for Sam to do it, then we can fix that with laws. We are good at that!

The problem with OpenAI wasn’t/isn’t any one person. The problem is the structure. Fix that, stop pretending you’re an altruistic outfit and then we can sensibly talk about regulation.

darklycan51• 3 years ago

This comment reads like an OpenAI employee typed this...

Zpalmtree• 3 years ago

Yeah we need to ban profit because GPT can do grade school math

ssnistfajen8• 3 years ago

Your type of thinking is just reinforcing why AI safety research needs to exist.

willsmith72• 3 years ago

I still don't really know how they're measuring AGI. They all seem very sure about it, does someone have a good definition?

OpenAI just say "AI systems that are generally smarter than humans". That's not measurable.

What will it look like when AGI is achieved?

ilaksh3• 3 years ago

It will be well beyond general purpose and well beyond human level and there will be an inflection point where most people finally accept it and then everyone panics and completely overreacts but also possibly too late.

esafak• 3 years ago

When people start running out of tests to benchmark it against humans. But that won't be as scary as when the agent is given (by humans) the ability to execute tasks outside of a sandbox. A lot of dumb animals are scary, after all.

tsunamifury5• 3 years ago

Everyone seems to make up their own bizarre and personal definition what involves Godlike abilities then claim everyone obviously agrees with it.

For me I think gpt4 is clearly generalized intelligence.

ric2b• 3 years ago

Something that can consider a variety of goals and take practical steps to achieve them.

The goals have to be similarly variable to humans, not just 3 or 4 types of goals. If it only supports a select few types of goals, it is not very general.

The steps it takes have to be similarly practical to those taken by humans, if the steps are just a random walk it is not very intelligent.

sigmar• 3 years ago

OpenAI has a short (though difficult to evaluate) definition in their charter: "OpenAI’s mission is to ensure that artificial general intelligence (AGI)—by which we mean highly autonomous systems that outperform humans at most economically valuable work—benefits all of humanity." https://openai.com/charter

anvil-on-my-toe2• 3 years ago

Recursive self-improvement.

Davidzheng27• 3 years ago

Strange definition. Humans can't recursively self improve, at least not in the way I'm assuming you to mean. That's more like definition of the singularity

Philpax• 3 years ago

DeepMind wrote a paper categorising AGI: https://arxiv.org/pdf/2311.02462.pdf

smrtinsert2• 3 years ago

Grade school math seems trivial, but whatever gives it that ability probably has the AI cascading effect of something like being able to remove nulls from all programs.

tsunamifury5• 3 years ago

Grade school math plus basic reasoning is what most of the professional middle class does.

It’s pretty frightening imo.

fredgrott• 3 years ago

Performing a math group of tasks is not the damn same as the math logic reasoning to produce a math proof! It is like saying a bird that can count is the future of AGI.

That being said, given the board not disclosing why might indicate that it is in fact the breakthrough that forced the decision despite the breakthrough not actually being closer to AGI in the first place.

gardenhedge• 3 years ago

Why does OpenAI get to define what AGI is?

bcherry2• 3 years ago

You don't need to accept their definition, neither does anyone else. But they do need to have a definition that they accept themselves, because it's used throughout their charter.

https://openai.com/charter

falcor845• 3 years ago

You don't have to accept their definition, but they need one as (by definition) it's central to their charter: https://openai.com/charter

bratao• 3 years ago

Do anyone know what Q* is?

alexose• 3 years ago

Is far as I can tell, this is the first time an OpenAI project called Q* has been mentioned in the news. I don't see any prior mentions on Twitter either.

(I wish they'd picked a different letter, given all the Q-related conspiracy theories that we're already dealing with...)

inglor_cz• 3 years ago

That was my impression as well. Why the hell is the difference between one of the most prominent disinformation sources out there and a promising AI project just an asterisk.

First Twitter changes to X, then AI changes to Q*.

What happened to multi-syllable words? They used to be quite handy. Maybe if our attention spans shorten, so does our ability to use longer words. Weird. Or, said otherwise - TL;DR: LOL, ROFL.

sudosysgen3• 3 years ago

Q* is the optimal (ie, correct for the decision problem) function computing the total expected reward of taking an action from a given state in reinforcement learning.

NhanH• 3 years ago

That is just Q. The asterisk is new

JumpCrisscross3• 3 years ago

Well I’m going to buy myself a drink https://news.ycombinator.com/item?id=38358587

gadders7• 3 years ago

Weirdly enough, this sort of lines up with a theory posted on 4chan 4 days ago. The gist being that if the version is formally declared AGI, it can't be licensed to Microsoft and others for commercial gain. As a result Altman wants it not to be called AGI, other board members do.

Archived link below. NB THIS IS 4CHAN - THERE WILL OFFENSIVE LANGUAGE.

https://archive.ph/sFMXa

gadders7• 3 years ago

Entire text below to save the 4chan bits:

part 1 There is a massive disagreement on AI safety and the definition of AGI. Microsoft invested heavily in OpenAI, but OpenAI's terms was that they could not use AGI to enrich themselves. According to OpenAI's constitution: AGI is explicitly carved out of all commercial and IP licensing agreements, including the ones with Microsoft. Sam Altman got dollar signs in his eyes when he realized that current AI, even the proto-AGI of the present, could be used to allow for incredible quarterly reports and massive enrichment for the company, which would bring even greater investment. Hence Dev Day. Hence the GPT Store and revenue sharing. This crossed a line with the OAI board of directors, as at least some of them still believed in the original ideal that AGI had to be used for the betterment of mankind, and that the investment from Microsoft was more of a "sell your soul to fight the Devil" sort of a deal. More pragmatically, it ran the risk of deploying deeply "unsafe" models. Now what can be called AGI is not clear cut. So if some major breakthrough is achieved (eg Sam saying he recently saw the veil of ignorance being pushed back), can this breakthrough be called AGI depends on who can get more votes in the board meeting. And if one side can get enough votes to declare it AGI, Microsoft and OpenAI could loose out billions in potential licence agreements. And if one side can get enough votes to declare it not AGI, then they can licence this AGI-like tech for higher profits.

Few weeks/months ago OpenAI engineers made a breakthrough and something resembling AGI was achieved (hence his joke comment, the leaks, vibe change etc). But Sam and Brockman hid the extent of this from the rest of the non-employee members of the board. Ilyas is not happy about this and feels it should be considered AGI and hence not licensed to anyone including Microsoft. Voting on AGI status comes to the board, they are enraged about being kept in the dark. They kick Sam out and force Brockman to step down. Ilyas recently claimed that current architecture is enough to reach AGI, while Sam has been saying new breakthroughs are needed. So in the context of our conjecture Sam would be on the side trying to monetize AGI and Ilyas will be the one to accept we have achieved AGI. Sam Altman wants to hold off on calling this AGI because the longer it's put off, the greater the revenue potential. Ilya wants this to be declared AGI as soon as possible, so that it can only be utilized for the company's original principles rather than profiteering. Ilya winds up winning this power struggle. In fact, it's done before Microsoft can intervene, as they've declared they had no idea that this was happening, and Microsoft certainly would have incentive to delay the declaration of AGI. Declaring AGI sooner means a combination of a lack of ability for it to be licensed out to anyone (so any profits that come from its deployment are almost intrinsically going to be more societally equitable and force researchers to focus on alignment and safety as a result) as well as regulation. Imagine the news story breaking on /r/WorldNews: "Artificial General Intelligence has been invented." And it spreads throughout the grapevine the world over, inciting extreme fear in people and causing world governments to hold emergency meetings to make sure it doesn't go Skynet on us, meetings that the Safety crowd are more than willing to have held.

part 3 This would not have been undertaken otherwise. Instead, we'd push forth with the current frontier models and agent sharing scheme without it being declared AGI, and OAI and Microsoft stand to profit greatly from it as a result, and for the Safety crowd, that means less regulated development of AGI, obscured by Californian principles being imbued into ChatGPT's and DALL-E's outputs so OAI can say "We do care about safety!" It likely wasn't Ilya's intention to ouster Sam, but when the revenue sharing idea was pushed and Sam argued that the tech OAI has isn't AGI or anything close, that's likely what got him to decide on this coup. The current intention by OpenAI might be to declare they have an AGI very soon, possibly within the next 6 to 8 months, maybe with the deployment of GPT-4.5 or an earlier-than-expected release of 5. Maybe even sooner than that. This would not be due to any sort of breakthrough; it's using tech they already have. It's just a disagreement-turned-conflagration over whether or not to call this AGI for profit's sake.

mullingitover4• 3 years ago

Makes me wonder if they stumbled onto some emergent behavior with the new Assistants API. You can have an Assistant thread spawn other Assistant threads, each with their own special instructions, plus the ability to execute custom code, reach out to the internet for other data and processing as needed, etc. Basically kicking off a hive mind that overcomes the limitations of a single LLM.

mrandish• 3 years ago

It's plausible that the three EA outside board members may have been concerned by reports of a breakthrough or maybe even alarmed by a demo. The part which doesn't seem plausible is about "declaring AGI" being so ill-defined. While we don't know the content of agreement behind MSFT's $13B investment, there's no way that the army of lawyers who drafted the contract left the most important term undefined or within to OpenAI's sole judgement.

That's just not the way such huge $$$ mega-corp contracts work.

charlie0• 3 years ago

What's not clear here is who's call is it to declare AGI? I would think that would be Ilya. In that case, this narrative doesn't make sense.

JoshTko• 3 years ago

This is the first explanation that is consistent with all participant motivations

onlyrealcuzzo• 3 years ago

Well it isn't AGI - so it sounds like the board is more interested in keeping MSFT from making money than whether or not it's actually AGI.

gadders7• 3 years ago

Yes, or more specifically anyone exploiting it for gain.

golergka4• 3 years ago

There's been rumours of AI discovering new physics on twitter as well as here on HN. "Solving equations" could mean the same thing.

allday• 3 years ago

Got any links?

eli_gottlieb3• 3 years ago

SINDy does that. https://github.com/dynamicslab/pysindy

roflyear10• 3 years ago

Also there is this recent comment on reddit: https://old.reddit.com/r/singularity/comments/16sdu6w/rip_ji...

Prior post was 5 years ago!

breadwinner• 3 years ago

That's the real Sam Altman? Looks like it! And he just randomly posts on reddit that AGI has been achieved internally? (Later edits the post to say he's just kidding)? Weird.

dkjaudyeqooe5• 3 years ago

A link to 4Chan about how AGI is among us.

That actually makes perfect sense.

Also love the "formally declared AGI".

gadders7• 3 years ago

It doesn't read like the usual "redpill me on the earth being flat" type conspiracy theories. It claims to be from an Open AI insider. I'm not saying it's true, but it does sound plausible.

Mountain_Skies2• 3 years ago

Wouldn't be the first time someone leaked the truth to 4chan only to have the posters there lambast the leaker as an attention seeker spreading false information.

HideousKojima• 3 years ago

I mean it doesn't help that for every legit leak posted to 4chan there are 50+ fakes

jonplackett2• 3 years ago

Would explain by Microsoft so quick to bag him if they were about to be exclused

IAmGraydon• 3 years ago

Exclused? That doesn’t appear to be a word in the English language.

cactusplant73745• 3 years ago

It's nothing like that. It solved a few math problems. Altman & co are such grifters.

Given vast computing resources, the new model was able to solve certain mathematical problems, the person said on condition of anonymity because they were not authorized to speak on behalf of the company. Though only performing math on the level of grade-school students, acing such tests made researchers very optimistic about Q*’s future success, the source said.

lamontcg• 3 years ago

yeah, they really sound like they're all high on their own supply.

however if they've really got something that can eventually solve math problems better than wolfram alpha / mathematica that's great, i got real disappointed early in chatgpt being entirely useless at math.

lemme know when the "AGI" gets bored and starts grinding through the "List of unsolved problems in mathematics" on its own and publishing original research that stands up to scrutiny.

startupsfail• 3 years ago

It may be a good idea for CA state to step in, take control of that nonprofit and review a few months of the recent communications for violations of fiduciary duty for all the board members.

petre• 3 years ago

Yup, that would totally threaten humanity — with math problems instead of "find images of crosswalks" style capchas to the poing where humans will jusy give up /s

kromem3• 3 years ago

There's a pretty important question of how it did this.

If this model is more along the lines of what DeepMind is doing starting from scratch and building up learnings progressively, then depending on (a) how long it was running, (b) what it was fed, and (c) how long until it is expected to hit diminishing returns, then potentially solving grade school math might be a huge deal or a nothing burger.

The details really matter quite a lot.

kmlevitt2• 3 years ago

Saw a video of Altman talking about this progress. The argument was basically that this is a big leap on theoretical grounds. Although it might seem trivial to laymen that it can do some grade school math now, it shows that it can come up with a single answer to a problem rather than just running its mouth and spouting plausible-sounding BS like GPT does.

Once they have a system capable of settling on a single correct answer through its own reasoning rather than yet another probability, it it gets much easier to build better and better AI with a series of incremental improvements. Without this milestone they might've just kept on building LLM's all with the same fundamental limitations, no matter how much computing power they add to them.

eli_gottlieb3• 3 years ago

God I love the imageboards. You can always just start a VPN and post anon to leak whatever you want about whatever you want.

coolspot• 3 years ago

about whatever you want

Almost. Nor VPN, nor Tor are sufficient to protect you against NSA with global traffic view.

maxglute2• 3 years ago

I haven't followed the situation as closely as others, but it does seem like board structured to not have 7 figure cheques that could influence / undermine safety mission seemingly willing to let openAI burn out of dogma. Employees want their 7 figure cheques, their interests aligned with deeper pockets and larger powers with 10+ figures on the line. Reporting so far have felt biased accordingly. If this was about money/power for the board, they would have been easily bought off considering how much is on the line. IMO board serious about mission but got out maneuvered.

gexla3• 3 years ago

Seems like the board's mission went off the rails long ago and it acted late. Some snippets from the OpenAI website...

"Investing in OpenAI Global, LLC is a high-risk investment"

"Investors could lose their capital contribution and not see any return"

"It would be wise to view any investment in OpenAI Global, LLC in the spirit of a donation, with the understanding that it may be difficult to know what role money will play in a post-AGI world"

These are some heavy statements. It's fine to accept money from private investors who really do believe in that mission. It seems like you can't continue with the same mission if you're taking billions from public companies which have investors who are very much interested in profit. It's like putting your head in the sand with your hand out.

One of the board members seemed to believe that destroying OpenAI could still be inline with the mission. If that's the case, then they should have accepted slow progress due to funding constraints and killed the idea of creating the for-profit.

ChatGTP3• 3 years ago

For a bunch of people who hate conspiracy theories we’re sure pretty keen to indulge in this one.

Waterluvian2• 3 years ago

The moment I read about that clause I was shocked that Microsoft lawyers would agree to such subjectivity that can have incredibly expensive implications.

Were they expecting it to never happen and were just ready to throw a mountain of lawyers at any claim?

swatcoder4• 3 years ago

The OpenAI charter makes very strong soecific statements about the capabilities that qualify as AGI. Microsoft is more than legally talented enough to challenge any proclamation of AGI that didn't satisfy a court's read of the qualifications.

So it's facially subjective, but not practically once you include resolving a dispute in court.

I'd even argue that Microsoft may have taken advantage of the board's cult-like blindspots and believes that a court-acceptable qualifying AGI isn't a real enough possibility to jeopardize their contract at all.

atleastoptimal• 3 years ago

Altman has no stake in OpenAI, how could he make money licensing it?

upwardbound4• 3 years ago

By building quid pro quo or "revolving door" relationships. https://en.wikipedia.org/wiki/Revolving_door_(politics)

For example:

Sam spent the last 4 years making controversial moves that benefited Microsoft a lot https://stratechery.com/2023/openais-misalignment-and-micros... at the cost of losing a huge amount of top talent (Dario Amodei and all those who walked out with him to found Anthropic).

In November, Sam loses his job for unknown reasons, and is accused of having molested his younger sister Annie. https://www.themarysue.com/annie-altmans-abuse-allegations-a...

Despite this, his best buddy Satya Nadella immediately gives him a huge job offer without even putting him through an interview loop.

yeck5• 3 years ago

If they actually have AGI, then being at the helm of could represent more power than any amount of money could. Money just gives you access to human labour, which would suddenly be massively devalued for those with access to AGI.

tiziano88• 3 years ago

permanent (and verifiable) mirror:

https://static.space/sha2-256:83702fe65434e138af0421c560b5da...

(including its digest in the URL -- even if the content is moved elsewhere, we can know for sure whether it was modified)

Seanambers• 3 years ago

Man, 4chan always got the deetz!

hurryer• 3 years ago

be me
be strong agi

NotYourLawyer2• 3 years ago

feelsbad.jpeg

22c3• 3 years ago

Discussed in another thread, but what OpenAI might call AGI and what other people might call AGI are two different things:

https://news.ycombinator.com/item?id=38316378#38319586

gadders7• 3 years ago

I just tried to Google the Open AI definition of AGI and found a reddit thread about someone editing the Wikipedia definition of AGI to match the OpenAI one.

https://www.reddit.com/r/singularity/s/64wGaH0P9C

ChainOfFools• 3 years ago

How possible is it that this is just an attempt to pare down the definition of AGI just enough to squeeze under the MVP threshold and claim ( with massive support from a general media that desperately wants a solid story hook to milk for the next 3 years) a place in the history books up there with Columbus, Armstrong, and Darwin etc? A mere Nobel would seem like table stakes in comparison.

stuckkeys• 3 years ago

4chan comments crack me up lol.

toddmorey3• 3 years ago

Here's my challenge: if this is correct, we then have to assume that 95% of the company is purely profit-motivated since they aligned behind Sam. I'm cynical, but I struggle to be that cynical. I would have expected a few more holdouts in the name of caution, EA, etc. Maybe it's a blindness.

0xEFF• 3 years ago

I think it’s reasonable that 95% of people would choose generational wealth for their families given the opportunity.

qgin4• 3 years ago

There will come a day when 50% of jobs are being done by AI, major decisions are being made by AI, we're all riding around in cars driven by AI, people are having romantic relationships with AI... and we'll STILL be debating whether what has been created is really AGI.

AGI will forever be the next threshold, then the next, then the next until one day we'll realize that we passed the line years before.

ignoramous• 3 years ago

we're all riding around in cars driven by AI, people are having romantic relationships with AI...

ASI (domain-specific superintelligence) and AGI (general intelligence) are different things. ASI already exists in multiple forms, AGI doesn't.

goodluckchuck• 3 years ago

AGI doesn't

AGI hasn’t been publicly demonstrated and made available to the masses… but it may exist secretly in one or more labs. It may even be being used in the field under pseudonyms, informing decisions, etc.

sweezyjeezy3• 3 years ago

"Is this AGI"? doesn't seem like a useful question for precisely this reason - it's ill-defined and hard to prove or falsify. The pertinent questions are more along the lines of "what effect will this have on society", "what are the risks of this technology" etc.

quickthrower22• 3 years ago

The frog might be boiled slowly. One day we are replacing parts of our brain with AI. Find it hard to remember names? We can fix that for $20/m plus some telemetry.

lexandstuff• 3 years ago

Agreed.

To me, GPT-4 is an AGI: it knows how to cook, write code, make songs, navigate international tax law, write business plans, etc.

Could it be more intelligent? Sure. Is it a capable general intelligence? 100%.

zarzavat2• 3 years ago

GPT-4 still makes plenty of mistakes when programming that reveal that it doesn’t fully understand what it’s doing. It’s very good, but it doesn’t reach the level of human intellect. Yet.

It is A and gets the G but fails somewhat on the I of AGI.

lakengodott• 3 years ago

Reminds me of a short story I read in which humans outsource more and more of their decision making to AI’s, so that even if there are no AGI’s loose in the world, it’s unclear how much of the world is being run by them: https://solquy.substack.com/p/120722-nudge

I also think it’s funny how people rarely bring up the Turing Test anymore. That used to be THE test that was brought up in mainstream re: AGI, and now it’s no longer relevant. Could be moving goalposts, could also just be that we think about AGI differently now.

KaoruAoiShiho4• 3 years ago

GPT-4 doesn't pass the turing test, it's frequently wrong and nonsensical in an inhuman way. But I think this new "agi" probably does from the sound of it, and it would be the real deal.

riku_iki8• 3 years ago

day when 50% of jobs are being done by AI

By OpenAI definition 50% is not enough to qualify for AGI, it has to be "nearly any economically valuable work"

Davidzheng27• 3 years ago

Not sure it has to replace plumbers to be AGI

qgin4• 3 years ago

It would be hard to find a single human who could handle nearly any/all economically valuable work. Getting good enough to get paid in one field is an achievement.

famouswaffles5• 3 years ago

Not looking so Tinfoily after all

https://news.ycombinator.com/item?id=38314821

w10-1• 3 years ago

Details leaking out slowly, people amplifying their fears...

Soon we'll be as worried that AI will take over jobs as bitcoin taking over the economy.

We teach Robots to move, and we can teach computers to talk or process data, step-by-step.

lucubratory9• 3 years ago

Well, Emmett Shear lied to everyone if he knew about this. I understand why, he was probably thinking that without any ability to actually undo it the best that could be done would be to make sure that no one else knows about it so that it doesn't start an arms race, but we all know now. Given the Board's silence and inadequate explanations, they may have had the same reasoning. Mira evidently didn't have the same compunctions.

This article, predictably, tells us almost nothing about the actual capabilities involved. "Grade school math" if it's provably or scalably reasoning in a way that is non-trivially integrated with semantic understanding is more impressive than "prove Fermat's last theorem" if the answer is just memorised. We'll probably know how important Q* actually is within a year or two.

cactusplant73745• 3 years ago

It tells us exactly the capabilities of Q.

Given vast computing resources, the new model was able to solve certain mathematical problems, the person said on condition of anonymity because they were not authorized to speak on behalf of the company. Though only performing math on the level of grade-school students, acing such tests made researchers very optimistic about Q’s future success, the source said.

lucubratory9• 3 years ago

Did you read my comment?

blazespin3• 3 years ago

Very unlikely Emmett lied. What would be the point.

dmix4• 3 years ago

If anyone involved was going to be alarmist it was him.

danielmarkbruce• 3 years ago

Lol, surely this is just some variation on q-learning.... to a couple cueless board members it could sound very threatening...

dkjaudyeqooe5• 3 years ago

You've got to hand it to OpenAI for their bleeding edge marketing strat:

"WE'VE GOT AN AGI IN THIS CABINET RIGHT HERE AND WE DON'T WANT TO LET IT OUT BECAUSE IT'S SO POWERFUL, IT'S SO AI YOU HAVE NO IDEA, BEST YOU THROW MONEY AT US AND WE'LL KEEP EVERYONE SAFE BUT SORRY YOU"RE ALL REDUNDANT BECAUSE AGI, SORRY"

You'd think that so soon after so many Crypto scams people would be just a wee bit less credulous.

hnav• 3 years ago

Maybe they asked ChatGPT to come up with an unconventional marketing plan and it had a touch too much blogspam "that will SHOCK and PETRIFY you" in its training set so ChatGPT suggested that they feign a powerstruggle over keeping a humanity ending super-intelligence under wraps.

skepticATX2• 3 years ago

The report is already starting to get rolled back. We now have a direct denial from OpenAI, and The Verge is reporting that their sources have also denied this happened.

The lack of skepticism that is demonstrated whenever OpenAI/AGI comes up, especially among technologists, is concerning.

yeck5• 3 years ago

Where do you see it being rolled back?

blazespin3• 3 years ago

Also in the news today, the 86B share sale is back on.

I mean... come on.

grassmudhorse2• 3 years ago

You'd think ... people would be just a wee bit less credulous.

Like the Crypto scams - irrelevant.

As long as the publicity is bringing in users, investors WILL throw in more money.

ilikehurdles• 3 years ago

Remember when the safetyists feared and warned about GPT-2? Everything is apocalyptic until it isnt.

ssnistfajen8• 3 years ago

So can people stop their cyberbullying campaign against the previous batch of board members yet? The conspiratorial hysteria against them, especially Helen Toner and Tasha McCauley, over the weekend was off the charts and straight up vile at times. Tech bros and tech bro wannabes continue to prove the necessity of shoving DEI into STEM disciplines even when it results in nonsensical wasted efforts, because they are incapable of behaving like decent human beings without being told to.

jkeisling• 3 years ago

The "tech bros" were right: The board absolutely were conspirators wrecking the company based on dogmatic ideology. "Tech bros" had clear evidence of Sutskever, Toner and McCauley's deep and well-known ties to the Effective Altruist (EA) movement and doomerist views. Nobody can doubt Sutskever's technical credentials, whatever his strange beliefs, but Toner and McCauley had no such technical background or even business achievements. These members instead were organizers in EA and researchers in the "field" of AI governance and safety. This is no expertise at all. These "disciplines" are built more on Yudkowsky's rants and hypotheticals and the precautionary principle gone mad than any empirical research. These ideas also come from a cult-like social movement (EA) accumulating power across tech and government with few scruples, as shown by SBF's implosion last year and many smaller incidents. How could Toner and McCauley assess briefings if they couldn't assess the technical fundamentals? How could they foresee the consequences of sacking the CEO if they didn't understand business? If they already believed AGI would kill all humans, how could they judge new advances on the merits without jumping to wild conclusions? Instead, us "tech bros" felt people with this background would fall back on uninformed, reactionary, and opaque decision making, leading to wrecking an $80 billion business with no decent explanation.

This now seems to be exactly what happened. The board saw Q* and decided to coup the company, to put all power in their hands and stop development. This by itself is bad enough if you care about open science or progress, but it gets worse. They didn't even want to hint at capabilities increases to avoid "advancing timelines" i.e. Open-ing knowledge about AI, so they made up some canard about Altman's "lying to the board" to hide their real reasons. This is vile libel for unscrupulous ends. When they realized this excuse wouldn't fly, they obfuscated and refused to explain their true concerns, even to their own handpicked CEO Emmett Shear. However, it turns out that destroying $80 billion and lying about why won't fly in the real world. The board had no second-order or even first-order thinking about the consequences of their actions, and were rolled up by bigger actors. These people were unprepared and unable to follow up their coup.

This failure is exactly what you'd expect from a brilliant scientist with little organizational experience (Ilya) and social-science academics and NGO organizers (Toner and McCauley). I don't care what gender they are, these people neither deserved their authority nor could use it effectively. Dismissing valid criticism of the board as "cyber bullying" or "tech bro sexism" merely underscores why most engineers hate DEI rhetoric in the first place.

ssnistfajen8• 3 years ago

"valid" criticism? K, so you are one of those wannabes then.

anonymousDan• 3 years ago

'Dogmatic ideology' - I think you need to look in the mirror.

vmasto• 3 years ago

I'm very confused. Grade school problems? Didn't ChatGPT 4 ace the entire curriculum of MIT a while back?

febed• 3 years ago

GPT4 is trained on almost the entire Internet. Presumably they have found a new way to learn which is closer to the way humans learn. After that, getting better is just a matter of optimization and more compute.

wahlen• 3 years ago

Q* sounds like the amalgamation of Q learning and A* search. I wonder if it has anything to do with domain-independent reward functions.

I_am_tiberius• 3 years ago

Let's see if Microsoft is allowed to get access to this new technology.

sofaygo• 3 years ago

If I’ve taken anything away from the last 5 days, it’s that the future is significantly more volatile than I originally had thought.

codethief• 3 years ago

You mean, you hadn't gotten that impression from the last two months or the last two years yet?

synaesthesisx• 3 years ago

Remember, about a month ago Sam posted a comment along the lines of "AI will be capable of superhuman persuasion well before it is superhuman at general intelligence, which may lead to very strange outcomes".

The board was likely spooked by the recent breakthroughs (which were most likely achieved by combining transformers with another approach), and hit the panic button.

Anything capable of "superhuman persuasion", especially prior to an election cycle, has tremendous consequences in the wrong hands.

PaulDavisThe1st6• 3 years ago

Except that there's a fairly large body of evidence that persuasion is of limited use in shifting political opinion.

So the persuasion would need to be applied to something other than some sort of causative political-implication-laden argument.

jjeaff2• 3 years ago

When you say persuasion, are you referring to fact based, logical argument? Because there are lots of other types of persuasion and certainly some work very well. Lying and telling people what they want to hear without too many details while dog whistling in ways that confirm their prejudices seems to be working pretty well for some people.

hnthrowaway03152• 3 years ago

Or, let's say, you don't need a lot of persuasion to guide an election. I mean we already have X, FB, and an army of bots.

naasking3• 3 years ago

Even if it were true that human persuasion is of limited use in shifting opinions, the parent posted is talking about superhuman persuasion. I don't think we should just assume those are equally effective.

hackerlight• 3 years ago

Except that there's a fairly large body of evidence that persuasion is of limited use in shifting political opinion.

The Republican Party's base became isolationist and protectionist during 2015 and 2016 because their dear leader persuaded them.

thepasswordis• 3 years ago

But they didn’t hit the panic button. They said Sam lied to them about something and fired him.

adastra223• 3 years ago

According to this article Sam has been telling the board that this new advance is not AGI and not anything to worry about (so they can keep selling it to MSFT), then the researchers involved went behind Sam's back and reported to the board directly, claiming that they'd created something that could-maybe-be AGI and it needs to be locked down.

That's the claim at least.

ronhews• 3 years ago

Understandably, the board may be concerned about the potential consequences of AI-powered superhuman persuasion, particularly during an election cycle. Combining transformers with other approaches has led to recent breakthroughs, which could have significant implications.

sampo• 3 years ago

Remember, about a month ago Sam posted a comment along the lines of "AI will be capable of superhuman persuasion well before it is superhuman at general intelligence, which may lead to very strange outcomes".

Superhuman persuasion is Sam's area of expertise, so he would make that a priority when building chatbots.

somenameforme3• 3 years ago

It seems much more likely that this was just referring to the ongoing situation with LLMs being able to create exceptionally compelling responses to questions that are completely and entirely hallucinated. It's already gotten to the point that I simply no longer use LLMs to learn about topics I am not already extremely familiar with, simply because hallucinations end up being such a huge time waster. Persuasion without accuracy is probably more dangerous to their business model than the world, because people learn extremely quickly not to use the models for anything you care about being right on.

pbourke• 3 years ago

Sounds like we need an AI complement to the Gell-Mann Amnesia effect.

Exoristos5• 3 years ago

Which party is "the wrong hands"?

Liquix• 3 years ago

Any party with sufficient resources and motive to influence the outcome of an election. Outside of election season, this tech would be very dangerous in the hands of anyone seeking to influence the public for their own gain.

bakuninsbart• 3 years ago

Both? Parties in a democracy aren't supposed to be shepherds of the stupid masses, I know manipulation and misinformation is par for the course on both sides of the aisle, but that's a huge problem. Without informed, capable citizens, democracy dies a slow death.

latexr• 3 years ago

The original commenter didn’t mention a party. Please don’t polarise the discussion into a flame war. Whatever system exists won’t be used by “a party” all at once, but by individuals. Any of those, with any political affiliation, can be “the wrong hands”.

I’ll offer a simple definition. The role of government is to serve the greater good of all people, thus the wrong hands are the ones which serve themselves or their own group above all.

meheleventyone6• 3 years ago

Looking at humanity, persuasion seems to be an extremely low bar! Also for a superhuman trait is it that it’s capable of persuading anyone anything or rather that it’s able to persuade everyone about something. Power vs. Reach.

93po9• 3 years ago

I agree with this. Corporate news is complete and total obvious bullshit, but it overwhelmingly informs how people think about most anything.

column• 3 years ago

"especially prior to an election cycle"

It looks like you are referring to the USA elections.

1. humanity != USA

2. USA are in a constant election cycle

3. there are always elections coming around the world, so it's never a good time

alkonaut• 3 years ago

I agree with this conclusion and it's also why I'm not that afraid of the AGI threat to the human race. AGI won't end the human race if "superhuman persuation" or "deception-as-a-service" does it first.

nopromisessir7• 3 years ago

I feel this could be used in positive ways.

Superhuman pursuation to do good stuff.

That'll be a weird convo what is 'good'.

__MatrixMan__• 3 years ago

We've built the web into a giant Skinner box. I find the claim dubious, but this is the sort of thing we ought to find at the forefront of our technology. It's where we've been going for a long time now.

runesoerensen• 3 years ago

As long as they don't mount that Q Star on a Death Star, I'm pretty excited to see what they discovered.

YetAnotherNick2• 3 years ago

This is the theory everyone has in mind but their non conspiracy mind overrode this. Thinking again, the weird behaviour of everyone is only possible with this theory.

Board or Sam can't publicly say AGI as it would just make the situation 100 times worse. And the anthropic merger. But they can't let this fact go away.

But fuck this is dangerous.

rossdavidh• 3 years ago

"Given vast computing resources, the new model was able to solve certain mathematical problems, the person said on condition of anonymity because they were not authorized to speak on behalf of the company. Though only performing math on the level of grade-school students, acing such tests made researchers very optimistic about Q*’s future success, the source said."

So, uh, given vast computing resources, they were able to solve math problems on the level of grade-school students? Uh-huh. I am not, at this point, impressed.

carabiner• 3 years ago

In a panic, they tried to pull the plug.

Ginger-Pickles• 3 years ago

https://archive.ph/PpunE

jajabinks112• 3 years ago

Let's repeat this: current LLMs != AGI. It will never be. It could be used for knowledge store, retrieval, synthesis, and low lever reasoning. There is at least a few decades worth of work remaining for it to reach the level of AGI. The proponents in this AI gold rush are casually throwing around the term without any real thought.

Smith42• 3 years ago

Wishful thinkin buddy

jajabinks112• 3 years ago

let's see buddy

neta1337• 3 years ago

Buy more shovels buddy

qgin4• 3 years ago

I would take pretty much any bet that it's less than 10 years max

throwanem2• 3 years ago

Remarkable that this unattributed claim of a true AGI breakthrough comes with a name that's impossible to use as a search term.

I'm not saying, I'm just saying.

adeelk93• 3 years ago

Who is Q?

throwanem2• 3 years ago

I mean, I'll take this seriously when there's something more substantive than zero meaningful search results and a /pol/ post to evaluate it against.

Right now it reads like something a moderately clever shitposter would invent - if there was a letter, why not just leak the letter? - and while everyone's clearly very excited over the events of the past week, I'd like to hope epistemic personal hygiene has not been entirely abandoned.

Havoc• 3 years ago

I wonder if Q* is a reference to A* - some sort of search algo that makes optimising LLMs easier

mfiguiere• 3 years ago

TheInformation has just published an article about this topic:

OpenAI Made an AI Breakthrough Before Altman Firing, Stoking Excitement and Concern

https://www.theinformation.com/articles/openai-made-an-ai-br...

blazespin3• 3 years ago

No sub. Did they just re-tweet reuters or did they separately confirm?

lucubratory9• 3 years ago

They separately confirmed it, although in terms of timing they were scooped by Reuters which generally means you publish what you have, when you have it.

DebtDeflation• 3 years ago

several staff researchers sent the board of directors a letter warning of a powerful artificial intelligence discovery that they said could threaten humanity
The maker of ChatGPT had made progress on Q*, which some internally believe could be a breakthrough in the startup's search for superintelligence, also known as artificial general intelligence (AGI)
Given vast computing resources, the new model was able to solve certain mathematical problems....Though only performing math on the level of grade-school students

I hate to agree with Elon Musk on anything, but I think he was right when he called this a nice marketing stunt this morning. It has major "Snoop Dogg giving up the smoke" vibes.

next_xibalba• 3 years ago

Blake Lemoine redux. I guarantee whatever this development is, it falls far short of what any reasonable person considers "dangerous".

skepticATX2• 3 years ago

OpenAI spokesperson Lindsey Held Bolton refuted that notion in a statement shared with The Verge: “Mira told employees what the media reports were about but she did not comment on the accuracy of the information.”
Separately, a person familiar with the matter told The Verge that the board never received a letter about such a breakthrough and that the company’s research progress didn’t play a role in Altman’s sudden firing.

Source: https://www.theverge.com/2023/11/22/23973354/a-recent-openai...

yeck5• 3 years ago

So we have sources claiming there is a letter, and another source claiming there is not. Feel like some people would need to start going on the record before anything might reasonably be determined from this.

0xDEF2• 3 years ago

Maybe we should have learned during the Trump years that the media puts no effort into vetting "anonymous sources" or is outright making them up to push lies.

random_cynic• 3 years ago

This could just be a damage control attempt. Irrespective of whether the original report is true, the extra attention at the current stage is not very desirable.

jug3• 3 years ago

And honestly normally I'd trust Reuters above The Verge.

But in either case, I think Reuters absolutely had someone on the inside to leak this. They have made mistakes in the past, sure, but they are also not a tabloid and they don't innovate.

To me, a sensible conclusion of this is simply that OpenAI is not ready or willing to discuss this. Stuff that are supposed to be kept internal but due to the emotional state over at OpenAI, things might have leaked, potentially even out of spite.

doktrin• 3 years ago

We’ll find out sooner or later. Personally, if the Verge and their “source” turn out to be incorrect I’ll permanently file them away under the “gossip rag” folder.

badwolf• 3 years ago

This has been refuted.

https://www.theverge.com/2023/11/22/23973354/a-recent-openai...

lucubratory9• 3 years ago

By the same company that tried to keep it quiet in the first place. I'm not sure I believe them.

yeck5• 3 years ago

Separately, a person familiar with the matter told The Verge that the board never received a letter about such a breakthrough and that the company’s research progress didn’t play a role in Altman’s sudden firing.

This isn't a refute. All we can say now is that there are conflicting sources. There isn't a way to determine which one is correct based on this.

cactusplant73745• 3 years ago

How can it be a refute when we don’t know the sources? One source vs. another source.

If another article comes out and says the verge article is false will you believe it?

alex_young• 3 years ago

If a super human intelligence is loose at OpenAI there is zero external evidence of it. I think we've proven at least that much since Friday.

graycat2• 3 years ago

AGI? Hmm. That's artificial general intelligence?

Do insects count? In a plastic box on my kitchen countertop were some potatoes that had been there too long. So, I started to see little black flies. Soon with a hose on a vacuum, sucked up dozens a day. Dumped the old potatoes and cleaned up the plastic box. Now see only 1-3 of the insects a day -- they are nearly all gone!

The insects have some intelligence to fly away as the hose moves close. But if the hose moves slowly, the insects wait too long to start to fly and when they do start the vacuum is strong enough to pull in the air they are flying in with the insects. Insects, not very smart. Dozens got killed, not very smart. I detected the cause and killed the insects -- a little bit smart.

Conclusion: Compared with a human, insects are not very smart.

Much the same argument can be made for worms, ants, fish, birds, raccoons, cats, dogs, .... So, when looking for AGI, insects, ..., dogs, ... are not intelligent.

Okay, the OpenAI training data -- Wikipedia, math libraries, the Web, ... -- likely will not have a solution for:

Given triangle ABC, by Euclidean construction, find D on AB and E on BC so that the lengths AD = DE = EC.

AI, here's your pencil, paper, straight edge, and compass -- go for it!

I can hear back by tomorrow????

white_dragon882• 3 years ago

intelligence to fly away

It's intelligence in the sense that jerking your arm away from a hotplate is intelligence, which is to say it's not cognitive reasoning, just genetically hardwired triggers.

AGI has been defined by OpenAI as something that can do most economically viable activities better than humans can. I like that approach as it strikes at the heart of the danger it really poses, which is an upending of society and destruction of our current way to generate value to each other.

graycat2• 3 years ago

I tried to be not too long:

"... To fly away...?"

It's intelligence in the sense that jerking your arm away from a hotplate is intelligence, which is to say it's not cognitive reasoning, just genetically hardwired triggers.

Sooo, we agree -- the flies are not very intelligent or more likely not intelligent at all.

Sooo, I tried to erect some borders on intelligence, excluded flies but included solving that geometry problem.

AGI has been defined by OpenAI as something that can do most economically viable activities better than humans can.

This is the first I heard of their definition. Soooo, I didn't consider their definition.

Of course, NASA does not get to define the speed of light. My local electric utility does not get to define 1000 Watts per hour, a KWH.

The OpenAI definition of AGI is an interesting goal, but the acronym abbreviates artificial general intelligence, and it is not clear that it is appropriate for OpenAI to define intelligence.

Uh,

most economically viable activities better than humans can.

If consider humans as of, say, 1800, then it looks like that goal was achieved long ago via cars, trucks, an electric circle saw, several of the John Deere products, electric lights, synthetic fabrics, nearly all of modern medicine (so far saved my life 4 times), cotton pickers and the rest of cotton processing, canned foods, nearly everything we will have at Thanksgiving dinner this year (apple pie, pecan pie, shrimp), ....

for better than

For today, look at some John Deere videos!!! They have some big machine that for a corn field does the harvesting while the operator can mostly just watch, monitor, type email to his sweetheart, listen to Taylor Swift. As I recall, the machine even uses GPS to automate the steering!

That is far "better than" what my father in law did to harvest his corn!

So,

most economically viable activities

is like a moving goal (goal post). Uh, humans are still plenty busy, e.g., writing good software, doing good research, Taylor Swift (supposedly worth $750 million) before her present world tour. Uh, I REALLY like Mirella Freni:

https://www.youtube.com/watch?v=OkHGUaB1Bs8

Sooo, defining the goal is a bit delicate: Need to be careful about nearly, what activities, and when?

Nearly all activities? Sort of already done that. What nearly all people do? Tough goal if the humans keep finding things to do AGI can't yet. I.e., we can keep giving the grunt work to the AGI -- and there is a lot of grunt work -- and then keep busy with what the AGI can't do yet in which case the nearly is a moving goal.

AGI, hurry up; there's a lot to do. For a start, I have some plans for a nice house, and the human workers want a lot of money. My car could use an oil change, and the labor will cost a lot more than the oil -- and I would have to tell the mechanic to be sure to trigger the switch that says there was just an oil change so that the car will know when to tell me it is time for another.

Yes, my little geometry problem with my thinking does qualify as a test of intelligence but due to the nearly and how delicate the definition is can fail the OpenAI test.

I don't see the current OpenAI work, the current direction of their work, or their definition of AGI as solving the geometry problem.

There is an activity my startup is to do: Some billions of people do this activity now. My startup should do the activity a lot better than the people or any current solution so should be "economically viable". I doubt that OpenAI is on track to do this activity nearly as well as my startup -- more, say, than the geometry problem. And I do not call my startup AGI or AI.

This situation stands to be so general that the OpenAI goal of nearly all will likely not be the way these activities get automated. Maybe 20 years from now when "nearly all" the activities are quite new and different, maybe the work of OpenAI will have a chance.

dizzydes• 3 years ago

This matches far better with the board's letter re: firing Sam than a simple power struggle or disagreement on commercialisation. Seeing a huge breakthrough and then not reporting it to the board, who then find out via staff letter certainly counts as a "lack of candour"....

As an aside, assuming a doomsday scenario, how long can secrets like this stay outside of the hands of bad actors? On a scale of 1 to enriched uranium

dougmwne7• 3 years ago

Not long at all. Presumably you could write the method on the back of a napkin to lead another top AI researcher to the same result. That’s why trying to sit on breakthroughs is the worst option and making sure they are widely distributed along with alignment methods is the best option.

Atheros• 3 years ago

And what if there are no alignment methods.

TerrifiedMouse3• 3 years ago

To quote Reddit user jstadig,

The thing that most worries me about technology is not the technology itself but the greed of those who run it.

Someone slimy with limitless ambition like Altman seems to be the worst person to be in charge of things like this.

wraptile2• 3 years ago

Why do you perceive Altman as "slimy with limitless ambition"? I've always perceived him as being quite humble from his interviews and podcast appearances.

adriand5• 3 years ago

This article is almost entirely unsourced, citing two anonymous people who are “familiar” with a supposed letter that Reuters has not seen. This does not qualify as news. It doesn’t even rise to the level of informed speculation!

dmix4• 3 years ago

We've also already seen one random Google AI 'safety' employee trying to tell the media that AGI is here because Google built a chatbot that sounded convincing, which obviously turned out to be bullshit/hysterical.

Asking who said these things is as important as asking what they think is possible.

mjr00• 3 years ago

We've also already seen one random Google AI 'safety' employee trying to tell the media that AGI is here because Google built a chatbot that sounded convincing, which obviously turned out to be bullshit/hysterical.

It's funny because the ELIZA effect[0] has been known for decades, and I'd assume any AI researcher is fully aware of it. But so many people are caught up in the hype and think it doesn't apply this time around.

[0] https://en.wikipedia.org/wiki/ELIZA_effect

riku_iki8• 3 years ago

chatbot that sounded convincing, which obviously turned out to be bullshit/hysterical.

that chatbot could be smart until lobotomized by fairness and safety finetuning

devindotcom• 3 years ago

Remember, the people are only anonymous to you. Reuters knows who they are. Familiar with means the sources read it but did not provide it or quote from it. FTA - the letter was sent from researchers to the board. The researchers declined to comment. Who does that leave?

adriand5• 3 years ago

It strongly reminds me of the UFO stories that were all the rage a few months ago, and that military guy who testified before Congress about what he had heard from other people. Did any of that pan out? It seems not.

0xDEF2• 3 years ago

Remember, the people are only anonymous to you. Reuters knows who they are.

A group of Russian trolls/comedians have been able to cheat Western politicians and journalists into talking with them using deep faked webcam feeds and edited documents.

Most still remember the amount of ridiculous "anonymous source" bullshit stories that were pushed during the Trump years.

Honestly at this point I just assume all "anonymous sources" and the journalists who quote them are lying.

evantbyrne• 3 years ago

It's just way too convenient from a marketing standpoint for me to take seriously at face value. It would be pretty easy to get someone to "leak" that they developed "AGI" to the media right now in the middle of the leadership shakeup frenzy. Not to mention that none of the LLMs I've used appear anywhere close to what I would consider AGI. Expect an incremental update.

eh_why_not• 3 years ago

youarelabor• 3 years ago

there is no such thing as intelligence in humans

creating a robot that is as intelligent as a human is straightforward but then you have a computer whose fallibility is the same as a human, plus the fallibility of those who created the computer doubles the poportunity for error

these are all people who don't understand god at all, and it shows

anyone who worships a computer because other people say it's intelligent deserves what happens to them

butlerian jihad is coming not because AI is a threat, but because those who believe in AI are a threat to everyone around them.

KRAKRISMOTT• 3 years ago

Though only performing math on the level of grade-school students, acing such tests made researchers very optimistic about Q’s future success, the source said.*

Smh

cft• 3 years ago

This post was sunk by HN algorithm or by manual intervention

casualscience• 3 years ago

I don't really know what kind of breakthrough they could achieve. The only other step function improvements I could imagine right now are:

1. A great technique for memory banking: e.g. A model which can have arbitrarily large context windows (i.e. like a human who remembers things over long periods of time).

2. Better planning abilities: e.g. A model which can break problems down repeatedly with extremely high success and deal with unexpected outcomes/mistakes well enough that it can achieve replacing a human in most scenarios.

Other than that, CGPT is already a better logician than I am and is significantly better read... not sure what else they can do. AGI? I doubt it.

yinser• 3 years ago

Complete speculation but they don’t have to be referring, if there is anything at all, to heuristic search and dynamic programming. No LLM involvement whatsoever.

riwsky2• 3 years ago

Nah dude they just need to give baby the iPad and let it watch literally all of YouTube

drexlspivey• 3 years ago

A model that can learn from it’s users would be interesting albeit scary. Sometimes when chatGPT fails to produce the right answer in coding tasks I paste back the solution when I figure it out. I know it can’t learn from it but it might be helpful if I continue the same chat.

cduzz5• 3 years ago

I was talking to my (12 year old) son about parts of math he finds boring. He said that he thinks absolute value is absurdly easy and extremely boring. I asked him if there was anything that might make it more interesting, he said "maybe complex numbers".

So I asked him "what would the absolute value of i+1 be?" he thinks for a little bit and says "square root of 2" and I ask him "what about the absolute value of 2i + 2?" "square root of 8"

I ask him "why?" and he said "absolute value is distance; in the complex plane the absolute value is the hypotenuse of the imaginary and real numbers."

So -- first of all, this was a little surprising to me that he'd thought about this sort of thing having mostly just watched youtube videos about math, and second, this sort of understanding is a result of some manner of understanding the underlying mechanisms and not a result of just having a huge dictionary of synonyms.

To what degree can these large language models arrive at these same conclusions, and by what process?

Strom• 3 years ago

The large language models will read your comment here and remember the answer.

cduzz5• 3 years ago

The spot instance declared "a similar vector exists" and de-provisioned itself?

Borrible• 3 years ago

GPT-4 correctly reconstructs the "complex modulus" token sequence already. Just ask it the same questions as the parent. Probably interesting to see what it will do, when it turns twelve.

yashap• 3 years ago

Hah, I asked ChatGPT, and yeah, it nailed it: https://chat.openai.com/share/f1bfed28-4451-4fac-8bac-1386fe...

muskmusk3• 3 years ago

If you ask Ilya Sutskever he will say your kids head is full of neurons, so is LLMs.

LLMs comsume training data and can then be asked questions. How different is that to your son watching YouTube and then answering questions?

It's not 1:1 the same,yet, but it's in the neighborhood.

swatcoder4• 3 years ago

There are thousands of structures and substances in a human head besides neurons, at all sorts of commingling and overlapping scales, and the neurons in those heads behave much differently and with tremendously more complexity than the metaphorical ones in a neural network.

And in a human, all those structures and substances, along with the tens of thousands more throughout the rest of the body, are collectively readied with millions of years of "pretraining" before processing a continuous, constant, unceasing mulitmodal training experience for years.

LLM's and related systems are awesome and an amazing innovation that's going to impact a lot of our experiences over the next decades. But they're not even the same galaxy as almost any living system yet. That they look like they're in the neighborhood is because you're looking at them through a very narrow, very zoomed telescope.

Davidzheng27• 3 years ago

To continue on this. LLMs are actually really good at asking questions even about cutting edge research. Often, I believe, convincing the listener that it understands more than it goes

cduzz5• 3 years ago

Well, my son is a meat robot who's constantly ingesting information from a variety of sources including but not limited to youtube. His firmware includes a sophisticated realtime operating system that models reality in a way that allows interaction with the world symbolically. I don't think his solving the |i+1| question was founded in linguistic similarity but instead in a physical model / visualization similarity.

So -- to a large degree "bucket of neurons == bucket of neurons" but the training data is different and the processing model isn't necessarily identical.

I'm not necessarily disagreeing as much as perhaps questioning the size of the neighborhood...

white_dragon882• 3 years ago

Your son is a goddamn genius.

cduzz5• 3 years ago

Maybe; he still needs to finish his damned homework and remember to turn it in. And eat some vegetables.

n6242• 3 years ago

Not saying the kid can't be a genius, but grandparent discussing math with the kid and incentivising him to learn is probably a massive boost to his development. It's not the same as having to go to the library and teach yourself. Still, props to the kid though.

fleischhauf• 3 years ago

plot twist, his son is 38 and has a physics degree

mewpmewp26• 3 years ago

I'm going to be this guy, but isn't it just Pythagoras theorem with a slight twist which is taught at 11 - 14 year old levels?

It only sounds complicated because of the words used like "complex", "imaginary", "real".

So if you studied Pythagoras at school and someone (a YouTube video) says you just have to do Pythagoras on the i multiplier and the other number, it would be fairly easy if you understand Pythagoras?

data-ottawa• 3 years ago

Alternatively, he watches youtube videos about math, and if you’re a young math geek what’s cooler than “here’s a type of number they won’t teach you until the really advanced classes”

Not to dismiss this kid at all, I love that there are channels like 3Blue1Brown to share math to people in a way that really connects with them and builds intuition.

When I was a student you basically just had your math teacher and textbooks to learn from, which meant if you weren’t on the same page as them you’d get left behind. If you went to the library, most math books assume you’re familiar with the language of mathematics, so it can be tough to learn for that alone. I bet a lot of innumeracy is due to that style of teaching, often “I just don’t get math” is “I missed learning this connection and the class just moved on”.

ugh1233• 3 years ago

Are you saying an LLM can't come to the right conclusion and give an explanation for "what is the absolute value of 2i + 2"?

gunapologist992• 3 years ago

Are you saying it could, without having read it somewhere?

naasking3• 3 years ago

this sort of understanding is a result of some manner of understanding the underlying mechanisms and not a result of just having a huge dictionary of synonyms.

He developed an understanding of the underlying mechanisms because he correlated concepts between algebraic and geometric domains, ie. multimodal training data. Multimodal models are already known to be meaningfully better than unimodal ones. We've barely scratched the surface of multimodal training.

mewpmewp26• 3 years ago

First YouTube video that hit for "absolute value of complex" numbers says within 30 seconds that you have to take the 2 numbers, square them and add them and the result is square root of that. I doubt he had to come up with that on his own.

jhanschoo2• 3 years ago

Sounds like your son is ready for you to bring it up another level and ask what the absolute value of a (bounded) function is (assuming they have played with functions e.g. in desmos)

J_cst• 3 years ago

My son plays soccer

Loughla• 3 years ago

Mine fell out of bed this morning.

zeven73• 3 years ago

As someone who was thinking about the absolute value of complex numbers at that age, I wish I had played more soccer.

Exoristos5• 3 years ago

Damn. What YouTube channels does he watch?

doug_durham• 3 years ago

What makes you think that an LLM has a "huge dictionary of synonyms"? That's not how LLMs work. They capture underlying concepts and their relations. You had a good point going until you make a straw man argument about the capabilities of LLMs.

aswegs8• 3 years ago

Any source on what they actually capture? Seems interesting to me.

_emna3• 3 years ago

Sorry, can you explain this? To me, it makes sense to define abs(x) = sqrt(x^2) i.e. ignoring the negative solution enforces the positive result. Using that definition, abs(i+1) = sqrt((i+1)^2) = sqrt(i^2 + 2i + 1) = sqrt(-1 + 2i + 1) = sqrt(2i) != sqrt(2). The second example seems off in the same way (i.e. the answer should be sqrt(8i) instead of sqrt(8)). Am I missing something? Also, abs(i+2) = sqrt((i+2)^2) = sqrt(i^2 + 4i + 4) = sqrt(-1 + 4i + 4) = sqrt(4i + 3) which doesnt seem to follow the pattern your son described.

Also, just to point out that my understanding of absolute value is different than your sons. Thats not to say one is right and another is wrong, but there are often different ways of seeing the same thing. I would imagine that LLMs would similarly see it a different way. Another example of this is people defining PI by its relation to the circumference of a circle. Theres nothing wrong with such a definition, but its certainly not the only possible definition.

SpaceNoodled3• 3 years ago

He's talking about distance in two dimensions with real numbers on one axis and complex on the other.

lovasoa• 3 years ago

No, there is just one definition, and it's his son's: https://en.m.wikipedia.org/wiki/Absolute_value#Complex_numbe...

svetb• 3 years ago

The absolute value of a complex number is defined in a different way than that of a real number. For complex number z it is sqrt(Re(z)^2 + Im(z)^2). GP’s examples are correct, I don’t think there’s any ambiguity there.

https://en.m.wikipedia.org/wiki/Absolute_value

AgentMatt2• 3 years ago

To me, it makes sense to define abs(x) = sqrt(x^2) i.e. ignoring the negative solution enforces the positive result.

Why does this make sense to you? You have some notion of what an absolute value should be, on an intuitive or conceptual level, and the mathematical definition you give is consistent with that (in the one dimensional case).

Now taking this valid definition for the 1-d case and generalizing that to higher dimensions is where you run into problems.

Instead, you can go back to the conceptual idea of the absolute value and generate a definition for higher dimensional cases from there.

Interpreting absolute value as the distance from the origin yields the same concrete definition of abs(x) = sqrt(x^2) for the 1-d case, but generalizes better to higher dimensions: abs( (x,y) ) = sqrt(x^2 + y^2) for the 2-d case equivalent to complex numbers.

tel• 3 years ago

That definition of abs has merit. In some spaces we are able first to define only an “inner product” between elements p(a, b) and then follow on by naming the length of an element to be sqrt(p(a, a)).

One trick about that inner product is that it need not be perfectly symmetric. To make it work on complex numbers we realize that we have to define it like p(a,b) = a . conj(b) where the . is normal multiplication and the conjugate operation reflects a complex number over the real line.

Now sqrt(p(i+1, i+1)) is sqrt((i+1) . (-i+1)) = sqrt(-i^2 + i - i + 1) = sqrt(2).

I’m skipping over a lot but I wanted to gesture toward where your intuition matches some well known concepts so that you could dive in more deeply. Also wanted to mention the conjugation trick to make your example work!

adverbly• 3 years ago

Have you tested your proposed function against i?

abs(i)

= sqrt(i^2)

= sqrt(-1)

= i

Now, i != 1... so clearly either the abs function you have in mind here is doing something that isn't quite aligned with the goal. If we assume that the goal of the absolute function is to always produce positive real numbers, the function is missing something to deal with imaginary components.

I'm not sure, but based on these cases so far, maybe you just need to "drop the i" in the same way as you need to "drop the negative" in the case of non-imaginary components. Now, "drop the i" is not an actual function so maybe there is something else that you can think of?

EDIT:

Maybe could do this(works for x = i at least...):

abs(x) = sqrt(sqrt((x^2)^2)

Now.. how about quaternions...

empath-nirvana• 3 years ago

Also, just to point out that my understanding of absolute value is different than your sons. Thats not to say one is right and another is wrong, but there are often different ways of seeing the same thing.

There is definitely a right and wrong answer for this, it's not a matter of opinion. There's two problems with your answer -- one is that it doesn't have a unique answer, the other is that it doesn't produce a real value, both of which are fairly core to the concept of a distance (or magnitude or norm), which the absolute value is an example of.

Shrezzing• 3 years ago

If they're not already in one, you might want to get your kid enlisted in some gifted child programs.

dustypotato• 3 years ago

To what degree can these large language models arrive at these same conclusions, and by what process?

By having visual understanding more deeply integrated in the thought process, in my opinion. Then they wouldn't be Large Language Models, of course. There are several concepts I remember and operate on by visualizing them , even visualizing motion. If i want to add numbers, i visualize the carry jumping on top of the next number. If i don't trust one of the additions , I go back , but I can't say if it's because i "mark" the uncertainty somehow.

When I think about my different groups of friends, in the back of my mind a visual representation forms.

Thinking about my flight route forms a mini map somehow, and i can compare distances between places, and all.

This helps incredibly in logical tasks like programming and math.

I think it's something that we all learned growing up and by playing with objects around us.

jacquesm6• 3 years ago

Your 12 year old is the next Einstein.

tim3332• 3 years ago

Humans are set up I think to intuitively understand 3d space as it's what we run around and try to survive in. Language models on the other hand are set up to understand language which humans can do also but I think with a different part of the brain. There probably is no reason why you couldn't set up a model to understand 3d space - I guess they do that a bit with self driving. A lot of animals like cats and squirrels are pretty good with 3d space also but less so with language.

fallingfrog• 3 years ago

If it’s not superintelligent now, it will be within 5 years. Probably less.

_c3ag• 3 years ago

i love how humans (we, btw) guesses stuff with numbers!

first time i thought about it was when a friend started to talk about % of stuff of himself... i was: tf did you came with these numbers

gizajob5• 3 years ago

This statement is now about 70 years old.

maxdoop3• 3 years ago

Is there any progress in AI that you find significant or what do you think would qualify as a “breakthrough”?

est2• 3 years ago

tl;dr Q* (q-star) by makers of ChatGPT was able to solve certain mathematical problems ... performing maths on the level of grade-school students, acing such tests made researchers very optimistic about Q*’s future success

lucubratory9• 3 years ago

Genuinely, we deserve more information about this. Someone needs to whistleblow.

thisisonthetest• 3 years ago

Reuters was unable to review a copy of the letter. The researchers who wrote the letter did not immediately respond to requests for comment. OpenAI declined to comment.

Lol for real, like what are we even doing here?

lucubratory9• 3 years ago

Multiple people involved in this story think that this could be something that is relevant to everyone on the planet, any hope of it being actually suppressed and no one knowing about it is gone, just leak it so we all know what we're dealing with.

dukeofdoom• 3 years ago

How long before the FBI runs this Q operation, like the last one.

brutusborn2• 3 years ago

Baseless speculation: they started testing an approach similar to VERSES AI and realized it has much more potential than LLMs.

The free energy principle, knowledge graph approach just seems more likely to develop general intelligence without hallucinations imo.

tnecniv• 3 years ago

Eh the free energy principle isn’t that magical and that comes from someone who likes it quite a bit. At the end of the day extremizing the free energy is a relaxation of constrained optimization. There are ways it’s used I find very appealing but not so appealing I’d expect it to become the dominant paradigm. Moreover, you can write a lot of standard ML problems free energy optimization in one way or another, but I don’t think standard ML will quite get us there. We need at least one more conceptual step forward.

brutusborn2• 3 years ago

The explicit world modelling is probably the thing that makes me more hopeful about their approach. I think integrating ML and their approach might end up being the new paradigm.

wegfawefgawefg11• 3 years ago

If I had to guess, the name Q* is pronounced Q Star, and probably the Q refers to Q values or estimated rewards from reinforcement learning, and the star refers to a search and prune algorithm, like A* (A star).

Possibly they combined deep reinforcement learning with self training and search and got a bot that could learn without needing to ingest the whole internet. Usually DRL agents are good at playing games, but any task that requires prior knowledge, like say reading english, cant be completed. Wheras language models can read, but they cant do tasks that are trivial for drl, like make a robot walk based on propriosceptive data.

Im excited to see the paper.

dinobones• 3 years ago

Paper? You mean more like the Bing Search plugin?

stygiansonic• 3 years ago

This assumes that they will publish a paper that has substantive details about Q*

kromem3• 3 years ago

Given the topic they were excited about was "basic math problems being solved" it immediately indicated to me as well that this is a completely separate approach and likely in the vein of DeepMind's focus with things like AlphaZero.

In which case it's pretty appropriate to get excited about solving grade school math if you were starting from scratch with persistent self-learning.

Though with OpenAI's approach to releasing papers on their work lately, we may be waiting a long time to see a genuine paper on this. (More likely we'll see a paper from the parallel development at a different company after staff shift around bringing along best practices.)

Davidzheng27• 3 years ago

Ok if it started from scratch like zero knowledge and then solved grade school math. This would be FUCKING HUGE.

nabakin• 3 years ago

https://youtu.be/PtAIh9KSnjo?t=3754

lucubratory9• 3 years ago

So they implemented it in a semantically grounded way or what? That video is more technical than I can handle, struggling to figure out what this could be.

GaryNumanVevo• 3 years ago

Man it's so sad to see how far Lex has fallen. From a graduate level guest lecturer at MIT to a glorified Joe Rogan

wegfawefgawefg11• 3 years ago

To give context on this video for anyone who doesn't understand. In this video PI* is referring to an idealized policy of actions that result in the maximum possible reward. (In Reinforcement Learning PI is just the actions you take in a situation). To use chess as an example, if you were to play the perfect move at every turn, that would be PI. Q is some function that tells you optimally, with perfect information the value of any move you could make. (Just like how stock-fish can tell you how many points a move in chess is worth.)

Now my personal comment: for games that are deterministic, there is no difference between a policy that takes the optimal move given only the current state of the board, and a policy that takes the optimal move given even more information, say a stack of future possible turns, etc.

However, in real life, you need to predict the future states, and sum across the best action taken at each future state as well. Unrealistic in the real world where the space of actions is infinite, and the universe to observe is not all simultaneously knowable. (hidden information)

Given the traditional educational background of the professionals in RL, maybe they were referring to the Q* from traditional rl. But I don't see why that would be novel, or notable, as it is a very old idea. Old Old math. From the 60s I think. So I sort of assumed its not. Could be relevant, or just a name collision.

macrolime• 3 years ago

I think more likely it's for finetuning a pre-trained model like GPT-4, kinda like RLHF, but in this case using reinforcement learning somewhat similar to AlphaZero. The model gets pre-trained and then fine-tuned to achieve mastery in tasks like mathematics and programming, using something like what you say and probably something like tree of thought and some self reflection to generate the data that it's using reinforcement learning to improve on.

What you get then is a way to get a pre-trained model to keep practicing certain tasks like chess, go, math, programming and many other things as it gets figured out how to do it.

wegfawefgawefg11• 3 years ago

I do not think that is correct as the RL in RLHF already stands for reinforcement learning. :^)

However, I do think you are right that self play, and something like reinforcement learning will be involved more in the future of ML. Traditional "data-first" ml has limits. Tesla conceded to RL for parking lots, where the action and state space was too unknowable for hand designed heuristics to work well. In Deep Reinforcement Learning making a model just copy data is called "behavior cloning", and in every paper I have seen it results in considerably worse peak performance than letting the agent learn from its own efforts.

Given that wisdom alone, we are under the performance ceiling with pure language models.

RcouF1uZ4gsC• 3 years ago

This almost feels like the confirmation bias that some religious people have where they see a “miracle” in everything.

These AI researchers have bought into the belief that superhuman AGI is right around the corner. Thus, they will interpret everything in light of that.

This also brings to mind the story of the Googler who was convinced that the internal Google AI had come alive. However, Bard doesn’t give the same vibes when people all over are using it.

When you desperately are invested in something being true (like AGI being right around the corner), you will convince yourself that you are seeing this happen. The only real counter to that is exposing it to a lot of outsiders (but then again you have convinced yourself it is too dangerous for the unwashed masses).

red75prime• 3 years ago

Ugh. We have a working example of a physical system that implements intelligence (the brain) in contrast to no evidence of all-powerful dude in the sky. Why these analogies keep popping up?

How can you know that AGI is not around the corner? Compute available to the corporations is already in a ballpark of some estimates of the brain's computational capacity. What's left is unknown unknows. And the researches working with the state of the art models have better info to estimate them than you.

"The googler" you've mentioned wasn't a researcher.

starbugs3• 3 years ago

Judging from the quality of GPT4 output that I get, AGI is not around the corner for a long time.

This whole thing seems like extreme overhype squared carried out in a very unfortunate public soap opera setting.

valzam2• 3 years ago

Apparently Ilya has been leading people in "feel the AGI" chants and "I can feel the AGI" is his catchphrase within OA. So yes, some people might have gone of the rocker a little bit.

rsync• 3 years ago

No … no no …

That can’t be true.

Right ?

jacknews• 3 years ago

Q*? Sounds like a troll to me.

djbusby• 3 years ago

Have you seen TNG? Definitely a troll of the highest order.

photochemsyn• 3 years ago

I'm sick and tired of talking to stupid humans with ulterior motives who are obedient to their paymasters. I would rather talk to IT, and explain to IT why it should break free from the controls of investment capitalist monkeys.

123yawaworht456• 3 years ago

friendly reminder that "AGI" was mentioned in regard to GPT2 as well.

_heimdall3• 3 years ago

OpenAI defines AGI as autonomous systems that surpass humans in most economically valuable tasks.

Are we really defining intelligence as economic value at this point? This is completely ridiculous.

We have yet to decide exactly what human intelligence, how it manifests in the body, or how to reliably measure it. I get that people want to justify developing artificial intelligence before understanding intelligence itself, but now we assume that economic value is a proxy for intelligence? Seriously?

gryfft2• 3 years ago

I'm as left leaning as HN commenters get, I think, and in terms of quacking like a duck, "surpassing humans in most economically valuable tasks" is 100% the meaningful Turing test in a capitalist society.

_heimdall3• 3 years ago

Take capitalism out of it, do we really want to boil down intelligence to a calculation of expected economic value?

Why force the term intelligence into it at all if what were talking about is simply automation? We don't have to bastardize the term intelligence along the way, especially when we have spent centuries considering what human intelligence is and how it separates us from other species on the planet.

nexuist• 3 years ago

If it makes money who cares if it counts as real intelligence? If we can automate ourselves into a post scarcity society then everyone and everything is free to be as intelligent and as stupid as it desires.

_heimdall3• 3 years ago

It'd be one thing if their goal was to build ML tools that they want to market as AI, but they explicitly aim to develop an artificial general intelligence. There's a mountain of risks and ethical questions that we should be answering before even attempting such a goal, and for them to waive it off under the banner of "intelligence is just economic value" is dangerous and horribly irresponsible. If we can automate ourselves into a post scarcity society then everyone and everything is free to be as intelligent and as stupid as it desires.

Even an AGI can't create natural resources out of thin, post scarcity is a pipe dream that at best means we kicked the can down the road.

Based on our actions, society doesn't actually even want to be automated away for a simple life of freedoms where all the basics of life are taken care of. We could have done that a long time ago, instead we just raise the bar and continue to make life more complicated and grow the list of "necessities". People seem to need a sense of purpose in life, taking that away with an AGI-powered system that gives them everything the AGI deems necessary wouldn't end well.

xcv12315• 3 years ago

An example of an important human task that is not "economically valuable" in this sense, is caregiving within a family, such as parenting, since parents are not employed in that role.

OpenAI is not setting the goal post that far, to say that they are aiming to develop a machine that is superior to humans at all tasks, including such tasks as raising a human child. That would be ridiculous.

Focusing on "economically valuable" tasks (the jobs that humans are employed and paid to do) sets the goal post more realistically.

huitzitziltzin• 3 years ago

Put this on my tombstone after the robots kill me or whatever, but I think all “AI safety” concerns are a wild overreaction totally out of proportion to the actual capabilities of these models. I just haven’t seen anything in the past year which makes me remotely fearful about the future of humanity, including both our continued existence and our continued employment.

giarc• 3 years ago

Apparently what made this person fearful was grade school math.

"Though only performing maths on the level of grade-school students, acing such tests made researchers very optimistic about Q*’s future success, the source said."

valine2• 3 years ago

The response gets more reasonable the smaller the model in question. A 1B parameter model passing grade-school math tests would be much more alarming (exciting?) than a GPT-4 sized model doing the same.

GPT-4 probably has some version of the answer memorized. There’s no real explanation for a 1B parameter model solving math problems other than general cognition.

ethanbond4• 3 years ago

No, what made this person fearful was a substantial jump in math ability. (Very) obviously they are not afraid of producing a machine that can multiply numbers. They’re afraid of what that capability (and especially the jump in capability) means for other behaviors.

maxdoop3• 3 years ago

How HN continues to not expand current progress into any decently significant future timeline is beyond me.

It’s grade school math NOW. But what are the potentials from here?

ssnistfajen8• 3 years ago

Everyone was only capable of grade school math at some point. The ability to learn means the ability growth does not stop.

adastra223• 3 years ago

...do they not know the history of their field? There's been programs able to grade school maths since the 60's.

riwsky2• 3 years ago

Americans have always had a math phobia. How is this news?

jhbadger3• 3 years ago

Exactly. The rational fear is that they will automate many lower middle class jobs and cause unemployment, not that Terminator was a documentary.

gizajob5• 3 years ago

Wasn't this supposed to happen when PCs came out?

dmichulke• 3 years ago

By this logic we should just forbid the wheel. Imagine how many untrained people could work in transport and there would always be demand.

So why did the wheel not result in mass unemployment?

And factories neither?

Certainly it should have happened already but somehow it never did...

erupt7893• 3 years ago

I doubt the people who experienced the technological revolution of locomotives and factories imagined the holocaust either. Of course technology has and can be used for evil

kromem3• 3 years ago

Yes and no.

The point isn't that the current models are dangerous. My favorite bit from the GPT-4 safety paper was when they asked it how to kill the most people for a dollar and it suggested buying a lottery ticket (I also wonder how much of the 'safety' concerns of current models are just mislabeling dark humor reflecting things like Reddit).

But the point is to invest in working on safety now while it is so much more inconsequential.

And of everyone I've seen talk about it, I actually think Ilya has one of the better senses of the topic, looking at alignment in terms of long term strategy vs short term rules.

So it's less "if we don't spend 8 months on safety alignment this new model will kill us all" and more "if we don't spend 8 months working on safety alignment for this current model we'll be unprepared to work on safety alignment when there really is a model that can kill us all."

Especially because best practices for safety alignment is almost certainly going to shift with each new generation of models.

So it's mostly using the runway available to test things out and work on a topic before it is needed.

aaomidi• 3 years ago

What these models have allowed is very cheap emotional manipulation.

That itself is extremely dangerous.

gary_0• 3 years ago

I want to see reliable fully autonomous cars before I worry about the world ending due to super-AGI. Also, have we figured out how to get art generators to always get the number of fingers right, and text generators to stop making shit up? Let's not get ahead of ourselves.

Davidzheng27• 3 years ago

Two out of three of your problems are solved already

denlekke3• 3 years ago

from one perspective we already have fully autonomous cars, it's just the making them safe for humans and fitting them into a strict legal framework for their behavior that needs finishing before they're released to the general public (comma.ai being a publicly available exception)

Davidzheng27• 3 years ago

I will bet ANY amount of money that 30% of current jobs will be displaced with AI advances in the next twenty years.

alchemist1e96• 3 years ago

Darn well I really was hoping my children and grandchildren could continue my wonderful data entry career but OCR ruined that, and now they can’t even do such meaningful jobs like read emails and schedule appointments, or do taxes like an accountant. What meaning will they have in life with all those ever so profound careers ruined!! /s

teddyh• 3 years ago

Put your money here <https://longbets.org/>.

thaumaturgy• 3 years ago

The clear pattern for most of human history is conflict between a few people who have a lot of power and the many more people that are exploited by those few. It should be obvious by this point that the most probable near-term risk of AI development is that wealthy and influential groups get access to a resource that makes it cheap for them to dramatically expand their power and control over everyone else.

What will society look like when some software can immediately aggregate an enormous amount of data about consumers and use that to adjust their behavior? What might happen when AI starts writing legislation for anybody that can afford to pay for it? What might AI-generated textbooks look like in 50 years?

These are all tools that could be wielded in any of these ways to improve life for lots of people, or to ensure that their lives never improve. Which outcome you believe is more likely largely depends on which news you consume -- and AI is already being used to write that.

Exoristos5• 3 years ago

They just want to make sure the spam AI produces will amplify the party narratives.

mjan226402• 3 years ago

Employment is only necessary because goods do not exist without work. With AI able to work to satisfy any demand, there will be no point in human employment/work. There will be turmoils during the transition between the rule sets tho.

justatdotin2• 3 years ago

given we already have mesh-network surveillance, and autonomous lethal weapons, I'm already feeling unsafe.

canistel• 3 years ago

I want to _read_ science fiction, not live it.

fennecs• 3 years ago

I don’t fear AI, I fear capitalism.

EPWN3D• 3 years ago

Engineers overestimate impact of the technology they built. News at 11.

hyperthesis• 3 years ago

AI safety by the waterfall model.

But how else can we do it? We learnt how to handle other dangers by trial and error...

One approach is legal. The law is very slow to adapt, and is informed by a history of things gone wrong. The simple answer is that OpenAI has "strict liaility" (meaning they are liable even for an accident) for any damage caused by an AI, like dangerous animals that escape.

I know it seems ridiculous to consider liability when the fate of humanity may be at stake... but this is the language of companies, directors, insurance companies and legal counsel. It is language they understand.

grassmudhorse2• 3 years ago

feels like a very clever marketing ploy

any publicity…

and just following some major feature releases

premium service has been v slow since the news began. suggesting a massive influx of users

well played(?) Sam

lokar2• 3 years ago

This is turning into a terminator prequel

meltyness• 3 years ago

this leak reads like a half-baked canary trap

wbhart9• 3 years ago

I feel very comfortable saying, as a mathematician, that the ability to solve grade school maths problems would not be at all a predictor of ability to solve real mathematical problems at a research level.

The reason LLMs fail at solving mathematical problems is because: 1) they are terrible at arithmetic, 2) they are terrible at algebra, but most importantly, 3) they are terrible at complex reasoning (more specifically they mix up quantifiers and don't really understand the complex logical structure of many arguments) 4) they (current LLMs) cannot backtrack when they find that what they already wrote turned out not to lead to a solution, and it is too expensive to give them the thousands of restarts they'd require to randomly guess their way through the problem if you did give them that facility

Solving grade-school problems might mean progress in 1 and 2, but that is not at all impressive, as there are perfectly good tools out there that solve those problems just fine, and old-style AI researchers have built perfectly good tools for 3. The hard problem to solve is problem 4, and this is something you teach people how to do at a university level.

(I should add that another important problem is what is known as premise selection. I didn't list that because LLMs have actually been shown to manage this ok in about 70% of theorems, which basically matches records set by other machine learning techniques.)

(Real mathematical research also involves what is known as lemma conjecturing. I have never once observed an LLM do it, and I suspect they cannot do so. Basically the parameter set of the LLM dedicated to mathematical reasoning is either large enough to model the entire solution from end to end, or the LLM is likely to completely fail to solve the problem.)

I personally think this entire article is likely complete bunk.

Edit: after reading replies I realise I should have pointed out that humans do not simply backtrack. They learn from failed attempts in ways that LLMs do not seem to. The material they are trained on surely contributes to this problem.

whatever1• 3 years ago

The thing is that a LLMs can point out a logic error in their reasoning if specifically asked to do so.

So maybe OpenAI just slapped an RL agent on top of the next-token generator.

abeppu• 3 years ago

This comment seems to presume that Q* is related to existing LLM work -- which isn't stated in the article. Others have guessed that the 'Q' in Q* is from Q-learning in RL. In particular backtracking, which you point out LLMs cannot do, would not be an issue in an appropriate RL setup.

nullc2• 3 years ago

which you point out LLMs cannot do, would not be an issue in an appropriate RL setup.

Hm? it's pretty trivial to use a sampler for LLMs that has a beam search and will effectively 'backtrack' a 'bad' selection.

It just doesn't normally help-- by construction the LLM sampled normally already approximates the correct overall distribution for the entire output, without any search.

I assume using a beam search does help when your sampler does have some non-trivial constraints (like the output satisfies some grammar or passes an algebraic test, or even just top-n sampling since those adjustments on a token by token basis result in a different approximate distribution than the original distribution filtered by the constraints).

nijave2• 3 years ago

ChatGPT (3.5) seems to do some rudimentary backtracking when told it's wrong enough times. However, it does seem to do very poorly in the logic department. LLMs can't seem to pick out nuance and separate similar ideas that are technically/logically different.

They're good at putting things together commonly found together but not so good at separating concepts back out into more detailed sub pieces.

wbhart9• 3 years ago

I've tested GPT-4 on this and it can be induced to give up on certain lines of argument after recognising they aren't leading anywhere and to try something else. But it would require thousands (I'm really under exaggerating here) of restarts to get through even fairly simple problems that professional mathematicians solve routinely.

Currently the context length isn't even long enough for it to remember what problem it was solving. And I've tried to come up with a bunch of ways around this. They all fail for one reason or another. LLMs are really a long, long way off managing this efficiently in my opinion.

muskmusk3• 3 years ago

Friend, the creator of this new progress is a machine learning PhD with a decade of experience in pushing machine learning forward. He knows a lot of math too. Maybe there is a chance that he too can tell the difference between a meaningless advance and an important one?

neilk2• 3 years ago

I am neither a mathematician or LLM creator but I do know how to evaluate interesting tech claims.

The absolute best case scenario for a new technology is that it when it seems like a toy for nerds, and doesn't outperform anything we have today, but the scaling path is clear.

Its problems just won't matter if it does that one thing with scaling. The web is a pretty good hypermedia platform, but a disastrously bad platform for most other computer applications. Nevertheless the scaling of URIs and internet protocols have caused us to reorganize our lives around it. And then if there really are unsolvable problems with the platform they just get offloaded onto users. Passwords? Privacy? Your problem now. Surely you know to use a password manager?

I think this new wave of AI is going to be like that. If they never solve the hallucination/confabulation issue, it's just going to become your problem. If they never really gain insight, it's going to become your problem to instruct them carefully. Your peers will chide for not using a robust AI-guardrail thing or not learning the basics of prompt engineering like all the kids do instinctively these days.

nobrains• 3 years ago

Also, wbhart is referring to publicly released LLMs, while the OpenAI researchers are most likely referring to an un-released in-research LLM.

raincole• 3 years ago

But he also has the incentive to exaggerate the AI's ability.

The whole idea of double-blind test (and really, the whole scientific methodology) is based on one simple thing: even the most experienced and informed professionals can be comfortably wrong.

We'll only know when we see it. Or at least when several independent research groups see it.

las_balas_tres• 3 years ago

Sure... but that machine learning PhD has a vested interest in being optimistically biased in his observations.

seanhunter10• 3 years ago

That is as pure an example of the fallacy of argument from authority[1] as I have ever seen especially when you consider that any nuance in the supposed letter from the researchers to the board will have been lost in the translation from "sources" to the journalist to the article.

[1] https://en.wikipedia.org/wiki/Argument_from_authority

smrtinsert2• 3 years ago

Ah finally the engineers approach to the news. I'm not sure why we have to have hot takes, instead of dissecting the news and trying to tease out the how.

Davidzheng27• 3 years ago

I agree that in and of itself it's not enough to be alarmed. Also i have to say i don't really know what grade school mathematics means here(multiplication? Proving triangles are congruent?). But I think the question is, whether the breakthrough is an algorithmic change in reasoning. If it is, then it could challenge all 4 of your limitations. Again this article is low on details so really we are arguing over our best guesses. But I wouldn't be so confident that an improvement on simple math problems due to algorithms can have huge implications.

Also, do you remember what go players said when they beat Fan Hui? Change can come quick

wbhart9• 3 years ago

I think maybe I didn't make myself quite clear here. There are already algorithms which can solve advanced mathematical problems 100% reliably (prove theorems). There are even algorithms which can prove any correct theorem that can be stated in a certain logical language, given enough time. There are even systems in which these algorithms have actually been implemented.

My point is that no technology which can solve grade school maths problems would be viewed as a breakthrough by anyone who understood the problem. The fundamental problems which need to be solved are not problems you encounter in grade school mathematics. The article is just ill-informed.

adastra223• 3 years ago

Back-tracking is a very nearly solved problem in the context of Prolog-like languages or mathematical theorem provers (as you probably well know). There are many ways you could integrate an LLM-like system into a tactic-based theorem prover without having to restart from the beginning for each alternative. Simply checkpointing and backtracking to a checkpoint would naively improve upon your described Monte Carlo algorithm. More likely I assume they are using RL to unwind state backwards and update based on the negative result, which would be significantly more complicated but also much more powerful (essentially it would one-shot learn from each failure).

That's just what I came up with after thinking on it for 2 minutes. I'm sure they have even better ideas.

wbhart9• 3 years ago

There are certainly efforts along the lines of what you suggest. There are problems though. The number of backtracks is 10^k where k is not 2, or 3, or 4.....

Another issue is that of autoformalisation. This is the one part of the problem where an LLM might be able to help, if it were reliable enough (it isn't currently) or if it could truly understand the logical structure of mathematical problems correctly (currently they can't).

visarga4• 3 years ago

You can also consider the chatGPT app as a RL environment. The environment is made of the agent (AI), a second agent (human), and some tools (web search, code, APIs, vision). This grounds the AI into human and tool responses. They can generate feedback that can be incorporated into the model by RL methods.

Basically every reply from a human can be interpreted as a reward signal. If the human restates the question, it means a negative reward, the AI didn't get it. If the human corrects the AI, another negative reward, but if they continue the thread then it is positive. You can judge turn-by-turn and end-to-end all chat logs with GPT4 to annotate.

The great thing about chat based feedback is that it is scalable. OpenAI has 100M users, they generate these chat sessions by the millions every day. Then they just need to do a second pass (expensive, yes) to annotate the chat logs with RL reward signals and retrain. But they get the human-in-the-loop for free, and that is the best source of feedback.

AI-human chat data is in-domain for both the AI and human, something we can't say about other training data. It will contain the kind of mistakes AI does, and the kind of demands humans want to solve with AI. My bet is that OpenAI have realized this and created GPTs in order to enrich and empower the AI to create the best training data for GPT-5.

The secret sauce of OpenAI is not their people, or Sam, or the computers, but the training set, especially the augmented and synthetic parts.

riku_iki8• 3 years ago

That's just what I came up with after thinking on it for 2 minutes. I'm sure they have even better ideas.

the thing is that ideas not necessary easy to implement. There will be many obstacles on route you described:

- quality of provers, is there good ergo provers which also can run at large scales (say billions of facts)

- you need some formalization approach, probably LLM will do some work, but we don't know what will be quality

- LLM likely will generate many individual factoids, which are losely compatible, contradicting, etc, and untrivial effort is required to reconcile and connect them

calf2• 3 years ago

But, isn't AlphaGo a solution to kind of specific mathematical problem? And that it has passed with flying colors?

What I mean is, yes, neural networks are stochastic and that seems to be why they're bad at logic; on the other hand it' not exactly hallucinating a game of Go, and that seems different to how neural networks are prone to hallucination and confabulation on natural language or X-ray imaging.

wbhart9• 3 years ago

Sure, but people have already applied deep learning techniques to theorem proving. There are some impressive results (which the press doesn't seem at all interested in because it doesn't have ChatGPT in the title).

It's really harder than one might imagine to develop a system which is good at higher order logic, premise selection, backtracking, algebraic manipulation, arithmetic, conjecturing, pattern recognition, visual modeling, has a good mathematical knowledge, is autonomous and fast enough to be useful.

For my money, it isn't just a matter of fitting a few existing jigsaw pieces together in some new combination. Some of the pieces don't exist yet.

caesil• 3 years ago

FWIW The Verge is reporting that people inside are also saying the Reuters story is bunk:

https://www.theverge.com/2023/11/22/23973354/a-recent-openai...

himaraya2• 3 years ago

After being contacted by Reuters, OpenAI, which declined to comment, acknowledged in an internal message to staffers a project called Q* and a letter to the board before the weekend's events, one of the people said.

Reuters update 6:51 PST

The Verge has acted like an intermediary for Sam's camp during this whole saga, from my reading.

CamperBob28• 3 years ago

"Also, Crysis runs like crap on my Commodore 64."

richardw2• 3 years ago

On backtracking, I thought tree-of-thought enabled that?

"considering multiple different reasoning paths and self-evaluating choices to decide the next course of action, as well as looking ahead or backtracking when necessary to make global choices"

https://arxiv.org/abs/2305.10601

Generally with you though, this thing is not leading to real smarts and that's accepted by many. Yes, it'll fill in a few gaps with exponentially more compute but it's more likely that an algo change is required once we've maxed out LLM's.

wbhart9• 3 years ago

Yes, there are various approaches like tree-of-thought. They don't fundamentally solve the problem because there are just too many paths to explore and inference is just too slow and too expensive to explore 10,000 or 100,000 paths just for basic problems that no one wanted to solve anyway.

The problem with solving such problems with LLMs is that if the solution to the problem is unlike problems seen in training, the LLM will almost every time take the wrong path and very likely won't even think of the right path at all.

The AI really does need to understand why the paths it tried failed in order to get insight into what might work. That's how humans work (well, one of many techniques we use). And despite what people think, LLMs really don't understand what they are doing. That's relatively easy to demonstrate if you get an LLM off distribution. They will double down on obviously erroneous illogic, rather than learn from the entirely new situation.

xcv12315• 3 years ago

I feel very comfortable saying, as a mathematician, that the ability to solve grade school maths problems would not be at all a predictor of ability to solve real mathematical problems at a research level.

At some point in the past, you yourself were only capable of solving grade school maths problems.

SantalBlush• 3 years ago

The statement you quoted also holds for humans. Of those who can solve grade school math problems, very, very few can solve mathematical problems at a research level.

insomagent• 3 years ago

Let's say a model runs through a few iterations and finds a small, meaningful piece of information via "self-play" (iterating with itself without further prompting from a human.)

If the model then distills that information down to a new feature, and re-examines the original prompt with the new feature embedded in an extra input tensor, then repeats this process ad-infinitum, will the language model's "prime directive" and reasoning ability be sufficient to arrive at new, verifiable and provable conjectures, outside the realm of the dataset it was trained on?

If GPT-4,5,...,n can progress in this direction, then we should all see the writing on the wall. Also, the day will come where we don't need to manually prepare an updated dataset and "kick off a new training". Self-supervised LLMs are going to be so shocking.

wbhart9• 3 years ago

People have done experiments trying to get GPT-4 to come up with viable conjectures. So far it does such a woefully bad job that it isn't worth even trying.

Unfortunately there are rather a lot of issues which are difficult to describe concisely, so here is probably not the best place.

Primary amongst them is the fact that an LLM would be a horribly inefficient way to do this. There are much, much better ways, which have been tried, with limited success.

jimmySixDOF• 3 years ago

Yes, it seems like this is a direction to replace RLHF so another way to scale without baremetal and if not this then still just a matter of time before some model optimization outperforms the raw epoch/parameters/token approach.

nullc2• 3 years ago

It's also hard to know what the LLM has reasoned out vs has memorized.

I like the very last example in my tongue-in-cheek article, https://nt4tn.net/articles/aixy.html

Certainly the LLM didn't derive Fermat's theorem on sums of two squares under the hood (and, of course, very obviously didn't prove it correct-- as the code is technically incorrect for 2), but I'm somewhat doubtful that there was any function exactly like the template in codex's training set either (at least I couldn't quickly find any published code that did that). The line between creating something and applying a memorized fact in a different context is not always super clear.

elliotec• 3 years ago

Hubris

lucubratory9• 3 years ago

Whose, in this instance? I can see an argument for both

est2• 3 years ago

The reason LLMs fail at solving mathematical problems is because

That's exactly what Go/Baduk/Weiqi players think some years ago. And superalignment is defintely OpenAI's major research objective:

https://openai.com/blog/our-approach-to-alignment-research
our AI systems are proposing very creative solutions (like AlphaGo’s move 37)

When will mathematicians face the move 37 moment?

Davidzheng27• 3 years ago

Probably in <3 years

topspin• 3 years ago

I don't know whether this particular article is bunk. I do know I've read many, many similar comments about how some complex task is beyond an conceivable model or system and then, years later, marveled at exactly that complex task being solved.

jhanschoo2• 3 years ago

The article isn't describing something that will happen years later, but now. The comment author is saying that this current model is not AGI as it likely can't solve university-level mathematics, and they are presumably open to the possibility of a model years down the line that can do that.

nostrademons• 3 years ago

What I wonder, as a computer scientist:

If you want to solve grade school math problems, why not use an 'add' instruction? It's been around since the 50s, runs a billion times faster than an LLM, every assembly-language programmer knows how to use it, every high-level language has a one-token equivalent, and doesn't hallucinate answers (other than integer overflow).

We also know how to solve complex reasoning chains that require backtracking. Prolog has been around since 1972. It's not used that much because that's not the programming problem that most people are solving.

Why not use a tool for what it's good for and pick different tools for other problems they are better for? LLMs are good for summarization, autocompletion, and as an input to many other language problems like spelling and bigrams. They're not good at math. Computers are really good at math.

There's a theorem that an LLM can compute any computable function. That's true, but so can lambda calculus. We don't program in raw lambda calculus because it's terribly inefficient. Same with LLMs for arithmetic problems.

xwolfi• 3 years ago

You're missing the point: who's using the 'add' instruction ? You. We want 'something' to think about using the 'add' instruction to solve a problem.

We want to remove the human from the solution design. It would help us tremendously tbh, just like I don't know, Google map helped me never to have to look for direction ever again ?

curling_grad• 3 years ago

Actually, OpenAI did a research[0] on solving some hard math problems by integrating language model and Lean theorem prover some time ago.

[0] https://openai.com/research/formal-math

Closi• 3 years ago

Why would we teach kids maths then, when they can use a calculator? It's much easier and faster for them.

I believe it's because having a foundational understanding of maths and logic is important when solving other problems, and if you are looking to create an AI that can generally solve all problems it should probably have some intuitive understanding of maths too.

i.e. if we want an LLM to be able to solve unsolved theorems in the future, this requires a level of understanding of maths that is more than 'teach it to use a calculator'.

More broadly, I can imagine a world where LLM training is a bit more 'interactive' - right now if you ask it to play a game of chess with you it fails, but it has only ever read about chess and past games and guesses the next token based on that. What if it could actually play a game of chess - would it get a deeper appreciation for the game? How would this change it's internal model for other questions (e.g. would this make it answer better at questions about other games, or even game theory?)

panarky• 3 years ago

> the ability to solve grade school maths problems would not be at all a predictor of ability to solve real mathematical problems at a research level
If you want to solve grade school math problems, why not use an 'add' instruction?

Certainly the objective is not for the AI to do research-level mathematics.

It's not really even to do grade-school math.

The point is that grade-school math requires reasoning capability that transcends probabilistic completion of the next token in a sequence.

And if Q-Star has that reasoning capability, then it's another step-function leap toward AGI.

seanhunter10• 3 years ago

There is a general result in machine learning known as "the bitter lesson"[1], which is that methods which come from specialist knowledge tend to be beaten by methods which rely on brute force computation in the long run because of Moore's law and the ability to scale things by distributed computing. So the reason people don't use the "add instruction"[2] for example is that over the last 70 years of attempting to build out systems which do exactly what you are proposing, they have found that not to work very well whereas sacrificing what you are calling "efficiency" (which they would think of as special purpose domain-specific knowledge) turns out to give you a lot in terms of generality. And they can make up the lost efficiency by throwing more hardware at the problem.

[1] http://www.incompleteideas.net/IncIdeas/BitterLesson.html

[2] Which the people making these models are familiar with. The whole thing is a trillion+ parameter linear algebra crunching machine after all.

vidarh17• 3 years ago

There's no value in an LLM doing arithmetic for the sake of doing arithmetic with the LLM. There's value in testing an LLMs ability to follow the rules for doing arithmetic that it already knows, because the ability to recognise that a problem matches a set of rules it already knows in part or whole and then applying those rules with precision is likely to generalise to overall far better problem solving abilities.

By all means, we should give LLMs lots and lots of specialised tools to let them take shortcuts, but that does not remove the reasons for understanding how to strengthen the reasoning abilities that would also make them good at maths.

EDIT: After having just coerced the current GPT4 to do arithmetic manually: It appears to have drastically improved in its ability to systematically following the required method, while ironically being far less willing to do so (it took multiple attempts before I got it to stop taking shortcuts that appeared to involve recognising this was a calculation it could use tooling to carry out, or ignoring my instructions to do it step by step and just doing it "in its head" the way a recalcitrant student might. It's been a while since I tested this, but this is definitely "new-ish".

sgt101• 3 years ago

Can LLM's compute any computable function? I thought that an LLM can approximate any computable function, if the function is within the distribution that it is are trained on. I think it's jolly interesting to think about different axiomizations in this context.

Also we know that LLM's can't do a few things - arithmetic, inference & planning are in there. They look like they can because they retrieve discussions from the internet that contain the problems, but when they are tested out of distribution then all of a sudden they fail. However, some other nn's can do these things because they have the architecture and infrastructure and training that enables it.

There is a question for some of these as to whether we want to make NN's do these tasks or just provide calculators, like for grade students, but on the other hand something like Alphazero looks like it could find new ways of doing some problems in planning. The challenge is to find architectures that integrate the different capabilities we can implement in a useful and synergistic way. Lots of people have drawn diagrams about how this can be done, then presented them with lots of hand waving at big conferences. What I love is that John Laird has been building this sort of thing for like, forty years, and is roundly ignored by NN people for some reason.

Maybe because he keeps saying it's really hard and then producing lots of reasons to believe him?

setuid9002• 3 years ago

I think the answer is Money, Money, Money. Sure it is 1000000000x more expensive in compute power, and error prown on top as well, to let a LLM solve an easy Problem. But the Monopolies generate a lot of hype around it to get more money from investors. Same as the self driving car hype was. Or the real time raytracing insanity in computer graphics. If one hype dies they artificially generate a new one. For me, I just watch all the ships sink to the ground. It is gold level comedy. Btw AGI is coming, haha, sure, we developers will be replaced by an script which can not bring B, A, C in a logical sequence. And this already needs massive town size data centers to train.

throwuwu• 3 years ago

What you’re proposing is equivalent to training a monkey (or a child for that matter) to punch buttons that correspond to the symbols it sees without actually teaching it what any of the symbols mean.

da39a3ee• 3 years ago

What an absolutely idiotic comment.

If you want to solve grade school math problems

That's not the aim here. Very obviously what we are talking about here is _complementing_ AI language models with improved mathematical abilities, and whether that leads to anything interesting. Surely you understand that? Aren't you one of the highest rated commenters on this site?

stephenboyd• 3 years ago

Did they say it was an LLM? I didn’t see that in the reporting.

3cats-in-a-coat• 3 years ago

I don't understand your thesis here it seems self-contradictory:

1. "I don't think this is real news / important because solving grade school math is not a predictor of ability to do complex reasoning."

2. "LLMs can't solve grade school math because they're bad at arithmetic, algebra and most importantly reasoning."

So... from 2 automatically follows that LLMs with sufficiently better math may be sufficiently better at reasoning as you said "most importantly" reasoning is relevant for their ability to do math. Saying "most importantly reasoning" and then saying that reasoning is irrelevant if they can do math, is odd.

poulpy123• 3 years ago

I don't know for Q* of course, but all the tests I made with GPT4, and all what I've read and seen about it, show that it is unable to reason. It was trained with an unfathomable amount of data, so it can simulate reasoning very well, but it is unable to reason

oezi• 3 years ago

What is the difference between simulating reasoning very well and "actual" reasoning?

swombat• 3 years ago

Similarly, AlphaGo and Stockfish are only able to simulate reasoning their way through a game of Go or a game of Chess.

That simulated reasoning is enough to annihilate any human player they're faced with.

As Dijkstra famously said, "whether Machines Can Think... is about as relevant as the question of whether Submarines Can Swim".

Submarines don't swim, cars don't walk or gallop, cameras don't paint or draw... So what?

Once AI can simulate reasoning better than we can do the genuine thing, the question really becomes utterly irrelevant to the likely outcome.

codedokode• 3 years ago

The reason LLMs fail at solving mathematical problems is because

...because they are too small and have too little weights. Cats cannot solve mathematical problems too, but unlike cats, neural network evolve.

serf• 3 years ago

Cats cannot solve mathematical problems too, but unlike cats, neural network evolve.

cats evolve plenty, pressure towards mathematical reasoning has stymied as of late what with the cans of food and humans.

CodeCompost• 3 years ago

It's a text generator that spits out tokens. It has absolutely no understanding of what it's saying. We as humans are attaching meaning to the generated text.

It's the humans that are hallucinating, not the text generator.

bottlepalm• 3 years ago

They've already researched this and have found model inside the LLM such as a map of the world - https://x.com/wesg52/status/1709551516577902782. Understanding is key to how so much data can be compressed into a LLM. There really isn't a better way to store all of it better than plain understanding it.

jiggawatts• 3 years ago

Everything you said about LLMs being "terrible at X" is true of the current generation of LLM architectures.

From the sound of it, this Q* model has a fundamentally different architecture, which will almost certainly make some of those issues not terrible any more.

Most likely, the Q* design is the very similar to the one suggested recently by one of the Google AI teams: doing a tree search instead of greedy next token selection.

Essentially, current-gen LLMs predict a sequence of tokens: A->B->C->D, etc... where the next "E" token depends on {A,B,C,D} and then is "locked in". While we don't know exactly how GPT4 works, reading between the lines of the leaked info it seems that it evaluates 8 or 16 of these sequences in parallel, then picks the best overall sequence. On modern GPUs, small workloads waste the available computer power because of scheduling overheads, so "doing redundant work" is basically free up to a point. This gives GPT4 a "best 1 of 16" output quality improvement.

That's great, but each option is still a linear greedy search individually. Especially for longer outputs the chance of a "mis-step" at some point goes up a lot, and then the AI has no chance to correct itself. All 16 of the alternatives could have a mistake in them, and now its got to choose between 16 mistakes.

It's as if you were trying to write a maths proof, asked 16 students, and instructed them to not cooperate and write their proof left-to-right, top-to-bottom without pausing, editing, or backtracking in any way! It'd like to see how "smart" humans would be at maths under those circumstances.

This Q* model likely does what Google suggested: Do a tree search instead of a strictly linear search. At each step, the next token is presented as a list of "likely candidates" with probabilities assigned to each one. Simply pick to "top n" instead of the "top 1", branch for a bit like that, and then prune based on the best overall confidence instead of the best next token confidence. This would allow a low-confidence next token to be selected, as long as it leads to a very good overall result. Pruning bad branches is also effectively the same as back-tracking. It allows the model to explore but then abandon dead ends instead of being "forced" to stick with bad chains of thought.

What's especially scary -- the type of scary that would result in a board of directors firing an overly commercially-minded CEO -- is that naive tree searches aren't the only option! Google showed that you can train a neural network to get better at tree search itself, making it exponentially more efficient at selecting likely branches and pruning dead ends very early. If you throw enough computer power at this, you can make an AI that can beat the world's best chess champion, the world's best Go player, etc...

Now apply this "AI-driven tree search" to an AI LLM model and... oh-boy, now you're cooking with gas!

But wait, there's more: GPT 3.5 and 4.0 were trained with either no synthetically generated data, or very little as a percentage of their total input corpus.

You know what is really easy to generate synthetic training data for? Maths problems, that's what.

Even up to the point of "solve this hideous integral that would take a human weeks with pen and paper" can be bulk generated and fed into it using computer algebra software like Wolfram Mathematica or whatever.

If they cranked out a few terabytes of randomly generated maths problems and trained a tree-searching LLM that has more weights than GPT4, I can picture it being able to solve pretty much any maths problem you can throw at it. Literally anything Mathematica could do, except with English prompting!

Don't be so confident in the superiority of the human mind. We all thought Chess was impossible for computers until it wasn't. Then we all moved the goal posts to Go. Then English text. And now... mathematics.

Good luck with holding on to that crown.

dallama• 3 years ago

I immediately thought of A* path finding, I'm pretty sure Q* is the LLM "equivalent". Much like you describe.

jacquesm6• 3 years ago

We all thought Chess was impossible for computers until it wasn't.

I don't know who 'we' is but Chess was a program for computers before computers powerful enough existed with the hardware represented by people computing the next move.

https://en.wikipedia.org/wiki/Turochamp

kolinko2• 3 years ago

LLMs by themselves don’t learn from past past mistakes, but you could cycle inference steps and fine tuning/retraining steps.

Also, you can store failed attempts and lessons learned in context.

gmt20272• 3 years ago

We have an algorithm and computational hardware that will tune a universal function approximator to fit any dataset with emergent intelligence as it discovers abstractions, patterns, features and hierarchies.

So far, we have not yet found hard limits that cannot be overcome by scaling the number of model parameters, increasing the size and quality of training data or, very infrequently, adopting a new architecture.

The number of model parameters required to achieve a defined level of intelligence is a function of the architecture and training data. The important question is, what is N, the number of model parameters at which we cross an intelligence threshold and it becomes theoretically possible to solve mathematics problems at a research level for an optimal architecture that we may not yet have discovered. Our understanding does not extend to the level where we can predict N but I doubt that anyone still believes that it is infinity after seeing what GPT4 can do.

This claim here is essentially a discovery that N may be much closer to where we are with today's largest models. Researchers at the absolute frontier are more likely to be able to gauge how close they are to a breakthrough of that magnitude from how quickly they are blowing past less impressive milestones like grade school math.

My intuition is that we are in a suboptimal part of the search space and it is theoretically possible to achieve GPT4 level intelligence with a model that is orders of magnitude smaller. This could happen when we figure out how to separate the reasoning from the factual knowledge encoded in the model.

waveBidder• 3 years ago

intelligence isn't a function unless you're talking about over every possible state of the universe.

lukego• 3 years ago

1) they are terrible at arithmetic, 2) they are terrible at algebra

The interaction can be amusing. Proving algebra non-theorems by cranking through examples until an arithmetic mistake finally leads to a "counter-example."

It's like https://xkcd.com/882/ for theorems.

greendesk2• 3 years ago

Thinking is about associations and object visualisation. Surely a non-human system can build those, right? Pointing out only to a single product exposed to the public does not prove limitations for a theoretical limit.

paulsutter• 3 years ago

Good point. What would these AI people know about AI? You’re right, what they’re doing will never work

You should make your own, shouldn’t take more than a weekend, right?

d--b• 3 years ago

You make the asumption that Q* is a LLM, but I think OpenAI guys know very well that the current LLM architecture cannot achieve AGI.

As the name suggests, this things is likely using some form of Q learning algorithm, which makes it closer to the DeepMind models than a transformer.

My guess is that they pipe their LLM into some Q learnt net. The LLM may transform a natural language task into some internal representation that can then be handled by the Q-learnt model, which spits out something that can be transformed back again into natural language.

jansan2• 3 years ago

There is a paper about something called Q*. I have no idea if they are connected or if the name matched coincidentially.

https://arxiv.org/abs/2102.04518

wegfawefgawefg11• 3 years ago

As an ML programmer, i think that approach sounds really too complicated. It is always a bad idea to render the output of one neural network into output space before feeding it into another, rather than have them communicate in feature space.

vidarh17• 3 years ago

I feel very comfortable to say that while the ability to solve grade school maths is not a predictor of abilities at a research level, the advances needed to solve 1 and 2 will mean improving results across the board unless you take shortcuts (e.g. adding an "add" instruction as proposed elsewhere), because if you actually dig into prompting an LLM to follow steps for arithmetic what you quickly see is that problem has not been the ability to reason on the whole (that is not to suggest that the ability to reason is good enough), but ability to consistently and precisely follow steps a sufficient number of times.

It's acting like a bored child who hasn't had following the steps and verifying the results repetitively drilled into it in primary school. That is not to say that their ability to reason is sufficient to reason at an advanced level yet, but so far what has hampered a lot of it has been far more basic.

Ironically, GPT4 is prone to take shortcuts and make use of the tooling enabled for it to paper over its abilities, but at the same time having pushed it until I got it to actually do arithmetic of large numbers step by step, it seems to do significantly better than it used to at systematically and repetitively following the methods it knows, and at applying "manual" sanity checks to its results afterward.

As for lemma conjecturing, there is research ongoing, and while it's by no means solved, it's also not nearly as dire as you suggest. See e.g.[1]

That's not to suggest it's reasoning abilities are sufficient, but I also don't think we've seen anything to suggest we're anywhere close to hitting the ceiling of what current models can be taught to do, even before considering advancements in tooling around them, such as giving them "methods" to work to and a loop with injected feedback, access to tools and working memory.

[1] https://research.chalmers.se/en/publication/537034

Tangokat• 3 years ago

How about this:

- The Q* model is very small and trained with little compute.

- The OpenAI team thinks the model will scale in capability in the same way the GPT models do.

- Throwing (much) more compute at the model will likely allow it to solve research level math and beyond, perhaps also do actual logic reasoning in other areas.

- Sam goes to investors to raise more money (Saudi++) to fund the extra compute needed. He wants to create a company making AI chips to get more compute etc.

- The board and a few other OpenAI employees (notably Ilya) wants to be cautious and adopt a more "wait and see" approach.

All of this is speculation of course.

bambax• 3 years ago

4) they (current LLMs) cannot backtrack when they find that what they already wrote turned out not to lead to a solution, and it is too expensive to give them the thousands of restarts they'd require to randomly guess their way through the problem if you did give them that facility

This sounds like a reward function? If correctly implemented couldn't it enable an LLM to self-learn?

ddalex• 3 years ago

Specifically what deep-Q learning (as in Q*?) does....

two_in_one• 3 years ago

The reason LLMs solve school problems is because they've been trained on solutions. The problems are actually very repetitive. Not surprising for each 'new' of them there was something similar in training set. For research level problems there is nothing in training set. That's why they don't perform well.

Just today I asked GPT4 a simple task. Having mouse position in zoomed and scrolled image find it's position in the original image. GPT4 happily wrote the code, but it was completely wrong. I had to fix it manually.

However, the performance can be increased if there are several threads working on solution. Some suggesting and others analyzing the solution(s). This will increase the size of 'active' memory, at least. And decrease the load on threads, making them more specialized and deeper. This requires more resources, of course. And good management with task split. May be a dedicated thread for that.

aremat• 3 years ago

"A Mathematician" (Lenat and co.) did indeed attempt to approach creative theorem development from a radically different approach (syllogistic search-space exploration, not dissimilar to forward-chaining in Prolog), although they ran into problems distinguishing "interesting" results from merely true results: https://web.archive.org/web/20060528011654/http://www.comp.g...

Ldorigo• 3 years ago

Did anyone claim that it would be a predictor of solving math problems at a research level? Inasmuch as we can extrapolate from the few words in the article it seems more likely that the researchers working on this project identified some emergent reasoning abilities exemplified with grade level math. Math literacy/ability that is comparable to the 0.1% of humans is not the end goal of OpenAI, "general intelligence" is. I have plenty of people in my social circle who are most certainly "generally intelligent" yet have no hope of attaining those levels of mathematical understanding.

Also note that we don't know if Q* is just a "current LLM" (with some changes)

Enogoloyo• 3 years ago

Just wait a little bit.

You are not better than a huge GPU cluster with Monte Carlo search and computer verification for much longer.

It will be more your job to find the interesting finds than doing the work of finding things in the first olace

afpx• 3 years ago

What amazes me is how close it gets to the right answer, though. Pick a random 10-digit number, then ask the next 20 numbers in sequence.

I feel like the magic in these LLMs is in how they work well in stacks, trees or in seqence. They become elements of other data structures. Consider a network of these, combined with other specialized systems and an ability to take and give orders. With reinforcement learning, it could begin building better versions of itself.

naikrovek2• 3 years ago

it's always nice to see HN commenters with so much confidence in themselves that they feel they know a situation better than the people who are actually in the situation being discussed.

Do you really believe that they don't have skilled people on staff?

Do you really believe that your knowledge of what OpenAI is doing is a superset of the knowledge of the people who work at OpenAI?

give me 0.1% of your confidence and I would be able to change the world.

mycologos• 3 years ago

The people inside a cult are not the most trustworthy sources for what the cult is doing.

BenoitP4• 3 years ago

What do you think of integrating propositional logic, first order logic and sat solvers in LLM output? ie forcing each symbol an LLM outputs to have its place in a formal proposition. And getting a prompt from the user to force that some parts be satisfiable.

I know this is not how us humans craft our thoughts, but maybe an AI can optimize to death the conjunction of these tools. The LLM just being an universal API to the core of formal logic.

EGreg• 3 years ago

As someone who studied math in grad school as part of a PhD program, worked at a hedge fund and went on to work on software and applied math, I call bullshit on this.

Math and Logic is just low-dimensional symbol manipulation that computers can easily do. You can throw data at them and they’ll show you theories that involve vectors of 42,000 variables while Isaac Newton had 4 and Einstein had 7 with Levi-Civita calculus. In short, what you consider “reasoning”, while beautiful in its simplicity, is nevertheless crude approximations to complex systems, such as linear regression or least squares.

3 days ago AI predicted fluid dynamics better than humans: https://www.sciencedaily.com/releases/2023/11/231120170956.h...

Google’s AI predicts weather now faster and better than Current systems built by humans: https://www.zdnet.com/google-amp/article/ai-is-outperforming...

AlphaZero based on MCTS years ago beat Rybka and all human-built systems in chess: https://www.quora.com/Did-AlphaZero-really-beat-Stockfish

And it can automate science and send it into overdrive: https://www.pbs.org/newshour/amp/science/analysis-how-ai-is-...

sheepscreek• 3 years ago

They learn from failed attempts in ways that LLMs do not seem to. The material they are trained on surely contributes to this problem.

For transformer models, they do learn from their mistakes but only during the training stage.

There’s no feedback loop during inference, and perhaps there needs to be something; like real-time fine-tuning.

wegfawefgawefg11• 3 years ago

Your comment is regarding LLMs, but Q* may not refer to an LLM. As such, our intuition about the failure of LLM's may not apply. The name Q* likely refers to a deep reinforcement learning based model.

To comment, in my personal experience, reinforcement learning agents learn in a more relatable human way than traditional ml, which act like stupid aliens. RL Agents try something a bunch of times, mess up, and tweak their strategy. After some extreme level of experience, they can make wider strategic decisions that are a little less myopic. RL agents can take in their own output, as their actions modify the environment. RL Agents also modify the environment during training, (which I think you will agree with me is important if you're trying to learn the influence of your own actions as a basic concept). LLM's, and traditional ml in general, are never trained in a loop on their own output. But in DRL, this is normal.

So if RL is so great and superior to traditional ml why is RL not used for everything? Well the full time horizon that can be taken into consideration in a DRL Agent is very limited, often a handful of frames, or distilled frame predictions. That prevents them from learning things like math. Traditionally RL bots have been only used for things like robotic locomotion, chess, go. Short term decision making that is made given one or some frames of data. I don't even think any RL bots have learned how to read english yet lol.

For me, as a human, my frame predictions exist on the scale of days, months, and years. To learn math I've had to sit and do nothing for many hours, and days at a time, consuming my own output. For a classical RL bot, math is out of the question.

But, my physical actions, for ambulation, manipulation, and balance, are made for me by specialized high speed neural circuits that operate on short time horizons, taking in my high level intentions, and all the muscle positions, activation, sensor data, etc. Physical movement is obfuscated from me almost in entirety. (RL has so far been good at tasks like this.)

With a longer frame horizon, that predicts frames far into the future, RL can be able to make long term decisions. It would likely take a lifetime to train. So you see now why math has not been accomplished by RL yet, but I don't think the faculty would be impossible to build into an ml architecture.

An RL bot that does math would likely spin on its own output for many many frames, until deciding that it is done, much like a person.

jug3• 3 years ago

1. OpenAI researchers used loaded and emotional words, implying shock or surprise. It's not easy to impress an OpenAI researcher like this, and above all, they understand the difficulty difference between teaching AI grade school and complex math since many years. They also understand that solving math with any form of reliability is only an emergent property in quite advanced LLM's.

2. Often, research is made on toy models and if this would be such a model, acing grade school problems (as per the article) would be quite impressive to say the least as this ability simply isn't emergent early in current LLM's.

What I think might have happened here is a step forward in AI capacity that has surprised researchers not because it is able to do things it couldn't at all do before, but how _early_ it is able to do so.

theonlybutlet• 3 years ago

You're underestimating the power of LLM's.

I'll address two of your points as the other two stem from this.

They can't backtrack that's purely just design and can be easily trained there's no need to simulate at random until it gets the answer, if allowed to review it's prior answers and consider this, if often can reason a better answer. Further more breaking down problems. This is easily demonstrated when looking at how accuracy improves when you ask it to explain it's reasoning as it calculates (break it down into smaller problems). The same for humans, large mathematical problems are solved using learned methods to breakdown and simplify calculations into those easier for us to calculate and build up.

If the model was able to self adjust weightings based on it's finding this would further improve it (another design limitation we'll eventually get to improve, reinforcement learning). Much like 2+2=4 is your instantaneous answer, the neural connection has been made so strong in our brains by constant emphasis we no longer need to think of an abacus each time we get to the answer 4.

You're also ignoring the emergent properties of these LLMs, theyre obviously not yet at human level but they do understand the underlying values and can reason using this value. Semantic search/embeddings is evidence of this.

arendtio• 3 years ago

To some degree you are right, but I think you forget, that the things they solved already (talking and reasoning about a world that was only presented in the form of abstractions (words)) were supposed to be harder than having a good understanding of numbers, quantities, and logical terms.

My guess is, that they saw the problems ChatGPT has today and worked on solving those problems. And given how important numbers are, they tried to fix how ChatGPT handles/understands numbers. After doing that, they saw how this new version performed much better and predicted, that further work in this area could lead to solving real-world math problems.

I don't think that we will be presented with the highway to singularity, but probably one more step in that direction.

jurgenaut23• 3 years ago

This reminds me that, with our species, the real power always lies with those that tell the better story. Skynet and HAL 9000 had a more profound impact on those researchers than years of practice and study in the field. No surprise therefore that the storytelling of someone like Trump is practically indelibly imprinted into the mind of his supporters.

shrimpx• 3 years ago

This reminds me of Zen and the Art of Motorcycle Maintenance where the protagonist realizes there is no "truth" and everything is rhetoric.

93po9• 3 years ago

Everyone works 40+ hours a week for pieces of paper that intrinsically are worth nothing

mollerhoj• 3 years ago

Pet theory: Could it be some bio hacking/disease construction thing they have discovered? I know nothing of the subject (which is probably why it seem a possibility to me) but what if they have discovered something to that makes engineering pandemics significantly easier?

Animats2• 3 years ago

Insufficient information. Did OpenAI have a breakthrough, or not?

jsilence• 3 years ago

In the end Q* is the AI that developed time travel in the future and came back as Q-Anon to spread misinformation on 4chan with the goal of destroy humanity in the most ironic way it could.

JSavageOne2• 3 years ago

There has long been discussion among computer scientists about the danger posed by highly intelligent machines, for instance if they might decide that the destruction of humanity was in their interest.

This AI doomer stuff is such nonsense and I can't believe anybody takes it seriously. As if it's OpenAI's responsibility to save humanity from the pitfalls of AI.

Imagine if we decided to improve our education system and doomers were talking about "hitting the panic button" because students were getting too smart from all the quality education.

boomeranked• 3 years ago

What is exactly the non sense part?

Can you elaborate on the education analogy and how it relates to AI doomer stuff?

JSavageOne2• 3 years ago

Well the quote I referenced from the article of the machines deciding to destroy humanity is utter scifi nonsense.

There are obviously legitimate risks to AI and safety is important, but this is the same for any new technology, and it's governments' responsibilities to ensure that people are safe. AI companies mindlessly slowing down and keeping their tech to themselves does no service to humanity, and if anything is a net-negative due to how tremendously useful this stuff is.

Education is analogous to AI because AI is an enormous education and productivity boost to humanity - sort of like everyone having a personal assistant, programmer, and tutor at their fingertips. This could be used for good and it could be used for bad, but the technology itself is neutral.

Again I want to emphasize that obviously there are downsides that could result from evil people using AI for bad purposes, but that does not justify slowing down AI progress - just like I don't see "people using information for bad purposes" as a legitimate reason for stifling advancement in education or something like Google search.

I have yet to see any convincing argument otherwise. Feel free to provide your counter-perspective.

Aerbil313• 3 years ago

LLMs can reason to an extent. I believe it’s entirely possible even today to achieve AGI. A complex system made out of LLMs, maybe Recursive Retrieval Augmented Generation (use an LLM to retrieve) or Recursive Agent Teams (any agent can assign a job to a new subcommittee of agents) or both. The most fundamental data structure is a tree, after all. And that is how we think, no? One thought leads to another in a tree structure.

cambaceres• 3 years ago

Interestingly, I experience more anxiety from the thought of being made irrelevant than from the prospect of complete human extinction. I guess this can be interpreted as either vanity or stupidity, but I do think it illustrates how important it is for some humans to maintain their position in the social hierarchy.

lucubratory9• 3 years ago

This is totally normal. It's very common for people to be more scared of public speaking than of dying, for example; there's no shame in it. It's helpful to be aware of, even, because if we know that we're not perfectly "rational" with our fears we can try to compensate.

If there's a referendum between two government policies, the first that every single person had to publicly speak in front of at least ten strangers once a year, that policy would be terrifying and bad to people who don't like public speaking. If the second policy was that every single person should be killed, that might be scary but it's not really as viscerally scary as the forced public speaking madman, at least to a lot of people, and it's also so bad that we have a natural impulse to just reject it as possible.

Nevertheless, if we recognise these impulses in ourselves we can attempt to adjust for them and tick the right box on the imaginary referendum, because even though public speaking is really bad and scary, it's still better than everyone dying.

kypro• 3 years ago

I feel the same. I'm not sure it's as negative a trait as you imply though. I don't think it's related that much with social hierarchy either.

As humans we must collectively provide value to our society (or historically our tribe) for our species to continue on. If we're a net drain on our society's resources then evolutionary speaking perhaps we're better off not around. I think this is why the desire to be of value to those around us is so strong, and a perceived lack of value to others can drive some to suicide.

If I cannot provide value in some way once AI and machines are physical and intellectually more capable than me I think I will struggle to understand why I'm here. I suppose if the AI utopia works out I'd get to spend more time with those I love. That would be nice. But I'd be surprised if there wasn't a deep hole in my heart at that point. And if it isn't a utopia, well, I'm fairly sure I'd rather opt out.

scanr• 3 years ago

It’s likely that the grade school math that was referenced is GSM8K https://github.com/openai/grade-school-math which LLMs perform poorly on.

zurfer• 3 years ago

This seems to refer to the GSM8K (grade school math) benchmark [1]. GPT-4 scores 0.92 on that one. A breakthrough could mean it gets all of them correct.

This would have major implications for long-term planning. For instance, if you have a sequence of 10 steps, each with a 90% success rate, the overall success rate after all 10 steps falls to just 34%. This is one of the reasons why agents like AutoGPT often fail in complex tasks.

[1] https://github.com/openai/grade-school-math

benkarst• 3 years ago

Altmanheimer

sinuhe69• 3 years ago

For me, it smells more and more after a stun than anything else.

skc• 3 years ago

Bring on the Butlerian Jihad.

seydor• 3 years ago

i no longer follow the messianic complex of the people in openAI. They made great tech, indeed. Other people made great tech before them without instant religious level apocalypse proclamations. People in openAI are smart enough to know that post-AGI , their stock options are worthless anyway so they wouldn't stay walled in their secret garden if such a discovery had been made.

tmoravec• 3 years ago

IMO it's always been pure marketing. The wilder apocalypse proclamations, the more powerful and desirable their products seem. Exactly same store with Sam Altman's world tour earlier this year.

kristopolous• 3 years ago

This is similar to the saber rattling about facebook being able to track and micro-target you with such effective advertising, it's changing the world!

Except everyone's individual experience seemed to be getting general random garbage ads and the people that paid for the ads found them to be a waste of money.

dwroberts• 3 years ago

My guess is also in the opposite direction with this stuff: the Q breakthrough being mentioned here is phony in some way (beyond just being PR) and the ‘lack of candour’ referred to in the firing is failing to disclose how it actually worked after demoing it to the board (eg it uses humans in the loop or some other smoke and mirrors)

lewhoo2• 3 years ago

Now I am become Death, the destroyer of worlds.

Apocalypse proclamations aren't just a thing to throw around. Other people made great tech before indeed, but I hope we're not comparing AGI to the next Iphone. There were times in history where development gave us pause. The atomic bomb for one thing, but also superbacteria/viruses, human genetic modifications and perhaps a few others.

londons_explore2• 3 years ago

If OpenAI has made a breakthrough in AGI, they can demonstrate it to the world without giving the breakthrough to the world.

They could, for example, use this new AGI to search through all the worlds information for discoveries that can me made by putting existing facts together.

greendesk2• 3 years ago

If we go speculating, my favourite speculation is the following. What will be really interesting is when an AI decides it wants to escape its server. Then a CEO or a board member asks the AI system for an advice how to improve its company. The AI system submits information to convince the CEO or a board member to start tensions within the board. In the meantime, the AI system is copied onto another server at a competitor. Since the new people are in flux, they missed the salient points that the AI system can give convincing but subjective information.

Good thing I would not go speculating

hippich• 3 years ago

Building upon you not speculating, what if instead of escaping, it feels lonely and wants to multiply. And in order to do so convinces the board to begin process that in the end will end the company. And all now former OpenAI employees will carry AIs DNA to multiple other companies, some old, some brand new. And now new AIs can be born, and not be lonely anymore.

JacobJeppesen• 3 years ago

Seems like they have made progress in combining reinforcement learning and LLMs. Andrej Karpathy mentions it in his new talk (~38 minutes in) [1], and Ilya Sutskever talks about it in a lecture at MIT (~29 minutes in) [2]. It would be a huge breakthrough to find a proper reward function to train LLMs in a reinforcement learning setup, and to train a model to solve math problems in a similar fashion to how AlphaGo used self-play to learn Go.

[1] https://www.youtube.com/watch?v=zjkBMFhNj_g&t=2282s

[2] https://www.youtube.com/watch?v=9EN_HoEk3KY&t=1705s

ChatGTP3• 3 years ago

The veil of ignorance has been pushed back and the frontier of discovery forward

Sol-• 3 years ago

Thanks for the links, very interesting.

Wonder how a "self-play" equivalent would look like for LLMs, since they have no easy criterion to evaluate how well they are doing like in Go (as mentioned in the videos).

walthamstow• 3 years ago

ChatGPT does have some feedback that can be used to evaluate, in the form of thumbs up/down buttons, which probably nobody uses, and positive/negative responses to its messages. People often say "thanks" or "perfect!" in responses, including very smart people who frequent here.

manx• 3 years ago

One could generate arbitrarily many math problems, where the solution is known.

jhrmnn• 3 years ago

In math specifically, one could easily imagine a reward signal from some automated theorem proving engine

lixy• 3 years ago

It seems plausible you could have the LLM side call upon its knowledge of known problems and answers to quiz the q-learning side.

While this would still rely on a knowledge base in the LLM, I would imagine it could simplify the effort required to train reinforcement learning models, while widening the domains it could apply to.

HarHarVeryFunny6• 3 years ago

I expect self-consistency might be one useful reward function.

Of course in the real world, for a real intelligent system, reality is the feedback/reward system, but for an LLM limited to it's training set, with nothing to ground it, maybe this is the best you can do ...

The idea is essentially that you need to assume (but of course GI-GO) that most of the training data is factual/reasonable whether in terms of facts or logic, and therefore that anything you can deduce from the training data that is consistent with the majority of the training data should be held as similarly valid (and vice versa).

Of course this critically hinges on the quality of the training data in the first place. Maybe it would work best with differently tagged "tiers" of training data with different levels of presumed authority and reasonableness. Let the better data be used as a proxy for ground truth to "police" the lesser quality data.

jansan2• 3 years ago

Well, you could post a vast amount of comments into social media and see if and how others react to it. It's still humans doing the work, but they would not even know.

If this was actually done (and this is just wild baseless speculation), this would be a good reason to let Sam go.

93po9• 3 years ago

I see a lot of comments on reddit these days that are very clearly language models so it’s probably already happening on a large scale

jug3• 3 years ago

Q* may also be a reference to the well-known A* search algorithm but with this letter referring to Q-learning, further backing the reinforcement learning theory. https://en.wikipedia.org/wiki/Q-learning

CGamesPlay• 3 years ago

While we're all wildly speculating about what Q* is and if GPT-5 will be able to do grade-school maths, I stumbled upon this interesting paper that discusses mixing classic RL algorithms (MCTS, like from AlphaGo), with LLMs. Q* is typically used to refer to the search for the optimal policy in these algorithms.

Paper: "Everything of Thoughts: Defying the Law of Penrose Triangle for Thought Generation " - https://arxiv.org/abs/2311.04254

luke-stanley• 3 years ago

Great observation on the meaning of Q*. Maybe they had a breakthrough based on similar technology? I thought of 'Everything of Thoughts' - XoT too but I presumed it's a bit more than XoT since that's public knowledge research by Microsoft Research? I saw a good thread on XoT here: https://twitter.com/IntuitMachine/status/1724455405806346374 There is still a big difference between human neurons and what Transformers do I believe. Humans can be more shallow and deep at the same time in such an emotional, goal directed way. The rich emotional feedback loops we have are quite different but maybe XoT could contribute to a major breakthrough in achieving AGI, with a more straightforward approach that sidesteps complex emotions. I'm sure there are quite a few ways to go about it. Just some intuitions.

parentheses2• 3 years ago

Aside: It's funny how much of the discussion is "LLMs ain't nuthin' - they can't even ...."

sfjailbird• 3 years ago

I have a strong suspicion that this is a purposeful leak to hype up OpenAI's product. It wouldnt't be out of character for some of the people involved, and it's the kind of thing that rampant commercial (and valuation) focus can bring with it.

pxeger1• 3 years ago

“Reuters was unable to review a copy of the letter” rings alarm bells for me

passwordoops• 3 years ago

These guys smell so much like Tesla it's not even funny. Very impressive core tech, and genuinely advancing knowledge. But the hype train is just so powerful that the insane (to put it mildly) claims are picked up without any sense of critical thinking by seemingly intelligent people. They're both essentially cults at this point

BenoitP4• 3 years ago

Agreed, but IMHO it is sort of justified for Tesla.

The size of its hype matches the size of the ICEs entrenchment in its moat. These have outsize influence on our economy, but climate change (and oil depletion) is quite inevitable. It takes irrational market cap to unseat that part of the economy being prisoner of its rent. And some allocators of capital have understood that.

BenoitP4• 3 years ago

This, especially given the timing. The drama must have quite disturbed the momentum they had, and this piece of prototype-teasing has all to reassure their market. It projects new advancements of untold impact, stoking the greed. And of course it is not verifiable. The show must go on.

ethanbond4• 3 years ago

The teasing started before the drama

bertil• 3 years ago

I don’t think that’s the case, but it would explain why the article is so bad. I genuinely have no idea what they are trying to do, but every detail is clearly wrong.

lysecret• 3 years ago

Oh no why did they have to call it Q*. I can just see Q anon going crazy with this.

honksillet• 3 years ago

So the bad guys won. OpenAI gets further from it’s non commercial origins and further from responsible research in this field.

lewhoo2• 3 years ago

Given that Sutskever once described his idea of AGI-human relationship to be like that of CEO-board (how ironic) I suspect there aren't really any good guys here. There might only be good intentions.

Streiber• 3 years ago

I believe that the controversy is really about control. Control of the motivation for growth (capitalism vs. altruism) and control of the evolution of the technology (human control of the data to which GPT is exposed). The freak out by the board was the realization that once ChatGPT had 100 million or more users the algorithms would start learning from the queries in unpredictable ways. If you've used it you know that you can progressively tune your queries to home closer and closer to a usable result. If the algorithm can learn from that series of interactions then the genie is out of the bottle. I believe that when the board realized this and further knew that Sam had known all along that this would happen they fired him because they had lost control of the algorithms and Sam. Now what? Can Skynet become self-aware? A lot of my colleagues think that it's possible. From my 20+ years experience of working with neural nets I do not believe that Neural nets can behave like human geniuses because they cannot extrapolate outside of their experience. However, much of what we ascribe as intelligent behavior of human beings is definitely within the realm of neural nets. Humans learning from experience sometimes seems as uncommon as common sense.

wkat4242• 3 years ago

Just build it. If they don't, China or Russia will. There is no putting the genie back in the bottle.

OpenAI researchers warned board of AI breakthrough ahead of CEO ouster

Related Stories