John Carmack talk at Upper Bound 2025

https://twitter.com/ID_AA_Carmack/status/1925710474366034326

by tosh • 1 year ago

557 342 1 year ago

kamranjon• 1 year ago

I was really excited when I heard Carmack was focusing on AI and am really looking forward to watching this when the video is up - but just from looking at the slides it seems like he tried to build a system that can play the Atari? Seems like a fun project, but curious what will come out of it or if there is an associated paper being released.

johnb23115• 1 year ago

Atari games are widely used in Reinforcement Learning (RL) research as a standard benchmark.

https://github.com/Farama-Foundation/Arcade-Learning-Environ...

The goal is to develop algorithms that generalize to other tasks.

sigmoid106• 1 year ago

They were highly used. OpenAI even included them in their RL Gym library back in the old days when they were still doing open research. But if you look at this leaderboard from 7 (yes, seven!) years ago [1], most of them were already solved way beyond human capabilities. But we didn't get a really useful general purpose algorithm out of it. As an AI researcher, I always considered Atari a fun academic exercise, but nothing more. Similar to how recognising characters using convnets was cool in the nineties and early 00s, but didn't give us general purpose image understanding. Only modern GPUs and massive training datasets did. Nowadays most cutting-edge RL game research focuses on much more advanced games like Minecraft which is thought to be better suited. But I'm pretty sure it's still not enough. Even role-playing GTA VI won't be. We probably need a pretty advanced physical simulation of the real world before we can get agents to handle the real world. But that means solving the problem of generating such an environment first, because you can't train on the actual real world due to the sample inefficiency of all current algorithms. Nvidia is doing some really interesting research in this direction by combining physics simulation and image generation models to simulate an environment, while getting accuracy and diversity at the same time into training data. But it still feels like some key ingredient is missing.

[1]https://github.com/cshenton/atari-leaderboard

cryptoz• 1 year ago

DeepMind’s original demos were also of Atari gameplay.

modeless• 1 year ago

He says they will open source it which is cool. I agree that I don't understand what's novel here. Playing with a physical controller and camera on a laptop GPU in real time is cool, and maybe that hasn't specifically been done before, but it doesn't seem surprising that it is possible.

If it is substantially more sample efficient, or generalizable, than prior work then that would be exciting. But I'm not sure if it is?

RetroTechie2• 1 year ago

Maybe that's exactly his goal: not to come up with something that beats the competition, but play with the architecture, get a feel for what works & what doesn't, how various aspects affect the output, and improve on that. Design more efficient architectures, or come up with something that has unique features compared to other models.

If so, scaling up may be more of a distraction rather than helpful (besides wasting resources).

I hope he succeeds in whatever he's aiming for.

tschillaci• 1 year ago

You will find many agents that solved (e.g., finished, reached high score) atari games, but there is still so much more work to do in the field. I wrote my Master's thesis on how to learn from few interactions with the game, so that if the algorithm is ported to actual robots they don't need to walk and fall for centuries before learning behaviors. I think there is more research to do on higher levels of generalization: when you know how to play a few video games, you quickly understand how to play a new one intuitively, and I haven't seen thorough research on that.

lo0dot0• 1 year ago

I can tell you right now without any research that video game designers reuse interface patterns and game mechanics that were already known when making new games. Those patterns and mechanics are also often analogies for real life allowing humans to intuitively play the games. If people can't play your game intuitively, they might say it's a bad game.

albertzeyer2• 1 year ago

His goal was not just to solve Atari games. That was already done.

His goal is to develop generic methods. So you could work with more complex games or the physical world for that, as that is what you want in the end. However, his insight is, you can even modify the Atari setting to test this, e.g. to work in realtime, and the added complexity by more complex games doesn't really give you any new additional insights at this point.

mike_hearn• 1 year ago

But how is this different to what NVIDIA have already done? They have robots that can achieve arbitrary and fluid actions in the real world by training NNs in very accurate GPU simulated environments using physics engines. Moving a little Atari stick around seems like not much compared to sorting through your groceries etc.

The approach NVIDIA are using (and other labs) clearly works. It's not going to be more than a year or two now before robotics is as solved as NLP and chatbots are today.

gadders• 1 year ago

I want smarter NPCs in games.

moralestapia2• 1 year ago

Here's what they built, https://x.com/ID_AA_Carmack/status/1925243539543265286

Quite exciting. Without diminishing the amazing value of LLMs, I don't think that path goes all the way to AGI. No idea if Carmack has the answer, but some good things will come out of that small research group, for sure.

petters• 1 year ago

Isn't that what Deepmind did 12 years ago?

willvarfar2• 1 year ago

Playing Atari games makes it easy to benchmark and compare and contrast his future research with Deepmind and more recent efforts.

moralestapia2• 1 year ago

IIRC Deepmind (and OpenAI and ...) have done this on software-only setups (emulators, TAS, etc); while this one has live input and actuators in the loop, so, kind of the same thing but operating in the physical realm.

I do agree that it is not particularly groundbreaking, but it's a nice "hey, here's our first update".

hombre_fatal• 1 year ago

He points that out in his notes and says DeepMind needed specialized training and/or 200M frames of training just to kinda play one game.

dusted3• 1 year ago

anywhere we can watch the presentation ? the slides alone are great, but if he's saying stuff alongside, I'd be interested in that too :)

vasco2• 1 year ago

Bro went his whole career and managed to somehow create a gig for himself where he gets the AI money while playing Atari. It's hard to increase the respect for someone who you already maxed out on but there we go. Carmack is a cool guy.

willvarfar2• 1 year ago

Although Carmack is the quintessential not-a-brogrammer.

dusted3• 1 year ago

I've never actually seen a brogrammer though, I've seen people who program only because they get money for it, I thought for a while those where it, but I'm not sure if I think they qualify either.

vasco2• 1 year ago

Expression of endearment in this case.

pyb2• 1 year ago

"... Since I am new to the research community, I made an effort" This means they've probably submitted a paper too.

epolanski3• 1 year ago

It states it's a research, not a product company.

diggan2• 1 year ago

To be fair, OpenAI is also a "research lab" rather than "product company" and they still sell products for $200/month, not sure the distinction matters in practice much today as long as the entity is incorporated somehow.

pyb2• 1 year ago

That's what I said

xiphias2• 1 year ago

A lot of the problems John mentioned (camera jpeg, latency, real time decisions) have been worked on by comma.ai for many years. He could have just used their stack and build on it the general learning parts that comma is not focusing on.

Flemlo• 1 year ago

And plenty of other people.

It's still a lot better to really learn and discover it yourself to really get it.

Also it's hard to determine how much time someone spent on particular topic.

prosunpraiser• 1 year ago

Reuse is not always necessary - sometimes things are just done for fun and exploration, not for appeasing thirsty VCs and grabbing that market share.

blitzar• 1 year ago

Reinventing the exact same thing and shouting from the rooftops about it is exactly how you appease thirsty VCs and grab that market share.

Cthulhu_2• 1 year ago

This is Carmack, who builds new things. He built one of the first 3D game engines based on the hard math.

WatchDog• 1 year ago

Carmack himself has done a lot of work on end to end latency, during his time at oculus.

brador• 1 year ago

I feel top level AI creation is beyond his skill set.

He’s a AAA software engineer but the prerequisites to build out cutting edge AI require deep formal math that is beyond his education and years at this point.

Nothing to stop him playing around with AI models though.

novosel• 1 year ago

There is no deep formal math in AI. It is a game of numbers.

All deep formal math is a boundary to a thing.

johnb23115• 1 year ago

The formal math takes a few months to learn. He is more than smart enough to figure that out.

kriro• 1 year ago

I think you overestimate the level of math required in AI and at the same time I think you underestimate the math skills of John. AI runs on GPUs, Quake 2 engine was one of the first to optimized for GPUs (OpenGL).

I'm pretty excited to see him in this domain. I think he'll focus on some DeepSeek style improvements.

horsellama• 1 year ago

this.

Having JC focusing on, say, writing a performant OSS CUDA replacement could be bigger than any of the last 20 announcements from openai/goggle/deepmind/etc

lyu07282• 1 year ago

This feels like an understatement. At the time, young me had the impression Carmack came first, then the industry created 3dfx/OpenGL to run his games better. I still have nothing but respect for his skills decades later.

cmpxchg8b3• 1 year ago

GLQuake was released 11 months before Quake 2.

Cthulhu_2• 1 year ago

What do you mean "beyond his skill set"? He effectively invented 3D gaming, which led to major leaps and investments into graphics cards which are now used for cryptocurrency and AI. He also did significant contributions into VR.

He's probably one of the most qualified people around.

whamlastxmas2• 1 year ago

He definitely did not invent 3D, he just pioneered it for PC gaming and got PC gaming up to speed to console gaming

jimbohn• 1 year ago

The math around machine learning is very manageable, and a lot of research in that area is throwing heuristics at a wall to see what sticks

threeseed2• 1 year ago

John Carmack:

So I asked Ilya, their chief scientist, for a reading list. This is my path, my way of doing things: give me a stack of all the stuff I need to know to actually be relevant in this space.

And he gave me a list of like 40 research papers and said, 'If you really learn all of these, you'll know 90% of what matters today! And I did. I plowed through all those things and it all started sorting out in my head.

foldr5• 1 year ago

What this misses is that research is a competitive endeavor. To succeed as a researcher you don’t just need to know the bare minimum required to do research in your field. You need to be able to do it better than most of the people you’re competing against. I know that HN as a collective has near-unlimited faith in Carmack’s abilities (and he is no doubt Very Smart). But he’s competing with other Very Smart people who have decades more experience of AI research.

To put it another way, the idea that John Carmack is going to do groundbreaking research in AI is roughly as plausible as the idea that Yann LeCun is going to make a successful AAA video game. Stranger things have happened, but I won’t be holding my breath.

abraxas• 1 year ago

I'd love to have a copy of that list. Just to see how much I've yet to absorb.

akomtu2• 1 year ago

AI creation is more like Alchemy than science, and breakthroughs come not from math background, but from intuition and a bit of math skills. Transformers behind the chatbots isn't a rocket science and were discovered almost by accident. The next breakthrough will come a similar way. I'd frankly bet on someone like Carmack than on some theoretical researcher who is churning out papers.

MrScruff3• 1 year ago

It's always a treat to watch a Carmack lecture or read anything he writes, and his notes here are no exception. He writes as an engineer, for engineers and documents all his thought processes and misteps in the exact detailed yet concise way you'd want a colleague to who was handing off some work.

One question I would have about the research direction is the emphasis on realtime. If I understand correctly he's doing online learning in realtime. Obviously makes for a cool demo and pulls on his optimisation background, and no doubt some great innovations will be required to make this work. But I guess the bitter lesson and recent history also tell us that some solutions may only emerge at compute levels beyond what is currently possible for realtime inference let alone learning. And the only example we have of entities solving Atari games is the human brain, of which we don't have a clear understanding of the compute capacity. In which case, why wouldn't it be better to focus purely on learning efficiency and relax the realtime requirement for now?

That's a genuine question by the way, definitely not an expert here and I'm sure there's a bunch of value to working within these constraints. I mean, jumping spiders solve reasonably complex problems with 100k neurons, so who knows.

suddenlybananas5• 1 year ago

It's because humans (and other animals) have enormous innate capacities and knowledge which makes learning new things much much simpler than if you start from scratch. It's not really because of human's computational capacity.

MrScruff3• 1 year ago

By innate do you mean evolved/instinctive? Surely even evolved behaviour must be expressed as brain function, and therefore would need a brain capable of handling that level of processing.

I don't think it's clear how much of a human brains function exists at birth though, I know it's theorised than even much of the sensory processing has to be learned.

xnx2• 1 year ago

enormous innate capacities and knowledge

Hundreds of millions of years of trial-and-error biological pre-training where survival/propagation is the reward function

Nopoint22• 1 year ago

There is just no reason to believe that we are born with some insanely big library of knowledge, and it sounds completely impossible. How would it be stored, and how would we even evolve it?

It just isn't needed. Just like you can find let's say kangaroos in the latent space of an image generator, so we learn abstract concepts and principles of how things work as a bonus of learning to process the senses.

Maybe a way to AGI could be figuring out how to combine a video generator with a LLM or something similar in a way that allows it to understand things intuitively, instead of doing just lots and lots of some statistical bullsit.

johnb23115• 1 year ago

From the notes:

"A reality check for people that think full embodied AGI is right around the corner is to ask your dancing humanoid robot to pick up a joystick and learn how to play an obscure video game."

throw_nbvc1234• 1 year ago

This sounds like a problem that could be solved around the corner with a caveat.

Games generally are solvable for AI because they have feedback loops and a clear success or failure criteria. If the "picking up a Joystick" part is the limiting factor, sure. But why would we want robots to use an interface (especially a modern controller) heavily optimized for human hands; that seems like the definition of a horseless carriage.

I'm sure if you compared a monkey and a dolphins performance using a joystick you'd get results that aren't really correlated with their intelligence. I would guess that if you gave robots an R2D2 like port to jack into and play a game, that problem could be solved relatively quickly.

ferguess_k3• 1 year ago

We don't really need AGI. We need better specialized AIs. Throw in a few specialized AIs and they will leave some impact in the society. That might not be that far away.

AndrewKemendo3• 1 year ago

This debate is exhausting because there's no coherent definition of AGI that people agree on.

I made a google form question for collecting AGI definitions cause I don't see anyone else doing it and I find it infinitely frustrating the range of definitions for this concept:

https://docs.google.com/forms/d/e/1FAIpQLScDF5_CMSjHZDDexHkc...

My concern is that people never get focused enough to care to define it - seems like the most likely case.

vonneumannstan• 1 year ago

Is this supposed to be a gotcha? We know these systems are typically trained using RL and they are exceedingly good at learning games...

nlitened• 1 year ago

the human brain, of which we don't have a clear understanding of the compute capacity

Neurons have finite (very low) speed of signal transfer, so just by measuring cognitive reaction time we can deduce upper bounds on how many _consecutive_ neuron connections are involved in reception, cognitive processing, and resulting reaction via muscles, even for very complex cognitive processes. And the number is just around 100 consecutive neurons involved one after another. So “the algorithm” could not be _that_ complex in the end (100x matmul+tanh?)

Granted, a lot of parallelism and feedback loops are involved, but overall it gives me (and many others) an impression that when the AGI algorithm is ever found, it’s “mini” version should be able to run on modest 2025 hardware in real time.

johnb23115• 1 year ago

(100x matmul+tanh?)

Biological neurons are way more complex than that. A single neuron has dentritic trees with subunits doing their own local computations. There are temporal dynamics in the firing sequences. There is so much more complexity in the biological networks. It's not comparable.

scajanus• 1 year ago

The granted is doing a lot of work there. In fact, if you imagine a computer being able to do similar tasks as human brain can in around 100 steps, it becomes clear that considering parallelism is absolutely critical.

kilpikaarna• 1 year ago

I'm sure there were offline rendering and 3D graphics workstation people saying the same about the comparatively crude work he was doing in the early 90s...

Obviously both Carmack and the rest of the world has changed since then, but it seems to me his main strength has always been in doing more with less (early id/Oculus, AA). When he's working in bigger orgs and/or with more established tech his output seems to suffer, at least in my view (possibly in his as well since he quit both Bethesda-id and Meta).

I don't know Carmack and can't claim to be anywhere close to his level, but as someone also mainly interested in realtime stuff I can imagine he also feels a slight disdain for the throw-more-compute-at-it approach of the current AI boom. I'm certainly glad he's not running around asking for investor money to train an LLM.

Best case scenario he teams up with some people who complement his skillset (akin to the game designers and artists at id back in the day) and comes up with a way to help bring some of the cutting edge to the masses, like with 3D graphics.

LarsDu885• 1 year ago

The thing about Carmack in the 90s... There was a lot of research going on around 3d graphics. Companies like SGI and Pixar were building specialized workstations for doing vector operations for 3d rendering. 3d was a thing. Game consoles with specialized 3d hardware would launch in 1994 with the Sega Saturn and the Sony Playstation (in Japan only for one year)

What Carmack did was basically get a 3d game running on existing COMMODITY hardware. The 386 chip that most people used for their excel spreadsheets did not do floating point operations well, so Carmack figured out how to do everything using integers.

May 1992 -> Wolfenstein 3d releases December 1993 -> Doom releases December 1994 -> Sony Playstation launches in Japan June 1996 -> Quake releases

So Wolfenstein and Doom were actually not really 3d games, but rather 2.5 games (you can't have rooms below other rooms). The first 3d game here is actually Quake which also eventually also got hardware acceleration support.

Carmack was the master of doing the seeminly impossible on super constrained hardware on virtually impossible timelines. If DOOM released in 1994 or 1995, would we still remember it in the same way?

Buttons8402• 1 year ago

his main strength has always been in doing more with less

Carmack builds his kingdom and then runs it well.

I makes me wonder how he would fare as an unknown Jr. developer with managers telling him "that's a neat idea, but for now we just need you to implement these Figma designs".

saejox• 1 year ago

What Carmack is doing is right. More people need to get away from training their models just with words. AI need the physicality.

NL807• 1 year ago

AI need the physicality.

which i found interesting, because i remember Carmack saying simulated environments are way forward and physical environments are too impractical for developing AI

SeanaldMcDnld• 1 year ago

Yeah in that way this demo seemed gimmicky like he acknowledged. He said in the past he would almost count people out if they weren’t training RL in a virtual environment. I agree, still happy he’s staying on the path of online continual learning though

programd• 1 year ago

Nvidia seems to think the same thing. Here's Jim Fan talking about a "physical Turing test" and how embodied AI is the way forward.

https://www.youtube.com/watch?v=_2NijXqBESI

He also talks needing large amounts of compute to run the virtual environments where you'll be training embodied AI. Very much worth watching.

johnb23115• 1 year ago

More people need to get away from training their models just with words.

They started doing that a couple of years ago. The frontier "language" models are natively multimodal, trained on audio, text, video, images. That is all in the same model, not separate models stitched together. The inputs are tokenized and mapped into a shared embedding space.

Gemini, GPT-4o, Grok 3, Claude 3, Llama 4. These are all multimodal, not just "language models".

timmg• 1 year ago

(If you know) how does that work?

Are the audio/video/images tokenized the same way as text and then fed in as a stream? Or is the training objective different than "predict next token"?

If the former, do you think there are limitations to "stream of tokens"? Or is that essentially how humans work? (Like I think of our input as many-dimensional. But maybe it is compressed to a stream of tokens in part of our perception layer.)

koolala3• 1 year ago

I wish he did this with VR environment instead like they mention at the start of the slides. A VR environment with a JPEG camera filter, physics sim, noise, robot simulation. If anyone could program that well its him.

Using real life robots is going to be a huge bottleneck for training hours no matter what they do.

qoez• 1 year ago

Interesting reply from an openai insider: https://x.com/unixpickle/status/1925795730150527191

andy_ppp3• 1 year ago

My bet is on Carmack.

speed_spread• 1 year ago

I suspect Carmack in the Dancehall with the BFG.

ramesh31• 1 year ago

What has he shipped in the last 20 years? Oculus is one thing, but that was firmly within his wheelhouse of graphics optimization. Abrash and co. handled the hardware side of things.

Carmack is a genius no doubt. But genius is the result of intense focused practice above and beyond anyone else in a particular area. Trying to extend that to other domains has been the downfall of so many others like him.

WithinReason2• 1 year ago

"Graphics Carmack" is a genius but that doesn't mean that "AI Carmack" is too.

cmpxchg8b3• 1 year ago

Appeal to authority is a logical fallacy. People often fall into the trap of thinking that because they are highly intelligent and an expert in one domain that this makes them an expert in one or more other domains. You see this all the time.

roflcopter693• 1 year ago

Funny, I was just commenting something similar here, see https://news.ycombinator.com/item?id=44071614

And I say this while most certainly not being as knowledgeable as this openai insider. So it even I can see this, then it's kinda bad, isn't it?

fmbb• 1 year ago

Can you explain which parts you think are bad and why?

johnb23115• 1 year ago

https://x.com/ID_AA_Carmack/status/1925973500327591979

zeroq3• 1 year ago

  >> "they will learn the same lesson I did"

Which is what? Don't trust Altman? x)

epr• 1 year ago

Actually no, it's not interesting at all. Vague dismissal of an outsider is a pretty standard response by insecure academic types. It could have been interesting and/or helpful to the conversation if they went into specifics or explained anything at all. Since none of that's provided, it's "OpenAI insider" vs John Carmack AND Richard Sutton. I know who I would bet on.

kadushka2• 1 year ago

He did go into specifics and explained his point. Or have you only read his first post?

ActivePattern• 1 year ago

It's a OpenAI researcher that's worked on some of their most successful projects, and I think the criticism in his X thread is very clear.

Systems that can learn to play Atari efficiently are exploiting the fact that the solutions to each game are simple to encode (compared to real world problems). Furthermore, you can nudge them towards those solutions using tricks that don't generalize to the real world.

lairv• 1 year ago

Alex Nichol worked on "Gotta Learn Fast" in 2018 which Carmack mentions in his talk, he also worked on foundational deep learning methods like CLIP, DDPM, GLIDE, etc. Reducing him to a "seething openai insider" seems a bit unfair

MattRix• 1 year ago

It’s not vague, did you only see the first tweet or the entire thread?

quadrature• 1 year ago

Do you have an X account, if you're not logged in you'll only see the first post in the thread.

handsclean• 1 year ago

It seems that you’ve only read the first part of the message. X sometimes aggressively truncates content with no indication it’s done so. I’m not sure this is complete, but I’ve recovered this much:

I read through these slides and felt like I was transported back to 2018.
Having been in this spot years ago, thinking about what John & team are thinking about, I can't help but feel like they will learn the same lesson I did the hard way.
The lesson: on a fundamental level, solutions to these games are low-dimensional. No matter how hard you hit them with from-scratch training, tiny models will work about as well as big ones. Why? Because there's just not that many bits to learn.
If there's not that many bits to learn, then researcher input becomes non-negligible.
"I found a trick that makes score go up!" -- yeah, you just hard-coded 100+ bits of information; a winning solution is probably only like 1000 bits. You see progress, but it's not the AI's.
In this simplified RL setting, you don't see anything close to general intelligence. The neural networks aren't even that important.
You won't see _real_ learning until you absorb a ton of bits into the model. The only way I really know to do this is with generative modeling.
A classic example: why is frame stacking just as good as RNNs? John mentioned this in his slides. Shouldn't a better, more general architecture work better?
YES, it should! But it doesn't, because these environments don't heavily encourage real intelligence.

cmiles74• 1 year ago

From a marketing perspective, this strikes me as a very predictable response.

jjulius3• 1 year ago

I appreciate how they don't tell us what lesson they learned.

dcre2• 1 year ago

It is a thread. You may have only seen the first tweet because Twitter is a user-hostile trash fire.

“The lesson: on a fundamental level, solutions to these games are low-dimensional. No matter how hard you hit them with from-scratch training, tiny models will work about as well as big ones. Why? Because there's just not that many bits to learn.”

https://unrollnow.com/status/1925795730150527191

alexey-salmin3• 1 year ago

Each of these games is low-dimensional and require not the "intelligence" but more like "reflexes", I tend to agree.

However making a system that can beat an unknown game does require generalization. If not real a intelligence (whatever that means) but at the level of say "a wolf".

Whether it can arise from RL alone is not certain, but it's there somewhere.

lancekey• 1 year ago

I think some replies here are reading the full twitter thread, while others (not logged in?) see only the first tweet. The first tweet alone does come off as a dismissal with no insight.

mannycalavera42• 1 year ago

indeed, this is pure walled garden sh*t

johnb23115• 1 year ago

Carmack replied to that https://x.com/ID_AA_Carmack/status/1925973500327591979

threeseed2• 1 year ago

Direct links:

https://docs.google.com/presentation/d/1GmGe9ref1nxEX_ekDuJX...

https://docs.google.com/document/d/1-Fqc6R6FdngRlxe9gi49PRvU...

roflcopter693• 1 year ago

Honestly, having gone through the slides, it's a bit painful to see Carmack "rediscover" stuff I've learned in a reinforcement learning lecture like ten years ago.

But don't get me wrong! Since this is a long-term research endeavor of his, I believe really starting from the basics is good for him and will empower him to bring something new to the table eventually.

I'm surprised though that he "only" came so far as of now. Maybe my slight idolization of Carmack made me kinda of blind to the fact that this kind of research is a mean beast after all and there is a reason that huuuuge research labs dump countless of man-decades into this kind of stuff with no guaranteed breakthroughs.

Cipater• 1 year ago

https://x.com/unixpickle/status/1925795730150527191

roflcopter693• 1 year ago

I was just going to answer https://news.ycombinator.com/item?id=44071595 who mentioned exactly the same tweet.

I'm nowhere as good at my craft as someone who works for openai, which the author of that tweet seems to be, but if even I can see this, then it's bad, isn't it?

andy_ppp3• 1 year ago

I still don’t think we have a clear enough idea of what a concept is to be able to think about AGI. And then being able to use concepts from one area to translate into another area, what is the process by which the brain combines and abstracts ideas into something new?

throw310822• 1 year ago

Known entities are recurring patterns (we give names to things that occur more than once, in the world or in our thoughts). Concepts are recurring thought patterns. Abstractions, relations, metaphors, are all ways of finding and transferring patterns from one domain to another.

andy_ppp3• 1 year ago

Sure, I understand what the terminology means but I don't believe we get to AGI without some ability to translate the learning of say using a mouse to using a trackpad in a simple way. Humans make these translations all the while, you know how to use a new room and the items in it automatically but I personally see the systems we have built are currently very brittle when they see new environments because they can't simplify everything to its fundamentals and then extrapolate back to more complex tasks. You could train a human on using an Android phone and give them an iPhone and they would do pretty well, if you did this with modern machine learning systems you will get an extremely high error rate. Or say you train an model on how to use a sword, I'm not convinced it would know how to use and ax or pair of crutches as a weapon.

Maybe it will turn out to simply be enough artificial neurons and everything works. But I don't believe that.

steveBK1236• 1 year ago

Another thought experiment - if OpenAI AGI was right around the corner, why are they wasting time/money/energy buying a product-less vanity hardware startup run by Ive?

Why not tackle robotics if anything. Or really just be the best AGI and everyone will be knocking on your door to license it in their hardware/software stacks, you will print infinite money.

soared• 1 year ago

Or have your AGI design products

steveBK1236• 1 year ago

All the more reason not to acquihire Ive for $6.5B, if true

tiahura• 1 year ago

Does AGI necessarily mean super-genius? Was KITT AGI? I'm not sure he could design products?

j_timberlake2• 1 year ago

This line of thought doesn't work, because any company approaching AGI might be actively trying to hide that information from regulators and the military. Being the 1st AGI company is actually pretty risky.

steveBK1236• 1 year ago

VCs are far too conditioned as hype men to hide the ball like that.

After generations of boastful over-promising, do you really believe THIS time they are underpromising?

foobiekr• 1 year ago

However their actual actions resemble companies who know AGI isn’t even on the horizon and moreso they are acting as exactly as if they believe the AI hype bubble is coming to an end and they need to dump the stuff into the public markets asap.

There really isn’t any other way to interpret OpenAI’s actions for the last few months.

Sure it could all be a feint to hide their amazing progress. Or it could be what it looks like.

Given the hype cycles of the last 20 years, I’m going with the second.

mindwok• 1 year ago

AGI is not enough. Seriously, imagine if they had an AGI in their ChatGPT interface. It’s not enough to do anything truly meaningful. It’s like a genius in the woods somewhere. For AGI to have an impact it needs to be everywhere.

steveBK1236• 1 year ago

Robotics to navigate the physical world seems more impactful than some pin/glasses product to provide a passive audio/visual interface to the chatbot doesn't seem so earth shattering either though.

What would you do with a 10x or 100x smarter Siri/Alexa? I still don't see my life changing.

Give me a robot that can legitimately do household errands like the dishes, laundry, etc.. now we are talking.

Jensson10• 1 year ago

Seriously, imagine if they had an AGI in their ChatGPT interface. It’s not enough to do anything truly meaningful

If they had that people would make agents with it and then it can do tons of truly meaningful things.

People try to make agents with the current one but its really difficult since its not AGI.

joshstrange• 1 year ago

Once AGI is accomplished I can’t imagine what else it would do but bootstrap itself up which, depending on compute, could scale quite far. OpenAI would only need to feed it compute for the most part.

I don’t think AGI is close, but once it happens it’s hard to imagine it not “escaping” (whenever we want to define that as).

typon• 1 year ago

They have a large robotics research group.

cma• 1 year ago

If Ive hype brings more investment valuation than it costs to acquire his company, and the time costs etc., it's a win for current investors regardless of if it is a good investment for the investors in the new round.

Flamentono22• 1 year ago

I find it interesting that he dismisses LLMs.

I would argue that if he wants to do AGI through RL, a LLM could be a perfect teacher or oracle.

After all i'm not walking around as a human and not having guidance. It should/could make RL a lot faster leveraging this.

My logical part / RL part does need the 'database'/fact part and my facts are trying to be as logical as possible but its just not.

akomtu2• 1 year ago

IMO, he's right. LLMs can't be AI because they don't create a model of observations to predict things, they just imitate observations based on their likeness to each other. When you play Quake, you create a simple model of the game physics and use that fast model to navigate the game. Your equivalent of LLM has a role too: it's a fuzzy detector of things you encounter in the game, sounds, images and symbols, but once detected, those things are fed into the fast and rigid physics model.

Flamentono22• 1 year ago

Yes but the LLM could tell the physics system that it is physics related.

Hey look you see a stone falling

mkoubaa• 1 year ago

It is worth trying out one of the many web based reaction time testers – you will find that you average over 160 milliseconds.

TIL JC has elite reflexes

2OEH8eoCRo0• 1 year ago

And nerves of steel

soci2• 1 year ago

Fundamentally, I believe in the importance of learning from a stream of interactive experience, as humans and animals do, which is quite different from the throw-everything-in-a-blender approach of pretraining an LLM. The blender approach can still be world-changingly valuable, but there are plenty of people advancing the state of the art there.

It's a shame that pretrained approach leads to such good enough result. The learning-from-experience, or what should be the "right" approach, will stagnate. I might be wrong, but it seems that aside from Carmack and a small team, "the world" is just not looking/investing on that side of the AI anymore.

However, I find it funny that Carmack is now researching for such approach. At the end of the day, he was the one who invented Portals, an algorithm to circumvent the need to reproduce the whole 3D world and therefore making 3D games computationally possible.

As a side note, I wonder what models are to come once we see the latest state of the art AI Video training technologies, in synch with the joystick movements from a real player. Maybe the results are so astonishing that even Carmack changes his mind on the subject.

EDIT::grammar & typos

anthonypasq• 1 year ago

i think you're overstating this. Yann LeCun (chief scientist at Meta) is firmly in this camp, and i think most companies trying to bring AI into the real world via some sort of robotics technology are thinking about and testing this approach.

soci2• 1 year ago

Thank you. You are right, most likely the ones working in the field haven't switched. But the truth is that big bucks are in pretrained technologies. As Carmack himself said, "there are plenty of people advancing the state of the art there".

koolala3• 1 year ago

Humans had 500 million * 8670 hours of Pre-Training.

I don't get why Carmack would say things should be learned in hours or upper bounds it to human lifetime.

flipnotyk• 1 year ago

I think there's a difference between "should be learned" and "should be able to be learned" here.

vlovich123• 1 year ago

I’m not necessarily convinced despite my human bias that it’s a superior mechanism. Humans work the way they do and learn the way they do in no small part because of biological limitations and a physical reality. It’s not clear that a virtual entity needs to face the same limitations, although clearly learning from feedback that’s not available to an AI is important. It is true though that humans are more energy efficient learners, but letting the AI experiment with the real world and get feedback that’s way may be the only missing piece rather than a problem with the “blender” approach.

tshaddox• 1 year ago

It's a shame that pretrained approach leads to such good enough result. The learning-from-experience, or what should be the "right" approach, will stagnate.

We’ll see. I’m skeptical that you’ll ever get novel theories like special and general relativity out of LLMs. For stuff like that I suspect you need the interactive learning approach, and perhaps more importantly, the ability to reject the current best theories and invent a replacement.

xnx2• 1 year ago

I'm surprised there's as much interest in looking at the structure/behavior of the biological brain, and less interest in considering the behavior of our vision system. Our brains are not CPUs, and our eyes are definitely not a grid of pixels with a fixed framerate.

lostmsu3• 1 year ago

I'm with OpenAI folks on this one: Atari just won't cut it for AGI. My layman intuition is that RL works well when rewards give good signal all the time. Until it does RL is basically random search. That's where massive data diversity like we have in text comes in handy.

In a game there might be a level with a door and a key, and because there's no reward for getting the key closer to the door, bridging this gap requires random search in a massive state space. But in the vast sea of scenarios that you can find in Common Crawl there's probably one, where you are 1 step from the key, and the key is 1 step from the door, so you get the reward signal from it without having to search an enormous state space.

You might say "but you have to search through the giant Common Crawl". Well yes, but while doing so you will get reward signal not just for the key and door problem, but for nearly every problem in the world.

The point is: pretraining teaches models to extract signal that can be used to explore solutions to hard search problems, and if you don't do that you are wasting your time enumerating giant state spaces.

lostmsu3• 1 year ago

You can actually easily test and overcome this by training a model simultaneously on a massive of text and Atari while carefully balancing learning rates between the two.

ploden2• 1 year ago

Why would AGI choose to be embodied? We talk about creating a superior intelligence and having it drive our cars and clean our homes. The scenario in Dan Simmons' Hyperion seems much more plausible: we invent AGI and it disappears into the cloud and largely ignores us.

fusionadvocate• 1 year ago

It doesn't need to be permanent. If humans could escape from their embodiment temporarily they would certainly do so. Being permanently bounded to a physical interface is definitely a disadvantage.

jwmcq• 1 year ago

Looking at other examples in sci-fi, perhaps to stop my body from pressing its off-switch?

ploden2• 1 year ago

With distributed backups in place, AIs will be much less worried about self-preservation than we are.

John Carmack talk at Upper Bound 2025

Related Stories