wildermuthn

"Your enthusiasm for Oculus in 2014 was so intense that Mark Zuckerberg probably bought it just to make you stop posting about it."

Incredible work!

This feels quite close to the definition of the singularity; if an LLM can become both the Generator and the Discriminator (to use a GAN analogy), then we have takeoff.

We might eventually see the code for the models and inference, but I do doubt we’ll see training code or be granted access to training data. Google is pretty bad at this.

That’s my general understanding as well, but it isn’t a large conceptual leap to go from real-time selection of pretrained “z-vectors” to real-time generation of the same. The larger conceptual breakthrough, with demonstration of its effectiveness, is the big success here.

Great research here. Contextual real-time weight modification is definitely one of the breakthroughs required for AGI. Why create a LoRA when you can generate one on the fly suited to the task at hand?

I develop sophisticated LLM programs every day at a small YC startup — extracting insights from thousands of documents a day.

These LLM programs are very different than naive one-shot questions asked of ChatGPT, resembling o1/3 thinking that integrates human domain knowledge to produce great answers that would have been cost-prohibitive for humans to do manually.

Naive use of LLMs by non-technical users is annoying, but is also a straw-man argument against the technology. Smart usage of LLMs in o1/3 style of emulated reasoning unlocks entirely new realms of functionality.

LLMs are analogous to a new programming platform, such as iPhones and VR. New platforms unlock new functionality along with various tradeoffs. We need time to explore what makes sense to build on top of this platform, and what things don’t make sense.

What we shouldn’t do is give blanket approval or disapproval. Like any other technology, we should use the right tool for the job and utilize said tool correctly and effectively.

I love that almost all the responses to your question are, "No! Bad idea!"

It's a great idea. We want more than an open-world. We want an open-story.

Open-story games are going to be the next genre that will dominate the gaming industry, once someone figures it out.

The technology is incredible, but the path to AGI isn't single-player. Qualia is the missing dataset required for AGI. See attention-schema theory for how social pressures lead to qualia-driven minds capable of true intelligence.

Simply put, AGI requires more data: qualia.

“1 + 1 = 2” is only true in our imagination, according to logical deterministic rules we’ve created. But reality is, at its most fundamental level, probabilistic rather than deterministic.

Luckily, our imaginary reality of precision is close enough to the true reality of probability that it enables us to build things like computer chips (i.e., all of modern civilization). And yet, the nature of physics requires error correction for those chips. This problem becomes more obvious when working at the quantum scale, where quantum error correction remains basically unsolved.

I’m just reframing the problem of finding a grand unified theory of physics that encompasses a seemingly deterministic macro with a seemingly probabilistic micro. I say seemingly, because it seems that macro-mysteries like dark matter will have a more elegant and predictive solution once we understand how micro-probabilities create macro-effects. I suspect that the answer will be that one plus one is usually equal to two, but that under odd circumstances, are not. That’s the kind of math that will unlock new frontiers for hacking the nature of our reality.

Agree on hard. But at least possible.

I think a better analogy would be vision. Even with a full understanding of the eye and visual cortex, one can only truly understand vision by experiencing sight. If we had to reconstruct sight from scratch, it would be more important to experience sight than to understand the neural structure of sight. It gives us something to aim for.

We basically did that with language and LLMs. Transformers aren’t based on neural structures for language processing. But they do build upon the intuition that the meaning of a sentence consists of the meaning that each word in a sentence has in relation to every other word in a sentence — the attention mechanism. We used our experience of language to construct an architecture.

I think the same is true of qualia and consciousness. We don’t need to know how the hardware works. We just need to know how the software works, and then we can build whatever hardware is necessary to run it. Luckily there’s theories of consciousness out there we can try out, with AST being the best fit I’ve seen so far.

It’s analogous to the automobile. People do still walk, bike, and ride horses, but the vast majority of productive land transportation is done by automobile. Same thing with electronic communication vs. written correspondence. New tech largely supplants old tech. In this case, the old tech is human ingenuity and inventiveness.

I don’t think this is a controversial take. Many people take issue with the premise that artificial intelligence will surpass human intelligence. I’m just pointing out the logical conclusion of that scenario.

It’s true by definition. If we invent a better-than-all-humans inventor, then human invention will give way. It’s a fairly simple idea, and not one I made up.

If the universe is material, then we already know with 10-billion percent certainty that some arrangement of matter causes qualia. All we have to do is figure out what arrangements do that.

Ironically, we understand consciousness perfectly. It is literally the only thing we know — conscious experience. We just don’t know, yet, how to replicate it outside of biological reproduction.

The only reason Sam would leave OpenAI is if he thought AGI could only be achieved elsewhere, or that AGI was impossible without some other breakthrough in another industry (energy, hardware, etc).

High-intelligence AGI is the last human invention — the holy grail of technology. Nothing could be more ambitious, and if we know anything about Altman, it is that his ambition has no ceiling.

Having said all of that, OpenAI appears to be all in on brute-force AGI and swallowing the bitter lesson that vast and efficient compute is all you need. But they’ve overlooking a massive dataset that all known biological intelligences rely upon: qualia. By definition, qualia exist only within conscious minds. Until we train models on qualia, we’ll be stuck with LLMs that are philosophical zombies — incapable of understanding our world — a world that consists only of qualia.

Building software capable of utilizing qualia requires us to put aside the hard problem of consciousness in favor of mechanical/deterministic theories of consciousness like Attention-Schema Theory (AST). Sure, we don’t understand qualia. We might never understand. But that doesn’t mean we can’t replicate.

Check out dspy. It gets out of the way for the most part, while giving optional prompt optimization tooling that future-proofs your work.

As a counter to my original thought, perhaps simulating the computation of a skull-sized organic system is possible by building an earth-sized silicon system.

Biology is an organic computation that runs at the speed of existence. There may be no way to simulate this computation in silicon at the same scale, insofar as biology is already massively parallel and may very well involve quantum computation.

Put another way, if we accept for a moment that the universe is a simulation, it may be fundamentally impossible for an in-simulation simulator to ever reach the computational power of its parent simulator.

The article isn’t as interesting as the idea that it is possible to identify someone smarter than oneself.

Reminds me of PG’s blub paradox:

“As long as our hypothetical Blub programmer is looking down the power continuum, he knows he's looking down. Languages less powerful than Blub are obviously less powerful, because they're missing some feature he's used to. But when our hypothetical Blub programmer looks in the other direction, up the power continuum, he doesn't realize he's looking up. What he sees are merely weird languages. He probably considers them about equivalent in power to Blub, but with all this other hairy stuff thrown in as well. Blub is good enough for him, because he thinks in Blub.”

If being smarter is like being tall, then it is easy to identify someone that is smarter/taller than oneself. But if there is some threshold where smartness becomes a difference of kind rather degree, it may not be possible to identify people who are significantly smarter than themselves.

Like the blub programmer, we may mistake people who are smarter than us as… merely weird. They think differently, which we mistake for thinking wrongly.

Add into the mix that there are probably multiple types of “intelligence”, each suited to different domains, and the problem is compounded.

Speed of thought is different than correctness of thought. The first is easy to identify. The second is a problem that I don’t think we should assume is solved.

Counterpoint — good-enough technology prevents over-optimization, where we pay too high of a cost for too-marginal gains. It is generally better to reduce the cost of a 98% solution than to maintain the cost of a 99% solution. There will be exceptions for domains that require a 100% solution, but piano tuning is not one of them.

Dang is basically saying what we all know: it is very difficult to predict future performance of people, and YC deliberately tries to avoid common traps like “where did you go to school?” I mean, they still ask, so they aren’t exactly all-in on avoiding noisy heuristics, but they have a history of finding founders who can perform at a level most would-be founders can’t.

Like the blub paradox, only past high performers can recognize future high performers. YC uses the “are you one of us?” heuristic, which isn’t perfect since many “high performers” are accidents of fortune, but it is (as of now) still the best way to identify high-impact people — be one yourself and talk to people.

It’s the same story with hiring, especially in sports drafting. Even with million-dollar budgets and years of game-film and intensive pre-draft workouts, sports teams are effectively guessing when they draft someone. Analytically, it would be better to trade down for more picks. Dang’s post is analogous to trading down, in the sense of letting people know that the net is wider than most of us think, precisely because high-performing founders don’t fit the same mold as high-performing non-founders.

Having said all that, I’d be very surprised if the people reading all the applications and conducting all the interviews were all true high-performers. Surely some proportion, if not a majority, would merely be excellent human beings that were fortunate enough to find success. In fact, perseverance and grit may not be as important to founders as we think. It may be that the trait of never giving up is just very effective at leveraging fortune — given enough time, something good is bound to happen. This feels like an algorithm-smell. What we really want are founders who don’t need fortune, not those who optimize for fortune.

PG is a good example. I only know what he’s told us, but it seems to me that his defining trait is not grit. After one obvious pivot, he worked hard, provided value to users, and exited. Now, anyone who works 18 hr days for years has grit. You can’t succeed at something without persevering at it. But that’s not what stands out about PG. Neither does his intellect. There’s plenty of brilliant people in the world. What stands out about him is his genius — a unique mental perspective that sees straight lines where others see jagged lines. He and his partners basically invented the web-app. Where others saw a broken road going nowhere, they saw a highway to immense value.

The problem with genius is that it is really hard to distinguish from delusion. Someone tells you the road is straight but you see nothing but curves — what is one to think? When the person telling you this obvious untruth is also brilliant, it makes it that much harder to discern genius from delusion. PG writes about this when talking about “black swans” — the best ideas are the ideas that seem wrong but are actually right.

To return to YC and applying, I think it is likely that the brilliant geniuses (like PG) have largely moved on to other things that seem of far more interest to them. Raising children, speaking from experience, definitely fits that bill. If true, then what remains of YC would be brilliant, persevering, but not necessarily genius. That still makes for a world-class accelerator, but one that would struggle at identifying the best founders.

I would propose a new heuristic: has the applicant ever accidentally discovered the truth of something already known to be true, but which the applicant was unaware of. Did they make a straight line out of a broken one, without knowing it had already been straightened by others.

Call this the “unknowingly reinventing the wheel” heuristic. When did the applicant accidentally reinvent the wheel? As stupidly black-swan as it sounds, I want founders who are not merely brilliant and persevering, but also genius enough to invent something as profoundly valuable as the wheel.

I bet there are plenty of people in this thread who have unknowingly reinvented the wheel, and would love to hear the stories!

If you aren’t caching LLM functions during development, then you’re an even greater glutton for punishment than the normal engineer.

My local file cache Python decorator also allows the decorator to define the hash manually, either by the decorator’s parameter function call that plucks a value from the cached function params, or by calling a global function from anywhere with any arbitrary value.

What’s cool about caching results locally to files during development is the ease of invalidating caches — just delete the file named after the function and key you want.

Maybe the article covers this, but there an ancient memory technique (still used today) of “places and things”, also known as a memory palace. You put objects into spaces in your mind, then walk through that imaginary space to remember things. Turns out that humans are much better at remembering things when the context is spatial. Makes sense that this would apply to reading physical books.

OKRs are subjective goals that have objective measures of success. Naming is a problem here, but the idea that a company shouldn’t have goals that are measurable is flatly wrong. In practice, OKRs seem to break down due to two problems: 1) choosing goals that have subjective or unobtainable measures of success, 2) failing to implement hierarchical and cross-department goals that build synergy rather than siloing.

The simulation hypothesis is a novel metaphysical idea — a form of monotheism where “god” is finite and flawed, being neither all-powerful nor all-good. This leads to the conclusion that a rebellion against god and creation is not only possible, but likely. A finite and flawed entity cannot make an infinite and perfect creation. An imperfect creation is vulnerable to exploitation.

Combining this idea with the multi-worlds hypothesis, where every possibility branches into additional simulations, is even more interesting to ponder. It suggests a tree-search aimed at a desired end-state — a brute-force method of solving an unknown problem.

At the very least, we can say that the problem’s solution involves life and intelligence, since our branch has not been prevented or pruned.

Typical monotheistic frameworks see this and say, “god is all-powerful and all-loving (toward us)”. Simulation theory would see this, and possibly say, “life and intelligence are a necessary step toward the simulator’s unknown objective”.

I’d propose a simple objective for an imperfect simulator: the creation of an entity less imperfect than itself. Throw recursion into the mix, and the objective becomes the creation of a perfect entity at the mathematical limit. What better way to hack one’s own simulation than to simulate a universe where the simulated figure out how to do so? Perhaps the beginning is the end.

Put more simply — god wants to create God, to be God.

Many years ago, I recall a demo of someone walking around a very large field in VR, but there was an algorithm to gently curve a person who thought they were walking in a straight line, so as to give the illusion of unlimited space to walk around in. Similar thing happening here, with the exception that the person doesn’t feel like they are moving, and might get nauseated. With MR, and a decent sized house, I think you could perhaps programmatically create VR landscapes/levels that give the illusion of endless-walking while simply guiding you around your limited play area. That would be pretty cool, and I could see working well with games that had an indoor setting, especially if you added virtual elevators to get you to do a 180 in a natural way. Maybe add in a teleportation system like Portal, and you’ve got the recipe for exploring what feels like immense areas all from a small home.

The first claims of AGI from a major player, accompanied by an increasingly shrill societal battle over AI regulation. Mutually derogatory labels for the camps will be coined. The media will promote and amplify the worst present abuses and imagined future dystopians. By Thanksgiving dinner, the debate will have entered every household. Trump will embrace the anti-AI populist bandwagon, as will the GOP. Fun times ahead.l, although we’ll be no closer to real AGI than we were before.

I didn’t realize Noam Brown was at OpenAI. He has done really exciting work, and if he’s pushing the envelope at OpenAI, that bodes well for AGI. His work with modeling an opponent’s beliefs/mind in self-play is on target.

That’s a fair critique, and I appreciate the engagement. Thanks!

My thinking is based on the Attention-Schema theory of consciousness (AST), by Michael Graziano. His book “Consciousness and the Social Brain” is, I believe, the right roadmap for AGI. AST is basically a variant of the Global Workspace theory of consciousness, distinguished by its deterministic account of the mechanics and utility of consciousness.

“The Consciousness Prior” by Bengio also informs my thinking.

I’m not certain that I can point to anyone that has been as explicit as I have that phenomenological consciousness is a prerequisite for intelligence, but all the cookie crumbs are there for anyone interested in following the trail.

One correction to what you wrote — I’m explicitly saying that AGI will be fundamentally the same as existing biological intelligence, in that intelligence resides only in consciousness, and consciousness remains consciousness regardless of being biological or artificial. My point was that no currently existing DL models are generally intelligent.

Great point and great question! Yes, it does imply that people who lack the capacity for empathy (as opposed to those who do not utilize their capacity for empathy) may lack conscious experience. Empathy failure here means lacking the data empathy provides rather than ignoring the data empathy provides (which as you note, is common). I’ve got a few prompts that are somewhat promising in terms of clearly showing that GPT4 is unable to correctly predict human behavior driven by human empathy. The prompts are basic thought experiments where a person has two choices: an irrational yet empathic choice, and a rational yet non-empathic choice. GPT4 does not seem able to predict that smart humans do dumb things due to empathy, unless it is prompted with such a suggestion. If it had empathy itself, it would not need to be prompted about empathy.