stalfie

Oh I wasn't talking about humans. I should probably have pointed that out. There's some scenarios like catastrophic crop failure and so on that might lead to that, but frankly I doubt it.

I was referring more to everything else. Corals, certain insects, polar bears, salamanders and so on. With some quick googling it appears that the "Bramble Cay Melomys" is the first species so far to be declared extinct because of climate change, but the number that seems to be thrown around is that an average of 18% of terrestrial life will be critically endangered by 2100 in a scenario with 2 degrees warming. I can't be bothered with figuring out what degree of academic consensus there is around that number, but I think it's reasonable to assume that there's at least some kind of consensus about "more than 1%".

Sure, here are some citations:

https://arxiv.org/abs/2401.03910?utm_source=chatgpt.com https://www.frontiersin.org/journals/psychology/articles/10.... https://arxiv.org/pdf/2408.04666 https://arxiv.org/abs/2402.00901 https://ar5iv.labs.arxiv.org/html/2407.11015 https://ar5iv.labs.arxiv.org/html/2202.05262 https://www.sciencedirect.com/science/article/abs/pii/S13646...

My point was not to argue one point or another about LLM intelligence, but to push against the notion that you have to be "delusional" to even argue that it is possible that LLMs can qualify as intelligent (although the op prefaced it with "actually"). That's mainly what irked me about the original comment, the arrogance of dismissing everyone even having the discussion as insane, as if there's no legitimate argument to be made.

And this was mainly the point I was arguing. However since we're on the topic, I also happen to think it's intellectually lazy to dismiss the Anthropic interpretability teams work as "delusional" simply because they have a conflict of interest. Of course, that is not an irrelevant fact, but much of their original work has since been replicated by independent entites (eg. https://arxiv.org/abs/2510.01246). Until they publish something that turns out to be fraudulent, I think it's reasonable to consider Anthropic's paid people a very relevant, and in fact excellent part of interpretability discourse.

Dismissing all opinions where there is a perceived conflict of interest is a pleasant cognitive bias to have, but reality is often more nuanced than that.

"Climate conspiracy"? Like you, mean, the conspiracy of climate scientists to publish facts to the best of their understanding?

I don't know what exact strawman you're arguing against, although I'm sure you can always find some idiots saying something like what you say. But scientific consensus has long been that it will lead to increasingly extreme weather and mass extinction, which we seem to be on track for. Of course, we can't know for sure what the consequences are untill we do the experiment, which in this case means potentially destroying large sections of the biosphere and living with increasingly destructive weather patterns. Surely that risk is worth at least legislating that hyperscalers need to spend some of their billions on solar panels?

No one knows how LLMs work. We know how the architecture works, but almost nothing about why. Saying "statistical next token prediction" tells you about as much about LLMs as saying "action potential thresholds" tells you about the brain. A true fact that explains very little.

And I'm sorry, but you're not up to date about interpretability literature, or for that matter philosophical discourse, if you think you have to be delusional to question whether LLMs are "intelligent", whatever you define that word to mean. The [latest publication](https://www.anthropic.com/research/global-workspace) from anthropics interpretability team purports that they see structures in Claude akin to those we think are associated with human consciousnesses experience in the brain. Are you going to dismiss the whole team as not being of "sound mind"?

Intelligence is a word with a somewhat unclear meaning to begin with, but you have move the goalposts pretty damn far to exclude LLMs at this point. They are certainly still lacking in some regards, but whether that disqualifies them for intelligence is very much a matter of debate.

I was referring more to the fact that no one predicted that next token predicting Transformers would go so far. Not about "AI" in general.

No one imagined LLMs in their current format, it was simply a result of discovering that scaling compute and tokens produced better and better results with the Transformer architecture. The inventors of the Transformer architecture were working on better translation, and probably did not imagine that their architecture would lead to modern LLMs.

Imagining something in advance is not necessary at all for scientific advancement. This is particularily true in AI, and no one expects to imagine what superintelligence is until after it is created. You set up your datasets, your architecture tweaks, and measure the results on some set of benchmarks. There never was a blueprint, no plan beyond the experiment itself. We're not even close to understanding the things we have already created, and yet we created them. So why expect anything else for the next step?

10 years is a long time. 10 years ago the Transformer architecture didn't exist. I would call it moderately unlikely at best. At the very least, I would say it's likely that development will require an entirely different skillet 10 years from now.

There's at least one benchmark that attempts to measure this, but it has been running for a year plus so it's quite infrequently updated now.

https://fiction.live/stories/Fiction-liveBench-Mar-25-2025/o...

Ironically, the article points out that the original authors publisher actually put out two DMCA notices to google last year, apparently with no effect.

I guess DMCA takedowns are only for the big fish fighting the good fight against car pirates.

Mmmm, not sure I agree with this, although this is a topic where we would have to do a lot of groundwork to formulate our positions precisely in order to ensure we're actually discussing the same thing. My counterargument is that verified mathematics does exist. A lot of mathematical models of physics predicted the existence of stuff that experiments later verified, the higgs boson, antimatter and gravitational waves comes to mind. Terrence Tao did in fact make MR machines go faster simply by finding better maths, and the tumors those machines see can be cut out and touched.

Yes, there are mathematical concepts that seem to exist purely in the realm of mathematics, but maths often touches reality in a consistent way that reflect experimental results. This seems to imply that there is more to mathematics than just internal consistency. And the parts that do not correspond to any observation right now, might just reach out and touch reality in the future. It is possible to create logically consistent systems that have nothing to do with reality, but this is not the mathematics that most mathematicians are thinking about.

Observation is the final arbiter of fact. Maybe we don't have a general system to verify ALL facts, but many facts are 100% verifiable, although not most of them. "Beyond reasonable doubt" is of course the highest level of fact as far as the scientific method is concerned, but some facts are so far beyond reasonable doubt that you might as well just call them true. In the average living human body, there is a particular clump of tissues that consistently corresponds a concept most experts would describe as a "heart", and it does in fact pump blood. True fact.

Once again, Russia turns out to be the reason we can't have nice things. War truly is a waste for everyone involved. Now that Russia is also helping North Korea to launch satellites (one so far), expect everything to get worse in the future.

I give it 2-10 years before one of the two threatens an imagined adversary with detonating a nuke in orbit, with the explicit intent of causing Kessler syndrome.

Well, I'd argue that this depends on the field you're investigating. Sometimes you have a way to identify objective reality and sometimes you don't. In mathematics the majority of the field is verifiable in this way. Coding a bit less as it's intersubjective, as and the ideal methodology is subject to taste.

But even in muddy fields of reality like medicine, there are objective facts to be found. When someone comes into an ER with chest pain, you often find a true, undeniable reason for why that is happening. If their lung has collapsed, a coronary artery is clogged or the aortic artery is dissecting, even if you don't find that out it tends to be clear in retrospect. The area of reality that becomes muddy is when use proxy signals to try to figure out who gets promoted to expensive/harmful examinations we can make final conclusions from, or the cases that don't fit cleanly into one bucket or the other. But very often, the gold standard truly is golden.

Of course, many realms of reality cannot be verified in this way. But I'd argue that there are quite a few that can.

I guess so. Just to be clear, I was talking about post-training methods for reasoning models here, not pre-training. I think "model as a judge" should actually do okay as a "sentiment analysis" style reward for expressing uncertainty. So if none of the thousands of reasoning traces you generate reach the validated answer, you run a judge to rate uncertainty and put those reasoning traces back into the training pool.

But I guess my logic breaks down here a bit, because if there is such a thing as a validated answer, then the correct answer is in fact never uncertainty. The correct answer is to continue post training until the model gets it right. So perhaps the real answer is to create RLVR tasks where the valid answer is "I don't know" and nothing else, like this benchmark does. Or maybe that doesn't work either, no matter how many you create.

I feel as though there is some kind of philosophical lesson to be had from how hard hallucinations are to get rid of. Maybe, similarly to humans, successful models are often "arrogant" in a sense. Perhaps you just never solve an Erdös problem without some degree of self deception that it's possible for you to do so. In this line of thinking, greatness in humans is actually not related to humility, but just being so good that you actually get things right when you try. Expressing humility is of course something great people tend to do, but I'm referring to what happens under the hood.

If you squint a bit, that's kinda the trend with models. The useful ones are not that much less likely to hallucinate, they are just good enough that they tend to get it right. This comparison is of course probably not even remotely correct, but at least it's fun to anthropomorphize a bit.

One thing I wonder about hallucinations, is that it seems on the surface that it is an easy problem for RLVR to target. Since you're already generating enormous amounts of reasoning traces which are verified by correct answers, just have "don't know" as an option as a valid answer, and on problems where none of the thousands of reasoning traces led to a correct answer, just promote the traces that led to the "don't know" answer as training data. Essentially teaching the model that "I don't know" is a valid answer.

Sam Altman himself had a blog post about this a while ago that seemed to suggest this thought, so I guess it's obvious to everyone. But if that is so I assume it's just not as easy in practice.

The criticism is also similar to those faced by Theranos. Survivorship bias is always a factor when looking backwards.

Excluding the cost of X-ray/CT/MRI machines, operating them, getting people to them and through them, sometimes injecting contrast, and sometimes dealing with side effects of said contrast, radiologists, I think. You can scale all of the above except interpretation. AI is the natural next thought for how to scale that part, but it's been thought that this would happen any moment for over a decade.

It's worse then that unfortunately. Even when invasive tests are positive, and we think we caught a cancer early, we know from population statistics that the reality is that often nothing would have happened. So we don't even truly know how to tell a cancer that will kill you from one won't. And we don't really know what it is that we don't know.

This is more true for some cancers then other though. Prostate, breast, and maybe melanoma are the worst in this regard. This is why prostate and breast cancer screening programmes are controversial, although the needle is swinging towards them being more useful as surgeries and treatments get better. Some other cancers like pancreatic cancer will always kill you eventually, so it's always good to catch them. It's a nuanced problem.

This whole issue is called "overdiagnosis", and personally I used to be obsessed with it. Being aware of it mostly caused a lot of hand wringing and grief, it's just easier to believe that every cancer you catch is a good thing. However, one of the broader issues is that we will never know what we don't know if we don't look. So there exists another perspective that all the suffering caused by overdiagnosis will eventually pay off in the long term. This is the "collect all the data for science/AI" perspective, and I've personally tentatively adopted it myself, although perhaps that's just because it's nicer to believe that you do some good even when you do harm. I think it's more likely that [novel cancer therapies](https://www.nature.com/articles/s41586-026-10738-7) will solve the "harm" part of treatment before we solve overdiagnosis.

The reality is that important breakthroughs are often entirely unrelated to the data for you are collecting, and even worse that possibly helpful data is locked away due to regulation and never used. This is kinda why I've come to make some kind of peace with private clinics scamming people with whole body MRIs, as I'm sure they're secretly selling the data which might lead to some good. However, they would probably do even more good if they didn't exist so they didn't jack up the prices for MRI machines by inflating demand. The marketing they do is the most morally reprehensible part of the whole deal, as it's usually just lying and creating health anxiety for profit. The fact that midjourney here is marketing themselves in this direction is giving me some serious Theranos vibes. Quick and cheap MRI equivalents would be really useful in the clinic, and it would have to spend a few decades there to prove it is useful before moving on to the "spa" stage. That they are trying to market a render of an idea directly to the wellness crowd firmly puts this in the "scam" folder for me. The fact that midjourney is mostly irrelevant now also fits well with this, making it likely that this is either a marketing stunt or a desperate pivot to get funded. Hopefully there are not that many suckers who will put their VC money down on this loosing bet.

Well, the Fable guardrails breaks this argument, as when you get booted down to Opus 4.8 it still happily responds (as does most other models after a "I'm not a doctor but..." hedge).

So you get to press the big red button anyways, you just get downgraded to stupider advice, increasing the risk of catastrophy and lawsuits.

But the whole point is that the models are so good now anyways that any stupidity will most likely be from the user misunderstanding the output...which already happens all the time with humans.

I got so frustrated with Fable refusing to make any medical diagnoses, which is the most recent iteration of a longer trend that has bothered me for years, that I made a blog to express how annoyed I am.

Posting it here in case anyone is interested in a non-coding perspective on model refusal behaviour.

And because I really want someone at anthropic to read this so I can test the goddamn thing.

Honestly, I have yet to see any evidence of data leak from private sources. I think one of the better example is "simple-bench", which at least used to be a low-key benchmark that I would assume would have been saturated quickly if the labs were secretly scooping up data from API requests. Yet it's been years and it has yet to be saturated.

It's easy to catch a data leak if you have private data. You know what the model is supposed to not know, and you can just ask to see if it does. Yet I have not seen or heard of a single case of this being documented. As far as I can tell the labs do in fact respect the request to opt out of training.

Update in case anyone reads this comment ever again.

I have found that I trigger the guardrails any time I ask for medical Q&A as a doctor, be it ECGs, case reports, and so on. But if I phrase it like I'm the patient ("help me interpret this ECG my doctor gave me"), then I usually get one or two answers out before hitting the guardrails.

It seems like the direction that triggers it is anything in the direction of making a diagnosis. As an MD, the fact that the paradigm of "LLMs shouldn't diagnose" has gone this far fills me with despair. The latest generation of LLMs are in fact truly excellent at diagnosis, and I know many of my colleagues, particularly those in primary care, regularly use LLMs to brainstorm. There is nothing wrong whatsoever with LLMs making diagnosis, the only caveat is that they have to be correct. This is the terrifying reality that MDs face every day and I get that the labs are hesitant about it, but as the current literature points to LLMs in fact being mostly superior to most doctors, ablating this capability is starting to get increasingly unethical. And frankly, it is also kind of insulting, both to MDs and patients, as it echoes paternalistic attitudes about medicine the field has been working for decades to move away from. Now those misguided attitudes have somehow become institutionalized as the dominant paradigm of "alignment". The nightmare scenario is that I have to be a "trusted" user in order to use the model for medicine. This gatekeeping of medical advice is profoundly unethical with regards to everyone that does not have immediate access to an MD.

And the whole thing makes even less sense when triggering the guardrails leads to a downgrade of the response by defaulting to Opus. How exactly is giving WORSE medical advice in any way related to safety and alignment? If anyone at anthropic ever reads this, please, please just abandon the paradigm that refusing to make diagnoses is in any way equivalent to alignment, it is profoundly misguided.

Tried to benchmark ECG interpretation capabilities, and I hit the guardrails no matter what I do.

Incredibly frustrating that medical performance seems to be a victim of "biological risk" guardrails.

This article describes how Transformers work, but not really how LLMs work. Explaining the underlying architecture gives you about as much insight into how a modern LLM behaves as an breakdown of neuronal biochemistry and a few pathways does for the brain. Meaning, almost no insight at all.

Last time I checked thoroughly (roughly two years ago), AI (in the form of small ML models) mostly outperformed radiologists in areas where the gold standard is "one level" above imagining wise. By that I mean that you train a model to detect on an X-ray what would normally need a CT. Or train it to see on a non-contrast CT what would normally need contrast or an MRI, or biopsy, and so on.

Essentially the cutting edge reaches up to 99% of human performance on the task it is trained, which is good enough for triage but not for a final diagnosis. However, magic sometimes happens when you train a model to detect something, which you already know is there, on an examination that is cheaper, faster or less invasive than the human"gold standard". Conveniently, this dataset exists since it's common to first do a cheap examination like an X-ray, and then escalate if nothing is found (or if something is found that you want to see better, or a number of other possibilities).

Examples of AI outperforming humans like this includes AI detecting sacral fractures on x-rays better than radiologists (who normally take a CT to conclusively exclude it), detecting potential precursors to pancreatic cancer on non-contrast CTs (where contrast or an MRI is usually required) and detecting an occluded coronary artery on an ECG without the archetypical "ST-elevation changes".

See the link below for references: https://pmc.ncbi.nlm.nih.gov/articles/PMC9478257/ https://www.nature.com/articles/s41591-023-02640-w https://rebelem.com/a-winning-hand-in-cardiology/

So AI, as a general rule, doesn't usually match or exceed the upper bound of the "gold standard" medical performance. But it tends to carry the quality of the upper bound downwards towards the faster, less expensive and invasive methods. In some cases, like in the case of EKGs, that's huge. In some cases it saves time, in some cases it decreases miss rates from tired radiologists or triages their review feed. And in some cases it's not very useful.

LLMs doesn't come close to specialized radiology models at the moment, because LLMs are more about applying knowledge than creating new correlations. Of course that's also hugely useful, but that's a bit of a different topic to unpack.

The counterpoint is that every company ever has based themselves on human effort they never paid for (usually). The entire scientific endeavour for example. Standing on the shoulders of giants and so on.

Ok, fair point about the lizardbrain jealousy, the choice of wording there was needlessly antagonistic. In my defense, even though that point might seem reductive, I don't mean it to be. I'd say the only reason words like"fair", even exists is because of that basic emotional response. We dress it up in higher order concepts, but I genuinely think everything about fair distribution boils down to human emotional responses that in different contexts are given names such as "greed" or "jealousy". And the tribal environment IS the environment our brains are adapted to. I don't think it's necessarily reductive to point out the very foundation of where the impulses from our cognitive medium is coming from. It's the basis of pretty much everything about human society. From a sibling being pissed because he got one less slice of cake, to intellectual property law. It all boils down to the same emotional programming.

And I just want to be absolutely clear on one thing, I am in no way shape or form mocking science. I am part of academia, and I see science as the one most valuable thing humanity is doing. And I am not convinced at all that the capitalist way things are organized are the best overall, but this is mostly because it incentivises locking down knowledge behind intellectual property. Personally, I am for radical openness of knowledge, with no pay walls, and that the reward structures for producing knowledge should be separate from how that knowledge is being used. Even the publication system is too greed-based in my opinion, if it was up to me, every lab would live stream their work and publish every thought and idea as it happens (with the possible exception of biosecurity adjacent stuff), and 10-20% of taxes going to science would be international law.

This issue is not what I was discussing, what I perceived us to be discussing was whether or not Jonas Salk should have been pissed at the pharmaceutical companies profiting of his vaccine. As it happened, Jonas Salk was charitable, and believed in the openness of science, and understood the fact that someone actually had to produce and distribute the polio vaccine, and that this costs money. And as it happened, society had given this mandate to private companies, that operate with profit margins. And if him insisting on having a cut, if doing so would result in even a single additional person dying that wouldn't have otherwise, which it probably would have, then he didn't insist. Also he was too busy doing science, he even commented later that his fame was partially an unwelcome distraction from that.

To me, this is the ideal of a scientist. Someone who knows they stand on the shoulders of giants, and doesn't fret about the fact that the people who are standing on his are closer to the sun. The difference between Jonas Salk and everyone complaining about not being rewarded for training tokens, is that Jonas Salk made the choice to not patent his invention, and here the labs made the choice for everyone else. Jonas Salk and many others are charitable, but not everyone is, hence the complaining. But if everyone is forced to be Salk against their will, is that really so bad?

I see the LLM labs as the pharmaceutical companies, occupying a societal mandate to actually produce, and all the perils that come with it. But "giving back" is not part of that mandate, and unfortunate as that might be, that is not their fault. They are tasked with production, competition and progress, and that is already so expensive that they are struggling to meet demand.

And, you know, if redistribution truly is as easy as you say, at this thought my brain also produces a tinkle of anger at the injustice of it. And if I look for somewhere to direct that anger, I even have a name and a face! Look at Sam Altman, that smug supercar driving bastard, profiting of the hard work of Stackoverflow commenters everywhere. Eat him! Like, not in a gay way, but like in eat the rich! Out with the guillotines!

To me, that's my lizardbrain talking. The reality of our more complex non-tribal society is that the corporate structures we have created were not tasked with distributing rewards fairly, they were tasked with competing no matter the cost. And so successful companies do that, because the ones that don't disappear.

And like it or not, this methodology seems to work, on the whole. Even though it also offends my scientist sensibilities, it turns out that humans are greedy, and so incorporating that impulse into a structure that is limited to soft power is a good idea (unlike classic communism, where the same powers that produced products could also kill you). And it's not Sams fault that this is how it is. "Just running a business" is the reward structure society has created, and it's not part of Sam's job description to break that mold and start rewarding people he doesn't have to. In fact it might even be illegal for him to do so if it doesn't reward investors somehow also (like PR-wise).

And the overall fact is that the labs are doing what they need to do in order to produce something entirely new, that no one was even sure would be useful before they created it. And they are the representatives of the capitalist way of doing things, and if a publicly funded LLM undercuts them, that is fine. But maybe you do actually need a trillion dollars to make useful LLMs. Overall it's a good thing that someone is at least trying with funds, because there was no one lining up to buy 200k H100 for even the most prestigious of publicly funded academic institution, and certainly not for sending a check to all authors on arXiv.

And so overall, capitalism seems to me to be doing a fine job, and I pay my subscription fee gladly. And when my lizardbrain provides it's hateful opinion, I think of Jonas Salk, and it doesn't seem so bad after all.

Well, is it true that they give back nothing? What about the compute? Pay checks? The value of the subscription you pay for? What about actual examples of things they have given back for free, like Whisper, which used to be SOTA and is still extremely useful. Occasional excellent research papers, particularly from anthropic?

My point about "moral panic" is that it leads to statements like "giving back nothing"... which are objectively untrue. They might not reward every person that contributed to the tokens they are training on, but doing so is extremely practically difficult, and hard to do fairly, and is also probably a waste of limited resources in terms of net human progress. All these companies are doing is exactly what society has set up as the capitalist methodology to get work done: gather investors, pay people, sell products, etc... As opposed for example to the communist party deciding that the state should fund your project, or to fight for a research grant, or some other methodology, which might or might not work as well.

The only curious part about capitalism is that some individuals get a disproportionate amount of the reward for work done. At a societal level, this is essentially a soft power redistribution system, but often also leads to obnoxious individuals with supercar collections. Whether this is an overall good or bad thing for human progress is really, really hard to say for sure. However, it has a tendency to promote a lizardbrain response evolved to promote resource sharing in tribal societies, which was the best overall strategy in that setting. Or in other words, it makes people jealous.

I'd say both are equally soulless, dualism is a little bit of a philosophical dead end.

Frankly, I don't particularly care much for the moral panic around capitalism. Capitalism has it's downsides for sure, but it's the system our society has chosen to motivate people, and it seems to work okay for many things. Does it matter if the AI model that solves your diagnosis, creates a life saving drug or solves an Erdös problem is made by a corporation or not? It bothers me none, progress is progress. As long as the authors of Textbooks everywhere wouldn't have otherwise invented LLMs a decade ago if they only had been given a little bit more money, then I'd say the money is going to the right place.

That sentence could easily be applied to the human baseline.

For the past year, I think I've watched more AI generated video content than movies in terms of hours spent. Some of it is quite good (eg. Neuralwiz)! Granted, I watch very few movies, but still, I'd say this kind of counts.