HN user
Plotting it this way is just a shift and rescaling. As the other commenter already showed this will lead to a plot that looks pretty much the same in terms of difference relative to the past data. I think standard deviation makes sense here since we don’t really have an innate sense of what is a meaningful change in ocean surface temperature and the point is to show that we are already far beyond normal here.
I’m not talking about the media release in the direct link. If you click through “Read the paper” they make the same comparisons.
This is cool but I don’t know if the comparisons to conscious awareness really make sense here. Their definition of the J-Space is basically the expectation of how much a final logits output would change as a result of a small change in a particular layer (see past work on information geometry). This seems more to me like showing there exists an abstract reasoning subspace which is generally shared across different contexts. I guess you can relate it to humans but I’d prefer a more direct claim in a paper rather than having to present things in this more fluffy way.
There’s debate over how much timing actually matters vs the rate of firing. Some people do believe in precise spike timing but I would say the general consensus is that spiking rate is a better measure of the current state of a system. There’s significant noise (as far as we can tell) in a neuron’s timing and it’s best modeled with a poisson process so we tend to think of it as rate coding which we can at least hand wave as viewing an RNN with a ReLU approximating.
Generally you can take a geometric view of this where certain features in a stimulus covary with neural activations in the same way they will with RNN “activations” which is at the real core of why people model things this way. The general idea being a dot product in an RNN can tell you something about what features are relevant for a task and we can look for hints of the same information being encoded in neural data. Certainly not everyone is in agreement on this though.
I feel like we’re all just operating on vibes at this point (including the government). These models are already powerful and could probably be used to do some bad things even without the next gen.
Anthropic has successfully made their vibe the bringers of the end of the world so please stop us. ChatGPT is in some weird middle ground where they’re no longer the evil company of a year ago and are trying to act like the mature one in the business. XAI is the vanity project of a jealous kid who wants to ignore anyone elses opinion but is falling behind. Gemini is viewed as the less cool cousin and no one pays attention to since google has generally avoided the end of the world talk (coincidentally I think gemini has the best non-coding model but no one seems to talk about that)
I think each company has been treated in the wider discourse based on their vibe and no one is operating on facts here since it’s all pretty obfuscated intentionally. I could see some things going very bad as a result of these models but I also don’t believe anyone who’s just pattern matching to their favorite scifi book here and we’re all playing into it. May as well go outside for a bit and enjoy the sun.
Oh good point, sorry got the years mixed up in my head somehow.
It’s not suppressed though, it’s just that the current evidence points in one way and much less in the other. There are plenty of review papers you can read to see how people weigh the evidence which you can of course disagree with: https://doi.org/10.1016/0010-0277(95)00678-r
I think it’s less that it’s impossible but more that we don’t have any clue as to what causes differences among groups and many people use such measured differences as evidence for pretty deplorable ideas. There’s much more evidence for social determiners than anything biological. Everyone outside of Africa shares a single ancestor 20,000 years ago. There’s far more genetic diversity within Africa than the rest of the world. That alone is often enough to disprove many theories regarding racial differences since our intuitive understanding of “genetic difference” is so flawed.
Past research has a eugenicist bias because early statisticians were eugenicists seeking evidence for the ideas. I would argue that’s why the social determiners research is valuable to help offset that.
I see. I wonder how this works out in terms of risk/reward. I suppose if you take extinction as -infinite cost than it would be the only issue worth thinking about. Where I think this line of thinking gets challenging is when you need to take in terms of a counter factual. A lot of these were already risks prior to AI (bioweapons, nukes, etc) so what’s the marginal increase in probability as a result of AI I guess is the question which matters. I could get more around this way of framing it than saying that AI itself is the problem. It’s just the being more capable as a species increases risks. I think a lot of these pushback comes from the fact that it’s often the CEO who stands to gain huge by saying his tool is going to end the world so we need public buyin to supporting it. If instead it was just framed as “general technological advancement” is dangerous but potentially worthwhile I think more would be on board.
I wonder if our difference in view could be an instance of the jagged nature of AI’s intelligence. I do computational research in a basic science so write code or build models basically all day that is (occasionally) novel. I would say that I’ve noticed exponential improvements in parts of my job but certainly not all. For example, if I’m trying to visualize a concept from a paper I now go straight to Codex, give it the paper, and describe a webapp which allows me to play with the model in a way that wasn’t possible one year ago (this is great for teaching btw). If I have a script that I want to generalize, add in better metrics, or setup for running on a cluster I use codex and it does great.
Where it fails me though is exactly when I’m doing something novel like developing a new model or trying to develop some new method to process data. I’ve tried many times to one shot these ideas with detailed descriptions of what I want, how I’d like to generate abstractions, etc and it almost always ends up changing what I want to what I can only describe as something which better matches its training data. It often quietly changes key details that means that I have to delete the whole thing and start over. Just today this happened. On this level of task I’ve found that my workflow and pace of iteration hasn’t really changed at all in the last year. I still have to go and explain in detail on a function by function level what I want in much the same way I did a year ago. While that’s obviously a harder task, it seems to me like the task this whole long term exponential argument hinges on. I obviously could be wrong and maybe LLM with eval loop will do all of this for us but it seems still quite bad at anything without a clear definition of “good”.
I’m personally much more concerned about autonomous weapons, surveillance, and people plugging these things into places they don’t belong to avoid responsibility than I am the general possibility of these models being smarter than me in every way but obviously I could be wrong on this and am just using it incorrectly, hence the question.
That’s interesting. I commented something about this elsewhere but to me part of the exponential argument that loses me though is that it can often seem like a way to distract from issues that already exist which we should be working to fix. Things like autonomous weapons or mass surveillance are already here and rather terrifying and I would hope that we would dedicate our time to fixing those rather than having industry leaders focus so much on hypotheticals. While I guess the hypothetical scenario could be so bad that we must focus on it, I imagine a world which can’t come up with a way to spread wealth more equally or prevent mass proliferation of surveillance technology through profit seeking behavior will not be able to handle a digital super intelligence. So I keep coming back to the question: why is all I hear these industry leaders talking about is the threat of extinction? Maybe it’s just news coverage but I would love to see a leading lab release research on the health effects of subaudible sound in datacenters or other immediately present issues which would build good will towards these further out concerns.
I don’t really think it has to do with general denial in this case. I think he’s likely right in that pushback against datacenters is partially the release of angst over the threats which these models pose but in an indirect fashion since power is so unevenly distributed that local political organizing is the only real mechanism for people to act through. I think everyone feels nervous about these tools. People are scared of autonomous weapons, hackers using agents, and any of the other present issues that already exist. I think some of the pushback that comes in these threads is the fact that people like Dario are so focused on the long-term view that they suppress all immediate threats which already exist. You don’t need to argue about exponential growth or “AI being better than humans at everything” to want to better regulate this technology.
I would argue the best way to safeguard against long-term threats is to start by focusing on the issues that already exist. If you can offset the health risks of local datacenters or issues of unequal distribution of wealth by creating a more equal society right now then you’re already on the path to handling these long term issues. To me, this distant focus only distracts from the already present issues and conceals effective policy in this moment. We do need to safeguard against AI risk and it’s already here. Don’t even get me started on the havoc which recommendation systems have caused in society in the last 15 years which we still don’t call AI because it doesn’t speak.
Tl;Dr: These essays can feel disingenuous because:
1) AI risk is already present without exponential growth. The exponential growth argument often feels like a distraction from the fixes we could put in to fix the current issues that are already here.
2) The people stating this argument often have billions of dollars to gain if it comes true. While they may be altruistic, I also don’t see them doing all that much to fix the issues that people are already claiming exist and instead continue onwards on their path by justifying that they are the only ones responsible enough to handle it if the super intelligence does arrive. By continuing down that path, if that day ever does arrive they’ll have ensured the existence of a system which is unable to handle it.
I know this is likely just for IPO hype but when I read things like this I sometimes wonder if I must be missing something. I use agents everyday and find them really useful and they save me a lot of headache. At the same time I find that if I let it self-direct at a high level at all it generally makes bad choices that cause me headaches later so I can’t really give them autonomy. Enough people seem to believe this exponential line of thinking though that I keep having to wonder: am I the one missing something here? Is there some magic tool that I haven’t found yet that will cure cancer?
I’m more referencing the narrative that climate change is being used as a political point to enforce a global political state which stifles innovation (for example read the Project 2025 document around page 417 https://static.heritage.org/project2025/2025_MandateForLeade...). They do make some fair points in there but the primary change to the EPA that is being proposed is focusing on decentralizing the response to climate change to the states. In a vacuum where climate change is a matter of air quality this isn’t a crazy idea but the reality is that this is a global issue which is inconvenient if you want to prioritize growth/innovation over all else and the general idea behind climate conservationism is a powerful political force which the whole world is trying to shape right now for their benefit. There is right wing conservation which largely focuses on local, narrow issues (such as whale habitats being used as a reason to cancel offshore wind). There is also left wing conservation which pushes the more global climate change view and more large scale changes such as carbon taxes, increased stimulus for renewable energy, and often more regulation. Multiple things are true here though: 1. A lot of our regulation is pointless and serves to just increase cost and lock-in existing players as dominant forces in markets. This definitely stifles innovation so there is pushback against climate related changes for fear this will continue or worsen. 2. Climate change is a powerful force in modern politics which serves as a binding force for many disparate factions and is creating many grassroots movements. The impact of these on innovation isn’t clear (nor is it the point) but increased climate awareness is certainly increasing pushback against greenhouse gas emission intensive industries such as datacenters (in their current forms until renewable capacity can catch up) 3. Climate change is very real and is already increasing the cost of doing business and being less aware of it isn’t going to help. This will also stifle innovation as costs mount in the coming years.
I see this move as an attempt to stifle 2 by hiding 3. It seems like a bet that if we don’t focus on the global issues (like a collapsing current) then much of the political power behind climate change disappears since we are frankly quite lucky in the US to have a beautiful, and largely clean environment/air. Grassroots movements and politically powerful messages are likely an annoyance for those who want to keep doing business as usual since it benefits them to do so. Not trying to make a strong claim that our current response to climate change or what anyone in particular is doing is right or wrong but it seems clear that climate change is a powerful message that pushes for much grander scale changes than most people are comfortable with especially those with strong private interests in ignoring it.
This strikes me as one of the recent moves by our political/capital class where they think that if they just remove the information that’s inconvenient for them, people will stop caring and let them do what they want. You only need to listen to the bosses who so many of us work for to know that they think climate change is just an inconvenience in the way of progress. Only time will tell if this strategy will work or not.
Agreed I usually use Gemini for explaining concepts and ChatGPT for getting things done on research projects.
I’m pretty skeptical that we’ll have actual convincing simulations of human brains any time soon. We’re still in the phase of trying to figure out how mice decide to turn left or right in a maze. Not to mention that the ethical/allowed ways of collecting human neural data are incredibly coarse especially the most common forms like fMRI or EEG (maybe some BCI will improve this but it’s still pretty rough technology). Most of our data as well is collected in incredibly controlled conditions (even “naturalistic” stimuli) so the data we operate on is still not particularly indicative of how people act in the real world. Maybe you mean simulations of a “mind” which behaves like people even if it’s not particularly accurate to the brain? What did you mean by this?
I’d also recommend reading Modernity and the Holocaust as a good intro to studies of the Holocaust through a similar lens. None of this is new
People is a broad term. Outside of major cities (where I live) libraries serve a very essential service for parents and their children and as a free communal space for the broader community. Our libraries are always full and a large part of the health of our area.
I recently moved over to codex after I couldn’t reup my membership and maintain access to clause code. I will say thus far I’ve found codex to work better and with less limits.
I think their point is that discounting the time estimates is more a constant shifting of the window of what we expect more than them being de-facto incorrect. They’re more off by degree (e.g. an XX% reduction vs complete extinction) than being worthless. As the example points out a large reduction can be very similar to an annihilation it’s just that we are only used to what we know so we constantly shift what is normal.
As someone actively researching in the neuroscience field these ideas are increasingly questionable. They do do a decent job of job of predicting neural data depending on your definition and if you compare them to hand built sets of features but we’re actually not even sure that will stay true. Especially in vision we already know that as models have scaled up they actually diverge more from humans and use quite different strategies. If you want them to act like humans or better reflect neural data you have to actively shape the training process to make that happen. There’s less we know about the language side of things currently though as that part of the field hasn’t yet really figured out exactly what they’re looking at yet because we generally know less about language in the brain vs vision. I think most vision scientists are on board with the idea that these things have really been diverging and have to be coerced to be useful. Language it’s more up in the air but there’s a growing wave of papers lately that seem to call the human LLM alignment idea into question. Personally I think the platonic representation idea is just a function of the convergence of training methods, data, and architectures all of these different labs are using. If you look at biological brains across species and even individuals within a species you see an incredible variety of strategies and representations that it seems ridiculous to me that anyone would suggest that there’s some base way to represent reality that is shared across everyone and every species. Here’s some articles that may be of interest if you’re curious:
[1] https://arxiv.org/pdf/2211.04533 [2] https://www.nature.com/articles/s41586-025-09631-6 [3] https://www.biorxiv.org/content/10.1101/2025.03.09.642245v1
I think you’re treating the people at these companies as more dumb/powerless than they really are. I used to work in big tech and quit after a few years due to similar concerns that are being raised here. I will tell you anecdotally that everyone there I worked with thought our company was a net negative on society and that our work was at best indifferent and in my case likely exploited weak labor laws in poor countries to overwork people who we never were allowed to speak with for cheap data labelling. Yes, there definitely was some organizational shuffling to make it hard for us to see. We all knew, we weren’t idiots. My personal favorite book on similar concepts is “Modernity and the Holocaust”.
I would argue the organizational tricks exist more for the benefit of the worker than the org itself. The “powerless software engineers” there wanted the excuse to accept the huge salary for very easy work. The organizational tricks don’t fool anyone it’s a favor to the workers at these companies. They exist specifically to ease the cognitive dissonance just enough continue doing your job so you can get paid as much as you want without having to take guilt home with you. I’d say the same is true of my friends in the aerospace defense world. Do you really think they can’t put two and two together and understand that their “flight stabilization module” isn’t going to be used to blow up some school in another country?Your argument is just giving these people the ease of conscious which they want.
On the decision front as well I’d say most of the actual decisions in my org were not actually being made by C-Suite level or even executives. The managerial class at these companies are playing a totally different game than engineers. All the managers care about is that they have good metrics to show their boss so they can get promoted before the person on their sister team. I didn’t interact with a single person above VP and on my projects (as a recent grad mind you) I couldn’t even get my product manager to make a decision on how my product should be implemented. Everyone in the managerial class in these large companies largely exists to provide the illusion that you have no power. Meanwhile they have no idea what anyone in their org is actually working on and as long as they get a nice number to show their boss at the end of the quarter they won’t bother looking too closely.
I think we’re going to have a reckoning in the near future where we’re going to have to come to terms with the fact that the surveillance state which we’re scared of has been designed and built by the “powerless” engineers. The world is too complex for executives to actually have any understanding of what’s being done beneath them. There is SO MUCH room for the average engineer to shape their work in a more positive direction but that would actually require taking ownership over their work and risk some mental connection to its implementation. The average big tech employee already exists on the precipice of too much cognitive dissonance so they can’t afford to try and change anything otherwise they’d be convinced to give up their mid 6 figure salary while already having a larger net worth than 99% of the world will achieve in their lifetime. You cannot equate the life of a software engineer at a large company with the struggle of the working class in any meaningful way. Having a large mortgage is not at all similar to living paycheck to paycheck with variable hours at multiple jobs.
I’m being a bit brutal here I know but I’m so tired of people making excuses for themselves and others for living a life devoid of responsibility. If what I’m saying has struck a chord with anyone who is in a similar spot as I was, I’d suggest strongly questioning your position in the world. I have since found a different career path where I have clear ownership over my work and direction and am much happier now. I’m not fixing the world or anything and took a huge paycut but I agree with the outcome of job and am actually willing to work hard without resentment. I also applaud those who fight against the indifference of their coworkers in these companies since I know they exist. If every worker took responsibility for their output I promise you the parasitic Google or Meta as we know it would not exist. We are not the victims here. If we were desperate coal miners, I’d agree with you, but we are a class of workers with a level of financial flexibility, education, and freedom in our work most of the world has never been able to dream of. The success of these companies exists in how much responsibility they can put in the hands of their engineers who ultimately make most of the meaningful decisions whether they’re willing to admit it or not.
Agreed. I’m sure some house members will vote for it because they only had a random staffer read the bill and heard that it gives them a good talking point in the next election. I just wanted to point out what’s maybe obvious to everyone that this won’t help kids. I’m sure this is being pushed by Meta/whatever other ad dependent business wants to pass off liability of verifying age with the added benefit to everyone in power that it’s easier to track everyone as a result.
I agree. Just was trying to point out that this has nothing to do with helping kids
Politicians will do any draconian measure to help kids except try and improve the lives of their parents so that they can actually dedicate time to parenting. Making it slightly harder to access the internet fixes nothing. What if instead of having the largest prison population in the world our government supported communities that make raising good children possible? Our society needs to lose this urge to diagnose each other and provide some forceful treatment and instead set sights on providing the pre-conditions for everyone to prosper and lead their version of a fulfilling life. Only then will we have functional, healthy children. I quite like what the mayor of Baltimore has been doing to revitalize his city and it seems to be leading to actual change there if you want a good example: https://m.youtube.com/watch?v=XQs59YY-e2I&pp=ygUXY2hhbm5lbCA...
There’s a lot of research out there about the general flexibility of the brain to adapt to whatever stimulus you pump into it. For example taxi cab drivers have larger areas in their hippocampus dedicated to place cells relative to the general population [1]. There’s also all kinds of work studying general flexibility of the brain in response to novel stimulus like the visual cortex of blind people being dedicated to auditory processing [2 is a broad review]. I guess you could argue that the ability to be flexible is intelligence but the timescales over which a brain functionally changes is longer than a general day to day flexibility. Maybe some brains come into an initial state that’s more predisposed to the set of properties that we deem as “intelligence” but development is so stimulus dependent that I think this definition of a fixed intelligence is functionally meaningless. There are definitely differences in what you can learn as you age but anyone stating we have any causal measure of innate intelligence is claiming far more than we actually have evidence for. We have far more evidence to suggest that we can train at least the appearance and usage of “intelligence”. After all no one is born capable of formal logical reasoning and it must be taught [3,4 kind of weak citations foe this claim but there’s a lot to suggest this that I don’t feel like digging up]
[1] https://pubmed.ncbi.nlm.nih.gov/17024677/ [2] https://www.annualreviews.org/content/journals/10.1146/annur... [3] https://psychologyfor.com/wason-selection-task-what-it-is-an... [4] https://www.tandfonline.com/doi/full/10.1080/14794802.2021.1...
Agreed. I find that people who argue that religion is necessary for ethics tend to ignore the history of their religion and the fact that the original text largely serves as a jumping off point for religious philosophers to connect older “secular” texts to this new religion. Modern Christianity is a complex combination of Platonic, Aristotelian, Syrian, and Roman ideals which are taken out of their original context to align with the Bible even though the original writers would say they knew nothing about Jesus. The base texts which many of these ideas are based on make almost no appeals to God and focus more on what it means to live a “good life”. To be fair a lot of great ethical arguments are made by Christian writers but I think that’s more just a consequence of their cultural upbringing and the fact that the thing the New Testament really added to the discussion was that your ethical responsibilities generalize beyond yourself and your friends/family.
Religious ethics are just as fluid and complex as secular ethics, it’s just that the concept of God makes people think they can claim that their way of thinking is the only one that’s real. I would guess if you self-reflect though you’d see that even within one lifetime the definition of what’s moral in a religious context changes as well.
Faux environmentalism is the new climate denialism. https://www.desmog.com/2025/08/27/ai-slop-websites-are-publi...
This is an interesting insight I hadn’t thought much about before. Reminds me a bit of some of the mechanistic interpretability work that looked at branch specialization in CNNs and found that architectures which had built in branches tended to have those branches specialize in a way that was consistent across multiple training runs [1]. Maybe the multi-headed and branching nature of transformers adds and inductive bias that is useful for stable training over larger scales.
[1] https://distill.pub/2020/circuits/branch-specialization/