scoring1774

Reflecting on this for some reason reminds me of this passage from Kurt Vonnegut's "Sirens of Titans". I hope we use these tools to unlock something within ourselves rather than mindlessly expanding outwards.

"Mankind, ignorant of the truths that lie within every human being, looked outward–pushed ever outward. What mankind hoped to learn in its outward push was who was actually in charge of all creation, and what all creation was all about.

Mankind flung its advance agents ever outward, ever outward. Eventually it flung them out into space, into the colorless, tasteless, weightless sea of outwardness without end.

It flung them like stones.

These unhappy agents found what had already been found in abundance on Earth—a nightmare of meaninglessness without end. The bounties of space, of infinite outwardness, were three: empty heroics, low comedy, and pointless death.

Outwardness lost, at last, its imagined attractions.

Only inwardness remained to be explored.

Only the human soul remained terra incognita.

This was the beginning of goodness and wisdom."

This is the first one of these announcements that has me actually scared of what comes next. Obviously these models have gotten smarter but this strikes me as the first time I've seen a model have a "paperclip factory" moment and perform non-trivial tasks to accomplish a clearly misaligned secondary goal.

It's remarkable that building a society based around having to do something so you can go do your hobbies at home after work has built tools like this. I still just want to play music so I hope we can control these enough to make that possible without detonating what I love.

I sometimes think this way but I do wonder whether this would be true in a world where individuals demand more control over the impacts of their work. It seems that the current problem is that the labor of the 98% (or whatever number you like) is well-aligned with the demands of the 2% without principles who seem to have a strong propensity to end up in leadership positions. It reminds me of the simulations people run where in a high-trust world scammers do well, while in a low-trust world they die off only to then lead to a new high-trust world.

If enough people make decisions like OP and only perform work that they believe in (for one reason or another) perhaps that redirects the path to power for the 2% such that they need to act in the interests of the 98% rather than the other way around. I think OP was very late to the train here and I had made a similar career switch in the past for similar reasons but I'm happy to see other people making decisions that, if nothing else, will make them feel that they are in control of the impacts of their time and hard-work. Not sure how things will balance out in the end/distant future but we do our best and try to lead a life we feel good about living.

This has been done: https://arxiv.org/abs/2510.04871v1

This is anecdotal but as a current PhD student who was doing research at a large tech company for a few years prior to this, the incentives as an individual are very different across the two programs. In tech even in a research role there was little to no incentive to dive deeper into potential high-risk, high-reward research because your career trajectory was determined by maximizing certain metrics for promotion cases. The general vibe among my coworkers was spend your day on the guaranteed progress projects and then go home. This was actively incentivized by leadership who asked for frequent progress updates especially as AI began to takeoff.

As a grad student so far though I've found the incentives to be very locally driven and the kind of research you can do is almost wholly determined by yourself and your advisor. This can be good or bad but if you find an advisor who is in a stable spot (tenured or nearly-tenured) and not a jerk they'll generally give you leeway to pursue what you believe to be high-impact work even if it doesn't align with the general consensus on what to do next, especially if you have proven credentials and a clear image of a research plan in mind. Additionally progress is largely driven by the individual so there's a larger personal motivation to really delve into a problem and be consumed by it. For me personally, I have access to significantly fewer resources than before but have gained the freedom and time to not be attached to the paper-mill or some measurable metric and am spending months of my time trying to get at a deeper problem than I ever would have been able to in industry. While this may be different than the usual narrative about academia, I think it's more true than people say since there are such huge variations in how academia works as a result of school, advisor, and the individual researchers themselves. The disgruntled tend to be those who complain the most while those happy with the field are busy doing other things. I'd compare my experience in academia thus far to the startup of the research world whereas the industry jobs (at least in tech) consume far more resources and are pressed to provide steady, measurable impact. Maybe it's upsetting that we do waste some resources on stupid research which does exist, but the odds of getting a researcher like Einstein dedicating 10 years to discovering relativity in an industry job are vanishingly small. I'll probably be unsuccessful but there are 100's of people in my field doing related but different approaches and this kind of swarm approach is more likely to give a fundamental discovery on a population level than the large alignment of goals found in private research who would do a great job building on any basic science discovered in academia. I don't think it's wasted resources if 99 researchers fail in different ways and 1 succeeds since traversing the tree is inherently valuable even if most of the leaf nodes are failure. That's far more likely to happen in academia imo than industry.

It's not that private sector funding is inherently worse, but in reality it is different and as such will lead to different results due to how people and our economic system at large work. While I'm sure there are exceptions where individuals at private research labs are highly-motivated and feel the push to go the extra mile and try to find some deeper truth than is necessary for their personal well-being, in my experience many doing research at these companies are apathetic as a direct result of the environment in which it's being conducted. It's hard to feel motivated to make a large step in basic science when you think it'll just be consumed by the large institution you exist within who's stock price you have no real effect on rather than being open-sourced for peoples' benefit. We should have diversity in how we fund science.