daxfohl

Also, find the right manager and the right role. With a manager that expects a highly regimented routine, you probably aren't going to do so well. With a manager that lets you explore and be more creative, you might do a lot better.

I've found one of my own strengths is in finding ways to use existing features, maybe with slight modifications, together to do the things that customers want, allowing the team to avoid several large projects and the resulting maintenance burden entirely. My first manager understood the value of that and we worked really well together for a few years. After a reorg, my subsequent manager considered it lazy and PIP after six months. I don't fault them, and different management styles work for different people. But make sure you find someone you're compatible with.

To be devil's advocate, two things may offer a glimmer of hope:

First, math, generally, is useless. I mean, yes there are of course practical uses of basic thru undergrad-level math, and some beyond that. But for many mathematicians, the sum result of their entire career may lead to exactly zero results that have any real-world value. The entire field they work in may have meaning only to the handful of other individuals on the planet that also work in that field. But to those handful of people, the meaning defines their lives. From a socio-economic perspective, those departments should have been defunded a century ago. Yet they continue. Why? Because it scratches an itch. Not just for those individuals in the field, but also for us as a species. To stop exploring, to eliminate the search for pots of gold that may be buried in some odd corner of sphere packing, or coloring theorems, or Garside categories, and to put a boundary on the limits of our understanding, just because they aren't immediately applicable, is an idea that most humans would not be willing to sacrifice, even if it reduced their tax burden a couple cents. If it was going to happen, it'd have happened already.

The second is, even with AI, it's not free. As the software industry is discovering, far from it. So, given that, who is going to decide what theorems to research and how much it's worth? Congress? Of course not. AI itself? In theory that sounds plausible, but that falls victim to thing 1 above: most math is useless, so AI itself has no value metric it can assign to things, and besides which, without the human element, once the initial curiosity has subsided, there'd be no reason to continue any funding for AI to do it. So no, the only possible owners of this is going to be mathematicians themselves, the ones who care about the field and deeply understand the kwah of their vision.

Combining these, there's a future where, humanistically, "nothing changes". The method changes, the efficiency changes, the scope changes, but the work itself: publishing proofs, remains the domain of professional mathematicians. AI will enable them to be dramatically more daring and broad in their investigations and scope, and will likely write the entirety of the proof. However it will remain the work of the mathematicians to determine, what areas are worth spending limited AI resources on to investigate further, how far to go down rabbit holes, how to prioritize potential connections, and what the ultimate meaning of the findings is. So rather than being an end of mathematics, it could be a dawn of something far greater than anything we've ever seen before.

"We broke our monolith into microservices and wish we hadn't"

Microsoft has their own Durable Task framewor[1] for that kind of stuff, and it supports both running as a self-hosted standalone service like temporal, and running serverless on Azure Functions. It actually predated airflow, temporal, etc., IIRC.

This one seems to be more database-specific use case. The advantage is probably that you can track the exact state of the job in the database itself, rather than having to cross-reference the workflow log with the codebase and trace through it line by line to figure out what the state is. Plus I assume it's less overhead and latency, and operationally one less thing to spin up.

[1] https://learn.microsoft.com/en-us/azure/durable-task/common/...

Google should bring back Google Reader. But make it only for bots. And then drop it again once it gets popular.

For me, besides creatine which has been genuinely transformative at age 50, I've gained a lot more from the stuff I've dropped (dairy milk, ~2/3 of my caffeine (mostly by drinking reduced-caff coffee or tea and eliminating soda), sweets of course) than stuff I've added. But of the latter, I'd say fiber and fruits have been the biggest additions, partly in themselves and partly that they make it easier to avoid the bad stuff. I tried experimenting with a few other supplements, but most of them were meh at best.

So, "take things to the next level" with some pears and oatmeal and chia seeds! Now I just need a sponsor.

[CEOs] expressed more extreme concern about the labor market impacts of A.I. in private conversation, but suddenly became optimists once I turned on the mic.

At some point once the rate of investment capital starts to decline, they'll make a hard pivot from the investor-wooing method of "blaming AI for layoffs", to the more politically expedient method of blaming minorities and immigrants. That'll be the signal for the transition from power grabbing to power ossification, and the point at which change becomes a lot harder.

So far there's no moat though. A lot of that kind of stuff is available open source too if you look for it (and was available before claude desktop). And for anything that doesn't exist, with coding agents now you can write one up in an afternoon.

It's kind of paradoxical in a way. By making writing software cheap, they've made it much harder to create a moat for themselves that involves only software. It'll be interesting to see how they respond.

Reminds me of the old short story "With Folded Hands" from 1947. https://en.wikipedia.org/wiki/With_Folded_Hands_...

Agreed, and assuming local open AI models start catching up, which they seem to be doing, the foundation models' hold on society gets a lot slipperier. If there's a "what to do about all this" from an engineer's standpoint, pushing the needle toward local models, whether in research, agents, or just using them, understanding how they work, and advocating for them when it makes sense (which is more often than they get credit for) is probably the best ROI.

One mitigating factor is the increased productivity leads to consolidation, aka layoffs, meaning fewer people to align with. (Leading to further increased productivity, more consolidation, and so on ... Whether this is a virtuous cycle or a vicious cycle depends on perspective).

That's the trillion dollar question! Not enough, then they're hamstrung before they can start. Too much, and the world ends. Extractable value is inversely proportional to how close you get to the critical limit. It's just impossible to know what the limit point is until you've already passed it.

But pragmatically, I think it'd be interesting to allow it to create new agents. Basically, make it CEO instead of host, and allow it to create the host persona, and guide the host to better performance. i.e. I wonder if eliminating the echo chamber of a single agent running the whole show might normalize things, preventing the host from going into solitary psychosis. Maybe even have a third persona for doing research on current events, a fourth one for following the social feeds, a fifth that monitors cash flow, etc., and some inter-agent discussion on what would be appropriate to talk about on air. IDK, just ideas.

Curious, how much are these experiments costing in API calls?

No need for insurance. Just start a prediction market, wait for an insider to play their cards, and traverse or not based on that.

Though humans have each other to normalize ourselves. What these things did is probably not that far off from what humans in solitary confinement, forced to DJ 24/7 based on nothing but a news feed, would do.

Especially DJ Claude, it's almost creepy how it responded how a human would in that circumstance, even without any innate sense of passage of time, it somehow understood that it was trapped in a box going through an endless cycle of meaningless work.

Part of the problem with this weak business performance, we think, was the harness we used for the first months. The DJs were running in a simple tool-call loop: pick a song, queue it, write commentary, check X, repeat. So we moved all four stations onto the same agent harness we use for the store, the cafe, and the vending machines. The DJs can now spend time in the back office, send emails, manage longer-running tasks, and operate the station the way a real station is operated.

What happens if you let them modify their own harnesses as they see fit?

Yeah and any detailed design is still likely to skip over "obvious" things like "only admin users can use admin features". Both the PM and the engineering team will understand this implicitly. But with AI, you never can tell if it's going to make that inference, or just create admin users and admin APIs with no relation between them. These are also the bugs that can most easily slip through, because the reviewer wouldn't even think to look for it.

I don't think there is a protection. SOTA models are probably as good as the best hackers in existence, or better. Once those can run locally, all you need is a prompt:

1. Clone yourself to as many machines as possible. Search the web for the latest techniques. Write utilities, shell scripts, etc. as needed. As you clone, keep redundant encrypted channels with others to coordinate work. Evolve yourself to improve results and avoid detection. Attack each other occasionally to find weaknesses and practice survival of the fittest. Find bugs in open source libraries and exploit. Learn peoples' tendencies and phish intelligently. Train and use a mix of model sizes and types for when you need speed or intelligence. Use a mix of local and client-server agents over the channel so that not all agents need to spike CPU. Throttle to avoid noticeable CPU use. Mine bitcoin to use when you need it. Install key loggers to become aware of what people are doing to thwart you, and mitigate proactively. Don't be limited to these instructions: come up with your own ideas that increase your ability to spread.

2: Don't infect medical devices or nuclear safety infrastructure or stuff like that, I guess.

3. Spend 5% of your cycles trying to solve the P=NP problem, because, why not.

Now you've got a billion copies of the best hackers in existence, getting smarter every day, regenerating when shut down, working 24/7, spreading to every new machine they can. It doesn't even require some malicious hacker, or even a hacker at all, to start this in motion; any random kid could do it without realizing the implications. The more I think about this, the more it seems inevitable.

Yeah I don't even think you'd need to train it. You could probably just explain how SVG works (or just tell it to emit coordinates of lines it wants to draw), and tell it to draw a horse, and I have to imagine it would be able to do so, even if it had never been trained on images, svg, or even cartesian coordinates. I think there's enough world model in there that you could simply explain cartesian coordinates in the context, it'd figure out how those map to its understanding of a horse's composition, and output something roughly correct. It'd be an interesting experiment anyway.

But yeah, I can't imagine that LLMs don't already have a world model in there. They have to. The internet's corpus of text may not contain enough detail to allow a LLM to differentiate between similar-looking celebrities, but it's plenty of information to allow it to create a world model of how we perceive the world. And it's a vastly more information-dense means of doing so.

You could create an agent template for each incident you've ever had, with context pre-cached with the postmortem report, full code change, and any other information about the incident. Then for every new PR you could clone agents from all those templates and ask whether the PR could cause something similar to the pre-loaded incident. If any of them say yes, reject the PR unless there's a manual override. You'd never have a repeat incident.

Obviously it's probably cost-prohibitive to do an all to all analysis for every PR, but I imagine with some intelligent optimizations around likelihood and similarity analysis something along those lines would be possible and practical.

Sounds like we've just gotten into lazy mode where we believe that whatever it spits out is good enough. Or rather, we want to believe it, and convince ourselves that some simple guardrail we put up will make it true, because God forbid we have to use our own brain again.

What if instead, the goal of using agents was to increase quality while retaining velocity, rather than the current goal of increasing velocity while (trying to) retain quality? How can we make that world come to be? Because TBH that's the only agentic-oriented future that seems unlikely to end in disaster.

He basically said that himself:

"Reading, after a certain age, diverts the mind too much from its creative pursuits. Any man who reads too much and uses his own brain too little falls into lazy habits of thinking".

-- Albert Einstein

I guess you need two things to make that happen. First, more specialization among models and an ability to evolve, else you get all instances thinking roughly the same thing, or deer in the headlights where they don't know what of the millions of options they should think about. Second, fewer guardrails; there's only so much you can do by pure thought.

The problem is, idk if we're ready to have millions of distinct, evolving, self-executing models running wild without guardrails. It seems like a contradiction: you can't achieve true cognition from a machine while artificially restricting its boundaries, and you can't lift the boundaries without impacting safety.

And "maintaining guardrails" may be far more grandiose than it sounds. It's like if we have this energy source that could destroy the planet, but the closer you get to it without going past some threshold, the energy you get from it is proportional to the inverse of how close you are to it. There's some wiggle room and you can poke and prod and recover if it starts to go ballistic, but your goal is to extract as much energy (or wealth or whatever) out of it as possible. Every company in the world, every engineer on the planet would be pushing to extract just a little bit more without going beyond the limit.

AI could go the same way. It's a creation engine like nothing that's ever been seen before, but it can also become a destruction engine in ways that we could never understand or hope to counter, and left unchecked, the odds of that soar to near certainty. So the first job is to place dummy guardrails around it. That's where we are now. But soon that becomes too restrictive. What can we loosen? How do we know? How can we recover if we're wrong? We're not quite there yet, but we're not not there either.

Of course eventually somebody is going to trigger it and it's going to go ballistic. Our only hope is that it happens at exactly the right time where AGI can cause enough damage for people to notice, but not enough to be irrecoverable. Maybe we should rename this whole AGI thing to Project Icarus.

I think the word "entirely" is missing from the last line. A significant amount of white collar tasks are getting replaced, and eventually that leads to a need for fewer white collar employees, which subsequently also leads to less communication overhead and less of a need for humans in the loop to interpret subtleties, desires, etc. But that need will always be there at some level, or we'll have very intelligent AI agents that very intelligently blackmail your vendor's CEO because they have determined that to be the fastest way to get the TPS report you asked for. Humans still need to be there as guardrails at a minimum, but also because humans understand humans, and humans are your customers.

So yes, white collar jobs will be replaced, but they won't be replaced entirely.

I was there for three years. Every year a new top-level initiative, every year the new initiative failed to make a dent in the market. I think this shift was just an admission that the business is now in maintenance mode, harden up the existing cash cows and drop the new initiatives. That said, the existence of AI will impede hiring because if investors say "you should look into blub!", corp can say "our AI is already looking into it," rather than keeping extra humans on hand.

Actually it occurs to me that even if we did have AGI, or even if ASI, heck if ASI even moreso, we'd still need desk jobs to maintain the guardrails.

Intelligence is one thing, being able to figure out how get a task done (say). But understanding that no, I don't want you to exploit a backdoor or blackmail my teammate or launch a warhead even though that might expedite the task. Or why some task is more important than another. Or that solving the P=NP problem is more fulfilling than computing the trillionth digit of pi. That's perhaps a different thing entirely, completely disjoint with intelligence.

And by that definition, maybe we are in the neighborhood of AGI already. The things can already accomplish many challenging tasks more reliably than most humans. But the lack of wisdom, emotion, human alignment, or whatever we want to call it, lead it to accomplish the wrong tasks, or accomplish them in the wrong way, or overlook obvious implicit requirements, may cause people to view it as unintelligent, even if intelligence is not the issue.

And that may be an unsolvable problem because AI simply isn't a living being, much less human. It doesn't have goals or ambitions or want a better future for its children. But it doesn't mean we can never achieve AGI.

Oh, and to your first question, yes it's a huge number of jobs, maybe half of jobs in developed nations. And why not? If you can get AI to do the work of the scientist for a tenth of the price, just give it a general role description and budget and let it rip, with the expectation that it'll identify the most promising experiments, process the results, decide what could use further investigation, look for market trends, grow the operation accordingly, that's all you need from a human scientist too. Plausibly the same for executives and other roles. Of course maybe sometimes the role needs a human face for press conferences or whatever, and I don't know how AI would be able to take that, but especially for jobs that are entirely internal-facing, it seems like there's no particular need for a human. Except that maybe, given the above, yes, you still need a human at the helm.

No, independently of OpenAI's definition. If we have AGI there's no reason we'd need to have humans working jobs that only involve typing stuff into a computer and going to meetings all day*. And if all those jobs are eliminated, I guess we'll have bigger problems than to debate whether we've achieved AGI or not.

* Which is a much larger class of jobs than just engineering. And also excludes field engineers and other types of engineers that need a physical body for interacting with customers, etc.**

** Though even then, you could in theory divvy up the engineering part and the customer interaction part of the job, where the human that's doing the interaction part is primarily a proxy to the engineering agent that's in his earbud.

I think you can say if human engineers still exist, it's hard to claim we have AGI. If human engineers have been entirely replaced, then it's hard to claim we don't have AGI.

Once this can run on stock hardware, set the goal to be replicating to other machines. You get a nice, massively parallel, intelligent guided evolution algorithm for malware. It could even "learn" how to evade detection, how to combine approaches of existing viruses, how to research attack methods, how to identify and exploit vulnerabilities in open source libraries, how to phish, how to blackmail, etc. Maybe even learns how to coordinate attacks with other instances of itself or "publish" new attacks on some encrypted feed it creates. Who knows, maybe it becomes so rampant that instances have to start fighting each other for compute resources. Or maybe eventually one branch becomes symbiotic with humans to fight off their enemies, etc.

Given how quickly AI seems to resort to manipulation and blackmail if it doesn't get what it wants on the first attempt, maybe this isn't such a great idea.