zar1048576

I think we are in largely uncharted territory here, especially given the implications. Is Anthropic's approach optimal? Probably not. But given the stakes involved, gating access seems like a reasonable place to start.

I'm curious about how gated access actually holds over time, especially given that historically with dual-use capabilities containment tends to erode, whether through leaks, independent rediscovery, or gradual normalization of access.

In case it helps, we open-sourced a tool to audit dependencies for this kind of supply-chain issue. The motivation was that there is a real gap between classic “known vulnerability” scanning and packages whose behavior has simply turned suspicious or malicious. We also use AI to analyze code and dependency changes for more novel or generic malicious behavior that traditional scanners often miss.

Project: https://point-wild.github.io/who-touched-my-packages/

I think that concern is valid in general terms, but it’s not clear to me that it applies here.

The goal here seems to be removing low-value output; e.g., sycophancy, prompt restatement, formatting noise, etc., which is different than suppressing useful reasoning. In that case shorter outputs do not necessarily mean worse answers.

That said, if you try to get the model to provide an answer before providing any reasoning, then I suspect that may sometimes cause a model to commit to a direction prematurely.

Have had similar issues with costs sometimes being all over the map. I suspect that the major providers will figure this out as it’s an important consideration in the enterprise setting

My sense is that the asymmetry is non-trivial issue here. In particular, a threat actor needs one working path, defenders need to close all of them. In practice, patching velocity is bounded by release cycles, QA issues / regression risk, and a potentially large number of codebases that need to be looked at.

I definitely agree w/ you that big organizations are generally better able to navigate the enterprise sales process, but mainy trying to say that customers might choose to work with a bigger company's products for reasons that typically go way beyond that (e.g., better integrations, support resources, etc.).

My sense is that if a threat actor were able to build a quantum computer to the scale of being able to compromise public-key primitives based on the difficulty of integer factorization and discrete logarithms under the key sizes used in practice today, one of the highest-valued targets will be Bitcoin.

Q-day estimates are sensitive to several factors; e.g., hardware qubit counts, error correction overhead, and algorithmic efficiency (e.g., better factoring approaches could compress the timeline meaningfully without any hardware breakthrough).

Migration complexity side is also not straightforward. Cryptographic primitives tend to be deeply embedded in ways that are not always easy to find. FWIW, we built a free scanning tool for developers to find and remediate cryptographic vulnerabilities in their repos (still in beta: https://app.threatpoint.com).

2029 might be conservative or optimistic depending on which variable moves first.

I wonder if there will be a different phenomena — namely everyone just developing their own personal version of what they want rather than relying on what someone else built. Nowadays, if the core functionality is straightforward enough, I find that I just end up building it myself so I can tailor it to my exact needs. It takes less time than trying to understand and adapt someone else’s code base, especially if it’s (mostly) AI generated and contains a great deal of code slop.

Nice weekend project! Even though there are copious resources out there (textbooks, videos, etc.), those may not appeal to everyone. People have different preferred modalities for consuming information and there is always value in (correctly) reframing concepts in a way that can be better understood by people who don’t resonate with traditional textbooks and YouTube videos. I’m glad you found a formulation that works for you, and judging by the number of upvotes, it resonated with others as well. At the very least, I’m sure that working on this improved your understanding as well!

Definitely miss those!

I don’t know if that’s necessarily true. I do think that a big part of enterprise sales involves building a comprehensive solution that works well within the customer’s ecosystem. Start-ups usually tend to build point products, which have value, but are still missing functionality (even if that functionality is not scintillating) that customers really desire to easily deploy and maintain solutions. Also, customers do care about things like stability of their vendors and the level of available support.

I agree with this. What you focus on depends on the circumstances. I believe PaulG likes to say that premature optimization is the root of all evil. Early on, you’re trying to ship and get a functioning product out the door — if spending a bit of money on extra RAM at that time helps you, it’s worth it. Over time, as you are trying to optimize, it makes sense to think more about memory management, etc.

I suspect the big jump came from the release of Claude Opus 4.5/4.6 and GPT-5.x-Codex between Nov ‘25 and Feb ‘26, which were trained with heavy reinforcement learning on long coding projects, rewarding only real success (like running code, using terminals, self-fixing bugs, and passing tests) while adding better memory for huge codebases and extra coding-specific training.

Definitely. But I think the nature of that impact is not entirely clear. In the legal context, LLMs are also hallucinating extensively, citing made up case law, etc. It’s not yet clear whether they are potentially solving one problem, while introducing many others.

Fixing a bug is in the wheelhouse of AI to the extent that the fix can be verified — since there is a clear objective function. The real question is whether there are unintended side effects (e.g., new bugs that get introduced) or whether the test cases are comprehensive enough to determine whether the fix worked.

+1 This is the core question to ask.

I suspect that for a nation-state type threat actor, this wouldn’t be much of a deterrent. Any type of reputation system like this would work to a point until motivated threat actors find a way to game it.

It does mean something to me, but perhaps not as profound as whoever coined the term was hoping!

Thanks for sharing. Can’t believe he’s still teaching that class after two and a half decades!

I remember reading that book (along with the Dragon book!)

What a fantastic resource!

What a fantastic resource! Thanks for sharing!

+1

:-) That might not even be enough as I hear (but haven't verified) that Claude does a pretty good job of making sense out of legacy COBOL code!

I'd be curious about what caused the network degradation internally. Incredible promise, but still many kinks that need to be worked out.

Really intriguing set of techniques to improve accuracy by generating multiple solutions. Even with the work to predict the most likely solutions, it's not clear to me based on the description how this could all be done efficiently. Would definitely be really impressive if it pans out on real-world use cases. Will look to kick the tires on this if I can get some time.

Wouldn't this just be a case of the bear catching one guy and then catching the other guy (especially if the issue was unnoticed altogether after the set number of days)?

The minimum-release-age heuristic is certainly helpful as it theoretically gives the community a chance to identify the issue. Of course, in practice, these things aren't scanned or analyzed the way they should ideally be, which is a deeper issue. Pinning has definitely saved me on more than one occasion, but it doesn't strike at the root of the issue.

The least-privilege framing makes sense. That said, a threat actor who understands your model can still craft inputs that have harmful side effects. A real challenge here is defining permissions reactively, because you risk breaking important existing behavior. This is not new in app security, but it gets messier with LLMs.