HN user

utilize1808

70 karma
Posts0
Comments56
View on HN
No posts found.

Also, in that case, there would likely be activations indicating that it is favoring a specific version.

Maybe you can illustrate a realistic scenario in which that would be a problem, otherwise I don't really understand what your point is in this context.

I doubt it. Can you definitively prove that you can reliably detect the kind of threat I described when model weights are released? Can you be sure that your detector won't miss *any* such sleeper attacks? If not, then that's a threat that will be used to justify the ban of models (open or not) that is not sanctioned by the US government. A model being open doesn't make a difference here.

The model is conditioned / pretrained to use a particular version of a library. The model doesn’t know why — it was just taught to use that version. To the model, it is just some insignificant detail in the grand scheme of things. It’s just like how a model would intuitively favour using English without explicit instructions — it’s not trying to sabotage other cultures, it’s just what it does.

They can just favour some specific versions of some library that's been compromised. Unlike introducing bugs / flaws directly in the source code, they can claim plausible deniability, and it's much easier to implement without compromising the general coding capabilities of the models.

Our dev arm (me included) do use Linux VDI. But most day-to-day business are done in Microsoft's ecosystem (Teams, SharePoint, Office, Copilot, etc). In the industry I am in, you would never be able to use any AI tools unless the trusted platforms offer them as part of their products (i.e. Microsoft's Copilot or Amazon's Bedrock) because InfoSec and Legal departments won't risk authorizing any other providers.

Are you more deterministic than an instruction-following black-box? I doubt it. Also, such organization practice mostly becomes a mechanical routine after the initial experimentation phase. Good riddance, I'd say.

I think for this kind of system to work, there has to be SOME kind of public/shared server to do the coordination. If the inviting node is behind a firewall then no amount of information can enable a guest node to connect to it without a node reachable by both.

Yes, that's what I meant. It's not that the Chinese will never be able to come up with models that superior --- it's simply that they will no longer have the incentive to open-source them once their models take the lead.

Training models are expensive. No one can do it for free for very long.

While that might be true, it is unlikely that open source models' capability will ever surpass frontier models --- if you have the best model (by some margin), then people will want to access it, even if it means going through compliance.

The government will just claim that unsanctioned models have the potential to deliberately introduce security vulnerabilities when working on IT projects (e.g. be trained to strongly yet covertly favoring introducing compromised dependencies when you are not looking).

Then laws will be made to forbid organizations who use models other than those from the sanctioned labs to participate in critical projects on national security concerns.

All of a sudden, no business would risk using open source models anymore.

You sweat because you are working with the CLI. Git is intrinsically "graphical". Use a good GUI client or higher level interface (maybe jj) to manipulate git graphs --- stop worrying about "how" (i.e. wrangling with CLI to achieve what you want) and focus more on "what".