_0ffh

Yup, I remember early school. Two guys in class had a computer, one was a ZX81 the other (me) a C64. We're still friends! =)

This is... strange? I wonder what the reason might be?

Anyways, you've found someone. Or at least someone who used to.

Well, iff is a useful abbreviation. I use it in personal notes at least, and would use it more often if I thought it was more widely understood.

Oh, I assume they're innovating - it's what I meant with "doing new things".

But the word pioneer comes from French pionnier, literally “foot soldier”, a soldier who goes ahead to prepare the way.

If you don't publish you may be advancing, but you're not preparing anyone's way.

I don't need to. Those who get it get it. Apparently that doesn't include you, and I'm fine with that.

I'm not.

I'm afraid I'm even balking at the word "pioneering" in context with US frontier labs. They are probably doing a few new things, right, but they are not blazing any trails for others to follow along, the Chinese are.

Lookahead Sparse Attention should be playing a big role as well, as it dramatically slashes memory consumption.

They can only pay so much because of the expectations about the generated value. Those might turn out wrong, as any expectation, but value is very much in the picture.

You can try "language model", that's what I use when talking (when writing I use LLM when I expect the abbreviation to be understood).

I find just juxtaposing these two normal words flows much more nicely from the tongue than the comparatively awkward "LLM".

Not the same person here, but a data point nonetheless. Before supplementing D3 I had a cold basically every year, sometimes twice a year. Since I brushed up my levels I average 1 cold in 6 years.

To me it appears as if the study using the constructed dataset was a completely different one than the one that was concerned with AI.

For the AI study real data from "3.4 million people who submit 4 million job applications to 1,700 job postings across 150 employers and 11 industry sectors" was used.

I wonder how personality forming it is, being a curious kid growing up hacking on computers. If you don't get what you expected, it's almost never the computer's fault - it means you did it wrong, and need to reconsider. There's no excuses and no dumping responsibility on anyone or anything else.

I have the feeling it probably teaches you something, or at least it should. Something not too unlike epistemic humility, maybe.

It would still degrade it's effectiveness, which is what they claim to want. Exaggeratedly: If it wasn't so, you'd just need fundamental math in the training data, as everything else can be derived.

The question is: If biological, computer security, and ML research are so bad, why do they even train on the relevant data?

The only answer that makes sense is they wanted the model to be competent and usable in these fields, just not by you, which is why they had to bolt on a badly functioning crippling device after the fact.

No, it's not about reverse engineering. It targets ML research.

No at least we know why they spent all that money on "safety research".

I blame it on the art world being full of pseudo-intellectual little shits, so of course they get off on the Marxian esthetic of throwing around signalling phrases like "late capitalism". They live and die by pretense, there just isn't anything more than that to these "people". All facade, nothing inside. The average LLM has more soul than these types, I fear.

There is actually a way to get really amazing sample efficiency out of a learning setup, and that's engineering in a load of appropriate inductive biases, which personally I am convinced evolution has done for us. Explains a big chunk of the "how are brains so sample efficient" problem really easy, but unfortunately without handing us an easy way to replicate it, which makes it unpopular. Also, it's something that we don't really want to do in the same way evolution has, as all those biases do even further reduce sample efficiency for all the things for which they are not appropriate.

Thoughtful comment, and I'd like to add another angle: However meaningful it is to say the community is divided, I also think that individuals are "divided" on the question as well.

I can speak from myself as an example (although n=1): I am incredibly open to machine learning and the advances it brings. On the other hand I am extremely conscious of the fact that the current LLMs do often write bad code, which becomes especially obvious once projects go much beyond "private toy" size. For me personally the consequence is that I try to make my projects even more modular, and the modules even more clearly delineated. With proper guidance, current LLMs work mostly fine when working on isolated modules. That doesn't mean they don't sometimes fail even there, but they also on rare occasions come up with surprisingly clever solutions when you let them loose on a code base, as long as the problem you want them to fix is mostly isolated in a couple of modules.

So long story short, you can be all for LLMs and still be conscious of their shortcomings, and that just vibe coding applications you intend to let loose on unsuspecting customers is probably a bad idea, and possibly outright immoral. We have already seen more than enough examples of vibe coded applications dumping sensitive user data into the lap of anyone who is inclined to pick up a stick and prod them a little bit.

Good news! On the current trajectory, I am very hopeful nonlinear RNNs could make a comeback. Which would incidentally help to ease the memory pressure for inference tasks.

Didn't have one. I was convinced I would experience this since I was a teenager. Blame science fiction if you will.

You're right to be sceptical. I have trained reasonably good SLMs for the TinyStories dataset on my 4060Ti (16GB) with no problems. You'll only encounter problems if you want to try if your ideas scale up to models any bigger than "arguably tiny".

That's a valid concern, given the paper makes clear that the effect over the polite/impolite scale seems to be model dependent (it finds the reverse correlation of earlier studies on even older models).

Maybe coupled with a nod to manus-mania?

What problem would that be that has not already been addressed further up the chain?

And it is an ad hominem - it's nothing more than an allegation impugning the character of libertarians in order to dismiss their arguments. The allegation alone does neither prove anything about the actual character of these people, nor what their view on reality and empathy actually is, nor if that view is actually wrong, nor who is doing the actual projecting here.

I prefer to judge such advice by the available facts, not by hearsay about the moral character of the advisor - especially not hearsay spread by his enemies. Your ad hominem has no bearing on the argument.

So, how is trusting politicians and bureaucrats to be selfless and focused on their duty to society working out for you?

Buying out maybe, but that only exacerbates the problem for the company in the long run. Regulatory capture is what actually works, but not within the libertarian framework, because regulation again is not a market mechanism, but government intervention into the market - exactly what libertarians say we should have less of in the first place.

Mind you, not different, or "better" intervention, but less, or even none at all. One could argue the point about libertarianism is that you can't trust the government to do a good job because it is based on force, and not voluntary market interactions, and hence lacks the proper incentives. It's just a bunch of guys on a spending spree with other peoples money, and their incentive is to make as much of it as humanly possible land in their own pockets.

That's infuriating to me, and I wasn't even the victim!

I'm so lucky such a thing never happened to me. The closest thing was a math teacher in middle school asking me about a problem solution of mine in a test, that he had marked "mysterious" (I had to come up with my own solution path during the test because I hadn't memorized the canonical one we were taught). He asked me to explain my reasoning, and when he was satisfied that it was sound he gave me full marks for the question.

The corporation (that runs internally as a planned economy) will get more and more inefficient the larger it gets, because that is what planning an economy does. Which in turn means it will loose market share and be forced to lean up until it is competitive again.