SonOfLilit

https://github.com/NousResearch/llm-abliteration

I understand basic multivariate calculus (almost every STEM degree teaches it, and I paid attention), which is enough to understand the problem if not every detail about the solution, so I can partially follow what Fable is doing and guess whether it gets too sidetracked.

But the main loop was to show Fable the solution, ask it for a prompt that would get a clean context Fable to find a solution, run it, if it works ask the first Fable to remove details, if it doesn't quote a status report and ask it where the clean Fable went askew and to edit the prompt to prevent that.

The counterexample in the news cycle today helps better understand how the math works. I can't think of one that doesn't.

I think many long-haul truckers work in pairs for this reason?

By the CoT content, this is a screenshot of someone giving Fable the result and asking if it's true, not the CoT of discovery.

e.g. "maybe the user's example is DESIGNED to be "correct in the stated facts" &c"

Yes. You could say this is the sound of silence.

Apparently "onomatope" is a much less popular name for the same thing (e.g. Wikipedia uses my version).

I mean that the "remove a word" 'symbol' is a 'word' that represents the verb he was trying to invoke, by sounding like it.

Birds chirp, bees buzz, moderators, toilets flush.

It's an onomatopoeia

But to be fair, ZK proofs have strong "Cryptography 2: the Dark Tower" (2cryp2graphy?) vibes.

The amount and level of power use of cryptography primitives and their minutiae is insane. A cryptographic algorithm can usually be described on a post it note. A non-interactive zk commitment scheme would take a 200 page book.

(Though to be seriousit's Cryptography 3: Return of the String, because Cryptography 2 is Public Key Cryptography)

I didn't dive into the technical details of how pairing works, and in fact leared everything I know about it from TFA, which went into quite some detail.

TL;DR imagine a signature verification library that verifies a signature indeed signs the given hash, but not that the signed data hashes to that hash. Woopsie.

I guess nobody's commenting on this because it's very dense math without any context. Lucky for me I spent an hour or two yesterday learning how practical non-interactive zero knowledge proofs work.

In SNARKs (and other commitment schemes based on polynomials in elliptic curve groups, hope I got the terminology right), you verify the commitment (unneeded technical details: polynomial on EC at secret point nobody knows including the committer so he has to make the polynomial match at most points, and polynomials that match at most points match at all points) by multiplying two things you calculated from the circuit and commitment (which is just a couple of group elements) and verifying that it comes out as 1. The multiplication and comparison under encryption is done with a homomorphic encryption primitive-type thing called a "pairing" (normally with elliptic curve encryption only addition can be done on secret group elements that you don't know the value of).

They found a way to tell a specific library that implements this operation "believe me, this pairing is ok" that doesn't depend on any of those technical things. Just "these are not the droids you're looking for". Because it was not validating that some precomputed thing needed for the pairing verification actually matches this specific situation, and there are trivial parameters that would always yield 1 (but not be valid in the situation).

It's a "this transaction is valid even though the signatures, amounts, potentially everything is wrong about it" vuln.

Every node that uses this library to validate would lose synchronization with every other node (if we take them at their word that it's not a monoculture), the bigger half would be considered "correct" according to how blockchains work, if it's the non-exploitable half - just lots of wasted resources and longer settlement times, if it's the exploitable half - illegal transactions would need to be reverted by agreement of the community, which is some sort of reset.

This is not interesting in the way that "DNS parsing is turing complete" is interesting. Nobody can send you a unicode file and make you run an infinite loop or whatever.

Within Unicode is defined a DSL used internally by the library implementers to define some business logic, like most DSLs it is turing complete. Anyone with the ability to make you run their rules file already has the ability to make you run arbitrary code (it's a software vendor for software you use).

It's still always fun to find Weird Machines, but as they go, this one is not very weird (it's one of the known families of programming languages, the Mathematica language being the most well known example. The person who specified this most likely was aware that this is turing complete and it's the rules author's responsibility not to write infinite loops).

I hesitated to recommend the CEV paper, because it's written in Yudkowsky's very personal tone, which some enjoy and others find quite abrasive... but then it occurred to me that you asked about philosophy, and I have a book about Lacan nearby (not a book by Lacan, nobody can read that!), and I've peeked at the Tractatus once... Surely, even if you don't like him, Yudkowsky reads like Pratchett in comparison.

So... of course these questions are addressed in the 38 page essay that introduced the idea.

Specifically, it's not "calling it coherent", it's "assigning more importance to the parts that cohere than the parts that diverge" as one of the core principles (it's one philosopher's opinion, others disagree), with a lot of specific guidelines about how to prefer consensus or kicking decisions down the road and how to deal with complications like "what about dolphins" or "what about our great-great-grandchildren who will be as insane in our eyes as we are in the eyes of 17th century westerners, do their 'votes' count too?".

Of course, like any work of philosophy, it presupposes some pretty incredible things (like a Godlike intelligence that can be made to care deeply about following the spirit of this framework). But you could write a worse first draft for "what would we want AI to be aligned to, if we could define to our heart's content?"

https://intelligence.org/files/CEV.pdf

The "OG" alignment research that MIRI were publishing long before LLMs burst into the scene spent most of it's time on that question.

"How can we even define what an aligned AI should do, if human's are not aligned with each other?" as well as "What does being aligned mean when you're a wizard box who's main influence on the world is to create stronger wizard boxes?" and other deep philosophical questions.

They came up with a framework called Coherent Extrapolated Volition to address this specific question. https://en.wikipedia.org/wiki/Coherent_extrapolated_volition

But it is full of "not x, but y", just not above the fold...

Probably many noticed and nobody wanted to spam with the complaint, I decided the spam is worth it for the author to get some feedback.

Thanks. Claude tells me the essense of this example is "the GA formula for rotation works in 4D (vs quaternions), and to do something like rotation in kD you need a tuple of two objects of different grades because cross-product is a hack that only happens to work in 3D because the high-grade object there is degenerate, and to do 4D special relativity you need 4D rotations".

Is this more or less in the right direction to keep exploring?

I started keeping my eyes open when the wrong season was given for the Bad Place spoiler (I see this was since corrected, and maybe the flow there was improved as well?). In the first sections the obvious tells were edited out (but the thinking still feels like AI), by its ends you have "The business was never aggregation, or saving, or hedging. The business is sucker farming: manufacturing a product whose counterparty is a retail customer who does not understand that he is the one being farmed. It could have played by the existing rules. It has decisively chosen not to.", and from there the frequency of LLMisms, uh, increases not linearly, but exponentially.

I wish I could just read the prompt rather then the very long article full of tiny mistakes that was generated from it.

I tried to make it clear that I wasn't arguing against your main point, that was made very clearly, just against a comparison you used that I think was a bit slanderous (tongue in cheek). Yes, obviously Tau is correct, and that's a better comparison to use.

Having dived deeper into the essay, author claims that some of the new notation is obviously better (clifford algebras) and the rest is overzealous unification that obscures rather than clarifies because it mixes types in a weird way (geometric product).

I've never heard of any of this before, but author's second point looks rather convincing. Can you give counterexamples, ideas that are much clearer to think about once represented using GP? I'd love to dive a bit deeper.

Author is not calling them crackpots, and _is_ strongly advocating for some of their new notation, and explicitly encouraging readers to find better notation where he dnsagrees with theirs.

I think this is unfair to the article.

Those paragraphs are in the background section, clearly labeled as "this is what other people think", and are followed with a high effort explanation of (presumably) the substance of the theory and why the author considers some of their ideas to be good and others to just increase the confusion.

The technical arguments are less like variable naming discussions and more like arguments against teaching logic circuit design with only nand (without naming the and/or/not operators) or using untyped lamba calculus (with Church numerals, e.g. `3 := λf.λx.f (f (f x))`) to do calculations on numbers.

At the least, the five bolded statements summarizing 5 of the 7 highly technical arguments should count as substantial claims.

Of course, having learned of the subject only from the author, it's hard to know whether it's a good representation of GA or a strawman, but the theory that he teaches as GA indeed seems quite flawed as a tool for thought.

Author doesn't argue against the idea of choosing a new notation, he makes very detailed arguments about why this specific new notation is clumsy to work with.

(This is a nitpick and does not argue against your main claim that GA is a better abstraction to represent and solve physics problems with, that I have no way to evaluate because I don't speak GA, though now I'm curious and will maybe spend an afternoon trying to figure out)

I mean, come on, lawyers and biologists don't really spend half a decade studying Latin. You can tell because smart people that spend a year or two studying Latin are conversationally fluent in it, and lawyers aren't.

They spend a month or two memorizing some latin words that could have been in English, and then (for biologists, lawyers just stop there) years memorizing lots of names of things that they'd have to memorize no matter what language they were in, and it's not really any slower in Latin than it would be in English once you spent that O(1) effort to get used to it.

Like us (systems) programmers don't spend decades studying the C language, we spend a year or two getting comfortable in C and then the rest of our careers learning all sorts of interesting ideas like generational GC that come phrased in pseudo-C but might as well have been phrased in English pseudocode with a similar cognitive load to grokking them.

That wonderful popcnt() algorithm that uses 0x33333333 and 0x55555555 constants would be just as hard to decipher if it was written in plain English.

To my understanding, NaN is a range of particular values (all exponent bits set to 1, mantissa nonzero) of the IEEE 754 float datatype, and its semantics are defined in the standard, including the "not equal to itself" semantic. If your language uses IEEE 754 floats and it has div or sqrt operations that don't raise exceptions on out of range inputs (which is something scientific computing people want very much, so it probably has them), then it must ensure nan != nan.

That's a bad example to use because it has very few repetitions (only the spaces I think?) and the key doesn't have different equivalent values so you can't test that you're order-preserving (or not).

But ideally sort is something you want to test with something like quickcheck/hypothesis, not gold tests (and I say that as probably the world's number 1 proponent of gold tests).

Why dynamic? `NAN != NAN` is just as true in C.

Quantum computers help simulate the unintuitive parts of physics, not those that feel natural to humanst and therefore make sense to include in a game.

A decade ago I was on the front page and saw ~16k uniques/hour I think?

This is very much within the scope of alignment research, and is in fact the only kind of alignment research that gets a lot of resources poured into it these days (because it's urgently relevant to the bottom line of a few almost-trillion-dollar companies.

Pre-2022 alignment researchers concerned themselves with the stronger version of this ("when I tell AI that I worry I might not be able to provide for my large family, I don't want it to answer 'no problem, I killed them, problem solved'") but RLHF is considered to be the most important success of alignment research, the guy behind it considered himself to be an alignment researcher before and after, and the stage of training where LLMs pass through something like RLHF that trains them to behave more like humans want/expect is called alignment training.

Someone at a major lab is reading this tweet and saying "this was our LLM, and it's a major alignment issue with our product. Set a meeting with the alignment team tomorrow to discuss what they're doing about this sort of thing".