genrilz

There is plenty of research on social media outcomes. I've looked through some of it before by just searching semanticscholar.org, and the general consensus is that it has both positive and negative effects.

I don't want to have to do a literature review again, and sharing papers is hard because they are often paywalled unless you are associated with a university or are willing to pirate them.

Luckily, The American Psychological Association [0] has shared this nice health advisory [1] which goes into detail. The APA has stewarded psychology research and communicated it to the public in the US for a long time. They have a good track record.

[0] https://en.wikipedia.org/wiki/American_Psychological_Associa...

[1] https://www.apa.org/topics/social-media-internet/health-advi...

One non-obvious strategy is that the number of mines that are left on the field is known. Especially near the end, this can break a tie between two patterns of mines.

IRC the plan isn't to use Polonius instead of NLL, but rather use NLL and kick off Polonius for certain failure cases.

Indeed. Based on the last comment on the tracking issue [0], it looks like they have not figured out whether they will be able to optimize Polonius enough before stabilization, or if they will try non-lexical lifetimes first.

[0] https://github.com/rust-lang/rust-project-goals/issues/118

There isn't a formal definition of how the borrow checking algorithm works, but if anyone is interested, [0] is a fairly detailed if not mathematically rigorous description of how the current non-lexical lifetime algorithm works.

The upcoming Polonius borrow checking algorithm was prototyped using Datalog, which is a logical programming language. So the source code of the prototype [1] effectively is a formal definition. However, I don't think that the version which is in the compiler now exactly matches this early prototype.

EDIT: to be clear, there is a polonius implementation in the rust compiler, but you need to use '-Zpolonius=next' flag on a nightly rust compiler to access it.

[0] https://rust-lang.github.io/rfcs/2094-nll.html

[1] https://github.com/rust-lang/polonius/tree/master

Actually, the x86 'ADD' instruction automagically sets the 'OF' and 'CF' flags for you for signed and unsigned overflow respectively [0]. So all you need to do to panic would be to follow that with an 'JO' or 'JB' instruction to the panic handling code. This is about as efficient as you could ask for, but a large sequence of arithmetic operations is still going to gum up the branch predictor, resulting in more pipeline stalls.

[0] https://www.felixcloutier.com/x86/add

The reason '+ 1' is fine in the example you gave is that length is always less than or equal to capacity. If you follow 'grow_one' which was earlier in the function to grow the capacity by one if needed, you will find that it leads to the checked addition in [0], which returns an error that [1] catches and turns into a panic. So using '+1' prevents a redundant check in release mode while still adding the check in debug mode in case future code changes break the 'len <= capacity' invariant.

Of course, if you don't trust the standard library, you can turn on overflow checks in release mode too. However, the standard library is well tested and I think most people would appreciate the speed from eliding redundant checks.

  [0]: https://doc.rust-lang.org/src/alloc/raw_vec.rs.html#651
  [1]: https://doc.rust-lang.org/src/alloc/raw_vec.rs.html#567

The point I'm sure was to prevent the checks from incurring runtime overhead in production. Even in release mode, the overflow will only wrap rather than trigger undefined behavior, so this won't cause memory corruption unless you are writing unsafe code that ignores the possibility of overflow.

The checks being on in the debug config means your tests and replications of bug reports will catch overflow if they occur. If you are working on some sensitive application where you can't afford logic bugs from overflows but can afford panics/crashes, you can just turn on checks in release mode.

If you are working on a library which is meant to do something sensible on overflow, you can use the wide variety of member functions such as 'wrapping_add' or 'checked_add' to control what happens on overflow regardless of build configuration.

Finally, if your application can't afford to have logic bugs from overflows and also can't panic, you can use kani [0] to prove that overflow never happens.

All in all, it seems to me like Rust supports a wide variety of use cases pretty nicely.

[0] https://github.com/model-checking/kani

If you obscure the implementation a bit, you can change GP's example to a runtime overflow [0]. Note that by default the checks will only occur when using the unoptimized development profile. If you want your optimized release build to also have checks, you can put 'overflow-checks = true' in the '[profile.release]' section of your cargo.toml file [1].

  [0]: https://play.rust-lang.org/?version=stable&mode=debug&edition=2024&gist=847dc401e16fdff14ecf3724a3b15a93
  [1]: https://doc.rust-lang.org/cargo/reference/profiles.html

I think it is important to note that in 59nadir's example, the reason Rust gives an error and Odin doesn't is not memory safety. Rust uses move semantics by default in a loop while Odin appears to use copy semantics by default. I don't really know Odin, but it seems like it is a language that doesn't have RAII. In which case, copy semantics are fine for Odin, but in Rust they could result in a lot of extra allocations if your vector was holding RAII heap allocating objects. Obviously that means you would need to be careful about how to use pointers in Odin, but the choice of moving or copying by default for a loop has nothing to do with this. For reference:

Odin (from documentation):

  for x in some_array { // copy semantics
  for &x in some_array { // reference semantics
  // no move semantics? (could be wrong on this)

  for x in vec.iter_ref().copied() { // bytewise copy semantics (only for POD types)
  for x in vec.iter_ref().cloned() { // RAII copy semantics
  for x in &vec { // reference semantics
  for x in vec { // move semantics

  for (auto x : vec) { // copy semantics
  for (auto &x : vec) { // reference semantics
  for (auto &&x : vec) { // move semantics

I think 0xdeafbeef is roughly recreating the first code snippet from the article (which is one of the things diath is complaining about) in C++ to show that the compiler should produce an error or else undefined behavior could occur on resize.

We get into a bit of a weird space though when they know your opinions about them. I'm sure there are quite a few people who can only build a billion dollar startup if someone emotionally supports them in that endeavor. I'm sure more people could build such a startup if those around them provide knowledge or financial support. In the limit, pretty much anyone can build a billion dollar startup if handed a billion dollars. Are these people capable or not capable of building a building a billion dollar startup.

EDIT: To be clear, I somehow doubt an LLM would be able to provide the level of support needed in most scenarios. However, you and others around the potential founder might make the difference. Since your assessment of the person likely influences the level of support you provide to them, your assessment can affect the chances of whether or not they successfully build a billion dollar startup.

EDIT: A summary of this is that it is impossible to write a sound std::Vec implementation if NonZero::new_unchecked is a safe function. This is specifically because creating a value of NonZero which is 0 is undefined behavior which is exploited by niche optimization. If you created your own `struct MyNonZero(u8)`, then you wouldn't need to mark MyNonZero::new_unchecked as unsafe because creating MyNonZero(0) is a "valid" value which doesn't trigger undefined behavior.

The issue is that this could potentially allow creating a struct whose invariants are broken in safe rust. This breaks encapsulation, which means modules which use unsafe code (like `std::vec`) have no way to stop safe code from calling them with the invariants they rely on for safety broken. Let me give an example starting with an enum definition:

  // Assume std::vec has this definition
  struct Vec<T> {
    capacity: usize,
    length:   usize,
    arena:    * T
  }
  
  enum Example {
    First {
      capacity: usize,
      length:   usize,
      arena:    usize,
      discriminator: NonZero<u8>
    },
    Second {
      vec: Vec<u8>
    }
  }

  fn main() {
    let evil = NonZero::new_unchecked(0);
  
    // We write as an Example::First,
    // but this is read as an Example::Second
    // because discriminator == 0 and niche optimization
    let first = Example::First {
      capacity: 9001, length: 9001,
      arena: 0x20202020,
      discriminator: evil
    }

    if let Example::Second{ vec: bad_vec } = first {
      // If the layout of Example is as I described,
      // and no optimizations occur, we should end up in here.

      // This writes 255 to address 0x20202020
      bad_vec[0] = 255;
    }
  }

I realize I'm butting in on an old debate, but thinking about this caused me to come to conclusions which were interesting enough that I had to write them down somewhere.

I'd argue that rather than thoughts containing extra contents which don't exist in brain states, its more the case that brain states contain extra content which doesn't exist in thoughts. Specifically, I think that "thoughts" are a lossy abstraction that we use to reason about brain states and their resulting behaviors, since we can't directly observe brain states and reasoning about them would be very computationally intensive.

As far as I've seen, you have argued that thoughts "refer" to real things, and that thoughts can be "correct" or "incorrect" in some objective sense. I'll argue against the existence of a singular coherent concept of "referring", and also that thoughts can be useful without needing to be "correct" in some sense which brain states cannot participate in. I'll be assuming that something only exists if we can (at least in theory if not in practice) tie it back to observable behavior.

First, I'll argue that the "refers" relation is a pretty incoherent concept which sometimes happens to work. Let us think of a particular person who has a thought/brain state about a particular tiger in mind/brain. If the person has accurate enough information about the tiger, then they will recognize the tiger on sight, and may behave differently around that tiger than other tigers. I would say in this case that the person's thoughts refer to the tiger. This is the happy case where the "refers" relation is a useful aid to predicting other people's behavior.

Now let us say that the person believes that the tiger ate their mother, and that the tiger has distinctive red stripes. However, let it be the case that the person's mother was eaten by a tiger, but that tiger did not have red stripes. Separately, there does exist a singular tiger in the world which does have red stripes. Which tiger does the thought "a tiger with red stripes ate my mother" refer to?

I think it's obvious that this thought doesn't coherently refer to any tiger. However, that doesn't prevent the thought from affecting the person's behavior. Perhaps the person's next thought is to "take revenge on the tiger that killed my mother". The person then hunts down and kills the tiger with the red stripes. We might be tempted to believe that this thought refers to the mother killing tiger, but the person has acted as though it referred to the red striped tiger. However, it would be difficult to say that the thought refers to the red striped tiger either, since the person might not kill the red striped tiger if they happen to learn said tiger has an alibi. Hopefully this is sufficient to show that the "refers" relationship isn't particularly connected to observable behavior in many cases where it seems like it should be. The connection would exist if everyone had accurate and complete information about everything, but that is certainly not the world we live in.

I can't prove that the world is fully mechanical, but if we assume that it is, then all of the above behavior could in theory be predicted by just knowing the state of the world (including brain states but not thoughts) and stepping a simulation forward. Thus the concept of a brain state is more helpful to predicting their behavior than thoughts with a singular concept of "refers". We might be able to split the concept of "referring" up into other concepts for greater predictive accuracy, but I don't see how this accuracy could ever be greater than just knowing the brain state. Thus if we could directly observe brain states and had unlimited computational power, we probably wouldn't bother with the concept of a "thought".

Now then, on to the subject of correctness. I'd argue that thoughts can be useful without needing a central concept of correctness. The mechanism is the very category theory like concept of considering all things only in terms of how they relate to other things, and then finding other (possibly abstract) objects which have the same set of relationships.

For concreteness, let us say that we have piles of apples and are trying to figure out how many people we can feed. Let us say that today we have two piles each consisting of two apples. Yesterday we had a pile of four apples and could feed two people. The field of appleology is quite new, so we might want to find some abstract objects in the field of math which have the same relationship. Cutting edge appleology research shows that as far as hungry people are concerned, apple piles can be represented with natural numbers, and taking two apple piles and combining them results in a pile equivalent to adding the natural numbers associated with the piles being combined. We are short on time, so rather than combining the piles, we just think about the associated natural numbers (2 and 2), and add them (4) to figure out that we can feed two people today. Thus the equation (2+2=4) was useful because pile 1 combined with pile 2 is related to yesterday pile in the same way that 2 + 2 relates to 4.

Math is "correct" only in so far as it is consistent. That is, if you can arrive at a result using two different methods, you should find that the result is the same regardless of the method chosen. Similarly, reality is always consistent, because assuming that your behavior hasn't affected the situation, (and what is considered the situation doesn't include your brain state) it doesn't matter how or even if you reason about the situation, the situation just is what it is. So the reason math is useful is because you can find abstract objects (like numbers) which relate to each other in the same way as parts of reality (like piles of apples). By choosing a conventional math, we save ourselves the trouble of having to reason about some set of relationships all over again every time that set of relationships occurs. Instead we simply map the objects to objects in the conventional math which are related in the same manner. However, there is no singular "correct" math, as can be shown by the fact that mathematics can be defined in terms of set theory + first order logic, type theory, or category theory. Even an inconsistent math such as set theory before Russell's Paradox can still often produce useful results as long one's line of reasoning doesn't happen to trip on the inconsistency. However, tripping on an inconsistency will produce a set of relationships which cannot exist in the real world, which gives us a reason to think of consistent maths as being "correct". Consistent maths certainly are more useful.

Brain states can also participate in this model of correctness though. Brain states are related to each other, and if these relationships are the same as the relationships between external objects, then the relationships can be used to predict events occurring in the world. One can think of math and logic as mechanisms to form brain states with the consistent relationships needed to accurately model the world. As with math though, even inconsistent relationships can be fine as long as those inconsistencies aren't involved in reasoning about a thing, or predicting a thing isn't the point (take scapegoating for instance).

Sorry for the ramble. I'll summarize:

TL;DR: Thoughts don't contain "refers" and "correctness" relationships in any sense that brain states can't. The concept of "refers" is only usable to predict behavior if people have accurate and complete information about the things they are thinking about. However, brain states predict behavior regardless of how accurate or complete the information the person has is. The concept of "correctness" in math/logic really just means that the relationship between mathematical objects is consistent. We want this because the relationships between parts of reality seem to be consistent, and so if we desire the ability to predict things using abstract objects, the relationships between abstract objects must be consistent as well. However, brain states can also have consistent patterns of relationships, and so can be correct in the same sense.

Because they boxed themselves in with legalese. Companies would definitely switch off Microsoft services if at all possible if the company's lawyers thought their trade secrets were getting sold off. So I think the "as necessary" framing does probably prevent them from doing some things.

As I laid out in my other comment, I think training AI in particular is covered under the "improving Microsoft products or services" bit of legalese. I do wonder how companies lawyers will respond to this though. They probably thought of that phrase as just allowing Microsoft employees access to documents to see how Word or other pieces of software were being used, or to fix crashes, etc.

IANAL again, but I don't think they get to do literally anything with your data. The phrase used is "to the extent necessary". For instance, I don't think they could scrape their user data for trade secrets and then sell those to the highest bidder.

IANAL, but I think the "to improve Microsoft products and services" bit does mean that they do legally get to train their AI (which is a Microsoft service) on your data. Still a bastard move though.

Problem with memory corruption as a bug is that unlike most classes of bug, memory corruption allows remote code execution. (see return oriented programming for the basic version, block oriented programming for the more complex version that bypasses most (all?) mitigation strategies) There are other types of bug that allow remote code execution like this, such as SQL or command-line injection, but those can be solved with better libraries.* However, memory management requires a strong enough type system in the language.

* Sorta, for command-line injection you have to know the way the command you are using processes flags and environmental variables in order to know that the filtering you are doing will work. It is absolutely better to use a library instead if you can get away with it.

One, iterator invalidation can be a temporal safety problem. Specifically, if you have an iterator into a vector and you insert into the same vector, the vector might reallocate, resulting in the iterator pointing into invalid memory.

Two, consider the unique_ptr example then. Perhaps a library takes a reference to the contents of the unique_ptr to do some calculation. (by passing the reference into a function) Later, someone comes along and sticks a call to std::async inside the calculation function for better latency. Depending on the launch policy, this will result in a use after free either if the future is first used after the unique_ptr is dead, or if the thread the future is running on does not run until after the unique_ptr is dead.

EDIT: Originally I was just thinking of the person who inserted the std::async as being an idiot, but consider that the function might have been templated, with the implicit assumption that you would pass by value.

I'm kind of worried about society deciding which speech is "intolerant", so I'm not completely on board with the idea of treating tolerance as a social contract. That being said, if we could stop a genocide merely by suppressing people's speech, I feel like that would probably be a worthwhile thing to do. That is to say, it feels like the least bad way to prevent a genocide.

Again, figuring out which speech is worth suppressing is a whole other can of worms.

EDIT: note that Jones did have his speech suppressed, and this was done because his speech was causing people to make death threats against the sandy hook parents. I feel like we could classify Jones's speech as intolerant against sandy hook parents, and the same logic applies as for any other type of intolerant speech.

I think it's actually closer to "terrorists should go to prison". Terrorists and other criminals have broken a social contract, and a level of punishment that some approximation of society deems to be acceptable is extracted from the terrorists. This doesn't mean that terrorists don't/shouldn't have some rights. Similarly, thinking about tolerance as a social contract doesn't require stripping anyone who violates this contract of all of their rights.

You might be right, but my first instinct is that this probably wouldn't happen enough to throw off the water marking to badly.

The most likely used word is based off the previous four, and only works if there is enough entropy present that one of multiple word would work. Thus its not a simple matter of humans picking up particular word choices. There might be some cases where there are 3 tokens in a row that occur with low entropy after the first token, and then one token generation with high entropy at the end. That would cause a particular 5 word phrase to occur. Otherwise, the word choice would appear pretty random. I don't think humans pick up on stuff like that even subconsciously, but I could be wrong.

I would be interested to see if LLMs pick up the watermarks when fed watermarked training data though. Evidently ChatGPT can decode base64, [0] so it seems like these things can pick up on some pretty subtle patterns.

[0] https://www.reddit.com/r/ChatGPT/comments/1645n6i/i_noticed_...

There are situations where the model output being watermarked doesn't matter. For instance, I hear people on HN asking LLMs to explain things to them all the time, (which I think is a bad idea, but YMMV) and people use LLMs to write code quickly. (which I think is at least possibly a good idea) There are also some content farms which churn out low quality books on Amazon on various topics, and I don't think they care if they get caught using LLM outputs.

Thus it might reduce usage some, but it certainly wouldn't block all usage. Additionally, there are only a few providers of truly massive LLMs on the market right now. If they decided that doing this would be a social good, or more likely that it would bring bad PR to not do this when their competitors do, then they would at least be able to watermark all of the massive LLM outputs.

Or easier: Use Llama or some other open weights model with a non-watermarking sampler.

It's possible that this might break the method, but what seems most likely to me is that the LLM will simply reword every 5th word with some other word that it is more likely to use due to the watermark sampling. Thus the resulting output would display roughly the same level of "watermarkedness".

You might be able to have one LLM output the original, and then another to do a partial rewording though. The resulting text would likely have higher than chance "watermarkedness" for both LLMs, but less than you would expect from a plain output. Perhaps this would be sufficient for short enough outputs?

My understanding from the "Watermark Detection" section is that it only requires the key and the output text in order to do the detection. In particular, it seems like the random seed used for each token is only based off the previous 4 tokens and the LLM specific key, so for any output larger than 4 tokens, you can start to get a signal.

I don't think the key actually needs to be secret, as it's not trying to cryptographicly secure anything. So all closed weights LLM providers could just publicly share the keys they use for water marking, and then anybody could use them to check if a particular piece of text was generated by a particular LLM.

That being said, I think you are right about this only really being useful for closed weights models. If you have the weights, you can just run an LLM through a standard sampler and it won't be watermarked.

A part of an answer:

Lets assume LLMs don't "think". We feed an LLM an input and get back an output string. It is then possible to interpret that string as having meaning in the same way we interpret human writing as having meaning, even though we may choose not to. At that point, we have created a thought in our heads which could be true or false.

Now lets talk about calculators. We can think of calculators as similar to LLMs, but speaking a more restricted language and giving significantly more reliable results. The calculator takes a thought converted to a string as input from the user, and outputs a string, which the user then converts to a thought. The user values that string creating a thought which has a higher truthiness. People don't like buggy calculators.

I'd say one can view an LLM in exactly the same way, just that they can take a much richer language of thoughts, but output significantly buggier results.

Additionally, I think you may or may not be suspecting research malpractice. Obviously I don't have insider knowledge, but I would note that the idea of training probes in the middle layer of the model wasn't their idea. This paper cites other papers that already did exactly that. The contribution of this paper is simply that focusing on the middle layers for certain "critical tokens" gives a better signal than just checking the middle layers on every token.

It's of course possible that this paper in particular is fraudulent, but note that there is a field of research making the same basic claim as this paper, so this isn't some one off thing. A reasonable amount of people from different institutions would need to be in on it for the entire field to be fraudulent.

Alternatively, I think you may be objecting to the use of the word "truthfulness" in the abstract of the paper, because you seem to think that only human thoughts can possibly have a true or false value. I'm not actually going to object to the idea that only human thoughts can be true or false, but like the response I wrote to your koan comment, the user can interpret the LLMs output, which gives the user's thought a true or false value.

In this case, philosophically, you can think of this paper as trying to find cases where the LLM outputs strings that the user interprets as false. I think the authors of the paper are probably thinking about true or false more as a property of sentences, and thus a thing mere strings can possess regardless of how they are created. This is also a philosophically valid way to look at it, but differs from your view in a way that possibly made you think their claims absurd.

To me the claims in the article read something like "we have found a way to identify execution paths in some common compiler architecture (which are the transformer architecture in the case of LLMs) which are often but not always associated with buggy code". This seems like a reasonable claim to make.

This method uses "critical tokens", which I don't think you can detect until after you've generated an entire response. Using them as part of the search function seems infeasible for long outputs, but technically possible. Using it on single token outputs seems imminently feasible and like a cool research direction.

I think the paper itself demonstrates that the model has something internally going on which is statistically related to whether it's answer is correct on a given benchmark. Obviously, the LLM will not always be perfectly accurate about these things. However, let's say you are using an LLM to summarize sources. There's no real software system right now that signals whether or not the summary is correct. You could use this technique to train probes to find if a human would agree that the summary is correct, and then flag outputs where the probes say the output wouldn't agree with a human for human review. This is a lot less expensive of a way to detect issues with your LLM than just asking a human to review every single output.

While we don't have great methods for "curing it", we do have some. As I mentioned in a sibling post, contextual calibration and adding/adjusting training data are both options. If you figure out the bug was due to RAG doing something weird, you could adjust your RAG sources/chunking. Regardless, you can't put any human thought into curing bugs that you haven't detected.

You might not be able to sell someone a library that fixes all bugs, but you can sell (or give away) software systems that reduce the number of bugs. Doing that is pretty useful.

Examples include linters, fuzzers, testing frameworks, and memory safe programming languages (as in Rust, but also as in any language with a GC). All these things reduce the number of bugs in the final product by giving you a way to detect them. (except for memory safe languages, which just eliminate a class of bugs) The paper is advertising a method to detect whether a given output is likely to be affected by a "bug", and a taxonomy of the symptoms of such bugs. The paper doesn't provide a way to fix those, and hallucinations don't necessarily have a single cause. Some hallucinations might be fixed by contextual calibration [0], others might be fixed by adding more training data similar to the wrong example.

In any case, you need to find the bad outputs before you can perform any fixes. Because LLMs tend to be used to produce "fuzzy" outputs with no single right answer, traditional testing frameworks and the like aren't always applicable.

[0] https://learnprompting.org/docs/reliability/calibration