mcyc

Lichess has a checkmate captcha that I think is cute.

It requires you to solve a mate-in-one puzzle to, e.g., post on the forums.

(Sorry, don't have a better link, there wasn't any non-technical I could find about it).

https://www.reddit.com/r/chess/comments/q19wgq/til_lichess_d...

Yeah, it is interesting to me that it is coming from the _city_'s government. I've seen sovereign AI things at the country level, but this is the first municipal one I have seen.

You are right about most tokenizers being heavily biased towards English, but the situation is not so bad for Portuguese. Here are some results on the Goldfish corpus [1] with a few different tokenizers. This measures #characters in corpus / #subwords in tokenized corpus.

```

Llama3

english, 0.216

portuguese, 0.285

italian, 0.287

greek, 0.592

```

```

Gemma4

english, 0.219

portuguese, 0.246

italian, 0.249

greek, 0.537

```

```

Kimi2.6

english, 0.214

portuguese, 0.310

italian, 0.308

greek, 0.716

```

Portuguese is worse than English certainly, but it is on par with Italian (which I think has more overlap with English) and much better than Greek (since it doesn't use the Latin script and is definitely not prioritized in the tokenizer construction).

On your second point, tokenizer transfer allows for extending/modifying a tokenizer without retraining the model from scratch. The simplest version of this is tokenizer extension + continual pretraining, where you just add a bunch more tokens to the vocab for the language/domain that you want to improve and train a little more. It's been done for Japanese [2] and Indic languages, but afaik not Portuguese.

So I think that continual pretraining for a large base model would have probably been fine for this case with huge cost savings. But it is good to have the ability to train your own base models, so I don't think this is such a bad idea.

-----------------------

[1] https://huggingface.co/datasets/goldfish-models/fish-food

[2] https://arxiv.org/abs/2404.17790

This is a fantastic guide! I did a lot of work on structured generation for my PhD. Here are a few other pointers for people who might be interested:

Some libraries:

- Outlines, a nice library for structured generation

  - https://github.com/dottxt-ai/outlines

  - https://github.com/guidance-ai/guidance

  - https://github.com/mlc-ai/xgrammar

  - This one has a lot of cool technical aspects that make it an interesting project

- Efficient Guided Generation for Large Language Models

  - By the outlines authors, probably the first real LLM constrained generation paper

  - https://arxiv.org/abs/2307.09702

  - A much more technical paper about constrained generation and implementation

  - https://arxiv.org/abs/2407.08103

  - A bit of self-promotion. We show where constrained generation can go wrong and discuss some techniques for the practitioner

  - https://openreview.net/pdf?id=DFybOGeGDS

- Fast, High-Fidelity LLM Decoding with Regex Constraints

  - Discusses adhering to the canonical tokenization (i.e., not just the constraint, but also what would be produced by the tokenizer)

  - https://vivien000.github.io/blog/journal/llm-decoding-with-regex-constraints.html

  - Also from the outlines team

  - This is about skipping inference during constrained generation if you know there is only one valid token (common in the canonical tokenization setting)

  - https://blog.dottxt.ai/coalescence.html

This is a nice attitude. I think HN is overall pretty nice for geeking out and also hearing other people geek out, but there is still a strain of elitism (not like StackExchange thankfully) and so I'm happy to see comments like this.

You can cross whitespace boundaries by setting flag `--split-on-whitespace` to false (it's true by default).

https://github.com/google/sentencepiece/blob/master/doc/opti...

Just a minor nit: SentencePiece is a library, not a tokenization algorithm. It implements two tokenization algorithms, Unigram and BPE.

BPE builds vocabularies from the base up so I assume you are talking about Unigram which starts with a big vocabulary and trims it.

The details of UnigramLM are here https://arxiv.org/pdf/1804.10959, and the part about vocabulary seeding is Section 3.2.

Basically, it just selects all substrings that appear in the corpus up to a certain length (and then maybe trims it a little by discarding rare substrings or something to reduce the initial size a bit and make things faster).

People may also be interested in Pynini [1], a python wrapper (+ a lot of additional ease-of-use functionality) of OpenFst [2] (a really great library for transducers).

There are some good tutorials in the form of homework assignments (from like Johns Hopkins and some others) that go through Pynini use cases.

[1] https://www.openfst.org/twiki/bin/view/GRM/Pynini

[2] https://www.openfst.org/

You might be interested in the Recurse Center (https://www.recurse.com/) and the experiences of people who have gone through it (they heavily encourage blogging about your time there so there is lots to read).

Note: I am not affiliated with the Recurse Center, just a big fan.

Thanks!

Our paper [1] is kind of a goofy adversarial thing where we thought "here's this cool metric, how can we break it?". The tokenizers we propose are definitely not tokenizers you should use in practice.

The original paper that proposes the metric is, imo, much more interesting theoretically [2].

[1] https://aclanthology.org/2024.lrec-main.1469/

[2] https://aclanthology.org/2023.acl-long.284/

It's two problems:

1) the sequence length increases too much. Idk what the average token length is for Llama, but imagine it's like 5+ bytes. Using individual bytes as tokens immediately makes the context 5x longer which is super bad for inference speed and memory requirements (since attention inference is quadratic in the length of the sequence).

2) individual bytes have essentially no meaning, so byte embeddings are harder to learn. Subword tokens aren't a perfect solution, but they definitely often have some standalone meaning where embeddings make sense.

I'll give another example from a recent paper that tries to eliminate tokenizers (this is a popular research direction) [1].

Figure 4 is a really good example of why byte-level models are wasting computation. Once part of a word is generated, most of the remaining bytes are assigned basically probability 1. But a byte-level model would still have to spend time decoding them. With a subword-level model most of these easy-to-decode bytes would be packed together in a single token so you don't have to decode them individually.

When model APIs bill by the token, this is an important consideration.

[1] https://arxiv.org/abs/2412.09871

NB: Can't edit my original reply.

Sorry actually I misread part of your comment in relation to the paper and confused δ and another parameter, K.

To clarify, δ is the number of tokens in the tokenized corpus and K is the size of the vocabulary.

So, if you are asking about why would they limit _K_, then my answer still applies (after swapping δ for K). But if you still mean "why do they pick some arbitrary δ as the limit of the size of the tokenized corpus", then I think the answer is just "because that makes it a decision problem".

Hi, I'm Cognetta from the above Cognetta et al. I can't answer all of your questions (and I can't speak for the authors of this paper ofc), but I will try to answer some.

Is a tokenizer that maximizes the compression of text (e.g. by identifying longer tokens that tend to be used whole) necessarily a better tokenizer, in terms of overall model performance? Compression might be a useful property for an objective function to consider... but then again maybe not, if it makes the problem NP-hard.

Compression isn't necessarily the best metric for language modeling quality [1][2][3], but there are some papers that find a correlation between it and quality [4] and also it has one important benefit: it reduces inference time by making the input sequences shorter (this is particularly important for transformers, because the runtime is quadratic in the sequence length).

If you imagine that with enough data, basically any reasonable tokenization algorithm would be ok (I think this is mostly true; there are definitely bad and "better" tokenizers and you see this very clearly in small data settings, but once you get into the trillions-of-tokens and 10s-of-billion-of-parameters setting, other things are going to matter more), then optimizing the tokenizer for compression is a good choice as it will provide tangible, practical benefits in the sense of reduced inference time.

I'm also not sure how realistic the limitation to "at most δ symbols" is. [...] But why not just keep adding tokens as needed, rather than imposing any preordained limit?

This is a pretty realistic limitation imo. Of course you can arbitrarily increase the vocabulary size, but there is a tradeoff between modeling quality, parameter count, and inference time. If you increase the vocabulary a bunch, your inference speed will probably improve (although now you have a much larger softmax at the end of your model, which isn't usually a bottleneck anymore, but still not great), parameter count will increase (due to the larger embedding table), and your modeling quality will go down (in that you have tokens which are so rare in the corpus that they are massively undertrained; this can cause big problems [5]).

So by constraining it to δ, you are basically setting a parameter budget for the vocabulary, and this is a pretty reasonable thing to do.

IIRC OpenAI's tokenizer has a vocabulary of around 52k subword strings.

Yeah, the size of the vocabulary varies a lot across models, but it isn't unusual to see significantly larger vocabularies these days (e.g., gemma has ~256k). However, these are still finite and very small compared to the corpus size.

How could you possibly choose a meaningful δ from first principles?

This is a really great question, and something that we don't know how to answer. A lot of work has tried to answer it [6][7], but it is very much an open question.

[1] https://arxiv.org/abs/2310.08754

[2] https://aclanthology.org/2023.acl-long.284/

[3] https://aclanthology.org/2024.emnlp-main.40/

[4] https://arxiv.org/abs/2403.06265

[5] https://aclanthology.org/2024.emnlp-main.649/

[6] https://aclanthology.org/2023.acl-long.284/

[7] https://aclanthology.org/2020.findings-emnlp.352/

It's just a convention for SIMD functions and types.

Suggested projects for the AT Protocol from the Bluesky devs.

A monthly writing challenge. See also: https://alpha.polymaths.social/@amin/statuses/01JBB8JKJMZ5ES...

A bit late, but I wrote up an example inference script here: https://gist.github.com/mcognetta/7a98e50859664b8efbb4ec094a...

It is a bit roundabout, since it involves converting maia models to onnx before loading into pytorch and some outdated versions of libraries (maia/lc0 are a little old). We were using this for transfer learning for a competition, so we needed some flexibility that we didn't know how to do quickly/easily in TF.

Hope this helps.

------------------

Personal note: given your interest in chess ai and your starcraft username, I think we would have a lot of shared interests. Feel free to reach out (info is in my profile).

See also: https://landing.google.co.jp/double-sided/ (in Japanese)

You may enjoy this article: https://arxiv.org/abs/2109.08203

The author treats the seed as a hyperparameter and searches for the one that performs best for training a CV model.

If you are interested in this field (exploring the limits of neural models using formal language theory), I help run a weekly seminar on it, Formal Languages and Neural Networks:

https://flann.super.site/

We have had many great speakers (most of them are recorded and available on YouTube) and have a welcoming Discord.

The Dean responded in the comments of this post.

https://statmodeling.stat.columbia.edu/2024/02/06/its-bezzle...

Ah, the other color schemes look really good (especially I like the heatmap one). However, the default stoplight one is nearly impossible for me to disambiguate. For example, in [1], I can't tell which of mine are closer at all (I don't think I am colorblind, but maybe I'm just in for a surprise today).

Anyway, overall, a very fun variant. Thanks for sharing!

[1] https://shorturl.at/cBJQ1

Interpolating with the game's specified final green value (34, 139, 34) "forest green" provides a nicer effect than true green (0, 255, 0).

https://twitter.com/good_in_theory/status/175088137157541934...

I also found the color scheme to be difficult to understand. A friend suggested interpolating between red and green depending on how close you are to the correct key. This is not too hard, since red is (255, 0, 0) and green is (0, 255, 0), so you can compute a distance (normalized to [0, 1]) and output (255 x d, 255 x (1-d), 0) to get the interpolated color.

It looks quite nice visually.

I wrote a small thread on it: https://twitter.com/good_in_theory/status/175079370720771734...

https://sigmoid.social/@mc/111821291510126156

Your videos are absolutely fantastic, thanks for making them!

SD1 uses a head dimension that is not supported by the old version of Flash Attention but is supported by this version.

The mean transduction in the sense of sequence-to-sequence models (transducing one sequence to another).

Attention based models don't necessarily need to be sequence to sequence. They can be classifiers, decoder only, etc. Attention is just one tool in the ML architecture toolkit.

While epsilon transitions often, but not always, cause this issue of not being able to complement the language by just flipping final/non-final states, it isn't always the case. For example, you can make an ambiguous NFA with no epsilon transitions by just doing epsilon removal, and then you still have the same problem but no non-trivial epsilon-reachable states!

On the other hand, you can have an unambiguous [1] NFA (for example, a DFA, but you insert some dummy epsilon transitions between a split up state) where you can just flip the final/non-final states and complement the language.

So, in the end, complementing languages described by NFAs without determinizing them first is a bit of a tricky problem.

[1] https://en.wikipedia.org/wiki/Unambiguous_finite_automaton --- a superset of DFAs, but they can have epsilon transitions

The parent comment asked specifically about negating the finality of each state, which I addressed. As for how to actually implement complement in a regex engine, there are certainly some strategies like the one you mentioned that could be used (but the point of my comment was that just swapping final/non-final states is not a valid one).

It's because this produces NFAs not DFAs (due to the epsilon arcs), and so negation can't be done by just flipping the final states.

Take, for example, the regex "a|(aa)+" (the set of even length strings of "a"'s or just one "a"). If you use the construction from the article, you get an NFA with basically two arms: one that recognizes "a" and one that recognizes "(aa)+". The initial state contains an epsilon arc to the start state of each of these arms. If you just flip the final/non-final states in each arm, the resulting language contains "a", since we are no longer accepting each even length "a" string, but now each odd length "a" string, of which "a" is a member. Thus the new language is not the proper complement of "a|(aa)+", which would not contain "a".