matthewolfe

What exactly doesn't "just work" anymore?

I believe SchemeFlow [0] is working on solving some of these problem, particularly with the insane reporting requirements. But of course, that still leaves the unions...

[0] https://www.schemeflow.com/

For argument's sake, suppose we live in a world where many high-quality models can be run on-device. Is there any concern from companies/model developers about exposing their proprietary weights to the end user? It's generally not difficult to intercept traffic (weights) sent to and app, or just reverse the app itself.

To echo the other replies, the tokenizer is definitely not the bottleneck. It just happens to be the first step in inference, so it's what I did first.

haha, I thought about it.

Fair chance I'm remembering it wrong :D

Alright, 0.1.1 should now be a true drop-in replacement. I'll write up some examples soon.

Should be the same. Both use Byte-Pair Encoding (BPE) as underlying algo.

Ah good catch. Updating this right now.

Modal's GPU glossary is a good overview about how GPUs work [0]. Karpathy's LLM overview is a good high level overview on LLMs [1]. 3b1b's video (and subsequent videos) on transformers was excellent at helping me understand the math at a high level [2]. This matrix multiplication optimization worklog helped me understand writing better CUDA (not for beginner intro though) [3].

During this process I also asked ChatGPT a lot of questions.

I'm definitely open to suggestions about "how to learn" with all the new tools we have. I felt this has not been straightforward to figure out.

[0] https://modal.com/gpu-glossary

[1] https://www.youtube.com/watch?v=7xTGNNLPyMI

[2] https://www.youtube.com/watch?v=wjZofJX0v4M

[3] https://siboehm.com/articles/22/CUDA-MMM

I'm working on incremental re-tokenizing next. Then I'll run some benchmarks against this crate too.

Agreed. A former mentor of mine told me a nice way of viewing software development:

1. Make it work. 2. Make it fast. 3. Make it pretty.

Transformers & LLMs have been developed to a point where they work quite well. I feel as though we're at a stage where most substantial progress is being made on the performance side.

Cool!

I've reached out to the guy who maintains Tiktoken to talk about this.

A lot of model-specific tokenizers have reference implementations ([0], [1]). Underlying them is a core algorithm like SentencePiece or Byte-pair encoding (BPE). Tiktoken and TokenDagger are BPE implementations. The wrapping "tokenizer" mostly deals with the quirks of the vocabulary and handling special tokens.

For this project, I think there is value in building some of these model-specific quirks into the library. Could see some minor performance gains and generally make it easier to integrate with. It's probably not too much work to keep up with newer models. Tokenizers change much less frequently.

[0] https://github.com/meta-llama/llama-models/blob/01dc8ce46fec...

[1] https://github.com/mistralai/mistral-common/tree/main/src/mi...

I probably will. Was hesitant initially, because adding PCRE2 as a dependency might cause issues to existing projects. I believe this was discussed briefly in a closed PR with other performance improvements.

The output should be identical, assuming no bugs.

The Tiktoken implementation takes a collection of all special tokens upon initialization and compiles them into a regex by joining them with `|` [0]. Then the actual encoding process checks for matches on this expression.

Models like Llama 4 define a list of 1,135 special tokens. Notably, 1,115 of those are "reserved" special tokens! So this yields a huge regexp of special tokens that shouldn't be considered at all.

TokenDagger does not do this. Instead, simple string matching is used. This works because we don't need to consider the entire special vocabulary every time. The caller of `encode` must explicitly define which special tokens should be considered [1]. So it's faster to check against the much smaller list we _know_ is being used.

[0] https://github.com/openai/tiktoken/blob/main/src/lib.rs#L476

[1] https://github.com/openai/tiktoken/blob/main/tiktoken/core.p...

Thanks, I clarified it.

Agreed. I figured nobody would use it otherwise.

I'm working on TokenDagger [0] a high performance implementation of OpenAI's Tiktoken. My benchmarks are showing 2-3x higher throughput, as well as ~4x faster tokenization for code samples on a single thread.

[0] https://github.com/M4THYOU/TokenDagger

I do this a lot for my work. A tool like this that can help get me to a nice starting point is huge. Instead of developing a mental model of the API in my head by manually looking through API requests/responses in ProxyMan, this can start me off much more quickly. From there, the edge cases can be worked out.

Great work, guys!

you'd have a very hard time arguing that making OSB illegal would only marginally impact the amount of sports gambling taking place.

"Offshore sportsbooks" are another thing to consider. These companies are not regulated in the US market, but still take online bets from people in the US. The ease of placing bets online does not go away with making OSB illegal. Just eliminates any consumer protection we could have had.

I also work in industry. I think a national self-exclusion scheme (like the UK has) would be huge.

I'm the founder of BeeBettor (YC S24). I've been working in this space for a while.

A lot of the points in the article are valid. I have two major issues with online sports betting (OSB) in the US.

1. Sports betting advertising before, during, and after games is horrendous. There is no way to watch sports without being bombarded. Obviously, this is a huge issue for problem gamblers. Sports become unwatchable.

2. Self-exclusion is impossible. There's 40+ sports betting apps available. There is no centralized body a person can say "hey don't let me bet anymore" and then be automatically restricted from betting across all apps. This is something I think we can help with in the near future.

So what can be done now? I don't think OSB is going to be redeclared illegal. I don't think that would be a good idea either. Millions of people have started sports betting. If it becomes illegal, it won't make them stop.

Happy to discuss this further. Email is in my profile.

Location: Waterloo, Ontario, Canada

Remote: Sure

Willing to relocate: Yes

Technologies: Ruby, Python, Go, C++, and plenty of other things

Résumé/CV: https://drive.google.com/file/d/1W2A58OSBQD9gJipIiTjXkFdBKmQ...

LinkedIn: https://www.linkedin.com/in/mathyou-wolfe

Email: m5wolfe [at] uwaterloo.ca

I'm a university student looking for Summer 2022 internships. I'm currently interning at Shopify. All I want to do is build really cool things :)

Also, if you want to just have a chat about anything, I would love to!

intractable for me, at least

almost all of them required skills well outside of my wheel house.

It seems like you have found good problems to address. Why not just try to learn more things that can make the solutions within your wheel house. Otherwise, nothing is stopping you from finding a partner with the right skill set.