marcelroed

Running the numbers now

It's usually not as binary as "hit" or "miss" with a prefix cache, and you need to know the token boundaries to know where the cache hit ends.

The current structures used for KV-caching in vLLM and SGLang work by chunking the KV-cache tokens into prefix trees, and you need to hash chunks of tokens in order to look up in these, meaning you need to be able to slice up your tokens by token count.

Again I have no idea what proprietary engines are doing, but this is why open source stuff needs to tokenize before cache lookup at least.

Time to first token refers to the time until the model outputs one token, which includes the time to process the entire prompt (doing prefill). The GPU time per token is much lower when doing prefill, so the significance of tokenization is higher.

I can add some benchmarks for compatibility mode in the future. I have a little more juice to squeeze out of the Python interop though, so not quite ready for it yet.

Numbers are for the Gigatoken API, but compatibility mode just means eating a bunch of Python overhead (creating lists, reading strings to bytes). You can expect a modest ~200-300x speedup with compatibility mode depending on how you use it.

You can, but this usually results in sequences with padding/truncation, since you won't know how many tokens your inputs map to before you actually tokenize them. This also makes shuffling difficult.

In practice every training project I've worked on does tokenization in a separate data processing phase.

Author here! In my case it's mostly pretraining experiments, where you might want to change your data mixture/filtering/processing of training data, and splits are usually done at a token-level instead of a text level. In this case we usually run for days on a huge number of CPUs to finish tokenizing something like DCLM.

From what I can tell it's also useful for inference when considering time-to-first-token (TTFT) as reported by fastokens.[0]

I'm not sure about the proprietary inference engines, but in the open source ones tokenization is done before looking up if a text sequence is present in the KV-cache. If you have a long prefix that's been seen before (say a system prompt), the time for tokenizing that will be a large part of your TTFT. The tokenizer cache should be warmed up in this case, so the throughput for Gigatoken would be significantly higher than reported in the repo.

[0] https://github.com/crusoecloud/fastokens

We have autograding for code through tests written by hand, and additionally do manual code audits if we see suspicious behavior. We also do grading the old-fashioned way for writeups.

We do indeed catch students who don't follow the honor code. It's very obvious from how the code looks, as well as the rate of progress. Since we use Modal for class submissions, we have code deltas for every time they run something on B200s. The diffs often contain something like 300 lines in 5 minutes, in which case we review and report based on how egregious/provable it looks.

TA here. Noted! I now have more resources to test more environments, and will do so whenever possible. I think freezing due to memory overuse is going to be a problem with anything you code yourself, but I do think we could be more rigorous with guiding people to achieve limited memory use for the tokenizer task.

IMO the cost of renting GPUs is a bit overstated in these comments. Generally almost all of the development can be done locally, and then ran for a short period of time using on-demand GPUs. For assignment 1, you can run everything on your local machine, even if you don't have a GPU. For A1 and A2, you can do (most of) the tasks with only a few hours of renting. Without being too careful using rental GPUs throughout will net you around $200 of a compute budget, but you can easily get this under $50 if you're willing to scale down many of the problems. I think we could work on making this clear and charting what these changes are.

If you have further feedback or encounter problems, feel free to open issues in the repos so we can resolve them! It's hard for us to fix issues we're not aware of.

We do provide resources for enrolled students. The online suggestions are for external students or Stanford students who we weren't able to admit.

TA here. Biggest changes are in the second assignment (distributed) where we added a bunch of memory, profiling and distributed tasks, as well as in the fifth assignment (alignment), where most of the RL tasks are fresh this year. Assignment 3 (scaling laws) was also completely updated, but in a way that might be difficult to run without substantial resources. I'm working on a way for external students to be able to run simulated experiments for free!

Assignment 1 (basics) has the most hours of preparation invested in it, and only minor modernization/bug fixes were necessary this year.

TA here. Definitely not! In fact we explicitly added sections in the first assignment to allow for scaling down to even local compute (M-series GPUs). For assignment 2 there are a few regions that require Triton support for your GPU, but everything can be adapted for much cheaper GPUs.

We were lucky enough to get Blackwell GPUs for Stanford students this year, which is why the writeups are written mostly around them.

I am only familiar with MLIR for accelerator-specific compilation, but my understanding is that by describing operations at a higher level, you don’t need the frontend to know what LLVM IR will lead to the best final performance. For instance you could say "perform tiled matrix multiplication" instead of "multiply and add while looping in this arbitrary indexing pattern", and an MLIR pass can reason about what pattern to use and take whatever hints you’ve given it. This is especially helpful when some implementations should be different depending on previous/next ops and what your target hardware is. I think there’s no reason Zig can’t do something like this internally, but MLIR is an existing way to build primitives at several different levels of abstraction. From what I’ve heard it’s far from ergonomic for compiler devs, though…

After trying this out on my iphone, I can say your conclusion is exactly correct. The icons look subtly out of focus in a way that's quite unsettling.

It’s worth noting that Shreyas (the first author) was a student with Tenenbaum at MIT before he went to Berkeley