Vision Transformers Need Registers

https://openreview.net/forum?id=2dnO3LLiJ1

by cscurmudgeon • 2 years ago

171 23 2 years ago

superkuh• 2 years ago

"Attention sinks" for vision models?

ttul• 2 years ago

According to the paper, the "registers" are additional learnable tokens that are appended to the input sequence of a Vision Transformer model during training.

They are added after the patch embedding layer, with a learnable value, similar to the [CLS] token and then at the end of the Vision Transformer, the register tokens are discarded, and only the [CLS] token and patch tokens are used as image representations.

The register tokens provide a place for the model to store, process and retrieve global information during the forward pass, without repurposing patch tokens for this role.

Adding register tokens removes the artifacts and high-norm "outlier" tokens that otherwise appear in the feature maps of trained Vision Transformer models.

Using register tokens leads to smoother feature maps, improved performance on dense prediction tasks, and enables better unsupervised object discovery compared to the same models trained without the additional register tokens.

This is a neat result. For just a 2% increase in inference cost, you can significantly improve ViT model performance. Close to a free lunch.

kadushka• 2 years ago

Interesting. One other potential benefit is an easier quantization of the activations.

macleginn• 2 years ago

There was an attempt to add several CLS tokens to BERT, with less spectacular results: https://arxiv.org/pdf/2210.05043

swyx• 2 years ago

are there lessons here for regular (non vision) transformers? sounds close to attention sinks/pause tokens?

johntb86• 2 years ago

For these tokens you first need to unembed the result of the final layer, the re-embed the resulting token on the next pass. Has anyone investigated passing the raw output of one pass to the input of the next?

hasmanean• 2 years ago

So is that what all the visual cues are in real life, things like fashion accessories, uniforms etc.?

larodi3• 2 years ago

This all token business is very shady, and the whole probability theory. You add token here and there and magic happens. Discreet math people can not take this lightly. Stochastic regexes is one thing, but this on a completely different level of mathematical debauchery.

Absolutely amazing this works.

akomtu• 2 years ago

The modern Alchemy.

programjames3• 2 years ago

Vision transformers are essentially just JPEG but with learned features rather than the Fourier transform.

samus• 2 years ago

I find it wild that the training process can do such things as forcing it repurpose background areas to begin with. The authors just observed abd optimized what the model was already doing by itself.

richdougherty• 2 years ago

Related? "Let's Think Dot by Dot: Hidden Computation in Transformer Language Models" https://arxiv.org/abs/2404.15758

Chain-of-thought responses from language models improve performance across most benchmarks. However, it remains unclear to what extent these performance gains can be attributed to human-like task decomposition or simply the greater computation that additional tokens allow. We show that transformers can use meaningless filler tokens (e.g., '......') in place of a chain of thought to solve two hard algorithmic tasks they could not solve when responding without intermediate tokens. However, we find empirically that learning to use filler tokens is difficult and requires specific, dense supervision to converge. We also provide a theoretical characterization of the class of problems where filler tokens are useful in terms of the quantifier depth of a first-order formula. For problems satisfying this characterization, chain-of-thought tokens need not provide information about the intermediate computational steps involved in multi-token computations. In summary, our results show that additional tokens can provide computational benefits independent of token choice. The fact that intermediate tokens can act as filler tokens raises concerns about large language models engaging in unauditable, hidden computations that are increasingly detached from the observed chain-of-thought tokens.
In this work, we demonstrate that transformers trained on the next-token prediction objective can achieve improved performance on certain tasks when given filler tokens, achieving perfect accuracy whereas the no-filler, immediate-answer setting achieves only low accuracy.

I wonder if we could get benefits from adding special computation/register tokens to text LLMs?

More discussion:

- https://news.ycombinator.com/item?id=40182695

- https://www.reddit.com/r/LocalLLaMA/comments/1cf2w5a/transfo...

joaquincabezas• 2 years ago

I was at ICLR and this was one of the best of this year, it was also evident during the poster session. Congrats to the authors!!

barbolo• 2 years ago

I’ve been using DinoV2 for some months now. I’ve tried the models with 4 register tokens along with CLS + patch tokens. I’ve several embeddings (tokens) from previous model (no registers) which are part of my solution, so I didn’t adopt the newer “register” models because the CLS tokens are not aligned between 0 registers and 4 registers models. It would be nice if the CLS and patch tokens were somehow aligned between those models.

ashvardanian• 2 years ago

We tested dozens (maybe >100) of papers/ideas over the last years in vision and multimodal perception and this is one of the rare cases, where everything worked well! Neat idea and paper!

This model, for example, uses 4 register tokens, and combines them with Matryoshka-style losses for training, resulting in super-compact 64-dimensional embeddings, in case anyone is looking for CLIP alternatives: https://huggingface.co/unum-cloud/uform3-image-text-english-...

Vision Transformers Need Registers

Related Stories