DeepSeek's multi-head latent attention and other KV cache tricks

https://www.pyspur.dev/blog/multi-head-latent-attention-kv-cache-paper-list

by t55 • 1 year ago

292 72 1 year ago

pyspur.dev

spps11• 1 year ago

When you say sequence length, does it only count the output tokens or are input tokens also included in that?

Thanks for the post, it was an excellent read!

t5529• 1 year ago

Thanks for reading! In most contexts (including this one), seq length encompasses both the initial input (prompt) tokens and the output tokens the model generates. It’s the total length of all tokens processed by the model so far.

maxmolecule• 1 year ago

Please do! Seeing that you used multiple research papers to back up this writing inspired me to use this in my current research project for the literature review and eventual write up.

The template will be hugely helpful for a non-programmer like me.

deepdarkforest• 1 year ago

Very clean writeup. On the attention sinks, you mention they enable "infinite-length sequence processing". What does that mean exactly in practice? Isn't deepseek still capped at 128k?

m348e912• 1 year ago

Agreed on the writeup itself. It's beautifully written and presented. Kudos to Jean Kaddour and anyone else that may have been involved in putting it together.

t5529• 1 year ago

Thank you so much, glad you liked it

t5529• 1 year ago

Thank you! Great question.

"Infinite-length sequence processing" in StreamingLLM refers to handling much longer sequences than the model's training window (e.g., millions of tokens), by combining a sliding window for recent tokens with fixed attention sinks from the start of the sequence.

I can't speak for DeepSeek, but if I had to guess, I'd say that the infinite context window isn’t practical because storing all past tokens eventually becomes too expensive.

amelius• 1 year ago

Not sure if I'm getting this. Is this cache implemented as part of the forward pass through the network, in a general Python datastructure like a dict? Or is the cache somehow part of the fabric of the neural network?

t5529• 1 year ago

The KV cache is typically stored in a data structure external to the trained weights—often a buffer or set of tensors kept alongside the model’s forward pass (e.g., in PyTorch, one might store it in a dictionary-like container). It’s not baked into the neural network parameters themselves; instead, it’s an auxiliary memory that holds precomputed key-value pairs so the model doesn’t have to re-encode past tokens on each new inference step.

anvuong2• 1 year ago

Neither. Think of it as something like redis or memcached. It's external to the program, and the program will run just fine without it. But it avoids a lot of duplicate works.

ahzhou• 1 year ago

It’s a tensor stored in GPU memory to improve inference throughput. Check out the PagedAttention (which introduces vLLM) paper for how most systems implement it nowadays.

evertedsphere2• 1 year ago

This blog post is mostly AI-generated using a PySpur workflow with minor human edits.

it's funny that this was clear about 5% in just due to the classic chatgpt-style format and tone

GalaxyNova• 1 year ago

It's blatantly obvious; nobody uses so many bullet points.

t5529• 1 year ago

True! We love bullet points! :)

pdntspa• 1 year ago

I do :(

It's a great and concise way to write!

t5529• 1 year ago

Fair point! Do you prefer a different format or tone? We really like the concise bullet point format :)

evertedsphere2• 1 year ago

it's not the bullet points per se, the general structure of the analysis has a certain vibe to it at a level deeper than that of just visual presentation

but this is something where it's up to you to decide what you want from your ghostwriting. my comments would not a system prompt make

TeMPOraL6• 1 year ago

Okay, so this is a PySpur ad, alright. Since I'm interested in this kind of tools, and I see on their GitHub that they don't have loops yet, I have to ask: does anyone know of a similar tool (node/DAG-based) that does support looping?

It seems to be a common problem; so far, I've played with Rivet, n8n, and "LLM Party" nodes for ComfyUI; they all seem to focus on everything other than allowing to conveniently loop the flows.

t5529• 1 year ago

We do support loops! :) Just haven't advertised it inside the Github readme yet.

One of the reasons why we started building pyspur is precisely because none of the other tools support loops!

If you need more support; shoot me an email; you can find it at the bottom of the github readme.

EDIT: Just added a screenshot to the readme.

visarga3• 1 year ago

so this is a PySpur ad, alright

They should have also posted the PySpur pipeline, it would be interesting to see the agentic flow they used in this article. I am doing a lot of this kind of worflows manually, copy pasting stuff, I'd like to have some tools to design AI flows. PySpur looks pretty interesting visually.

didgeoridoo• 1 year ago

It was going pretty well until the exclamation point at the end of the first paragraph.

t5529• 1 year ago

Replaced the exclamation point with a dot, hope it's better now!

karolist2• 1 year ago

What's specific to deepseek here that other models do not use, or are you just riding the keyword wave?

t5529• 1 year ago

DeepSeep proposed the multi-head latent attention technique! :)

As far as I know, they are the only ones using it so far

karolist2• 1 year ago

Fair point, thanks for clarification, it seems this was first proposed in https://arxiv.org/pdf/2405.04434? I was confused by your title mentioning DeepSeek but then first paragraph revert to "...language models like ChatGPT and DeepSeek faster at generating text".

pama• 1 year ago

Neat. Can you share the workflow that created this blog? What models did it use?

t5529• 1 year ago

Thanks! Yes, will push it as template to our repo (https://github.com/PySpur-Dev/pyspur) soon! We used o1 and claude3.5

8note• 1 year ago

hmm. after my engineer degree put all of the vector math in the form

k = Wx

seeing

k = xW

is jarring. Is there a reason for using horizontal vectors? Common for data science docs?

t5529• 1 year ago

It’s mostly a convention. In many deep learning frameworks (PyTorch, TensorFlow, etc.), inputs are stored with the “batch × length × hidden-dim” shape, effectively making the token embeddings row vectors. Multiplying “xW” is then the natural shape-wise operation. On the other hand, classical linear algebra references often treat vectors as column vectors and write “Wx.”

anvuong2• 1 year ago

Isn't batch-first a Pytorch thing? I started with Tensorflow and it's batch-last.

quanto• 1 year ago

You are in the right here. Horizontal vectors are common for (some) deep learning docs, but column factors are the literature standard elsewhere.

sifar• 1 year ago

It is more efficient to compute k = xW with the weights transposed than k = Wx.

narmiouh• 1 year ago

How were the Images in the blog generated?

seanvelasco• 1 year ago

getting tired of these blog posts that end with "this post is AI-generated" as if it's going to surprise us. it's getting repetitive. imo, articles should be prefaced if they're ai generated or not to make the reader not feel stupid after reading the whole thing

with that said, i love the content! will be bookmarking for future reference

spencerf• 1 year ago

I feel like we’re living in strange times where your comment appears to be AI generated as well. You complain about the surprise at the end and then offer up a similar structural surprise in your reply.

TeMPOraL6• 1 year ago

Strange times indeed, given that I naturally write comments structured similarly to GP. Hell, I'm probably even more of an LLM with human face than GP, because I capitalize the first words in my sentences, exactly like ChatGPT does.

t5529• 1 year ago

Hi, OP here. My intention wasn't to "gotcha" anyone by mentioning that in the end, it was simply to be upfront. Many blog posts/content put out these days are obviously 100% AI-generated, yet it's never being mentioned. This one was probably 80%/20% (I still did many manual edits).

Glad you overall liked it!

esperent3• 1 year ago

At the start of the article there's a very clear profile picture and name of the author: Jean Kaddour.

If you want to be upfront, you should mention at the start that it's written by AI instead of showing this fake author.

This would give people the choice on whether to read it.

Putting it at the end is just to give you plausible deniability. Clearly your intention is to present this as if it was written by this Mr. Kaddour, which is a lie.

EDIT: they removed the fake author in response to this comment

llmthrow102• 1 year ago

I'd rather eat sand than read an AI-generated article. If you don't care enough to write it, I don't care enough to read it.

visarga3• 1 year ago

I don't like formatting in bullet points and listicles much, but the contents are pretty good, they cover many papers in a lightweight way, you can get a decent overview in 10 minutes for what would take hours to research.

t5529• 1 year ago

the contents are pretty good, they cover many papers in a lightweight way, you can get a decent overview in 10 minutes for what would take hours to research.

Exactly, I'm still surprised it works so well.

Also, which formatting do you prefer? I explicitly prompted it to write everything in bullet points because I find it more digestible

esperent3• 1 year ago

How long would it take you to generate your own equally good summary of these papers using Claude? Maybe 30 seconds?

t5529• 1 year ago

Hi, OP here; this article helped me a lot to understand better KV caches, which is ultimately why I co-wrote it with AI + read it several times before posting

yellow_lead• 1 year ago

each token needs to "look at" or "attend to" all other tokens to understand the context.
First token: Look at 1 token (cost: O(1^2))

Umm, is this right? There is not 1 token existing before generating the first token, so how do you look at it? AI slop?

Vampiero• 1 year ago

Umm, is this right?

No way to know until you painstakingly verify every single assertion that the AI made! The author of this article certainly didn't, and the content was good enough to them.

Trust me, AGI is almost there.

t5529• 1 year ago

I did :)

Yes, 80% of the post is generated, yet, I still reviewed everything!

And if I had written everything from scratch, this would have probably taken a week rather than a few hours.

t5529• 1 year ago

The phrase, “the first token looks at 1 token,” is simply a shorthand for the self-attention step when the sequence length is one. Although there are no preceding tokens, we still treat it as an O(1^2) operation where the first token effectively attends to itself (or a special [BOS] token). This approach preserves the big-O analysis when summing over all tokens.

DeepSeek's multi-head latent attention and other KV cache tricks

Related Stories