HN user
I published a video that explains Self-Attention and Multi-head attention in a different way -- going from intuition, to math, to code starting from the end-result and walking backward to the actual method.
Hopefully this sheds light on this important topic in a way that is different than other approaches and provides the clarity needed to understand Transformer architecture. It starts at 41:22 in the below video.
This video tutorial provides an intuitive, in-depth breakdown of how an LLM learns language and uses that learning to generate text. Key concepts shown below are covered in a way that is both broad and deep, while still keeping the material accessible without losing technical rigor:
* Historical context for LLMs and GenAI
* Training an LLM -- 100K overview
* What does an LLM learn during training?
* Inferencing an LLM -- 100K overview
* 3 steps in the LLM journey from pre-training to serving
* Word Embeddings -- representing text in numeric format
* RMS Normalization -- the sound engineer of the Transformer
* Benefits of RMS Normalization over Layer Normalization
* Rotary Position Encoding (RoPE) -- making the Transformer aware of token position
* Masked Self-Attention -- making the Transformer understand context
* How RoPE generalizes well making long-context LLMs possible
* Understanding what Causal Masking is (intuition and benefit)
* Multi-Head Attention -- improving stability of Self Attention
* Residual Connections -- improving stability of learning
* Feed Forward Network
* SwiGLU Activation Function
* Stacking
* Projection Layer -- Next Token Prediction
* Inferencing a Large Language Model
* Step by Step next token generation to form sentences
* Perplexity Score -- how well did the model does
* Next Token Selector -- Greedy Sampling
* Next Token Selector -- Top-k Sampling
* Next Token Selector -- Top-p/Nucleus Sampling
* Temperature -- making an LLM's generation more creative
* Instruction finetuning -- aligning an LLM's response
* Learning going forward
Could you share which 3 static analysis tools you use?
We found Coverity to be prohibitively expensive; but unfortunately there isn't any real alternative!
You are saying this out of the research you have done yourself, right? And yes, I have done the research and also have experienced it first hand -- something that Stanford doctors could not do (and an army of 3 different specializations nonetheless) was cured using Homeopathy. Is this placebo effect?
It is a shame that most people are unaware of the efficacy of Homeopathy to treat ADD, ADHD and autism-spectrum disorders. Do some research and you'll see how hundreds of thousands of people have been cured of these ailments using Homeopathy alone.
Thanks for the pointer. Are there any known open-source implementations of the same?
But it doesn't really describe the LSM data structure!
The implementation seems to use Log-Structured Merge Trees.
The only paper on this data structure seems to be: http://goo.gl/CVF1l
This paper is poorly written and quite honestly not useful to implement an LSM tree. Does anyone know of a better paper than this one?
Is there a more useful technical paper on Fractal Trees? Better yet is there a open source implementation of the same?
What would be the licensing model if the product is open source, but is NOT free (i.e., customers are charged for an open-source product in which BDB is used)?
Perl followed by C
C