Comparing compute cost versus FlashAttention-2 is not very honest to me.
FlashAttention-2 is not used anymore for at least 2y.
This architecture would have been a massive improvement 3 years ago, but it is a ~solved~ problem IMO.
@samuelberthe https://github.com/samber meet.hn/city/47.2186371,-1.5541362/Nantes