There is a fairly new ASIC named "Sohu" that is purpose-built for transformers. They have some bold claims that are impressive if true.
I found a short discussion[2] you may find useful.
[2] https://www.lesswrong.com/posts/qhpB9NjcCHjdNDsMG/new-fast-t...