Just so you know, I'm the one who wrote RWKV.f90. After making a lightweight autograd package that's a lot like PyTorch, but in Go (I called it spaGO), I am now passionately working on adapting the same concepts to the Fortran language.
But first things first, I want to make sure that using your GPU is as easy as pie. That's my next challenge.
As one of the authors, I'd like to clarify: the equations of the RWKV model enable computational parallelization, provided that the sequence is predetermined. This parallelization occurs during both the training and inference stages, specifically during the prompt reading process (consider it an "encoding"), right before the generation (or decoding phase).
I'm grateful to anyone who feels like commenting, critiquing, or even better getting involved to improve this Machine Translation implementation which is currently very slow (almost unusable except for very short sentences).