dzhulgakov

You can try Mixtral live at https://app.fireworks.ai/ (soon to be faster too)

Warning: the implementation might be off as there's no official one. We at Fireworks tried to reverse-engineer model architecture today with the help of awsome folks from the community. The generations look reasonably good, but there might be some details missing.

If you want to follow the reverse-engineering story: https://twitter.com/dzhulgakov/status/1733330954348085439

In this experimental release with prebuilt binaries it’s about 5Mb per architecture. This includes all operators for inference (that is forward only). We’re working on selective compilation so that you can build a smaller bundle with only a subset of ops that you use. With that for common CNNs it should get to 1-2 Mb range or even smaller.

The recurrent network work is under way - we are revisiting the designs to consciously balance between performance and API niceness. We'll share more tech details in the upcoming days. Right now, this is an example RNN:

https://github.com/caffe2/caffe2/blob/master/caffe2/python/e...

We use it for machine translation so the perf is nice.

If you have design feedbacks please let us know - we'll be really grateful.

Right now we're training models in server-side environment and run highly optimized inference on pre-trained model in products (with focus on inference optimization).

The framework itself allows fine-tuning and training of the model on the mobile device too, but more work is required to enable particular use cases.