chaoyu

onnx is not a good option for LLM type of autoregressive generation

In my opinion, Automatic1111 is more of a development tool to experiment with different pipelines on your local GPU, or an internal tool serving a few users.

OneDiffusion project aims to solve a very different problem, which is bringing a SD pipeline to serve production traffic and scale in the cloud. For example, configuring a large diffusion model to run on multi-GPU: https://huggingface.co/blog/deploy-deepfloydif-using-bentoml

Check out BentoML, which is the underlying serving framework used by OpenLLM, and it supports other type of models and modality such as images and videos.

OpenLLM itself is under Apache 2 license, which does NOT restrict commercial use. However, OpenLLM as a framework can be extended to support other LLMs which may come with additional restrictions.

OpenLLM plan to provide an OpenAI-compatible API, which allows you to even use OpenAI's python client to talk to OpenLLM, user just need to change to Base URL to point to your OpenLLM server. This feature is working-in-progress.

Fine-tuning is coming up in the next release!

You can actually try it out on the main branch :P

Looking forward to it!

OpenLLM is adding a OpenAI-compatible API layer, which will make it even easier to migrate LLM apps built around OpenAI's API spec. Feel free to join our Discord community and discuss more!

Smaller models are likely more efficient to run inference and doesn't necessarily need the latest GPU. Larger language model trend to have better performance over more different type of tasks. But for a specific enterprise use case, either distilling a large model or use large model to help with training a smaller model can be quite helpful in getting things to production - where you may need cost-efficiency and lower latency.

The OpenLLM team is actively exploring those techniques for streamlining the fine-tuning process and making it accessible!

OpenLLM in comparison focuses more on building LLM apps for production. For example, the integration with LangChain + BentoML makes it easy to run multiple LLMs in parallel across multiple GPUs/Nodes, or chain LLMs with other type of AI/ML models, and deploy the entire pipeline on Kubernete (via Yatai or BentoCloud).

Disclaimer: I helped build BentoML and OpenLLM.

BentoML.ai | ML Engineer, Backend Engineer | Full-time | Bay Area or Remote | Python, Kubernetes, MLOps platform, Data Infra, Tensorflow, PyTorch, etc

BentoML is an open-source framework for machine learning model serving & deployment https://github.com/bentoml/BentoML

We are a venture backed startup behind the BentoML open source project, and we are looking for engineers who are passionate about building Open Source, MLOps, ML Platform or developer tools. Email [ chaoyu at bentoml.ai] if you are interested.

Job descriptions: https://angel.co/company/bentoml

What does BentoML do?

* Package models trained with any ML framework and reproduce them for model serving in production

* Package once and deploy anywhere for real-time API serving or offline batch serving

* High-Performance API model server with adaptive micro-batching support

* Central storage hub with Web UI and APIs for managing and accessing packaged models

* Modular and flexible design allowing advanced users to easily customize

BentoML is a framework for serving, managing and deploying machine learning models. It is aiming to bridge the gap between Data Science and DevOps, and enable data science teams to continuously deliver prediction services to production.

BentoML.ai | Open Source Evangelist / Technical Writer | San Francisco or Remote | Full time or Contract | http://docs.bentoml.org/

BentoML is an open-source platform for high-performance machine learning model serving. We are a small team building the open-source project BentoML as well as a BentoML based SaaS product for enterprise teams. https://github.com/bentoml/BentoML

We are looking for a technical writer with a Machine Learning background, ideally, someone who has experience in either open-source evangelism or solution architect in related space, to help with creating tutorials, blog posts, and documentation for our open-source project BentoML.

Email chaoyu [at] bentoml.ai for more information.

I'm actually building a "modular open-source company/product" in the MLOps space:

BentoML https://docs.bentoml.org/en/latest/

BentoML(https://github.com/bentoml/BentoML) may help you with the process of building endpoints with both Deep learning models and logistic regression/tree models, and it automatically helps you to containerize the API server into docker image that's ready for production deployment.

It also provides OpenAPI spec for your API endpoint, which allows you to generate API client in Java, for your backend/app teams.

hi Aaron, I'm one of the BentoML aurthors - great suggestion on pipreqs, will look into incorparating that into BentoML!

It should be very straightforward adding support for saving/loading Statsmodels in BentoML. In fact you should also be able to just use the existing "PickleArtifact" in BentoML for statsmodel predictors too. We will add an example notebook for working with Statsmodels library soon!

Graphpipe solves a very unique problem when building ML model serving system, although BentoML is trying to solve a very different problem. We think it would be interesting to support GraphPipe's flatbuffer format in BentoML's REST API model server down the line if people are interested in that.

Our quick start guide notebook on Google Colab is also a great place to get started!

https://colab.research.google.com/github/bentoml/BentoML/blo...

Thanks for sharing the project Kevlar1818.

BentoML author here - we are building BentoML to empower Data Scientists to ship prediction services instead of delivering "models" to dev teams. We proposed a workflow that made it easy for data scientists to create and test prediction services and then deploy them to cloud platforms such as AWS Lambda, SageMaker or Docker/Kubernetes.

Happy to answer more questions, cheers!