ajhai

https://x.com/ajhai/status/1899528923303809217 something I have been working on for a few months now.

It is inference latency most of the time. These VLA models take in an image + state + text and spit out a set of joint angle deltas.

Depending on the model being used, we may get just one set of joint angle deltas or a series of them. In order to be able to complete a task, it will need to capture images from the cameras, current joint angles and send them to the model along with the task text to get the joint angle changes we will need to apply. Once the joint angles are updated, we will need to check if the task is complete (this can come from the model too). We run this loop till the task is complete.

Combine this with the motion planning that has to happen to make sure the joint angles we are getting do not result in colliding with the surroundings and are safe, results in overall slowness.

Building a wheeled robot with arms to help automate household chores - https://x.com/ajhai/status/1891933005729747096

I have been working with LLMs and VLMs to automate browser based workflows among other things for the last couple of years. Given how good the vision models have gotten lately, the perception problem is solved to level where it opens up a lot of possibilities. Manipulation is not generally solved yet but there is a lot of activity in the field and there are promising approaches to solve (OpenVLA, π0). Given these, I'm trying to build an affordable robot that can help around with household chores using language and vision models. Idea is to ship capable enough hardware that can do a few things really well with the currently available models and keep upgrading the AI stack as manipulation models get better over time.

I've tested Q4 on M1 and it works though the quality may not likely be the same as you'd expect as others have pointed out on the issue.

You can already run these models locally with Ollama (ollama run llama3.1:latest) along with at places like huggingface, groq etc.

If you want a playground to test this model locally or want to quickly build some applications with it, you can try LLMStack (https://github.com/trypromptly/LLMStack). I wrote last week about how to configure and use Ollama with LLMStack at https://docs.trypromptly.com/guides/using-llama3-with-ollama.

Disclaimer: I'm the maintainer of LLMStack

You can actually do this with LLMStack (https://github.com/trypromptly/LLMStack) quite easily in a no-code way. Put together a guide to use LLMStack with Ollama last week - https://docs.trypromptly.com/guides/using-llama3-with-ollama for using local models. It lets you load all your files as a datasource and then build a RAG app over it.

For now it still uses openai for embeddings generation by default and we are updating that in the next couple of releases to be able to use a local model for embedding generation before writing to a vector db.

Disclosure: I'm the maintainer of LLMStack project

If anyone is looking to try it out quick without local installation, we added Llama-8B model to Promptly playground. Please check it out at https://trypromptly.com/playground.

Sorry missed this. It was hidden behind login before. It should now be reachable.

If you are looking to play with the model without installing it locally, we've added it our playground at https://trypromptly.com/playground.

Put together a guide on how to do this with your own avatar and posted at https://news.ycombinator.com/item?id=39053304

We can get a lot done with vector db + RAG before having to finetune or custom models. There are a lot of techniques to improve RAG performance. Captured a few of them a while back at https://llmstack.ai/blog/retrieval-augmented-generation.

We have recently added support to query data from SingleStore to our agent framework, LLMStack (https://github.com/trypromptly/LLMStack). Out of the box performance performance when prompting with just the table schemas is pretty good with GPT-4.

The more domain specific knowledge needed for queries, the harder it has gotten in general. We've had good success `teaching` the model different concepts in relation to the dataset and giving it example questions and queries greatly improved performance.

Gemini Pro compares to GPT 3.5.. their biggest model that competes with GPT-4 is called Gemini Ultra and they say that's coming early next year.

https://github.com/trypromptly/LLMStack - started working on this as a wrapper over OpenAI's endpoints for another product and it gradually became this.

Another project I worked on for my own use was a network isolated, lightweight video monitoring system. Around 5 years ago, I was looking to install a camera in our living room. I couldn't find anything I trusted that worked completely offline without some companion app pinging their servers. So I bought a basic IP camera on Amazon that supports rtsp and a raspberry pi. Created a fenced wifi network and added the camera to it.

Had an FFmpeg process read camera stream on demand and write to local buffers. Wrote a simple python server to listen for incoming connections on a different interface and stream the video on API requests. Then built an android app that talks to the python server to stream video on demand.

Also installed motion (https://github.com/Motion-Project/motion) on raspberry pi to detect motion in the video and store those snippets to local storage. With motion running, the adapter I was using wasn't delivering enough power resulting in storage occasionally unmounting and raspberry pi restarting taking the camera system offline. With motion detection disabled, the entire setup ran reliably for many years.

This will hopefully improve the startup times for FFmpeg when streaming from virtual display buffers. We use FFmpeg in LLMStack (low-code framework to build and run LLM agents) to stream browser video. We use playwright to automate browser interactions and provide that as tool to the LLM. When this tool is invoked, we stream the video of these browser interactions with FFmpeg by streaming the virtual display buffer the browser is using.

There is a noticeable delay booting up this pipeline for each tool invoke right now. We are working on putting in some optimizations but improvements in FFmpeg will definitely help. https://github.com/trypromptly/LLMStack is the project repo for the curious.

I've been using Django as my main choice for web projects for over ten years. The reason I like it so much is because it comes with a lot of built-in features that one needs to ship web projects to production. For example, I was first attracted to Django because of its admin interface and its straightforward views and templating system.

Over the years, Django has kept up with changes in web development. An example of this is when database migrations, which used to be a separate project, were integrated into Django itself. The Django community is also strong with great ecosystem projects like DRF for APIs, Django Channels for real-time features, and social-auth for social sign-ins.

My recent use of Django is in (https://github.com/trypromptly/LLMStack). We use Django Channels for WebSocket support, DRF for APIs, and ReactJS for the frontend.

We built https://github.com/trypromptly/LLMStack to serve exactly this persona. A low-code platform to quickly build RAG pipelines and other LLM applications.

Kudos to the team for a very detailed notebook going into things like pipeline evaluation wrt performance and costs etc. Even if we ignore the framework specific bits, it is a great guide to follow when building RAG systems in production.

We have been building RAG systems in production for a few months and have been tinkering with different strategies to get the most performance out of these pipelines. As others have pointed out, vector database may not be the right strategy for every problem. Similarly there are things like lost in the middle problems (https://arxiv.org/abs/2307.03172) that one may have to deal with. We put together our learnings building and optimizing these pipelines in a post at https://llmstack.ai/blog/retrieval-augmented-generation.

https://github.com/trypromptly/LLMStack is a low-code platform we open-sourced recently that ships these RAG pipelines out of the box with some app templates if anyone wants to try them out.

There are a lot of things that goes on in production to scale any service to handle that level of requests. But Django as a web framework is good at what it does. It comes in with most things that one needs to put web apps in production.

We recently open-sourced an LLM apps platform (https://github.com/trypromptly/LLMStack) that is entirely built with django as backend (drf for APIs, channels for websockets and reactjs for frontend).

We did write one just yesterday that talks about rags and some techniques to improve their performance in production at https://llmstack.ai/blog/retrieval-augmented-generation

Django has been my go to framework for any new web project I start for more than a decade. Its batteries-included approach meant that one could go pretty far with just Django alone. Included admin interface and the views/templating setup was what first drew me to the project.

Django project itself has kept pace with recent developments in web development. I still remember migrations being an external project, getting merged in and the transition that followed. Ecosystem is pretty powerful too with projects like drf, channels, social-auth etc., covering most things we need to run in production.

https://github.com/trypromptly/LLMStack is a recent project I built entirely with Django. It uses django channels for websockets, drf for API and reactjs for the frontend.

We recently opensourced a similar platform for building workflows by chaining LLMs visually along with LocalAI support.

Check it out at https://github.com/trypromptly/LLMStack. Like you said, it was fairly easy to integrate LocalAI and is a great project.

You can do this with LLMStack. We are adding the datasource handlers to pull data from Google docs etc., Slack integration is built in. More info at https://llmstack.ai/docs/apps/integrations/slack. Happy to chat if you have questions.

Would you mind expanding why it was tough to get started with Retool?

We are building https://github.com/trypromptly/LLMStack, a low-code platform to build LLM apps with a goal of making it easy for non-tech people to leverage LLMs in their workflows. Would love to learn about your experience with retool and incorporate some of that feedback into LLMStack.

Most of the use cases we are seeing LLMs unlock look trivial in isolation when looked at it from content generation point of view but become super powerful when mixed into workflows. This is evident from all the startups we are seeing lately in this space and LLM integrations into existing products like Zapier, Hubspot and now Retool AI.

Granted these are still early days for LLMs, but it looks they are primarily going to revolutionize how knowledge work is done at a scale similar to what computers have done. There are exciting projects coming out every week putting LLMs into the workflows of more and more knowledge workers. open-interpreter is the recent of the bunch.

Similar to other workflow automation projects in this space, we started building LLMStack (https://github.com/trypromptly/LLMStack) with a goal of scaling knowledge work pieplines. Exciting times ahead!

Not directly imo. Zapier focus is still on workflow automation and retool is on internal tools. This directly competes with a bunch of startups that recently launched including an open-source project we put out a few weeks back (https://github.com/trypromptly/LLMStack)

Congrats on shipping. Looks pretty great! A few years ago I was chatting with a friend of mine about a similar idea. I remember there were a lot of discussions around content moderation and biased nature of comments section of certain news websites around that time. The idea was to come up with a decentralized way to maintain comments indexed by the urls and anyone with a certain chrome extension can participate in these comments section completely unmoderated by the site owners.

Eventually there ought to be some community moderation but the goal was to keep it unbiased. If your project takes off, first thing you will have to deal with is going to be spam.

Building quizzes, tests, and lesson plans from curriculum materials

Example prompts that OpenAI shared here are a great start. However I think these use-cases are better served as micro apps built on top of these prompts. For example, a teacher will keep coming back to use this prompt with same/similar set of responses most of the year. On top of that, enriching the context with additional information pulled from local sources will quickly become a need.

ChatGPT's custom instructions will help with not having to repeat prompts but the interface falls short when it comes to repeat narrow use cases. This is where imo LLM apps shine. A simple app built with langchain or some low-code platforms and providing local data from a vector store can be super powerful.

We recently open-sourced LLMStack (https://github.com/trypromptly/LLMStack), a platform that allows users to build these micro apps to automate their workflows. Our goal is to make these workflows sharable so someone can download a yaml file for this prompt and chain and start using it in their job.

I previously submitted this a few days ago as https://news.ycombinator.com/item?id=37290863 but without any context. Reposting it with background around why we decided to make it open-source and our path forward.

The only way I can think of is to provide a comprehensive list of everything the tool was trained on.

There are some startups working in the space that essentially plan to do something like this. https://www.konfer.ai/aritificial-intelligence-trust-managem... is one I know of that is trying to solve this. They enable these foundation model providers to maintain an inventory of training sources so they can easily deal with coming regulations etc.