correa_brian

lol

super cool. just tried it. love the sassy comments

I literally feel like I'm doing drugs when I'm watching short-form videos.

cool idea

hell yeah

I'm on an M2 and it takes <5 minutes to index a 2hr movie. If you're trying to index a lot of media at once, we will queue it up to be indexed. We also do smart sampling to detect similar frames so if it two talking heads vs. a lot of different shots, it will process faster. In that case the audio is more valuable for the talking heads.

The semantic search queries typically take 100-250ms.

Yeah, exactly. We capture the semantic meaning of each frame and complement the filename/metadata, so both options work.

Hey everyone, I'm brian, one of the makers of cosmos, a desktop app that makes your entire media collection, including external hard drives, searchable by using local ML models.

With your catalog indexed, you can use existing content to generate videos (text-to-video and image-to-video) using Veo 3. To try this out you'll need to bring your own Gemini API key. Obviously this part is not private since you are using Google's AI, but the generations get saved to your desktop and imo it's less clunky than the Google Videos UI. We also added a prompt pre-processing step to enrich the original user input. We use Gemini to create a structured JSON prompt that includes detailed information on lighting, audio, characters, and mood, to name it few. In my experience this makes it easier to preserve continuity in your scenes.

I want to experiment with some local generation models soon so Cosmos can function 100% offline (I've read good things about Wan 2.1 and Stable Diffusion). I really like working with local models (also using Whisper for audio to text transcription) and think long-term everyone will want at least some portion of their data managed by private, offline models.

If you are curious about building something like this for yourself, below is a rough outline: - Pick a platform or a cross-platform tool for your build (we started with Electron and eventually moved to Tauri) - Select your ML models. There are plenty of open-source image and text embedding models (Clip, Siglip, Nomic) - Design a media processing pipeline that won't fry your users' computer (pro tip: you're going to want to throttle indexing when CPU utilization gets too high) - Experiment with well-known open-source media tools like ImageMagick and FFmpeg. This is more than enough to extract frame, clip videos, or anything else you might want to do with a piece of media in your pre/post-processing - Database choice: There are lots of choices for DBs, but in my experience simpler is better. We started with Redis (it was overkill) and eventually migrated to sqlite with a vector embedding extension. Haven't tried Qdrant, Pinecone, or Chromadb, but sqlite works great for this use case. - If you want to support online AI platforms like OpenAI or Anthropic then you'll need to manage API keys and HTTP requests to these services (or maybe MCP? Don't know much about that yet).

Demo https://www.youtube.com/watch?v=qHPl_n-HlP4

fixed! thanks again

woah, you're right. thanks for catching! fixing that now

lets goooo. thank you! i also made it red so it's easier to catch. appreciate the feedback.

You can record directly on the site! I can make it more prominent but there's a Microphone button under the file drop. Lmk if you don't see it.

That's what it used to be. Now it's bytedance's premiere/veed competitor

Ended up swapping out redis for sqlite-vec. Works great. Thanks

:just-blaze

What part?

We're using redis vector modules for cosine similarity. I'm sure there's more to optimize there. Your project sounds cool. How'd you reverse engineer apple's model?

The UX was a small part of the re-write. Since we started from scratch we had a blank slate and figured we'd go all in on Tauri/Rust/React vs. trying continue with Electron in the mix.

I just wasn't able to get the similarity results to match redis. Probably my own error, but that's why I opted for it instead of SQLite. We'll revisit at some point.

Thanks! We went with Rust because we weren't able to tune sqlite with a vector search extension to give me the results we wanted. I'm sure it's possible to use it instead of Redis but that's an optimization for another day. I'll check out Neon.

That's pretty sick. Nice work. What's it called? We're working on the app store submission this week.

Hey Alex - thanks for commenting. I'd be interested in giving this all another look. Like we mentioned in the post + comments, We went with redis because that's what got us the results we wanted. I'm open to revisiting our vector storage if it spares us needing to manage a separate db process.

We love it here. We're staying together forever

Their vector search modules are the reason I embedded it. I tried sqlite and wasn't getting good search results with the vss extension. Maybe I wasn't configuring it correctly. Redis was just better than the alternatives (qdrant, sqlite, duckdb with vector search).

Nice. How long did the migration take?

seems like a lot of people in the comment disagree. good to know it exists

:chefs-kiss

Would have loved to use sqlite if I could get good results. Maybe I botched the implementation.

Appreciate the feedback! We haven't setup the infrastructure for a trial but maybe in the future.

That's cool you built a similar tool - what kept you from releasing it?

Plan is to ship a Windows and Linux version in the next few months if there's enough demand.

We've gotten our users through various launches on HN and reddit with some minimal linkedin promotion. It's been mostly word of mouth, which has been very promising to see.

Re: the electron and video processing performances - there's a lot to dive into. I don't claim to be an Electron expert, so maybe it was our misuse of workers that created a bottleneck. As part of the migration to rust we also implemented scene detection to help reduce the number of frames we indexed and this helped reduce processing loads a lot. We also added some GPU acceleration flags on ffmpeg that gave us meaningful gains. Batching processing image embedding generation was also a good improvement to a point, before it started crashing our model instance.

why?