HN user
adam, i'd like to get in touch and would love to run the benachmark with mixedbread as a search backend. we are doing this right now with a lot of compliance companies. would be very curious how it improves quality/cost e2e
yes, your are right. what heading would you have taken here?
everything worth writing, you should write yourself
ah whoops, I'll fix it. ty!
The ndcg loss is minimal 90.26 -> 89.65. This means it maintains most of the quality.
to which email did you send it? can u send it to support please?
this is the reason why we report ndcg and not recall. ndcg respects fine grained details so you get the an overview of how much details you are trading off since it would hurt the ranking.
yes exactly.
slop complaining about other slop
very bad take. with most modern multomodal models you get way better performance then going to text first
this might be interesting: https://www.theinformation.com/articles/chatgpt-doctors-star...
$150M RR on just ads, +3x from August. On <1M users.
a good system (like openevidence) indexes every paper released and semantic search can incredible helpful since the the search api of all those providers are extremely limited in terms of quality.
now you get why those system are not cheap. keeping indexes fresh, maintaining high quality at large scale and being extremely precise is challenging. by having distributed indexes you are at the mercy of the api providers and i can tell you from previous experience that it won't be 'currently accurate'.
for transparency: i am building a search api, so i am biased. but i also build medical retrieval systems for some time.
you should check mixedbread out. we support indexing multimodal data and making data ready for ai. we are adding video and audio support by the end of the year. might be interesting for the OP as well.
we have couple investigative journalists and lawyers using us for a similar usecase.
or deberta but nevertheless super interesting!
For everyone wondering how good this and other benchmarks are:
- the OmniAI benchmark is bad
- Instead check OmniDocBench[1] out
- Mistral OCR is far far behind most Open Source OCR models and even further behind then Gemini
- End to End OCR is still extremely tricky
- composed pipelines work better (layout detection -> reading order -> OCR every element)
- complex table parsing is still extremely difficult
we have extremely processing heavy jobs where user upload large collection of files (audios, pdfs, videos etc.) and expect to get fast processing. its just that we need to fan out sometimes, since a lot of our users a sensitive to processing times.
We have extremely processing heavy jobs where user upload large collection of files (PDFs, audios, videos etc.) and expect to get fast processing.
Hetzner is really great until you try to scale with them. We started building our service on top of Hetzner and had couple 100s of VMs running and during peak time we had to scale them to over 1000 VMs. And here couple of problems started, you get pretty often IPs which are black listed, so if you try to connect to services hosted by Google, AWS like S3 etc. you can't reach them. Also at one point there were no VMs available anymore in our region, which caused a lot of issues.
But in general if you don't need to scale crazy Hetzner is amazing, we still have a lot of stuff running on Hetzner but fan out to other services when we need to scale.
yeah but if people would like to double check the results it would be nice to have the actual benchmark. especially given that your playground is broken...
"We ran into an error processing your request. Please try again"
guys please release the benchmark or the benchmark code. like this is just "trust me bro"
I think we are going through the langchain era for agents. the world will look really different in couple month and the stack will be wildly different and more unified.
Thats actually not correct. Embeddings can handle relationships like “without” or “not.” when trained for it. You need to scale up the training massively to make it generalize it well. The current version of Mixedbread Search supports negatives like "tshirt without stripes". You can check it out on our launch video [1]. We are working on a way more generalized model, which should be able to capture relationships, emotions and much more. The current models are just limited.
yes, in which one would be interested?
there is a lot coming
We are right now not including the artist name. Which will be done in the next iteration of the model (next week). Right now the search is only based on what the model can "see". And it seems like that the model does not understand the art of Mark Rothko.
The next version can see the image and read the metadata.
A bit more context: We are include everything in the latent space (embeddings) without trying to maintain multiple indexes and hack around things. There is still a huge mountain to climb. But this one seems really promising.
We will add a feedback form tomorrow morning. For now please feel free to write to aamir at domain name of the page. thank you so much! this helps us a lot.
Thank you for pointing this out. We will add this tomorrow morning.
If you find more such cases please feel free to send them over to aamir at domain name of the Show HN. I would love to see those cases and see how we can improve on them. Thank you so much for the feedback.
thank you for reporting these. we will improve on them for the next iteration.
We have a lot more things coming up soon. It just took us some time building Mixedbread Search.