OpenAI's Assistant API can't handle multiple files

https://www.tonic.ai/blog/rag-evaluation-series-validating-openai-assistants-rag-performance

by Ephil012 • 3 years ago

13 3 3 years ago

kaoD• 3 years ago

It seems that OpenAI has brief time periods during which their RAG system is able to work properly

This is the problem of running tests behind a huge black box over the internet: are you even measuring properly? Who knows.

OpenAI has been under huge load lately so it's possible (even likely) that whatever microservice the actual LLM was connecting to for retrieval was just failing miserably and not properly reporting the failure.

I've seen this myself even for a single document: GPT was failing to retrieve some obvious data and it just didn't properly communicate it to me, just told me something along the lines of "the info is not there" or spitting plausibly-sounding results from the LLM inference instead of the info that was actually in the PDF. After retrying (and a long wait) it worked -- I can see why this effect can be exacerbated when retrieving from multiple files, which is probably implemented as multiple calls over to the microservice handling the retrieval, resulting in more chances to fail retrieval.

drcode2• 3 years ago

the conundrum is that OpenAI is lowering the barrier to entry for this stuff so much, the option for most people unfortunately is between using these black box tools or not doing the thing at all

drcode2• 3 years ago

I'm getting results that roughly match your findings

It's weird, we tend to think of the OpenAI people as programming gods now, but the kr api seems pretty obviously botched at the moment (since I can run use third party tools that use gpt4 with another embedding database, along with the same data, and get much better results)

OpenAI's Assistant API can't handle multiple files

Related Stories