Ask HN: Custom AI LLM Solutions vs. Open Source Tools?

Has anybody else chosen the route of developing a fully customized solution to integrate LLMs into their workflow instead of using some of the available options (AutoGPT, AgentGPT, LangChain, and others)?

I'd love to hear about your experiences and any challenges you faced. So far, my most challenging issue has been creating an output format that doesn't fail 20-30% of the time.