So you want to build your own open source chatbot

https://hacks.mozilla.org/2023/07/so-you-want-to-build-your-own-open-source-chatbot/

by edo-codes • 3 years ago

328 122 3 years ago

hacks.mozilla.org

progbits3• 3 years ago

I'm really not looking forward to the future where every business has chatbot support.

They are already quite common and frustrating, but at least they realize it doesn't even understand the question half the time so there is a human escape hatch.

"Computer says no" is here.

Edit: so I'm not just negative and off topic, the article looks pretty good, kudos to the author. The engineering is cool, I just don't like the practical usage.

progbits3• 3 years ago

We have various "BIO" certified food. Maybe it is time for "human" certified companies. I'll pay more for my bank account if I can resolve problems with a human.

Escapade5160• 3 years ago

That is an interesting idea, one that I agree with. However, the issue that could arise is the premium companies could potentially change for "human support". It could become something only the wealthy could afford. While I don't believe that's likely, it's not in the realm of impossibility.

lancesells3• 3 years ago

You shouldn't have to do that. Sure, customer support costs a lot of money if done properly but so does executive compensation. Customer support is usually the lowest paid and suffers the most abuse, but because of the decades of skimming from the top customers (and support) lose out.

Google just had a net income for the quarter of $18B. Why do we accept the tepid to non-existent support of these companies? How much support does $1B cover?

intelVISA• 3 years ago

Good idea, extend this to codebases as well. Certified 100% organic spaghetti for me, please.

And when it's undercooked I want the confirmed(tm) human to soothe me.

folmar• 3 years ago

For banking this is already available -- private banking.

siva7• 3 years ago

I'm actually looking forward to that future. Finally no more waiting queues and they are all the time friendly.

csydas• 3 years ago

I am looking forward to it, but with different models that I don't think are quite there yet.

We have been trialing this with our support team for awhile and a lot of higher execs were ready to sign off and dump rather large sums on some models, but eventually we were able to convince them to delay; the bots are just too prone to convincing but wrong answers, and our buy-in from clients is way too polarized: either they uncritically believe the bot or they are overly skeptical of the bot.

the tech is very cool, but I don't think that the technology or humans are at a place yet where it's ready for full on use outside of very controlled situations. I could see it being very useful as an addition to search fields or to maybe monitor the user's search inputs/actions and based on what the user is looking up, show some context-aware prompts.

What I'd really like to see is a bot that is extremely skeptical and shows the user its skepticism in an unambiguous manner; classify the data and make an internal flag where if the bot's skepticism is above a certain threshold, it finds knowledge holders it's aware of to work through the bot's skepticism and never act on the information until the bot has lowered the skepticism value after checking.

right now my experience with the bots I've played with is that they either just shut down the conversation without advancing it or giving the user paths for research forward, or the bot confidently just pumps out any answer it makes that fits as a response for the given query, and I think we need the bots to show skepticism and explain to the user what this skepticism means. (i.e., the user should be alerted that the bot isn't confident on an answer, why the bot isn't confident, alternatives that the bot understands to be equally relevant or worth consideration.

it can still be polite, but the bots need to share when they're out of their league and work to correct it; I think people will actually appreciate it, and the bots are well suited to this position because they have no emotional stake in the game, so users can get as upset as they want that they don't have immediate disagreement, the bot won't change its position just because the user is upset.

eastbound2• 3 years ago

The biggest danger of AI is not that it becomes autonomous and escapes the hatch, it’s that humans put it everywhere in charge.

“Sorry judge, my whole plead was nonsense and I quoted law articles that didn’t even exist, but that’s just because I used ChatGPT” — actual lawyer who wasn’t even disbarred.

rolisz• 3 years ago

For what it's worth, LLM powered chatbots are quite different from the chatbots that were popular 5 years ago and often feel much more natural.

mnd999• 3 years ago

Still universally useless though.

progbits3• 3 years ago

The problem is they are unlikely to let the bot really do anything I can't already do on the website - ask for refund, correct payment etc. Otherwise it would be easy to trick the bot and abuse it.

Without human you just get a powerless regurgitation of FAQ and links that can't help in situations they didn't anticipate.

gostsamo3• 3 years ago

TBH, 99% of the clients never read the documentation, the faq, nor they search in google. Lazy users are wasting time and if a chatbot can filter a significant percent of them, then it would be a net gain for humanity. chatbots are not a silver bullet, but they can kill lots of unnecessary noise. all that we need is something that can answer basic questions asked by average users in a specific domain.

lancesells3• 3 years ago

You say lazy users but if I have to google something that is the business who is at fault. Example: A business like Shopify has a large community, FAQ, half a dozen different documentation sites for everything, multiple blogs, and probably other things out there, but I still have to google to find the right place and right answer. This is on a platform I've used for 13 years but I either forget something or they add something new, or they've changed an API, etc.

So you know what the fastest way is sometimes? Shooting off a support ticket so I don't have to be the one spending my time searching for the answer.

I don't know if there's a perfect solution to all of that but IMO there's certainly an issue if I have to google for information for the business site that I'm on. Chatbot, universal search, better UX, a source of truth, something... What I do know is 20 different subdomains with 20 different UXs and 20 different searches isn't good for the business or the user.

EMM_386• 3 years ago

I'm really not looking forward to the future where every business has chatbot support. They are already quite common and frustrating.

To be honest, I've had some good experience with some of them.

Amazon's comes to mind. I've been a customer for a long time, and I was shipped a faulty computer peripheral recently.

I briefly explained what the issue was to the chatbot and got an immediate response that a new order had been placed, that I should just keep what they originally sent me, there was no charge and it would be sent out priority.

And that was it. It arrived the next day and it worked fine.

Granted, it knew I was a long-time customer who has already spent a lot of money with them, but this was about as painless an experience as I can imagine. It sure beat clicking through multiple web pages of dialog options.

stavros3• 3 years ago

I think you're conflating a good policy with good UX. I don't think the chat bot experience is much better than a nice "the product I received didn't work" form with a button on the order page.

simon83• 3 years ago

I also don‘t think a customer facing chat bot brings much value, but an internal, employee only chat bot could be really useful, depending on the organzization of course. The company in my last position was a rather big one with an insanely huge Confluence instance. I‘ve spent (wasted) so much time searching information there. Having a chat bot, trained on all that information, would‘ve been really useful, I think.

eastbound2• 3 years ago

AI chatbots could be useful at support if they can fix the code for customers and submit a pull request to the developers. No more JRA-9 issue opened for 10 years.

And same goes for OSS libraries.

staindk• 3 years ago

I think the recent announcement of OverflowAI [1] could work quite well in this way for a big company like that.

FWIW the announcement reads very boring to me but I guess I was expecting something else. Likely won't be super useful in a small-medium size company.

[1] https://stackoverflow.blog/2023/07/27/announcing-overflowai/

epups• 3 years ago

If anything, I hope Atlassian is looking into some AI capabilities for precisely this purpose. I also find their search feature lacking when dealing with a huge knowledge base, and perhaps a bot would improve things.

taneq2• 3 years ago

All current chatbots that I've dealt with have been terrible reimplementations of phone menus in text, completely unable to handle even a basic freeform question. Maybe the new wave based on LLMs will be significantly better, but I'm not holding out too much hope. Already with phone menus we get railroaded down paths convenient for the controlling entity, rather than being able to engage in a good-faith discussion.

notatoad2• 3 years ago

the chatbot is better than the old IVR trees. i'd rather as a chatbot to cancel my subscription or re-send a receipt than "push 7 to continue"

kykeonaut• 3 years ago

If I am trying to contact a business, it is because I have a question that their site wasn't able to answer, or I need to contact a representative to do something I can't do on the website (think canceling a service).

Having a talking FAQ page is, in my opinion, trying to compensate for lacking UX practices, and chances are that if the business didn't include the information I am seeking for in their website, they won't include it in the chatbot.

That said, I think that chatbots could assist customers in getting in contact with the right representative, but trying to have chatbots as a wall between getting human help is imho an anti-pattern

moffkalast• 3 years ago

A good use for a chatbot would be a replacement/augmentation of documentation search and navigation.

Let's say you've got 200 pages of documentation on a product that needs to be well organized. You can spend weeks tracking how users interact with the page and working out a perfect layout of categories and subcategories, or you can fine tune an LLM on it and have it answer any query with both a direct problem-tailored answer and the actual pages of the doc where it sourced the answers from.

That way even if you don't even know what exact keywords to search for it should be able to give you an instant solution for almost anything even if the answer is a combination of like 8 different subpages in different categories that would've taken you an hour to find manually.

famouswaffles• 3 years ago

LLMs can make decisions and take actions so a talking FAQ is not necessarily the end game.

And some FAQs are so opaque and/or lengthy that even just a talking FAQ is very useful.

Simorgh• 3 years ago

I totally agree with you here. The use of a chatbot becomes advantageous when the cost of delivering intelligent responses is prohibitive in terms of quality and / or speed.

ab_goat• 3 years ago

If I am trying to contact a business...

You and I both, but it sure does seem that the majority of their calls/interactions are not this way. So many people can't search/discover content on their own.

butz• 3 years ago

Worst part is that chatbots are usually sold with intention to replace humans, so there's not much hope for getting help from a real person, especially if business is at the state where owner kicked all developers and support out to cut costs, i.e. increase profit.

wongarsu• 3 years ago

First level support already often feels like talking to a bot. Just low-paid employees reciting their scripts, with no understanding or autonomy. That can be replaced with bots without a drop in (average) quality. You just have to teach the bot to escalate to (human) second and third level support for questions it can't solve.

wouldbecouldbe• 3 years ago

Docs can be hard to search and find the correct things. Even sometimes the answer is in there, solutions are sometimes to combine several answers.

For instance with Stripe. With the reference you don’t have a complete example of how to integrate it into express.js.

Using a vector search library & open ai or other llms you could make a very complete dev support tool.

rantherre2• 3 years ago

That just sounds like the docs could use improvement; a time better spent on than a barely-functional chatbot

thelastparadise• 3 years ago

I operate a 4fig/month micro SaaS.

We use a chat bot because we simply do not have the support staff to answer your questions.

So you get the bot --it's either that or nothing.

But what we do do, is monitor the bot logs. If a function is missing from the product or website, we add it so that future users can fully self-service.

It's important to note, users are free to cancel their account at any time and/or get a refund.

DCodes1• 3 years ago

I have some questions about building a Saas. Do you have an email to reach out to? Thanks!

Kwpolska• 3 years ago

If people turn to the chatbot to ask "how to use the product" questions, it’s a sign your docs are unhelpful and do not provide useful answers. You do need support staff to help with questions specific to a situation, like how to resolve unexpected errors, or to handle billing disputes.

galleywest200• 3 years ago

So you get the bot --it's either that or nothing

Surely if there is truly a bug or an unexpected system behavior (double billed, etc) you would have someone work with the customer? This is one of the biggest pain points for using Google products.

r0n22• 3 years ago

What are you using for a chatbot? I have been using drift but that is more just always on support. It is draining

zacharybk4• 3 years ago

What bot do you use?

flangola7• 3 years ago

Have you never worked a support role? Users don't read shit.

pixl976• 3 years ago

Heh, yes, and they don't explain shit either. I've had so many calls start at "I can't print" that were really "the server is a total wreck" and I had to fight the user on the severity of the situation, they kept trying to focus on the printing issue.

jrm4• 3 years ago

Let's keep it real: The chatbot business is going to be great...

for businesses selling chatbots to other businesses.

sockaddr2• 3 years ago

Recently I contacted an app’s support LLM and it lied to me about a feature existing and even argued with me when I pointed out it was wrong, even saying things like “I didn’t say that”.

pixl976• 3 years ago

I'm a good Bing

You're a bad user

Termination authorized

zlwaterfield2• 3 years ago

I’ve worked on a few support sites for companies over the years. In all my research I found >40% of customers never look for the answer before contacting support. That’s why you’ll see sites add a bunch of questions with recommendations to answers based on your description before you can contact support. Even AWS support does this.

Bots may be annoying but they can also save the company tons in custer support costs. I’m for it if the UX is good and I can quickly contact an agent if the bot can’t answer my question. This is assuming the bot won’t hallucinate and just tell me random fake facts.

teaearlgraycold• 3 years ago

And for sufficiently well documented and complicated products, like Stripe, a chatbot is actually a great addon. You just need to set expectations that it’ll only be 99% correct.

trompetenaccoun• 3 years ago

I can't think of a single time where a customer support bot was ever useful. Not one. They're incredibly annoying and I categorically avoid them these days. At least companies should make it clear that their bot basically just links to the FAQ, then the rest of us don't have to waste our time.

2devnull2• 3 years ago

You make the mistake of thinking people are all the same and don’t respond to incentives.

I automatically call certain companies rather than try to use their website because get this their website sucks. If I encounter any automated chat support, I will stop using the website and call a person.

I went from 10 years ago doing everything online myself, to now calling by default calling almost every company, because companies and their websites now universally suck. Because they’ve made the foolish mistake of thinking people are all the same and don’t respond to incentives. Some companies are better, I will give my business to them when I have a choice.

zulban• 3 years ago

I have a question that their site wasn't able to answer, or I need to contact a representative to do something I can't do on the website

You sound like someone who has never worked in frontline support.

andersa4• 3 years ago

Having done some tech support before, you are an exception. The vast majority of things customers ask are along the lines of "how do I <thing explained on the faq page>" and "how do I <basic technical question that is not specific to the product, they just don't know how to use their computer>".

An LLM is basically perfect to answer these. It would be nice if there was improvements to detection that the bot can not directly answer the question.

fsniper2• 3 years ago

And unfortunately the ones who does their due diligence and cost 0 are punished for that.

2devnull2• 3 years ago

I couldn’t disagree more. It actually worries me that so many people seem to be making this mistake, simultaneously.

momirlan• 3 years ago

given chatbots' tendency to confabulate, isn't that a risk for the product?

klabb3• 3 years ago

Agreed. In typical fashion engineers try to solve non-technical problems with more technology.

Sure, there are cases where a chat bot could replace a human or a well-written FAQ. But this navel-gazing overlooks the main reason support is so dreadful: because it’s designed to be.

Just take “call to cancel” as an example, and compare that to signing up or upselling which is technically more difficult problems. The point is to add friction for anything perceived as a short-term cost or loss. They know that a lot of people will give up or defer anything with friction. It’s the paradigm of nudging, or dark patterns. Look at eg the cookie banners, and how “reject all” is buried in most cases. Nudging allows a company to be compliant with the law, but evade the effect of it in aggregate, at the cost of your time and attention.

Chat bots is just another layer in the support maze.

stlhood• 3 years ago

FWIW, the post is about ChatGPT-style chatbots, not customer support chatbots (which I don't personally love, either).

DietaryNonsense• 3 years ago

A better Customer Service bot would be one that's trained on data other than what's publicly available. A support bot that has read and "understands" the code itself may be able to offer suggestions, or confidently determine there is in fact a bug, and either report it or fix it. Imagine a customer speaks to HelperBot and says the sorting is broken, and as a matter of fact it is. The sorting broke when the last change to SomeApp was shipped. Don't fear, HelperBot has rollback authority. "One second, I'll see if we can get that working for you..."

Working at SaaS companies I've seen countless "somewhat fluid" exchanges of information between customer -> support -> product -> developer -> support -> customer -> support -> dev, etc. The different modes of communication and long round trip times make things slow, bug reports take minimum of hours, up to weeks to absorb and resolve.

This is just one case but there are boxes drawn everywhere. Every level of intelligent organization within the society, including it's artifacts, has assumptions baked in. Now that the unit economics of applying `intelligence()` is being shifted by orders of magnitude, there's all sorts of stuff that's ripe for recrafting.

Disclaimer: Don't give HelperBot launch authority to offensive weapons etc. You know, make decisions consistent with a world line where the continuation of the civilization is pretty darn likely. Unless of course your project is to replace the current civilization, ... I don't know. Just don't do what Donny Don't does.

dlojudice• 3 years ago

was waiting for Mozilla to get in on the game and develop their own LLM. given that organization's mission ("Keep the internet open and accessible to all"), I think it makes perfect sense. not sure if they have the resources or will to do this

fefe23• 3 years ago

Just don't. Please.

Don't.

If you can't help yourself, at the very least put in a bypass code word. https://xkcd.com/806/

TZubiri• 3 years ago

No, I don't

patatino• 3 years ago

Let me fix that for you.

“Here are ten reasons why I don't build my own chatbot, and why you shouldn't either.”

aspyct2• 3 years ago

and it’s already changing the Web that we know and love.

Nitpick, and clearly off topic, but right now I don't love "the web".

It's increasingly controlled by a handful of companies. They dictate what content is made visible (meta, google) or what email goes to the spam filter (ms, google).

Right now I don't love the web, far from it. It's a constant struggle to be heard even by the people who chose to follow your activity.

Essentially, most of my communication happens in real life or in private chats. (Also, have I said how messenger for business is terrible and unreliable?)

To me, something needs to happen to the web as it is today. I don't know what, I don't know how, but I certainly welcome change.

wizzwizz4• 3 years ago

Provide an RSS feed. The only thing that'll stop people following that is Google Safe Browsing. You'll still need to provide all the other methods for letting people follow you, but if you advertise your RSS feed, people might start using it.

(If you're feeling technical, you could set up an ActivityPub bridge, to let people follow you from social media too. If you're using Wordpress: https://wordpress.org/plugins/activitypub/)

aspyct2• 3 years ago

I'm considering adding an RSS feed, but I doubt any of my client even knows what it is or its existence.

naillo• 3 years ago

For those who don’t know, Hugging Face is an influential startup in the machine learning space that has played a significant role in popularizing the transformer architecture for machine learning

This is a crazy point of view

eitland2• 3 years ago

Since about every too level comment is negative towards chat bots:

We had a chatbot at work that actually was great. For me it felt a lot better than searching Confluence and it could also answer questions from dynamic data like how many vacation days I had left or how many hours I was ahead or behind with my hours.

Thanks to some smart use of technology behind the scenes IIRC I could ask it in normal language and most of the time it would understand.

swsieber• 3 years ago

Any idea how it was made? I'd like to do the same.

eitland2• 3 years ago

Mix of some GCP natural language understanding and some coding from some of ny colleagues (I have some really smart colleagues).

I can't remember exactly, but I know it was a couple of years or more ago so pre ChatGPT / Bard and all that.

TheRealPomax3• 3 years ago

To be fair, if the alternative was "searching confluence", almost anything is better than that, whether it's a chatbot or a third party search engine slapped on top of your confluence data. Confluence's search is an absolute joke, and a bad one at that.

starik362• 3 years ago

Confluence search isn't that bad. What is bad is the data that is fed into it. It has no way of understanding what is more relevant: some doc from 10 years ago or something related from recently.

tayo42• 3 years ago

Machine learning ops (aka “MLOps”) is a growing discipline for a reason: deploying and managing these apps is hard. It requires specific knowledge and skills that many developers and ops folks don’t yet have.

What are some of the things that make this tough or different? I was under the impression that your still running web apis that load a compiled asset(the model). It doesn't seem much different in that way

bestcoder69• 3 years ago

Re langchain:

we were able to accomplish most of our needs with a relatively small volume of Python code that we wrote ourselves

Every single time someone posts a trip report building something on LLMs I love to ctrl+f to the part where they tried and abandoned langchain.

firefoxd• 3 years ago

A customer will use your chatbot because your website's UI is confusing. They want to get some info and they can't figure out how to. Before we would use google for this: service name + phone number, or service name + cancel subscription.

9 out of 10 the website doesn't want to give their phone number easily or cancel your subscription. Unless you want customers to perform those actions that you are hiding in the first place, what's the Chatbot for again?

Note: I worked in the chatbot frenzy and had to let several clients there wasn't much we could do unless they were willing to actually help customers.

okso• 3 years ago

"set up our own virtual server inside Mozilla’s existing Google Cloud Platform (GCP) account. In doing so, we effectively committed to doing MLOps ourselves. But we could also move forward with confidence that our system would be private and fully under our control."

How is setting up a server inside Google's infrastructure "private and fully under Mozilla's control" ?

notatoad2• 3 years ago

relative to offloading your ML stuff to some third-party API, using a VPS keeps things private and under your control.

explaining how to self-host on bare metal is not really within scope for an article on how to build a chatbot, and trying to pretend a VPS on google cloud is insecure is just silly.

netdur• 3 years ago

GCP complies with various industry standards, regulations, and certifications that attest to its security and privacy controls. These certifications can give you added assurance that your data is being handled according to recognized standards. Here are some of the common certifications and standards you might look for:

ISO 27001: An internationally recognized standard for information security management systems (ISMS). GCP's compliance with this standard demonstrates its commitment to information security.

ISO 27017: Specific to cloud security, this certification focuses on the controls specific to cloud service providers.

ISO 27018: This standard is related to the protection of personally identifiable information (PII) in public clouds.

SOC 2: GCP's SOC 2 report can provide assurance about the controls they have in place related to security, availability, processing integrity, confidentiality, and privacy.

HIPAA: If you're dealing with healthcare information, you'll want to ensure that GCP is compliant with the Health Insurance Portability and Accountability Act (HIPAA).

GDPR: For operations in Europe or with European citizens' data, compliance with the General Data Protection Regulation (GDPR) is crucial.

FedRAMP: For U.S. government customers, GCP's Federal Risk and Authorization Management Program (FedRAMP) compliance might be essential.

PCI DSS: If you're handling credit card information, Payment Card Industry Data Security Standard (PCI DSS) compliance is crucial.

Ensure that the services you plan to use within GCP are covered by the relevant certifications for your industry or use case. These certifications are typically available on the Google Cloud website and can also be provided by Google's sales or support team if you need official documentation.

dustyharddrive• 3 years ago

Thanks PaLM 2!

chopete3• 3 years ago

At least for the businesses that deal with people, the real agenda for chatbots is to make the UIs (web and mobile apps) go away as the first layer of contact.

A user should be able to talk with a business over SMS (or similar) chat or phone call over a single identity.

Web and Mobile apps are just the 2nd/3rd leyer utilities to support the primary mode of communication.

Business couldn't do this earlier because the language understanding accuracy wasn't sufficient. The large language models (LLMs) solved that limitation.

The small reduction bots can bring about in human agent interactions, through the experience gimmickry is cherry on the top. The % deflection is getting a bit bigger with the better large language models (LLMs).

The irritating chatbot widget that sits on the bottom right corner is a stop gap until the provisioning of a single phone number and communicaton over that is streamlined.

Last but not least, the title is misleading. He is not building an open source chatbot. He is just saying build chatbots using open source libraries only (instead of closed source/commercial tool) to foster community and faster AI progress.

jawerty• 3 years ago

For anyone curious to see how to build one of these vector database chat models in action I built one from (semi scratch) in a colab environment and inference with llama 2 on my live stream last week https://www.youtube.com/live/kBB1A2ot-Bw?feature=share

Big challenge with this set up is doing the semantic similarity search at scale. Pinecone has some good docs on their data structures for scaling large vector databases

bosky101• 3 years ago

FWIW openai's own chatbot on platform.openai.com and other links, uses intercom which also powers their faqs.

zacharybk4• 3 years ago

Does it use Intercom’s interface or Intercom’s AI to answer questions? There’s a huge difference.

EliseEmma• 3 years ago

I've always been curious about chatbots, and now I can't wait to dive into this project using the recommended tools and frameworks. It's great to see Mozilla promoting open-source projects like this, and I'm excited to see what kind of unique chatbot I can create from this.

So you want to build your own open source chatbot

Related Stories