Most website chatbots invent answers when they don't have one. This one answers only from the business's own documents, says where the answer came from, and says "I don't know" when it can't find it. It is live on this site: look for the button at the bottom right.
The story so far
-
2 Oct
Looking at what exists
I compared the chatbot plugins that already exist for WordPress and the hosted services. The complaints that repeat are two: answers that are made up, and doubts about privacy. None of the ones I saw makes a point of checking itself before answering.
-
2 Oct
Three bets to prove
Written down before building anything: a bot in Spanish that says "I don't know" and cites its source beats the current ones; a "gatekeeper" in front of and behind the search reduces invented answers; and there are businesses in Europe that want their data to stay there. None of the three is proven yet.
-
2 Oct
Testing the gatekeeper
I tried an open model that is meant to act as that gatekeeper, with 32 sentences of my own in Spanish. It saw 9 out of 10 times whether a text answered the question, and spotted all 10 attempts to trick it. Deciding whether it needed to search at all was weaker: 6 out of 12, and 9 out of 12 with a better wording. It is a small sample, a hint and not a measurement.
-
11 Oct
What I will promise
The edge is not speed. I will only promise what can be measured: if it does not find it in your documents, it says it does not know, backed by real accuracy figures from the test. It answers as it writes, in a close tone like talking to a person, and it can say it is an automatic assistant, because European rules seem to require it.
-
13 Oct
The basic version, built
Phase one works: it reads a PDF, answers with the source and says "I don't know" when it finds nothing. With automated tests, and a spending cap.
-
13 Oct
The first big test
I wrote 120 questions about a real document, the national collective agreement for hospitality: 90 with an answer in it and 30 without. The first version got 72 of the 90 right, said "I don't know" correctly 29 times out of 30 and invented one answer.
-
13 Oct
Making it read better
Several changes to how it finds and checks what it reads. The best run reached 81 of 90, 30 of 30 and nothing invented. I then wrote 40 new questions blind, before touching anything else: 24 of the 28 answerable ones right, and all 12 unanswerable ones met with "I don't know".
-
14 Oct
The version I chose
After comparing several combinations, I chose the one that makes the fewest mistakes. On the 112 answerable questions: 106 right, 3 half right, 3 "I don't know" when it did know, none wrong, and in the 38 unanswerable ones nothing invented. On 40 brand new questions it got 40 of 40. In total 134 of 140 right (95.7 %) and 50 of 50 "I don't know" well said. A zero out of 140 is not a real zero: it could be up to 2 %. It is one document, marked by an automatic grader.
-
14 Oct
The gatekeeper stays outside
I tested adding a gatekeeper on top. It did not improve the answers, and one of its checks made them worse. For now it is not part of the chatbot.
-
14 Oct
Reading a business's website
It can now read the whole website of a business by itself, and it does it safely. First test on noteveryone.com, 22 pages and 40 questions written beforehand: 23 right, 5 half right (all for leaving out a detail, none with a false fact), 0 wrong, and 12 of 12 "I don't know" correct. I fixed the reader by looking at the mistakes of those same questions, so that figure is not blind.
-
16 Oct
A chat for my own sites
A chat window, with the look I chose, for this site and for Cofferdock. It answers in the language of the question and never speaks as if it were me. Questions written beforehand: on this site 23 right, 5 half right, 0 wrong, 12 of 12 unanswerable well met; on Cofferdock 24 right, 5 half right, 0 wrong. It has limits per visitor, a spending cap and an off switch.
-
18 Oct
It is live
The chat assistant is now on this site, on every public page, in English and Spanish and with the look I chose. It answers about the projects, the notes and how to reach me, and when it does not know it says so and offers my email. It says it is an automatic assistant that can make mistakes, and the privacy page explains what happens to what visitors type.
-
Next
Keeping it up to date
Next: re-reading the site from time to time so it knows about every change, giving it the memory of the conversation so it understands a follow-up question, and watching the real questions people ask. Then the final name, other websites and the WordPress plugin.
It's live on this very site: use the chat button at the bottom right of any page. The name is still provisional.