This guide explains how to use Orlo's Chatbot Topic Report to analyse the confidence levels of your chatbot's answers to inbound user queries.
Things to consider ✅
- Users will need to be an Admin in Orlo to access the Chatbot Analytics Report, which can be configured in the 'User Settings' in Orlo.
- Orlo's Chatbot works across Facebook, X (Twitter), and Live Chat.
Benefits 🙌
- Improve Answer Accuracy - The Topic Report allows users to quickly identify areas where your chatbot may be providing uncertain or low-confidence answers to user queries. This insight enables you to prioritise updates and refinements to your chatbot's Metabase for those specific topics.
How to Run a Topic Report
1) Navigate to the 'Settings' dropdown and select the Chatbot icon in the 'Company Settings' area.
2) Click on the 'Topic Report' button for the chatbot for which you want to run the report.
3) Once you are in the Topic Report, you can customise the report in the following ways:
- Feedback - Positive, Negative, and None
- Handover - Yes/ No
- Confidence Levels
- Keywords
- Topics
- URLs
- Chatbot Widgets
Data in the Report
- User Question: The exact text of the question asked by the user.
- Matched Topic: The specific topic in your Metabase that the chatbot identified as relevant to the user's question.
- Answer Confidence (0-1): A score indicating how certain the chatbot was in its answer (1 being very confident, 0 being not confident at all).
- User Feedback: The feedback provided by the user after the chatbot's response (Positive, Negative, or no feedback given).
- Handed Over to Agent: Indicates whether the conversation was ultimately transferred to a human agent (Yes or No).
- Number of Occurrences: The number of times this specific combination of user question, matched topic, etc., occurred within the reporting period.