Understanding Bot Analytics

Analytics Platform for Virtual Assistants:

Now it is easier to understand and evaluate your Virtual Assistant's users, usage, and performance with our interactive dashboard that comes with a whole lot of valuable visual insights.

Aim:

To help you better understand how your Virtual Assistants are performing, whether the conversation flows are working as intended and which ones need improvement.

Analytics available now:

Usage Analytics:

• Added the Charts on Assistant Hit-Rates and Resolutions-Statuses to provide insights on how the assistant is handling the conversations.
  • The charts will show the proportion of the reasons responsible for hits and misses on the assistant.
  • It will also drill down to present the resolution status of the issues handled by the assistant.
• Added the Idle Misses and Response Misses tables to drill down on exact user messages, assistant responses, and associated flow to understand where Idle misses or response misses are happening.

User Analytics:

• Added the Number of Users graphs to show the number of existing and new users who accessed your assistant.
• Added the Returning Users graphs to show how much percentage of previous users returned on that day to your assistant.
• Added the Retention Matrix to better understand the user behavior and present the pattern of how many prior users are returning for the following 12 days.

Conversation Analytics:

• Added the Intents by the number of Utterances charts to present the intents that the user's messages matched most frequently.
• Added the Intents by the number of Conversations charts to show the intents that were involved in the most conversations.
• Added the Top Intents by High Confidence charts to provide insight into how confident your top-performing intents are at matching utterances. See which intents are the most confident.
• Added the Top Intents by Low Confidence charts to provide insight into the intents with the lowest confidence scores.

Training Analytics:

• Added the Stats Summary to provide the composition of keywords added and associated with each knowledge base, intent, entities, Q&As, and triggering phrases for training.
• Added the Performance Summary to better evaluate and understand the performance of intents during Model Training. 
  • The graphs and charts will provide insights on Accuracy, Precision, Recall, and F1 score for intent during the training, helping you understand which intents need improvement to better train the assistant.
  • The Insights will also drill down on true positives, true negatives, false positives, and false negatives to provide insights on what exactly needs improvement and how to improve those intents.
• Added the Training History graphs to present the performance history of each training of the assistant to help monitor the assistant's improvement/degradation over the course of different deployment iterations.
• Added the Confusion Matrix to report on the number of confusions occurring between every two intents. The higher the confusion number, the more likely they are overlapping; hence recommended to either modify or delete one of the confused intents.
• Added the Confusion Report on exactly which phrases in predicted intents are causing the overlap with actual intent to provide insights on which phrases for these intents need to be changed.