By Mark Chatterton, Co-Founder, inGenious AI
InGenious AI, which designs and builds AI-powered conversational interfaces for clients, was an early adopter of the Facebook Messenger Customer Chat Plugin (released as a beta in November 2017). Unfortunately, for one client’s customer-support chatbot based on the plugin, our design’s lack of support for free-text entry led to the default fallback message (AKA, the standard error response) being triggered 57% of the time, leading to a frustrating user experience and an unacceptable abandonment rate of 74%. This was a classic case of a bot doing more harm than good, and we had to act quickly to stop the “bleeding.”
Using Chatbase to quickly and easily analyze that client’s chatbot performance, we were able to reduce not-handled messages (those with misunderstood, unsupported, or missed intents) by 72%, and subsequently the early-abandonment rate by 85%. Customers are now much happier with what was a very broken bot, and that makes for a much happier client!
Eventually, we helped the client deploy the chatbot to its website as well in order to complement their renewed marketing campaign with interactive and automated customer support. But after completing the implementation using the chat plugin, we were surprised by the disparity in the number of users interacting with it via free-text entry versus quick-reply buttons and image carousels: Whereas on the mobile app 93% of chatbot users preferred to interact using buttons and images, on the website that rate was only 39%. This huge difference in the preferred method of interaction suggested that the bot’s original quick-reply conversational flow as initially designed for the mobile app was a poor fit for the website; we simply hadn’t anticipated the huge jump in free-text messages.
Because the chatbot immediately identifies itself as a robot, many customers are willing to send messages a few times in different ways, consequently providing us a rich “paper trail” for analysis to continue enhancing the experience further. If working manually, it would take our team days, if not weeks, to comb through transcripts in the Facebook Pages Inbox for not-handled messages - not to mention verifying exactly when users had abandoned the chatbot. With the bot failing miserably and customers abandoning the client in droves for competitors, this was time we could not afford to lose!
Fortunately, we had integrated the chatbot with Chatbase from the start. Chatbase streamlined the analysis of our failing chatbot by grouping and showing a count of similar messages by intent. This report reduced the number of messages to read by over 66% - a great first step.
We also used the Session Flow view to work out when conversations were going off the rails, and found another big problem: most exits were occuring within the first three chatbot interactions. We were losing customers fast, and it was looking like an uphill battle to get back to the original early-abandonment rate of 3.8%.
We quickly determined that our top priority should be to tweak initial welcome messages and conversational-flow design to better support free-text entry. We also generated and integrated a list of common misspellings, as we could see people continually struggling with the spelling of some product names - which initially was one of the major drivers for using quick replies and image carousels. Those two changes alone helped reduce the overall not-handled rate by 33% very quickly, which was far from a complete victory but strong progress nonetheless.
Next, we took a deeper dive into conversational-flow exits and their corresponding not-handled messages to get a better sense of where the customer experience was going wrong. At that point, we discovered that critical information requested by customers was buried quite deep in the conversation flow. In response, we optimized the flow based on these learnings and the new data coming in from the updated free-text conversational design and misspellings. Those changes cut the not-handled rate to just under 16% and the abandonment rate to 11%. There was still work to do, but we had returned the bot to competency!
As shown in this particular example, with Chatbase, we can quickly and easily pinpoint the source of poor experiences to fix failing bots, as well as prove to clients that our optimizations lead to measurably better customer satisfaction. Without it, it would take a huge effort to do those things manually.
About Chatbase
Chatbase gives builders of conversational interfaces (or bots) sophisticated tools for creating better, and stickier, consumer experiences than ever before--leading to better conversion rates and retention. Chatbase is a cloud service that easily integrates with any bot platform and type, voice or text, and is free to use.
Among other features, Chatbase uniquely relies on Google’s machine learning capabilities to automate the identification of bot problems and opportunities that would otherwise take a lot of time, leading to faster optimizations and better bot accuracy.
Chatbase is brought to you by Area 120, an incubator for early-stage products operated by Google.
InGenious AI, which designs and builds AI-powered conversational interfaces for clients, was an early adopter of the Facebook Messenger Customer Chat Plugin (released as a beta in November 2017). Unfortunately, for one client’s customer-support chatbot based on the plugin, our design’s lack of support for free-text entry led to the default fallback message (AKA, the standard error response) being triggered 57% of the time, leading to a frustrating user experience and an unacceptable abandonment rate of 74%. This was a classic case of a bot doing more harm than good, and we had to act quickly to stop the “bleeding.”
Using Chatbase to quickly and easily analyze that client’s chatbot performance, we were able to reduce not-handled messages (those with misunderstood, unsupported, or missed intents) by 72%, and subsequently the early-abandonment rate by 85%. Customers are now much happier with what was a very broken bot, and that makes for a much happier client!
Diagnosis: Unanticipated free-text messages
The chatbot in question was originally built for the Facebook Messenger mobile app to improve the customer-service experience. This bot was considered highly successful, with a conversion rate of 36%, default fallback triggered by only 4.7% of messages, and an early-abandonment rate of 3.8%.Eventually, we helped the client deploy the chatbot to its website as well in order to complement their renewed marketing campaign with interactive and automated customer support. But after completing the implementation using the chat plugin, we were surprised by the disparity in the number of users interacting with it via free-text entry versus quick-reply buttons and image carousels: Whereas on the mobile app 93% of chatbot users preferred to interact using buttons and images, on the website that rate was only 39%. This huge difference in the preferred method of interaction suggested that the bot’s original quick-reply conversational flow as initially designed for the mobile app was a poor fit for the website; we simply hadn’t anticipated the huge jump in free-text messages.
Because the chatbot immediately identifies itself as a robot, many customers are willing to send messages a few times in different ways, consequently providing us a rich “paper trail” for analysis to continue enhancing the experience further. If working manually, it would take our team days, if not weeks, to comb through transcripts in the Facebook Pages Inbox for not-handled messages - not to mention verifying exactly when users had abandoned the chatbot. With the bot failing miserably and customers abandoning the client in droves for competitors, this was time we could not afford to lose!
Fortunately, we had integrated the chatbot with Chatbase from the start. Chatbase streamlined the analysis of our failing chatbot by grouping and showing a count of similar messages by intent. This report reduced the number of messages to read by over 66% - a great first step.
Finding and plugging design gaps
Our next step was to prioritize the discovery of those not-handled messages that were causing abandonment. With Chatbase providing that service automatically, it was quite obvious we were in trouble, with over half of not-handled messages causing frustrated customers to leave.We also used the Session Flow view to work out when conversations were going off the rails, and found another big problem: most exits were occuring within the first three chatbot interactions. We were losing customers fast, and it was looking like an uphill battle to get back to the original early-abandonment rate of 3.8%.
We quickly determined that our top priority should be to tweak initial welcome messages and conversational-flow design to better support free-text entry. We also generated and integrated a list of common misspellings, as we could see people continually struggling with the spelling of some product names - which initially was one of the major drivers for using quick replies and image carousels. Those two changes alone helped reduce the overall not-handled rate by 33% very quickly, which was far from a complete victory but strong progress nonetheless.
Chatbase stats showing a significant reduction in not-handled messages
Automated analytics for the win
We are continually analyzing customer messages and making small tweaks as needed and now have the chatbot performing at close to its original level. (The new Transcripts feature should be really useful by providing more context around not-handled messages.) And, we’ve learned a lot along the way about how people interact with the chatbot through different channels - for example, when using a keyboard on a desktop versus on a mobile phone.As shown in this particular example, with Chatbase, we can quickly and easily pinpoint the source of poor experiences to fix failing bots, as well as prove to clients that our optimizations lead to measurably better customer satisfaction. Without it, it would take a huge effort to do those things manually.
About Chatbase
Chatbase gives builders of conversational interfaces (or bots) sophisticated tools for creating better, and stickier, consumer experiences than ever before--leading to better conversion rates and retention. Chatbase is a cloud service that easily integrates with any bot platform and type, voice or text, and is free to use.
Among other features, Chatbase uniquely relies on Google’s machine learning capabilities to automate the identification of bot problems and opportunities that would otherwise take a lot of time, leading to faster optimizations and better bot accuracy.
Chatbase is brought to you by Area 120, an incubator for early-stage products operated by Google.
Comments
Post a Comment