How Quartz for Facebook Messenger achieves high retention (with help from Chatbase)

By Karen Hao, Data Analyst at Quartz

In 2016, Quartz broke new ground by re-imagining the news as a conversation. Our app let readers enjoy the top headlines of the day through a seamless chat experience. It was a defining moment for the media industry—and for conversational interfaces.

Two years later, the Quartz Bot Studio once again pushed the envelope with the release of Quartz for Messenger—a new bot experience that takes conversational journalism to the next level. The Quartz bot provides a personalized messaging experience, picking up cues on when to send you content and remembering your preferences over time.

Since its launch, the reception has been overwhelmingly positive. Whereas the vast majority of bots struggle with retention, our users have remained loyal. The industry average for retention after one week is 4%, reports Forbes. But six weeks after launching Quartz for Messenger, we’re still going strong with a 27.2% retention rate.


Retention Cohort Report from Chatbase


The recipe for our long-term success can be broken down into three key ingredients:
  • We play to our strengths as storytellers.
  • We offer brand new experiences that cannot be attained anywhere else.
  • Above all, we respect our users’ time.
Let’s dive into each point. I’ll conclude with some details about how we use Chatbase to help us understand how well we use these ingredients, and what we should do to improve results over time.

Play to your strengths

At Quartz, we are first and foremost storytellers. When we write, we strive to embody the tone of a knowledgeable and witty friend. We always design for these elements—story and voice—regardless of the product.

In the context of a chatbot experience, that means the narrative and tone of our content delivery is just as important as the content itself. Consider this snippet of conversation about kale in our bot:


Quartz has a personable and humorous voice that’s fun to chat with and feels authentic to our brand. It also uses strong narrative principles to order the flow of conversation: an enticing hook that grabs readers (“Would you allow a vegetable like kale to select your future spouse?”), an escalation of information (“It gets better”), and a bigger-picture takeaway that relates the topic back to the reader (“Kale traditions are alive and well outside of Scotland”).

Equally important, we give readers their own voice in the buttons, allowing them to emotionally react to the experience. Depending on their reaction, we adapt the script so they feel understood and heard.

Avoid redundancy

The second key to our success is we never have redundant experiences across our products. Our daily morning newsletter, the Quartz Daily Brief, offers different content than our website; our app offers different content than our email.

When we designed Quartz for Messenger, we followed the same principle. Rather than regurgitate our app for a different platform, we created fresh experiences, including:
  • Obsessions: deep dives into the most fascinating corners of the global economy
  • Projects: adventures designed to make your days easier and more interesting, like a 12-day mindfulness challenge or a recipe experiment
A common pitfall among chatbots is that companies will recreate the same experience that they already offer somewhere else. Perhaps a chatbot allows customers to purchase a product that they could also buy on the company’s website or allows readers to peruse an article available elsewhere. This lowers the tolerance chatbot users have for errors or poorly designed experiences—the moment they hit a snag, they revert back to the other option. If, in contrast, the chatbot unlocks new content designed native to the conversational interface, users are more loyal and willing to put up with imperfection.

Respect the user

Finally, the number one rule that Quartz has always abided by is to have the utmost respect for our readers’ time. This is the underlying driver behind our signature style—short and pithy content that’s dense on high-quality information—and is what led us to create a new approach to notifications that sets Quartz for Messenger apart from our app.

Whereas our app sends notifications at set times each day, our bot personalizes the timing of notifications to each reader’s schedule and preferences. The goal is to create as seamless an experience as possible, anticipating when to give readers the right content to fit the rhythm of their day. This, we believe, engenders higher loyalty from our users than tactics designed to grab and retain their attention. In the long run, the latter is far more disruptive and doesn’t serve Quartz’s ultimate mission to educate and inform.

Using Chatbase to increase our retention

There is always room for improvement, so we use Chatbase to keep a close eye on our retention rate.  Since launching the bot, we have been carefully tracking other important metrics on Chatbase to establish a performance baseline. In the coming months, we plan to iterate on our design and conduct careful A/B tests to bring an even better experience to our users.

Here are our top three strategies:
  • Resolve not-handled messages. One of the most helpful features of Chatbase is its ability to surface not-handled messages. Working closely with the engineering teams at Chatbase and Dexter, the bot-building and deployment platform that we use for our backend, we updated the integration to include a full message report. The not-handled messages give us visibility into any failure points of our conversational experiences as well as the most common user requests. That allows us resolve bugs, improve our NLP capabilities, and produce more content in alignment with user interests. For example, we’re already seeing user requests for news from specific regions or about specific topics.
  • Minimize exit rates. Using the Session Flow analysis tool, we plan to identify where users have the highest exit rates. What we find could lead us to rework a specific conversation or make us rethink what types of conversations users most enjoy. And with any changes we make, it will be important to track whether or not it actually lowers the exit rates.
  • Maximize completion rates. We will use the funnel analysis tool to identify which experiences have the highest and lowest completion rates. What can we learn from the experiences with the best completion rates, and how might we apply those learnings to the rest? How do we rework experiences with low completion rates to be more engaging, informative, and seamless?
Through this three-pronged approach, we expect to achieve even higher retention rates over time. Of course, we haven’t figured out everything and are still learning and iterating along the way. If you have thoughts, we’d love to hear from you. You can reach the Quartz Bot Studio at bots@qz.com.

About Chatbase

Chatbase gives builders of conversational interfaces (or bots) sophisticated tools for creating better, and stickier, consumer experiences than ever before--leading to better conversion rates and retention. Chatbase is a cloud service that easily integrates with any bot platform and type, voice or text, and is free to use.

Among other features, Chatbase uniquely relies on Google’s machine learning capabilities to automate the identification of  bot problems and opportunities that would otherwise take a lot of time, leading to faster optimizations and better bot accuracy.

Chatbase is brought to you by Area 120, an incubator for early-stage products operated by Google.
















Comments