Over the past year, we have been developing survey chatbots as part of our HumanListening™ platform. We have noted the successes of this technology in many marketing applications, and feel there is great potential for this technology to fundamentally change how a lot of research is conducted.

This article summarises our experience so far, highlighting important design decisions and rationales. We’ll also discuss a fun project we created to support our yearly Christmas Charity appeal, and showcase some of the possibilities of this technology.

By now, most of us would have had some experience with a chatbot. In marketing, chatbots are employed to guide a conversation with a participant towards some set goal. This goal could be to place a pizza order, or perform first line service support or navigate a complicated website. This serves as a cost-cutting measure to reduce the need of expensive frontline service resourcesto deal with low complexity enquiries.

On the other hand, in the field of market research, chatbots provide an opportunity to generate enhanced market insights and provide respondents an a better feedback experience.

Broadly speaking there are two approaches to chatbot design – deep learning powered ‘generative’ chatbots and more traditional dictionary and routine based approaches. Both of these approaches can be said to fall under the umbrella of creating some ‘AI’ – artificial intelligence. There is a lot of excitement about the potential of the deep learning approach, and we’ve seen some successes such as Amazon’s Alexa, and some embarrassing failures such as Microsoft’s ‘Tay’. These two approaches differ in that one is ‘black box’ and the second is ‘clear box’, meaning we have greater control in terms of reliability and consistency of action.

In the marketing sphere we’ve seen some further successes which tend to follow the second approach such as the Domino’s pizza ordering app and mental health chatbot applications. There are several instances of hybrid approaches, where a combination of chatbots using deep learning, dictionary approach, followed up by live human support deliver great results, such as in the Zendesk support platform.

Introducing Evolve’s ‘Eve’

Our approach has been modelled primarily on the ‘dictionary’ approach for conversations, with an additional machine learning layer to inform improvement opportunities. Our solution is called the Evolved Voice Engine (‘Eve’ for short). We were tempted to call Eve a ‘survey bot’ but felt that it didn’t do justice to the technical challenges and sophistication that we need to make her. In essence, this technology will enable a simulated conversation within a survey to create deeper open-ended comments, within a defined contextual domain.

Already, Eve is leading to step changes in insights, for our clients, including:

• A natural ‘conversation’ with customers, creating higher levels of engagement, stronger response rates and positive brand impact.
• A dynamic way to learn from people rather than using prescriptive questionnaires.
• At a meta data level, a system that recognises and explores emerging themes, you know what’s going on before your competitors
• Deeper insights from iterative probing; effectively, ‘qualitative at scale’

What metrics are we using to mark success?

To achieve these goals, we have implemented a data driven approach to development. We have identified three key metrics by which to judge the effectiveness of Eve including;

• Information quality – This relates to our ability to predict overall satisfaction or NPS from text data. We would expect to show a greater strength of relationship between data collected through Eve compared to a simple open-ended text box. This is because as we encourage users to provide more detail and probe on additional topics, we obtain richer information for further text analysis and machine learning. On a recent employee engagement survey, we saw a 2.5 times increase in open ended responses with equivalent depth of insight and actionability.
• Respondent engagement metric – Respondent feedback on how much they agree/disagree that the survey helps them feel ‘understood and listened to’. It is our hope that respondents enjoy and appreciate the chatbot approach compared to a traditional long form survey. On a recent customer experience project, we saw equivalent levels of engagement for a survey that was of 4 minutes average duration versus 12 minutes with no equivalent loss of insight.
• Survey completion rate – Similar to the above, this is more of a behavioural metric which helps us determine respondent engagement. It is early days yet, but we are seeing higher repeat survey rates on longitudinal studies.

Latest features

We’ve built several features which can be used to direct the conversational flow, such as how many and which topics a respondent is discussing and length of the conversation. We also use tactical machine learning approaches to measure the emotionality content and sentiment of a conversation. We have created sets of dictionary approaches to manage common scenarios such as getting concrete answers from respondents such as yes, no, or to stop a conversation. We also have capabilities to detect ‘danger words’ where we ought to show more concern and potentially elevate a conversation to a human for actioning.

Given the possibilities of chatbots and the early encouraging results, we’re looking forward to being able to roll out this technology as part of our HumanListening™ platform. Talk to us today about how we can incorporate a chatbot survey design into your market research program.