What do users want?

(hint: more than we expected)

In our health chatbot askNivi, the first step in the user journey is intent classification—what does she want to know? In a previous post, we discussed our approach to answering this question.

At the time we wrote that piece, the challenge was that our algorithm did not know how to handle a substantial chunk of SMS messages. Word was spreading about askNivi, and we started getting a wider range of questions that were tough to classify with the existing model. For instance, we had not seen questions like:

What is the probability of getting cervical cancer when you have sex at around maybe 18 to 20 years?

We needed our crack team of humans to go back and label a growing pile of unclassified messages to further train the system, so we built an MTurk-style app with Shiny and Flexdashboard (see here for a tutorial with code).

After classifying the messages, we had a better picture of the full distribution of topics. In addition to the questions about contraception we expected based on our marketing (mint circles), we also fielded questions about topics we did not advertise much if at all (rose circles), such as sexual health and sexually transmitted diseases. This chart is guiding how we prioritize the development of new automated conversation modules.

While we got our start catering to the health needs of adolescent girls and young women, we quickly found a high demand among men and responded by expanding our service to meet their needs as well. So what do men want to know?

Compared to our female users, men are more interested in questions related to sexual health, relationships, and STIs. But overall, contraception is still the dominant theme, even among men. We chat with loads of men who want to support their partners in family planning:

My girl fears all the other contraceptives…she prefers taking P2. What are the risk?

Listen, analyze, refine. That’s our process.