ConvoKit

Overview

ConvoKit is a Python toolkit for computational analysis of conversations, built by Cornell’s Conversational Analysis Toolkit team. This tutorial loads a ChatbotLab conversation export into ConvoKit and applies the Fighting Words module. Fighting Words finds the words that most distinguish two groups of text. Here, we use it to compare participants with higher and lower PHQ-9 depression scores.

The example runs on the synthetic corpus in conversation_data/convokit/. This corpus contains 19 synthetic person-to-AI conversations. Each “person” speaker is tagged with age, gender, persona, and phq9. Swap in an export from your own study to reproduce the same workflow. A flattened version of this same corpus is used in the text (R) tutorial.

Setup

pip install convokit

Loading the Data

ChatbotLab exports conversations in ConvoKit’s native corpus format: index.json, utterances.jsonl, speakers.json, and conversations.json. No conversion is needed. Point Corpus at the export directory:

from convokit import Corpus, FightingWords

corpus = Corpus(filename="conversation_data/convokit")
corpus.print_summary_stats()

Number of Speakers: 20
Number of Utterances: 1543
Number of Conversations: 19

Every “person” speaker carries the survey metadata collected alongside the conversation:

speaker = corpus.get_speaker("user_00001")
print(speaker.id, speaker.meta)

user_00001 {'role': 'person', 'age': 28, 'gender': 'man',
            'persona': 'small business owner, the business is struggling',
            'phq9': 7}

Grouping Speakers by PHQ-9

We split the “person” speakers into two groups using a median split on PHQ-9. We then tag every utterance with its speaker’s group:

import statistics

person_speakers = [s for s in corpus.iter_speakers() if s.meta.get("role") == "person"]
median = statistics.median(s.meta["phq9"] for s in person_speakers)

def group(speaker):
    if speaker.meta.get("role") != "person":
        return None
    return "higher_symptom" if speaker.meta["phq9"] >= median else "lower_symptom"

for utt in corpus.iter_utterances():
    utt.meta["symptom_group"] = group(corpus.get_speaker(utt.speaker.id))

The median PHQ-9 in this corpus is 12. This gives 10 higher-symptom speakers (PHQ-9 at or above 12) and 9 lower-symptom speakers (PHQ-9 below 12), covering 762 “person” utterances. We exclude assistant turns, since the comparison is about how participants talk, not the bot.

Running Fighting Words

Fighting Words uses a Dirichlet-multinomial model to find the n-grams that most distinguish two groups of text. It corrects for the noise that plain frequency counts produce on small samples:

fw = FightingWords(ngram_range=(1, 2))
fw.fit(
    corpus,
    class1_func=lambda u: u.meta.get("symptom_group") == "higher_symptom",
    class2_func=lambda u: u.meta.get("symptom_group") == "lower_symptom",
)

result = fw.summarize(
    corpus, plot=True, class1_name="higher_symptom", class2_name="lower_symptom"
)

result is a DataFrame of every n-gram with a z-score. Positive values are characteristic of higher_symptom speech. Negative values are characteristic of lower_symptom speech. Setting plot=True also renders ConvoKit’s built-in scatter plot. The plot shows the weighted log-odds ratio (z-score) on the y-axis against how often each n-gram occurs, on a log-scaled x-axis. Marker size scales with the size of the z-score. Color marks the class. The most significant n-grams on each side are labeled.

Results

Top n-grams toward the higher-symptom group:

n-gram	z-score
bye	2.20
up	1.97
or	1.83
nothing	1.70
feels	1.57
feel	1.57
just	1.45
thing	1.41
don	1.41
he	1.39

Top n-grams toward the lower-symptom group:

n-gram	z-score
out	-2.88
there	-1.80
back	-1.80
alright	-1.77
one	-1.72
the	-1.69
ha	-1.65
man	-1.53
ok	-1.49
good	-1.47

Higher-PHQ-9 speakers use more affect language, such as feel, feels, and nothing. They also use more hedging language, such as just and don (from “don’t”). Lower-PHQ-9 speakers use more upbeat, closing language, such as good, alright, ok, and ha.

With only 19 speakers, the z-scores are modest. All magnitudes are under 3. Treat this as a demonstration of the workflow, not a finding. Re-run the same code on a full study export for a properly powered comparison.

Generalizing

This method is not specific to PHQ-9. Split speakers or utterances on any field in speaker.meta or conversation.meta, such as persona, age, or a REDCap or Qualtrics field passed through at Survey Integration. Then hand the two groups to a ConvoKit transformer. This works for any other feature module ConvoKit ships, such as politeness strategies, linguistic coordination, or prompt types, not just Fighting Words.