Two-stage LLM survey coding: turning hundreds of open-ended answers into categories and themes without a predefined scheme

STEP 1a: Give the model all the answers → it generates a list of primary categories with descriptions (separate prompt)
STEP 1b: Give the model the same answers + the category list from step 1a → it assigns each answer one or more categories with an explanation (separate prompt)
STEP 2a: Give the model the list of primary categories from step 1a → it groups them into larger themes with descriptions (separate prompt)
STEP 2b: Give the model the category list + the theme list → it assigns each category to a theme with an explanation (separate prompt)

All four steps are separate requests; the output of each step is copied into the next prompt as input.

Task: HR ran a survey of 300 employees with the open question "Why would you consider leaving the company?" They need to understand the main reasons without a predefined category list.

Prompt (step 1a):

You are an experienced researcher specializing in qualitative text analysis.
Your task is to analyze employee survey answers and identify suitable categories.
Study the answers below and find recurring themes, motives or intentions.
At the end, output a table listing the categories you found with a short description (1-2 sentences) for each.

Here are the employee answers:
1. [answer 1]
2. [answer 2]
...
n. [answer n]

Output the table in Markdown, one row per category.
First column: category name; second column: category description.

Result: The model produces a table of 5-10 categories (e.g., "low pay," "no career growth," "conflicts with management") with descriptions. Then this list goes into the step 1b prompt, where the model sorts all 300 answers into these categories with a short explanation for each. After that, steps 2a and 2b merge the categories into 2-3 big themes (e.g., "financial dissatisfaction" and "management problems").

An LLM does poorly if you ask it to produce a final classification of hundreds of answers in one go: it gets confused, repeats similar categories in different words, loses consistency over a long list. That's the weakness: the model doesn't hold a single "registry" of categories in its head unless you give it one explicitly.

But the model is excellent at spotting patterns in text and applying a ready-made category list to new text; that's essentially a classification task, where it's stable. Splitting into "come up with categories" (creation) and "sort into categories" (application) uses exactly this strength: at each step the model does one simple thing instead of juggling creation and application at once.

Control levers: - Number of answers per request → if there are many (500+), split them into batches and after each one ask the model to add new categories only if the old ones don't fit (as in the original: "priority is to use the categories already created") - Asking for an explanation ("in 1-2 sentences, why") → drop it if you only need the result without justification; it will speed up and shorten the answer - Abstraction level of themes in step 2a → explicitly ask for "not too narrow, not too broad"; agreement with human logic depends heavily on this

STEP 1a — Generating categories:
You are an experienced researcher specializing in {analysis_type, e.g. "qualitative analysis of customer reviews"}.
Your task is to analyze the texts below and identify suitable categories.
Find recurring themes, motives or intentions.
Output a Markdown table: a "Category" column and a "Category description" column (1-2 sentences).

Here are the texts:
{text_list}

STEP 1b — Assigning categories:
Your task is to assign each text to one or more categories from the list below.
Explain in 1-2 sentences why that particular category was chosen.

Here are the texts:
{text_list}

Here are the categories:
{category_list_from_step_1a}

Output a Markdown table: "Text number" | "Category(ies), separated by semicolons" | "Explanation".

STEP 2a — Generating themes:
Your task is to analyze the list of primary categories and identify broader, unifying themes.
Choose an appropriate level of generalization — not too narrow, but not too broad either.
Output a Markdown table: "Theme" | "Theme description".

Here are the primary categories:
{category_list_from_step_1a}

STEP 2b — Assigning categories to themes:
Your task is to assign each primary category to the most fitting theme.
Explain your choice in 1-2 sentences.

Here are the primary categories:
{category_list_from_step_1a}

Here are the themes:
{theme_list_from_step_2a}

Output a Markdown table: "Category" | "Theme" | "Explanation".

Put your own data into {analysis_type} and {text_list}: it can be a survey, reviews, interviews, comments.

🚀 Quick start — paste into the chat:

Here is the four-step template for analyzing open-ended texts. Adapt it to my task: {your task}.
Ask me questions to fill in the fields.

[paste the template above]

The LLM will ask how much text you have, how granular the categories should be and what language you need the output in, because that determines how to split the texts into batches and what abstraction level to set for the themes.

⚠️ Abstraction hurts accuracy: at the level of specific categories the LLM agrees with humans fairly well, but when generalizing categories into themes, agreement drops: the model generalizes its own way, not the way a human does.

⚠️ False signal from noise: the model tends to find meaning even in unclear or effectively empty answers, where a human would write "unclear." Check short and odd answers by hand.

⚠️ There's no single right answer: the method was compared against humans, not against a "correct" gold standard, and the humans disagree with each other almost as much as with the model. That's not a model error, it's the nature of qualitative analysis.

⚠️ Results depend heavily on the question type: for emotionally charged or vague topics (e.g., "how to improve your financial situation") agreement is lower than for concrete questions (e.g., about mobility barriers).

The researchers took a real survey of almost 3000 doctoral students across Europe and selected 10% of the answers (903 of them) to six open questions: about access to education, finances, well-being, mobility and overall impressions. Five human coders hand-coded these answers using a standard qualitative analysis method, while GPT-5 was run in parallel through the same four-step prompting process.

Agreement was measured with the Adjusted Rand Index, essentially a metric that checks whether the same pairs of answers land in the same group for both the human and the model, regardless of what the group is called. The agreement came out moderate: 0.61 at the category level and 0.54 at the theme level. Meanwhile, the humans agreed with each other at 0.68, and the model with itself (between coding and themes) at 0.76.

A curious detail: wherever the coders themselves (both humans and the model) were internally inconsistent, human-model agreement dropped too. So the problem isn't always the model; it's often the ambiguous data itself. At the end, the researchers simply showed the humans the LLM's results and asked their opinion: all five coders said the result reflected their own logic and that they would trust the model with it.

Bergmann, L., Gheorghiu, R., Gvritishvili, A., Mican, A., Stewart, C., Tolonen-Weckström, T. — LLMs for Survey Text Analysis: A Performance Comparison Between Humans and GPT-5 on Inductive Content Analysis. The data and analysis code are open: osf.io/rgkeh. The method is based on the procedure from Bergmann et al. (2025).