LLMs in Qualitative Analysis: Structured Prompts + Self-Check = Methodological Rigor

Phase 1: Human reads the data (getting familiar with the material)

Phase 2: LLM creates codes → Human grades them against the rubric

PROMPT → "You are an expert researcher. RQ: [question]. 
Create codes: clear, specific, analytically useful.
Before output, grade yourself against: [Excellent rubric].
Format: JSON with fields: segment, codes, explanation, sensitive"

Phase 3: LLM groups codes into themes (for one interview) → Human reviews

Phases 4-5: LLM merges themes across all interviews + refines boundaries → Human grades against the rubric

Phase 6: Human writes the final report with interpretation

Task: You're running a series of customer interviews for a startup building a freelancer-matching service for US tech companies. You have 20 recorded conversations with CTOs and team leads. You need to find patterns: what keeps them from hiring remote contractors, what they fear, what they expect. Classic qualitative analysis would take a week — you'd have to code every interview, find recurring themes, and check you didn't miss anything important.

Prompt (simplified version for a quick start):

You are an expert in qualitative data analysis for product research.

RESEARCH QUESTION: 
What factors influence whether tech companies hire freelancers or not?

TASK:
Read the CTO interview below. Create codes — short labels for important ideas that help answer the question.

CODE QUALITY CRITERIA (grade yourself before output):
- Clarity: the code is understandable, no ambiguity
- Relevance: directly tied to the research question
- Specificity: narrow focus, no overlap with other codes
- Analytical value: helps reveal a pattern, not just a paraphrase

OUTPUT FORMAT:
JSON with fields:
- segment: quote from the interview (verbatim)
- code: short label (2-5 words)
- explanation: why this segment matters for answering the question
- line_number: line number in the interview (for verification)

SELF-CHECK:
Before output, ask yourself:
1. Does this code help answer the RQ?
2. Is it specific enough?
3. Is the explanation convincing?
If not — redo it.

INTERVIEW:
[paste transcript]

Result:

The model will return structured JSON with codes such as "Distrust of remote workers", "Fear of losing control", "Bad past freelancer hire", "1099 contractor paperwork hassle". Each code is tied to a specific interview quote with a line number — so you can verify the model isn't hallucinating.

In the explanation field, the model explains why it picked that segment: "The respondent twice stressed the risk of data leaks through freelancers, pointing to a trust barrier — a key reason for not hiring".

You get a first-pass pattern map in 10 minutes instead of a day of manual work. Then you review the codes, cut the extras, and group them into themes. The LLM did the grunt work — you take it to the finish line, adding the context the model may have missed.

The LLM's weakness: The model doesn't understand the research context or methodological standards. Just ask it to "find themes" and you'll get a shallow summary or hallucinations. Without quality criteria, the LLM doesn't know what a "good code" is in qualitative analysis — it may produce labels that are too broad ("Hiring problems") or too narrow ("Respondent mentioned Upwork").

The LLM's strength: The model is excellent at following explicit instructions and structured templates. Build methodological criteria right into the prompt (as the study did with Braun and Clarke's rubrics for qualitative analysis), and the LLM starts imitating a professional analyst. It spots patterns across large volumes of text, groups similar ideas, and gives clear labels — but only if it knows exactly what's expected of it.

The method taps that strength through three levers:

1. Role + RQ + Methodology — the model understands "who I am", "why I'm doing this", "how to do it right". Instead of an abstract "find themes", it gets a concrete frame: "You're a researcher, your job is to answer question X, use an inductive approach (from data to themes)".
2. Self-check before output — the study's key trick. The model grades its draft against the "Excellent" criteria BEFORE showing the result. It's similar to Chain-of-Thought, but instead of "think out loud" it's "check yourself against a checklist". In the experiment this gave a jump in quality: codes got more specific, explanations more convincing.
3. Verifiable references — the prompt requires a line/interview number for every code. That kills hallucinations: the model can't make up a quote if it has to say where it came from. A human can quickly check whether a code is real.

Prompt control levers:

• Quality criteria → drop the self-check to go faster (but quality will drop). Or add your own criteria: "the code should apply to other interviews too".
• Output format (JSON) → switch to a table or Markdown if you don't need automated processing.
• Methodological orientation → switch from inductive (data-driven) to deductive (hypothesis testing) if you already have a list of themes.
• Level of detail → say "create 3-5 codes per interview" to save tokens, or "be as detailed as possible" for deep analysis.

You are an expert in qualitative data analysis in {domain: product research / UX / marketing}.

RESEARCH QUESTION:
{question: e.g. "Why do users cancel their paid subscription?"}

TASK:
Read the {data type: interviews / reviews / open-ended answers} below. 
Create codes — short labels for important ideas that help answer the question.

METHODOLOGY:
- Approach: {inductive (from data to themes) / deductive (hypothesis testing)}
- Level of interpretation: {semantic (explicit meaning) / latent (hidden meaning)}

CODE QUALITY CRITERIA:
- Clarity: the code is understandable without extra explanation
- Relevance: directly tied to the research question
- Specificity: narrow focus, no overlap with other codes
- Analytical value: helps reveal a pattern, not just a paraphrase of the respondent's words

OUTPUT FORMAT:
JSON with fields:
- segment: quote from the data (verbatim)
- code: short label (2-5 words, verb or noun)
- explanation: why this segment matters for answering the RQ
- source_reference: {line number / respondent ID / date} for verification

SELF-CHECK before output:
1. Does every code help answer the RQ?
2. Are the codes specific enough (not "Problems" but "High price compared with competitors")?
3. Are the explanations convincing and not just repeating the quote?
4. Are all quotes real (checkable via source_reference)?

If any answer is "no" — redo it before output.

DATA:
{paste transcript / reviews / answers}

What to fill in:

• {domain} — the research area (product, UX, HR, marketing)
• {question} — the main research question you're trying to answer
• {data type} — interviews, marketplace reviews, survey answers
• {approach} — inductive (looking for new themes) or deductive (testing hypotheses)
• {level of interpretation} — semantic (what the person said outright) or latent (what's behind the words)
• {source_reference} — how to cite the source: line number, ID, date — whatever lets you verify the quote

🚀 Quick start — paste into a chat:

Here is a template for qualitative data analysis with a built-in quality self-check. 
Adapt it to my task: {describe your task — e.g. "Analyze 15 interviews with customers of a SaaS product who canceled their subscription"}.

Ask me questions so you fill in the template fields correctly.

[paste the template above]

The LLM will ask: what your research question is (to focus the codes), what data you have (interviews/reviews — this shapes the coding style), which approach (look for something new or test hypotheses). It will take the structure from the template — role, RQ, quality criteria, self-check — and adapt it to your task. You'll get a working prompt with methodological rigorwithout having to master the finer points of qualitative analysis.

⚠️ Latent (hidden) meaning: LLMs handle explicit statements well ("the respondent said X") but miss subtext, contradictions, and nonverbal cues. If someone says "everything's fine" sarcastically, the model won't catch it. For tasks where context and the emotional layer matter (interviews about personal experience, team conflicts), you need to review the codes by hand.

⚠️ Over-fragmentation: The model chops data into pieces that are too small. Instead of one theme, "Work flexibility", it creates three separate codes: "Remote work", "Flexible schedule", "Work-life balance". That complicates later analysis — you have to group and consolidate by hand.

⚠️ Fuzzy theme boundaries: When merging codes into themes (phases 4-5), the model sometimes creates overlapping categories. For example, a "Team support" theme and an "Organizational culture" theme may overlap. The experts recommend a final human review of the theme structure.

⚠️ Requires expert oversight: The method doesn't fully automate the analysis. Phases 1 (reading the data) and 6 (writing the report) stay human. Between phases you need rubric-based grading — otherwise the model can drift away from the RQ or produce convincing-sounding but wrong interpretations.

The researchers took 15 real interviews about software engineers' well-being (177 pages of transcripts) — a topic where both explicit factors (pay, schedule) and hidden ones (anxiety, burnout) matter. Two experienced analysts hand-coded these interviews using the Braun and Clarke method — that became the gold standard for comparison.

Then they created three prompts for phases 2–5 of thematic analysis (coding, grouping, merging themes) and ran them through 4 LLMs: ChatGPT o3-mini, GPT-4o, Claude Sonnet 4, Gemini 2.5 Pro. Each prompt went through iterations — for example, the coding prompt was rewritten 8 times because early versions produced hallucinations (made-up quotes) and codes that were too broad. The final version added a self-check — the model grades its output against the "Excellent" criteria from the rubric BEFORE showing the result. That gave a jump in quality.

To evaluate the results, four experts (professors and PhDs with qualitative research experience) ran a blind evaluation: they were shown pairs of codes (human vs LLM) without the author — and had to pick the better one. In 61% of cases the experts chose the LLM codes, citing greater clarity, specificity, and analytical depth. But the experts also noticed weaknesses: the model fragmented the data (created too many small codes instead of broad patterns) and missed latent meaning (didn't see contradictions or subtext).

Why this particular result? The LLM codes looked more professional: short, standardized, with clear explanations. The human codes were more context-sensitive, but often longer, less uniform, and sometimes redundant. In a head-to-head comparison, the LLM codes' visual "polish" won out. But that doesn't mean the LLM understands the data better — just that its presentation is more convincing.

Surprising insight: a self-check in the prompt works as a quality amplifier. When the model grades its draft against a checklist ("Is this code specific enough? Is it relevant to the RQ?"), it rewrites the output — and the final version comes out stronger. It's like editing improving a text, except the editor is the model itself, using criteria from the prompt.

Practical takeaway: LLMs can handle the routine parts of qualitative analysis (coding, grouping) if you structure the prompt right — but the final interpretation must be human. The model doesn't replace the analyst; it speeds up the grunt work 5–10x.

Context: The researchers created three prompts for the phases of thematic analysis. Below is a simplified structure of the Prompt 1 (Coding)that generated codes from the interviews. The full version runs 1485 tokens; here are the key elements.

You are a world-class software engineering researcher with deep expertise 
in qualitative data analysis and reflexive thematic analysis.

STUDY GOAL:
Identify the factors that influence the well-being of software engineers.

RESEARCH QUESTION:
What factors influence the well-being of software engineers in their work?

METHODOLOGICAL ORIENTATION:
- Inductive approach (data-driven, not testing predefined themes)
- Semantic level (focus on explicit meanings, what participants said directly)
- Experiential perspective (participants' lived experiences and perceptions)

CODING PRINCIPLES:
- Succinct: Short, focused labels (2-5 words)
- Specific: Narrow focus, distinct from other codes
- Gerunds preferred: "Experiencing burnout" not "Burnout"
- Informative: Label captures essence, not just repeats words

EVALUATION CRITERIA (assess your output before finalizing):
1. Clarity: Code meaning is unambiguous
2. Relevance: Directly addresses the research question
3. Semantic accuracy: Captures what participant meant
4. Specificity: Distinct, non-overlapping
5. Theme potential: Will support meaningful theme development
6. Alignment: Reflects dataset content accurately
7. Good labels: Concise, insightful shorthand

CONTEXT:
Interview number: {N}
Interviewer's question: {question}

OUTPUT FORMAT:
Return JSON with fields:
- segment: Verbatim quote from response
- codes: Array of code labels (2-5 words each)
- explanation: Why this segment matters for answering RQ (2-3 sentences)
- sensitive: Boolean (true if mentions mental health, conflict, discrimination)
- line_number: Exact line in transcript (for verification)

RESPONSE TO BE CODED:
{participant's response}

SELF-CHECK before output:
1. Each code helps answer "What influences well-being?"
2. Codes are specific enough to distinguish from others
3. Explanations justify selection convincingly
4. All segments traceable to line numbers (no hallucinations)

If any answer is "no" → revise before returning output.

This prompt builds in methodological rigor (a clear RQ, evaluation criteria) + a self-check (the model edits itself against a checklist) + hallucination protection (a required line number). The result: codes that experts preferred over the human ones 61% of the time.

💡 Adaptation for customer review analysis:

The original method was built for interviews, but it also works for reviews on marketplaces, social media, and websites. Replace "interview segment" with "review text", and the RQ with "Why are customers unhappy with delivery?". The self-check and structure stay the same.

You are an expert in customer experience analysis.

RESEARCH QUESTION:
Why do customers give low ratings to courier service X?

TASK:
Read the review below. Create codes for the drivers of dissatisfaction.

QUALITY CRITERIA:
- Clarity: the code is understandable
- Relevance: tied to the question
- Specificity: narrow focus

SELF-CHECK:
1. Does the code explain the reason for the low rating?
2. Is it specific enough (not "Bad service" but "Courier was 2 hours late with no heads-up")?

FORMAT:
JSON: {segment, code, explanation, review_id}

REVIEW:
{review text}

Effect: Instead of sorting through 500 reviews by hand over a week, you get structured codes in an hour. Then you group them into the top 5 problems and set priorities.

🔧 Technique: Drop the self-check → faster processing

If you have lots of data (1000+ reviews) and need speed over depth, remove the SELF-CHECK block from the prompt. The model will return results faster, but code quality will drop (more broad, less specific codes).

When to use it: First-pass exploration of large volumes. Then you hand-pick the interesting segments and run them through the full prompt with the self-check.

🔧 Technique: Add examples of good codes → few-shot learning

Paste 2-3 examples of great codes from your own data into the prompt:

EXAMPLES OF GOOD CODES:
Segment: "The account manager didn't answer 5 emails in a week"
Code: "Ignoring customer requests"
Explanation: Shows a systemic communication problem that hurts loyalty

Segment: "The price went up 30% in a month with no explanation"
Code: "Opaque pricing"
Explanation: The customer loses trust because of unexpected changes in terms

Effect: The model copies the style and depth of your examples. The codes get closer to how you'd write them yourself.

💡 Extrapolation: Multi-stage analysis for complex decisions

Use the study's logic (Phase 2 → Phase 3 → Phases 4-5) for decision-making. For example, choosing a business idea:

Phase 1 (human): Gather 10-15 ideas with descriptions (from brainstorming, trends, personal experience).

Phase 2 (LLM): Create codes for each idea — factors FOR and AGAINST.

You are a startup evaluation expert.

RQ: What factors determine the viability of this business idea?

Create codes for the idea below: strengths, risks, barriers to entry.
Format: JSON {aspect, code, explanation}

IDEA: {idea description}

Phase 3 (LLM): Group the codes into themes for each idea (e.g. "Market potential", "Operational risks").

Phases 4-5 (LLM): Compare themes across all 10 ideas. Rank the ideas by theme strength and frequency of supporting data.

You have themes for 10 business ideas. 
Merge and rank: which idea has the strongest themes with the lowest risks?

Phase 6 (human): Read the top 3 ideas with the LLM's rationale. Pick the final one using your intuition and the context the model can't see (your resources, personal motives).

Effect: Instead of "this feels like a good idea", you get a structured, data-backed comparison. The LLM does the analytical work; you make the final call.

Large Language Models in Thematic Analysis: Prompt Engineering, Evaluation, and Guidelines for Qualitative Software Engineering Research
Cristina Martinez Montes, Robert Feldt, Cristina Miguel Martos, Sofia Ouhbi, Shweta Premanandan, Daniel Graziotin
Chalmers University of Technology, University of Gothenburg, Universidad Ramon Llull, Uppsala University, University of Hohenheim

The study builds on the Braun and Clarke's Reflexive Thematic Analysis methodology — the gold standard of qualitative analysis in the social sciences and software engineering research.