AHP prompting: how to make an LLM explain decisions through explicit criteria and weights instead of guessing

STEP 1: Define 3-7 decision criteria → a list of criteria with definitions
STEP 2: Compare the criteria with each other pairwise → a comparison matrix + numeric weights
STEP 3: Score each option on each criterion → a weighted score + the final choice

All three steps can be done in one conversation as sequential messages (this is more reliable) or squeezed into one prompt (worse on weaker models, but better on strong ones like GPT-4o-mini).

Task: You need to pick one of three candidates for head of sales, based on résumés and interview results.

Prompt (step 1):

I have three candidates for head of sales:
Candidate A: 8 years of experience, but none in our niche
Candidate B: 3 years of experience, but specifically in our niche, strong references
Candidate C: 5 years of experience, managed a team of 20, average revenue results

Identify 4-5 criteria worth using to evaluate candidates for this role. 
Give a short definition of each criterion.

Prompt (step 2, after the model's answer):

Now compare these criteria with each other pairwise: 
for each pair, say which criterion is more important and by how much (on a scale of 1 to 9).
Based on these comparisons, calculate each criterion's weight as a percentage.

Prompt (step 3):

Score each candidate on each criterion (scale 1-10).
Calculate each candidate's final weighted score using the criteria weights.
Name the best candidate and explain the decision through the weights and scores.

Result: The model produces three separate, connected answers. The first is a list of criteria with definitions. The second is a comparison matrix and weight percentages (e.g., "niche experience" 35%, "management skills" 25%, and so on). The third is a table of candidate scores and the final pick, with an explicit explanation of why that candidate scored highest.

When you ask an LLM to "just pick the best option," the model generates an answer that sounds convincing but isn't required to be logically traceable: it isn't computing weights in its head, it's predicting the next plausible token. There's no way to check why the choice is what it is.

But LLMs handle small, concrete steps well: list the criteria, compare two things with each other, calculate a simple proportion. Each of these steps is a task the model solves reliably.

The method takes the complex task "make a decision across several criteria" and chops it into such small steps. Each step is grounded in the previous one (the conversation remembers what was said earlier), so the final decision can be traced backward: you can see which weights and scores it was built from.

Control levers: - Number of criteria (3-7 in the original) → 3 is enough for simple everyday decisions; up to 7 for complex business decisions - Comparison scale (in AHP, the Saaty scale from 1 to 9) → you can simplify it to "1-5" or to words ("slightly more important," "far more important"); accuracy drops only slightly, but the prompt gets simpler - One prompt vs. a step-by-step dialogue → for powerful models (GPT-4o-mini, Claude, GPT-4) you can risk giving the whole instruction at once; for simpler models, always split it into separate messages

Help me make a decision using the AHP method (Analytic Hierarchy Process).

Situation: {description of the decision and options}
Options: {option 1}, {option 2}, {option 3}

Step 1: Identify {number} criteria that matter for this decision. Give a short definition of each.

Don't move on to the next step until I confirm the list of criteria.

After the criteria are confirmed, the second prompt:

Now compare these criteria pairwise: which of the two is more important and by how much (scale 1-9, where 1 means equally important and 9 means the criterion is incomparably more important). 
Build a comparison table and calculate each criterion's final weight as a percentage.

After the weights, the third prompt:

Score each option on each criterion (scale 1-10).
Calculate each option's weighted total score (score × criterion weight, summed over all criteria).
Name the best option and explain the decision through the resulting scores and weights.

What to fill in: {description of the decision} — the gist of the question you need to settle; {option 1, 2, 3} — the specific alternatives; {number} — how many criteria you want (3 for simple tasks, 5-7 for complex ones).

🚀 Quick start — paste into the chat:

Here is the template for making a decision with the AHP method. Adapt it to my task: {your task}.
Ask me questions to fill in the fields.

[paste the template above]

The LLM will ask what your options are and how many criteria you want to use, because without that it can't build the comparison matrix. Then it will walk you through all three steps on its own.

⚠️ The model can lose to a simple baseline. On weaker models (Mistral, Llama3), a complex multi-step AHP instruction sometimes produces worse results than just "pick the best option" with no structure at all. Before trusting the method, test it on a couple of sample decisions to make sure the structure didn't break the logic.

⚠️ A single prompt with the full instruction is unstable. If you try to cram the whole AHP process into one message (without splitting it into steps), the model may forget part of the instructions or mix up the order of actions, especially on less powerful models.

⚠️ The model's criteria don't always match human ones. In the study, the model forced to follow the AHP instruction strictly in one go matched the criteria chosen by human experts less often than the model working in a free step-by-step dialogue. It's worth asking the model to explain why each criterion was chosen and checking the logic yourself.

The researchers created a new dataset, LEGAL-AHP: they took legal questions from the well-known LegalBench benchmark and had law students manually annotate them following AHP rules: identify criteria, compare them pairwise, calculate weights. The result was 60 reliable question-answer pairs with verified consistency (a check that a person didn't compare criteria contradictorily: if "A is more important than B" and "B is more important than C," then "A" can't be less important than "C").

They then compared four models (Llama3, Qwen2.5, Mistral, GPT-4o-mini) across four strategies: no AHP, a single prompt with the full AHP instruction, a step-by-step dialogue, and a multi-agent system with separate roles for each stage. It turned out that the strong model (GPT-4o-mini) consistently benefits from structuring: accuracy rose from 73% to 82% in the best configuration. The weak models, on the contrary, often lost quality, with accuracy dropping to 37-43%.

The surprise: the criteria the model generated in the step-by-step dialogue matched human ones more often than criteria from the single AHP prompt, even though final decision accuracy was higher with the single prompt on strong models. In other words, the model can give the right final answer for the "wrong" (non-human) reasons. The method was also tested outside law, on university rankings (QS, US News), and the approach carried over, although the gain depended on the specific model and dataset.