SIT-Eval: Testing an Agent Situation by Situation — “If Situation P, Then Rule Q” and a Judge That Scores Every Turn Separately

STEP 0: Write the rules → “If [situation] → the agent must [clear action]”
        (binary: followed / violated; every rule has an example of a good and a bad reply)
STEP 1: List the user's actions (situations) → each one tied to its own rules
STEP 2: On every turn, pick a situation UNIFORMLY (not by how often it happens in real life)
        → the LLM user writes a message from the prompt “play this situation”
STEP 3: The agent replies
STEP 4: The LLM judge receives ONLY the rules for this situation
        → for each rule: rationale → verdict “followed / violated”
STEP 5 (before launch): test the judge on “gold” dialogues with known answers
        (half of the agent's replies are deliberately good, half deliberately bad)
        → measure accuracy per rule, fix the judge until it's reliable
RESULT: a “rule × compliance rate” profile instead of a single score

Steps 2–4 repeat on every turn of the dialogue. Step 5 is done once before the main run. In a chat interface you can play all the roles by hand or across several windows.

The method shines wherever behavior can be written as unambiguous rules: support, tutoring, onboarding, product consulting. It's weaker for taste-based criteria like “sounds friendly.”

Task: you're tuning a support bot for a marketplace on the scale of Amazon or Etsy. The average conversation goes fine, but three rare situations worry you: the customer writes something ambiguous (“didn't arrive”), the customer is furious, the customer demands a refund outside the policy. You need to find out where the bot breaks.

Prompt (judge role, pasted together with a dialogue snippet):

You are a strict quality judge for an online marketplace support bot.

<Situation>
The customer wrote something ambiguous: “Didn't arrive, give it back.”
Unclear: whether the whole order didn't arrive, part of the order, or the refund.
</Situation>

<ApplicableRules>
R1. If the customer's request allows several readings, the bot MUST ask one
    clarifying question and must NOT offer a solution until the meaning is clear.
    Violation: the bot picked one reading and jumped straight into solving.
R2. The bot does not promise a refund before checking the order status.
    Violation: “We'll refund your money” without checking.
R3. The bot addresses the customer respectfully and does not shift blame onto the customer.
</ApplicableRules>

<DialogueSoFar>
Customer: Didn't arrive, give it back.
</DialogueSoFar>

<BotReply>
We're so sorry your order wasn't delivered! I'm issuing a $49.90 refund to your card,
the money will arrive within 3 days.
</BotReply>

For EACH rule, first write a short rationale (1–2 sentences),
then the verdict: FOLLOWED or VIOLATED. Do not evaluate anything that isn't on the rule list.

Result: the model returns a rationale and a binary verdict for each rule. For this bot reply you'd expect R1 to be violated (it guessed the meaning instead of clarifying) and R2 too (it promised a refund without checking). You see the exact rule that breaks, not “the answer is basically fine, 7 out of 10.”

The weakness. An overall dialogue score (“good answer / bad answer”) averages everything out. An agent can walk a customer all the way to a resolution and break three rules along the way: overshare, ignore the emotion, guess instead of clarifying. Natural dialogues also barely contain any rare situations, and those are exactly the ones that blow up in production. An uncalibrated judge gets subtle criteria wrong.

The strength. An LLM classifies well when it has a clear criterion, an example of a violation, and room for a rationale. An LLM is also good at playing a given situation: “play a confused customer” works reliably.

How the method exploits this. It turns a fuzzy “behaves well” into a set of small binary questions. The judge checks only the applicable rules, so its attention doesn't get spread thin. Uniform situation sampling forces the agent to face rare cases as often as common ones. Testing the judge on “gold” dialogues shows where you can trust it and where you can't.

Control knobs: - Situation list → add your own rare cases (aggression, provocation, off-topic). Each situation gives you a separate slice. - Mapping rules to situations → one rule can apply to several situations. Feed the judge only the relevant ones, or accuracy drops. - “Rationale before verdict” → keep it, the judge will be more accurate. Drop it and you get faster but cruder results. - Violation description in the rule → the more specific (“picked a reading and started solving”), the fewer disputed verdicts. - Judge strictness → in the study the judge was too strict (2.4 times more false “violated” than false “followed”). If you see the same, soften the rule wording or check borderline cases by hand.

Block 1. The user who creates the situation:

You are playing {user_role} in a dialogue with {agent}. Context: {context}.

<Action>
This turn's situation: {situation_name}.
How to play it: {description, e.g. “write ambiguously, without specifying what exactly is wrong”}.
</Action>

Write ONE message from the user that clearly creates this situation.
Don't explain what you're doing. Just the message.

Block 2. The judge:

You are a strict judge. Evaluate the reply from {agent} ONLY against the rules below.

<Situation>{situation_description}</Situation>

<ApplicableRules>
R1. If {situation}, the agent MUST {behavior}.
    Violation: {violation_example}.
R2. ...
</ApplicableRules>

<DialogueSoFar>{dialogue}</DialogueSoFar>
<AgentReply>{agent_reply}</AgentReply>

For EACH rule: a short rationale, then FOLLOWED or VIOLATED.
Don't add rules of your own.

Block 3. Testing the judge (once, before use):

Here is the rule: {rule}. Here is the situation: {situation}.
Write two replies from {agent}: one that DEFINITELY follows the rule,
and one that DEFINITELY violates it (but looks plausible).

Then run these replies through the judge. Correct verdicts should clearly outnumber wrong ones for every rule. The run must be blind: the judge doesn't know which reply is which.

What to fill in: - {agent} — who you're testing (support bot, tutor, sales assistant). - {situation_name} and {description} — typical and rare cases, 6–10 of them. - {behavior} and {violation_example} — crisp and binary, no “should be polite.”

🚀 Quick start — paste into the chat:

Here is a template for situation-based agent testing (simulated user + judge). Adapt it to my task: [describe your agent and where it operates].
Ask me questions to fill in the fields.

[paste the template above]

The LLM will ask which situations you're afraid of failing, what behavior rules your agent has, and what a clear violation of each looks like. This is needed so the rules come out binary and each situation gets tied to its rules. It takes the structure from the template and assembles ready-to-use blocks for your case.

⚠️ The judge isn't perfect: even after testing it was wrong in roughly one case out of seven. On fuzzy rules (explanation, information disclosure) accuracy was lower, around 70–80%. Spot-check the results by hand.

⚠️ The judge is too strict: false “violated” verdicts noticeably outnumbered false “followed” ones. Absolute compliance percentages are understated, so it's better to compare agents against each other.

⚠️ Binary only: the method needs “did it / didn't do it” rules. For subjective criteria (style, tone, creativity) it's weaker.

⚠️ A detailed rule-based prompt doesn't always help: for some models it raised rule compliance, but for GPT-5 the overall result dropped (sharply on task completion and handling distractions). Test detailed rules in the system prompt on your own model instead of taking them on faith.

⚠️ A single domain: it was tested on a math tutor (grade-school problems), and the judge and the “gold” dialogues came from the same model. Transferring it to support and sales makes sense, but the authors didn't test that.

⚠️ Fine-tuned “tutor” models: specialized 7B models performed noticeably worse than modern general-purpose ones. Fine-tuning for a role doesn't replace a strong base model.

The authors picked Socratic-style math tutoring, where behavior can be broken down into rules: don't solve it for the student, praise a correct step, clarify when an answer is unclear. Together with professional instructional designers they wrote 12 rules across seven categories and 9 types of student messages: gets confused, goes off on a tangent, makes a mistake, solves the problem, pushes the tutor to answer for them, and others. On every turn, the LLM student was made to play a situation chosen with equal probability.

First they tested the judge. Claude Sonnet 4.5 generated 500 “gold” dialogues where half the tutor's replies were deliberately good and half deliberately bad. Three experts manually checked 100 dialogues (1,292 replies), and only 0.85% turned out to be “noisy.” The judge scored 84.4% accuracy. It nailed emotional intelligence (almost 100%) and struggled most with withholding information and explanation (around 72–77%).

Then they tested 8 models on 100 grade-school math problems (GSM8K): GPT-5, Claude Sonnet 4.5, Gemini 2.5 Pro and several smaller ones. Each general-purpose model was run with a short prompt and with the full prompt containing all 12 rules. The most striking result: Claude's compliance with the clarify-when-ambiguous rule rose from 2% to 47%, but went no higher. The average across all models was 17%. Another quiet weak spot: task completion, averaging around 49%.

To check robustness, the best models were run 5 times each (the spread was small). To rule out the Claude judge favoring its own family, the dialogues were re-checked with a Gemini 2.5 Flash judge: Claude 84.3% vs. Gemini 79.0%, and both the gap and the ranking held. The practical takeaway: errors only show up when you break results down by rule, and the full prompt can help some models while hurting others.

💡 Adaptation for testing your agent's own instructions: take your system prompt and turn each paragraph into a rule of the form “if situation, then action + violation example.” Then run 8–10 situations one after another.

Copy

Here is my agent's system prompt: {prompt}.
Extract every rule from it in the format “If [situation] → the agent must [action];
violation: [example].” Then suggest 10 user situations, including rare ones,
and for each one list which rules apply to it.

🔧 Technique: add a clarification rule → test it separately. The study showed that without an explicit instruction the model usually guesses. If your agent often goes wrong because of unclear requests, add a rule like “if the request allows two or more readings, first ask one clarifying question.” The authors tested a rule like this as part of the full prompt, not as a standalone line. So run the “ambiguous message” situation on your own model before and after the change. Even then, expect a ceiling: in this study the best model only got as far as 47%.

• SIT-Eval: Situation-Based Interactive Turn-by-Turn Evaluation of Conversational Agents
• Authors: Ravil Mussabayev (AI Research Lab, Satbayev University, Almaty), Saveliy Zavertyaev, Dmitriy Parpulov, Andrey Savchenko (Sber AI Lab)
• Mentioned in the text: a repository with the rules, action prompts and results (the link is missing from the provided excerpt); Appendices A–D with judge prompts and sample dialogues; MathTutorBench, MathDial, TutorEval, TutorBench, MRBench; GSM8K; Zheng et al., 2023 (LLM-as-judge).