TL;DR
By default an LLM grades everything the same way — through a standard set of criteria ("helpfulness", "coherence", "safety"). It doesn't matter what's in front of it: an investor pitch deck or a code debugging session. AdaRubric breaks this pattern with a two-step scheme: first, ask the model to come up with evaluation criteria specifically for the task, then grade against them.
The problem is that the same criteria don't catch different kinds of failure. You can't grade a pitch deck on "grammar and structure" — what matters is unit economics, the competitive landscape, the viability of the business model. If you ask an LLM to "evaluate this", it applies a one-size-fits-all yardstick and gives one-size-fits-all feedback. Worse, if three criteria look good and one critical one is bad, the overall score hides the failure.
AdaRubric solves this in two steps: generate a rubric (5 task-specific dimensions with weights and 5-level criteria), then grade each dimension separately — so high scores on some can't mask a failure on others.
Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.
Method Blueprint
Two separate requests in the chat:
STEP 1 (separate prompt): task description →
the LLM generates 5 quality dimensions
specific to THIS particular task,
with weights and 5-level criteria (JSON)
STEP 2 (separate prompt): material + rubric from step 1 →
the LLM grades each dimension in isolation,
records failures separately even when the overall score is high,
outputs a weighted total + a list of critical weaknesses
Example in Practice
Task: Jake is a founder building a ghost kitchen (a delivery-only kitchen) in Chicago. He's written a 15-page business plan. He wants an honest assessment before meeting an early-stage investor from a startup accelerator — what will fail, what works, what to watch out for.
Prompt — Step 1 (rubric generation):
Task: evaluate a business plan for opening a ghost kitchen (delivery-only kitchen)
in Chicago. Target audience: an early-stage investor (accelerator / angel investor).
The business runs through DoorDash and Uber Eats, focuses on office districts,
average order $15–25.
Generate an evaluation rubric for this task:
- 5 key quality dimensions specific to a ghost kitchen
at the pre-seed stage (don't write generic "structure" and "grammar")
- For each dimension: a weight (summing to 1.0) +
descriptions of levels 1, 3 and 5 with concrete examples
Format — JSON:
{
"dimensions": [
{
"name": "...",
"weight": 0.X,
"criteria": {
"1": "...",
"3": "...",
"5": "..."
}
}
]
}
Prompt — Step 2 (grading):
Use the rubric above to evaluate the business plan:
[paste the text or key sections of the business plan]
Grading rules:
1. Grade each dimension IN ISOLATION from the others
2. If any dimension scores ≤ 2, that's a critical failure:
put it in a separate block at the top of the answer, don't bury it in the total
3. Overall score = weighted sum.
But show critical failures separately as RED FLAGS —
even if the overall score looks acceptable
Answer format:
- 🚨 Critical failures (if any)
- Score for each dimension: points + one sentence on why
- Overall score and the main takeaway
Result:
Step 1 will produce a JSON rubric with 5 dimensions — for example: Unit economics, Operational scalability, Competitive positioning, Delivery-platform dependency risk, Path to profitability. The criteria will be specific: not "a good financial model" but "CAC/LTV calculated with the platform's 30%+ commission factored in, plus sensitivity to changes in the average order".
Step 2 will show the score for each dimension separately. If unit economics fail (score 1–2) while everything else looks fine, it pops up as a red flag at the top instead of drowning in an average of 3.5.
Why It Works
The LLM's weakness: When you ask it to evaluate something without clear criteria, the model applies a pattern from training — the set of most common evaluation criteria for similar texts. A business plan gets the same criteria as a short essay: "structure, arguments, logic". The criteria aren't bad — they're just the wrong ones.
The LLM's strength: The model is great at generating task-specific criteria if you ask explicitly. It knows what a good ghost-kitchen business plan should look like — unit economics, platform dependency, operational bottlenecks. The problem was never the knowledge; it's that nobody asked it to pick the right yardstick first.
How the method uses this: The first step "pulls" the right yardstick out into the open by generating the rubric. The second step applies it to each dimension in isolation. Isolation is the key: a high marketing score physically can't "lift" failing unit economics when you grade them separately.
Control levers:
| Parameter | Effect |
|---|---|
| Number of dimensions (5 by default) | Fewer (3) — faster, for simple tasks; More (7+) — duplication, confusion |
| Dimension weights | You can set them manually in step 1: "unit economics weight at least 0.35" |
| Critical failure threshold (≤2) | Tighten it to ≤3 for high-risk tasks (medicine, law) |
| Context in step 1 | The more precisely the task is described, the more precise the dimensions. Specify the audience, the goal, the constraints |
Prompt Template
Step 1 — rubric generation:
Task: {task_description_and_context}
Generate an evaluation rubric for this task:
- {number_of_dimensions} key quality dimensions
specific to this particular task and its goal
- Do NOT use generic criteria ("structure", "grammar", "completeness") —
only what matters specifically here
- For each dimension: a weight (weights sum to 1.0) +
descriptions of levels 1 (failure), 3 (acceptable), 5 (excellent)
with concrete examples
Format — JSON:
{
"dimensions": [
{
"name": "...",
"weight": 0.X,
"criteria": {"1": "...", "3": "...", "5": "..."}
}
]
}
Step 2 — grading:
Use the rubric above. Evaluate:
{material_to_evaluate}
Rules:
1. Grade each dimension in isolation
2. A dimension scoring ≤ {failure_threshold} = critical failure:
move it to the top as a separate block
3. Total = weighted sum.
RED FLAGS — separately, even if the average score looks fine.
Format: critical failures → score per dimension → total
What to plug in:
- {task_description_and_context} — what you're evaluating, for whom, in what context, with what goal
- {number_of_dimensions} — 5 by default, 6–7 for complex tasks
- {material_to_evaluate} — a text, plan, code, strategy
- {failure_threshold} — usually 2, 3 for critical tasks
🚀 Quick start — paste into the chat:
Here is the AdaRubric template — a two-step evaluation with an adaptive rubric.
Adapt it to my task: {your task}.
Ask me questions to fill in the fields.
[paste the template above]
The LLM will ask what exactly needs evaluating, for which audience, and what the goal is — because without that context it can't generate specific dimensions (instead of "unit economics" you'd get "financial literacy"). It will take the pattern from the template and pick criteria for your specific task.
Limitations
⚠️ Simple tasks: Don't use it to check a fact or answer a simple question — two steps add needless overhead where a direct request is enough.
⚠️ Edge cases: The model sometimes misses niche industry criteria that only an expert would know (like "CAPTCHA handling" for web agents or "AML/KYC compliance under the Bank Secrecy Act" for fintech). For high-stakes tasks, explicitly add in step 1: "also include these criteria: [list]".
⚠️ Quality depends on the task description: The fuzzier the context in step 1, the more generic the dimensions. Garbage in, garbage out.
⚠️ Long materials: When evaluating large texts (30+ pages), the model may not keep all the dimensions in context. Better to grade section by section.
Resources
AdaRubric: Task-Adaptive Rubrics for LLM Agent Evaluation — Liang Ding, Alibaba Group
Code: github.com/alphadl/AdaRubrics
Benchmarks: WebArena, ToolBench, AgentBench, SWE-bench Lite