Mixture-of-Prompts (MoP): smart check routing for deep analysis

STAGE 1: Feature Analysis (Routing)
→ Analyze the input data
→ Identify the possible problem types
→ Pick the relevant checks

STAGE 2: Sequential Check (for each selected check)
→ Step 2.1: Analysis Prompts — infer the state
   (values, types, relationships, paths in the data)
→ Step 2.2: Determination Prompt — verify the propositions
   P1: [proposition 1] → YES/NO
   P2: [proposition 2] → YES/NO
   ...
→ Step 2.3: Combine the results logically
   There's a problem if: (P1 AND NOT P2) OR P3

STAGE 3: Aggregation
→ Collect all problems found
→ Rank them by severity
→ Output the final result

Important: Routing and the Sequential Check can run in a single chat as a few consecutive messages.

Task: Check a services agreement for hidden financial risks before signing.

Prompt:

I'm reviewing a legal services agreement. Check it for financial risks in two stages.

[AGREEMENT TEXT]

STAGE 1 — Identify possible risks:
Which of these risk types are PRESENT in the agreement?
1. Uncapped price escalation
2. Hidden additional costs
3. Early termination penalties
4. Financial liability for the provider's actions

For each: POSSIBLE (mentioned) or NOT POSSIBLE (not mentioned)

STAGE 2 — Detailed check of POSSIBLE risks:

For type #1 (if POSSIBLE):
P1: The agreement specifies a price escalation mechanism (YES/NO)
P2: The provider can raise the price by more than 15% a year (YES/NO)
P3: There is a cap on escalation (YES/NO)

HIGH RISK if: P1 AND P2 AND NOT P3

[same for the other risk types]

Final conclusion:
- HIGH risks found, with quotes
- Overall risk level: Critical/High/Medium/Low

Result:

The model will first filter the risk types (for example, it decides types #1 and #3 are present and #2 and #4 are not), then check in detail only the relevant ones via propositions. In the end you get a list of specific risks with quotes from the agreement and an overall rating. Each proposition is checked separately (which cuts down hallucinations), then combined according to the logic.

LLMs handle simple questions well, like "does the text mention X?" or "is statement P true?". But they handle complex multi-factor checks poorly, like "find every possible problem of types A, B, C, D, E". The more factors the model has to juggle at once, the higher the odds it misses something — or "sees" a problem that isn't there.

Routing fixes false positives: instead of running all 30-40 problem types back to back (which inevitably leads to hallucinations), we filter first — activating only the checks that make sense for the data at all. If the agreement says nothing about intellectual property, why check IP-related risks? It's like a doctor who takes your symptoms first and then orders tests — not every test in the book, only the relevant ones.

Multiple Propositions fix false negatives: we decompose a complex check into a series of simple questions. The LLM answers each one separately (where it's strong), and then we combine the answers ourselves using logic. This moves the complex logic outside the model — the model only infers facts, and we make the decision with an explicit formula. For example: "there's a problem if P1 AND NOT P2" — we control that, not the LLM.

Control knobs:

• Number of propositions — fewer for simple checks (saves tokens), more for critical ones (lowers the risk of misses)
• Combination logic — AND/OR/NOT between propositions, tailored to your use case
• Routing depth — you can do two-level filtering: categories first, then subcategories
• Answer format — add "confidence" or a "quote from the source" to each proposition

Check {object_of_analysis} for {problem_type} using a two-stage approach.

[DATA]

STAGE 1 — ROUTING (filtering possible problems):

Which of these problem categories are PRESENT in the data?
1. {category_1}
2. {category_2}
3. {category_3}
...

For each: POSSIBLE (signs present) / NOT POSSIBLE (no signs)

STAGE 2 — DETAILED CHECK (only for POSSIBLE categories):

For category #{N}:
P1: {proposition_1} — YES/NO
P2: {proposition_2} — YES/NO
P3: {proposition_3} — YES/NO

Criterion: {combination_formula}
For example: PROBLEM if (P1 AND NOT P2) OR P3

[repeat for each POSSIBLE category]

STAGE 3 — SUMMARY:
- Problems found, with evidence
- Overall risk rating

Placeholders:

• {object_of_analysis} — what you're checking (an agreement, a business plan, code, an article)
• {problem_type} — what you're looking for (risks, errors, contradictions, vulnerabilities)
• {category_N} — problem types to filter on
• {proposition_N} — simple verifiable facts (yes/no)
• {combination_formula} — AND/OR/NOT logic between propositions

Paste into the chat:

Here's a Mixture-of-Prompts template for in-depth checks.
Adapt it to my task: [describe the task].

Ask me questions to fill in:
- Which problem categories to filter in STAGE 1
- Which propositions to check in STAGE 2
- What logic to use for combining them

[paste the template above]

The LLM will ask which problem types matter (for routing), which specific signals to look for (for the propositions), and how to combine them (for the final decision). It takes the structure from the template and fills it in for your task — you get a working prompt without picking apart the mechanics by hand.

⚠️ Requires structured thinking: To write the right propositions and the logic for combining them, you need a solid grasp of the subject area. Unfamiliar tasks may take a few iterations.

⚠️ Not for simple tasks: If the check fits in one question ("does the text mention X?"), the overhead of routing + multiple propositions isn't worth it. The method is for complex multi-factor checks.

⚠️ More tokens: Each proposition is a separate question to the LLM. With lots of propositions (10+), the request cost goes up. For critical checks that's justified; for high-volume ones you'll need to optimize.

⚠️ Depends on filtering quality: If the routing stage misses a relevant problem category, it won't get checked in STAGE 2. Don't be too strict in the first-stage filter.

The researchers used an internal company dataset of real code review cases — 345 programs with confirmed security vulnerabilities and 337 "clean" programs with no issues. It covered C/C++, Java, Python and Shell — 38 vulnerability categories in total from the CWE standard (Common Weakness Enumeration).

For each vulnerability category they hand-designed an expert prompt — a chain of 1-3 analysis prompts (which infer the program state) and a determination prompt (the final decision via propositions). For Memory Leak, for example: first analyze call relationships and the functionality of external APIs, then build data flow paths from memory allocation to the end of the function, and only then check "is there at least one path where the memory isn't freed".

They compared against 8 baseline approaches: static analysis (CppCheck, FlawFinder), fine-tuned models (CodeReviewer, T5-Review), and prompting (instruction prompts, prompts with CWE information). They tested on two LLMs: Qwen-2.5 (a regular model) and DeepSeek R1 (a "deep thinking" model).

Why the results came out this way: Static analysis showed high precision (94%) but terrible recall (7.69% for CppCheck) — these tools only find what's explicitly written into their rules and miss everything else. Fine-tuned models flopped completely (recall 0.28-2.85%) — because in open repositories only 1% of code review comments are about security, so there isn't enough training data. Plain instruction prompting got F1=45-48%, but with a pile of false positives — the model "found" SQL injection where there wasn't even a database call.

Mixture-of-Prompts reached F1=64% — the best result, because routing filtered out irrelevant checks (fewer hallucinations), and multiple propositions broke complex logic into simple questions (fewer misses). Also important: in a real production pipeline the acceptance rate rose from 39% to 84% — developers started accepting 2x+ more comments, because there were fewer false alarms.

Surprisingly, DeepSeek R1 with "deep thinking" was not much better than plain Qwen-2.5 on this task — F1 differs by 2-3%. That suggests prompt architecture matters more than the model. Another interesting bit: when Multiple Propositions were removed from iCodeReviewer, F1 dropped from 64% to 60% — so decomposing into propositions delivers a concrete, measurable gain.

💡 Adaptation for evaluating business ideas

Instead of checking code for vulnerabilities — checking a business idea for viability. Routing determines which aspects are relevant at all (B2B/B2C, software/hardware, marketplace/SaaS), then for each aspect we check propositions.

STAGE 1 — Identify the business type:
- B2B or B2C?
- Product or service?
- Recurring revenue or one-off sales?

STAGE 2 — Check for B2B SaaS (if that's the type identified):
P1: There's a clear customer persona with a pain point worth >$10k a year
P2: The solution takes >2 weeks to integrate (high switching cost)
P3: Competitors are already on the market with a similar solution
P4: Unit economics: LTV/CAC > 3

HIGH POTENTIAL if: P1 AND P2 AND P4 AND NOT P3

💡 Adaptation for reviewing marketing copy

Route by risk type (legal, reputational, ethical), then check specific triggers.

STAGE 1 — Possible risks in the copy:
1. Legal (misleading claims, guaranteed results)
2. Reputational (insulting competitors, controversial metaphors)
3. Ethical (fear-based manipulation, pressure to buy)

STAGE 2 — For legal risks (if any):
P1: There are claims like "100% results" without a disclaimer
P2: Comparisons with competitors without evidence
P3: Money-back guarantees without conditions

RISK if: P1 OR P2 OR P3

🔧 Technique: Confidence threshold for expert activation

In the original, routing is black and white (POSSIBLE/NOT POSSIBLE). Let's add some shades:

STAGE 1 — Estimate the likelihood of each problem type:
1. {category_1} — High (>70%) / Medium (30-70%) / Low (<30%)
2. {category_2} — High / Medium / Low
...

STAGE 2 — Check in detail only HIGH and MEDIUM likelihood.

Effect: Saves tokens — unlikely categories don't get checked. But the risk of missing a problem goes up.

🔧 Technique: Weighted Propositions to reflect importance

Not all propositions are equally important. Let's add weights:

P1: {critical_proposition} — weight 3
P2: {important_proposition} — weight 2
P3: {nice_to_have_proposition} — weight 1

Score = (P1×3 + P2×2 + P3×1) / 6

PROBLEM if Score > 0.5

Effect: A flexible score instead of rigid AND/OR. You can tune it to your own risk tolerance.

iCodeReviewer: Improving Secure Code Review with Mixture of Prompts — Yun Peng (The Chinese University of Hong Kong), Kisub Kim (DGIST), Linghan Meng, Kui Liu (Huawei Technologies)

The paper describes an application to code review, but the Mixture-of-Prompts architecture works for any deep-analysis task.