Apply to my task
arXiv:2507.22358 ★ 88 Jul 30, 2025

Magentic-UI: Become the Agent's Manager Instead of Waiting for a Black-Box Miracle

Key result

The problem: an LLM agent takes 15 steps on a complex task and messes up on step 7, and you only find out once it has "finished" everything. All that work, wasted. Magentic-UI lets you insert yourself into the process at every meaningful step: first you ask for a plan and edit it, then the agent moves forward but stops at the right points and waits for your approval. The shift: you don't write one giant prompt hoping for a miracle; you take on the role of project manager, and the LLM becomes a contractor who reports back at key stages. The interaction stops being a lottery.

Research 2507.22358

2 Why This Example Works.

This example applies the study's methodology effectively to tasks that demand accuracy and fact verification:

  1. Co-planning for research: Asking for a "research plan" forces the LLM to structure its approach. It lets the user confirm the LLM understood the task and will look for information in the right places, instead of just stitching together the first facts it finds online.
  2. Lower hallucination risk (Reliability): The "checkpoint" that requires sources (cite sources for every point) is a practical implementation of the "Answer Verification" and "Action Guards" principles. The user doesn't just trust the LLM; they get a chance to check facts at an intermediate stage. That is critical for topics where accuracy matters and helps fight made-up facts (hallucinations).
  3. Building trust step by step: By breaking a complex task into manageable, verifiable steps, the user builds trust in the process. Instead of getting a big "black box" of text at the end and guessing what's true and what isn't, the user takes an active part in creating it and controls quality at every stage.

Key Aspects of the Research:

The study shows that fully autonomous LLM agents often fail at complex tasks. Instead of full autonomy, it proposes a human-in-the-loop model where the user and the agent work as a team. To do this, the authors built Magentic-UI, a system that implements co-planning, co-execution of tasks and human oversight.

Key Result: Involving a human in the process through these mechanisms significantly raises the task success rate compared to fully autonomous systems.


The Method in a Nutshell:

The core idea is to stop treating the LLM as a "black box" that should execute a complex instruction perfectly on the first try. Instead, you become the LLM's manager or partner.

For an everyday user, the approach based on the study's findings looks like this:

  1. Co-planning: Don't hand the LLM the whole complex task at once. First ask it to draft a step-by-step action plan.
    • Example command: "Before you start writing, draft a detailed plan for how you'll do it. Show me the plan for approval."
  2. Edit the plan: Review the proposed plan. If it's inaccurate, incomplete or heading the wrong way, fix it right there in the chat. You know the context and the end goal better.
    • Example command: "The plan is good, but in step 2 search the Microsoft website instead of Amazon. And drop step 4, we don't need it."
  3. Controlled execution (Co-tasking & Action Guards): Tell it to execute the approved plan. For the most important steps, build "checkpoints" into the instruction: moments where the LLM must stop and ask for your permission or choice.
    • Example command: "Now execute the plan. When you get to step 3 (choosing a hotel), give me 3 options with prices and wait for my pick before continuing."
  4. Save what works (Memory): If a conversation and prompt structure worked well for a certain type of task, save that template for future use.

This approach turns a one-shot prompt into a guided conversation where you steer the LLM, fix its mistakes on the fly and make sure the result matches your expectations.


Practical Applicability:

  • Direct applicability: Very high. Any user can start applying co-planning in their chats with ChatGPT, Claude and other LLMs right away. Instead of one giant prompt, break the interaction into stages:

    1. "Draft a plan for [my task]"
    2. "Plan approved. Start on step 1"
    3. "Great. Before you do step 2, show me the options..." This carries the Magentic-UI logic straight into a regular chat.
  • Conceptual value: Huge. The study gives users a new "mental model": an LLM is a powerful but inexperienced intern. You can't leave it unsupervised with a complex task. You need to guide it, give clear instructions, check intermediate results and correct course. That understanding dramatically improves how effectively you work with it.

  • Adaptation potential: Maximum. The adaptation is simple: the user mentally takes on the role of the "Orchestrator" from Magentic-UI. They decide when to ask for a plan, when to approve it and when to step in. The whole complex dialogue that Magentic-UI automates, the user reproduces by hand in a regular chat, steering the LLM through a series of short, clear commands.


1 Practical Example:

You are an experienced social media manager. Your task is to create a one-week content plan for the Instagram account of a coffee shop called "Cozy Corner".

**STAGE 1: CO-PLANNING**

Your first and **only** task at this step is to propose a step-by-step plan for **HOW** you will build the content plan. The plan should cover target audience analysis, defining content pillars, and generating ideas for posts and stories.

**IMPORTANT:** Do not generate the content plan itself! Just show me your plan of action for approval. Wait for my further instructions.

(After the LLM proposes a plan, the user adjusts it and gives the next command)

Great, your plan of action is approved.

**STAGE 2: EXECUTING THE PLAN WITH A CHECKPOINT**

Now start executing the plan. When you reach the "generate post ideas" step, stop.

Give me 5-7 specific post ideas as a table with the columns: "Idea", "Content Pillar", "Format (post/reels)".

**IMPORTANT:** After you show the table of ideas, **wait for my choice**. Do not continue until I tell you which ideas I like.

2 Why It Works:

This prompt works because it simulates the key principles of the Magentic-UI study:

  1. Decomposition and co-planning: Instead of asking for a finished content plan right away (which may not match the user's vision), we make the LLM first lay out the process of creating it. That surfaces and fixes logic errors at the earliest stage.
  2. Human oversight (Action Guard): The command IMPORTANT: ... Wait for my further instructions creates an artificial "checkpoint". The LLM can't "run ahead" and churn out content based on ideas the user might not like. That saves time and makes the final result more relevant.
  3. Guided dialogue: The prompt turns a monologue (one request) into a guided dialogue where the user acts as project lead and the LLM as the contractor who reports back at every key stage.

1 Another Practical Example

You are my personal research assistant. I need to prepare a short summary of how microplastics affect human health, written for non-specialists.

**STEP 1: RESEARCH PLAN**

First, develop and present your research plan. The plan should include:
- Key questions that need answers.
- Types of sources you plan to use (scientific papers, WHO reports, popular publications).
- The structure of the final document.

**Critically important:** Do not start searching for information or writing text. Just show me the plan for sign-off.

(The user gets the plan, approves it and gives the next command)

Plan signed off. It's excellent.

**STEP 2: GATHERING INFORMATION WITH VERIFICATION**

Now start executing the plan. Gather information for each item.

**Checkpoint:** When you find information on the item "Main ways microplastics enter the body", stop. Present that information as short bullet points and **cite sources** for every point.

I'll check them, and only after my approval will you move on to the next item in the plan.

Usefulness Score: 92

02

Main Evaluation Criteria

  • A. Relevance to prompting techniques: Yes, but at a higher, conceptual level. The study offers not specific "magic phrases" but whole interaction strategies (co-planning, co-tasking) that users can implement through dialogue.
  • B. Better conversational answers: Yes, that's the main goal of the study. Human-in-the-loop methods directly aim to raise success rates on complex, multi-step tasks, which leads to a higher-quality, more relevant final result.
  • C. Direct practical applicability: High, but it takes some thought from the user. Users can't install Magentic-UI themselves, but they can simulate its principles in a regular chat with GPT-4, Claude and other advanced models by asking for a plan before execution, requiring confirmation and so on. No code or special tools needed.
  • D. Conceptual value: Very high. The study fundamentally changes the "mental model" of working with LLMs: from simple "request-response" to collaboration and managing an agent. It explains why LLMs fail at complex tasks and how a human can fix that by acting as "pilot" or "manager".
  • E. New useful practice: The work falls into several key clusters at once:
    • 1. Wording techniques: Offers the meta-technique of co-planning, an advanced form of task decomposition.
    • 3. Prompt structure optimization: The idea of an editable plan is essentially advanced structuring of a complex request.
    • 6. Context and memory: The concept of "memory" as saving successful plans for reuse belongs squarely in this cluster.
    • 7. Reliability and stability: Action guards and user verification of answers are direct ways to raise reliability and reduce risk.

Practicality checklist (+15 points): * Gives ready-made phrases/constructions for prompts? Yes, conceptually. "First draft a plan and show it to me for approval". * Explains where to place important information in a prompt? Yes, in the co-planning phase. * Shows how to structure complex requests? Yes, by building a step-by-step plan. * Reveals non-obvious LLM behavior? Yes, their inability to plan long-term and their tendency to "get stuck" without human help. * Offers ways to improve answer consistency/accuracy? Yes, through plan approval and intermediate checks.

03

2 Numerical Usefulness Score

Arguments for a high score (92/100): The study offers not just another trick but a fundamental shift in how to work with LLMs on complex tasks. Co-planning and co-tasking are extremely valuable for anyone who has watched an LLM "lose the plot" on multi-step instructions. The principles are universal and can be adapted right away in any advanced chatbot (ChatGPT, Claude, etc.) with no technical skills. The work gives users a fishing rod rather than a fish: an understanding of how to manage an LLM like a manager, not just a customer.

Counterarguments (why the score could be lower): * The study focuses on describing a specific system (Magentic-UI) that everyday users don't have access to. Users have to do the mental work of adapting the ideas rather than just copying ready-made prompts. * Part of the study covers technical implementation (AutoGen, Docker), which is irrelevant to a broad audience. * The main focus is LLM agents that take real-world actions (browsing, working with files), a narrower scenario than general chat requests.

Counterarguments (why the score could be higher): * The human-in-the-loop approach may be the single most important skill for using LLMs effectively in the future. Learning it has huge long-term value, beyond knowing individual prompting techniques. * The paper's principles directly address one of the biggest LLM problems, their "brittleness" and inability to stick to complex logic, which makes the study critical for practical use.


Research digest

Key result

The problem: an LLM agent takes 15 steps on a complex task and messes up on step 7, and you only find out once it has "finished" everything. All that work, wasted. Magentic-UI lets you insert yourself into the process at every meaningful step: first you ask for a plan and edit it, then the agent moves forward but stops at the right points and waits for your approval. The shift: you don't write one giant prompt hoping for a miracle; you take on the role of project manager, and the LLM becomes a contractor who reports back at key stages. The interaction stops being a lottery.

Principle

Split any complex task into three clear stages. First: demand a plan, not the result. The LLM shows HOW it will solve the task, and you see the logic before it goes off the rails. Second: edit the plan right in the chat until it's precise. That takes two minutes and saves an hour of rework. Third: give the go-ahead, but with explicit stopping points. "When you get to step 3, stop, show me the options and wait for my choice." The LLM doesn't guess your preferences. It asks.

Why it works

An LLM doesn't know when it took a wrong turn. Without intermediate feedback, it marches confidently toward the wrong finish, in a confident tone to boot. Stopping points solve the main problem: a mistake gets fixed at step 3 instead of discovered at step 15. A separate bonus: hallucinations. Requiring sources at an intermediate stage turns blind trust into a verifiable process: instead of getting a wall of text and guessing what's true, you check the claims while there are three of them, not thirty.

When to apply

Tasks where a mid-course mistake is expensive: building content strategies and plans, research that demands factual accuracy, multi-step analysis (financial, marketing, technical), writing long structured documents, especially when the result goes beyond just you. NOT a fit for simple, unambiguous requests: there the planning step just slows you down.

Mini recipe

1. Demand the plan first: start with "Before executing, draft a step-by-step plan. Don't start the task, just show me the plan for sign-off." The LLM structures its approach and you see the logic. 2. Fix the plan while you still can: once you have the plan, check whether it understood the task. Edit specifically: "In step 2 search site X; drop step 4, we don't need it." 3. Set stopping points: when you hand over the task for execution, spell it out: "When you get to [specific step], stop. Show me the options and wait for my choice." The LLM won't move on without your signal. 4. Save the template: if the structure worked, save the prompt for similar tasks. Don't reinvent it every time.

Examples

[BAD]: `Create a one-week content plan for a coffee shop's Instagram account` [GOOD]: Stage 1: `You are an experienced social media manager. Task: a one-week content plan for the coffee shop "Cozy Corner". Your only job right now is to draft a plan for HOW you will create it: audience analysis, choosing content pillars, format types. Don't write the content plan itself. Just show me the work plan and wait for my approval.` (After the plan is edited and approved) Stage 2: `Plan approved. Go ahead. When you get to generating post ideas, stop. Give me a table: idea, content pillar, format (post or short video). Wait for my choice; don't write any copy until I mark which ideas I'm taking.`

In plain words

The point of Magentic-UI is that today's AI agents too often turn into "black boxes": you give them a task, they spin some gears under the hood and hand back a finished (often wonky) result. The researchers realized autonomy is a trap and proposed Human-in-the-loop Agentic Systems. Instead of just executing a command, the agent first builds a dynamic interface for agreeing on the steps with you. It's not just a chat, it's a full control panel where you see the model's logic and can correct it before it makes a mess.

It's like hiring a contractor to remodel your apartment. A typical agent is the contractor who silently disappears for a month and ends up painting the walls pink because "that's his vision". Magentic-UI is the contractor who first brings you a mood board and an estimate, asks you to sign off on the colors, and only picks up the roller after your "OK". Technically the job is the same, but the odds of waking up in a pink nightmare drop to near zero, because you control the decision points.

It rests on three methods: co-planning, intermediate verification and an adaptive interface. The agent doesn't rush off to write the post; it rolls out the structure first, say, a week-long plan for a coffee shop. It flags the critical moments: "here we lean into coziness, there we push the promo". You don't rewrite everything from scratch, you just point at a specific block and say: "swap this one". Magentic-UI makes the process transparent and turns AI from a moody contractor into a predictable tool.

Although the study wrestled with interfaces and code, the principle of agentic planning applies everywhere, from social media and marketing to complex data analysis. It's a universal pattern for any situation where a mistake costs more than zero. If you ask a model to build a financial plan or a brand strategy, you don't need a "final file" in one second. You need an interactive draft where you can cut the nonsense early. Autonomy is overrated; control is the new standard.

Bottom line: stop expecting magic and perfect results from AI on the first prompt. The future belongs to systems that ask for confirmation at every turn. Use the Magentic-UI approach: make the model plan and show you the structure first, and only then generate content. It's the only way to stop getting typical AI slop and start shipping genuinely good work. Whoever learns to conduct agents like an orchestra wins.

AI lab assistant

Apply this study to your task

Describe what you're working on. AI takes the method and the measured results from this paper and adapts them to your case, or writes a ready prompt built on them.

Based on Magentic-UI: Become the Agent's Manager Instead of Waiting for a Black-Box Miracle arXiv:2507.22358

Sign in to apply this study New accounts get 50 free credits.

Get the week’s best papers by email

One email on Sundays with the highest-rated new techniques. Unsubscribe with one click.