OpenGCUA: the agent writes a short program instead of a list of clicks and decides for itself when to look at the screen

EVERY STEP (one model call):
  1. Choose what to observe → screenshot / window element tree / command output / nothing
  2. Choose how to act → ONE program in <code>…</code>
     (clicks + terminal + API + if / for / assert)
  3. Update working state → plan, what's done, what to keep in context

Before the program: one line on "what it's for"
After execution: program output (+ any requested screen views) → next step

Everything happens in one system prompt; the harness runs the code.

Task: The bookkeeper at a small online store moves 40 rows of an order export from Excel into a shipping service's web dashboard every week. The service has no API, only a form on its website. The form has three fields, a "Session expiring" popup appears now and then, and each row needs confirmation that the order was saved.

Prompt (system instruction for an agent with computer access; template below):

Task: transfer the orders from ~/Documents/orders_week.xlsx into the form 
on the dashboard page that's open in the browser.

For each row, fill in the "Order number", "Address" and "Weight, kg" fields 
and click "Save".

How to work:
- Read the Excel data through the terminal (python + openpyxl), not by eyeballing the screen.
- Process all 40 rows in one program with a loop, not one row per call.
- Inside the loop: if a "Session expiring" window appears, close it and continue.
- After "Save", check that the row's status changed to "Saved". 
  If not, add the order number to an error list and move on.
- At the end of the program, print how many were saved and the error list.
- Request a screenshot only if the program returned errors or you can't tell 
  what's on the screen.

Result: The agent reads the file through the terminal and writes one program that loops over the rows. It handles the popup and checks the status after each save. The output is a short summary: how many orders were saved and which rows failed. It requests screenshots only on failures. In the best case, 40 rows fit into a handful of calls instead of a hundred.

The weakness. When a model outputs a ready-made list of actions, it decides everything up front without seeing the result of the first step. If something goes wrong on screen, the list no longer fits. If the model outputs one action at a time, every step costs a full call, and the harness sends a screenshot after each one by rule. Lots of calls, a ballooning context, everything slow and expensive.

The strength. Modern models write code well. They express conditions, loops and checks precisely. Code in a single program reacts to intermediate results on its own. You don't need to call the model to find out the dialog closed.

How the method exploits this. A program replaces the batch of actions: "a batch is a program without control flow." And the decisions "when to look" and "what to remember" go to the model, because only it knows what this particular step needs. The effect in the paper: higher accuracy and fewer steps for every model tested. The gap between the strongest and weakest model shrank by more than half.

Prompt control knobs: - Interfaces (screen, terminal, API, browser): add whichever you have and hint which is for what. Files and data go through the terminal, apps without an API through the screen. - Examples in the prompt (condition, loop, check): in the original they show the model how to combine clicks, commands and checks. Give examples from your own task. - The screenshot rule: "request only when necessary" saves context. In the paper this raised the cache hit rate by 10–14 pp with no loss in score. For unreliable interfaces, allow more frequent requests. - The "what this program is for" line: gives you a readable action log. Drop it if you need to save tokens. - Program size: "one program" versus "in pieces." For risky operations (sending, deleting), cap the size and add a check after each one.

The paper moves the full system prompt to Appendix C; it isn't in the main text. Below is a template assembled from the paper's description: interfaces and output format (one program in a <code> block), examples with conditions and checks, the program's purpose, screenshots on request, state tracking.

You are an agent that carries out tasks on the user's computer.

<Interfaces>
- Screen: {pyautogui.click(x, y), pyautogui.write("..."), pyautogui.hotkey(...)}
- Terminal: bash("command") — returns output and exit code
- Browser: {Playwright, if needed}
- {other interfaces: API, window element tree}
</Interfaces>

<Output>
At each step, output ONE Python program in a <code>...</code> block.
Before the block, write one line on what the program is for.
The program may combine clicks, terminal commands and calculations, 
and use if, for, variables and result checks.
</Output>

<Rules>
1. Pick the interface for the step: files and data via the terminal, 
   apps without an API via the screen.
2. Bundle related operations into one program; don't split them into tiny calls.
3. Condition: if a dialog or error might appear, handle it in the program.
4. Loop: do repetitive operations in a loop.
5. Check: after an important action, verify the result (assert or a condition), 
   not just that the action ran.
6. Screenshots don't arrive automatically. At the end of your answer write: 
   SCREENSHOT: yes / no. Request one only when you can't tell the state without it.
7. Maintain working state: the plan, what's done, what's left. 
   Update it at the start of every answer.
</Rules>

<Examples>
{Example 1: condition — "if the dialog is visible, close it"}
{Example 2: loop — "for each row, fill in three fields"}
{Example 3: check — "verify the status is now 'Saved'"}
</Examples>

Task: {task}

What to fill in: take {interfaces} from your environment. Write the three examples in <Examples> using your own typical operations. Describe {task} with a "done" criterion.

🚀 Quick start: paste this into the chat:

Here is a system prompt template for an agent that operates a computer 
and writes a program instead of a list of actions. Adapt it to my task: [your task]. 
Ask me questions to fill in the fields.

[paste the template above]

The LLM will ask which interfaces are available (terminal, browser, API), what typical failures happen (dialogs, timeouts), and what "task done" looks like. It needs this to write three examples with a condition, a loop and a check based on your operations. Those examples are what set the pattern for the agent.

⚠️ You need a code-based harness: the method works when there's an environment that extracts code from the answer, checks the syntax and runs it. In a chat without code execution and without computer access, "a program instead of clicks" gets you nothing. It partly carries over to off-the-shelf agents (Claude Code, Cursor), but you can't set up the harness itself with a single instruction.

⚠️ The full prompt isn't in the main text: the original system prompt is in Appendix C. The template above was assembled from the description in the main body of the paper, not copied verbatim.

⚠️ Check the outcome, not an intermediate step: in one of the real cases analyzed, the program compared an email draft and decided whether to click "Send." It never checked delivery. A program does more per call, so an error without an outcome check costs more.

⚠️ Models differ a lot: on real macOS, one model spent dozens of times more tokens on reasoning than another and completed fewer tasks. The authors attribute this to excess reasoning on simple operations, poor adaptation to real-world conditions, and weak skill at writing computer-control code. The method doesn't level everything out.

⚠️ Real devices are preliminary results: the main numbers come from the OSWorld-v2 benchmark, and the real-device testbed was checked on just four macOS tasks. Treat it as a warning, not as proof.

⚠️ The parts' contributions aren't cleanly separated: the big gain comes from comparing the full stack against the weakest variant (screen only, action batches). Over an already-hybrid agent with a terminal, the code approach adds noticeably less.

A team from Alibaba took eight frontier models, including GPT-6 Astra and Claude Opus 5, and ran them on OSWorld-v2 (a suite of computer tasks). No model training was needed. Only the harness and the prompt changed. They compared four configurations: "screen only, action batches," "screen + terminal," "full stack with code," and ablations of each part separately.

The full stack improved all eight models: on average +19.6 points of partial score and −53% steps per task. Over the "screen + terminal" hybrid, code added another +6.1 points and −14.8% steps. For Qwen3.8-Max that's +5.2 points and −17% steps. The decisions "when to look at the screen" and "how to maintain state" were handed to the model. The "when to look" decision alone raised the cache hit rate by 10.5 and 14.4 pp for two models with no drop in score.

The team then tested a real Mac under their harness on four tasks. This revealed that the model itself matters a lot: with the same harness, reasoning-token spend differed by a factor of 19.4. The practical takeaway: the harness helps everyone but doesn't replace model quality, and excess reasoning on simple operations costs money and time.

This is not from the paper; it's carrying the principle over to tools the reader already has.

🔧 Technique: a "script instead of a series of commands" rule in CLAUDE.md / AGENTS.md → fewer calls and an outcome check

## How to handle repetitive operations
- If an action needs to happen more than twice, write one script 
  with a loop instead of calling commands one by one.
- If a step can fail (file locked, dialog, timeout), 
  handle it inside the script.
- After the script, verify the result (file count, status, tests), 
  not just that it finished without an error.
- Before running, write one line on what the script does.

🔧 Technique: remove the "what this program is for" line → fewer tokens; keep it → a readable log. For routine tasks where you don't read the log, drop it. For sensitive operations (sending emails, payments), keep it so you see the intent before the run.

• OpenGCUA: Toward a Fully Open Stack for Generalist Computer Use
• GitHub: https://github.com/Tongyi-MAI/OpenGCUA
• Project site: https://tongyi-mai.github.io/OpenGCUA/
• Authors: Hanzhang Zhou, Long Li, Quyu Kong, Lei Wang, Panrong Tong et al., Alibaba Token Hub, Alibaba Group
• Related work cited in the paper: Qwen-UI-Agent (Zhou et al., 2026), Agents' Last Exam (Sun et al., 2026), OSWorld-v2 (Yuan et al., 2026)