PhyEvolve: a three-role pipeline that fills in a scene's physics before the prompt ever reaches the video generator

STEP 1. DIRECTOR: scene + events → table "event | type | what it can change"
        (physical action / environment change / visibility change / scene constraint)
        → self-check

STEP 2. PHYSICAL EVOLVER: 
        2a. mechanisms: (evidence, activation condition, mechanism, expected response)
        2b. transition chain: before → conditions → process → after → depends on #…
        → separate check of the records

STEP 3. COMPOSER: full replacement prompt → check against records
        → fix ONLY flagged sentences → repeat (up to 3 times)
        → no new mechanisms allowed

→ the final prompt goes to the video generator instead of the original

You can do it all in one prompt or in three messages in a row.

Task: You're making a Domino's-style ad for a pizza place in Veo or Kling. The pizza goes into the oven, the camera pans to the face of a waiting customer, then comes back to the oven. The generator will most likely show the same raw pizza: while the camera was on the customer, "nothing happened."

Prompt:

You work as a three-role pipeline. My task is to improve a prompt for a video generator.

Original prompt:
"A chef in a red apron slides a raw pepperoni pizza into a wood-fired oven and closes the oven door. The camera smoothly pans to the face of a young guy at a table: he checks his watch and smiles. The camera returns to the oven, the chef opens the door and pulls out the pizza."

<Director>
List the events in the prompt. For each, give its type: physical action / environment change / visibility change / scene constraint. For each, state what it CAN change: physical conditions, object state, visibility. Rule: the event "the camera pans" changes visibility only and does not start, stop, or reset any physical process. Check that your labeling matches the text.
</Director>

<PhysicalEvolver>
1. For each physical process, write the mechanism: (evidence from the text, activation condition, mechanism, expected response).
2. Build a transition chain: state before → active conditions → process → state after → mechanism numbers → which earlier transitions it depends on. Track visibility separately from physical state. Processes whose conditions stay active keep going off-screen too. If the text doesn't imply that something changes, mark it "undetermined."
3. Check the records: are the mechanisms grounded in the text, and is state continuity preserved?
</PhysicalEvolver>

<Composer>
Assemble a new, self-contained prompt for the video generator. Keep all objects, actions, event order, and pacing. Add the physical states and changes needed to connect the events. Then check the draft against the records: find sentences where a required state is missing, a causal link is broken, or the original request is altered. Rewrite only those sentences and leave the rest alone. Don't invent new physical mechanisms. Repeat the check up to three times. In your answer, show: the Director's table, the Physicist's records, the final prompt.
</Composer>

Result: The model returns three blocks. First, an event table where "camera pans" is marked "visibility only." Second, mechanism records (heating, cheese melting, browning) and a state chain in which the pizza keeps baking while the camera is on the customer. Third, a full prompt where, when the camera returns, the pizza is already done: golden crust, melted cheese. You paste that into the generator instead of the original.

The weakness. A short prompt describes the scene and the actions but leaves mechanisms, conditions, and consequences "off-screen." The generator follows what's written. If an off-screen process isn't mentioned, it doesn't evolve. On top of that, "not visible" easily blurs into "not happening."

The strength. A language model is good at breaking familiar physical processes down into causes and stages. Make it write them out explicitly and it won't skip them. It's also good at checking text against existing records: "this sentence forgot a state."

How the method uses this. The analysis is split into stages, and each stage's output is recorded as structure. Every next state builds on the previous one, so "the pizza got baked" carries through the shots where it isn't visible. The final prompt can be checked against the records rather than eyeballed. Edits are surgical: accepted sentences stay untouched, so nothing good gets broken.

Control knobs: - Check-repeat limit (up to 3 in the paper) — lower it for simple scenes to save time. - Director's event types — add your own types (sound, lighting, character reaction) if the scene is more complex. - The "undetermined" line — keep it so the model doesn't invent things that aren't in the text. Drop it if you want more creative detail. - Composer's ban on new mechanisms — loosen it if you want freewheeling direction rather than precision. - Showing intermediate records — keep it to see where the model got the physics wrong. Ask for the final prompt only to get results faster.

The paper contains no verbatim prompts. The template is built from the method description: roles, record fields, and check rules are preserved.

You work as a three-role pipeline. Improve a prompt for a video generator.

Original prompt:
"{original_prompt}"

First frame (if any): {first_frame_description}

<Director>
1. List entities and their initial states from the text{and_frame} only. Don't reconstruct the past or predict outcomes.
2. Find the sentences with events. For each, give its type: physical action / environment change / visibility change / scene constraint.
3. For each event, state what it CAN change: physical conditions, object state, visibility.
4. Rule: an event that only hides or reveals an object ({observation_examples}) changes visibility but does not start, stop, or reset physics. If the text explicitly states a physical effect of the event, keep it.
5. Check: does the labeling match the text, the event types, and the causal roles?
</Director>

<PhysicalEvolver>
1. Mechanisms. For each mechanism record: (evidence — support from the text; condition — activation condition; mechanism — physical mechanism; response — expected response).
2. Transitions. For each transition record: state before; active conditions; process of change; state after; mechanism numbers; which earlier transitions it depends on. The "after" state becomes the "before" of the next transition.
3. Track physical state and visibility SEPARATELY. If an event changes only visibility, processes with active conditions continue off-screen, and when the object comes back into view, describe its already-changed state.
4. If the text doesn't determine a change, mark it "undetermined."
5. In a separate pass, check: grounding of mechanisms in the text, conditions and responses, state continuity, transition dependencies, consistency with the Director's constraints. Fix the records that fail.
</PhysicalEvolver>

<Composer>
1. Assemble a self-contained replacement prompt. Keep objects, actions, event order, scene constraints, and pacing. Add the physical conditions and state changes that connect the events.
2. Check the draft against the Director's and Physicist's records. Flag sentences that: skip a required state, violate a causal constraint, or alter the original request.
3. Rewrite only the flagged sentences. Leave accepted ones alone. You may spell out mechanisms already recorded in the transitions, but you may not introduce new ones.
4. Repeat the check no more than {repeat_count} times. If the check still fails, return the latest version.
</Composer>

Show: the Director's table, the Physicist's records, the final prompt.

What to fill in: - {original_prompt} — your usual short video prompt. - {first_frame_description} — if you're generating from an image, describe it in words. If not, delete the line. - {observation_examples} — what hides or reveals the object in your scene: the camera, a curtain, lights going off, smoke. - {repeat_count} — 2–3.

🚀 Quick start — paste into a chat:

Here's the PhyEvolve template. Adapt it to my task: {describe the video scene}.
Ask me questions to fill in the fields.

[paste the template above]

The model will ask what events happen in the scene, what gets hidden from the camera, and which processes should keep running off-screen. That's what the Director needs to separate visibility from physics correctly. It will take the structure from the template and assemble a ready-made prompt for your scene.

⚠️ Video only: The method was tested on a video generator and physical scenes: water, fire, mechanics, materials. No gains were shown for images, text, or general tasks. You'd have to port it yourself.

⚠️ Tested on one generator: All numbers come from a single video "engine" (Minimax-H3). The effect wasn't tested on Veo, Kling, or Sora. It could be smaller or larger.

⚠️ Faithfulness to the request is still a weak spot: Physics improves, but the generator still often fails to do exactly what was asked (on the hard scene set, semantic accuracy stayed low). The method doesn't guarantee "it came out exactly as ordered."

⚠️ No check against the video: All the improvement happens before generation. The finished video never goes back into the pipeline. If the generator ignored a good prompt, the method won't catch it.

⚠️ Physics errors in the records break the result: The planner model can get a mechanism wrong, and the error flows into the prompt. Read the Physicist's records, especially for unusual scenes.

⚠️ In the paper this is an automated pipeline: The authors run it as a program on several GPUs. In a chat you reproduce the logic by hand, without the paper's exact settings.

⚠️ Some areas lag behind: On optics, the competing method NEWTON came out stronger.

The authors from Canva Research tested the method on four physics benchmark sets: "invisible" evolution (StEvo-Bench) and physical common sense (PhyGenBench, PhyWorldBench, VideoPhy-2). The video generator was held fixed; only the prompt changed. They compared against direct generation and two agentic competitors (NEWTON, Co-Director). The planner was the open model Qwen3-VL-32B.

Averaged over the four benchmarks, PhyEvolve beat direct generation by 14.9 points and the best competitor by 7.9. The most striking result: on "invisible" processes, success rose from 17.7% to 45.7%.

Then the authors switched off the roles one at a time. Without the Physicist, success dropped to 29.0%, the biggest loss. Without the Director, the main metric barely moved (45.3%), but camera and action control slipped. Without the Composer, losses were small (about 1–3 points), although in one illustration a glass ended up visible behind the curtain.

The surprise: the heavy lifting is done by the explicit physics step, not by "clever" prompt assembly. Practical takeaway: if you don't have time for all three steps, don't skip the second one. The authors also swapped the planner. A 9B-parameter model gave nearly the same result as a 30B one (77.3% vs. 79.6%), so the method doesn't need a top-tier model. In a human preference study, PhyEvolve videos were preferred in more than 60% of comparisons.

The provided text contains no verbatim prompts. Here's the paper's formalization of the pipeline and the records:

D = Director(P, I0),
(M, T) = Evolver(P, D),
P* = Composer(P, D, M, T),
V = G(P*, I0)

Mechanism record:  m_i = (e_i, c_i, φ_i, r_i)
  e = supporting evidence, c = activation condition,
  φ = physical mechanism, r = expected response

State transition:  s_k --(M_k, c_k)--> s_{k+1},   o_k = O(s_k, v_k)
  s = physical state, v = visibility condition, o = what is observable

Context: This is how the authors describe the data handoff between roles and the structure of the records. State and visibility are tracked separately, which is why a process keeps going while the object is hidden.

💡 Adaptation for storyboards and scripts: The "visibility ≠ event" rule works for text too. Have the model label what's happening "off-screen" in the story: while the hero is in another scene, everyone else's processes keep running.

Here's a TV series script: {script}. 
Step 1: list the events and mark which ones change the world and which change only what the viewer sees.
Step 2: for every storyline that drops out of frame, record the state "before → what keeps going → state after."
Step 3: check whether the scenes stay consistent when we return to that storyline. Fix only the inconsistent spots.

🔧 Technique: edit only flagged spots → don't break what works. The Composer's idea is handy in everyday editing too: "First flag the sentences that violate criterion X. Rewrite only those. Leave everything else alone and don't paraphrase it." That keeps edits surgical, and the good parts stay untouched.

• Paper: "PhyEvolve: Multi-Agent State-Evolution Planning for Physics-Aware Video Generation"
• Authors: Yang-Hao Zhou, Lixing Tan, Zhiqi Li, Guang Yang, Xiao Yang (Canva Research)
• Benchmarks: StEvo-Bench, PhyGenBench, PhyWorldBench, VideoPhy-2
• Competitors: NEWTON, Co-Director
• Generator: Minimax-H3; planner: Qwen3-VL-32B-Instruct