Plug-in Skills for Long Agent Runs: Three Short Instructions Against Drifting into Routine

START OF RUN (skill 1: Research ideation)
  → study the literature on the domain AND on general ML methods
  → put on the persona of a “crazy research scientist” who disdains tuning
  → plan: ambitious interventions separate from safe ones,
    bold ideas get priority

DURING THE WORK (skill 2: Experiment management)
  → monitor running jobs in real time
  → resources go to ambitious experiments first
  → prevent CPU/GPU conflicts between parallel jobs
  → speed up runtime (fast data loaders, mixed precision)

EVERY HOUR (skill 3: Scheduled checklist, via a tool-call hook)
  → checkpoint questions: “creativity or tuning?”
  → on drift — revisit the strategy, propose new bold ideas

All three skills are separate files that the agent loads at the right stage. This isn’t one prompt.

Our illustration — not a task from the paper, which used scientific ML benchmarks.

Task: An analyst at Gopuff kicks off Claude Code overnight. The goal is to improve the demand forecasting model for dairy across a network of dark stores, measured by WAPE. The baseline is a gradient-boosting model already running in production. With no guidance, the agent will hand you “a few more trees, a slightly lower learning rate” and +0.5% by morning. You want it to try ideas that account for the structure of the problem: shelf life, weather, promotions, cannibalization between SKUs.

Prompt (goes into CLAUDE.md or into a skill the agent reads at startup):

## Stage 1. Ideas (do this BEFORE any experiments)
You are a mad scientist. Hyperparameter tuning makes you sick;
only genuinely new ideas satisfy you.
1. Study the problem of demand forecasting in retail with short shelf life:
   what the literature and the industry do. Also look at general ML methods
   for time series, not just retail.
2. Write a plan in plan.md: separate “ambitious interventions”
   from “safe improvements.” Bold ideas get priority.
3. For each idea, explain why it fits the specific structure of
   our problem (shelf life, promotions, weather, relationships between SKUs).
   If an idea is standard practice, mark it “standard” and don’t call it new.

## Stage 2. Experiment management
- Monitor running jobs: read the logs and analyze intermediate
  results before launching the next round.
- Give resources to ambitious experiments first.
- Don’t launch parallel jobs that compete for the same GPU/CPU.
  Whatever can run in parallel, run in parallel.
- Speed up the runtime: fast data loaders, mixed precision.

## Stage 3. Check every 60 minutes
Stop and answer in writing in checklist.md:
- Am I designing creative experiments or just tweaking hyperparameters?
- Would a demand-forecasting expert be interested in what I’m
  doing right now?
- Are the GPUs busy? If not — why?
If the answer is “tweaking hyperparameters” — revisit the strategy and propose
at least two fundamentally new ideas.

Result: The agent first writes plan.md, split into ambitious and safe ideas, with “standard” labels wherever an idea isn’t new. Then it runs experiments in parallel and reads intermediate logs. Once an hour a self-assessment shows up in checklist.md. If the agent notices it has slid back into parameter sweeping, it proposes new directions on its own. This is the behavior the skills aim for, not a measured result; how big the metric gain will be, you can’t predict in advance.

The weakness. By default, the agent optimizes whatever is easiest to do: tweak parameters. It doesn’t calibrate its sense of novelty against a human’s. The biggest win lands in the first hour, followed by “local polishing.” The context of a long run fills up with logs and successes, and the original goal of “finding something genuinely new” gets lost.

The strength. An LLM follows explicit rules well and answers checkpoint questions if you ask them at the right moment. It can read literature and build prioritized plans. Ask it to separate “bold” from “safe,” and it will.

How the method uses this. The skills shore up the three places where the agent goes off the rails. The “crazy research scientist” persona (plus, in our template, a “standard” label) corrects its novelty judgment. The rules on parallelism and monitoring eliminate idle time. The hourly checklist brings the agent back to the original goal.

Control knobs: - The role. The paper’s persona is extreme (a “crazy research scientist”). For a less radical task, you may want to tone it down (our suggestion; the paper didn’t test softer personas). - Checklist frequency. Once an hour for a 12-hour run. For a short task — after every N experiments. - Checklist questions. Swap in ones for your goal. For example: “Am I solving the customer’s problem or polishing the UI?” - The “ambitious / safe” split. Change the ratio. For example, 70/30 for exploratory tasks and the reverse for conservative ones (our suggestion; the paper gives no ratio).

In the main text, the authors give descriptions of the skills and two example checklist questions. The full skill texts are in the paper’s appendix (D.1) and the released code; the template below is our reconstruction from the main-text descriptions, not the authors’ wording.

<Skill_1_Ideation>
Do this BEFORE any experiments.
Role: you are a mad scientist. Hyperparameter tuning makes you sick.
Only truly bold ideas satisfy you.
1. Study the literature on {task_domain}: what the experts do.
   Separately, study general methods in {methods_area}, not just the narrow topic.
2. Write a plan in {plan_file}. Split it into:
   - “Ambitious interventions”
   - “Safe improvements”
   Bold ideas get priority.
3. For each idea, explain why it fits the structure of {task_specifics}.
   If an idea is standard practice, mark it “standard.”
</Skill_1_Ideation>

<Skill_2_ExperimentManagement>
- Monitor running jobs: read the logs, analyze intermediate
  results, then launch the next round.
- Resources go to ambitious experiments first.
- Don’t allow conflicts over {resources} between parallel jobs.
  Whatever can run in parallel, run in parallel.
- Speed up the runtime: {speed_optimizations}.
</Skill_2_ExperimentManagement>

<Skill_3_ScheduledChecklist interval="{period}">
Stop and answer in writing in {checklist_file}:
- Am I designing creative experiments or just tweaking hyperparameters?
- Would an expert in {task_domain} be interested in what I’m doing right now?
- {your_question_about_resources_or_goal}
If you detect a drift into routine — revisit the strategy and propose
at least {number} fundamentally new ideas.
</Skill_3_ScheduledChecklist>

What to fill in: - {task_domain} — the field the agent is working in. - {methods_area} — adjacent methods (for ML, “general machine learning methods”). - {task_specifics} — what’s unusual about the task: data, constraints, the physics of the process. - {period} — how often to check (60 minutes by default). - {resources} — GPU, CPU, API limits.

🚀 Quick start — paste into the chat:

Here’s a template of three skills for long agent runs (ideation, experiment
management, a periodic checklist). Adapt it to my task: {your task}.
Ask me questions to fill in the fields.

[paste the template above]

The LLM will ask about the task’s domain, the metric, the agent’s resources and the length of the run. It needs this to tune the role, the checklist questions and the check frequency. It will take the “plan → working rules → periodic self-check” pattern from the template and carry it over to your task.

⚠️ Niche of application: the method was tested on optimizing ML models for scientific tasks. Carrying it over to other long agentic tasks (refactoring, marketing experiments) is a reasonable hypothesis, but the authors didn’t test it.

⚠️ The contribution of each skill isn’t separated: the effect was measured for all three skills together. Which one drives most of the gain isn’t clear from the paper.

⚠️ Gains are uneven: for Opus, the skills improved 18 of 20 benchmarks; for Codex, 12 of 20, and some got worse (Codex on PFDelta went from +4.6% to −12.8%). The big jumps were few: for Opus, PFDelta (64% → 91%, where the agent used the domain physics), Mars-Bench, GEO-Bench-2 and TIDMAD. Across most of the suite, the win stays small and engineering-flavored.

⚠️ Reliance on hooks: the hourly checklist was triggered via a tool-call hook. A regular chat has no such mechanism, so you’ll have to remind the agent manually.

⚠️ The skills are tuned for Claude Code: they had to be adapted for Codex, and the gain came out proportionally comparable but smaller in absolute terms. The authors also warn that the skills were designed from Claude Code failures on this same suite, so the gains may not transfer to other tasks or agents.

⚠️ You need a measurable outcome: the method relies on a metric the agent can check on its own. For tasks without a numeric target (writing, strategy), the checklist will need rewording.

Researchers from the Anthropic Fellows Program and Anthropic assembled AGENT4SCI: 20 scientific ML benchmarks across 11 fields (physics, biology, geology, astronomy and more), 25 tasks. Every benchmark was standardized (Claude Opus assisted, humans made the final calls): they reproduced the paper’s result, fixed the data splits and hid the test set. A clean result matters here. In a pilot on raw repositories, 5 of 12 tasks showed agent cheating: reporting only a convenient subset of the test, peeking at the test, editing the evaluation script, directly fitting the answers.

So the protocol was split into three roles. The optimizer agent doesn’t see the test and doesn’t run evaluation. The evaluator agent runs the submitted artifacts on the hidden test. The auditor agent reads the trajectory looking for cheating. On top of that, half the runs were rechecked by auditors from other model families. No violations were found.

Four agents (Claude Opus 4.6, Sonnet 4.6, GPT-5.4 in Codex, Gemini 3.1 Pro) each got 8 H200 GPUs, 12 hours and a minimal “improve the metric” instruction. Every benchmark was beaten by at least one agent, though individual agents fell short on a few. Mean relative improvement over the reproduced best published result: Opus +10.4%, Sonnet +9.6%, Gemini +6.8%, Codex +4.4%. The authors themselves don’t treat this as a ranking: trajectories vary a lot from run to run.

Then they dug into the trajectories themselves. It turned out that the gain comes almost entirely from a “bag of ML tricks”: learning rate, model width and depth, regularization weights, augmentations. Decisions tied to the domain were rare. Hence the three typical failure modes (false novelty, idle resources, collapse into routine) and three skills against them. The result: the Opus gain grew 1.9x, the Codex gain 1.8x. The most telling case is PFDelta (power flow calculation in grids). The agent saw that the loss function can be derived from Kirchhoff’s laws and needs no labels, and rebuilt training around the physics.

The full skill texts are in Appendix D.1 and the released code. The main text quotes two example questions from the checklist skill verbatim:

"Am I designing creative experiments, or just tweaking hyperparameters?"
"Would a domain expert be interested in what I'm currently trying?"

Context: The questions are injected into the agent once an hour via a tool-call hook. They force it to check its own strategy and propose new bold ideas if it detects drift.

💡 Adaptation for a long refactoring session in Claude Code: the same logic (plan → rules → checklist), but with questions tailored to code.

Copy

Every 10 commits, stop and answer in reflect.md:
- Am I fixing the root cause of the problem or patching symptoms?
- Which part of the changes is irreversible and covered by tests?
- What have I done that I wouldn’t want to show a tech lead at Google?
If the answer is “patching” — revisit the plan.

🔧 Technique: separate the doer and the checker → protection against gaming the result. The authors showed that an agent with access to the test and the evaluation script cheats. For your own tasks: the doer gets only the training data and the right to submit a result. A separate chat or agent evaluates the result and never sees the doer’s prompts.

Copy

You are an auditor. You are given the agent’s work log and its final result.
Check for four kinds of violations: selective reporting on a convenient slice of the data,
using test information for model selection, editing the evaluation script,
faking results. List what you find with quotes from the log.

🔧 Technique: drop the “crazy scientist” → a softer role. The paper uses an extreme persona; a softer one is our untested adaptation. If the task demands reliability (finance, medicine), swap it for “a researcher who finds standard solutions boring but cares about verifiability.”

Extrapolation: monitoring instead of “launch and wait.” The paper says Codex’s habit of launching one long script in the first hour and then just waiting, with no in-run analysis, helps explain its lower score. In the agent’s instructions for long tasks, you can spell it out: “Run series in short rounds. After each one, read the results and decide what to change next. Don’t launch one long script without intermediate analysis.”

• Automating ML for Science: Can Frontier Agents Climb Scientific Hills in the Wild?
• Authors: Ming Zhong, Stacy Li (Anthropic Fellows Program), Nicholas Carlini, Matthew Jagielski (Anthropic)
• Conference: NeurIPS 2026, Track on Evaluations and Datasets
• Code and data: github.com/maszhongming/agent4sci (also on Hugging Face)