🛠️ AutoSaddler: Automatic Harness Optimization with Durable Updates from Agent Execution Traces

AutoSaddler automatically improves LLM-agent harnesses by diagnosing execution traces, applying structured updates to prompts, tools, and middleware, and selecting changes that generalize.

Preliminary results report the following test Pass@1 scores across benchmarks and agent harnesses:

✨ Highlights

• Full-harness optimization: searches over prompts, tool definitions and implementations, middleware hooks, and agent-loop logic.
• In-depth diagnosis: deeply debugs execution traces and the harness codebase to identify root causes rather than relying on shallow reflection.
• Structured intervention: targets prompts, tools, and middleware through an explicit patch taxonomy and phased Capability-to-Steering schedule instead of unconstrained editing.
• Generalization-aware selection: validates updates beyond the motivating trajectories and uses reflection with an evolution DAG (EvoDAG) to retain broadly useful lessons.
• Durable execution: records append-only events, immutable provenance, resumable state, and content-addressed candidates.

🛠️ Installation

AutoSaddler requires Python 3.12-3.14, uv, and Git.

git clone https://github.com/microsoft/AutoSaddler.git
cd AutoSaddler
uv sync --extra dev

🚀 Quick Start

Run the deterministic, credential-free V2 template to exercise the optimization engine, event store, candidate evolution, and output projections:

uv run python -m autosaddler.v2.cli \
  --config configs/v2/local_template.yaml \
  --run-id local-template

🧭 Versions

• V2 (current): the durable, plugin-based implementation.
• V1 (legacy): the research-quality implementation used for the experiments in the arXiv paper.

🧩 How It Works

AutoSaddler formulates harness optimization as offline mini-batch learning. It uses three session types across the optimization lifecycle:

1. Diagnosis-Patch: inspects failed traces and the harness codebase, then proposes structured Capability patches (code or infrastructure) and Steering patches (textual behavior changes).
2. Reflection: compares pre- and post-patch traces, classifies fixed, regressed, still-failing, and still-passing cases, and records reusable lessons.
3. Evolution: consults the full EvoDAG to synthesize candidates from successful components and lessons across lineages.

🎯 Supported Harnesses and Benchmarks

⚙️ Configuration

Every V2 config starts with schema_version: autosaddler/v2. A scenario plugin is the adapter between AutoSaddler's generic optimization engine and a specific harness/benchmark pair.

🔬 Reproducing the Included GAIA2 Smoke Run

The smoke config exercises the real optimization pipeline on seven GAIA2 scenarios.

1. Prepare the repositories

Use a sibling layout with AutoSaddler/, Meta-ARE/, meta_are_data/, and a working_dir/.

2. Provision the benchmark inputs

uv run --extra meta-are-setup python scripts/meta_are/provision_gaia2_scenarios.py ...
uv run --extra meta-are-setup python scripts/meta_are/provision_demo_filesystem.py ...

3. Configure providers and run

export OPENAI_API_KEY="..."
export ANTHROPIC_API_KEY="..."
cd ../working_dir
uv run --project ../AutoSaddler \
  python -m autosaddler.v2.cli \
  --config ../AutoSaddler/configs/v2/meta_are_smoke.yaml \
  --run-id "$RUN_ID"

📝 Citation

@misc{park2026autosaddlerautomaticharnessoptimization,
  title={AutoSaddler: Automatic Harness Optimization with Durable Updates from Agent Execution Traces},
  author={Sungho Park and Wonjoong Kim and Rongyuan Tan and Jue Zhang and Wook-Shin Han and Pengfei Gao and Chanyoung Park and Yongqiang Yao and Rao Fu and Elsie Nallipogu and Qingwei Lin and Saravan Rajmohan and Dongmei Zhang},
  year={2026},
  eprint={2608.23041},
  archivePrefix={arXiv},
  primaryClass={cs.AI},
  url={https://arxiv.org/abs/2608.23041},
}