Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ Tests are in `tests/` with shared fixtures in `tests/conftest.py`. Supporting ma
- `uv run python scripts/smoke_harness.py --budget 1` — run CLI smoke harness.
- `uv run python scripts/score_check.py` — score regression check.
- `uv run python scripts/score_check.py --update` — update baselines after improvement.
- `uv run python scripts/live_integration.py --phase green --artifact FILE --model openai/gpt-4o-mini --budget 15 --objective "..." --evaluator-command bash evaluators/eval.sh` — GREEN phase optimization.
- `uv run python scripts/live_integration.py --phase green --artifact FILE --model openai/gpt-5.6-sol --budget 15 --objective "..." --evaluator-command bash evaluators/eval.sh` — GREEN phase optimization.

## Coding Style & Naming Conventions
Target Python is `>=3.10`. Follow existing style:
Expand Down
14 changes: 9 additions & 5 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,20 +10,20 @@ uv run pytest
uv run optimize-anything --help

# Optimize
uv run optimize-anything optimize seed.txt --evaluator-command bash eval.sh --model openai/gpt-4o-mini --objective "Improve quality"
uv run optimize-anything optimize seed.txt --evaluator-command bash eval.sh --model openai/gpt-5.6-sol --objective "Improve quality"

# Generate evaluator (default type: judge)
uv run optimize-anything generate-evaluator seed.txt --objective "Score quality" > eval.py

# Score one artifact
uv run optimize-anything score artifact.txt --judge-model openai/gpt-4o-mini --objective "Score clarity"
uv run optimize-anything score artifact.txt --judge-model openai/gpt-5.6-luna --objective "Score clarity"

# Analyze for quality dimensions
uv run optimize-anything analyze artifact.txt --judge-model openai/gpt-4o-mini --objective "Quality"
uv run optimize-anything analyze artifact.txt --judge-model openai/gpt-5.6-luna --objective "Quality"

# Validate across providers
uv run optimize-anything validate artifact.txt \
--providers openai/gpt-4o-mini anthropic/claude-sonnet-4-5 google/gemini-2.0-flash \
--providers openai/gpt-5.6-luna anthropic/claude-sonnet-5 gemini/gemini-3.6-flash \
--objective "Score quality" \
--intake-file intake.json
```
Expand Down Expand Up @@ -70,18 +70,22 @@ uv run optimize-anything validate artifact.txt \
Notes:
- Exactly one evaluator source: `--evaluator-command` OR `--evaluator-url` OR `--judge-model`.
- Early stop auto-activates when budget > 30.
- The proposer defaults to `openai/gpt-5.6-sol` after checking `--model` and `OPTIMIZE_ANYTHING_MODEL`; generated judge and composite evaluators default to `openai/gpt-5.6-luna`.
- LLM judge calls use each provider's sampling defaults unless an explicit temperature is supplied.

## Validate subcommand

`validate` runs LLM-judge scoring for one artifact across 2+ providers and reports mean/stddev/min/max.

```bash
uv run optimize-anything validate runs/run-20260303-130000/best_artifact.txt \
--providers openai/gpt-4o-mini anthropic/claude-sonnet-4-5 google/gemini-2.0-flash \
--providers openai/gpt-5.6-luna anthropic/claude-sonnet-5 gemini/gemini-3.6-flash \
--objective "Score clarity, constraints, and robustness" \
--intake-file intake.json
```

Gemini uses LiteLLM's `gemini/` provider prefix.

## Plugin structure

```text
Expand Down
32 changes: 17 additions & 15 deletions EXAMPLES.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ Optimization output examples should follow this structure:
```bash
optimize-anything optimize parse_duration.py \
--evaluator-command bash eval_tests.sh \
--model openai/gpt-4o-mini \
--model openai/gpt-5.6-sol \
--objective "Pass all duration parsing tests" \
--budget 20
```
Expand All @@ -61,7 +61,7 @@ Sample output excerpt:
```bash
optimize-anything optimize error_template.txt \
--evaluator-url http://localhost:8080/evaluate \
--model openai/gpt-4o-mini \
--model openai/gpt-5.6-sol \
--objective "Make API error messages clear and actionable" \
--budget 15
```
Expand All @@ -72,9 +72,9 @@ optimize-anything optimize error_template.txt \

```bash
optimize-anything optimize support_prompt.txt \
--judge-model openai/gpt-4o-mini \
--judge-model openai/gpt-5.6-luna \
--objective "Improve clarity, constraints, and tone" \
--model openai/gpt-4o-mini \
--model openai/gpt-5.6-sol \
--budget 30
```

Expand All @@ -84,10 +84,10 @@ optimize-anything optimize support_prompt.txt \

```bash
optimize-anything optimize prompt.txt \
--judge-model openai/gpt-4o-mini \
--judge-model openai/gpt-5.6-luna \
--objective "Generalize across user intents" \
--dataset data/train.jsonl \
--model openai/gpt-4o-mini \
--model openai/gpt-5.6-sol \
--budget 120 \
--workers 6 \
--cache --run-dir runs
Expand All @@ -101,11 +101,11 @@ With validation set:

```bash
optimize-anything optimize prompt.txt \
--judge-model openai/gpt-4o-mini \
--judge-model openai/gpt-5.6-luna \
--objective "Generalize to unseen examples" \
--dataset data/train.jsonl \
--valset data/val.jsonl \
--model openai/gpt-4o-mini \
--model openai/gpt-5.6-sol \
--budget 150 \
--cache --cache-from runs/run-20260303-120000 \
--run-dir runs
Expand All @@ -117,21 +117,23 @@ optimize-anything optimize prompt.txt \

```bash
optimize-anything validate runs/run-20260303-130000/best_artifact.txt \
--providers openai/gpt-4o-mini anthropic/claude-sonnet-4-5 google/gemini-2.0-flash \
--providers openai/gpt-5.6-luna anthropic/claude-sonnet-5 gemini/gemini-3.6-flash \
--objective "Score clarity, correctness, and robustness" \
--intake-file intake.json
```

Gemini uses LiteLLM's `gemini/` provider prefix.

Sample output excerpt:

```json
{
"artifact_file": "runs/run-20260303-130000/best_artifact.txt",
"objective": "Score clarity, correctness, and robustness",
"providers": [
{"provider": "openai/gpt-4o-mini", "score": 0.82, "reasoning": "..."},
{"provider": "anthropic/claude-sonnet-4-5", "score": 0.79, "reasoning": "..."},
{"provider": "google/gemini-2.0-flash", "score": 0.81, "reasoning": "..."}
{"provider": "openai/gpt-5.6-luna", "score": 0.82, "reasoning": "..."},
{"provider": "anthropic/claude-sonnet-5", "score": 0.79, "reasoning": "..."},
{"provider": "gemini/gemini-3.6-flash", "score": 0.81, "reasoning": "..."}
],
"mean": 0.8066666667,
"stddev": 0.0152752523,
Expand All @@ -147,8 +149,8 @@ Sample output excerpt:
```bash
optimize-anything optimize --no-seed \
--objective "Draft a concise support policy prompt" \
--model openai/gpt-4o-mini \
--judge-model openai/gpt-4o-mini \
--model openai/gpt-5.6-sol \
--judge-model openai/gpt-5.6-luna \
--budget 25
```

Expand All @@ -161,7 +163,7 @@ optimize-anything optimize --no-seed \
```bash
optimize-anything optimize strategy.md \
--evaluator-command bash eval_unbounded.sh \
--model openai/gpt-4o-mini \
--model openai/gpt-5.6-sol \
--objective "Maximize reward" \
--score-range any
```
Expand Down
2 changes: 1 addition & 1 deletion PROTOCOL.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ Protocol v2 extends the input payload with optional metadata and example context
{
"_protocol_version": 2,
"candidate": "<text>",
"task_model": "openai/gpt-4o-mini",
"task_model": "openai/gpt-5.6-luna",
"example": {
"...": "dataset example object"
}
Expand Down
30 changes: 18 additions & 12 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,9 +18,9 @@ optimize-anything generate-evaluator seed.txt \

# 4) Optimize
optimize-anything optimize seed.txt \
--judge-model openai/gpt-4o-mini \
--judge-model openai/gpt-5.6-luna \
--objective "Improve clarity and specificity" \
--model openai/gpt-4o-mini \
--model openai/gpt-5.6-sol \
--budget 20 \
--workers 4 \
--cache \
Expand All @@ -30,6 +30,10 @@ optimize-anything optimize seed.txt \

CLI stdout returns a JSON summary — see [Result Contract](#result-contract) for the full shape.

### Model defaults

The proposer defaults to `openai/gpt-5.6-sol` after checking `--model` and `OPTIMIZE_ANYTHING_MODEL`. Generated judge and composite evaluators default to `openai/gpt-5.6-luna`. LLM judge calls use each provider's sampling defaults unless an explicit temperature is supplied.

## How It Works

optimize-anything runs a GEPA (Guided Evolutionary Prompt Algorithm) loop: propose → evaluate → reflect, repeating until budget is exhausted or early stopping kicks in.
Expand All @@ -55,11 +59,11 @@ Use `--dataset` for multi-task optimization (one evaluator call per example). Ad

```bash
optimize-anything optimize prompt.txt \
--judge-model openai/gpt-4o-mini \
--judge-model openai/gpt-5.6-luna \
--objective "Generalize across customer request types" \
--dataset data/train.jsonl \
--valset data/val.jsonl \
--model openai/gpt-4o-mini \
--model openai/gpt-5.6-sol \
--budget 120 --workers 6 --cache --run-dir runs
```

Expand All @@ -69,20 +73,22 @@ Cross-check one artifact with multiple judge providers:

```bash
optimize-anything validate result.txt \
--providers openai/gpt-4o-mini anthropic/claude-sonnet-4-5 google/gemini-2.0-flash \
--providers openai/gpt-5.6-luna anthropic/claude-sonnet-5 gemini/gemini-3.6-flash \
--objective "Score clarity, constraints, and robustness" \
--intake-file intake.json
```

Gemini uses LiteLLM's `gemini/` provider prefix.

### Seedless mode

No seed file required; GEPA bootstraps from objective.

```bash
optimize-anything optimize --no-seed \
--objective "Draft a concise, testable API prompt" \
--model openai/gpt-4o-mini \
--judge-model openai/gpt-4o-mini
--model openai/gpt-5.6-sol \
--judge-model openai/gpt-5.6-luna
```

`--no-seed` requires both `--objective` and `--model`.
Expand Down Expand Up @@ -110,7 +116,7 @@ to copying the prior fitness cache.
```bash
optimize-anything optimize seed.txt \
--evaluator-command bash eval.sh \
--model openai/gpt-4o-mini \
--model openai/gpt-5.6-sol \
--budget 150 \
--cache --cache-from runs/run-20260303-120000 \
--run-dir runs \
Expand All @@ -126,7 +132,7 @@ For command/HTTP evaluators:
```bash
optimize-anything optimize seed.txt \
--evaluator-command bash eval.sh \
--model openai/gpt-4o-mini \
--model openai/gpt-5.6-sol \
--score-range any
```

Expand Down Expand Up @@ -226,7 +232,7 @@ The plugin includes three skills that Claude Code can invoke automatically:
/optimize-anything:quick prompt.txt "improve clarity and specificity"
→ runs analyze + optimize with sensible defaults, shows diff

/optimize-anything:validate result.txt --providers openai/gpt-4o anthropic/claude-sonnet-4-5
/optimize-anything:validate result.txt --providers openai/gpt-5.6-luna anthropic/claude-sonnet-5 gemini/gemini-3.6-flash
→ cross-checks the result with multiple judges
```

Expand Down Expand Up @@ -278,12 +284,12 @@ Pass it inline or from a file:
# Inline
optimize-anything optimize seed.txt \
--intake-json '{"quality_dimensions": ["clarity", "specificity"], "hard_constraints": ["max 100 words"]}' \
--judge-model openai/gpt-4o-mini
--judge-model openai/gpt-5.6-luna

# From file
optimize-anything optimize seed.txt \
--intake-file intake.json \
--judge-model openai/gpt-4o-mini
--judge-model openai/gpt-5.6-luna
```

`optimize-anything intake` normalizes and validates these keys:
Expand Down
18 changes: 10 additions & 8 deletions WALKTHROUGH.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,17 +57,17 @@ echo '{"_protocol_version":2,"candidate":"test"}' | python evaluators/eval.py

```bash
uv run optimize-anything score seed.txt \
--judge-model openai/gpt-4o-mini \
--judge-model openai/gpt-5.6-luna \
--objective "Score clarity and constraints"
```

## Step 6: Optimize

```bash
uv run optimize-anything optimize seed.txt \
--judge-model openai/gpt-4o-mini \
--judge-model openai/gpt-5.6-luna \
--objective "Improve clarity and specificity" \
--model openai/gpt-4o-mini \
--model openai/gpt-5.6-sol \
--budget 40 \
--cache --run-dir runs --diff
```
Expand All @@ -80,28 +80,30 @@ Notes:

```bash
uv run optimize-anything score runs/run-<TIMESTAMP>/best_artifact.txt \
--judge-model anthropic/claude-sonnet-4-5 \
--judge-model anthropic/claude-sonnet-5 \
--objective "Score clarity, constraints, and usefulness"
```

## Step 8: Validate with multiple providers (new `validate` flow)

```bash
uv run optimize-anything validate runs/run-<TIMESTAMP>/best_artifact.txt \
--providers openai/gpt-4o-mini anthropic/claude-sonnet-4-5 google/gemini-2.0-flash \
--providers openai/gpt-5.6-luna anthropic/claude-sonnet-5 gemini/gemini-3.6-flash \
--objective "Score clarity, constraints, and robustness"
```

Gemini uses LiteLLM's `gemini/` provider prefix.

## Step 9: Iterate (dataset/generalization optional)

For multi-task optimization:

```bash
uv run optimize-anything optimize seed.txt \
--judge-model openai/gpt-4o-mini \
--judge-model openai/gpt-5.6-luna \
--objective "Generalize across support scenarios" \
--dataset data/train.jsonl --valset data/val.jsonl \
--model openai/gpt-4o-mini \
--model openai/gpt-5.6-sol \
--budget 120 --cache --run-dir runs
```

Expand All @@ -114,7 +116,7 @@ evaluators, cap concurrency with workers:
uv run optimize-anything optimize seed.txt \
--evaluator-command bash evaluators/eval.sh \
--objective "Improve quality" \
--model openai/gpt-4o-mini \
--model openai/gpt-5.6-sol \
--budget 100 \
--workers 8
```
Expand Down
6 changes: 3 additions & 3 deletions commands/analyze.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,14 +10,14 @@ Use an LLM to discover relevant quality dimensions for a given artifact and opti
## Usage

```bash
optimize-anything analyze SEED_FILE --judge-model openai/gpt-4o-mini --objective "Quality"
optimize-anything analyze SEED_FILE --judge-model openai/gpt-5.6-luna --objective "Quality"
```

## Example

```bash
optimize-anything analyze my-prompt.txt \
--judge-model openai/gpt-4o-mini \
--judge-model openai/gpt-5.6-luna \
--objective "Score for clarity and persuasiveness"
```

Expand All @@ -26,7 +26,7 @@ After dimension discovery, proceed directly to optimization using the returned `

```bash
optimize-anything optimize my-prompt.txt \
--judge-model openai/gpt-4o-mini \
--judge-model openai/gpt-5.6-luna \
--objective "Score for clarity and persuasiveness" \
--intake-json '<paste intake_json from analyze>' \
--budget 50 --diff --run-dir runs/ --early-stop
Expand Down
4 changes: 2 additions & 2 deletions commands/quick.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,10 +9,10 @@ Run a no-questions-asked fast optimization.

## Behavior (do not ask follow-up questions)
1. Run analysis to discover dimensions:
- `optimize-anything analyze <file> --judge-model openai/gpt-4o-mini --objective "<objective>"`
- `optimize-anything analyze <file> --judge-model openai/gpt-5.6-luna --objective "<objective>"`
- If analyze fails, skip dimension discovery and run optimize with `--judge-model` directly using the objective as-is.
2. Run optimization using LLM judge with:
- `--judge-model openai/gpt-4o-mini`
- `--judge-model openai/gpt-5.6-luna`
- `--budget 50`
- `--diff`
- `--early-stop`
Expand Down
4 changes: 2 additions & 2 deletions commands/score.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ Score a single artifact file using a command evaluator, HTTP evaluator, or LLM j

```
optimize-anything score SEED_FILE --evaluator-command bash eval.sh
optimize-anything score SEED_FILE --judge-model openai/gpt-4o-mini --objective "Score clarity"
optimize-anything score SEED_FILE --judge-model openai/gpt-5.6-luna --objective "Score clarity"
```

## Example
Expand All @@ -21,7 +21,7 @@ optimize-anything score my-prompt.txt \
--evaluator-command bash evaluators/clarity.sh

optimize-anything score my-prompt.txt \
--judge-model openai/gpt-4o-mini \
--judge-model openai/gpt-5.6-luna \
--objective "Score for persuasiveness"
```

Expand Down
Loading
Loading