Spice Framework UI

A runnable Axum dashboard template for Spice agent evaluations.

It provides the browser surface around your project-specific Spice suite:

• searchable test catalog with tags and availability
• exact single-case runs and consensus runs
• server-sent progress events
• persisted per-turn Spice traces
• deterministic assertion results
• model-judge score, threshold, and reason
• a distinct Judge unavailable state for provider/authentication errors
• cancellation and single-active-run protection
• local-only listening by default

The repository is a GitHub template. Create a repository from it, replace the demo integration, and keep the server and UI unchanged.

Run the demo

cargo run

Open http://127.0.0.1:3030. The included deterministic weather agent exercises tool calls, traces, assertions, consensus, and judgment without requiring an API key.

To use the built-in Spice JevJudge instead of the deterministic demo judge:

TYPESAFE_API_KEY=... cargo run

Optional listener settings:

SPICE_UI_HOST=127.0.0.1 SPICE_UI_PORT=3030 cargo run

Do not expose the server publicly without adding authentication. Evaluation traces commonly contain prompts, tool arguments, observations, and model output.

Adapt it to your agent

The integration boundary is intentionally one trait in src/project.rs:

#[async_trait]
pub trait EvaluationProject: Send + Sync + 'static {
    fn cases(&self) -> Vec<CaseSummary>;

    async fn run_case(
        &self,
        request: CaseRun,
        progress: ProgressSink,
        cancel: CancellationToken,
    ) -> Result<RunArtifact, String>;
}

1. Copy src/demo.rs to a module for your project.
2. Return your test catalog from cases().
3. In run_case(), build exactly the requested TestCase, configure Runner, install an optional Judge, and run your AgentUnderTest.
4. Set RunnerConfig::trace_dir; load those trace JSON files into RunArtifact::traces so the browser can render complete turns.
5. Forward your runtime's thought/tool/observation events through ProgressSink::emit for live progress. The persisted Spice trace remains the source of truth after completion.
6. Thread the provided CancellationToken into your model and tool runtime.
7. Replace DemoProject in src/main.rs.

The included src/demo.rs is a complete, compiling reference implementation. It also demonstrates how to select JevJudge when TYPESAFE_API_KEY exists and use MockJudge for a credential-free demo.

Result semantics

The UI deliberately distinguishes three outcomes:

• Passed — every assertion and configured judge passed.
• Failed — an assertion or completed judge verdict rejected the run.
• Judge unavailable — deterministic assertions passed, but the judge returned an infrastructure/provider error such as HTTP 401. No semantic verdict was produced, so the UI does not call it a Jev failure or show the framework's placeholder 0.0 as a real score.

Runner and report data are not rewritten. The UI only classifies presentation state; raw reasons and JSON assertion records remain visible.

HTTP surface

Method	Path	Purpose
GET	/	Dashboard
GET	/api/cases	Catalog
POST	/api/runs	Start { "case_id": "...", "once": true }
GET	/api/runs/{id}	Current/final snapshot
GET	/api/runs/{id}/events	SSE snapshots
POST	/api/runs/{id}/cancel	Request cancellation

Only one run executes at a time. This is conservative for agents that share a browser, keyboard, workspace, or local service.

Verify

cargo test

The test suite runs the demo through the real Spice runner and checks that judge provider errors are rendered as unavailable rather than semantic failures.

License

MIT