Experiments

A collection of heterogeneous, reproducible experiments. Each experiment lives in its own folder under experiments/ and documents its hypothesis, setup, methodology, metrics, results, and conclusions.

Experiments

Experiment	Status	Goal
jev-vs-llm	In progress	Compare System One decision models and LLMs on accuracy, calibration, latency, scaling, and end-to-end workflows.
model-capability-benchmark	Implemented / hardening	Compare local and API models with compact task-vertical suites, deterministic evidence, paired statistics, runtime/cost planning, and neutral capability reporting.
vlm-capability-benchmark	Planned	Compare local/open and API vision-language models on document, chart, visual reasoning, grounding, and UI-understanding tasks.
image-generation-benchmark	Planned	Compare image-generation/editing models with reproducible prompt suites, objective checks where meaningful, blind human evaluation, and visual side-by-side reports.
redactguard-local-anonymization	Initial implementation	Compare local Korgis models on RedactGuard PII recall, leakage, over-redaction, robustness, and latency.
diarization-benchmark	Implemented / real-model validation pending	Compare local open speaker-diarization models on DER/JER, speaker-count accuracy, latency and resource efficiency on Apple Silicon.

Multimodal roadmap

The VLM and image-generation tracks share a small multimodal extension of benchmark-core and are designed to be implemented in parallel after that foundation is stable.

See experiments/MULTIMODAL_PARALLEL_PLAN.md.

First real-model execution steps are documented in experiments/MULTIMODAL_SMOKE_RUNBOOK.md.