optiflow

Observe first. Prove relationships. Plan safely.

optiflow is a local-first Rust CLI for inventorying messy media collections, proving byte-identical duplicate groups, calculating reclaimable storage, and producing immutable review plans.

The published v0.1.1 product is read-only with respect to source media. Current development source adds approved exact-duplicate quarantine on Linux and an unvalidated macOS/APFS same-volume checkpoint. It cannot permanently delete, replace, or optimize source files.

Current capabilities

The latest published release is v0.1.1, from source b82599a, published and independently verified on 2026-09-26. All three native targets passed the read-only external-drive pilot. The verification record binds the signed bundle, qualification receipts, and independent Linux installation. The capabilities below describe that release's source and its documented CLI/library boundaries. Future main changes are not automatically part of this immutable release; v0.1.0 predates the later read-only additions.

• Scan one or more files and directories without modifying them.
• Exclude hidden trees, symbolic links, filesystem crossings, and optiflow's own state directory by default.
• Classify files by inspected content rather than filename extension.
• Collect optional container, stream, codec, dimensions, duration, sample-rate, channel, and bitrate metadata through ffprobe JSON.
• Derive deterministic read-only lossless-PNG review opportunities from current, provider-bound evidence without creating outputs or estimating savings.
• Persist observations and reusable analysis in a local SQLite database.
• Narrow exact-duplicate candidates by byte length before complete hashing.
• Calculate complete streaming BLAKE3-256 hashes only for size candidates.
• Produce exact-duplicate groups with transparent evidence.
• Calculate potential reclaimable storage without deleting anything.
• Publish schema-versioned run, report, and plan artifacts as coherent, marker-sealed sets.
• Generate deterministic review plans with apply-time safety preconditions.
• Emit human-readable or stable JSON output for flow and other pipelines.
• Load explicitly selected, operator-locked extension manifests for typed inspectors, analyzers, policy contributors, planners, validators, report/export providers, and read-only lifecycle observers.

Development addition: approved execution and bounded quarantine

Current source adds explicit execution plans, approval records and apply --dry-run for selected exact duplicates. It rechecks file identity, metadata, full hashes, direct byte equality and capacity, and saves versioned validation/recovery evidence. Bounded quarantine adds a separate v2 journal and an approved mutation path. The macOS/APFS checkpoint is limited to same-volume moves and recovery, refuses cross-volume plans pending a safe ACL metadata backend, and still needs native qualification. These features are not in the published v0.1.1 bundle. New development runs have operator-initiated status, resume, restore and owned cleanup with append-only v3 recovery evidence; older v2 runs remain inspection-only. Explicitly selected, restored cross-filesystem quarantine copies may be previewed and separately authorized for irreversible finalization under additive v4 evidence on Linux only. This is absent from v0.1.1 and never applies to the restored original source.

The draft also authors a standalone macOS copy-property review contract for explicit ACL and metadata declarations. It does not observe or copy files, change historical recovery hashes, or enable cross-volume execution. Its synthetic checks and the native backend qualification remain unrun.

Development addition: source-preserving PNG candidates

The #94 implementation on main adds a separately installed, exact-version OxiPNG v10.2.1 adapter for bounded, static noninterlaced 8-bit RGB/RGBA PNG candidates. It validates source and candidate bytes independently, publishes a private candidate/evidence set, and inspects or discards incomplete staging after an interruption. It neither replaces nor deletes the source. Peak provider memory and provider workdir usage are unmeasured; the signed v0.1.1 release contains none of this path.

Development addition: video candidate planning

#146 starts with a pure FFmpeg recipe planner for one declared video subset. It distinguishes stream-copy from explicitly configured lossy encoding and records what must still be validated. The synthetic request and input recipe are development aids; no video optimizer CLI, provider execution, candidate publication or replacement is implemented by this checkpoint.

The follow-on video provider observation adds read-only source/provider binding and richer FFprobe-reported facts, plus an internal untrusted encoder-capture primitive. It is unqualified development source. Provider memory/scratch enforcement, complete candidate validation and video publication remain unavailable; no video optimizer CLI is exposed.

The next resource checkpoint adds output-cap termination, nonblocking pipe cleanup and one shared observation deadline on Linux/macOS. It removes the runner's extra stdin copy. This code is unrun; provider memory and total scratch quotas remain outstanding.

Media capability matrix

The published v0.1.1 release does not produce optimized media or apply replacements. The #94 development implementation produces source-preserving PNG candidates only. Inventory, optional probing, review evidence, candidate validation, production and replacement are separate capabilities.

Legend: ✓ merged implementation; ~ conditional or limited subset; P planned only; — no implementation or selected format-specific profile. Each status links to its evidence or precise roadmap boundary.

Files	Inventory / exact dedup	Optional probe	Review evidence	Candidate validation	Candidate production	Apply / replace	Maturity / evidence
Any regular file (baseline)	✓	—	—	—	—	P	Format-independent inventory
PNG	✓	~	~	~	P	P	Released review/validator; #94 production in development
JPEG	✓	~	—	P	P	P	Probe eligible; encoding planned
GIF	✓	~	—	—	—	P	No GIF producer selected
SVG	✓	—	—	—	—	P	Inventory only
WebP	✓	~	—	P	P	P	Delivery-format evaluation
AVIF / HEIF	✓	~	—	P	P	P	AVIF evaluation; HEIF inventory/probe
JPEG XL	✓	~	—	P	P	P	Delivery-format evaluation
TIFF, RAW, other images	✓	~	—	—	—	P	Recognition varies; preserve RAW
Audio	✓	~	—	P	P	P	Stream inspection; transforms planned
Video	✓	~	—	P	P	P	Stream inspection; transforms planned

The baseline row covers arbitrary readable regular files; media rows add the listed conditional capabilities. Classification inspects bytes, not suffixes. The locked infer recognizers only establish classification eligibility; optional ffprobe coverage depends on the exact installed provider and the input. SVG has no built-in image recognizer. RAW and container recognition is partial. P does not mean supported: the modern-format entries are evaluation candidates, and planned apply refers to the shared transaction engine, not a promise of optimization for every row.

PNG review opportunities do not prove that a candidate can be made smaller. The separate byte validator covers static noninterlaced 8-bit RGB/RGBA with a documented metadata subset, exact sample/chunk preservation and strictly smaller encoded output. It does not run during scans. The development #94 producer invokes this validator before publishing a candidate and never claims physical savings. See the capability evidence and optimizer strategy and candidate protocol.

Requirements

• Rust stable with edition 2024 support
• macOS or Linux
• Optional: ffprobe for media stream inventory
• Optional: jq for the repository smoke test

On macOS:

brew install ffmpeg jq rust

Quick start

cargo build --release

./target/release/optiflow doctor

./target/release/optiflow scan "/path/to/Media"

The scan prints its run identifier and writes immutable artifacts beneath the local state directory:

runs/<run-id>/
├── effective-policy.json
├── run.json
└── report.json

Generate a separate, review-only exact-duplicate plan:

./target/release/optiflow plan exact-duplicates \
  --run "<run-id>"

The generated plan-exact-duplicates.json explicitly declares "mutates_files": false. Its lexicographically first keep_path is only a stable review default; it is not presented as the objectively best copy.

Commands

optiflow doctor
optiflow scan <inputs...>
optiflow report <run-or-report-path>
optiflow plan exact-duplicates --run <run-or-report-path>
optiflow cache status
optiflow config validate
optiflow config show
optiflow config explain <setting>
optiflow extensions list --manifest <FILE> --lock <FILE>
optiflow extensions inspect <EXTENSION_ID> --manifest <FILE> --lock <FILE>
optiflow extensions doctor --manifest <FILE> --lock <FILE>

The #94 development branch additionally exposes the source-preserving candidate commands (absent from the v0.1.1 release):

optiflow candidate png --source <FILE> --oxipng <FILE>
optiflow candidate status --id <UUID>
optiflow candidate recover --id <UUID>

Global options:

--state-directory <DIRECTORY>  Override persistent local state
--config <FILE>                Select one explicit configuration file
--no-config                    Disable configuration-file loading
--output-format <FORMAT>       Select human or JSON command-result output
--json                         Compatible alias for --output-format json

Scan policy options:

--follow-symlinks      Follow symbolic links
--no-follow-symlinks   Explicitly keep link following disabled
--include-hidden       Include hidden files and directories
--exclude-hidden       Explicitly exclude hidden paths
--cross-filesystems    Cross filesystem boundaries
--stay-on-filesystem   Explicitly preserve filesystem boundaries
--probe                Explicitly enable optional ffprobe inspection
--no-probe             Skip optional ffprobe metadata extraction

The defaults are intentionally conservative for external drives and large archives.

Configuration follows compiled defaults < user file < nearest project optiflow.toml < recognized environment < explicit CLI. See Configuration and effective policy for the strict optiflow.config.v1 schema, exact locations and variables, supported settings, path rules, provenance, fingerprints, and locked invariants.

JSON pipeline contract

Every JSON invocation emits one optiflow.command-result.v1 envelope. The envelope carries the typed outcome, exact process exit code, coverage, diagnostics, committed artifact references, and domain result. Its schema is schemas/command-result.schema.json.

Machine results and committed domain artifacts have independent identifiers:

Artifact	Identifier	Schema
Command result	optiflow.command-result.v1	schemas/command-result.schema.json
Configuration	optiflow.config.v1	schemas/config-v1.schema.json
Effective policy	optiflow.effective-policy.v1	schemas/effective-policy-v1.schema.json
Artifact set	optiflow.artifact-set.v1	schemas/artifact-set-v1.schema.json
Run	optiflow.run.v5	schemas/run.schema.json
Report	optiflow.report.v6	schemas/report.schema.json
Plan	optiflow.plan.v5	schemas/plan.schema.json
Media profile evidence	optiflow.media-profile-evidence.v1	schemas/media-profile-evidence-v1.schema.json
Extension manifest	optiflow.extension-manifest.v1	schemas/extension-manifest-v1.schema.json
Extension lock	optiflow.extension-lock.v1	schemas/extension-lock-v1.schema.json
Extension invocation	optiflow.extension-invocation.v1	schemas/extension-invocation-v1.schema.json
Extension result	optiflow.extension-result.v1	schemas/extension-result-v1.schema.json
PNG candidate evidence (development #94)	optiflow.png-candidate-evidence.v1	schemas/png-candidate-evidence-v1.schema.json
PNG candidate artifact set (development #94)	optiflow.png-candidate-artifact-set.v1	schemas/png-candidate-artifact-set-v1.schema.json

Use --output-format json (or --json) for subprocess integration. JSON owns stdout; human primary output uses stdout and human diagnostics use stderr. Complete success is 0, while a valid result with reduced coverage is 3. See CLI outcome contract for the complete stable exit matrix, stream rules, signal behavior, and shell examples.

State locations

• macOS: ~/Library/Application Support/optiflow
• Linux: $XDG_STATE_HOME/optiflow or ~/.local/state/optiflow
• Override: --state-directory or OPTIFLOW_STATE_DIRECTORY

The primary database stays on a local filesystem. optiflow uses SQLite's rollback journal rather than assuming WAL-safe behavior on removable or network volumes.

Safety boundary

An exact group in v0.1.0 means equal byte length plus an equal complete BLAKE3 content hash. Before a future destructive operation, every plan precondition requires optiflow to:

1. Re-read filesystem metadata.
2. Reject files changed since planning.
3. Recalculate the complete BLAKE3 hash.
4. Perform byte-for-byte confirmation.
5. Refuse mutation if any precondition fails.

See Safety Model for the complete invariant set. The safe extension SDK defines the separate provider declaration, operator trust, deterministic resolution, and core result- acceptance boundaries. Extensions receive no source mutation, destructive, signing, or publication authority. The artifact-set commit protocol defines coherent scan/plan publication, marker validation, and crash recovery. The handle-bound observation protocol documents how OptiFlow rejects replacement and in-read races without mixing evidence. The media-profile evidence contract defines the first lossless-PNG review profile, exact provider provenance, limitation semantics, and why an opportunity is neither an output nor a savings guarantee.

Ecosystem integration

optiflow is independently installable and useful as a standalone CLI. In the Ego Hygiene suite, flow is the unified orchestration facade: it invokes optiflow through the CLI and consumes the versioned JSON artifacts. Sibling tools do not embed optiflow's source or take an unversioned Rust dependency on its default branch.

Development

task validate
task performance:check

Equivalent commands:

cargo fmt --all -- --check
cargo clippy --locked --all-targets --all-features -- -D warnings
cargo test --locked --all-targets
task contracts
task extensions:examples
./scripts/smoke-test.sh
python3 scripts/run-performance-baseline.py \
  --binary "target/release/optiflow" \
  --budgets "performance/budgets-v1.json" \
  --output "target/performance-baseline.json" \
  --enforce

The end-to-end test uses synthetic files in a temporary directory, includes spaces and Unicode in paths, and verifies that both duplicate inputs remain unchanged after scanning and planning.

The performance baseline uses a separate synthetic fixture to enforce tolerant release-mode ceilings for cold discovery, complete candidate hashing, warm-cache reuse, peak resident memory, and committed artifact size. It retains three independent trials, uses median wall time and worst-case resource/artifact evidence, and never reads user media or invokes optional probes.

Dependency admission, security reporting, supported binary targets, signed release verification, and immutable rollback are defined in the release and dependency policy and security policy.

Documentation

The product documentation is built with Zensical from the checked-in Markdown under docs/. Its Python environment is pinned independently from the Rust crate with uv.lock.

task docs:serve
task docs:build
task site:serve

The strict documentation build validates internal links and anchors. The site build generates an architecture portal from the canonical root corpus, then composes it with the LaunchKit-derived landing source, Zensical output, and canonical JSON Schemas into a collision-checked dist/ artifact. See Site publication architecture for the boundary between the landing page, architecture, documentation, generated API reference, schemas, repository intelligence, and release guidance.

Project documents

• Complete architecture document inventory
• Architecture portal source and generation contract
• Purpose, vision, and principles
• Architecture
• MVP specification
• Safety model
• Artifact-set commit protocol
• Handle-bound observation protocol
• State model
• JSON contract
• Media-profile evidence
• Media capabilities and optimizer strategy
• CLI outcome contract
• Configuration and effective policy
• Safe extension SDK
• Generated extension capability reference
• Development model
• Performance budgets
• Release and dependency policy
• Operator-ready release path
• Security policy
• Site publication architecture
• Cloud-native placement
• Roadmap

License

MIT