chore: promote main to production

.dockerignore

@@ -1,3 +1,6 @@
+# Local reports, captures and scratch files
+.user/
+
 ### Python template
 # Byte-compiled / optimized / DLL files
 __pycache__/

.gitignore

@@ -1,3 +1,4 @@
+/.user/
 __pycache__
 .pytest_cache
 .DS_Store

.pre-commit-config.yaml

@@ -23,23 +23,24 @@ repos:
 
   # flake8 python linter with all the fixins
   - repo: https://github.com/PyCQA/flake8
-    rev: 6.0.0
+    rev: 7.3.0
     hooks:
       # additional configuration of flake8 and extensions in pyproject.toml
       - id: flake8
         additional_dependencies:
           [
-            flake8-annotations,
-            flake8-bandit,
-            flake8-bugbear,
-            flake8-black,
-            flake8-docstrings,
-            flake8-import-order,
-            darglint,
-            mypy,
-            pycodestyle,
-            pydocstyle,
-            Flake8-pyproject
+            flake8-annotations==3.2.0,
+            flake8-bandit==4.1.1,
+            bandit==1.9.4,
+            flake8-bugbear==24.12.12,
+            flake8-black==0.4.0,
+            black==23.1.0,
+            flake8-docstrings==1.7.0,
+            flake8-import-order==0.19.2,
+            darglint==1.8.1,
+            pycodestyle==2.14.0,
+            pydocstyle==6.3.0,
+            Flake8-pyproject==1.2.4
           ]
 
   # removed unused imports and variables

AGENTS.md

@@ -0,0 +1,104 @@
+# Repository guide
+
+Python 3.11 / Flask service that converts raster images to SVG using NumPy,
+OpenCV, FAISS, scikit-image and Potrace. Requests can return raw SVG or upload it
+to S3. Start with `README.md` for setup and usage.
+
+## Code map
+
+- `vectorizing/__init__.py`: app factory, request validation, cropping and API responses.
+- `vectorizing/solvers/binary/`: binary masks and tracing.
+- `vectorizing/solvers/color/`: quantization, cluster cleanup, ordered masks and tracing.
+- `vectorizing/geometry/` and `vectorizing/svg/`: path conversion, bounds and SVG serialization.
+- `vectorizing/util/`: image loading, color-mode normalization and size limits.
+- `vectorizing/server/`: environment settings, S3, logging and timing.
+- `vectorizing/tests/`: API contracts, image gallery and algorithm regressions.
+- `scripts/`: local infrastructure, dependency generation and offline comparison/benchmark tools.
+
+## Development and checks
+
+Prefer Docker Compose; it supplies Moto S3 and needs no AWS account or `.env` file.
+Run from the repository root:
+
+```sh
+docker compose up --build
+pre-commit run --all-files
+docker compose --profile testing run --build --rm test
+```
+
+For a focused test run:
+
+```sh
+docker compose --profile testing run --build --rm test \
+  python -m pytest vectorizing/tests/test_color_bitmap_rims.py
+```
+
+- The API defaults to `http://localhost:8000` in Compose.
+- Application source is copied into the image, not bind-mounted. Rebuild after edits.
+- Use `docker compose down` to stop services you started; Moto data is discarded.
+  Isolate project names and image tags when testing alongside other sessions/worktrees.
+- Native execution needs the Python dependencies and system libraries from the
+  Dockerfile/dev-container setup. Package imports require `PORT` and `S3_BUCKET`;
+  integration tests also require a distinct `S3_TEST_BUCKET` and a local S3 endpoint.
+- Preserve normal AWS behavior unless `AWS_ENDPOINT_URL` is explicitly configured.
+  Use Moto for validation, not live buckets.
+- CI runs the pre-commit hooks with Python 3.11 and the Compose test command.
+
+## Output and behavior contracts
+
+- Keep smooth anti-aliasing and transparent exteriors. Do not hide seams with SVG
+  strokes or `shape-rendering="crispEdges"`, or reduce normal tracing quality.
+- Color masks use bounded overlaps, not full underpainting. The current rim is
+  two processed-image pixels; zoom enlarges it and very thin features may remain
+  covered. Do not promise universally exact holes and seam-free rendering.
+- Preserve palette/centroid ordering, dtypes and overflow behavior, connectivity,
+  and independent mask storage when optimizing arrays. Rim conversion runs forward
+  because the next mask must still be cumulative.
+- Preserve public defaults, response/error shapes, return containers, SVG paint
+  order and numeric formatting during refactors or performance work.
+- Image readers currently wrap all loading failures, including interrupts, and URL
+  downloads have no timeout. These are tested legacy contracts, not recommendations;
+  change them deliberately with updated tests, never incidentally during lint cleanup.
+- Comments and docstrings should explain behavior and invariants, not edit history.
+
+## Regression evidence
+
+- Add focused tests for changed behavior. Keep reference implementations independent
+  of the optimized code they check.
+- The gallery can auto-create missing PNG baselines. A newly seeded baseline does
+  **not** prove equivalence; do not delete or replace baselines just to make tests pass.
+- For output-preserving changes, capture before editing (or use a trusted baseline
+  checkout), then compare SVG bytes, RGBA pixels, palettes, dimensions and bounds
+  with `scripts/vectorization_outputs.py`. See the README for commands.
+- For layer/geometry changes, check layer deletion, zero alpha and half opacity,
+  transparent backgrounds and thin features, including `aftermath.png`. An assembled
+  render alone cannot establish editability.
+- Use `scripts/benchmark_vectorization.py` for measured performance claims. Retain
+  raw samples, check equivalent outputs, and report runtime/hardware context. Its
+  timing excludes input decoding, HTTP/S3 and rasterization; process peak RSS includes
+  startup and warmups. Local results are not production or concurrent-load guarantees.
+
+## Style, dependencies and artifacts
+
+- Follow `.pre-commit-config.yaml` and `pyproject.toml`: Black, 88 columns, isort,
+  annotations and NumPy-style docstrings. Avoid broad lint suppressions and keep
+  correctness/security checks enabled. Passing lint is not a static-type-check claim.
+- Prefer existing dependencies. Edit `requirements/prod.in` or `requirements/dev.in`,
+  then use `scripts/compile_requirements.sh` and `scripts/compile_envs.sh` when dependency
+  updates are needed. The latter requires the configured Conda `dev` environment.
+  Keep system dependency setup consistent with `scripts/install_system_dependencies.sh`
+  and the Dockerfile.
+- Keep generated captures, benchmarks, investigations and scratch source under `.user/`
+  or temporary directories, not tracked code. `.user/` is excluded from Git and Docker.
+  Historical notes there may be stale; verify them against current source.
+- Keep visual evidence local unless upload is explicitly requested. Never include
+  credentials in reports or commits.
+
+## Pull requests
+
+Keep descriptions short and plain-language. Fetch `.github/pull_request_template.md`
+from the `Kittl/.github` repository rather than inventing a replacement.
+Leave owner attestations unchecked without explicit approval. Do not add agent/tool
+badges. After addressing human review feedback, reply but leave the thread open
+unless the user explicitly asks to resolve it; uncertain reviewer identity counts
+as human.

README.md

@@ -53,6 +53,39 @@ pre-commit run --all-files
 The first execution might take a bit longer, as it will set up the virtual environment
 where the linter and the formatter will run.
 
+## Output comparisons and benchmarks
+
+These developer commands use local fixtures and need no AWS credentials or Moto.
+Run them in the dev container or an environment with `requirements/dev.txt` and
+its system libraries installed. They do not change application code or dependencies.
+
+Capture each version into a different directory, then compare:
+
+```bash
+python scripts/vectorization_outputs.py capture .user/before --source-root /path/to/baseline
+python scripts/vectorization_outputs.py capture .user/after
+python scripts/vectorization_outputs.py compare .user/before .user/after
+python scripts/benchmark_vectorization.py --output .user/benchmark.json --case aftermath-16
+```
+
+- Repeat `--case` to select fixtures; `--help` lists them. The default is all 15 cases.
+- Comparison checks SVG bytes, decoded RGBA pixels, colors, dimensions, bounds and input identity. Exit codes: `0` equal, `1` changed, `2` invalid/incomplete artifacts. Nonempty output directories and existing benchmark reports are never overwritten.
+- Benchmark timing covers the solver, SVG serialization and bounds, not file decoding, HTTP/S3 or rasterization. Each sample uses a fresh process; peak RSS includes imports, input preparation, warmups and output checks. Linux and macOS memory units are normalized to MiB.
+- Add `--baseline-root /path/to/baseline` to benchmark two trusted checkouts in alternating order, with exact-output checks. `--repeats` defaults to 3 and `--warmups` to 1 per process. Output mismatches exit `1` but still save the report; timings have no pass/fail threshold.
+- Reports include runtime/source fingerprints and raw samples. `--label` can record commit IDs. Checkouts share the installed dependencies; this is not a dependency-isolated or production/concurrency benchmark. Use only trusted source roots.
+
+For Docker, build the current scripts and mount only the output directory:
+
+```bash
+mkdir -p .user
+docker build -t vectorizing:tools .
+docker run --rm --user "$(id -u):$(id -g)" -v "$PWD/.user:/results" vectorizing:tools \
+  python scripts/vectorization_outputs.py capture /results/candidate
+```
+
+A separate baseline checkout must also be mounted when using `--source-root` or
+`--baseline-root`. Generated artifacts under `.user/` are excluded from Git and Docker builds.
+
 ## Server
 
 The server has a single endpoint that receives `POST` requests.
@@ -111,6 +144,9 @@ To run tests against the local Moto service, run:
 docker compose --profile testing run --build --rm test
 ```
 
+Tests are split into `test_*.py` files in `vectorizing/tests`. Compose and CI
+run them all. `color_reference.py` keeps the old code for before/after checks.
+
 When running pytest outside Compose, configure the AWS variables for an S3-compatible service first.
 
 ### Required setup

docker-compose.yml

@@ -57,4 +57,4 @@ services:
     depends_on:
       moto-init:
         condition: service_completed_successfully
-    command: python -m pytest vectorizing/tests
+    command: python -m pytest

pyproject.toml

@@ -16,7 +16,12 @@ extend-ignore = [
     'I100',
     'I101',
     'I202',
+    # Black controls whitespace in complex slices.
+    'E203',
+    # The containing class already determines the type of self.
+    'ANN101',
 ]
+per-file-ignores = ['vectorizing/tests/*:S101']
 import-order-style = "google"
 docstring-convention = "numpy"
 strictness = "short"

pytest.ini

@@ -1,3 +1,3 @@
 [pytest]
 pythonpath = .
-python_files = test.py test_*.py *_test.py
+testpaths = vectorizing/tests

scripts/__init__.py

@@ -0,0 +1 @@
+"""Local development and maintenance commands."""

scripts/_vectorization.py

@@ -0,0 +1,143 @@
+"""Shared fixtures and offline solver execution for developer commands."""
+
+import hashlib
+import math
+import os
+import platform
+import sys
+from collections.abc import Callable
+from importlib.metadata import version
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parents[1]
+# Explicit cases keep developer reports independent of pytest's module layout.
+CASES = {
+    "1px_binary": ("1px.jpg", "binary", None),
+    "1px_color": ("1px.jpg", "color", 9),
+    "black_rectangle_binary": ("black_rectangle.png", "binary", None),
+    "black_rectangle_color": ("black_rectangle.png", "color", 5),
+    "bubbles": ("bubbles.png", "color", 5),
+    "geo_logo": ("geo_logo.png", "binary", None),
+    "shapes": ("shapes.png", "color", 4),
+    "shapes_2": ("shapes_2.png", "color", 7),
+    "white_text": ("white_text.png", "binary", None),
+    "aftermath-2": ("aftermath.png", "color", 2),
+    "aftermath-6": ("aftermath.png", "color", 6),
+    "aftermath-16": ("aftermath.png", "color", 16),
+    "aftermath-64": ("aftermath.png", "color", 64),
+    "empty-binary": ("empty.png", "binary", None),
+    "empty-color": ("empty.png", "color", 6),
+}
+
+
+def file_hash(path: Path) -> str:
+    """Hash file bytes without retaining the entire file in memory."""
+    with path.open("rb") as source:
+        return hashlib.file_digest(source, "sha256").hexdigest()
+
+
+def fingerprint(svg: str, details: dict[str, object]) -> dict[str, object]:
+    """Record exact SVG bytes alongside palette, dimensions, bounds and input."""
+    return {"svg_sha256": hashlib.sha256(svg.encode("utf-8")).hexdigest(), **details}
+
+
+def source_hash(source_root: Path) -> str:
+    """Fingerprint runtime Python sources without requiring Git in the container."""
+    digest = hashlib.sha256()
+    for path in sorted((source_root / "vectorizing").rglob("*.py")):
+        relative = path.relative_to(source_root)
+        if "tests" not in relative.parts:
+            digest.update(str(relative).replace(os.sep, "/").encode("utf-8") + b"\0")
+            digest.update(path.read_bytes() + b"\0")
+    return digest.hexdigest()
+
+
+def environment() -> dict[str, object]:
+    """Report the numerical runtime and native thread counts without credentials."""
+    import cv2
+    import faiss
+
+    packages = (
+        "numpy",
+        "Pillow",
+        "scipy",
+        "scikit-image",
+        "faiss-cpu",
+        "opencv-python-headless",
+        "pypotrace",
+        "skia-pathops",
+        "CairoSVG",
+    )
+    return {
+        "python": platform.python_version(),
+        "platform": platform.platform(),
+        "machine": platform.machine(),
+        "cpus": os.cpu_count(),
+        "faiss_threads": faiss.omp_get_max_threads(),
+        "opencv_threads": cv2.getNumThreads(),
+        "versions": {name: version(name) for name in packages},
+    }
+
+
+def prepare_case(
+    name: str,
+    source_root: Path,
+) -> Callable[[], tuple[str, dict[str, object]]]:
+    """Load a fixture once and return a fresh solver/SVG/bounds run for its checkout."""
+    source_root = source_root.resolve()
+    if not (source_root / "vectorizing" / "__init__.py").is_file():
+        raise ValueError(f"Not a vectorizing checkout: {source_root}")
+    sys.path.insert(0, str(source_root))
+    # Package imports require these settings; no server or S3 client is created.
+    os.environ.setdefault("PORT", "8000")
+    os.environ.setdefault("S3_BUCKET", "unused-vectorization-tools")
+
+    import numpy as np
+    from PIL import Image
+
+    import vectorizing
+    from vectorizing.geometry.bounds import compound_paths_bounds
+    from vectorizing.server.timer import Timer
+    from vectorizing.solvers.binary.BinarySolver import BinarySolver
+    from vectorizing.solvers.color.ColorSolver import ColorSolver
+    from vectorizing.svg.markup import generate_SVG_markup
+    from vectorizing.util.read import convert_RGB_A
+
+    if Path(vectorizing.__file__).resolve().parent != source_root / "vectorizing":
+        raise ValueError(
+            "A different checkout is already imported; use a fresh process",
+        )
+    filename, solver, count = CASES[name]
+    path = source_root / "vectorizing" / "tests" / "images" / filename
+    with Image.open(path) as source:
+        image = convert_RGB_A(source).copy()
+    source_info = {
+        "image": filename,
+        "sha256": file_hash(path),
+        "solver": solver,
+        "color_count": count,
+    }
+
+    def run() -> tuple[str, dict[str, object]]:
+        result = (
+            BinarySolver(image).solve()
+            if solver == "binary"
+            else ColorSolver(image, count, Timer()).solve()
+        )
+        paths, colors, width, height = result
+        bounds = compound_paths_bounds(paths)
+        # Empty artwork has infinite bounds; strings keep the report valid JSON.
+        bounds = {
+            key: float(value) if math.isfinite(value) else str(float(value))
+            for key, value in bounds.items()
+        }
+        palette = np.asarray(colors)
+        return generate_SVG_markup(*result), {
+            "input": source_info,
+            "colors": palette.tolist(),
+            "colors_dtype": str(palette.dtype),
+            "dimensions": [width, height],
+            "bounds": bounds,
+        }
+
+    return run