SAM 2.1 Hiera Small Intel® OpenVINO AITK Recipe

[Anirudh-Swaminathan]

This commit contains the SAM 2.1 Hiera Small Intel® OpenVINO AITK Recipe for generating ONNX Encapsulated OVIR models

[Copilot]

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

This PR adds an Intel® OpenVINO-based Olive workflow for optimizing and running SAM 2.1 Hiera Small (vision encoder + mask decoder), plus utilities and docs updates to support OpenVINO/WinML execution provider registration and inference samples.

Changes:

• Add SAM 2.1 OpenVINO Olive workflows (configs + Python scripts) including calibration data preparation and ONNX encapsulated OVIR generation.
• Add OpenVINO/WinML EP registration helper and inference utilities + notebooks for running OVIR models in ONNX Runtime.
• Update repository documentation/config indices to surface the new OpenVINO recipes and runtime support.

Reviewed changes

Copilot reviewed 33 out of 34 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
sd-legacy-stable-diffusion-v1-5/aitk/sd_utils/ov.py	Adds a helper to register ORT execution providers via a WinML worker script.
sam2.1-hiera-small/aitk/sam2_ov_workflow.py	Adds a one-shot script to run data prep + both Olive workflows.
sam2.1-hiera-small/aitk/sam2_ov_workflow.json.config	Adds AITK template metadata for the OpenVINO conversion workflow.
sam2.1-hiera-small/aitk/sam2_ov_workflow.json	Adds Olive workflow wrapper config that runs the AitkPython script.
sam2.1-hiera-small/aitk/sam21_vision_encoder_ov.json	Adds OpenVINO conversion + quantization + encapsulation recipe for the vision encoder.
sam2.1-hiera-small/aitk/sam21_mask_decoder_ov.json	Adds OpenVINO conversion + quantization + encapsulation recipe for the mask decoder.
sam2.1-hiera-small/aitk/prep_ov_quant_data.py	Adds COCO128 download/extract and decoder input generation for calibration.
sam2.1-hiera-small/aitk/ov_model_utils.py	Adds PyTorch wrapper modules + example inputs for exporting/quantizing SAM2.1 submodels.
sam2.1-hiera-small/aitk/ov_inference_utils.py	Adds ORT inference utilities for prompted segmentation using encoder+decoder OVIR models.
sam2.1-hiera-small/aitk/ov_encoder_quant.py	Adds Olive dataset + transform for encoder calibration.
sam2.1-hiera-small/aitk/ov_decoder_quant.py	Adds Olive dataset + transform for decoder calibration.
sam2.1-hiera-small/aitk/model_project.config	Registers the new OpenVINO workflow template in the project.
sam2.1-hiera-small/aitk/info.yml	Exposes an OpenVINO recipe entry and marks status as Ready.
sam2.1-hiera-small/aitk/inference_sample.ipynb	Updates the inference notebook to support both QNN and OpenVINO paths.
sam2.1-hiera-small/aitk/README.md	Documents OpenVINO quantization + inference environment setup and usage.
sam2.1-hiera-small/aitk/.gitignore	Ignores the downloaded sample image.
sam2.1-hiera-small/OpenVINO/winml.py	Adds WinML-based EP discovery and ORT provider library registration.
sam2.1-hiera-small/OpenVINO/sam2_ov_workflow.py	Adds OpenVINO-specific wrapper script for data prep + Olive runs.
sam2.1-hiera-small/OpenVINO/sam21_vision_encoder_ov.json	Adds OpenVINO recipe for vision encoder in the OpenVINO subfolder.
sam2.1-hiera-small/OpenVINO/sam21_mask_decoder_ov.json	Adds OpenVINO recipe for mask decoder in the OpenVINO subfolder.
sam2.1-hiera-small/OpenVINO/prep_ov_quant_data.py	Adds COCO128 data prep (OpenVINO folder variant).
sam2.1-hiera-small/OpenVINO/ov_model_utils.py	Adds model wrappers (OpenVINO folder variant).
sam2.1-hiera-small/OpenVINO/ov_inference_utils.py	Adds inference utilities (OpenVINO folder variant).
sam2.1-hiera-small/OpenVINO/ov_encoder_quant.py	Adds encoder calibration dataset/transform (OpenVINO folder variant).
sam2.1-hiera-small/OpenVINO/ov_decoder_quant.py	Adds decoder calibration dataset/transform (OpenVINO folder variant).
sam2.1-hiera-small/OpenVINO/info.yml	Adds OpenVINO recipe metadata for the OpenVINO folder.
sam2.1-hiera-small/OpenVINO/inference_sample.ipynb	Adds an OpenVINO-only inference notebook.
sam2.1-hiera-small/OpenVINO/README.md	Documents how to run OpenVINO quantization + inference in the OpenVINO folder.
sam2.1-hiera-small/OpenVINO/.gitignore	Ignores sample image and COCO artifacts in the OpenVINO folder.
README.md	Updates top-level model/recipe listings to include the new SAM2.1 OpenVINO path(s).
.aitk/requirements/requirements-IntelNPU-SAM.txt	Adds dependencies used for SAM2.1 OpenVINO/SAM tooling.
.aitk/docs/guide/ModelList.md	Adds Intel CPU/GPU/NPU options for SAM2.1 Hiera Small.
.aitk/configs/model_list.json	Adds Intel runtime tags for the SAM2.1 Hiera Small model entry.
.aitk/configs/checks.json	Updates check counters to reflect added configs/files.

Comments suppressed due to low confidence (2)

sd-legacy-stable-diffusion-v1-5/aitk/sd_utils/ov.py:1

• register_execution_providers() uses os.path.abspath(...) but os is not imported in the shown module imports, which will raise NameError at runtime. Add an import os (module-level or inside the function) consistent with the file’s import style.
  sam2.1-hiera-small/aitk/sam2_ov_workflow.py:1
• There appear to be two parallel copies of the OpenVINO workflow implementation (under sam2.1-hiera-small/aitk/ and sam2.1-hiera-small/OpenVINO/) with extremely similar scripts/configs. This duplication increases the risk of drift and inconsistent bug fixes. If both entry points are required, consider consolidating the shared implementation into one module and having the other location import/call into it, or generate one set from the other during packaging.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[xieofxie]

Please follow others using Convert to Intel CPU/NPU/GPU

[xieofxie]

Please define ExecutionProvider earlier to pass it into this

[xieofxie]

Add

import sys

NOTEBOOK_DIR = Path(__file__).parent if "__file__" in globals() else Path.cwd()
PROJECT_ROOT = NOTEBOOK_DIR.parents[1]
sys.path.insert(0, str(PROJECT_ROOT))

since the inference_sample will be copied into history folder to run

[xieofxie]

please follow

olive-recipes/sam2.1-hiera-small/aitk/sam2_qnn_workflow.py

Line 33 in 32fd807

oliveJson["cache_dir"] = cache_dir

to

• copy config into history folder
• update cache_dir to a shared cache folder
• update output_dir to save model into history folder

[xieofxie]

To keep it workable from Readme, you could check if the arguments exist or not

olive-recipes/sam2.1-hiera-small/aitk/sam2_qnn_workflow.py

Line 14 in 32fd807

def parse_arguments():