From f90184da0c3ec1de0d7513def117880d913217e5 Mon Sep 17 00:00:00 2001 From: sumanaga Date: Tue, 29 Sep 2026 13:54:38 +0530 Subject: [PATCH 1/5] docs(stream-density): document TARGET_FPS precedence and measurement window Clarify how the effective target FPS is resolved for each camera: TARGET_FPS overrides every stream, then the camera's targetFps, then its fps, and finally the 14.95 default. Add the MEASUREMENT_WINDOW_SECONDS and INIT_DURATION settle-time controls, including their defaults and the per-window pass/fail step logic, and note that the resolved values are recorded in stream_density.log. Signed-off-by: sumanaga --- .../use-cases/loss-prevention/performance.md | 29 +++++++++++++++++-- 1 file changed, 26 insertions(+), 3 deletions(-) diff --git a/docs_src/use-cases/loss-prevention/performance.md b/docs_src/use-cases/loss-prevention/performance.md index dc3b699..603184f 100644 --- a/docs_src/use-cases/loss-prevention/performance.md +++ b/docs_src/use-cases/loss-prevention/performance.md @@ -148,11 +148,20 @@ make benchmark-stream-density ### Custom Target FPS -Set per-stream target FPS using `lane_config.cameras[].targetFps` in `camera_to_workload_*.json`. +Set the target FPS for each camera using the `targetFps` field in the selected `camera_to_workload_*.json` file. Choose the value based on workload criticality and the source FPS. -Set `targetFps` based on workload criticality and source FPS. +If `targetFps` is not defined for a camera (or is not a positive value), the benchmark uses that camera's `fps` field instead. If neither field provides a positive value, it falls back to the default target FPS, currently `14.95`. -If `targetFps` is not present for a stream (or is invalid/non-positive), the stream falls back to `TARGET_FPS`. +To override the camera configuration, set the `TARGET_FPS` environment variable. When supplied, `TARGET_FPS` applies the same target to every stream and takes precedence over both `targetFps` and `fps`. + +In summary, the effective target FPS for each camera is resolved in this order: + +1. `TARGET_FPS` environment variable, when explicitly supplied (overrides the camera configuration for every stream). +2. The camera's `targetFps` in `camera_to_workload_*.json`. +3. The camera's `fps`, used when `targetFps` is missing or not a positive value. +4. The default target FPS, `14.95`, when neither camera field provides a positive value. + +The resolved target FPS and its source are recorded in `stream_density.log` so benchmark results can be audited. CLI examples: @@ -164,11 +173,25 @@ make TARGET_FPS=13.5 benchmark-stream-density make PIPELINE_SCRIPT=yolo11n_effnetb0.sh TARGET_FPS=13.5 benchmark-stream-density ``` +### Measurement Window & Settle Time + +`MEASUREMENT_WINDOW_SECONDS` controls how long FPS samples are collected for each density step. Its default is **100 seconds**. Each step adds a lane after one passing window, the final count is confirmed by two passing windows in a row, and a lane is only given up after two failing windows in a row. Each window is evaluated independently; windows are not combined into one longer window. + +`INIT_DURATION` sets the settle time, giving the pipelines time to stabilize before measurement starts. A longer settle time lets the pipelines stabilize more, and a longer measurement window can reduce short-term measurement noise, but both increase the total benchmark duration. The resolved settle time is also printed in the stream-density result summary. + +```sh +make benchmark-stream-density \ + INIT_DURATION=120 \ + MEASUREMENT_WINDOW_SECONDS=60 +``` + ### Stream Density Environment Variables | Variable | Description | Values | |:---------|:------------|:--------| | `TARGET_FPS` | Minimum FPS threshold | `14.95` (default), `13.5`, `20.0` | +| `MEASUREMENT_WINDOW_SECONDS` | Duration FPS samples are collected per density step | `100` (default), `60` | +| `INIT_DURATION` | Settle time before measurement starts | `120`, `60` | | `OOM_PROTECTION` | Prevent out-of-memory crashes | `1` (enabled), `0` (disabled) | > ⚠️ **Warning**: Setting `OOM_PROTECTION=0` may crash your system requiring a hard reboot. From 5e92a1d3b34cb2b7dfe56f0f72b91c2eb6ff4d34 Mon Sep 17 00:00:00 2001 From: sumanaga Date: Tue, 29 Sep 2026 16:34:38 +0530 Subject: [PATCH 2/5] docs(stream-density): document TARGET_FPS precedence and measurement window Update the loss-prevention benchmarking docs to match the new stream-density behavior. Explain that the effective target FPS resolves in order: the TARGET_FPS env override for every stream, the camera's targetFps, its fps, then the 14.95 default, and note that the resolved value and source are logged to stream_density.log. Add the MEASUREMENT_WINDOW_SECONDS and INIT_DURATION settle-time controls, including their defaults and the per-window pass/fail step logic, to both performance.md and lp-benchmarking.md. Also tidy table formatting and whitespace in lp-benchmarking.md. Signed-off-by: sumanaga --- .../loss-prevention/lp-benchmarking.md | 190 ++++++++---------- .../use-cases/loss-prevention/performance.md | 15 +- 2 files changed, 87 insertions(+), 118 deletions(-) diff --git a/docs/user-guide/loss-prevention/lp-benchmarking.md b/docs/user-guide/loss-prevention/lp-benchmarking.md index 99f0db9..a47b10c 100644 --- a/docs/user-guide/loss-prevention/lp-benchmarking.md +++ b/docs/user-guide/loss-prevention/lp-benchmarking.md @@ -7,19 +7,16 @@ Test your Loss Prevention and Automated Self-Checkout pipeline performance on va **Goal**: Run a basic performance test to verify your system works correctly ### 1. Initialize Performance Tools - ```bash make update-submodules ``` -### 2. Run Quick Benchmark - +### 2. Run Quick Benchmark ```bash make benchmark-quickstart ``` **What this does:** - - Tests GPU performance with 6 different loss prevention workloads - Runs headless (no display needed) - Uses pre-built Docker images for faster setup @@ -32,27 +29,25 @@ make benchmark-quickstart ### Basic Performance Testing **Default Benchmark Command:** - ```bash make benchmark ``` **Configuration:** - -- Single pipeline instance (`PIPELINE_COUNT=1`) +- Single pipeline instance (`PIPELINE_COUNT=1`) - CPU-only processing (`WORKLOAD_DIST=workload_to_pipeline.json`) - Standard camera setup (`CAMERA_STREAM=camera_to_workload.json`) - No visual output (`RENDER_MODE=0`) ### Environment Variables Reference -| Category | Variable | Description | Common Values | -| ---------------- | ---------------- | ------------------------------------- | ----------------------------------------------------------------------------------------------------- | -| **Display** | `RENDER_MODE` | Show/hide visual output | `0` (headless), `1` (visual) | -| **Performance** | `PIPELINE_COUNT` | Number of parallel pipeline instances | `1`, `2`, `4` (higher = more stress) | -| **Hardware** | `WORKLOAD_DIST` | Target processing hardware | `workload_to_pipeline_cpu.json`, `workload_to_pipeline_gpu.json`, `workload_to_pipeline_gpu-npu.json` | -| **Camera Setup** | `CAMERA_STREAM` | Camera configuration | `camera_to_workload.json`, `camera_to_workload_full.json` | -| **Build** | `REGISTRY` | Use pre-built vs local images | `true` (faster), `false` (custom builds) | +| Category | Variable | Description | Common Values | +|----------|----------|-------------|---------------| +| **Display** | `RENDER_MODE` | Show/hide visual output | `0` (headless), `1` (visual) | +| **Performance** | `PIPELINE_COUNT` | Number of parallel pipeline instances | `1`, `2`, `4` (higher = more stress) | +| **Hardware** | `WORKLOAD_DIST` | Target processing hardware | `workload_to_pipeline_cpu.json`, `workload_to_pipeline_gpu.json`, `workload_to_pipeline_gpu-npu.json` | +| **Camera Setup** | `CAMERA_STREAM` | Camera configuration | `camera_to_workload.json` | +| **Build** | `REGISTRY` | Use pre-built vs local images | `true` (faster), `false` (custom builds) | | **Build** | `REGISTRY` | Use pre-built vs local images | `true` (faster), `false` (custom builds) | @@ -60,45 +55,34 @@ make benchmark ### Camera Stream Configurations -**Standard Setup** (`camera_to_workload.json`): - -| Camera | Workloads | -| :----- | :--------------------------------------------- | -| cam1 | items_in_basket + multi_product_identification | -| cam2 | hidden_items + product_switching | -| cam3 | fake_scan_detection | - -**Full Workload Testing** (`camera_to_workload_full.json`): - -| Camera | Workload | -| :----- | :--------------------------- | -| cam1 | items_in_basket | -| cam2 | hidden_items | -| cam3 | fake_scan_detection | -| cam4 | multi_product_identification | -| cam5 | product_switching | -| cam6 | sweet_heartening | +**Full Workload Testing** (`camera_to_workload.json`): +| Camera | Workload | +|:-------|:---------| +| cam1 | items_in_basket | +| cam2 | hidden_items | +| cam3 | fake_scan_detection | +| cam4 | multi_product_identification | +| cam5 | product_switching | +| cam6 | sweet_heartening | ### Hardware Distribution Options -| Configuration | File | Best For | -| :---------------- | :---------------------------------- | :-------------------------------------- | -| **CPU Only** | `workload_to_pipeline_cpu.json` | Testing, development environments | -| **GPU Only** | `workload_to_pipeline_gpu.json` | Production, high performance | -| **Mixed GPU/NPU** | `workload_to_pipeline_gpu-npu.json` | Latest Intel hardware | -| **Heterogeneous** | `workload_to_pipeline_hetero.json` | Maximum performance across all hardware | -| **Default Mixed** | `workload_to_pipeline.json` | Balanced CPU/GPU/NPU distribution | +| Configuration | File | Best For | +|:--------------|:-----|:---------| +| **CPU Only** | `workload_to_pipeline_cpu.json` | Testing, development environments | +| **GPU Only** | `workload_to_pipeline_gpu.json` | Production, high performance | +| **Mixed GPU/NPU** | `workload_to_pipeline_gpu-npu.json` | Latest Intel hardware | +| **Heterogeneous** | `workload_to_pipeline_hetero.json` | Maximum performance across all hardware | +| **Default Mixed** | `workload_to_pipeline.json` | Balanced CPU/GPU/NPU distribution | ## Advanced Performance Testing (15-30 minutes) ### GPU Performance Testing - ```bash -make benchmark WORKLOAD_DIST=workload_to_pipeline_gpu.json CAMERA_STREAM=camera_to_workload_full.json +make benchmark WORKLOAD_DIST=workload_to_pipeline_gpu.json CAMERA_STREAM=camera_to_workload.json ``` ### Multi-Pipeline Stress Testing - ```bash # Test with 2 parallel pipelines make PIPELINE_COUNT=2 benchmark @@ -108,40 +92,35 @@ make PIPELINE_COUNT=4 benchmark ``` ### Custom Hardware Configuration - ```bash # Test heterogeneous workload distribution -make benchmark WORKLOAD_DIST=workload_to_pipeline_hetero.json CAMERA_STREAM=camera_to_workload_full.json REGISTRY=false +make benchmark WORKLOAD_DIST=workload_to_pipeline_hetero.json CAMERA_STREAM=camera_to_workload.json REGISTRY=false ``` ### Automated Self-Checkout Performance - ```bash # Object detection workload CAMERA_STREAM=camera_to_workload_asc_object_detection.json WORKLOAD_DIST=workload_to_pipeline_asc_object_detection_gpu.json make benchmark -# Age verification workload +# Age verification workload CAMERA_STREAM=camera_to_workload_asc_age_verification.json WORKLOAD_DIST=workload_to_pipeline_asc_age_verification_gpu.json make benchmark ``` ## Viewing Results ### Generate Consolidated Metrics - ```bash make consolidate-metrics ``` **Output**: `benchmark/metrics.csv` containing: - - FPS (frames per second) for each pipeline -- CPU/GPU/NPU utilization percentages +- CPU/GPU/NPU utilization percentages - Memory usage statistics - Power consumption data - Latency measurements ### View Results - ```bash cat benchmark/metrics.csv ``` @@ -151,24 +130,31 @@ cat benchmark/metrics.csv **Goal**: Find the maximum number of parallel pipelines your system can handle while maintaining target performance. ### Basic Stream Density Test - ```bash make benchmark-stream-density ``` **Default behavior:** - - Tests until FPS of any stream drops below its own target FPS from config JSON (`targetFps`); if not set, default `14.95` is used. - Uses OOM protection to prevent system crashes - Reports maximum sustainable pipeline count ### Custom Target FPS -Set per-stream target FPS using `lane_config.cameras[].targetFps` in `camera_to_workload_*.json`. +Set the target FPS for each camera using the `targetFps` field in the selected `camera_to_workload_*.json` file. Choose the value based on workload criticality and the source FPS. -Set `targetFps` based on workload criticality and source FPS. +If `targetFps` is not defined for a camera (or is not a positive value), the benchmark uses that camera's `fps` field instead. If neither field provides a positive value, it falls back to the default target FPS, currently `14.95`. -If `targetFps` is not present for a stream (or is invalid/non-positive), the stream falls back to `TARGET_FPS`. +To override the camera configuration, set the `TARGET_FPS` environment variable. When supplied, `TARGET_FPS` applies the same target to every stream and takes precedence over both `targetFps` and `fps`. + +In summary, the effective target FPS for each camera is resolved in this order: + +1. `TARGET_FPS` environment variable, when explicitly supplied (overrides the camera configuration for every stream). +2. The camera's `targetFps` in `camera_to_workload_*.json`. +3. The camera's `fps`, used when `targetFps` is missing or not a positive value. +4. The default target FPS, `14.95`, when neither camera field provides a positive value. + +The resolved target FPS and its source are recorded in `stream_density.log` so benchmark results can be audited. CLI examples: @@ -180,19 +166,32 @@ make TARGET_FPS=13.5 benchmark-stream-density make PIPELINE_SCRIPT=yolo11n_effnetb0.sh TARGET_FPS=13.5 benchmark-stream-density ``` +### Measurement Window & Settle Time + +`MEASUREMENT_WINDOW_SECONDS` controls how long FPS samples are collected for each density step. Its default is **100 seconds**. Each step adds a lane after one passing window, the final count is confirmed by two passing windows in a row, and a lane is only given up after two failing windows in a row. Each window is evaluated independently; windows are not combined into one longer window. + +`INIT_DURATION` sets the settle time, giving the pipelines time to stabilize before measurement starts. A longer settle time lets the pipelines stabilize more, and a longer measurement window can reduce short-term measurement noise, but both increase the total benchmark duration. The resolved settle time is also printed in the stream-density result summary. + +```sh +make benchmark-stream-density \ + INIT_DURATION=120 \ + MEASUREMENT_WINDOW_SECONDS=60 +``` + ### Stream Density Environment Variables -| Variable | Description | Values | -| :--------------- | :---------------------------- | :-------------------------------- | -| `TARGET_FPS` | Minimum FPS threshold | `14.95` (default), `13.5`, `20.0` | -| `OOM_PROTECTION` | Prevent out-of-memory crashes | `1` (enabled), `0` (disabled) | +| Variable | Description | Values | +|:---------|:------------|:--------| +| `TARGET_FPS` | Minimum FPS threshold | `14.95` (default), `13.5`, `20.0` | +| `MEASUREMENT_WINDOW_SECONDS` | Duration FPS samples are collected per density step | `100` (default), `60` | +| `INIT_DURATION` | Settle time before measurement starts | `120`, `60` | +| `OOM_PROTECTION` | Prevent out-of-memory crashes | `1` (enabled), `0` (disabled) | -> **Warning:** Setting `OOM_PROTECTION=0` may crash your system requiring a hard reboot. +> ⚠️ **Warning**: Setting `OOM_PROTECTION=0` may crash your system requiring a hard reboot. Note: stream-density caches camera-target mappings using `(config_path, file_mtime)` and auto-refreshes when the camera config changes; unit tests for this behavior are in the `performance-tools` repository at `benchmark-scripts/stream_density_test.py`. ### Expected Output - ``` Total averaged FPS per stream: 15.210442307692306 for 26 pipeline(s) ``` @@ -200,29 +199,25 @@ Total averaged FPS per stream: 15.210442307692306 for 26 pipeline(s) ## Visualization & Analysis ### Generate Performance Graphs - ```bash make plot-metrics ``` **Output**: `benchmark/plot_metrics.png` showing: - -- CPU Usage Over Time -- NPU Utilization Over Time -- GPU Usage for each GPU device found +- 🧠 CPU Usage Over Time +- ⚙️ NPU Utilization Over Time +- 🎮 GPU Usage for each GPU device found ### Useful Maintenance Commands - ```bash make validate-all-configs # Validate configuration files make clean-images # Remove dangling Docker images -make clean-containers # Remove stopped containers +make clean-containers # Remove stopped containers +make clean-all # Remove all unused Docker resources +``` make clean-all # Remove all unused Docker resources ``` -make clean-all # Remove all unused Docker resources - -```` ## Custom Configuration (Advanced) ### Creating Custom Workloads @@ -236,7 +231,7 @@ The application is highly configurable via JSON files in the `configs/` director "cameras": [ { "camera_id": "cam1", - "fileSrc": "sample-media/video1.mp4", + "fileSrc": "sample-media/video1.mp4", "targetFps": 12.5, "workloads": ["items_in_basket", "multi_product_identification"], "region_of_interest": {"x": 100, "y": 100, "x2": 800, "y2": 600} @@ -244,33 +239,21 @@ The application is highly configurable via JSON files in the `configs/` director ] } } -```` - -#### Pipeline Configuration (`workload_to_pipeline.json`) +``` +#### Pipeline Configuration (`workload_to_pipeline.json`) ```json { "workload_pipeline_map": { "items_in_basket": [ - { - "type": "gvadetect", - "model": "yolo11n", - "precision": "INT8", - "device": "CPU" - }, - { - "type": "gvaclassify", - "model": "efficientnet-v2-b0", - "precision": "INT8", - "device": "CPU" - } + {"type": "gvadetect", "model": "yolo11n", "precision": "INT8", "device": "CPU"}, + {"type": "gvaclassify", "model": "efficientnet-v2-b0", "precision": "INT8", "device": "CPU"} ] } } ``` -### Add Custom Workloads - +### To Add Custom Workloads: 1. Edit `configs/camera_to_workload.json` to add your camera and assign workloads 2. Edit `configs/workload_to_pipeline.json` to define the pipeline for your workload 3. Place your video files in `performance-tools/sample-media/` and update the `fileSrc` path @@ -280,24 +263,21 @@ The application is highly configurable via JSON files in the `configs/` director ## Detailed Hardware Distribution (Reference) ### Heterogeneous Configuration Breakdown - The `workload_to_pipeline_hetero.json` distributes workloads across multiple processing units: -| Workload | Object Detection | Classification | Inference | -| :--------------------------- | :--------------- | :------------- | :-------- | -| items_in_basket | GPU | GPU | - | -| hidden_items | GPU | CPU | - | -| fake_scan_detection | GPU | CPU | - | -| multi_product_identification | GPU | CPU | - | -| product_switching | GPU | GPU | - | -| sweet_heartening | NPU | - | NPU | - -### Mixed Configuration Details +| Workload | Object Detection | Classification | Inference | +|:---------|:----------------|:---------------|:----------| +| items_in_basket | GPU | GPU | - | +| hidden_items | GPU | CPU | - | +| fake_scan_detection | GPU | CPU | - | +| multi_product_identification | GPU | CPU | - | +| product_switching | GPU | GPU | - | +| sweet_heartening | NPU | - | NPU | +### Mixed Configuration Details The `workload_to_pipeline.json` balances workloads across available hardware: - - **CPU**: items_in_basket, multi_product_identification, sweet_heartening -- **GPU**: product_switching, hidden_items +- **GPU**: product_switching, hidden_items - **NPU**: fake_scan_detection ## Project Structure (Reference) @@ -318,7 +298,6 @@ The `workload_to_pipeline.json` balances workloads across available hardware: If your organization requires proxy settings for internet access: #### Shell Session Proxy - ```bash export http_proxy=http://: export https_proxy=http://: @@ -328,7 +307,6 @@ export NO_PROXY=localhost,127.0.0.1,::1 ``` #### System-Wide Proxy (`/etc/environment`) - ```bash sudo nano /etc/environment # Add the following lines: @@ -340,9 +318,7 @@ NO_PROXY=localhost,127.0.0.1,::1 ``` #### Docker Proxy Configuration - Create `/etc/systemd/system/docker.service.d/http-proxy.conf`: - ```bash sudo mkdir -p /etc/systemd/system/docker.service.d sudo nano /etc/systemd/system/docker.service.d/http-proxy.conf @@ -355,4 +331,4 @@ Environment="no_proxy=localhost,127.0.0.1,::1" # Reload and restart sudo systemctl daemon-reload sudo systemctl restart docker -``` +``` \ No newline at end of file diff --git a/docs_src/use-cases/loss-prevention/performance.md b/docs_src/use-cases/loss-prevention/performance.md index 603184f..a47b10c 100644 --- a/docs_src/use-cases/loss-prevention/performance.md +++ b/docs_src/use-cases/loss-prevention/performance.md @@ -46,7 +46,7 @@ make benchmark | **Display** | `RENDER_MODE` | Show/hide visual output | `0` (headless), `1` (visual) | | **Performance** | `PIPELINE_COUNT` | Number of parallel pipeline instances | `1`, `2`, `4` (higher = more stress) | | **Hardware** | `WORKLOAD_DIST` | Target processing hardware | `workload_to_pipeline_cpu.json`, `workload_to_pipeline_gpu.json`, `workload_to_pipeline_gpu-npu.json` | -| **Camera Setup** | `CAMERA_STREAM` | Camera configuration | `camera_to_workload.json`, `camera_to_workload_full.json` | +| **Camera Setup** | `CAMERA_STREAM` | Camera configuration | `camera_to_workload.json` | | **Build** | `REGISTRY` | Use pre-built vs local images | `true` (faster), `false` (custom builds) | | **Build** | `REGISTRY` | Use pre-built vs local images | `true` (faster), `false` (custom builds) | @@ -55,14 +55,7 @@ make benchmark ### Camera Stream Configurations -**Standard Setup** (`camera_to_workload.json`): -| Camera | Workloads | -|:-------|:----------| -| cam1 | items_in_basket + multi_product_identification | -| cam2 | hidden_items + product_switching | -| cam3 | fake_scan_detection | - -**Full Workload Testing** (`camera_to_workload_full.json`): +**Full Workload Testing** (`camera_to_workload.json`): | Camera | Workload | |:-------|:---------| | cam1 | items_in_basket | @@ -86,7 +79,7 @@ make benchmark ### GPU Performance Testing ```bash -make benchmark WORKLOAD_DIST=workload_to_pipeline_gpu.json CAMERA_STREAM=camera_to_workload_full.json +make benchmark WORKLOAD_DIST=workload_to_pipeline_gpu.json CAMERA_STREAM=camera_to_workload.json ``` ### Multi-Pipeline Stress Testing @@ -101,7 +94,7 @@ make PIPELINE_COUNT=4 benchmark ### Custom Hardware Configuration ```bash # Test heterogeneous workload distribution -make benchmark WORKLOAD_DIST=workload_to_pipeline_hetero.json CAMERA_STREAM=camera_to_workload_full.json REGISTRY=false +make benchmark WORKLOAD_DIST=workload_to_pipeline_hetero.json CAMERA_STREAM=camera_to_workload.json REGISTRY=false ``` ### Automated Self-Checkout Performance From a4ea91d526a9a8aa4c0f63229eddcc31b527eb5e Mon Sep 17 00:00:00 2001 From: sumanaga Date: Fri, 16 Oct 2026 07:57:58 +0530 Subject: [PATCH 3/5] docs(stream-density): clarify benchmark thresholds and timing terms Make FPS target precedence, the acceptance threshold, and warmup defaults clear so users can configure stream-density runs using accurate guidance. Standardize terminology and remove duplicated or malformed examples to reduce confusion when following benchmark instructions. Signed-off-by: sumanaga --- .../loss-prevention/lp-benchmarking.md | 161 ++++++++++-------- 1 file changed, 93 insertions(+), 68 deletions(-) diff --git a/docs/user-guide/loss-prevention/lp-benchmarking.md b/docs/user-guide/loss-prevention/lp-benchmarking.md index a47b10c..01215fd 100644 --- a/docs/user-guide/loss-prevention/lp-benchmarking.md +++ b/docs/user-guide/loss-prevention/lp-benchmarking.md @@ -7,20 +7,23 @@ Test your Loss Prevention and Automated Self-Checkout pipeline performance on va **Goal**: Run a basic performance test to verify your system works correctly ### 1. Initialize Performance Tools + ```bash make update-submodules ``` -### 2. Run Quick Benchmark +### 2. Run Quick Benchmark + ```bash make benchmark-quickstart ``` **What this does:** -- Tests GPU performance with 6 different loss prevention workloads + +- Benchmarks the six workloads in the default camera configuration using the default mixed CPU/GPU/NPU workload mapping - Runs headless (no display needed) -- Uses pre-built Docker images for faster setup -- Automatically generates consolidated metrics +- Uses registry images by default (`REGISTRY=false` builds the benchmark image locally) +- Generates consolidated metrics in `benchmark/metrics.csv` **Expected results**: You'll see FPS metrics and resource utilization for each workload. @@ -29,60 +32,63 @@ make benchmark-quickstart ### Basic Performance Testing **Default Benchmark Command:** + ```bash make benchmark ``` **Configuration:** -- Single pipeline instance (`PIPELINE_COUNT=1`) -- CPU-only processing (`WORKLOAD_DIST=workload_to_pipeline.json`) + +- Single pipeline instance (`PIPELINE_COUNT=1`) +- Mixed CPU/GPU/NPU processing (`WORKLOAD_DIST=workload_to_pipeline.json`) - Standard camera setup (`CAMERA_STREAM=camera_to_workload.json`) - No visual output (`RENDER_MODE=0`) ### Environment Variables Reference -| Category | Variable | Description | Common Values | -|----------|----------|-------------|---------------| -| **Display** | `RENDER_MODE` | Show/hide visual output | `0` (headless), `1` (visual) | -| **Performance** | `PIPELINE_COUNT` | Number of parallel pipeline instances | `1`, `2`, `4` (higher = more stress) | -| **Hardware** | `WORKLOAD_DIST` | Target processing hardware | `workload_to_pipeline_cpu.json`, `workload_to_pipeline_gpu.json`, `workload_to_pipeline_gpu-npu.json` | -| **Camera Setup** | `CAMERA_STREAM` | Camera configuration | `camera_to_workload.json` | -| **Build** | `REGISTRY` | Use pre-built vs local images | `true` (faster), `false` (custom builds) | - -| **Build** | `REGISTRY` | Use pre-built vs local images | `true` (faster), `false` (custom builds) | +| Category | Variable | Description | Common Values | +| ---------------- | ---------------- | -------------------------------------- | ----------------------------------------------------------------------------------------------------- | +| **Display** | `RENDER_MODE` | Show/hide visual output | `0` (headless), `1` (visual) | +| **Performance** | `PIPELINE_COUNT` | Number of parallel pipeline instances | `1`, `2`, `4` (higher = more stress) | +| **Hardware** | `WORKLOAD_DIST` | Target processing hardware | `workload_to_pipeline_cpu.json`, `workload_to_pipeline_gpu.json`, `workload_to_pipeline_gpu-npu.json`, `workload_to_pipeline_hetero.json` | +| **Camera Setup** | `CAMERA_STREAM` | Camera configuration | `camera_to_workload.json` | +| **Build** | `REGISTRY` | Use pre-built vs local images | `true` (faster), `false` (custom builds) | ## Workload Configuration Options ### Camera Stream Configurations **Full Workload Testing** (`camera_to_workload.json`): -| Camera | Workload | -|:-------|:---------| -| cam1 | items_in_basket | -| cam2 | hidden_items | -| cam3 | fake_scan_detection | -| cam4 | multi_product_identification | -| cam5 | product_switching | -| cam6 | sweet_heartening | + +| Camera | Workload | +| :----- | :--------------------------- | +| cam1 | items_in_basket | +| cam2 | hidden_items | +| cam3 | fake_scan_detection | +| cam4 | multi_product_identification | +| cam5 | product_switching | +| cam6 | sweet_heartening | ### Hardware Distribution Options -| Configuration | File | Best For | -|:--------------|:-----|:---------| -| **CPU Only** | `workload_to_pipeline_cpu.json` | Testing, development environments | -| **GPU Only** | `workload_to_pipeline_gpu.json` | Production, high performance | -| **Mixed GPU/NPU** | `workload_to_pipeline_gpu-npu.json` | Latest Intel hardware | -| **Heterogeneous** | `workload_to_pipeline_hetero.json` | Maximum performance across all hardware | -| **Default Mixed** | `workload_to_pipeline.json` | Balanced CPU/GPU/NPU distribution | +| Configuration | File | Best For | +| :---------------- | :---------------------------------- | :-------------------------------------- | +| **CPU Only** | `workload_to_pipeline_cpu.json` | Testing, development environments | +| **GPU Only** | `workload_to_pipeline_gpu.json` | Production, high performance | +| **Mixed GPU/NPU** | `workload_to_pipeline_gpu-npu.json` | Assign selected workloads to GPU or NPU | +| **Heterogeneous** | `workload_to_pipeline_hetero.json` | Maximum performance across all hardware | +| **Default Mixed** | `workload_to_pipeline.json` | CPU/GPU/NPU distribution used by default | ## Advanced Performance Testing (15-30 minutes) ### GPU Performance Testing + ```bash make benchmark WORKLOAD_DIST=workload_to_pipeline_gpu.json CAMERA_STREAM=camera_to_workload.json ``` ### Multi-Pipeline Stress Testing + ```bash # Test with 2 parallel pipelines make PIPELINE_COUNT=2 benchmark @@ -92,35 +98,40 @@ make PIPELINE_COUNT=4 benchmark ``` ### Custom Hardware Configuration + ```bash # Test heterogeneous workload distribution make benchmark WORKLOAD_DIST=workload_to_pipeline_hetero.json CAMERA_STREAM=camera_to_workload.json REGISTRY=false ``` ### Automated Self-Checkout Performance + ```bash # Object detection workload CAMERA_STREAM=camera_to_workload_asc_object_detection.json WORKLOAD_DIST=workload_to_pipeline_asc_object_detection_gpu.json make benchmark -# Age verification workload +# Age verification workload CAMERA_STREAM=camera_to_workload_asc_age_verification.json WORKLOAD_DIST=workload_to_pipeline_asc_age_verification_gpu.json make benchmark ``` ## Viewing Results ### Generate Consolidated Metrics + ```bash make consolidate-metrics ``` **Output**: `benchmark/metrics.csv` containing: + - FPS (frames per second) for each pipeline -- CPU/GPU/NPU utilization percentages +- CPU/GPU/NPU utilization percentages - Memory usage statistics - Power consumption data - Latency measurements ### View Results + ```bash cat benchmark/metrics.csv ``` @@ -130,12 +141,14 @@ cat benchmark/metrics.csv **Goal**: Find the maximum number of parallel pipelines your system can handle while maintaining target performance. ### Basic Stream Density Test + ```bash make benchmark-stream-density ``` **Default behavior:** -- Tests until FPS of any stream drops below its own target FPS from config JSON (`targetFps`); if not set, default `14.95` is used. + +- Requires every stream to reach at least 95% of its effective target FPS; by default targets come from each camera's `targetFps`, then `fps`, with a fallback of `15`. - Uses OOM protection to prevent system crashes - Reports maximum sustainable pipeline count @@ -143,7 +156,7 @@ make benchmark-stream-density Set the target FPS for each camera using the `targetFps` field in the selected `camera_to_workload_*.json` file. Choose the value based on workload criticality and the source FPS. -If `targetFps` is not defined for a camera (or is not a positive value), the benchmark uses that camera's `fps` field instead. If neither field provides a positive value, it falls back to the default target FPS, currently `14.95`. +If `targetFps` is not defined for a camera (or is not a positive value), the benchmark uses that camera's `fps` field instead. If neither field provides a positive value, it falls back to the default target FPS of `15`. To override the camera configuration, set the `TARGET_FPS` environment variable. When supplied, `TARGET_FPS` applies the same target to every stream and takes precedence over both `targetFps` and `fps`. @@ -152,9 +165,9 @@ In summary, the effective target FPS for each camera is resolved in this order: 1. `TARGET_FPS` environment variable, when explicitly supplied (overrides the camera configuration for every stream). 2. The camera's `targetFps` in `camera_to_workload_*.json`. 3. The camera's `fps`, used when `targetFps` is missing or not a positive value. -4. The default target FPS, `14.95`, when neither camera field provides a positive value. +4. The default target FPS, `15`, when neither camera field provides a positive value. -The resolved target FPS and its source are recorded in `stream_density.log` so benchmark results can be audited. +The benchmark passes when each stream reaches at least 95% of its effective target. The resolved target FPS and its source are recorded in `stream_density.log`. CLI examples: @@ -162,59 +175,62 @@ CLI examples: # Test for different performance thresholds make TARGET_FPS=13.5 benchmark-stream-density -# With custom pipeline configuration -make PIPELINE_SCRIPT=yolo11n_effnetb0.sh TARGET_FPS=13.5 benchmark-stream-density +# With explicit camera and workload configurations +make CAMERA_STREAM=camera_to_workload.json WORKLOAD_DIST=workload_to_pipeline_gpu.json TARGET_FPS=13.5 benchmark-stream-density ``` ### Measurement Window & Settle Time -`MEASUREMENT_WINDOW_SECONDS` controls how long FPS samples are collected for each density step. Its default is **100 seconds**. Each step adds a lane after one passing window, the final count is confirmed by two passing windows in a row, and a lane is only given up after two failing windows in a row. Each window is evaluated independently; windows are not combined into one longer window. +`MEASUREMENT_WINDOW_SECONDS` controls how long FPS samples are collected for each density step. Its default is **100 seconds**, and values below 100 are rejected because the benchmark requires at least 100 samples per pipeline log. Each step adds a lane after one passing window, the final count is confirmed by two passing windows in a row, and a lane is only given up after two failing windows in a row. Each window is evaluated independently; windows are not combined into one longer window. -`INIT_DURATION` sets the settle time, giving the pipelines time to stabilize before measurement starts. A longer settle time lets the pipelines stabilize more, and a longer measurement window can reduce short-term measurement noise, but both increase the total benchmark duration. The resolved settle time is also printed in the stream-density result summary. +`INIT_DURATION` sets the settle time, defaulting to **10 seconds**, before measurement starts. A longer settle time lets the pipelines stabilize more, and a longer measurement window can reduce short-term measurement noise, but both increase the total benchmark duration. The resolved settle time is also printed in the stream-density result summary. ```sh make benchmark-stream-density \ INIT_DURATION=120 \ - MEASUREMENT_WINDOW_SECONDS=60 + MEASUREMENT_WINDOW_SECONDS=120 ``` ### Stream Density Environment Variables -| Variable | Description | Values | -|:---------|:------------|:--------| -| `TARGET_FPS` | Minimum FPS threshold | `14.95` (default), `13.5`, `20.0` | -| `MEASUREMENT_WINDOW_SECONDS` | Duration FPS samples are collected per density step | `100` (default), `60` | -| `INIT_DURATION` | Settle time before measurement starts | `120`, `60` | -| `OOM_PROTECTION` | Prevent out-of-memory crashes | `1` (enabled), `0` (disabled) | +| Variable | Description | Values | +| :-------------------------- | :------------------------------------------------------ | :-------------------------------- | +| `TARGET_FPS` | Override the target FPS for all streams; unset uses camera configuration | Unset by default, `13.5`, `20.0` | +| `MEASUREMENT_WINDOW_SECONDS` | Duration FPS samples are collected per density step (minimum 100 seconds) | `100` (default), `120` | +| `INIT_DURATION` | Settle time before measurement starts | `10` (default), `120`, `60` | +| `OOM_PROTECTION` | Prevent out-of-memory crashes | `1` (enabled), `0` (disabled) | +| `PASS_TOLERANCE_RATIO` | Fraction of each target FPS required to pass | `0.95` (default), up to `1.0` | -> ⚠️ **Warning**: Setting `OOM_PROTECTION=0` may crash your system requiring a hard reboot. +> **Warning:** Setting `OOM_PROTECTION=0` may crash your system requiring a hard reboot. Note: stream-density caches camera-target mappings using `(config_path, file_mtime)` and auto-refreshes when the camera config changes; unit tests for this behavior are in the `performance-tools` repository at `benchmark-scripts/stream_density_test.py`. ### Expected Output + ``` -Total averaged FPS per stream: 15.210442307692306 for 26 pipeline(s) +Stream density completed: use case density lanes, stream density streams. For the detailed report, see benchmark/stream_density.log. ``` ## Visualization & Analysis ### Generate Performance Graphs + ```bash make plot-metrics ``` **Output**: `benchmark/plot_metrics.png` showing: -- 🧠 CPU Usage Over Time -- ⚙️ NPU Utilization Over Time -- 🎮 GPU Usage for each GPU device found + +- CPU Usage Over Time +- NPU Utilization Over Time +- GPU Usage for each GPU device found ### Useful Maintenance Commands + ```bash make validate-all-configs # Validate configuration files make clean-images # Remove dangling Docker images -make clean-containers # Remove stopped containers -make clean-all # Remove all unused Docker resources -``` +make clean-containers # Remove stopped containers make clean-all # Remove all unused Docker resources ``` @@ -225,13 +241,14 @@ make clean-all # Remove all unused Docker resources The application is highly configurable via JSON files in the `configs/` directory: #### Camera Configuration (`camera_to_workload.json`) + ```json { "lane_config": { "cameras": [ { "camera_id": "cam1", - "fileSrc": "sample-media/video1.mp4", + "fileSrc": "sample-media/video1.mp4", "targetFps": 12.5, "workloads": ["items_in_basket", "multi_product_identification"], "region_of_interest": {"x": 100, "y": 100, "x2": 800, "y2": 600} @@ -241,7 +258,8 @@ The application is highly configurable via JSON files in the `configs/` director } ``` -#### Pipeline Configuration (`workload_to_pipeline.json`) +#### Pipeline Configuration (`workload_to_pipeline.json`) + ```json { "workload_pipeline_map": { @@ -254,6 +272,7 @@ The application is highly configurable via JSON files in the `configs/` director ``` ### To Add Custom Workloads: + 1. Edit `configs/camera_to_workload.json` to add your camera and assign workloads 2. Edit `configs/workload_to_pipeline.json` to define the pipeline for your workload 3. Place your video files in `performance-tools/sample-media/` and update the `fileSrc` path @@ -263,21 +282,24 @@ The application is highly configurable via JSON files in the `configs/` director ## Detailed Hardware Distribution (Reference) ### Heterogeneous Configuration Breakdown + The `workload_to_pipeline_hetero.json` distributes workloads across multiple processing units: -| Workload | Object Detection | Classification | Inference | -|:---------|:----------------|:---------------|:----------| -| items_in_basket | GPU | GPU | - | -| hidden_items | GPU | CPU | - | -| fake_scan_detection | GPU | CPU | - | -| multi_product_identification | GPU | CPU | - | -| product_switching | GPU | GPU | - | -| sweet_heartening | NPU | - | NPU | +| Workload | Object Detection | Classification | Inference | +| :--------------------------- | :--------------- | :------------- | :-------- | +| items_in_basket | CPU | CPU | - | +| hidden_items | GPU | GPU | - | +| fake_scan_detection | GPU | GPU | - | +| multi_product_identification | NPU | NPU | - | +| product_switching | CPU | CPU | - | +| sweet_heartening | NPU | - | NPU | + +### Mixed Configuration Details -### Mixed Configuration Details The `workload_to_pipeline.json` balances workloads across available hardware: + - **CPU**: items_in_basket, multi_product_identification, sweet_heartening -- **GPU**: product_switching, hidden_items +- **GPU**: product_switching, hidden_items - **NPU**: fake_scan_detection ## Project Structure (Reference) @@ -286,7 +308,7 @@ The `workload_to_pipeline.json` balances workloads across available hardware: - `docker/` — Dockerfiles for containers - `download-scripts/` — Model and video download scripts - `src/` — Pipeline runner scripts -- `performance-tools/benchmark-scripts/results/` — Performance test results +- `benchmark/` — Benchmark logs, consolidated metrics, and plots (default `RESULTS_DIR`) - `Makefile` — Build automation commands --- @@ -298,6 +320,7 @@ The `workload_to_pipeline.json` balances workloads across available hardware: If your organization requires proxy settings for internet access: #### Shell Session Proxy + ```bash export http_proxy=http://: export https_proxy=http://: @@ -307,6 +330,7 @@ export NO_PROXY=localhost,127.0.0.1,::1 ``` #### System-Wide Proxy (`/etc/environment`) + ```bash sudo nano /etc/environment # Add the following lines: @@ -319,6 +343,7 @@ NO_PROXY=localhost,127.0.0.1,::1 #### Docker Proxy Configuration Create `/etc/systemd/system/docker.service.d/http-proxy.conf`: + ```bash sudo mkdir -p /etc/systemd/system/docker.service.d sudo nano /etc/systemd/system/docker.service.d/http-proxy.conf From 07b7fe766ce0c44f8f560bbdf5f472dbd2bdf7c3 Mon Sep 17 00:00:00 2001 From: sumanaga Date: Wed, 21 Oct 2026 08:57:15 +0530 Subject: [PATCH 4/5] docs(stream-density):Performance.md updates Signed-off-by: sumanaga --- .../use-cases/loss-prevention/performance.md | 161 ++++++++++-------- 1 file changed, 93 insertions(+), 68 deletions(-) diff --git a/docs_src/use-cases/loss-prevention/performance.md b/docs_src/use-cases/loss-prevention/performance.md index a47b10c..01215fd 100644 --- a/docs_src/use-cases/loss-prevention/performance.md +++ b/docs_src/use-cases/loss-prevention/performance.md @@ -7,20 +7,23 @@ Test your Loss Prevention and Automated Self-Checkout pipeline performance on va **Goal**: Run a basic performance test to verify your system works correctly ### 1. Initialize Performance Tools + ```bash make update-submodules ``` -### 2. Run Quick Benchmark +### 2. Run Quick Benchmark + ```bash make benchmark-quickstart ``` **What this does:** -- Tests GPU performance with 6 different loss prevention workloads + +- Benchmarks the six workloads in the default camera configuration using the default mixed CPU/GPU/NPU workload mapping - Runs headless (no display needed) -- Uses pre-built Docker images for faster setup -- Automatically generates consolidated metrics +- Uses registry images by default (`REGISTRY=false` builds the benchmark image locally) +- Generates consolidated metrics in `benchmark/metrics.csv` **Expected results**: You'll see FPS metrics and resource utilization for each workload. @@ -29,60 +32,63 @@ make benchmark-quickstart ### Basic Performance Testing **Default Benchmark Command:** + ```bash make benchmark ``` **Configuration:** -- Single pipeline instance (`PIPELINE_COUNT=1`) -- CPU-only processing (`WORKLOAD_DIST=workload_to_pipeline.json`) + +- Single pipeline instance (`PIPELINE_COUNT=1`) +- Mixed CPU/GPU/NPU processing (`WORKLOAD_DIST=workload_to_pipeline.json`) - Standard camera setup (`CAMERA_STREAM=camera_to_workload.json`) - No visual output (`RENDER_MODE=0`) ### Environment Variables Reference -| Category | Variable | Description | Common Values | -|----------|----------|-------------|---------------| -| **Display** | `RENDER_MODE` | Show/hide visual output | `0` (headless), `1` (visual) | -| **Performance** | `PIPELINE_COUNT` | Number of parallel pipeline instances | `1`, `2`, `4` (higher = more stress) | -| **Hardware** | `WORKLOAD_DIST` | Target processing hardware | `workload_to_pipeline_cpu.json`, `workload_to_pipeline_gpu.json`, `workload_to_pipeline_gpu-npu.json` | -| **Camera Setup** | `CAMERA_STREAM` | Camera configuration | `camera_to_workload.json` | -| **Build** | `REGISTRY` | Use pre-built vs local images | `true` (faster), `false` (custom builds) | - -| **Build** | `REGISTRY` | Use pre-built vs local images | `true` (faster), `false` (custom builds) | +| Category | Variable | Description | Common Values | +| ---------------- | ---------------- | -------------------------------------- | ----------------------------------------------------------------------------------------------------- | +| **Display** | `RENDER_MODE` | Show/hide visual output | `0` (headless), `1` (visual) | +| **Performance** | `PIPELINE_COUNT` | Number of parallel pipeline instances | `1`, `2`, `4` (higher = more stress) | +| **Hardware** | `WORKLOAD_DIST` | Target processing hardware | `workload_to_pipeline_cpu.json`, `workload_to_pipeline_gpu.json`, `workload_to_pipeline_gpu-npu.json`, `workload_to_pipeline_hetero.json` | +| **Camera Setup** | `CAMERA_STREAM` | Camera configuration | `camera_to_workload.json` | +| **Build** | `REGISTRY` | Use pre-built vs local images | `true` (faster), `false` (custom builds) | ## Workload Configuration Options ### Camera Stream Configurations **Full Workload Testing** (`camera_to_workload.json`): -| Camera | Workload | -|:-------|:---------| -| cam1 | items_in_basket | -| cam2 | hidden_items | -| cam3 | fake_scan_detection | -| cam4 | multi_product_identification | -| cam5 | product_switching | -| cam6 | sweet_heartening | + +| Camera | Workload | +| :----- | :--------------------------- | +| cam1 | items_in_basket | +| cam2 | hidden_items | +| cam3 | fake_scan_detection | +| cam4 | multi_product_identification | +| cam5 | product_switching | +| cam6 | sweet_heartening | ### Hardware Distribution Options -| Configuration | File | Best For | -|:--------------|:-----|:---------| -| **CPU Only** | `workload_to_pipeline_cpu.json` | Testing, development environments | -| **GPU Only** | `workload_to_pipeline_gpu.json` | Production, high performance | -| **Mixed GPU/NPU** | `workload_to_pipeline_gpu-npu.json` | Latest Intel hardware | -| **Heterogeneous** | `workload_to_pipeline_hetero.json` | Maximum performance across all hardware | -| **Default Mixed** | `workload_to_pipeline.json` | Balanced CPU/GPU/NPU distribution | +| Configuration | File | Best For | +| :---------------- | :---------------------------------- | :-------------------------------------- | +| **CPU Only** | `workload_to_pipeline_cpu.json` | Testing, development environments | +| **GPU Only** | `workload_to_pipeline_gpu.json` | Production, high performance | +| **Mixed GPU/NPU** | `workload_to_pipeline_gpu-npu.json` | Assign selected workloads to GPU or NPU | +| **Heterogeneous** | `workload_to_pipeline_hetero.json` | Maximum performance across all hardware | +| **Default Mixed** | `workload_to_pipeline.json` | CPU/GPU/NPU distribution used by default | ## Advanced Performance Testing (15-30 minutes) ### GPU Performance Testing + ```bash make benchmark WORKLOAD_DIST=workload_to_pipeline_gpu.json CAMERA_STREAM=camera_to_workload.json ``` ### Multi-Pipeline Stress Testing + ```bash # Test with 2 parallel pipelines make PIPELINE_COUNT=2 benchmark @@ -92,35 +98,40 @@ make PIPELINE_COUNT=4 benchmark ``` ### Custom Hardware Configuration + ```bash # Test heterogeneous workload distribution make benchmark WORKLOAD_DIST=workload_to_pipeline_hetero.json CAMERA_STREAM=camera_to_workload.json REGISTRY=false ``` ### Automated Self-Checkout Performance + ```bash # Object detection workload CAMERA_STREAM=camera_to_workload_asc_object_detection.json WORKLOAD_DIST=workload_to_pipeline_asc_object_detection_gpu.json make benchmark -# Age verification workload +# Age verification workload CAMERA_STREAM=camera_to_workload_asc_age_verification.json WORKLOAD_DIST=workload_to_pipeline_asc_age_verification_gpu.json make benchmark ``` ## Viewing Results ### Generate Consolidated Metrics + ```bash make consolidate-metrics ``` **Output**: `benchmark/metrics.csv` containing: + - FPS (frames per second) for each pipeline -- CPU/GPU/NPU utilization percentages +- CPU/GPU/NPU utilization percentages - Memory usage statistics - Power consumption data - Latency measurements ### View Results + ```bash cat benchmark/metrics.csv ``` @@ -130,12 +141,14 @@ cat benchmark/metrics.csv **Goal**: Find the maximum number of parallel pipelines your system can handle while maintaining target performance. ### Basic Stream Density Test + ```bash make benchmark-stream-density ``` **Default behavior:** -- Tests until FPS of any stream drops below its own target FPS from config JSON (`targetFps`); if not set, default `14.95` is used. + +- Requires every stream to reach at least 95% of its effective target FPS; by default targets come from each camera's `targetFps`, then `fps`, with a fallback of `15`. - Uses OOM protection to prevent system crashes - Reports maximum sustainable pipeline count @@ -143,7 +156,7 @@ make benchmark-stream-density Set the target FPS for each camera using the `targetFps` field in the selected `camera_to_workload_*.json` file. Choose the value based on workload criticality and the source FPS. -If `targetFps` is not defined for a camera (or is not a positive value), the benchmark uses that camera's `fps` field instead. If neither field provides a positive value, it falls back to the default target FPS, currently `14.95`. +If `targetFps` is not defined for a camera (or is not a positive value), the benchmark uses that camera's `fps` field instead. If neither field provides a positive value, it falls back to the default target FPS of `15`. To override the camera configuration, set the `TARGET_FPS` environment variable. When supplied, `TARGET_FPS` applies the same target to every stream and takes precedence over both `targetFps` and `fps`. @@ -152,9 +165,9 @@ In summary, the effective target FPS for each camera is resolved in this order: 1. `TARGET_FPS` environment variable, when explicitly supplied (overrides the camera configuration for every stream). 2. The camera's `targetFps` in `camera_to_workload_*.json`. 3. The camera's `fps`, used when `targetFps` is missing or not a positive value. -4. The default target FPS, `14.95`, when neither camera field provides a positive value. +4. The default target FPS, `15`, when neither camera field provides a positive value. -The resolved target FPS and its source are recorded in `stream_density.log` so benchmark results can be audited. +The benchmark passes when each stream reaches at least 95% of its effective target. The resolved target FPS and its source are recorded in `stream_density.log`. CLI examples: @@ -162,59 +175,62 @@ CLI examples: # Test for different performance thresholds make TARGET_FPS=13.5 benchmark-stream-density -# With custom pipeline configuration -make PIPELINE_SCRIPT=yolo11n_effnetb0.sh TARGET_FPS=13.5 benchmark-stream-density +# With explicit camera and workload configurations +make CAMERA_STREAM=camera_to_workload.json WORKLOAD_DIST=workload_to_pipeline_gpu.json TARGET_FPS=13.5 benchmark-stream-density ``` ### Measurement Window & Settle Time -`MEASUREMENT_WINDOW_SECONDS` controls how long FPS samples are collected for each density step. Its default is **100 seconds**. Each step adds a lane after one passing window, the final count is confirmed by two passing windows in a row, and a lane is only given up after two failing windows in a row. Each window is evaluated independently; windows are not combined into one longer window. +`MEASUREMENT_WINDOW_SECONDS` controls how long FPS samples are collected for each density step. Its default is **100 seconds**, and values below 100 are rejected because the benchmark requires at least 100 samples per pipeline log. Each step adds a lane after one passing window, the final count is confirmed by two passing windows in a row, and a lane is only given up after two failing windows in a row. Each window is evaluated independently; windows are not combined into one longer window. -`INIT_DURATION` sets the settle time, giving the pipelines time to stabilize before measurement starts. A longer settle time lets the pipelines stabilize more, and a longer measurement window can reduce short-term measurement noise, but both increase the total benchmark duration. The resolved settle time is also printed in the stream-density result summary. +`INIT_DURATION` sets the settle time, defaulting to **10 seconds**, before measurement starts. A longer settle time lets the pipelines stabilize more, and a longer measurement window can reduce short-term measurement noise, but both increase the total benchmark duration. The resolved settle time is also printed in the stream-density result summary. ```sh make benchmark-stream-density \ INIT_DURATION=120 \ - MEASUREMENT_WINDOW_SECONDS=60 + MEASUREMENT_WINDOW_SECONDS=120 ``` ### Stream Density Environment Variables -| Variable | Description | Values | -|:---------|:------------|:--------| -| `TARGET_FPS` | Minimum FPS threshold | `14.95` (default), `13.5`, `20.0` | -| `MEASUREMENT_WINDOW_SECONDS` | Duration FPS samples are collected per density step | `100` (default), `60` | -| `INIT_DURATION` | Settle time before measurement starts | `120`, `60` | -| `OOM_PROTECTION` | Prevent out-of-memory crashes | `1` (enabled), `0` (disabled) | +| Variable | Description | Values | +| :-------------------------- | :------------------------------------------------------ | :-------------------------------- | +| `TARGET_FPS` | Override the target FPS for all streams; unset uses camera configuration | Unset by default, `13.5`, `20.0` | +| `MEASUREMENT_WINDOW_SECONDS` | Duration FPS samples are collected per density step (minimum 100 seconds) | `100` (default), `120` | +| `INIT_DURATION` | Settle time before measurement starts | `10` (default), `120`, `60` | +| `OOM_PROTECTION` | Prevent out-of-memory crashes | `1` (enabled), `0` (disabled) | +| `PASS_TOLERANCE_RATIO` | Fraction of each target FPS required to pass | `0.95` (default), up to `1.0` | -> ⚠️ **Warning**: Setting `OOM_PROTECTION=0` may crash your system requiring a hard reboot. +> **Warning:** Setting `OOM_PROTECTION=0` may crash your system requiring a hard reboot. Note: stream-density caches camera-target mappings using `(config_path, file_mtime)` and auto-refreshes when the camera config changes; unit tests for this behavior are in the `performance-tools` repository at `benchmark-scripts/stream_density_test.py`. ### Expected Output + ``` -Total averaged FPS per stream: 15.210442307692306 for 26 pipeline(s) +Stream density completed: use case density lanes, stream density streams. For the detailed report, see benchmark/stream_density.log. ``` ## Visualization & Analysis ### Generate Performance Graphs + ```bash make plot-metrics ``` **Output**: `benchmark/plot_metrics.png` showing: -- 🧠 CPU Usage Over Time -- ⚙️ NPU Utilization Over Time -- 🎮 GPU Usage for each GPU device found + +- CPU Usage Over Time +- NPU Utilization Over Time +- GPU Usage for each GPU device found ### Useful Maintenance Commands + ```bash make validate-all-configs # Validate configuration files make clean-images # Remove dangling Docker images -make clean-containers # Remove stopped containers -make clean-all # Remove all unused Docker resources -``` +make clean-containers # Remove stopped containers make clean-all # Remove all unused Docker resources ``` @@ -225,13 +241,14 @@ make clean-all # Remove all unused Docker resources The application is highly configurable via JSON files in the `configs/` directory: #### Camera Configuration (`camera_to_workload.json`) + ```json { "lane_config": { "cameras": [ { "camera_id": "cam1", - "fileSrc": "sample-media/video1.mp4", + "fileSrc": "sample-media/video1.mp4", "targetFps": 12.5, "workloads": ["items_in_basket", "multi_product_identification"], "region_of_interest": {"x": 100, "y": 100, "x2": 800, "y2": 600} @@ -241,7 +258,8 @@ The application is highly configurable via JSON files in the `configs/` director } ``` -#### Pipeline Configuration (`workload_to_pipeline.json`) +#### Pipeline Configuration (`workload_to_pipeline.json`) + ```json { "workload_pipeline_map": { @@ -254,6 +272,7 @@ The application is highly configurable via JSON files in the `configs/` director ``` ### To Add Custom Workloads: + 1. Edit `configs/camera_to_workload.json` to add your camera and assign workloads 2. Edit `configs/workload_to_pipeline.json` to define the pipeline for your workload 3. Place your video files in `performance-tools/sample-media/` and update the `fileSrc` path @@ -263,21 +282,24 @@ The application is highly configurable via JSON files in the `configs/` director ## Detailed Hardware Distribution (Reference) ### Heterogeneous Configuration Breakdown + The `workload_to_pipeline_hetero.json` distributes workloads across multiple processing units: -| Workload | Object Detection | Classification | Inference | -|:---------|:----------------|:---------------|:----------| -| items_in_basket | GPU | GPU | - | -| hidden_items | GPU | CPU | - | -| fake_scan_detection | GPU | CPU | - | -| multi_product_identification | GPU | CPU | - | -| product_switching | GPU | GPU | - | -| sweet_heartening | NPU | - | NPU | +| Workload | Object Detection | Classification | Inference | +| :--------------------------- | :--------------- | :------------- | :-------- | +| items_in_basket | CPU | CPU | - | +| hidden_items | GPU | GPU | - | +| fake_scan_detection | GPU | GPU | - | +| multi_product_identification | NPU | NPU | - | +| product_switching | CPU | CPU | - | +| sweet_heartening | NPU | - | NPU | + +### Mixed Configuration Details -### Mixed Configuration Details The `workload_to_pipeline.json` balances workloads across available hardware: + - **CPU**: items_in_basket, multi_product_identification, sweet_heartening -- **GPU**: product_switching, hidden_items +- **GPU**: product_switching, hidden_items - **NPU**: fake_scan_detection ## Project Structure (Reference) @@ -286,7 +308,7 @@ The `workload_to_pipeline.json` balances workloads across available hardware: - `docker/` — Dockerfiles for containers - `download-scripts/` — Model and video download scripts - `src/` — Pipeline runner scripts -- `performance-tools/benchmark-scripts/results/` — Performance test results +- `benchmark/` — Benchmark logs, consolidated metrics, and plots (default `RESULTS_DIR`) - `Makefile` — Build automation commands --- @@ -298,6 +320,7 @@ The `workload_to_pipeline.json` balances workloads across available hardware: If your organization requires proxy settings for internet access: #### Shell Session Proxy + ```bash export http_proxy=http://: export https_proxy=http://: @@ -307,6 +330,7 @@ export NO_PROXY=localhost,127.0.0.1,::1 ``` #### System-Wide Proxy (`/etc/environment`) + ```bash sudo nano /etc/environment # Add the following lines: @@ -319,6 +343,7 @@ NO_PROXY=localhost,127.0.0.1,::1 #### Docker Proxy Configuration Create `/etc/systemd/system/docker.service.d/http-proxy.conf`: + ```bash sudo mkdir -p /etc/systemd/system/docker.service.d sudo nano /etc/systemd/system/docker.service.d/http-proxy.conf From e78da6892e65495136139bb32fb0d446504da503 Mon Sep 17 00:00:00 2001 From: sumanaga Date: Thu, 22 Oct 2026 08:58:19 +0530 Subject: [PATCH 5/5] docs(benchmark): standardize timing terminology Use consistent terms for measurement intervals and warmup periods so readers can understand benchmark timing and acceptance criteria without ambiguity. This improves clarity across both performance guides. Signed-off-by: sumanaga --- docs/user-guide/loss-prevention/lp-benchmarking.md | 8 ++++---- docs_src/use-cases/loss-prevention/performance.md | 8 ++++---- 2 files changed, 8 insertions(+), 8 deletions(-) diff --git a/docs/user-guide/loss-prevention/lp-benchmarking.md b/docs/user-guide/loss-prevention/lp-benchmarking.md index 01215fd..7f4f81c 100644 --- a/docs/user-guide/loss-prevention/lp-benchmarking.md +++ b/docs/user-guide/loss-prevention/lp-benchmarking.md @@ -179,11 +179,11 @@ make TARGET_FPS=13.5 benchmark-stream-density make CAMERA_STREAM=camera_to_workload.json WORKLOAD_DIST=workload_to_pipeline_gpu.json TARGET_FPS=13.5 benchmark-stream-density ``` -### Measurement Window & Settle Time +### Measurement interval and warmup period -`MEASUREMENT_WINDOW_SECONDS` controls how long FPS samples are collected for each density step. Its default is **100 seconds**, and values below 100 are rejected because the benchmark requires at least 100 samples per pipeline log. Each step adds a lane after one passing window, the final count is confirmed by two passing windows in a row, and a lane is only given up after two failing windows in a row. Each window is evaluated independently; windows are not combined into one longer window. +`MEASUREMENT_WINDOW_SECONDS` controls how long FPS samples are collected for each density step. Its default is **100 seconds**, and values below 100 are rejected because the benchmark requires at least 100 samples per pipeline log. Each step adds a lane after one passing measurement interval, the final count is confirmed by run acceptance criterion (two consecutive passes), and a lane is only given up after two failing measurement interval in a row. Each measurement interval is evaluated independently; measurement interval are not combined into one longer measurement interval. -`INIT_DURATION` sets the settle time, defaulting to **10 seconds**, before measurement starts. A longer settle time lets the pipelines stabilize more, and a longer measurement window can reduce short-term measurement noise, but both increase the total benchmark duration. The resolved settle time is also printed in the stream-density result summary. +`INIT_DURATION` sets the warmup period, defaulting to **10 seconds**, before measurement starts. A longer warmup period lets the pipelines stabilize more, and a longer measurement interval can reduce short-term measurement noise, but both increase the total benchmark duration. The resolved warmup period is also printed in the stream-density result summary. ```sh make benchmark-stream-density \ @@ -197,7 +197,7 @@ make benchmark-stream-density \ | :-------------------------- | :------------------------------------------------------ | :-------------------------------- | | `TARGET_FPS` | Override the target FPS for all streams; unset uses camera configuration | Unset by default, `13.5`, `20.0` | | `MEASUREMENT_WINDOW_SECONDS` | Duration FPS samples are collected per density step (minimum 100 seconds) | `100` (default), `120` | -| `INIT_DURATION` | Settle time before measurement starts | `10` (default), `120`, `60` | +| `INIT_DURATION` | warmup period before measurement starts | `10` (default), `120`, `60` | | `OOM_PROTECTION` | Prevent out-of-memory crashes | `1` (enabled), `0` (disabled) | | `PASS_TOLERANCE_RATIO` | Fraction of each target FPS required to pass | `0.95` (default), up to `1.0` | diff --git a/docs_src/use-cases/loss-prevention/performance.md b/docs_src/use-cases/loss-prevention/performance.md index 01215fd..7f4f81c 100644 --- a/docs_src/use-cases/loss-prevention/performance.md +++ b/docs_src/use-cases/loss-prevention/performance.md @@ -179,11 +179,11 @@ make TARGET_FPS=13.5 benchmark-stream-density make CAMERA_STREAM=camera_to_workload.json WORKLOAD_DIST=workload_to_pipeline_gpu.json TARGET_FPS=13.5 benchmark-stream-density ``` -### Measurement Window & Settle Time +### Measurement interval and warmup period -`MEASUREMENT_WINDOW_SECONDS` controls how long FPS samples are collected for each density step. Its default is **100 seconds**, and values below 100 are rejected because the benchmark requires at least 100 samples per pipeline log. Each step adds a lane after one passing window, the final count is confirmed by two passing windows in a row, and a lane is only given up after two failing windows in a row. Each window is evaluated independently; windows are not combined into one longer window. +`MEASUREMENT_WINDOW_SECONDS` controls how long FPS samples are collected for each density step. Its default is **100 seconds**, and values below 100 are rejected because the benchmark requires at least 100 samples per pipeline log. Each step adds a lane after one passing measurement interval, the final count is confirmed by run acceptance criterion (two consecutive passes), and a lane is only given up after two failing measurement interval in a row. Each measurement interval is evaluated independently; measurement interval are not combined into one longer measurement interval. -`INIT_DURATION` sets the settle time, defaulting to **10 seconds**, before measurement starts. A longer settle time lets the pipelines stabilize more, and a longer measurement window can reduce short-term measurement noise, but both increase the total benchmark duration. The resolved settle time is also printed in the stream-density result summary. +`INIT_DURATION` sets the warmup period, defaulting to **10 seconds**, before measurement starts. A longer warmup period lets the pipelines stabilize more, and a longer measurement interval can reduce short-term measurement noise, but both increase the total benchmark duration. The resolved warmup period is also printed in the stream-density result summary. ```sh make benchmark-stream-density \ @@ -197,7 +197,7 @@ make benchmark-stream-density \ | :-------------------------- | :------------------------------------------------------ | :-------------------------------- | | `TARGET_FPS` | Override the target FPS for all streams; unset uses camera configuration | Unset by default, `13.5`, `20.0` | | `MEASUREMENT_WINDOW_SECONDS` | Duration FPS samples are collected per density step (minimum 100 seconds) | `100` (default), `120` | -| `INIT_DURATION` | Settle time before measurement starts | `10` (default), `120`, `60` | +| `INIT_DURATION` | warmup period before measurement starts | `10` (default), `120`, `60` | | `OOM_PROTECTION` | Prevent out-of-memory crashes | `1` (enabled), `0` (disabled) | | `PASS_TOLERANCE_RATIO` | Fraction of each target FPS required to pass | `0.95` (default), up to `1.0` |