Skip to content

docs(stream-density): document TARGET_FPS precedence and measurement window #120

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
sumanaga wants to merge 5 commits into
base: main
Choose a base branch
from
Open

docs(stream-density): document TARGET_FPS precedence and measurement window #120

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
f90184d
docs(stream-density): document TARGET_FPS precedence and measurement …
sumanaga Sep 29, 2026
5e92a1d
docs(stream-density): document TARGET_FPS precedence and measurement …
sumanaga Sep 29, 2026
a4ea91d
docs(stream-density): clarify benchmark thresholds and timing terms
sumanaga Oct 16, 2026
272b920
Merge branch 'main' into main
sumanaga Oct 16, 2026
07b7fe7
docs(stream-density):Performance.md updates
sumanaga Oct 21, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
123 changes: 62 additions & 61 deletions docs/user-guide/loss-prevention/lp-benchmarking.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,10 +20,10 @@ make benchmark-quickstart

**What this does:**

- Tests GPU performance with 6 different loss prevention workloads
- Benchmarks the six workloads in the default camera configuration using the default mixed CPU/GPU/NPU workload mapping
- Runs headless (no display needed)
- Uses pre-built Docker images for faster setup
- Automatically generates consolidated metrics
- Uses registry images by default (`REGISTRY=false` builds the benchmark image locally)
- Generates consolidated metrics in `benchmark/metrics.csv`

**Expected results**: You'll see FPS metrics and resource utilization for each workload.

Expand All @@ -40,35 +40,25 @@ make benchmark
**Configuration:**

- Single pipeline instance (`PIPELINE_COUNT=1`)
- CPU-only processing (`WORKLOAD_DIST=workload_to_pipeline.json`)
- Mixed CPU/GPU/NPU processing (`WORKLOAD_DIST=workload_to_pipeline.json`)
- Standard camera setup (`CAMERA_STREAM=camera_to_workload.json`)
- No visual output (`RENDER_MODE=0`)

### Environment Variables Reference

| Category | Variable | Description | Common Values |
| ---------------- | ---------------- | ------------------------------------- | ----------------------------------------------------------------------------------------------------- |
| **Display** | `RENDER_MODE` | Show/hide visual output | `0` (headless), `1` (visual) |
| Category | Variable | Description | Common Values |
| ---------------- | ---------------- | -------------------------------------- | ----------------------------------------------------------------------------------------------------- |
| **Display** | `RENDER_MODE` | Show/hide visual output | `0` (headless), `1` (visual) |
| **Performance** | `PIPELINE_COUNT` | Number of parallel pipeline instances | `1`, `2`, `4` (higher = more stress) |
| **Hardware** | `WORKLOAD_DIST` | Target processing hardware | `workload_to_pipeline_cpu.json`, `workload_to_pipeline_gpu.json`, `workload_to_pipeline_gpu-npu.json` |
| **Camera Setup** | `CAMERA_STREAM` | Camera configuration | `camera_to_workload.json`, `camera_to_workload_full.json` |
| **Build** | `REGISTRY` | Use pre-built vs local images | `true` (faster), `false` (custom builds) |

| **Build** | `REGISTRY` | Use pre-built vs local images | `true` (faster), `false` (custom builds) |
| **Hardware** | `WORKLOAD_DIST` | Target processing hardware | `workload_to_pipeline_cpu.json`, `workload_to_pipeline_gpu.json`, `workload_to_pipeline_gpu-npu.json`, `workload_to_pipeline_hetero.json` |
| **Camera Setup** | `CAMERA_STREAM` | Camera configuration | `camera_to_workload.json` |
| **Build** | `REGISTRY` | Use pre-built vs local images | `true` (faster), `false` (custom builds) |

## Workload Configuration Options

### Camera Stream Configurations

**Standard Setup** (`camera_to_workload.json`):

| Camera | Workloads |
| :----- | :--------------------------------------------- |
| cam1 | items_in_basket + multi_product_identification |
| cam2 | hidden_items + product_switching |
| cam3 | fake_scan_detection |

**Full Workload Testing** (`camera_to_workload_full.json`):
**Full Workload Testing** (`camera_to_workload.json`):

| Camera | Workload |
| :----- | :--------------------------- |
Expand All @@ -85,16 +75,16 @@ make benchmark
| :---------------- | :---------------------------------- | :-------------------------------------- |
| **CPU Only** | `workload_to_pipeline_cpu.json` | Testing, development environments |
| **GPU Only** | `workload_to_pipeline_gpu.json` | Production, high performance |
| **Mixed GPU/NPU** | `workload_to_pipeline_gpu-npu.json` | Latest Intel hardware |
| **Mixed GPU/NPU** | `workload_to_pipeline_gpu-npu.json` | Assign selected workloads to GPU or NPU |
| **Heterogeneous** | `workload_to_pipeline_hetero.json` | Maximum performance across all hardware |
| **Default Mixed** | `workload_to_pipeline.json` | Balanced CPU/GPU/NPU distribution |
| **Default Mixed** | `workload_to_pipeline.json` | CPU/GPU/NPU distribution used by default |

## Advanced Performance Testing (15-30 minutes)

### GPU Performance Testing

```bash
make benchmark WORKLOAD_DIST=workload_to_pipeline_gpu.json CAMERA_STREAM=camera_to_workload_full.json
make benchmark WORKLOAD_DIST=workload_to_pipeline_gpu.json CAMERA_STREAM=camera_to_workload.json
```

### Multi-Pipeline Stress Testing
Expand All @@ -111,7 +101,7 @@ make PIPELINE_COUNT=4 benchmark

```bash
# Test heterogeneous workload distribution
make benchmark WORKLOAD_DIST=workload_to_pipeline_hetero.json CAMERA_STREAM=camera_to_workload_full.json REGISTRY=false
make benchmark WORKLOAD_DIST=workload_to_pipeline_hetero.json CAMERA_STREAM=camera_to_workload.json REGISTRY=false
```

### Automated Self-Checkout Performance
Expand Down Expand Up @@ -158,34 +148,58 @@ make benchmark-stream-density

**Default behavior:**

- Tests until FPS of any stream drops below its own target FPS from config JSON (`targetFps`); if not set, default `14.95` is used.
- Requires every stream to reach at least 95% of its effective target FPS; by default targets come from each camera's `targetFps`, then `fps`, with a fallback of `15`.
- Uses OOM protection to prevent system crashes
- Reports maximum sustainable pipeline count

### Custom Target FPS

Set per-stream target FPS using `lane_config.cameras[].targetFps` in `camera_to_workload_*.json`.
Set the target FPS for each camera using the `targetFps` field in the selected `camera_to_workload_*.json` file. Choose the value based on workload criticality and the source FPS.

If `targetFps` is not defined for a camera (or is not a positive value), the benchmark uses that camera's `fps` field instead. If neither field provides a positive value, it falls back to the default target FPS of `15`.

To override the camera configuration, set the `TARGET_FPS` environment variable. When supplied, `TARGET_FPS` applies the same target to every stream and takes precedence over both `targetFps` and `fps`.

Set `targetFps` based on workload criticality and source FPS.
In summary, the effective target FPS for each camera is resolved in this order:

If `targetFps` is not present for a stream (or is invalid/non-positive), the stream falls back to `TARGET_FPS`.
1. `TARGET_FPS` environment variable, when explicitly supplied (overrides the camera configuration for every stream).
2. The camera's `targetFps` in `camera_to_workload_*.json`.
3. The camera's `fps`, used when `targetFps` is missing or not a positive value.
4. The default target FPS, `15`, when neither camera field provides a positive value.

The benchmark passes when each stream reaches at least 95% of its effective target. The resolved target FPS and its source are recorded in `stream_density.log`.

CLI examples:

```bash
# Test for different performance thresholds
make TARGET_FPS=13.5 benchmark-stream-density

# With custom pipeline configuration
make PIPELINE_SCRIPT=yolo11n_effnetb0.sh TARGET_FPS=13.5 benchmark-stream-density
# With explicit camera and workload configurations
make CAMERA_STREAM=camera_to_workload.json WORKLOAD_DIST=workload_to_pipeline_gpu.json TARGET_FPS=13.5 benchmark-stream-density
```

### Measurement Window & Settle Time

`MEASUREMENT_WINDOW_SECONDS` controls how long FPS samples are collected for each density step. Its default is **100 seconds**, and values below 100 are rejected because the benchmark requires at least 100 samples per pipeline log. Each step adds a lane after one passing window, the final count is confirmed by two passing windows in a row, and a lane is only given up after two failing windows in a row. Each window is evaluated independently; windows are not combined into one longer window.

`INIT_DURATION` sets the settle time, defaulting to **10 seconds**, before measurement starts. A longer settle time lets the pipelines stabilize more, and a longer measurement window can reduce short-term measurement noise, but both increase the total benchmark duration. The resolved settle time is also printed in the stream-density result summary.

```sh
make benchmark-stream-density \
INIT_DURATION=120 \
MEASUREMENT_WINDOW_SECONDS=120
```

### Stream Density Environment Variables

| Variable | Description | Values |
| :--------------- | :---------------------------- | :-------------------------------- |
| `TARGET_FPS` | Minimum FPS threshold | `14.95` (default), `13.5`, `20.0` |
| `OOM_PROTECTION` | Prevent out-of-memory crashes | `1` (enabled), `0` (disabled) |
| Variable | Description | Values |
| :-------------------------- | :------------------------------------------------------ | :-------------------------------- |
| `TARGET_FPS` | Override the target FPS for all streams; unset uses camera configuration | Unset by default, `13.5`, `20.0` |
| `MEASUREMENT_WINDOW_SECONDS` | Duration FPS samples are collected per density step (minimum 100 seconds) | `100` (default), `120` |
| `INIT_DURATION` | Settle time before measurement starts | `10` (default), `120`, `60` |
| `OOM_PROTECTION` | Prevent out-of-memory crashes | `1` (enabled), `0` (disabled) |
| `PASS_TOLERANCE_RATIO` | Fraction of each target FPS required to pass | `0.95` (default), up to `1.0` |

> **Warning:** Setting `OOM_PROTECTION=0` may crash your system requiring a hard reboot.

Expand All @@ -194,7 +208,7 @@ Note: stream-density caches camera-target mappings using `(config_path, file_mti
### Expected Output

```
Total averaged FPS per stream: 15.210442307692306 for 26 pipeline(s)
Stream density completed: use case density <lanes> lanes, stream density <streams> streams. For the detailed report, see benchmark/stream_density.log.
```

## Visualization & Analysis
Expand All @@ -220,16 +234,14 @@ make clean-containers # Remove stopped containers
make clean-all # Remove all unused Docker resources
```

make clean-all # Remove all unused Docker resources

````
## Custom Configuration (Advanced)

### Creating Custom Workloads

The application is highly configurable via JSON files in the `configs/` directory:

#### Camera Configuration (`camera_to_workload.json`)

```json
{
"lane_config": {
Expand All @@ -244,32 +256,22 @@ The application is highly configurable via JSON files in the `configs/` director
]
}
}
````
```

#### Pipeline Configuration (`workload_to_pipeline.json`)

```json
{
"workload_pipeline_map": {
"items_in_basket": [
{
"type": "gvadetect",
"model": "yolo11n",
"precision": "INT8",
"device": "CPU"
},
{
"type": "gvaclassify",
"model": "efficientnet-v2-b0",
"precision": "INT8",
"device": "CPU"
}
{"type": "gvadetect", "model": "yolo11n", "precision": "INT8", "device": "CPU"},
{"type": "gvaclassify", "model": "efficientnet-v2-b0", "precision": "INT8", "device": "CPU"}
]
}
}
```

### Add Custom Workloads
### To Add Custom Workloads:

1. Edit `configs/camera_to_workload.json` to add your camera and assign workloads
2. Edit `configs/workload_to_pipeline.json` to define the pipeline for your workload
Expand All @@ -285,11 +287,11 @@ The `workload_to_pipeline_hetero.json` distributes workloads across multiple pro

| Workload | Object Detection | Classification | Inference |
| :--------------------------- | :--------------- | :------------- | :-------- |
| items_in_basket | GPU | GPU | - |
| hidden_items | GPU | CPU | - |
| fake_scan_detection | GPU | CPU | - |
| multi_product_identification | GPU | CPU | - |
| product_switching | GPU | GPU | - |
| items_in_basket | CPU | CPU | - |
| hidden_items | GPU | GPU | - |
| fake_scan_detection | GPU | GPU | - |
| multi_product_identification | NPU | NPU | - |
| product_switching | CPU | CPU | - |
| sweet_heartening | NPU | - | NPU |

### Mixed Configuration Details
Expand All @@ -306,7 +308,7 @@ The `workload_to_pipeline.json` balances workloads across available hardware:
- `docker/` — Dockerfiles for containers
- `download-scripts/` — Model and video download scripts
- `src/` — Pipeline runner scripts
- `performance-tools/benchmark-scripts/results/` — Performance test results
- `benchmark/` — Benchmark logs, consolidated metrics, and plots (default `RESULTS_DIR`)
- `Makefile` — Build automation commands

---
Expand Down Expand Up @@ -340,7 +342,6 @@ NO_PROXY=localhost,127.0.0.1,::1
```

#### Docker Proxy Configuration

Create `/etc/systemd/system/docker.service.d/http-proxy.conf`:

```bash
Expand All @@ -355,4 +356,4 @@ Environment="no_proxy=localhost,127.0.0.1,::1"
# Reload and restart
sudo systemctl daemon-reload
sudo systemctl restart docker
```
```
Loading
Loading