Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 35 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -191,6 +191,13 @@ The `itential.monitoring` uses the following community Prometheus exporter roles
- [`prometheus.prometheus.node_exporter`](https://prometheus-community.github.io/ansible/branch/main/node_exporter_role.html#ansible-collections-prometheus-prometheus-node-exporter-role)
- [`prometheus.prometheus.process_exporter`](https://prometheus-community.github.io/ansible/branch/main/process_exporter_role.html#ansible-collections-prometheus-prometheus-process-exporter-role)

In addition, `itential.monitoring` ships its own exporter role — the `itential.monitoring.itential_platform_exporter` role installs the
[itential-job-metrics-exporter](https://github.com/itential/job-metrics-exporter), which connects
directly to the Itential Platform MongoDB replica set and exposes job/task lifecycle metrics.
Unlike the exporters above, it is downloaded from GitHub releases rather than installed via the
`prometheus.prometheus` collection. Refer to the [role README](roles/itential_platform_exporter/README.md)
for full configuration details and MongoDB prerequisites.

Each exporter is a lightweight Go application that exposes the metrics on a standard HTTP endpoint. Each exporter requires a port to be opened so that Prometheus can access the metrics that are being exposed by the exporter. The following ports will be utilized by these roles:

| Exporter | Default Port | Description |
Expand All @@ -199,6 +206,7 @@ Each exporter is a lightweight Go application that exposes the metrics on a stan
| [process exporter](https://github.com/ncabatoff/process-exporter) | 9256 | The process exporter is installed on `platform` and `gateway` hosts and will expose individual processes from Itential Platform and IAG. |
| [mongodb exporter](https://github.com/percona/mongodb_exporter) | 9216 | The mongo exporter is installed on `mongodb` hosts and will expose information about the MongoDB installation and any replica sets. |
| [redis exporter](https://github.com/oliver006/redis_exporter) | 9121 | The redis exporter is installed on `redis` hosts and will expose information about the Redis installation and any replica sets. |
| [itential platform exporter](https://github.com/itential/job-metrics-exporter) | 9477 | The Itential Platform exporter is installed on `itential_platform_exporter` hosts and will expose job/task lifecycle metrics read from the Itential Platform MongoDB replica set. |

#### Process Exporter Notes

Expand Down Expand Up @@ -267,8 +275,9 @@ All Prometheus variables are handled by the `prometheus.prometheus.prometheus` r

### Exporters Role Variables

All exporter variables are handled by the exporter roles. Refer to the documentation links in the
[Exporter Roles](#exporter-roles) section.
All community exporter variables are handled by the exporter roles. Refer to the documentation
links in the [Exporter Roles](#exporter-roles) section. The `itential_platform_exporter` role's
variables are its own — see [Itential Platform Exporter Role Variables](#itential-platform-exporter-role-variables) below.

### Redis Exporter Requirements

Expand Down Expand Up @@ -337,6 +346,21 @@ all:
| `alloy_tls_enabled` | Boolean | Enable TLS for the Alloy → Loki push connection | `false` |
| `alloy_tls_ca_file` | String | Path to the CA certificate used to verify the Loki server cert (required when `alloy_tls_enabled: true`) | `/etc/alloy/certs/ca.crt` |

### Itential Platform Exporter Role Variables

| Variable | Type | Description | Default Value |
| :------- | :--- | :---------- | :------------ |
| `itential_platform_exporter_version` | String | Exporter version to install | `1.0.1` |
| `itential_platform_exporter_listen_address` | String | Address/port the exporter listens on | `:9477` |
| `itential_platform_exporter_mongo_uri` | String | Full MongoDB connection string (takes precedence over the individual `_mongo_host`/`_mongo_port`/... fields) | `""` |
| `itential_platform_exporter_mongo_password` | String | MongoDB password — **must be set via Ansible Vault** | `""` |
| `itential_platform_exporter_tls_enabled` | Boolean | Enable TLS on the exporter's own HTTP listener | `false` |
| `itential_platform_exporter_change_stream_enabled` | Boolean | Enable real-time MongoDB change stream counters (requires a replica set) | `true` |
| `itential_platform_exporter_polling_enabled` | Boolean | Enable background aggregation polling (use when change streams are unavailable) | `false` |

See the [role README](roles/itential_platform_exporter/README.md) for the full variable list and
required MongoDB setup steps (dedicated read-only user and indexes).

## Building Your Inventory

### ELK Stack Inventory
Expand Down Expand Up @@ -461,7 +485,8 @@ all:
### Prometheus / Grafana Inventory

To install and configure Prometheus and Grafana, add `prometheus` and `grafana` groups and hosts to
your inventory (in addition to the other Itential-related groups and hosts).
your inventory (in addition to the other Itential-related groups and hosts). To also install the
Itential Platform Exporter, add an `itential_platform_exporter` group and host.

### Example Inventory

Expand Down Expand Up @@ -511,6 +536,12 @@ all:
- python3.9
{% endraw %}

itential_platform_exporter:
hosts:
<EXPORTER-HOST>:
vars:
itential_platform_exporter_mongo_uri: "mongodb://prometheus:<MONGODB-PROMETHEUS-PASSWORD>@<MONGODB-HOST-1>:27017,<MONGODB-HOST-N>:27017/itential?replicaSet=rs0&authSource=admin"

prometheus:
hosts:
<PROMETHEUS-HOST>:
Expand Down Expand Up @@ -663,4 +694,5 @@ You can also selectively execute portions of the role by using the following tag
| `process_exporter_install` | This will execute the tasks to install the process exporter. The process exporter is installed on `platform` and `gateway` hosts. |
| `mongodb_exporter_install` | This will execute the tasks to install the mongo exporter. The mongo exporter is installed on `mongodb` hosts. |
| `redis_exporter_install` | This will execute the tasks to install the redis exporter. The redis exporter is installed on `redis` hosts. |
| `itential_platform_exporter_install` | This will execute the tasks to install the Itential Platform exporter. It is installed on `itential_platform_exporter` hosts. |
| `grafana_install` | This will execute the tasks to install Grafana. |
7 changes: 7 additions & 0 deletions playbooks/prometheus_exporters.yml
Original file line number Diff line number Diff line change
Expand Up @@ -94,6 +94,13 @@
- ansible_facts.services["firewalld.service"].state == "running"
- ansible_facts.services["firewalld.service"].status == "enabled"

- name: Install Itential Platform Exporter
hosts: itential_platform_exporter
become: true
roles:
- role: itential.monitoring.itential_platform_exporter
tags: itential_platform_exporter_install

- name: Install process exporter
hosts: platform*, gateway
become: true
Expand Down
79 changes: 79 additions & 0 deletions roles/itential_platform_exporter/CLAUDE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
# itential_platform_exporter role

## Purpose
Installs the [itential-job-metrics-exporter](https://github.com/itential/job-metrics-exporter) —
a Prometheus exporter that connects directly to the Itential Platform MongoDB replica set and
exposes job/task lifecycle metrics (start/complete/error/cancel counters and status gauges).
Downloaded as a prebuilt binary from GitHub releases (not built from source), deployed with a
systemd service. This is unrelated to Itential Platform's own built-in `/prometheus_metrics`
endpoint (the `iap_exporter` scrape job in the `prometheus` role) — it is a separate process
that reads directly from MongoDB.

## Key files
| File | Purpose |
|---|---|
| `defaults/main.yml` | All role variables with defaults |
| `tasks/main.yml` | Three blocks: `itential_platform_exporter_install`, `itential_platform_exporter_configure`, `always` |
| `templates/config.yaml.j2` | Exporter config — MongoDB connection, TLS, logging, change stream, polling |
| `templates/itential-job-metrics-exporter.service.j2` | Systemd service unit with hardening options |
| `handlers/main.yml` | Single handler: `Restart Itential Platform Exporter` |

## Task block structure
`tasks/main.yml` is split into three tagged blocks:

- **`itential_platform_exporter_install`** — creates a dedicated system user/group, checks the
installed binary version via `--version`, downloads the release binary only if the pinned
version differs, opens the exporter port in firewalld if the service is running.
- **`itential_platform_exporter_configure`** — templates `config.yaml` and the systemd unit to
disk; both notify the restart handler.
- **`always`** — flushes handlers, starts/enables the service, asserts `ActiveState == active`,
then polls `{{ itential_platform_exporter_metrics_path }}` (HTTPS when
`itential_platform_exporter_tls_enabled`) with 12 retries (5s delay).

## Binary install
Unlike `loki` (zip archive), the upstream release assets are raw binaries — one file per arch,
no archive. The role maps `ansible_architecture` (`aarch64` → `arm64`, everything else → `amd64`)
and downloads directly to `{{ itential_platform_exporter_install_dir }}`. The version check
compares `itential_platform_exporter_version` against the installed `--version` output string;
the download/install task is skipped if it already matches.

## MongoDB prerequisites (not automated by this role)
The exporter requires a MongoDB **replica set** and a dedicated read-only user, plus two indexes
on the `itential` database. These must be created manually — this role only installs and
configures the exporter process itself:

```javascript
use admin
db.createUser({
user: "prometheus",
pwd: "<password>",
roles: [{ role: "read", db: "itential" }]
})
```

```javascript
db.jobs.createIndex({ status: 1, _id: 1 }, { name: "itential_status", background: true })
db.tasks.createIndex(
{ status: 1, "metrics.server_id": 1 },
{ name: "itential_job_metrics_exporter_task_status_server", background: true }
)
```

The `jobs` index may already exist — it is typically created by the Itential Platform
application itself.

## Variables that should be overridden
- `itential_platform_exporter_version` — pin to a specific release; defaults to `1.0.1`
- `itential_platform_exporter_mongo_uri` — full connection string (recommended for replica
sets); when unset, the individual `itential_platform_exporter_mongo_host`/`_port`/`_username`/
`_password`/`_database`/`_auth_source` fields are used instead
- `itential_platform_exporter_mongo_password` — **must be set via Ansible Vault**; empty by default
- `itential_platform_exporter_change_stream_enabled` — requires MongoDB running as a replica
set; set `itential_platform_exporter_polling_enabled: true` instead (and disable change
stream) if it is not

## Not wired into the Prometheus scrape config
This role does not add a scrape job to the `prometheus` role's `scrape_configs.j2`. If the
Itential Platform Exporter should be scraped, that template needs a new target list (e.g. hosts
in a dedicated group, or the `platform` group) — this is a separate change to the `prometheus`
role and was intentionally left out of scope here.
117 changes: 117 additions & 0 deletions roles/itential_platform_exporter/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,117 @@
# itential.monitoring.itential_platform_exporter

Installs and configures the [itential-job-metrics-exporter](https://github.com/itential/job-metrics-exporter),
a Prometheus exporter that connects directly to the Itential Platform MongoDB replica set and
exposes job/task lifecycle metrics. Deployed as a binary (downloaded from GitHub releases)
running under a dedicated systemd service.

## Requirements

- RHEL/Rocky Linux 8/9, Amazon Linux 2023, or Oracle Linux 8/9 (amd64 or arm64)
- MongoDB 4.2+ running as a **replica set** (required for change streams)
- A dedicated MongoDB read-only user on the `itential` database, plus two indexes — see
[MongoDB Setup](#mongodb-setup) below (not automated by this role)
- Outbound internet access to `github.com` for binary download (or pre-stage the binary)
- Firewalld managed automatically if the service is running

## MongoDB Setup

Create a read-only user:

```javascript
use admin
db.createUser({
user: "prometheus",
pwd: "<password>",
roles: [{ role: "read", db: "itential" }]
})
```

Create the required indexes on the `itential` database:

```javascript
db.jobs.createIndex({ status: 1, _id: 1 }, { name: "itential_status", background: true })
db.tasks.createIndex(
{ status: 1, "metrics.server_id": 1 },
{ name: "itential_job_metrics_exporter_task_status_server", background: true }
)
```

The `jobs` index (`itential_status`) may already exist — it is typically created by the
Itential Platform application itself.

## Role Variables

| Variable | Type | Description | Default |
|----------|------|-------------|---------|
| `itential_platform_exporter_version` | String | Exporter version to install | `1.0.1` |
| `itential_platform_exporter_listen_address` | String | Address/port the exporter listens on | `:9477` |
| `itential_platform_exporter_metrics_path` | String | Metrics endpoint path | `/metrics` |
| `itential_platform_exporter_config_dir` | String | Config directory | `/etc/itential-job-metrics-exporter` |
| `itential_platform_exporter_install_dir` | String | Directory for the binary | `/usr/local/bin` |
| `itential_platform_exporter_user` | String | System user the exporter runs as | `itential_platform_exporter` |
| `itential_platform_exporter_group` | String | System group | `itential_platform_exporter` |
| `itential_platform_exporter_service_name` | String | Systemd service name | `itential-job-metrics-exporter` |
| `itential_platform_exporter_mongo_uri` | String | Full MongoDB connection string (takes precedence over the individual fields below) | `""` |
| `itential_platform_exporter_mongo_host` | String | MongoDB host (used when `_mongo_uri` is unset) | `localhost` |
| `itential_platform_exporter_mongo_port` | Integer | MongoDB port | `27017` |
| `itential_platform_exporter_mongo_username` | String | MongoDB username | `prometheus` |
| `itential_platform_exporter_mongo_password` | String | MongoDB password — **must be set via Ansible Vault** | `""` |
| `itential_platform_exporter_mongo_database` | String | MongoDB database name | `itential` |
| `itential_platform_exporter_mongo_auth_source` | String | MongoDB auth source database | `admin` |
| `itential_platform_exporter_mongo_tls_enabled` | Boolean | Enable TLS for the MongoDB connection | `false` |
| `itential_platform_exporter_mongo_tls_ca_file` | String | Path to the CA file used to verify MongoDB's cert | `/etc/itential-job-metrics-exporter/ca.pem` |
| `itential_platform_exporter_tls_enabled` | Boolean | Enable TLS on the exporter's HTTP listener | `false` |
| `itential_platform_exporter_tls_cert_file` | String | Path to the exporter server certificate | `/etc/itential-job-metrics-exporter/server.crt` |
| `itential_platform_exporter_tls_key_file` | String | Path to the exporter server private key | `/etc/itential-job-metrics-exporter/server.key` |
| `itential_platform_exporter_log_level` | String | Log level (`debug`\|`info`\|`warn`\|`error`) | `info` |
| `itential_platform_exporter_log_format` | String | Log format (`json`\|`text`) | `json` |
| `itential_platform_exporter_change_stream_enabled` | Boolean | Enable real-time MongoDB change stream counters (requires a replica set) | `true` |
| `itential_platform_exporter_polling_enabled` | Boolean | Enable background aggregation polling (use when change streams are unavailable) | `false` |
| `itential_platform_exporter_polling_interval` | String | Polling interval | `60s` |

## Tags

| Tag | Description |
|-----|-------------|
| `itential_platform_exporter_install` | Install binary, create user/group/config dir, open firewall port |
| `itential_platform_exporter_configure` | Deploy `config.yaml` and systemd service file |

## TLS

Both the MongoDB connection and the exporter's own HTTP listener support TLS independently.
Certificates must be pre-placed on the host before running the playbook — this role does not
deploy them.

```yaml
itential_platform_exporter:
vars:
itential_platform_exporter_tls_enabled: true
itential_platform_exporter_tls_cert_file: /etc/itential-job-metrics-exporter/server.crt
itential_platform_exporter_tls_key_file: /etc/itential-job-metrics-exporter/server.key
```

## Inventory

Add an `itential_platform_exporter` group to your inventory, and set the MongoDB password via
Ansible Vault:

```yaml
itential_platform_exporter:
hosts:
<EXPORTER-HOST>:
vars:
itential_platform_exporter_mongo_uri: "mongodb://prometheus:{{ vault_itential_platform_exporter_mongo_password }}@<mongo-host-1>:27017,<mongo-host-2>:27017,<mongo-host-3>:27017/itential?replicaSet=rs0&authSource=admin"
```

## Playbook

```bash
ansible-playbook itential.monitoring.itential_platform_exporter -i <inventory>
```

## Not wired into the Prometheus scrape config

This role does not add a scrape job for the exporter to the `prometheus` role's
`scrape_configs.j2`. If you want Prometheus to scrape it automatically, that template needs a
new target list — this is a separate change outside the scope of this role.
62 changes: 62 additions & 0 deletions roles/itential_platform_exporter/defaults/main.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
# Copyright (c) 2026, Itential, Inc
# GNU General Public License v3.0+ (see LICENSE or https://www.gnu.org/licenses/gpl-3.0.txt)
---
itential_platform_exporter_version: "1.0.1"
itential_platform_exporter_arch: "{{ 'arm64' if ansible_architecture == 'aarch64' else 'amd64' }}"

# Network
itential_platform_exporter_listen_address: ":9477"
itential_platform_exporter_metrics_path: "/metrics"
itential_platform_exporter_port: "{{ itential_platform_exporter_listen_address.split(':') | last }}"

# Filesystem paths
itential_platform_exporter_config_dir: /etc/itential-job-metrics-exporter
itential_platform_exporter_install_dir: /usr/local/bin
itential_platform_exporter_binary_name: itential-job-metrics-exporter

# System user/group
itential_platform_exporter_user: itential_platform_exporter
itential_platform_exporter_group: itential_platform_exporter

# Service
itential_platform_exporter_service_name: itential-job-metrics-exporter

# MongoDB connection.
# Set itential_platform_exporter_mongo_uri to use a full connection string (recommended
# for replica sets). When unset, the individual host/port/username/... fields below are used.
itential_platform_exporter_mongo_uri: ""
itential_platform_exporter_mongo_host: "localhost"
itential_platform_exporter_mongo_port: 27017
itential_platform_exporter_mongo_username: "prometheus"
itential_platform_exporter_mongo_password: ""
itential_platform_exporter_mongo_database: "itential"
itential_platform_exporter_mongo_auth_source: "admin"

# MongoDB TLS
itential_platform_exporter_mongo_tls_enabled: false
itential_platform_exporter_mongo_tls_ca_file: "{{ itential_platform_exporter_config_dir }}/ca.pem"
itential_platform_exporter_mongo_tls_insecure_skip_verify: false

# Exporter TLS — server-side TLS for the HTTP listener.
# Certs must be pre-placed on the host; this role does not deploy them.
itential_platform_exporter_tls_enabled: false
itential_platform_exporter_tls_cert_file: "{{ itential_platform_exporter_config_dir }}/server.crt"
itential_platform_exporter_tls_key_file: "{{ itential_platform_exporter_config_dir }}/server.key"

# Logging
itential_platform_exporter_log_level: "info"
itential_platform_exporter_log_format: "json"

# Timeout for the one-time bootstrap poll run at startup when change_stream is enabled
itential_platform_exporter_slow_query_timeout: "4m"

# Real-time change stream counters. Requires MongoDB to run as a replica set.
itential_platform_exporter_change_stream_enabled: true
itential_platform_exporter_change_stream_initial_load: true
itential_platform_exporter_change_stream_initial_load_timeout: "2m"

# Background aggregation polling. Use when change streams are not available
# (MongoDB not running as a replica set). Ignored when change_stream is enabled.
itential_platform_exporter_polling_enabled: false
itential_platform_exporter_polling_interval: "60s"
itential_platform_exporter_polling_query_timeout: "55s"
Loading