diff --git a/README.md b/README.md index 62b8fde..e33be75 100644 --- a/README.md +++ b/README.md @@ -191,6 +191,13 @@ The `itential.monitoring` uses the following community Prometheus exporter roles - [`prometheus.prometheus.node_exporter`](https://prometheus-community.github.io/ansible/branch/main/node_exporter_role.html#ansible-collections-prometheus-prometheus-node-exporter-role) - [`prometheus.prometheus.process_exporter`](https://prometheus-community.github.io/ansible/branch/main/process_exporter_role.html#ansible-collections-prometheus-prometheus-process-exporter-role) +In addition, `itential.monitoring` ships its own exporter role — the `itential.monitoring.itential_platform_exporter` role installs the +[itential-job-metrics-exporter](https://github.com/itential/job-metrics-exporter), which connects +directly to the Itential Platform MongoDB replica set and exposes job/task lifecycle metrics. +Unlike the exporters above, it is downloaded from GitHub releases rather than installed via the +`prometheus.prometheus` collection. Refer to the [role README](roles/itential_platform_exporter/README.md) +for full configuration details and MongoDB prerequisites. + Each exporter is a lightweight Go application that exposes the metrics on a standard HTTP endpoint. Each exporter requires a port to be opened so that Prometheus can access the metrics that are being exposed by the exporter. The following ports will be utilized by these roles: | Exporter | Default Port | Description | @@ -199,6 +206,7 @@ Each exporter is a lightweight Go application that exposes the metrics on a stan | [process exporter](https://github.com/ncabatoff/process-exporter) | 9256 | The process exporter is installed on `platform` and `gateway` hosts and will expose individual processes from Itential Platform and IAG. | | [mongodb exporter](https://github.com/percona/mongodb_exporter) | 9216 | The mongo exporter is installed on `mongodb` hosts and will expose information about the MongoDB installation and any replica sets. | | [redis exporter](https://github.com/oliver006/redis_exporter) | 9121 | The redis exporter is installed on `redis` hosts and will expose information about the Redis installation and any replica sets. | +| [itential platform exporter](https://github.com/itential/job-metrics-exporter) | 9477 | The Itential Platform exporter is installed on `itential_platform_exporter` hosts and will expose job/task lifecycle metrics read from the Itential Platform MongoDB replica set. | #### Process Exporter Notes @@ -267,8 +275,9 @@ All Prometheus variables are handled by the `prometheus.prometheus.prometheus` r ### Exporters Role Variables -All exporter variables are handled by the exporter roles. Refer to the documentation links in the -[Exporter Roles](#exporter-roles) section. +All community exporter variables are handled by the exporter roles. Refer to the documentation +links in the [Exporter Roles](#exporter-roles) section. The `itential_platform_exporter` role's +variables are its own — see [Itential Platform Exporter Role Variables](#itential-platform-exporter-role-variables) below. ### Redis Exporter Requirements @@ -337,6 +346,21 @@ all: | `alloy_tls_enabled` | Boolean | Enable TLS for the Alloy → Loki push connection | `false` | | `alloy_tls_ca_file` | String | Path to the CA certificate used to verify the Loki server cert (required when `alloy_tls_enabled: true`) | `/etc/alloy/certs/ca.crt` | +### Itential Platform Exporter Role Variables + +| Variable | Type | Description | Default Value | +| :------- | :--- | :---------- | :------------ | +| `itential_platform_exporter_version` | String | Exporter version to install | `1.0.1` | +| `itential_platform_exporter_listen_address` | String | Address/port the exporter listens on | `:9477` | +| `itential_platform_exporter_mongo_uri` | String | Full MongoDB connection string (takes precedence over the individual `_mongo_host`/`_mongo_port`/... fields) | `""` | +| `itential_platform_exporter_mongo_password` | String | MongoDB password — **must be set via Ansible Vault** | `""` | +| `itential_platform_exporter_tls_enabled` | Boolean | Enable TLS on the exporter's own HTTP listener | `false` | +| `itential_platform_exporter_change_stream_enabled` | Boolean | Enable real-time MongoDB change stream counters (requires a replica set) | `true` | +| `itential_platform_exporter_polling_enabled` | Boolean | Enable background aggregation polling (use when change streams are unavailable) | `false` | + +See the [role README](roles/itential_platform_exporter/README.md) for the full variable list and +required MongoDB setup steps (dedicated read-only user and indexes). + ## Building Your Inventory ### ELK Stack Inventory @@ -461,7 +485,8 @@ all: ### Prometheus / Grafana Inventory To install and configure Prometheus and Grafana, add `prometheus` and `grafana` groups and hosts to -your inventory (in addition to the other Itential-related groups and hosts). +your inventory (in addition to the other Itential-related groups and hosts). To also install the +Itential Platform Exporter, add an `itential_platform_exporter` group and host. ### Example Inventory @@ -511,6 +536,12 @@ all: - python3.9 {% endraw %} + itential_platform_exporter: + hosts: + : + vars: + itential_platform_exporter_mongo_uri: "mongodb://prometheus:@:27017,:27017/itential?replicaSet=rs0&authSource=admin" + prometheus: hosts: : @@ -663,4 +694,5 @@ You can also selectively execute portions of the role by using the following tag | `process_exporter_install` | This will execute the tasks to install the process exporter. The process exporter is installed on `platform` and `gateway` hosts. | | `mongodb_exporter_install` | This will execute the tasks to install the mongo exporter. The mongo exporter is installed on `mongodb` hosts. | | `redis_exporter_install` | This will execute the tasks to install the redis exporter. The redis exporter is installed on `redis` hosts. | +| `itential_platform_exporter_install` | This will execute the tasks to install the Itential Platform exporter. It is installed on `itential_platform_exporter` hosts. | | `grafana_install` | This will execute the tasks to install Grafana. | diff --git a/playbooks/prometheus_exporters.yml b/playbooks/prometheus_exporters.yml index ff2cd7d..3a43d76 100644 --- a/playbooks/prometheus_exporters.yml +++ b/playbooks/prometheus_exporters.yml @@ -94,6 +94,13 @@ - ansible_facts.services["firewalld.service"].state == "running" - ansible_facts.services["firewalld.service"].status == "enabled" +- name: Install Itential Platform Exporter + hosts: itential_platform_exporter + become: true + roles: + - role: itential.monitoring.itential_platform_exporter + tags: itential_platform_exporter_install + - name: Install process exporter hosts: platform*, gateway become: true diff --git a/roles/itential_platform_exporter/CLAUDE.md b/roles/itential_platform_exporter/CLAUDE.md new file mode 100644 index 0000000..cb901a7 --- /dev/null +++ b/roles/itential_platform_exporter/CLAUDE.md @@ -0,0 +1,79 @@ +# itential_platform_exporter role + +## Purpose +Installs the [itential-job-metrics-exporter](https://github.com/itential/job-metrics-exporter) — +a Prometheus exporter that connects directly to the Itential Platform MongoDB replica set and +exposes job/task lifecycle metrics (start/complete/error/cancel counters and status gauges). +Downloaded as a prebuilt binary from GitHub releases (not built from source), deployed with a +systemd service. This is unrelated to Itential Platform's own built-in `/prometheus_metrics` +endpoint (the `iap_exporter` scrape job in the `prometheus` role) — it is a separate process +that reads directly from MongoDB. + +## Key files +| File | Purpose | +|---|---| +| `defaults/main.yml` | All role variables with defaults | +| `tasks/main.yml` | Three blocks: `itential_platform_exporter_install`, `itential_platform_exporter_configure`, `always` | +| `templates/config.yaml.j2` | Exporter config — MongoDB connection, TLS, logging, change stream, polling | +| `templates/itential-job-metrics-exporter.service.j2` | Systemd service unit with hardening options | +| `handlers/main.yml` | Single handler: `Restart Itential Platform Exporter` | + +## Task block structure +`tasks/main.yml` is split into three tagged blocks: + +- **`itential_platform_exporter_install`** — creates a dedicated system user/group, checks the + installed binary version via `--version`, downloads the release binary only if the pinned + version differs, opens the exporter port in firewalld if the service is running. +- **`itential_platform_exporter_configure`** — templates `config.yaml` and the systemd unit to + disk; both notify the restart handler. +- **`always`** — flushes handlers, starts/enables the service, asserts `ActiveState == active`, + then polls `{{ itential_platform_exporter_metrics_path }}` (HTTPS when + `itential_platform_exporter_tls_enabled`) with 12 retries (5s delay). + +## Binary install +Unlike `loki` (zip archive), the upstream release assets are raw binaries — one file per arch, +no archive. The role maps `ansible_architecture` (`aarch64` → `arm64`, everything else → `amd64`) +and downloads directly to `{{ itential_platform_exporter_install_dir }}`. The version check +compares `itential_platform_exporter_version` against the installed `--version` output string; +the download/install task is skipped if it already matches. + +## MongoDB prerequisites (not automated by this role) +The exporter requires a MongoDB **replica set** and a dedicated read-only user, plus two indexes +on the `itential` database. These must be created manually — this role only installs and +configures the exporter process itself: + +```javascript +use admin +db.createUser({ + user: "prometheus", + pwd: "", + roles: [{ role: "read", db: "itential" }] +}) +``` + +```javascript +db.jobs.createIndex({ status: 1, _id: 1 }, { name: "itential_status", background: true }) +db.tasks.createIndex( + { status: 1, "metrics.server_id": 1 }, + { name: "itential_job_metrics_exporter_task_status_server", background: true } +) +``` + +The `jobs` index may already exist — it is typically created by the Itential Platform +application itself. + +## Variables that should be overridden +- `itential_platform_exporter_version` — pin to a specific release; defaults to `1.0.1` +- `itential_platform_exporter_mongo_uri` — full connection string (recommended for replica + sets); when unset, the individual `itential_platform_exporter_mongo_host`/`_port`/`_username`/ + `_password`/`_database`/`_auth_source` fields are used instead +- `itential_platform_exporter_mongo_password` — **must be set via Ansible Vault**; empty by default +- `itential_platform_exporter_change_stream_enabled` — requires MongoDB running as a replica + set; set `itential_platform_exporter_polling_enabled: true` instead (and disable change + stream) if it is not + +## Not wired into the Prometheus scrape config +This role does not add a scrape job to the `prometheus` role's `scrape_configs.j2`. If the +Itential Platform Exporter should be scraped, that template needs a new target list (e.g. hosts +in a dedicated group, or the `platform` group) — this is a separate change to the `prometheus` +role and was intentionally left out of scope here. diff --git a/roles/itential_platform_exporter/README.md b/roles/itential_platform_exporter/README.md new file mode 100644 index 0000000..17ff94d --- /dev/null +++ b/roles/itential_platform_exporter/README.md @@ -0,0 +1,117 @@ +# itential.monitoring.itential_platform_exporter + +Installs and configures the [itential-job-metrics-exporter](https://github.com/itential/job-metrics-exporter), +a Prometheus exporter that connects directly to the Itential Platform MongoDB replica set and +exposes job/task lifecycle metrics. Deployed as a binary (downloaded from GitHub releases) +running under a dedicated systemd service. + +## Requirements + +- RHEL/Rocky Linux 8/9, Amazon Linux 2023, or Oracle Linux 8/9 (amd64 or arm64) +- MongoDB 4.2+ running as a **replica set** (required for change streams) +- A dedicated MongoDB read-only user on the `itential` database, plus two indexes — see + [MongoDB Setup](#mongodb-setup) below (not automated by this role) +- Outbound internet access to `github.com` for binary download (or pre-stage the binary) +- Firewalld managed automatically if the service is running + +## MongoDB Setup + +Create a read-only user: + +```javascript +use admin +db.createUser({ + user: "prometheus", + pwd: "", + roles: [{ role: "read", db: "itential" }] +}) +``` + +Create the required indexes on the `itential` database: + +```javascript +db.jobs.createIndex({ status: 1, _id: 1 }, { name: "itential_status", background: true }) +db.tasks.createIndex( + { status: 1, "metrics.server_id": 1 }, + { name: "itential_job_metrics_exporter_task_status_server", background: true } +) +``` + +The `jobs` index (`itential_status`) may already exist — it is typically created by the +Itential Platform application itself. + +## Role Variables + +| Variable | Type | Description | Default | +|----------|------|-------------|---------| +| `itential_platform_exporter_version` | String | Exporter version to install | `1.0.1` | +| `itential_platform_exporter_listen_address` | String | Address/port the exporter listens on | `:9477` | +| `itential_platform_exporter_metrics_path` | String | Metrics endpoint path | `/metrics` | +| `itential_platform_exporter_config_dir` | String | Config directory | `/etc/itential-job-metrics-exporter` | +| `itential_platform_exporter_install_dir` | String | Directory for the binary | `/usr/local/bin` | +| `itential_platform_exporter_user` | String | System user the exporter runs as | `itential_platform_exporter` | +| `itential_platform_exporter_group` | String | System group | `itential_platform_exporter` | +| `itential_platform_exporter_service_name` | String | Systemd service name | `itential-job-metrics-exporter` | +| `itential_platform_exporter_mongo_uri` | String | Full MongoDB connection string (takes precedence over the individual fields below) | `""` | +| `itential_platform_exporter_mongo_host` | String | MongoDB host (used when `_mongo_uri` is unset) | `localhost` | +| `itential_platform_exporter_mongo_port` | Integer | MongoDB port | `27017` | +| `itential_platform_exporter_mongo_username` | String | MongoDB username | `prometheus` | +| `itential_platform_exporter_mongo_password` | String | MongoDB password — **must be set via Ansible Vault** | `""` | +| `itential_platform_exporter_mongo_database` | String | MongoDB database name | `itential` | +| `itential_platform_exporter_mongo_auth_source` | String | MongoDB auth source database | `admin` | +| `itential_platform_exporter_mongo_tls_enabled` | Boolean | Enable TLS for the MongoDB connection | `false` | +| `itential_platform_exporter_mongo_tls_ca_file` | String | Path to the CA file used to verify MongoDB's cert | `/etc/itential-job-metrics-exporter/ca.pem` | +| `itential_platform_exporter_tls_enabled` | Boolean | Enable TLS on the exporter's HTTP listener | `false` | +| `itential_platform_exporter_tls_cert_file` | String | Path to the exporter server certificate | `/etc/itential-job-metrics-exporter/server.crt` | +| `itential_platform_exporter_tls_key_file` | String | Path to the exporter server private key | `/etc/itential-job-metrics-exporter/server.key` | +| `itential_platform_exporter_log_level` | String | Log level (`debug`\|`info`\|`warn`\|`error`) | `info` | +| `itential_platform_exporter_log_format` | String | Log format (`json`\|`text`) | `json` | +| `itential_platform_exporter_change_stream_enabled` | Boolean | Enable real-time MongoDB change stream counters (requires a replica set) | `true` | +| `itential_platform_exporter_polling_enabled` | Boolean | Enable background aggregation polling (use when change streams are unavailable) | `false` | +| `itential_platform_exporter_polling_interval` | String | Polling interval | `60s` | + +## Tags + +| Tag | Description | +|-----|-------------| +| `itential_platform_exporter_install` | Install binary, create user/group/config dir, open firewall port | +| `itential_platform_exporter_configure` | Deploy `config.yaml` and systemd service file | + +## TLS + +Both the MongoDB connection and the exporter's own HTTP listener support TLS independently. +Certificates must be pre-placed on the host before running the playbook — this role does not +deploy them. + +```yaml +itential_platform_exporter: + vars: + itential_platform_exporter_tls_enabled: true + itential_platform_exporter_tls_cert_file: /etc/itential-job-metrics-exporter/server.crt + itential_platform_exporter_tls_key_file: /etc/itential-job-metrics-exporter/server.key +``` + +## Inventory + +Add an `itential_platform_exporter` group to your inventory, and set the MongoDB password via +Ansible Vault: + +```yaml +itential_platform_exporter: + hosts: + : + vars: + itential_platform_exporter_mongo_uri: "mongodb://prometheus:{{ vault_itential_platform_exporter_mongo_password }}@:27017,:27017,:27017/itential?replicaSet=rs0&authSource=admin" +``` + +## Playbook + +```bash +ansible-playbook itential.monitoring.itential_platform_exporter -i +``` + +## Not wired into the Prometheus scrape config + +This role does not add a scrape job for the exporter to the `prometheus` role's +`scrape_configs.j2`. If you want Prometheus to scrape it automatically, that template needs a +new target list — this is a separate change outside the scope of this role. diff --git a/roles/itential_platform_exporter/defaults/main.yml b/roles/itential_platform_exporter/defaults/main.yml new file mode 100644 index 0000000..eb2c46a --- /dev/null +++ b/roles/itential_platform_exporter/defaults/main.yml @@ -0,0 +1,62 @@ +# Copyright (c) 2026, Itential, Inc +# GNU General Public License v3.0+ (see LICENSE or https://www.gnu.org/licenses/gpl-3.0.txt) +--- +itential_platform_exporter_version: "1.0.1" +itential_platform_exporter_arch: "{{ 'arm64' if ansible_architecture == 'aarch64' else 'amd64' }}" + +# Network +itential_platform_exporter_listen_address: ":9477" +itential_platform_exporter_metrics_path: "/metrics" +itential_platform_exporter_port: "{{ itential_platform_exporter_listen_address.split(':') | last }}" + +# Filesystem paths +itential_platform_exporter_config_dir: /etc/itential-job-metrics-exporter +itential_platform_exporter_install_dir: /usr/local/bin +itential_platform_exporter_binary_name: itential-job-metrics-exporter + +# System user/group +itential_platform_exporter_user: itential_platform_exporter +itential_platform_exporter_group: itential_platform_exporter + +# Service +itential_platform_exporter_service_name: itential-job-metrics-exporter + +# MongoDB connection. +# Set itential_platform_exporter_mongo_uri to use a full connection string (recommended +# for replica sets). When unset, the individual host/port/username/... fields below are used. +itential_platform_exporter_mongo_uri: "" +itential_platform_exporter_mongo_host: "localhost" +itential_platform_exporter_mongo_port: 27017 +itential_platform_exporter_mongo_username: "prometheus" +itential_platform_exporter_mongo_password: "" +itential_platform_exporter_mongo_database: "itential" +itential_platform_exporter_mongo_auth_source: "admin" + +# MongoDB TLS +itential_platform_exporter_mongo_tls_enabled: false +itential_platform_exporter_mongo_tls_ca_file: "{{ itential_platform_exporter_config_dir }}/ca.pem" +itential_platform_exporter_mongo_tls_insecure_skip_verify: false + +# Exporter TLS — server-side TLS for the HTTP listener. +# Certs must be pre-placed on the host; this role does not deploy them. +itential_platform_exporter_tls_enabled: false +itential_platform_exporter_tls_cert_file: "{{ itential_platform_exporter_config_dir }}/server.crt" +itential_platform_exporter_tls_key_file: "{{ itential_platform_exporter_config_dir }}/server.key" + +# Logging +itential_platform_exporter_log_level: "info" +itential_platform_exporter_log_format: "json" + +# Timeout for the one-time bootstrap poll run at startup when change_stream is enabled +itential_platform_exporter_slow_query_timeout: "4m" + +# Real-time change stream counters. Requires MongoDB to run as a replica set. +itential_platform_exporter_change_stream_enabled: true +itential_platform_exporter_change_stream_initial_load: true +itential_platform_exporter_change_stream_initial_load_timeout: "2m" + +# Background aggregation polling. Use when change streams are not available +# (MongoDB not running as a replica set). Ignored when change_stream is enabled. +itential_platform_exporter_polling_enabled: false +itential_platform_exporter_polling_interval: "60s" +itential_platform_exporter_polling_query_timeout: "55s" diff --git a/roles/itential_platform_exporter/handlers/main.yml b/roles/itential_platform_exporter/handlers/main.yml new file mode 100644 index 0000000..d4a7636 --- /dev/null +++ b/roles/itential_platform_exporter/handlers/main.yml @@ -0,0 +1,9 @@ +# Copyright (c) 2026, Itential, Inc +# GNU General Public License v3.0+ (see LICENSE or https://www.gnu.org/licenses/gpl-3.0.txt) +--- +- name: Restart Itential Platform Exporter + ansible.builtin.systemd: + name: "{{ itential_platform_exporter_service_name }}" + state: restarted + enabled: true + daemon_reload: true diff --git a/roles/itential_platform_exporter/tasks/main.yml b/roles/itential_platform_exporter/tasks/main.yml new file mode 100644 index 0000000..2428991 --- /dev/null +++ b/roles/itential_platform_exporter/tasks/main.yml @@ -0,0 +1,111 @@ +# Copyright (c) 2026, Itential, Inc +# GNU General Public License v3.0+ (see LICENSE or https://www.gnu.org/licenses/gpl-3.0.txt) +--- +- name: Install Itential Platform Exporter + tags: itential_platform_exporter_install + block: + - name: Create itential_platform_exporter group + ansible.builtin.group: + name: "{{ itential_platform_exporter_group }}" + system: true + + - name: Create itential_platform_exporter user + ansible.builtin.user: + name: "{{ itential_platform_exporter_user }}" + group: "{{ itential_platform_exporter_group }}" + system: true + shell: /sbin/nologin + home: "{{ itential_platform_exporter_config_dir }}" + create_home: false + + - name: Ensure config directory exists + ansible.builtin.file: + path: "{{ itential_platform_exporter_config_dir }}" + state: directory + owner: "{{ itential_platform_exporter_user }}" + group: "{{ itential_platform_exporter_group }}" + mode: "0750" + + - name: Check installed Itential Platform Exporter version + ansible.builtin.command: + cmd: "{{ itential_platform_exporter_install_dir }}/{{ itential_platform_exporter_binary_name }} --version" + register: itential_platform_exporter_installed_version + changed_when: false + failed_when: false + + - name: Download and install Itential Platform Exporter binary + when: itential_platform_exporter_version not in (itential_platform_exporter_installed_version.stdout | default('')) + block: + - name: Download Itential Platform Exporter binary + ansible.builtin.get_url: + url: "https://github.com/itential/job-metrics-exporter/releases/download/v{{ itential_platform_exporter_version }}/itential-job-metrics-exporter-linux-{{ itential_platform_exporter_arch }}" + dest: "{{ itential_platform_exporter_install_dir }}/{{ itential_platform_exporter_binary_name }}" + owner: root + group: root + mode: "0755" + notify: Restart Itential Platform Exporter + + - name: Gather service facts + ansible.builtin.service_facts: + + - name: Open Itential Platform Exporter port in firewall + ansible.posix.firewalld: + port: "{{ itential_platform_exporter_port }}/tcp" + permanent: true + state: enabled + zone: public + immediate: true + when: + - ansible_facts.services["firewalld.service"] is defined + - ansible_facts.services["firewalld.service"].state == "running" + - ansible_facts.services["firewalld.service"].status == "enabled" + +- name: Configure Itential Platform Exporter + tags: itential_platform_exporter_configure + block: + - name: Deploy config.yaml + ansible.builtin.template: + src: config.yaml.j2 + dest: "{{ itential_platform_exporter_config_dir }}/config.yaml" + owner: "{{ itential_platform_exporter_user }}" + group: "{{ itential_platform_exporter_group }}" + mode: "0640" + notify: Restart Itential Platform Exporter + + - name: Deploy Itential Platform Exporter systemd service + ansible.builtin.template: + src: itential-job-metrics-exporter.service.j2 + dest: "/etc/systemd/system/{{ itential_platform_exporter_service_name }}.service" + owner: root + group: root + mode: "0644" + notify: Restart Itential Platform Exporter + +- name: Ensure Itential Platform Exporter is running + tags: always + block: + - name: Flush all handlers + ansible.builtin.meta: flush_handlers + + - name: Enable and start Itential Platform Exporter + ansible.builtin.systemd: + name: "{{ itential_platform_exporter_service_name }}" + state: started + enabled: true + daemon_reload: true + + - name: Assert that Itential Platform Exporter is running + ansible.builtin.systemd: + name: "{{ itential_platform_exporter_service_name }}" + register: itential_platform_exporter_status + failed_when: itential_platform_exporter_status.status.ActiveState != "active" + + - name: Wait for Itential Platform Exporter to be ready + ansible.builtin.uri: + url: "{{ 'https' if itential_platform_exporter_tls_enabled | bool else 'http' }}://localhost:{{ itential_platform_exporter_port }}{{ itential_platform_exporter_metrics_path }}" + validate_certs: false + status_code: 200 + register: itential_platform_exporter_ready + until: itential_platform_exporter_ready.status == 200 + retries: 12 + delay: 5 diff --git a/roles/itential_platform_exporter/templates/config.yaml.j2 b/roles/itential_platform_exporter/templates/config.yaml.j2 new file mode 100644 index 0000000..aa3a3e0 --- /dev/null +++ b/roles/itential_platform_exporter/templates/config.yaml.j2 @@ -0,0 +1,41 @@ +{{ ansible_managed | comment }} + +mongo: +{% if itential_platform_exporter_mongo_uri %} + uri: "{{ itential_platform_exporter_mongo_uri }}" +{% else %} + host: "{{ itential_platform_exporter_mongo_host }}" + port: {{ itential_platform_exporter_mongo_port }} + username: "{{ itential_platform_exporter_mongo_username }}" + password: "{{ itential_platform_exporter_mongo_password }}" + database: "{{ itential_platform_exporter_mongo_database }}" + auth_source: "{{ itential_platform_exporter_mongo_auth_source }}" +{% endif %} + + tls: + enabled: {{ itential_platform_exporter_mongo_tls_enabled | bool | lower }} + ca_file: "{{ itential_platform_exporter_mongo_tls_ca_file }}" + insecure_skip_verify: {{ itential_platform_exporter_mongo_tls_insecure_skip_verify | bool | lower }} + +exporter: + listen_address: "{{ itential_platform_exporter_listen_address }}" + metrics_path: "{{ itential_platform_exporter_metrics_path }}" + slow_query_timeout: "{{ itential_platform_exporter_slow_query_timeout }}" + tls: + enabled: {{ itential_platform_exporter_tls_enabled | bool | lower }} + cert_file: "{{ itential_platform_exporter_tls_cert_file }}" + key_file: "{{ itential_platform_exporter_tls_key_file }}" + +log: + level: "{{ itential_platform_exporter_log_level }}" # debug | info | warn | error + format: "{{ itential_platform_exporter_log_format }}" # json | text + +change_stream: + enabled: {{ itential_platform_exporter_change_stream_enabled | bool | lower }} + initial_load: {{ itential_platform_exporter_change_stream_initial_load | bool | lower }} + initial_load_timeout: "{{ itential_platform_exporter_change_stream_initial_load_timeout }}" + +polling: + enabled: {{ itential_platform_exporter_polling_enabled | bool | lower }} + interval: "{{ itential_platform_exporter_polling_interval }}" + query_timeout: "{{ itential_platform_exporter_polling_query_timeout }}" diff --git a/roles/itential_platform_exporter/templates/itential-job-metrics-exporter.service.j2 b/roles/itential_platform_exporter/templates/itential-job-metrics-exporter.service.j2 new file mode 100644 index 0000000..030ece3 --- /dev/null +++ b/roles/itential_platform_exporter/templates/itential-job-metrics-exporter.service.j2 @@ -0,0 +1,47 @@ +{{ ansible_managed | comment }} + +[Unit] +Description=Itential Job Metrics Prometheus Exporter +Documentation=https://github.com/itential/job-metrics-exporter +After=network-online.target +Wants=network-online.target + +[Service] +Type=simple +User={{ itential_platform_exporter_user }} +Group={{ itential_platform_exporter_group }} + +ExecStart={{ itential_platform_exporter_install_dir }}/{{ itential_platform_exporter_binary_name }} \ + --config {{ itential_platform_exporter_config_dir }}/config.yaml + +# Restart on failure. StartLimitBurst caps restarts to 5 attempts per +# StartLimitIntervalSec window to avoid hammering MongoDB during an outage. +Restart=on-failure +RestartSec=10s +StartLimitIntervalSec=120s +StartLimitBurst=5 + +StandardOutput=journal +StandardError=journal +SyslogIdentifier=itential-job-metrics-exporter + +# Security hardening — safe for a stateless HTTP server process that only +# needs outbound TCP to MongoDB and inbound TCP for scrapes. +NoNewPrivileges=true +ProtectSystem=strict +ProtectHome=true +PrivateTmp=true +PrivateDevices=true +ProtectKernelTunables=true +ProtectKernelModules=true +ProtectControlGroups=true +RestrictAddressFamilies=AF_INET AF_INET6 +RestrictNamespaces=true +LockPersonality=true +MemoryDenyWriteExecute=true +CapabilityBoundingSet= +AmbientCapabilities= +ReadOnlyPaths={{ itential_platform_exporter_config_dir }} + +[Install] +WantedBy=multi-user.target diff --git a/roles/prometheus/CLAUDE.md b/roles/prometheus/CLAUDE.md index ea1a732..7741878 100644 --- a/roles/prometheus/CLAUDE.md +++ b/roles/prometheus/CLAUDE.md @@ -19,6 +19,7 @@ Thin wrapper around the community `prometheus.prometheus.prometheus` role that a | `gateway` | `node_exporter`, `process_exporter` | | `mongodb` | `node_exporter`, `mongo_exporter` | | `redis_*` (all groups matching `^redis_.*`) | `node_exporter`, `redis_exporter` | +| `itential_platform_exporter` | `itential_platform_exporter` | | `vault` | `node_exporter` | The scrape config file is written to `{{ prometheus_config_dir }}/scrape_configs/itential.yml`. `prometheus_config_dir` is provided by the upstream `prometheus.prometheus.prometheus` role. @@ -29,6 +30,9 @@ Each exporter target uses `inventory_hostname:default_port` unless the host defi - `process_exporter_web_listen_address` - `redis_exporter_web_listen_address` - `mongodb_exporter_web_listen_address` +- `itential_platform_exporter_web_listen_address` — note this is a scrape-target-only override, + distinct from the `itential_platform_exporter` role's own `itential_platform_exporter_listen_address` + variable (which configures the exporter's actual bind address, e.g. `:9477`) When the override is set, its full value (host:port) is used as-is. diff --git a/roles/prometheus/defaults/main.yml b/roles/prometheus/defaults/main.yml index 258f46e..720d0de 100644 --- a/roles/prometheus/defaults/main.yml +++ b/roles/prometheus/defaults/main.yml @@ -6,3 +6,4 @@ prometheus_node_exporter_web_listen_port: 9100 prometheus_redis_exporter_web_listen_port: 9121 prometheus_mongodb_exporter_web_listen_port: 9216 prometheus_itential_platform_web_listen_port: 3000 +prometheus_itential_platform_exporter_web_listen_port: 9477 diff --git a/roles/prometheus/templates/scrape_configs.j2 b/roles/prometheus/templates/scrape_configs.j2 index 894ae41..b6e881c 100644 --- a/roles/prometheus/templates/scrape_configs.j2 +++ b/roles/prometheus/templates/scrape_configs.j2 @@ -5,6 +5,7 @@ {%- set node_export_targets = [] %} {%- set mongodb_exporter_targets = [] %} {%- set redis_exporter_targets = [] %} +{%- set itential_platform_exporter_targets = [] %} {%- if 'platform' in groups -%} {% for host in groups['platform'] %} @@ -76,6 +77,17 @@ {% endfor %} {%- endfor %} +{%- if 'itential_platform_exporter' in groups -%} +{% for host in groups['itential_platform_exporter'] %} +{% if 'itential_platform_exporter_web_listen_address' in hostvars[host] %} +{% set itential_platform_exporter_web_listen_address = hostvars[host].itential_platform_exporter_web_listen_address %} +{% else %} +{% set itential_platform_exporter_web_listen_address = host + ":" + prometheus_itential_platform_exporter_web_listen_port | string %} +{% endif %} +{{- itential_platform_exporter_targets.append( itential_platform_exporter_web_listen_address ) -}} +{% endfor %} +{% endif %} + {%- if 'vault' in groups -%} {% for host in groups['vault'] %} {% if 'node_exporter_web_listen_address' in hostvars[host] %} @@ -134,3 +146,12 @@ scrape_configs: {% endfor %} {% endif %} +{% if itential_platform_exporter_targets | length > 0 %} + - job_name: itential_platform_exporter + static_configs: + - targets: +{% for target in itential_platform_exporter_targets %} + - {{ target }} +{% endfor %} + +{% endif %}