feat: notifications to webhooks, Ascender and AWX

CHANGELOG.md

@@ -6,6 +6,10 @@ project adheres to [Semantic Versioning](https://semver.org/).
 
 ## [Unreleased]
 
+### Added
+
+- Notifications: a `notifications` item sends `drift_detected`, `out_of_sync`, `sync_succeeded`, `sync_failed` and `back_in_sync` to a webhook (any receiver, with bearer, basic or header auth, a body template and optional HMAC signing) or to an Ascender or AWX job template (its webhook receiver or its launch API), one message per run, queued and retried, never with a field's value; `GET /api/v1/notifications` shows each one's last send and `POST /api/v1/notifications/{id}/test` sends a test (see `docs/notifications.md`).
+
 ### Changed
 
 - **Breaking (who runs the image):** the image no longer bundles a demo configuration or connectors (`/app/config`, `/app/connectors`); mount your configuration at `/app/config` or set `CONFIG_DIR`, and take connectors from a project such as [unified-connectors](https://github.com/OpusProjects/unified-connectors), which now holds the SDK and the `kv` example.

CLAUDE.md

@@ -45,14 +45,17 @@ src/
 │   ├── desired.rs            # desired-state files: parse, load a folder, validate, order
 │   ├── plan.rs               # THE DIFF: only declared fields compared → Operation list
 │   ├── run.rs                # SyncStatus, RunRecord, PlanSnapshot, AppStateRegistry
+│   ├── notification.rs       # Notification: type (webhook/ascender/awx), events, its block
 │   ├── yaml.rs               # YAML → JSON with exact scalars ("0644" stays a string)
 │   ├── credential.rs, api_key.rs, ldap.rs, project.rs, sync_health.rs
 ├── application/
 │   ├── reconcile.rs          # THE LOOP: validate → list → copy → diff → apply → re-list
 │   ├── projects.rs           # pull a checkout (bounded by timeout_seconds)
+│   ├── notify.rs             # event per run (detect), message, request per type, queue + worker
 │   ├── scripts.rs            # per-run script path resolution into checkouts
 │   ├── credentials.rs, login.rs, config.rs (reload)
-├── ports/                    # connector, state_store, git, secrets, directory, config_store, venv
+├── ports/                    # connector, state_store, git, secrets, directory, config_store, venv,
+│                             #   notifier (sends one built HTTP request)
 └── adapters/
     ├── in/http/              # applications (+ state copies), connectors, projects,
     │                         #   credentials, settings, users (auth), config, login, session,
@@ -61,6 +64,7 @@ src/
     └── out/
         ├── connectors/process.rs  # ConnectorPort → script (scrubbed env, APPLICATION_CONFIG, timeout kills)
         ├── state/fs.rs         # StateStorePort → <state.dir>/<application>/<kind>.yaml
+        ├── notify/http.rs      # HttpSenderPort → reqwest (no redirects followed)
         ├── git/, secrets/, directory/, config/, python.rs, process_env.rs
 tests/                        # integration tests; tests/connectors/ holds test connector scripts
                               #   (kv.py + the SDK, copied from unified-connectors)

README.md

@@ -27,6 +27,7 @@ network, and a change made by hand on a system shows up as drift.
 - **Self-describing connectors**: a script says what it manages, each type's key and what goes first
 - **One application, one system**: its address and credentials live in config, never in git
 - **Run history**: which commit was applied, when, and the result of every operation
+- **Notifications**: drift, applied commits and failures to a webhook or an Ascender or AWX job
 - **Config as code**: one item per file, any folders; `name` is the identity
 - **Strict validation**: a typo in desired state or config fails before any system is touched
 - **Web UI and LDAP sign-in**: the same UI family and sign-in as unified-api, on its own address
@@ -41,6 +42,7 @@ network, and a change made by hand on a system shows up as drift.
 | [How it works](docs/how-it-works.md)      | The reconcile loop: pull, list, diff, apply, and what each status means |
 | [Connectors](docs/connectors.md)                | The script contract, the Python SDK, and writing a connector               |
 | [Configuration](docs/configuration.md)    | Every YAML file field by field, and desired-state files                 |
+| [Notifications](docs/notifications.md)    | Drift, syncs and failures to a webhook or an Ascender or AWX job template |
 | [API](docs/api.md)                        | All routes with authentication and examples                            |
 | [Architecture](docs/architecture.md)      | The layers, the dependency rule, and what is shared with unified-api   |
 | [Configuration API](docs/config-api.md)   | Pushing the config directory over HTTP and reloading it live            |

docs/api.md

@@ -18,6 +18,8 @@ session also need its `X-CSRF-Token`). Every failure is `{"error": "..."}`.
 | `/api/v1/applications/{id}/state/{kind}` | GET | key granted the app  | The system's current settings, as of its last check |
 | `/api/v1/search?q=`                   | GET    | any key (its apps)   | Find an object by key or value: its application, kind, whether git declares it |
 | `/api/v1/connectors`                     | GET    | admin                | Connectors: script, project, timeout, who uses them |
+| `/api/v1/notifications`               | GET    | admin                | Notifications: events, destination, last send      |
+| `/api/v1/notifications/{id}/test`     | POST   | admin                | Send a test message now, once; answers the outcome |
 | `/api/v1/projects`                    | GET    | admin                | Git checkouts, their health, what uses them        |
 | `/api/v1/projects/{id}/sync`          | POST   | admin                | Pull a project now                                 |
 | `/api/v1/credentials`                 | GET    | admin                | Credentials: where from, whether they resolve      |

docs/configuration.md

@@ -20,8 +20,10 @@ as startup does, and exits.
 │   └── desired-state.yaml
 ├── credentials/
 │   └── cred-isilon.yaml
-└── api_keys/
-    └── key-pipeline.yaml
+├── api_keys/
+│   └── key-pipeline.yaml
+└── notifications/
+    └── ntf-failures.yaml
 ```
 
 | Top-level key  | Holds                                                         |
@@ -31,6 +33,7 @@ as startup does, and exits.
 | `projects`     | Git repositories: desired state, and connector scripts           |
 | `credentials`  | Where each credential's secrets come from (never the values)  |
 | `api_keys`     | API keys by name and role (the values come from env vars)     |
+| `notifications` | What runs report, and where: webhooks, Ascender and AWX job templates |
 
 - **The key says what an item is, never the folder.** A file in `connectors/` holding `applications:` is
   read as an application.
@@ -169,6 +172,23 @@ api_keys:
 `UNIFIED_GITOPS_KEY`, if set, is one more admin key. With no keys at all the API is open, which is
 logged loudly at startup.
 
+## notifications
+
+Which events, from which applications, go where. The full reference is [notifications.md](notifications.md):
+
+```yaml
+# notifications/ntf-ascender-drift.yaml
+notifications:
+  - name: ntf-ascender-drift
+    type: ascender                     # webhook | ascender | awx
+    events: [drift_detected]
+    applications: [isilon-prod]        # empty = every application
+    ascender:
+      url: "https://ascender.example.com"
+      job_template: 123
+      credentials: [cred-ascender-jt123]   # its token is the template's webhook key
+```
+
 ## Desired-state files
 
 In the application's folder, one kind per file:

docs/notifications.md

@@ -0,0 +1,154 @@
+# Notifications
+
+A notification tells another system what happened to an application: drift found, a commit applied,
+a sync that failed. It is a configuration item like the others, under `notifications:`. Every send
+goes out from this instance, so nothing has to reach in.
+
+- [Events](#events)
+- [Types](#types)
+- [The message](#the-message)
+- [Ascender and AWX](#ascender-and-awx)
+- [Sending, retries and what is kept](#sending-retries-and-what-is-kept)
+
+---
+
+## Events
+
+Each run produces at most one event, so one change never arrives twice under two names.
+
+| Event | When | Fires |
+| --- | --- | --- |
+| `drift_detected` | Differences at a commit already applied: someone changed the system outside git | On entering drift; with `auto_sync`, every revert |
+| `out_of_sync` | A new commit waits to be applied (`auto_sync` off) | On entering the state |
+| `sync_succeeded` | A sync applied at least one change from git: a commit or a rollback | Every such sync |
+| `sync_failed` | A check or sync could not finish, or some operations failed | On entering the state |
+| `back_in_sync` | In sync again after a failure or a difference | On entering the state |
+
+- **Drift and commits are told apart by commit:** differences found at the commit the application
+  was last in sync at were made on the system; at a newer commit, they come from git.
+- **With `auto_sync`, a reverted drift is one `drift_detected`** with `reverted: true`, not a
+  `drift_detected` plus a `sync_succeeded`.
+- **History is in memory:** after a restart, the first run fires only failures and applied
+  changes, so a restart does not report every pending difference again.
+
+## Types
+
+The `type` names the receiver, and its settings live in the block of the same name.
+
+```yaml
+# notifications/ntf-failures.yaml
+notifications:
+  - name: ntf-failures
+    description: "Any application that fails"
+    type: webhook
+    events: [sync_failed, back_in_sync]
+    applications: []                     # empty = every application
+    webhook:
+      url: "https://alerts.example.com/hooks/unified-gitops"
+      method: POST                       # POST (default), PUT or PATCH
+      credentials: [cred-alerts]
+      auth: bearer                       # bearer (default) | basic | none | {header: X-Name}
+      headers:
+        X-Source: unified-gitops
+      # body: '{"text": "{{application}}: {{event}}"}'   # optional; default is the message
+      # sign: {}                         # optional HMAC-SHA256 of the body
+      timeout_seconds: 30
+      retries: 3
+```
+
+| `auth` | What is sent | Credential field |
+| --- | --- | --- |
+| `bearer`, or left out with a credential | `Authorization: Bearer <token>` | `token` |
+| `basic` | `Authorization: Basic …` | `username`, `password` |
+| `{header: X-Name}` | `X-Name: <token>` | `token` |
+| `none` | nothing | — |
+
+- **`body`** is a template: `{{field}}` takes any field of [the message](#the-message), plus
+  `{{revision_short}}`. Text is written escaped for a JSON string (write the quotes yourself); lists,
+  numbers and booleans as JSON. A name the message does not have fails the configuration.
+- **`sign`** adds `X-Hub-Signature-256: sha256=<hex>` over the body, from the credential field
+  `secret`. `header`, `prefix` and `secret_field` change those.
+- **Never in the YAML:** a token, a password, or a URL that carries one. They belong in
+  credentials, which come from env vars or Vault.
+
+The `ascender` and `awx` types start a job template; see [Ascender and AWX](#ascender-and-awx).
+
+## The message
+
+The default body, and the fields a `body` template can use:
+
+```json
+{
+  "event": "drift_detected",
+  "notification": "ntf-ascender-drift",
+  "application": "portainer",
+  "description": "Portainer settings, environments and users",
+  "status": "synced",
+  "previous_status": "synced",
+  "cause": "drift",
+  "reverted": true,
+  "revision": "39d6efb30883db193fcda905e0aa3def7b5d1f30",
+  "applied": 1,
+  "changes": [
+    {"kind": "user", "key": "someone.else", "op": "delete", "fields": [], "ok": true}
+  ],
+  "errors": [],
+  "at": "2026-10-26T15:42:10Z",
+  "url": "https://gitops.example.com/applications/portainer"
+}
+```
+
+- **`changes`** names each object, its operation and the names of the fields that changed, never
+  their values: a value can be a secret, and the message leaves this instance.
+- **`cause`** is `commit`, `drift` or `rollback`; **`reverted`** is set on `drift_detected` only.
+- **`errors`**: the first five lines of a failure, for `sync_failed`.
+- **`url`** links the application in the UI when `server.url` is set.
+
+## Ascender and AWX
+
+Both start a job template (or a workflow job template) in two ways. Their routes are the same today;
+they are separate types so each follows its own product.
+
+```yaml
+# notifications/ntf-ascender-drift.yaml
+notifications:
+  - name: ntf-ascender-drift
+    description: "Drift report in Ascender"
+    type: ascender                       # or awx, with an awx: block
+    events: [drift_detected]
+    applications: [portainer]
+    ascender:
+      url: "https://ascender.example.com"   # the controller, without /api
+      job_template: 123                  # or workflow_job_template: 45
+      trigger: webhook                   # webhook (default) | launch
+      credentials: [cred-ascender-jt123]
+      # extra_vars: {report: drift}      # launch only
+```
+
+| `trigger` | Route | The credential's `token` | The playbook reads the message as |
+| --- | --- | --- | --- |
+| `webhook` | `/api/v2/job_templates/{id}/gitlab/` | The template's webhook key | `ascender_webhook_payload` (Ascender), `tower_webhook_payload` (both) |
+| `launch` | `/api/v2/job_templates/{id}/launch/` | A user's OAuth token | `unified_gitops`, beside `extra_vars` |
+
+- **`webhook`** needs the template's **Webhooks** option on, with service **GitLab**: the
+  controller then gives it a key, compared as is, and no user is needed. Any event is accepted.
+- **`launch`** needs a user allowed to run the template, and **Prompt on launch** for its variables.
+  Without it the controller drops `extra_vars`; the send reports that as a failure instead of
+  starting a job without the event.
+- The last send records the job a `launch` started.
+
+## Sending, retries and what is kept
+
+- **A queue, not the run:** a run queues its event and moves on; a worker sends, eight at a time,
+  so a slow receiver never holds up a check or a sync. A full queue (1000) drops the newest message,
+  counted in the metrics.
+- **Retries:** an unanswered send, a `429` or a `5xx` is retried `retries` times, waiting 2, 4, 8 …
+  up to 60 seconds. A `4xx` is not: a wrong key or template does not fix itself.
+- **Redirects are not followed:** the body and its credential go only where the configuration says.
+- **What is kept:** each notification's last send (event, application, result, status, job,
+  error), in memory, on `GET /api/v1/notifications`. `POST /api/v1/notifications/{id}/test`
+  sends a test message now, once, and answers the outcome.
+- **Metrics:** `unified_gitops_notifications_total{notification,event,outcome}` (`success`,
+  `error`, `dropped`) and `unified_gitops_notification_duration_seconds`.
+- **Reload:** notifications reload like applications. A message keeps the definition it was queued
+  with.

docs/observability.md

@@ -27,6 +27,8 @@ stops running pushes nothing, so only a scrape-time gauge shows the silence.
 | `unified_gitops_project_sync_last_success_age_seconds`    | project                  | Seconds since the last good pull              |
 | `unified_gitops_connector_calls_total`                       | application, connector, command, outcome | Calls to the systems: `list` reads, `apply` writes, `describe` starts a run |
 | `unified_gitops_connector_call_duration_seconds`             | application, connector, command | How long each call to a system took (histogram) |
+| `unified_gitops_notifications_total`                      | notification, event, outcome | Messages sent (`success`), failed after retries (`error`), or dropped by a full queue |
+| `unified_gitops_notification_duration_seconds`            | notification             | How long a send took, retries included (histogram) |
 | `unified_gitops_http_requests_total`                      | method, path, status     | Requests, by matched route                    |
 | `unified_gitops_http_request_duration_seconds`            | method, path             | Latency histogram                             |
 | `unified_gitops_config_generation`                        |                          | Bumped by every applied reload                |

src/adapters/in/http/config.rs

@@ -53,6 +53,7 @@ pub struct ConfigSummary {
     pub credentials: usize,
     pub projects: usize,
     pub api_keys: usize,
+    pub notifications: usize,
 }
 
 #[derive(Serialize, ToSchema)]
@@ -116,6 +117,7 @@ pub struct ReloadInfo {
     pub connectors: DeltaInfo,
     pub projects: DeltaInfo,
     pub credentials: DeltaInfo,
+    pub notifications: DeltaInfo,
     /// How many API keys are in force after the reload
     pub api_keys: usize,
 }
@@ -251,6 +253,7 @@ fn summarize(cfg: &AppConfig) -> ConfigSummary {
         credentials: cfg.credentials.len(),
         projects: cfg.projects.len(),
         api_keys: cfg.api_keys.len(),
+        notifications: cfg.notifications.len(),
     }
 }
 
@@ -263,6 +266,7 @@ fn reload_info(report: &reload_use_case::ReloadReport, api_keys: usize) -> Reloa
         connectors: (&report.connectors).into(),
         projects: (&report.projects).into(),
         credentials: (&report.credentials).into(),
+        notifications: (&report.notifications).into(),
         api_keys,
     }
 }

src/adapters/in/http/index.rs

@@ -30,6 +30,9 @@ pub struct ApiIndex {
     /// Admin-only; absent for a restricted key
     #[serde(skip_serializing_if = "Option::is_none")]
     pub connectors: Option<&'static str>,
+    /// Admin-only; absent for a restricted key
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub notifications: Option<&'static str>,
     /// Admin-only route; absent for a restricted key
     #[serde(skip_serializing_if = "Option::is_none")]
     pub projects: Option<&'static str>,
@@ -71,6 +74,7 @@ pub async fn api_index(
         applications: "/api/v1/applications",
         search: "/api/v1/search",
         connectors: admin.then_some("/api/v1/connectors"),
+        notifications: admin.then_some("/api/v1/notifications"),
         projects: admin.then_some("/api/v1/projects"),
         // Same condition the config handlers enforce: the routes exist in the
         // router either way, and answer 403 when no store is configured.

src/adapters/in/http/mod.rs

@@ -12,6 +12,7 @@ pub mod html;
 pub mod index;
 pub mod login;
 pub mod metrics;
+pub mod notifications;
 pub mod openapi;
 pub mod projects;
 pub mod routes;