From 130fca1d2fcadfc75a0ac3c7f71c42bb75b2dea3 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Dan=20Gr=C3=B8ndahl?= Date: Mon, 7 Sep 2026 12:01:20 +0200 Subject: [PATCH 1/8] docs: document how the k8s reporter handles job and cronjob pods Closes #387 --- tutorials/report_k8s_envs.md | 44 ++++++++++++++++++++++++++++++++++++ 1 file changed, 44 insertions(+) diff --git a/tutorials/report_k8s_envs.md b/tutorials/report_k8s_envs.md index 8c06ad6..ec9d968 100644 --- a/tutorials/report_k8s_envs.md +++ b/tutorials/report_k8s_envs.md @@ -171,6 +171,50 @@ kosli snapshot k8s k8s-tutorial \ +## What gets reported + +A Kubernetes snapshot is a list of **pods**, not of workload kinds. For each pod the reporter records the pod name, its namespace, its container image digests, its creation timestamp, and its owner references. The reporter has no awareness of `Deployment`, `StatefulSet`, `Job`, or `CronJob` objects — a Job pod is reported exactly like a Deployment pod. + +Only pods in certain phases are reported: + +| Pod phase | Reported | Notes | +| :--- | :---: | :--- | +| `Running` | Yes | | +| `Failed` | Yes | Skipped, with a warning, if any of its containers has no image ID. | +| `Succeeded` | No | Where a completed Job pod ends up. | +| `Pending` | No | No image digests exist yet. | +| `Unknown` | No | | + +Owner references are stored on the snapshot, so the `Job` or `CronJob` that owns a pod does reach Kosli. They are not in the table output of [`kosli get snapshot`](/client_reference/kosli_get_snapshot) — read them from the JSON: + +```shell +kosli get snapshot k8s-tutorial --output json | jq '.artifacts[].pods' +``` + +### Jobs and CronJobs + +Because reporting is driven by pod phase, a `Job` or `CronJob` pod is captured only while it is running: + +* A job that starts and finishes between two snapshots never appears at all. With the Helm chart's default `*/5 * * * *` schedule, a job that completes in under five minutes is likely to be missed. +* A job that happens to be running when a snapshot is taken appears in that snapshot and is gone from the next one. +* A completed job leaves no trace. `Succeeded` pods are never reported, so nothing in the environment records that the run happened. + +Whether a captured run shows up as environment churn depends on the image: + +* If the job runs an image that nothing else in the environment runs, each captured run produces one snapshot where the artifact started and a later one where it exited. +* If the job runs the same image as a long-running workload, only the instance count changes. Instance-count-only differences do not create a snapshot, so the run is invisible. + + +If the job's image was never attested to a Kosli flow, it is reported as an artifact with no provenance. Under an [environment policy](/policy-reference/environment_policy) that requires provenance, snapshots taken while a job was running are non-compliant and snapshots taken between runs are compliant — so compliance appears to flicker. + + +### Keep job pods out of an environment + +The reporter filters by namespace only; there is no way to exclude pods by owner kind. Two options: + +* **Run jobs in their own namespace.** Then either exclude that namespace with `--exclude-namespaces`, or report it to a separate Kosli environment so job churn does not affect the compliance of your long-running workloads. +* **Attest the job to a flow instead.** Environment snapshots answer "what is running right now"; they are the wrong tool for "what ran, when, and did it succeed". Create a [flow](/getting_started/flows) for the job, [begin a trail](/getting_started/trails) for each run, and attest its outcome. Unlike snapshots, this captures every run no matter how briefly it ran. + ## Running multiple reporters If you are considering running more than one reporter against the same cluster, the table below summarizes which setups produce meaningful snapshots and which don't. From 8e6ff7581aa7ff48cc354f2f1385c1f294944f1b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Dan=20Gr=C3=B8ndahl?= Date: Mon, 7 Sep 2026 12:10:06 +0200 Subject: [PATCH 2/8] docs: address review on k8s job/cronjob section --- tutorials/report_k8s_envs.md | 28 ++++++++++++++++++++++------ 1 file changed, 22 insertions(+), 6 deletions(-) diff --git a/tutorials/report_k8s_envs.md b/tutorials/report_k8s_envs.md index ec9d968..e761079 100644 --- a/tutorials/report_k8s_envs.md +++ b/tutorials/report_k8s_envs.md @@ -173,14 +173,16 @@ kosli snapshot k8s k8s-tutorial \ ## What gets reported -A Kubernetes snapshot is a list of **pods**, not of workload kinds. For each pod the reporter records the pod name, its namespace, its container image digests, its creation timestamp, and its owner references. The reporter has no awareness of `Deployment`, `StatefulSet`, `Job`, or `CronJob` objects — a Job pod is reported exactly like a Deployment pod. +The reporter collects **pods**, and is blind to workload kind: it has no awareness of `Deployment`, `StatefulSet`, `Job`, or `CronJob` objects, so a Job pod is reported exactly like a Deployment pod. For each pod it records the pod name, its namespace, its container image digests, its creation timestamp, and its owner references. + +In the snapshot those pods are grouped under the artifact whose image they run — a snapshot is a list of artifacts, each carrying the pods running it. That is why the churn rules below turn on image digests rather than on individual pods. Only pods in certain phases are reported: | Pod phase | Reported | Notes | | :--- | :---: | :--- | | `Running` | Yes | | -| `Failed` | Yes | Skipped, with a warning, if any of its containers has no image ID. | +| `Failed` | Yes* | *Skipped, with a warning, if any of its containers has no image ID. | | `Succeeded` | No | Where a completed Job pod ends up. | | `Pending` | No | No image digests exist yet. | | `Unknown` | No | | @@ -188,7 +190,10 @@ Only pods in certain phases are reported: Owner references are stored on the snapshot, so the `Job` or `CronJob` that owns a pod does reach Kosli. They are not in the table output of [`kosli get snapshot`](/client_reference/kosli_get_snapshot) — read them from the JSON: ```shell -kosli get snapshot k8s-tutorial --output json | jq '.artifacts[].pods' +kosli get snapshot k8s-tutorial \ + --api-token \ + --org \ + --output json | jq '.artifacts[].pods' ``` ### Jobs and CronJobs @@ -208,11 +213,22 @@ Whether a captured run shows up as environment churn depends on the image: If the job's image was never attested to a Kosli flow, it is reported as an artifact with no provenance. Under an [environment policy](/policy-reference/environment_policy) that requires provenance, snapshots taken while a job was running are non-compliant and snapshots taken between runs are compliant — so compliance appears to flicker. -### Keep job pods out of an environment +### Handling job workloads + +The reporter filters by namespace only; there is no way to exclude pods by owner kind. Three options: + +* **Run jobs in their own namespace,** then either leave that namespace out of reporting or give it its own Kosli environment, so job churn does not affect the compliance of your long-running workloads. With the Helm chart, both are per-entry namespace selectors under `reporterConfig.environments`: `excludeNamespaces` on your main entry, plus a second entry whose `namespaces` is the job namespace if you want it reported separately. A second entry is not a second reporter, so the caveats in [Running multiple reporters](#running-multiple-reporters) do not apply. See the [chart configuration reference](/helm/k8s_reporter/configuration). With the CLI, use `--exclude-namespaces`. +* **Waive provenance for the job's image** if you want the job pods in the environment but not the compliance flicker. An environment policy's `artifacts.provenance.exceptions` drops the provenance requirement for artifacts matching a policy expression: -The reporter filters by namespace only; there is no way to exclude pods by owner kind. Two options: + ```yaml + artifacts: + provenance: + required: true + exceptions: + - if: ${{ matches(artifact.name, "^my-job:.*") }} + ``` -* **Run jobs in their own namespace.** Then either exclude that namespace with `--exclude-namespaces`, or report it to a separate Kosli environment so job churn does not affect the compliance of your long-running workloads. + See [environment policy](/policy-reference/environment_policy). * **Attest the job to a flow instead.** Environment snapshots answer "what is running right now"; they are the wrong tool for "what ran, when, and did it succeed". Create a [flow](/getting_started/flows) for the job, [begin a trail](/getting_started/trails) for each run, and attest its outcome. Unlike snapshots, this captures every run no matter how briefly it ran. ## Running multiple reporters From 783dee79fc8dd1ea89c54da5487b3ac598d8e903 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Dan=20Gr=C3=B8ndahl?= Date: Mon, 7 Sep 2026 12:19:24 +0200 Subject: [PATCH 3/8] docs: refine k8s job/cronjob section after second review --- tutorials/report_k8s_envs.md | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/tutorials/report_k8s_envs.md b/tutorials/report_k8s_envs.md index e761079..b39f22b 100644 --- a/tutorials/report_k8s_envs.md +++ b/tutorials/report_k8s_envs.md @@ -200,27 +200,33 @@ kosli get snapshot k8s-tutorial \ Because reporting is driven by pod phase, a `Job` or `CronJob` pod is captured only while it is running: -* A job that starts and finishes between two snapshots never appears at all. With the Helm chart's default `*/5 * * * *` schedule, a job that completes in under five minutes is likely to be missed. +* A job that starts and finishes between two snapshots never appears at all. Roughly, a run is captured about as often as its runtime divides into the snapshot interval — so on the Helm chart's default `*/5 * * * *` schedule a job that runs for a few seconds is almost always missed, while one that runs for most of the interval is almost always caught. * A job that happens to be running when a snapshot is taken appears in that snapshot and is gone from the next one. * A completed job leaves no trace. `Succeeded` pods are never reported, so nothing in the environment records that the run happened. Whether a captured run shows up as environment churn depends on the image: * If the job runs an image that nothing else in the environment runs, each captured run produces one snapshot where the artifact started and a later one where it exited. -* If the job runs the same image as a long-running workload, only the instance count changes. Instance-count-only differences do not create a snapshot, so the run is invisible. +* If the job runs the same image as a long-running workload, only the instance count changes. Instance-count-only differences do not create a snapshot, so the run is invisible — and because the report is discarded, the pod's owner references are not stored either. Owner references only reach Kosli for runs that produce a snapshot. If the job's image was never attested to a Kosli flow, it is reported as an artifact with no provenance. Under an [environment policy](/policy-reference/environment_policy) that requires provenance, snapshots taken while a job was running are non-compliant and snapshots taken between runs are compliant — so compliance appears to flicker. + +The reporter deployed by the Helm chart is itself a `CronJob`, and at the default whole-cluster scope it is running whenever it takes a snapshot — so it appears in its own snapshots. Kosli ignores the reporter's image (`ghcr.io/kosli-dev/cli`) when deciding whether a snapshot is worth saving, so the reporter's own pods never create snapshots or start and exit events. The reporter is *not* exempt from compliance evaluation, though: under a policy that requires provenance it counts as an artifact without provenance — persistently, not intermittently. Leave the reporter's namespace out of reporting, or waive provenance for its image. + + ### Handling job workloads The reporter filters by namespace only; there is no way to exclude pods by owner kind. Three options: -* **Run jobs in their own namespace,** then either leave that namespace out of reporting or give it its own Kosli environment, so job churn does not affect the compliance of your long-running workloads. With the Helm chart, both are per-entry namespace selectors under `reporterConfig.environments`: `excludeNamespaces` on your main entry, plus a second entry whose `namespaces` is the job namespace if you want it reported separately. A second entry is not a second reporter, so the caveats in [Running multiple reporters](#running-multiple-reporters) do not apply. See the [chart configuration reference](/helm/k8s_reporter/configuration). With the CLI, use `--exclude-namespaces`. +* **Run jobs in their own namespace,** then either leave that namespace out of reporting or give it its own Kosli environment, so job churn does not affect the compliance of your long-running workloads. With the Helm chart, both are per-entry namespace selectors under `reporterConfig.environments`: `excludeNamespaces` on your main entry, plus a second entry whose `namespaces` is the job namespace if you want it reported separately. If your main entry already lists `namespaces` or `namespacesRegex`, drop the job namespace from that list instead — the include and exclude selectors are mutually exclusive within one entry. A second entry is not a second reporter, so the caveats in [Running multiple reporters](#running-multiple-reporters) do not apply. See the [chart configuration reference](/helm/k8s_reporter/configuration). With the CLI, use `--exclude-namespaces` when reporting the whole cluster, or simply omit the job namespace from `--namespaces`. * **Waive provenance for the job's image** if you want the job pods in the environment but not the compliance flicker. An environment policy's `artifacts.provenance.exceptions` drops the provenance requirement for artifacts matching a policy expression: ```yaml + _schema: https://docs.kosli.com/schemas/policy/v1 + artifacts: provenance: required: true From 791a21a81929c500cf80b976d7031c5c441e2cee Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Dan=20Gr=C3=B8ndahl?= Date: Mon, 7 Sep 2026 12:25:33 +0200 Subject: [PATCH 4/8] docs: cover failed job pods and reporter namespace in k8s section --- tutorials/report_k8s_envs.md | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-) diff --git a/tutorials/report_k8s_envs.md b/tutorials/report_k8s_envs.md index b39f22b..73267e8 100644 --- a/tutorials/report_k8s_envs.md +++ b/tutorials/report_k8s_envs.md @@ -198,11 +198,12 @@ kosli get snapshot k8s-tutorial \ ### Jobs and CronJobs -Because reporting is driven by pod phase, a `Job` or `CronJob` pod is captured only while it is running: +Because reporting is driven by pod phase, a `Job` or `CronJob` pod is captured only while it is running — or, if it fails, until the failed pod is cleaned up: -* A job that starts and finishes between two snapshots never appears at all. Roughly, a run is captured about as often as its runtime divides into the snapshot interval — so on the Helm chart's default `*/5 * * * *` schedule a job that runs for a few seconds is almost always missed, while one that runs for most of the interval is almost always caught. +* A job that starts and finishes between two snapshots never appears at all. As a rough guide, a run is captured about as often as its runtime divides into the snapshot interval — so on the Helm chart's default `*/5 * * * *` schedule a job that runs for a few seconds is almost always missed, while one that runs for most of the interval is almost always caught. That guide assumes the job starts at an arbitrary point in the interval. A `CronJob` whose schedule shares a period with the reporter's does not sample randomly at all: it is caught on nearly every run or on nearly none, depending on which of the two fires first. * A job that happens to be running when a snapshot is taken appears in that snapshot and is gone from the next one. -* A completed job leaves no trace. `Succeeded` pods are never reported, so nothing in the environment records that the run happened. +* A job that **fails** is the exception. `Failed` is a terminal phase, so the pod stays in it — and keeps being reported — until Kubernetes garbage-collects it: the Job's `ttlSecondsAfterFinished`, or for a `CronJob` the failed Job retained by `failedJobsHistoryLimit` (default `1`). Until then a failed run is not a flicker; it parks a stopped artifact in the environment. +* A job that succeeds leaves no trace. `Succeeded` pods are never reported, so nothing in the environment records that the run happened. Whether a captured run shows up as environment churn depends on the image: @@ -210,11 +211,17 @@ Whether a captured run shows up as environment churn depends on the image: * If the job runs the same image as a long-running workload, only the instance count changes. Instance-count-only differences do not create a snapshot, so the run is invisible — and because the report is discarded, the pod's owner references are not stored either. Owner references only reach Kosli for runs that produce a snapshot. -If the job's image was never attested to a Kosli flow, it is reported as an artifact with no provenance. Under an [environment policy](/policy-reference/environment_policy) that requires provenance, snapshots taken while a job was running are non-compliant and snapshots taken between runs are compliant — so compliance appears to flicker. +If the job's image was never attested to a Kosli flow, it is reported as an artifact with no provenance. Under an [environment policy](/policy-reference/environment_policy) that requires provenance, snapshots taken while a job was running are non-compliant and snapshots taken between runs are compliant — so compliance appears to flicker. A failed run is worse than a flicker: the `Failed` pod is reported in every snapshot until it is garbage-collected, holding the environment non-compliant for as long as it survives. -The reporter deployed by the Helm chart is itself a `CronJob`, and at the default whole-cluster scope it is running whenever it takes a snapshot — so it appears in its own snapshots. Kosli ignores the reporter's image (`ghcr.io/kosli-dev/cli`) when deciding whether a snapshot is worth saving, so the reporter's own pods never create snapshots or start and exit events. The reporter is *not* exempt from compliance evaluation, though: under a policy that requires provenance it counts as an artifact without provenance — persistently, not intermittently. Leave the reporter's namespace out of reporting, or waive provenance for its image. +The reporter deployed by the Helm chart is itself a `CronJob`, and at the default whole-cluster scope it is running whenever it takes a snapshot — so it appears in its own snapshots. Kosli ignores the reporter's image (`ghcr.io/kosli-dev/cli`) when deciding whether a snapshot is worth saving, so the reporter's own pods never create snapshots or start and exit events. The reporter is *not* exempt from compliance evaluation, though: under a policy that requires provenance it counts as an artifact without provenance — persistently, not intermittently. Either waive provenance for its image, or install the reporter into a namespace of its own and exclude that namespace: + +```shell +helm install kosli-reporter kosli/k8s-reporter -n kosli --create-namespace -f tutorial-values.yaml +``` + +The install command earlier on this page has no `-n`, so the reporter shares whichever namespace you are currently in — excluding *that* would drop your own workloads from the environment along with it. ### Handling job workloads From bb4dc94efa8083dd88b57dbe92094ffba0579259 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Dan=20Gr=C3=B8ndahl?= Date: Mon, 7 Sep 2026 12:31:44 +0200 Subject: [PATCH 5/8] docs: fix reporter namespace install and failed-pod lifetime --- tutorials/report_k8s_envs.md | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/tutorials/report_k8s_envs.md b/tutorials/report_k8s_envs.md index 73267e8..1f65dc4 100644 --- a/tutorials/report_k8s_envs.md +++ b/tutorials/report_k8s_envs.md @@ -202,7 +202,7 @@ Because reporting is driven by pod phase, a `Job` or `CronJob` pod is captured o * A job that starts and finishes between two snapshots never appears at all. As a rough guide, a run is captured about as often as its runtime divides into the snapshot interval — so on the Helm chart's default `*/5 * * * *` schedule a job that runs for a few seconds is almost always missed, while one that runs for most of the interval is almost always caught. That guide assumes the job starts at an arbitrary point in the interval. A `CronJob` whose schedule shares a period with the reporter's does not sample randomly at all: it is caught on nearly every run or on nearly none, depending on which of the two fires first. * A job that happens to be running when a snapshot is taken appears in that snapshot and is gone from the next one. -* A job that **fails** is the exception. `Failed` is a terminal phase, so the pod stays in it — and keeps being reported — until Kubernetes garbage-collects it: the Job's `ttlSecondsAfterFinished`, or for a `CronJob` the failed Job retained by `failedJobsHistoryLimit` (default `1`). Until then a failed run is not a flicker; it parks a stopped artifact in the environment. +* A job that **fails** is the exception. `Failed` is a terminal phase, so the pod stays in it — and keeps being reported — for as long as it exists, and nothing removes it by default. `ttlSecondsAfterFinished` is unset unless you set it, so a standalone Job's failed pods survive until the Job is deleted; a `CronJob` always retains its most recent failed Job (`failedJobsHistoryLimit`, default `1`). A Job with `restartPolicy: Never` that keeps failing leaves one `Failed` pod per attempt, up to `backoffLimit` (default `6`), and each is reported. Until they are gone, a failed run is not a flicker: Kosli keeps showing the dead pods' artifact as running in the environment. * A job that succeeds leaves no trace. `Succeeded` pods are never reported, so nothing in the environment records that the run happened. Whether a captured run shows up as environment churn depends on the image: @@ -215,13 +215,17 @@ If the job's image was never attested to a Kosli flow, it is reported as an arti -The reporter deployed by the Helm chart is itself a `CronJob`, and at the default whole-cluster scope it is running whenever it takes a snapshot — so it appears in its own snapshots. Kosli ignores the reporter's image (`ghcr.io/kosli-dev/cli`) when deciding whether a snapshot is worth saving, so the reporter's own pods never create snapshots or start and exit events. The reporter is *not* exempt from compliance evaluation, though: under a policy that requires provenance it counts as an artifact without provenance — persistently, not intermittently. Either waive provenance for its image, or install the reporter into a namespace of its own and exclude that namespace: +The reporter deployed by the Helm chart is itself a `CronJob`, and at the default whole-cluster scope it is running whenever it takes a snapshot — so it appears in its own snapshots. Kosli ignores the reporter's image (`ghcr.io/kosli-dev/cli`) when deciding whether a snapshot is worth saving, so the reporter's own pods never create snapshots or start and exit events. The match is on the image name, so if you override `image.repository` to mirror the reporter image into your own registry, its pods stop being ignored and each run shows up as an artifact starting and exiting. The reporter is *not* exempt from compliance evaluation, though: under a policy that requires provenance it counts as an artifact without provenance — persistently, not intermittently. Either waive provenance for its image, or install the reporter into a namespace of its own and exclude that namespace — subject to the same include/exclude constraint as the [first option below](#handling-job-workloads). + +The API token secret is namespace-scoped, so a dedicated namespace needs its own copy of it. Create both before installing: ```shell -helm install kosli-reporter kosli/k8s-reporter -n kosli --create-namespace -f tutorial-values.yaml +kubectl create namespace kosli +kubectl create secret generic kosli-api-token -n kosli --from-literal=apikey= +helm install kosli-reporter kosli/k8s-reporter -n kosli -f tutorial-values.yaml ``` -The install command earlier on this page has no `-n`, so the reporter shares whichever namespace you are currently in — excluding *that* would drop your own workloads from the environment along with it. +The commands earlier on this page pass no `-n`, so they put the secret and the reporter in whichever namespace you are currently in — usually alongside the workloads you came here to report. Excluding *that* namespace would drop those workloads from the environment too. ### Handling job workloads From 4871515c853484c895f2a0cbfaf467c9708f441f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Dan=20Gr=C3=B8ndahl?= Date: Mon, 7 Sep 2026 14:07:20 +0200 Subject: [PATCH 6/8] docs: move k8s snapshot mechanics out of the tutorial into an explanation page --- config/navigation.json | 1 + tutorials/report_k8s_envs.md | 76 ++------------------- understand_kosli/kubernetes_snapshots.md | 85 ++++++++++++++++++++++++ 3 files changed, 90 insertions(+), 72 deletions(-) create mode 100644 understand_kosli/kubernetes_snapshots.md diff --git a/config/navigation.json b/config/navigation.json index e680d77..642f154 100644 --- a/config/navigation.json +++ b/config/navigation.json @@ -11,6 +11,7 @@ "understand_kosli/risks", "understand_kosli/controls", "understand_kosli/how_kosli_works", + "understand_kosli/kubernetes_snapshots", "understand_kosli/glossary", "understand_kosli/ai_docs_access" ] diff --git a/tutorials/report_k8s_envs.md b/tutorials/report_k8s_envs.md index 1f65dc4..d3a64a7 100644 --- a/tutorials/report_k8s_envs.md +++ b/tutorials/report_k8s_envs.md @@ -173,80 +173,11 @@ kosli snapshot k8s k8s-tutorial \ ## What gets reported -The reporter collects **pods**, and is blind to workload kind: it has no awareness of `Deployment`, `StatefulSet`, `Job`, or `CronJob` objects, so a Job pod is reported exactly like a Deployment pod. For each pod it records the pod name, its namespace, its container image digests, its creation timestamp, and its owner references. +A snapshot lists the **pods** running in scope. The reporter is blind to workload kind — a `Job` pod is reported exactly like a `Deployment` pod — and only pods in the `Running` and `Failed` phases are reported; `Succeeded` and `Pending` pods are not. -In the snapshot those pods are grouped under the artifact whose image they run — a snapshot is a list of artifacts, each carrying the pods running it. That is why the churn rules below turn on image digests rather than on individual pods. +That matters most for short-lived workloads. A `Job` or `CronJob` pod is captured only while it is running, so a job that starts and finishes between two snapshots never appears, and a successful run leaves no trace in the environment. A pod that *fails* is the exception — it keeps being reported until Kubernetes cleans it up. -Only pods in certain phases are reported: - -| Pod phase | Reported | Notes | -| :--- | :---: | :--- | -| `Running` | Yes | | -| `Failed` | Yes* | *Skipped, with a warning, if any of its containers has no image ID. | -| `Succeeded` | No | Where a completed Job pod ends up. | -| `Pending` | No | No image digests exist yet. | -| `Unknown` | No | | - -Owner references are stored on the snapshot, so the `Job` or `CronJob` that owns a pod does reach Kosli. They are not in the table output of [`kosli get snapshot`](/client_reference/kosli_get_snapshot) — read them from the JSON: - -```shell -kosli get snapshot k8s-tutorial \ - --api-token \ - --org \ - --output json | jq '.artifacts[].pods' -``` - -### Jobs and CronJobs - -Because reporting is driven by pod phase, a `Job` or `CronJob` pod is captured only while it is running — or, if it fails, until the failed pod is cleaned up: - -* A job that starts and finishes between two snapshots never appears at all. As a rough guide, a run is captured about as often as its runtime divides into the snapshot interval — so on the Helm chart's default `*/5 * * * *` schedule a job that runs for a few seconds is almost always missed, while one that runs for most of the interval is almost always caught. That guide assumes the job starts at an arbitrary point in the interval. A `CronJob` whose schedule shares a period with the reporter's does not sample randomly at all: it is caught on nearly every run or on nearly none, depending on which of the two fires first. -* A job that happens to be running when a snapshot is taken appears in that snapshot and is gone from the next one. -* A job that **fails** is the exception. `Failed` is a terminal phase, so the pod stays in it — and keeps being reported — for as long as it exists, and nothing removes it by default. `ttlSecondsAfterFinished` is unset unless you set it, so a standalone Job's failed pods survive until the Job is deleted; a `CronJob` always retains its most recent failed Job (`failedJobsHistoryLimit`, default `1`). A Job with `restartPolicy: Never` that keeps failing leaves one `Failed` pod per attempt, up to `backoffLimit` (default `6`), and each is reported. Until they are gone, a failed run is not a flicker: Kosli keeps showing the dead pods' artifact as running in the environment. -* A job that succeeds leaves no trace. `Succeeded` pods are never reported, so nothing in the environment records that the run happened. - -Whether a captured run shows up as environment churn depends on the image: - -* If the job runs an image that nothing else in the environment runs, each captured run produces one snapshot where the artifact started and a later one where it exited. -* If the job runs the same image as a long-running workload, only the instance count changes. Instance-count-only differences do not create a snapshot, so the run is invisible — and because the report is discarded, the pod's owner references are not stored either. Owner references only reach Kosli for runs that produce a snapshot. - - -If the job's image was never attested to a Kosli flow, it is reported as an artifact with no provenance. Under an [environment policy](/policy-reference/environment_policy) that requires provenance, snapshots taken while a job was running are non-compliant and snapshots taken between runs are compliant — so compliance appears to flicker. A failed run is worse than a flicker: the `Failed` pod is reported in every snapshot until it is garbage-collected, holding the environment non-compliant for as long as it survives. - - - -The reporter deployed by the Helm chart is itself a `CronJob`, and at the default whole-cluster scope it is running whenever it takes a snapshot — so it appears in its own snapshots. Kosli ignores the reporter's image (`ghcr.io/kosli-dev/cli`) when deciding whether a snapshot is worth saving, so the reporter's own pods never create snapshots or start and exit events. The match is on the image name, so if you override `image.repository` to mirror the reporter image into your own registry, its pods stop being ignored and each run shows up as an artifact starting and exiting. The reporter is *not* exempt from compliance evaluation, though: under a policy that requires provenance it counts as an artifact without provenance — persistently, not intermittently. Either waive provenance for its image, or install the reporter into a namespace of its own and exclude that namespace — subject to the same include/exclude constraint as the [first option below](#handling-job-workloads). - -The API token secret is namespace-scoped, so a dedicated namespace needs its own copy of it. Create both before installing: - -```shell -kubectl create namespace kosli -kubectl create secret generic kosli-api-token -n kosli --from-literal=apikey= -helm install kosli-reporter kosli/k8s-reporter -n kosli -f tutorial-values.yaml -``` - -The commands earlier on this page pass no `-n`, so they put the secret and the reporter in whichever namespace you are currently in — usually alongside the workloads you came here to report. Excluding *that* namespace would drop those workloads from the environment too. - - -### Handling job workloads - -The reporter filters by namespace only; there is no way to exclude pods by owner kind. Three options: - -* **Run jobs in their own namespace,** then either leave that namespace out of reporting or give it its own Kosli environment, so job churn does not affect the compliance of your long-running workloads. With the Helm chart, both are per-entry namespace selectors under `reporterConfig.environments`: `excludeNamespaces` on your main entry, plus a second entry whose `namespaces` is the job namespace if you want it reported separately. If your main entry already lists `namespaces` or `namespacesRegex`, drop the job namespace from that list instead — the include and exclude selectors are mutually exclusive within one entry. A second entry is not a second reporter, so the caveats in [Running multiple reporters](#running-multiple-reporters) do not apply. See the [chart configuration reference](/helm/k8s_reporter/configuration). With the CLI, use `--exclude-namespaces` when reporting the whole cluster, or simply omit the job namespace from `--namespaces`. -* **Waive provenance for the job's image** if you want the job pods in the environment but not the compliance flicker. An environment policy's `artifacts.provenance.exceptions` drops the provenance requirement for artifacts matching a policy expression: - - ```yaml - _schema: https://docs.kosli.com/schemas/policy/v1 - - artifacts: - provenance: - required: true - exceptions: - - if: ${{ matches(artifact.name, "^my-job:.*") }} - ``` - - See [environment policy](/policy-reference/environment_policy). -* **Attest the job to a flow instead.** Environment snapshots answer "what is running right now"; they are the wrong tool for "what ran, when, and did it succeed". Create a [flow](/getting_started/flows) for the job, [begin a trail](/getting_started/trails) for each run, and attest its outcome. Unlike snapshots, this captures every run no matter how briefly it ran. +For the full model — how job pods affect environment churn and compliance, and what to do about it — see [Kubernetes environment snapshots](/understand_kosli/kubernetes_snapshots). ## Running multiple reporters @@ -274,3 +205,4 @@ From here you can: * Query your environment with [`kosli list snapshots`](/client_reference/kosli_list_snapshots) and [`kosli get snapshot`](/client_reference/kosli_get_snapshot) * [Compare snapshots to see what changed](/client_reference/kosli_diff_snapshots) * Trace a running artifact back to its git commit with the [From commit to production](/tutorials/following_a_git_commit_to_runtime_environments) tutorial +* Understand what snapshots do and don't capture in [Kubernetes environment snapshots](/understand_kosli/kubernetes_snapshots) diff --git a/understand_kosli/kubernetes_snapshots.md b/understand_kosli/kubernetes_snapshots.md new file mode 100644 index 0000000..458cc12 --- /dev/null +++ b/understand_kosli/kubernetes_snapshots.md @@ -0,0 +1,85 @@ +--- +title: 'Kubernetes environment snapshots' +description: 'What the Kosli Kubernetes reporter captures, which pod phases it reports, and what that means for Job and CronJob workloads.' +--- + +A Kubernetes environment snapshot records what was running in your cluster at a moment in time. This page explains what the reporter actually collects, which pods it skips, and why short-lived workloads such as `Job` and `CronJob` behave differently from long-running ones. To set reporting up in the first place, see [Report Kubernetes environments to Kosli](/tutorials/report_k8s_envs). + +## What gets reported + +The reporter collects **pods**, and is blind to workload kind: it has no awareness of `Deployment`, `StatefulSet`, `Job`, or `CronJob` objects, so a Job pod is reported exactly like a Deployment pod. For each pod it records the pod name, its namespace, its container image digests, its creation timestamp, and its owner references. + +In the snapshot those pods are grouped under the artifact whose image they run — a snapshot is a list of artifacts, each carrying the pods running it. That is why the churn rules below turn on image digests rather than on individual pods. + +Only pods in certain phases are reported: + +| Pod phase | Reported | Notes | +| :--- | :---: | :--- | +| `Running` | Yes | | +| `Failed` | Yes* | *Skipped, with a warning, if any of its containers has no image ID. | +| `Succeeded` | No | Where a completed Job pod ends up. | +| `Pending` | No | No image digests exist yet. | +| `Unknown` | No | | + +Owner references are stored on the snapshot, so the `Job` or `CronJob` that owns a pod does reach Kosli. They are not in the table output of [`kosli get snapshot`](/client_reference/kosli_get_snapshot) — read them from the JSON: + +```shell +kosli get snapshot \ + --api-token \ + --org \ + --output json | jq '.artifacts[].pods' +``` + +## Jobs and CronJobs + +Because reporting is driven by pod phase, a `Job` or `CronJob` pod is captured only while it is running — or, if it fails, until the failed pod is cleaned up: + +* A job that starts and finishes between two snapshots never appears at all. As a rough guide, a run is captured about as often as its runtime divides into the snapshot interval — so on the Helm chart's default `*/5 * * * *` schedule a job that runs for a few seconds is almost always missed, while one that runs for most of the interval is almost always caught. That guide assumes the job starts at an arbitrary point in the interval. A `CronJob` whose schedule shares a period with the reporter's does not sample randomly at all: it is caught on nearly every run or on nearly none, depending on which of the two fires first. +* A job that happens to be running when a snapshot is taken appears in that snapshot and is gone from the next one. +* A job that **fails** is the exception. `Failed` is a terminal phase, so the pod stays in it — and keeps being reported — for as long as it exists, and nothing removes it by default. `ttlSecondsAfterFinished` is unset unless you set it, so a standalone Job's failed pods survive until the Job is deleted; a `CronJob` always retains its most recent failed Job (`failedJobsHistoryLimit`, default `1`). A Job with `restartPolicy: Never` that keeps failing leaves one `Failed` pod per attempt, up to `backoffLimit` (default `6`), and each is reported. Until they are gone, a failed run is not a flicker: Kosli keeps showing the dead pods' artifact as running in the environment. +* A job that succeeds leaves no trace. `Succeeded` pods are never reported, so nothing in the environment records that the run happened. + +Whether a captured run shows up as environment churn depends on the image: + +* If the job runs an image that nothing else in the environment runs, each captured run produces one snapshot where the artifact started and a later one where it exited. +* If the job runs the same image as a long-running workload, only the instance count changes. Instance-count-only differences do not create a snapshot, so the run is invisible — and because the report is discarded, the pod's owner references are not stored either. Owner references only reach Kosli for runs that produce a snapshot. + + +If the job's image was never attested to a Kosli flow, it is reported as an artifact with no provenance. Under an [environment policy](/policy-reference/environment_policy) that requires provenance, snapshots taken while a job was running are non-compliant and snapshots taken between runs are compliant — so compliance appears to flicker. A failed run is worse than a flicker: the `Failed` pod is reported in every snapshot until it is garbage-collected, holding the environment non-compliant for as long as it survives. + + + +The reporter deployed by the Helm chart is itself a `CronJob`, and at the default whole-cluster scope it is running whenever it takes a snapshot — so it appears in its own snapshots. Kosli ignores the reporter's image (`ghcr.io/kosli-dev/cli`) when deciding whether a snapshot is worth saving, so the reporter's own pods never create snapshots or start and exit events. The match is on the image name, so if you override `image.repository` to mirror the reporter image into your own registry, its pods stop being ignored and each run shows up as an artifact starting and exiting. + +The reporter is *not* exempt from compliance evaluation, though: under a policy that requires provenance it counts as an artifact without provenance — persistently, not intermittently. Either waive provenance for its image, or install the reporter into a namespace of its own and exclude that namespace — subject to the same include/exclude constraint as the [first option below](#handling-job-workloads). + +The API token secret is namespace-scoped, so a dedicated namespace needs its own copy of it. Create both before installing: + +```shell +kubectl create namespace kosli +kubectl create secret generic kosli-api-token -n kosli --from-literal=apikey= +helm install kosli-reporter kosli/k8s-reporter -n kosli -f values.yaml +``` + +The install steps in the [tutorial](/tutorials/report_k8s_envs) pass no `-n`, so they put the secret and the reporter in whichever namespace you are currently in — usually alongside the workloads you came there to report. Excluding *that* namespace would drop those workloads from the environment too. + + +## Handling job workloads + +The reporter filters by namespace only; there is no way to exclude pods by owner kind. Three options: + +* **Run jobs in their own namespace,** then either leave that namespace out of reporting or give it its own Kosli environment, so job churn does not affect the compliance of your long-running workloads. With the Helm chart, both are per-entry namespace selectors under `reporterConfig.environments`: `excludeNamespaces` on your main entry, plus a second entry whose `namespaces` is the job namespace if you want it reported separately. If your main entry already lists `namespaces` or `namespacesRegex`, drop the job namespace from that list instead — the include and exclude selectors are mutually exclusive within one entry. A second entry is not a second reporter, so the caveats in [Running multiple reporters](/tutorials/report_k8s_envs#running-multiple-reporters) do not apply. See the [chart configuration reference](/helm/k8s_reporter/configuration). With the CLI, use `--exclude-namespaces` when reporting the whole cluster, or simply omit the job namespace from `--namespaces`. +* **Waive provenance for the job's image** if you want the job pods in the environment but not the compliance flicker. An environment policy's `artifacts.provenance.exceptions` drops the provenance requirement for artifacts matching a policy expression: + + ```yaml + _schema: https://docs.kosli.com/schemas/policy/v1 + + artifacts: + provenance: + required: true + exceptions: + - if: ${{ matches(artifact.name, "^my-job:.*") }} + ``` + + See [environment policy](/policy-reference/environment_policy). +* **Attest the job to a flow instead.** Environment snapshots answer "what is running right now"; they are the wrong tool for "what ran, when, and did it succeed". Create a [flow](/getting_started/flows) for the job, [begin a trail](/getting_started/trails) for each run, and attest its outcome. Unlike snapshots, this captures every run no matter how briefly it ran. From a9a8ef6d4f3f94d8ab776645a4822491b8a1593c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Dan=20Gr=C3=B8ndahl?= Date: Mon, 7 Sep 2026 14:22:52 +0200 Subject: [PATCH 7/8] docs: move k8s reporting page under Administration and fix Environments group --- .../managing_environments/kubernetes_reporting.md | 2 +- administration/managing_environments/overview.md | 2 +- config/navigation.json | 6 +++--- tutorials/report_k8s_envs.md | 4 ++-- 4 files changed, 7 insertions(+), 7 deletions(-) rename understand_kosli/kubernetes_snapshots.md => administration/managing_environments/kubernetes_reporting.md (99%) diff --git a/understand_kosli/kubernetes_snapshots.md b/administration/managing_environments/kubernetes_reporting.md similarity index 99% rename from understand_kosli/kubernetes_snapshots.md rename to administration/managing_environments/kubernetes_reporting.md index 458cc12..2308901 100644 --- a/understand_kosli/kubernetes_snapshots.md +++ b/administration/managing_environments/kubernetes_reporting.md @@ -1,5 +1,5 @@ --- -title: 'Kubernetes environment snapshots' +title: 'Understand Kubernetes reporting' description: 'What the Kosli Kubernetes reporter captures, which pod phases it reports, and what that means for Job and CronJob workloads.' --- diff --git a/administration/managing_environments/overview.md b/administration/managing_environments/overview.md index 5727f0b..b78fa9f 100644 --- a/administration/managing_environments/overview.md +++ b/administration/managing_environments/overview.md @@ -1,5 +1,5 @@ --- -title: Managing Environments +title: Managing environments with Terraform description: Learn how to manage Kosli environments via Terraform, including creating and importing physical and logical environments. --- diff --git a/config/navigation.json b/config/navigation.json index 642f154..cb90004 100644 --- a/config/navigation.json +++ b/config/navigation.json @@ -11,7 +11,6 @@ "understand_kosli/risks", "understand_kosli/controls", "understand_kosli/how_kosli_works", - "understand_kosli/kubernetes_snapshots", "understand_kosli/glossary", "understand_kosli/ai_docs_access" ] @@ -59,9 +58,10 @@ ] }, { - "group": "Managing Environments", + "group": "Environments", "pages": [ - "administration/managing_environments/overview" + "administration/managing_environments/overview", + "administration/managing_environments/kubernetes_reporting" ] }, { diff --git a/tutorials/report_k8s_envs.md b/tutorials/report_k8s_envs.md index d3a64a7..b1470b2 100644 --- a/tutorials/report_k8s_envs.md +++ b/tutorials/report_k8s_envs.md @@ -177,7 +177,7 @@ A snapshot lists the **pods** running in scope. The reporter is blind to workloa That matters most for short-lived workloads. A `Job` or `CronJob` pod is captured only while it is running, so a job that starts and finishes between two snapshots never appears, and a successful run leaves no trace in the environment. A pod that *fails* is the exception — it keeps being reported until Kubernetes cleans it up. -For the full model — how job pods affect environment churn and compliance, and what to do about it — see [Kubernetes environment snapshots](/understand_kosli/kubernetes_snapshots). +For the full model — how job pods affect environment churn and compliance, and what to do about it — see [Understand Kubernetes reporting](/administration/managing_environments/kubernetes_reporting). ## Running multiple reporters @@ -205,4 +205,4 @@ From here you can: * Query your environment with [`kosli list snapshots`](/client_reference/kosli_list_snapshots) and [`kosli get snapshot`](/client_reference/kosli_get_snapshot) * [Compare snapshots to see what changed](/client_reference/kosli_diff_snapshots) * Trace a running artifact back to its git commit with the [From commit to production](/tutorials/following_a_git_commit_to_runtime_environments) tutorial -* Understand what snapshots do and don't capture in [Kubernetes environment snapshots](/understand_kosli/kubernetes_snapshots) +* See what snapshots do and don't capture in [Understand Kubernetes reporting](/administration/managing_environments/kubernetes_reporting) From 8cdb81dd4e2c64fca244d54da967f21a78429c78 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Dan=20Gr=C3=B8ndahl?= Date: Mon, 7 Sep 2026 14:29:44 +0200 Subject: [PATCH 8/8] docs: move multiple-reporters guidance to the k8s reporting page --- .../kubernetes_reporting.md | 20 ++++++++++++++++++- tutorials/report_k8s_envs.md | 16 +-------------- 2 files changed, 20 insertions(+), 16 deletions(-) diff --git a/administration/managing_environments/kubernetes_reporting.md b/administration/managing_environments/kubernetes_reporting.md index 2308901..0f32b05 100644 --- a/administration/managing_environments/kubernetes_reporting.md +++ b/administration/managing_environments/kubernetes_reporting.md @@ -68,7 +68,7 @@ The install steps in the [tutorial](/tutorials/report_k8s_envs) pass no `-n`, so The reporter filters by namespace only; there is no way to exclude pods by owner kind. Three options: -* **Run jobs in their own namespace,** then either leave that namespace out of reporting or give it its own Kosli environment, so job churn does not affect the compliance of your long-running workloads. With the Helm chart, both are per-entry namespace selectors under `reporterConfig.environments`: `excludeNamespaces` on your main entry, plus a second entry whose `namespaces` is the job namespace if you want it reported separately. If your main entry already lists `namespaces` or `namespacesRegex`, drop the job namespace from that list instead — the include and exclude selectors are mutually exclusive within one entry. A second entry is not a second reporter, so the caveats in [Running multiple reporters](/tutorials/report_k8s_envs#running-multiple-reporters) do not apply. See the [chart configuration reference](/helm/k8s_reporter/configuration). With the CLI, use `--exclude-namespaces` when reporting the whole cluster, or simply omit the job namespace from `--namespaces`. +* **Run jobs in their own namespace,** then either leave that namespace out of reporting or give it its own Kosli environment, so job churn does not affect the compliance of your long-running workloads. With the Helm chart, both are per-entry namespace selectors under `reporterConfig.environments`: `excludeNamespaces` on your main entry, plus a second entry whose `namespaces` is the job namespace if you want it reported separately. If your main entry already lists `namespaces` or `namespacesRegex`, drop the job namespace from that list instead — the include and exclude selectors are mutually exclusive within one entry. A second entry is not a second reporter, so the caveats in [Running multiple reporters](#running-multiple-reporters) do not apply. See the [chart configuration reference](/helm/k8s_reporter/configuration). With the CLI, use `--exclude-namespaces` when reporting the whole cluster, or simply omit the job namespace from `--namespaces`. * **Waive provenance for the job's image** if you want the job pods in the environment but not the compliance flicker. An environment policy's `artifacts.provenance.exceptions` drops the provenance requirement for artifacts matching a policy expression: ```yaml @@ -83,3 +83,21 @@ The reporter filters by namespace only; there is no way to exclude pods by owner See [environment policy](/policy-reference/environment_policy). * **Attest the job to a flow instead.** Environment snapshots answer "what is running right now"; they are the wrong tool for "what ran, when, and did it succeed". Create a [flow](/getting_started/flows) for the job, [begin a trail](/getting_started/trails) for each run, and attest its outcome. Unlike snapshots, this captures every run no matter how briefly it ran. + +## Running multiple reporters + +If you are considering running more than one reporter against the same cluster, the table below summarizes which setups produce meaningful snapshots and which don't. + +| Scenario | Supported | Explanation | +| :--- | :---: | :--- | +| Two orgs, separate environments, overlapping namespaces | Yes | Different environments → independent snapshots. | +| One org, two environments, overlapping namespaces | Yes | Same as above. | +| One org, **same environment**, two reporters with overlapping namespaces | No | Snapshots toggle between each reporter's view. No data is deleted, but diffs between consecutive snapshots become meaningless. | +| One org, same environment, two reporters with **disjoint** namespaces | No | Each snapshot only reflects one reporter's namespaces, so diffs compare unrelated scopes. | + + +A single Kosli environment must have exactly one reporter feeding it. Snapshots are never overwritten or deleted, but if two reporters take turns updating the same environment: + +* Diffs between consecutive snapshots compare unrelated views of the cluster. +* The environment history shows artifacts continuously stopping and starting as each report toggles which namespaces are visible. + diff --git a/tutorials/report_k8s_envs.md b/tutorials/report_k8s_envs.md index b1470b2..4f39b6d 100644 --- a/tutorials/report_k8s_envs.md +++ b/tutorials/report_k8s_envs.md @@ -181,21 +181,7 @@ For the full model — how job pods affect environment churn and compliance, and ## Running multiple reporters -If you are considering running more than one reporter against the same cluster, the table below summarizes which setups produce meaningful snapshots and which don't. - -| Scenario | Supported | Explanation | -| :--- | :---: | :--- | -| Two orgs, separate environments, overlapping namespaces | Yes | Different environments → independent snapshots. | -| One org, two environments, overlapping namespaces | Yes | Same as above. | -| One org, **same environment**, two reporters with overlapping namespaces | No | Snapshots toggle between each reporter's view. No data is deleted, but diffs between consecutive snapshots become meaningless. | -| One org, same environment, two reporters with **disjoint** namespaces | No | Each snapshot only reflects one reporter's namespaces, so diffs compare unrelated scopes. | - - -A single Kosli environment must have exactly one reporter feeding it. Snapshots are never overwritten or deleted, but if two reporters take turns updating the same environment: - -* Diffs between consecutive snapshots compare unrelated views of the cluster. -* The environment history shows artifacts continuously stopping and starting as each report toggles which namespaces are visible. - +A single Kosli environment must have exactly one reporter feeding it. Pointing two reporters at the same environment does not delete data, but it makes diffs between consecutive snapshots meaningless. For the supported and unsupported topologies, see [Running multiple reporters](/administration/managing_environments/kubernetes_reporting#running-multiple-reporters). ## What you've accomplished