Skip to content
103 changes: 103 additions & 0 deletions administration/managing_environments/kubernetes_reporting.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,103 @@
---
title: 'Understand Kubernetes reporting'
description: 'What the Kosli Kubernetes reporter captures, which pod phases it reports, and what that means for Job and CronJob workloads.'
---

A Kubernetes environment snapshot records what was running in your cluster at a moment in time. This page explains what the reporter actually collects, which pods it skips, and why short-lived workloads such as `Job` and `CronJob` behave differently from long-running ones. To set reporting up in the first place, see [Report Kubernetes environments to Kosli](/tutorials/report_k8s_envs).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggestion — this push moved ## Running multiple reporters onto the page, but neither this intro nor the description frontmatter (line 3) mentions it. Both enumerate the page's contents — "what the reporter actually collects, which pods it skips, and why short-lived workloads such as Job and CronJob behave differently" — so the page now under-describes itself by a whole section.

It matters more than a stale summary usually would, because tutorials/report_k8s_envs.md:184 now sends readers here specifically for reporter topologies. That arriving intent is the one thing the opening paragraph doesn't acknowledge, and the description is what shows in search results.

Suggested change
A Kubernetes environment snapshot records what was running in your cluster at a moment in time. This page explains what the reporter actually collects, which pods it skips, and why short-lived workloads such as `Job` and `CronJob` behave differently from long-running ones. To set reporting up in the first place, see [Report Kubernetes environments to Kosli](/tutorials/report_k8s_envs).
A Kubernetes environment snapshot records what was running in your cluster at a moment in time. This page explains what the reporter actually collects, which pods it skips, why short-lived workloads such as `Job` and `CronJob` behave differently from long-running ones, and which multi-reporter setups produce meaningful snapshots. To set reporting up in the first place, see [Report Kubernetes environments to Kosli](/tutorials/report_k8s_envs).

The description needs the same extension — e.g. "…what that means for Job and CronJob workloads, and which multi-reporter setups are supported."

Fix this →


## What gets reported

The reporter collects **pods**, and is blind to workload kind: it has no awareness of `Deployment`, `StatefulSet`, `Job`, or `CronJob` objects, so a Job pod is reported exactly like a Deployment pod. For each pod it records the pod name, its namespace, its container image digests, its creation timestamp, and its owner references.

In the snapshot those pods are grouped under the artifact whose image they run — a snapshot is a list of artifacts, each carrying the pods running it. That is why the churn rules below turn on image digests rather than on individual pods.

Only pods in certain phases are reported:

| Pod phase | Reported | Notes |
| :--- | :---: | :--- |
| `Running` | Yes | |
| `Failed` | Yes* | *Skipped, with a warning, if any of its containers has no image ID. |
| `Succeeded` | No | Where a completed Job pod ends up. |
| `Pending` | No | No image digests exist yet. |
| `Unknown` | No | |

Owner references are stored on the snapshot, so the `Job` or `CronJob` that owns a pod does reach Kosli. They are not in the table output of [`kosli get snapshot`](/client_reference/kosli_get_snapshot) — read them from the JSON:

```shell
kosli get snapshot <your-environment> \
--api-token <your-api-token-here> \
--org <your-kosli-org-name> \
--output json | jq '.artifacts[].pods'
```

## Jobs and CronJobs

Because reporting is driven by pod phase, a `Job` or `CronJob` pod is captured only while it is running — or, if it fails, until the failed pod is cleaned up:

* A job that starts and finishes between two snapshots never appears at all. As a rough guide, a run is captured about as often as its runtime divides into the snapshot interval — so on the Helm chart's default `*/5 * * * *` schedule a job that runs for a few seconds is almost always missed, while one that runs for most of the interval is almost always caught. That guide assumes the job starts at an arbitrary point in the interval. A `CronJob` whose schedule shares a period with the reporter's does not sample randomly at all: it is caught on nearly every run or on nearly none, depending on which of the two fires first.
* A job that happens to be running when a snapshot is taken appears in that snapshot and is gone from the next one.
* A job that **fails** is the exception. `Failed` is a terminal phase, so the pod stays in it — and keeps being reported — for as long as it exists, and nothing removes it by default. `ttlSecondsAfterFinished` is unset unless you set it, so a standalone Job's failed pods survive until the Job is deleted; a `CronJob` always retains its most recent failed Job (`failedJobsHistoryLimit`, default `1`). A Job with `restartPolicy: Never` that keeps failing leaves one `Failed` pod per attempt, up to `backoffLimit` (default `6`), and each is reported. Until they are gone, a failed run is not a flicker: Kosli keeps showing the dead pods' artifact as running in the environment.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Improvement — two numbers in this bullet are ones a reader plans capacity and cleanup around, and both are off.

  1. backoffLimit is the number of retries, so a Job that keeps failing makes backoffLimit + 1 attempts — seven Failed pods at the default, not six. "up to backoffLimit (default 6)" undercounts by one.
  2. Those pods all run the same image, so Kosli reports them as one artifact with an instance count of seven, not as seven artifacts. Saying so explicitly is what ties this bullet to the churn rules three paragraphs down, which turn on exactly that distinction.
Suggested change
* A job that **fails** is the exception. `Failed` is a terminal phase, so the pod stays in it — and keeps being reported — for as long as it exists, and nothing removes it by default. `ttlSecondsAfterFinished` is unset unless you set it, so a standalone Job's failed pods survive until the Job is deleted; a `CronJob` always retains its most recent failed Job (`failedJobsHistoryLimit`, default `1`). A Job with `restartPolicy: Never` that keeps failing leaves one `Failed` pod per attempt, up to `backoffLimit` (default `6`), and each is reported. Until they are gone, a failed run is not a flicker: Kosli keeps showing the dead pods' artifact as running in the environment.
* A job that **fails** is the exception. `Failed` is a terminal phase, so the pod stays in it — and keeps being reported — for as long as it exists, and nothing removes it by default. `ttlSecondsAfterFinished` is unset unless you set it, so a standalone Job's failed pods survive until the Job is deleted; a `CronJob` always retains its most recent failed Job (`failedJobsHistoryLimit`, default `1`). A Job with `restartPolicy: Never` that keeps failing leaves one `Failed` pod per attempt`backoffLimit` retries plus the first try, so seven at the default — and every one of them is reported, as instances of the same artifact. Until they are deleted, a failed run is not a flicker: Kosli keeps showing the dead pods' artifact as running in the environment.

Fix this →

* A job that succeeds leaves no trace. `Succeeded` pods are never reported, so nothing in the environment records that the run happened.

Whether a captured run shows up as environment churn depends on the image:

* If the job runs an image that nothing else in the environment runs, each captured run produces one snapshot where the artifact started and a later one where it exited.
* If the job runs the same image as a long-running workload, only the instance count changes. Instance-count-only differences do not create a snapshot, so the run is invisible — and because the report is discarded, the pod's owner references are not stored either. Owner references only reach Kosli for runs that produce a snapshot.

<Warning>
If the job's image was never attested to a Kosli flow, it is reported as an artifact with no provenance. Under an [environment policy](/policy-reference/environment_policy) that requires provenance, snapshots taken while a job was running are non-compliant and snapshots taken between runs are compliant — so compliance appears to flicker. A failed run is worse than a flicker: the `Failed` pod is reported in every snapshot until it is garbage-collected, holding the environment non-compliant for as long as it survives.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Improvement — this <Warning> still says the Failed pod is reported "until it is garbage-collected", which is the framing line 39 exists to correct: nothing garbage-collects it by default (ttlSecondsAfterFinished is unset, and a CronJob's failedJobsHistoryLimit replaces the retained Job rather than removing it).

The mismatch matters because the callout is the sentence a reader skims and acts on: as written it says "wait and it clears itself", when the bullet above says "it stays until you delete it". The callout should carry the stronger claim.

Suggested change
If the job's image was never attested to a Kosli flow, it is reported as an artifact with no provenance. Under an [environment policy](/policy-reference/environment_policy) that requires provenance, snapshots taken while a job was running are non-compliant and snapshots taken between runs are compliant — so compliance appears to flicker. A failed run is worse than a flicker: the `Failed` pod is reported in every snapshot until it is garbage-collected, holding the environment non-compliant for as long as it survives.
If the job's image was never attested to a Kosli flow, it is reported as an artifact with no provenance. Under an [environment policy](/policy-reference/environment_policy) that requires provenance, snapshots taken while a job was running are non-compliant and snapshots taken between runs are compliant — so compliance appears to flicker. A failed run is worse than a flicker: nothing deletes a `Failed` pod by default, so it is reported in every snapshot until you remove it — holding the environment non-compliant until then.

Fix this →

</Warning>

<Note>
The reporter deployed by the Helm chart is itself a `CronJob`, and at the default whole-cluster scope it is running whenever it takes a snapshot — so it appears in its own snapshots. Kosli ignores the reporter's image (`ghcr.io/kosli-dev/cli`) when deciding whether a snapshot is worth saving, so the reporter's own pods never create snapshots or start and exit events. The match is on the image name, so if you override `image.repository` to mirror the reporter image into your own registry, its pods stop being ignored and each run shows up as an artifact starting and exiting.

The reporter is *not* exempt from compliance evaluation, though: under a policy that requires provenance it counts as an artifact without provenance — persistently, not intermittently. Either waive provenance for its image, or install the reporter into a namespace of its own and exclude that namespace — subject to the same include/exclude constraint as the [first option below](#handling-job-workloads).

The API token secret is namespace-scoped, so a dedicated namespace needs its own copy of it. Create both before installing:

```shell
kubectl create namespace kosli
kubectl create secret generic kosli-api-token -n kosli --from-literal=apikey=<your-kosli-api-token>
helm install kosli-reporter kosli/k8s-reporter -n kosli -f values.yaml
```

The install steps in the [tutorial](/tutorials/report_k8s_envs) pass no `-n`, so they put the secret and the reporter in whichever namespace you are currently in — usually alongside the workloads you came there to report. Excluding *that* namespace would drop those workloads from the environment too.
</Note>
Comment on lines +51 to +65

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Improvement — the remedy in this <Note> still doesn't do what the prose promises, and now that the content lives on its own page there's an easy fix for both halves.

  1. Line 54 says "install the reporter into a namespace of its own and exclude that namespace", but the three commands only do the first half: they create kosli, put the secret there, and install with -n kosli using an unchanged values.yaml. Nothing tells the reader to add excludeNamespaces: ["kosli"] to the reporterConfig.environments entry — and at the whole-cluster scope this page assumes (line 52), the reporter is then in a new namespace and still reporting itself, so the provenance failure the Note exists to explain is untouched. The include/exclude cross-reference explains the constraint but never says to make the edit.
  2. This is now four paragraphs and a three-command procedure inside a callout, on a page whose other callout is one paragraph. CLAUDE.md asks for <Tip>/<Note>/<Warning> "sparingly", and a callout this size stops reading as an aside — it reads as the section it actually is. A ## The reporter's own pods heading between "Jobs and CronJobs" and "Handling job workloads" would also make it linkable, which the Note's own #handling-job-workloads back-reference suggests you want.

Suggested shape: promote to a section, keep paragraph 1 as-is, and name the values edit before the commands — e.g. "Add excludeNamespaces: ["kosli"] to the environment entry, then create the namespace and secret before installing:".

Fix this →


## Handling job workloads

The reporter filters by namespace only; there is no way to exclude pods by owner kind. Three options:

* **Run jobs in their own namespace,** then either leave that namespace out of reporting or give it its own Kosli environment, so job churn does not affect the compliance of your long-running workloads. With the Helm chart, both are per-entry namespace selectors under `reporterConfig.environments`: `excludeNamespaces` on your main entry, plus a second entry whose `namespaces` is the job namespace if you want it reported separately. If your main entry already lists `namespaces` or `namespacesRegex`, drop the job namespace from that list instead — the include and exclude selectors are mutually exclusive within one entry. A second entry is not a second reporter, so the caveats in [Running multiple reporters](#running-multiple-reporters) do not apply. See the [chart configuration reference](/helm/k8s_reporter/configuration). With the CLI, use `--exclude-namespaces` when reporting the whole cluster, or simply omit the job namespace from `--namespaces`.
* **Waive provenance for the job's image** if you want the job pods in the environment but not the compliance flicker. An environment policy's `artifacts.provenance.exceptions` drops the provenance requirement for artifacts matching a policy expression:

```yaml
_schema: https://docs.kosli.com/schemas/policy/v1

artifacts:
provenance:
required: true
exceptions:
- if: ${{ matches(artifact.name, "^my-job:.*") }}
```

See [environment policy](/policy-reference/environment_policy).
* **Attest the job to a flow instead.** Environment snapshots answer "what is running right now"; they are the wrong tool for "what ran, when, and did it succeed". Create a [flow](/getting_started/flows) for the job, [begin a trail](/getting_started/trails) for each run, and attest its outcome. Unlike snapshots, this captures every run no matter how briefly it ran.

## Running multiple reporters

If you are considering running more than one reporter against the same cluster, the table below summarizes which setups produce meaningful snapshots and which don't.

| Scenario | Supported | Explanation |
| :--- | :---: | :--- |
| Two orgs, separate environments, overlapping namespaces | Yes | Different environments → independent snapshots. |
| One org, two environments, overlapping namespaces | Yes | Same as above. |
| One org, **same environment**, two reporters with overlapping namespaces | No | Snapshots toggle between each reporter's view. No data is deleted, but diffs between consecutive snapshots become meaningless. |
| One org, same environment, two reporters with **disjoint** namespaces | No | Each snapshot only reflects one reporter's namespaces, so diffs compare unrelated scopes. |

<Warning>
A single Kosli environment must have exactly one reporter feeding it. Snapshots are never overwritten or deleted, but if two reporters take turns updating the same environment:

* Diffs between consecutive snapshots compare unrelated views of the cluster.
* The environment history shows artifacts continuously stopping and starting as each report toggles which namespaces are visible.
</Warning>
2 changes: 1 addition & 1 deletion administration/managing_environments/overview.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
title: Managing Environments
title: Managing environments with Terraform
description: Learn how to manage Kosli environments via Terraform, including creating and importing physical and logical environments.
---

Expand Down
5 changes: 3 additions & 2 deletions config/navigation.json
Original file line number Diff line number Diff line change
Expand Up @@ -58,9 +58,10 @@
]
},
{
"group": "Managing Environments",
"group": "Environments",
"pages": [
"administration/managing_environments/overview"
"administration/managing_environments/overview",
"administration/managing_environments/kubernetes_reporting"
]
Comment on lines +61 to 65

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggestion — worth re-checking this placement now that it has moved. The page is explanation: its title starts "Understand", its description is "What the Kosli Kubernetes reporter captures…", and four of its five sections are mechanics and consequences rather than steps. doc-write's placement table maps "a concept, or the reasoning behind a design" to Documentation ▸ Understand Kosli, and Administration to "a task an org admin performs (users, roles, auth, org-wide settings)" — which is what its three sibling groups (Authentication & access, Users & roles, Managing Custom Attestation Types) all are.

The group it landed in also now holds two unrelated things: a Terraform how-to (which needed its title changed to Managing environments with Terraform to make room) and reporter mechanics. A reader who wonders why their CronJob never appears will reach this page from the tutorial link, but won't find it browsing Understand Kosli next to how_kosli_works, which is where the shallower version of the same model already lives.

If the intent was to keep it near environment operations, Documentation ▸ Tutorials ▸ Reporting environments — beside tutorials/report_k8s_envs itself — is closer to that than Administration, and doesn't require the sibling page to be retitled.

Fix this →

},
{
Expand Down
21 changes: 8 additions & 13 deletions tutorials/report_k8s_envs.md
Original file line number Diff line number Diff line change
Expand Up @@ -171,23 +171,17 @@ kosli snapshot k8s k8s-tutorial \
</Tab>
</Tabs>

## Running multiple reporters
## What gets reported

If you are considering running more than one reporter against the same cluster, the table below summarizes which setups produce meaningful snapshots and which don't.
A snapshot lists the **pods** running in scope. The reporter is blind to workload kind — a `Job` pod is reported exactly like a `Deployment` pod — and only pods in the `Running` and `Failed` phases are reported; `Succeeded` and `Pending` pods are not.

| Scenario | Supported | Explanation |
| :--- | :---: | :--- |
| Two orgs, separate environments, overlapping namespaces | Yes | Different environments → independent snapshots. |
| One org, two environments, overlapping namespaces | Yes | Same as above. |
| One org, **same environment**, two reporters with overlapping namespaces | No | Snapshots toggle between each reporter's view. No data is deleted, but diffs between consecutive snapshots become meaningless. |
| One org, same environment, two reporters with **disjoint** namespaces | No | Each snapshot only reflects one reporter's namespaces, so diffs compare unrelated scopes. |
That matters most for short-lived workloads. A `Job` or `CronJob` pod is captured only while it is running, so a job that starts and finishes between two snapshots never appears, and a successful run leaves no trace in the environment. A pod that *fails* is the exception — it keeps being reported until Kubernetes cleans it up.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggestion — "until Kubernetes cleans it up" carries the same implication of automatic cleanup that the explanation page corrects (understand_kosli/kubernetes_snapshots.md:39: nothing removes a Failed pod by default). A tutorial reader who only reads this summary will wait for a cleanup that never comes.

Suggested change
That matters most for short-lived workloads. A `Job` or `CronJob` pod is captured only while it is running, so a job that starts and finishes between two snapshots never appears, and a successful run leaves no trace in the environment. A pod that *fails* is the exception — it keeps being reported until Kubernetes cleans it up.
That matters most for short-lived workloads. A `Job` or `CronJob` pod is captured only while it is running, so a job that starts and finishes between two snapshots never appears, and a successful run leaves no trace in the environment. A pod that *fails* is the exception — nothing deletes it by default, so it keeps being reported until you remove it.

Fix this →


<Warning>
A single Kosli environment must have exactly one reporter feeding it. Snapshots are never overwritten or deleted, but if two reporters take turns updating the same environment:
For the full model — how job pods affect environment churn and compliance, and what to do about it — see [Understand Kubernetes reporting](/administration/managing_environments/kubernetes_reporting).

* Diffs between consecutive snapshots compare unrelated views of the cluster.
* The environment history shows artifacts continuously stopping and starting as each report toggles which namespaces are visible.
</Warning>
## Running multiple reporters

A single Kosli environment must have exactly one reporter feeding it. Pointing two reporters at the same environment does not delete data, but it makes diffs between consecutive snapshots meaningless. For the supported and unsupported topologies, see [Running multiple reporters](/administration/managing_environments/kubernetes_reporting#running-multiple-reporters).

## What you've accomplished

Expand All @@ -197,3 +191,4 @@ From here you can:
* Query your environment with [`kosli list snapshots`](/client_reference/kosli_list_snapshots) and [`kosli get snapshot`](/client_reference/kosli_get_snapshot)
* [Compare snapshots to see what changed](/client_reference/kosli_diff_snapshots)
* Trace a running artifact back to its git commit with the [From commit to production](/tutorials/following_a_git_commit_to_runtime_environments) tutorial
* See what snapshots do and don't capture in [Understand Kubernetes reporting](/administration/managing_environments/kubernetes_reporting)