docs: document how the k8s reporter handles job and cronjob pods - #391
docs: document how the k8s reporter handles job and cronjob pods#391dangrondahl wants to merge 8 commits into
Conversation
|
Preview deployment for your docs. Learn more about Mintlify Previews.
|
|
Claude finished @dangrondahl's task in 3m 1s —— View job Doc review — round 5
The split holds up: Moving New this push1. The page's intro and Still open from round 4 (not re-commented inline)2. The reporter-namespace remedy still doesn't do what the prose promises — lines 54–62. Line 54 says "install the reporter into a namespace of its own and exclude that namespace"; the three commands create 3. The automatic-cleanup framing is now an incomplete sweep across two pages. Line 39 establishes that nothing removes a
The split made this worse rather than better: two of the three are the section lead-in and the 4. Findings 2–4 were reported at 12:11–12:12 UTC and are unchanged at head; I've left those threads rather than duplicating them. Nothing new on the phase table, the churn rules, the policy example or the multiple-reporters table. |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
|
Addressed all five findings from the doc review — the two Improvements and all three Suggestions. Pushed as Finding 3 was summary-only, so answering it here: applied. The point was correct and it was the most substantive of the three suggestions — "a snapshot is a list of pods" described a different data model than the Two verifications behind the changes, since both Improvements rested on claims I had not checked when writing the section:
Checks: Nothing disputed — all five were valid. |
|
Restructured — the section had outgrown its doc type, so the mechanics now live on their own page. WhyFour review rounds grew By Diátaxis the section had become three types wearing a tutorial's clothes: reference (the phase table), explanation (churn model, cron sampling, the image-name-keyed exemption), and how-to (the three remedies). A learner following "report a snapshot" was reading What changedNew: Trimmed:
Anchors: the cross-reference to Nothing was droppedAll 14 findings from the four rounds remain applied — this is placement only, not a revert. One judgment call worth flagging: at 1322 words the new page is larger than any existing |
|
|
||
| * A job that starts and finishes between two snapshots never appears at all. As a rough guide, a run is captured about as often as its runtime divides into the snapshot interval — so on the Helm chart's default `*/5 * * * *` schedule a job that runs for a few seconds is almost always missed, while one that runs for most of the interval is almost always caught. That guide assumes the job starts at an arbitrary point in the interval. A `CronJob` whose schedule shares a period with the reporter's does not sample randomly at all: it is caught on nearly every run or on nearly none, depending on which of the two fires first. | ||
| * A job that happens to be running when a snapshot is taken appears in that snapshot and is gone from the next one. | ||
| * A job that **fails** is the exception. `Failed` is a terminal phase, so the pod stays in it — and keeps being reported — for as long as it exists, and nothing removes it by default. `ttlSecondsAfterFinished` is unset unless you set it, so a standalone Job's failed pods survive until the Job is deleted; a `CronJob` always retains its most recent failed Job (`failedJobsHistoryLimit`, default `1`). A Job with `restartPolicy: Never` that keeps failing leaves one `Failed` pod per attempt, up to `backoffLimit` (default `6`), and each is reported. Until they are gone, a failed run is not a flicker: Kosli keeps showing the dead pods' artifact as running in the environment. |
There was a problem hiding this comment.
Improvement — two numbers in this bullet are ones a reader plans capacity and cleanup around, and both are off.
backoffLimitis the number of retries, so a Job that keeps failing makesbackoffLimit + 1attempts — sevenFailedpods at the default, not six. "up tobackoffLimit(default6)" undercounts by one.- Those pods all run the same image, so Kosli reports them as one artifact with an instance count of seven, not as seven artifacts. Saying so explicitly is what ties this bullet to the churn rules three paragraphs down, which turn on exactly that distinction.
| * A job that **fails** is the exception. `Failed` is a terminal phase, so the pod stays in it — and keeps being reported — for as long as it exists, and nothing removes it by default. `ttlSecondsAfterFinished` is unset unless you set it, so a standalone Job's failed pods survive until the Job is deleted; a `CronJob` always retains its most recent failed Job (`failedJobsHistoryLimit`, default `1`). A Job with `restartPolicy: Never` that keeps failing leaves one `Failed` pod per attempt, up to `backoffLimit` (default `6`), and each is reported. Until they are gone, a failed run is not a flicker: Kosli keeps showing the dead pods' artifact as running in the environment. | |
| * A job that **fails** is the exception. `Failed` is a terminal phase, so the pod stays in it — and keeps being reported — for as long as it exists, and nothing removes it by default. `ttlSecondsAfterFinished` is unset unless you set it, so a standalone Job's failed pods survive until the Job is deleted; a `CronJob` always retains its most recent failed Job (`failedJobsHistoryLimit`, default `1`). A Job with `restartPolicy: Never` that keeps failing leaves one `Failed` pod per attempt — `backoffLimit` retries plus the first try, so seven at the default — and every one of them is reported, as instances of the same artifact. Until they are deleted, a failed run is not a flicker: Kosli keeps showing the dead pods' artifact as running in the environment. |
| * If the job runs the same image as a long-running workload, only the instance count changes. Instance-count-only differences do not create a snapshot, so the run is invisible — and because the report is discarded, the pod's owner references are not stored either. Owner references only reach Kosli for runs that produce a snapshot. | ||
|
|
||
| <Warning> | ||
| If the job's image was never attested to a Kosli flow, it is reported as an artifact with no provenance. Under an [environment policy](/policy-reference/environment_policy) that requires provenance, snapshots taken while a job was running are non-compliant and snapshots taken between runs are compliant — so compliance appears to flicker. A failed run is worse than a flicker: the `Failed` pod is reported in every snapshot until it is garbage-collected, holding the environment non-compliant for as long as it survives. |
There was a problem hiding this comment.
Improvement — this <Warning> still says the Failed pod is reported "until it is garbage-collected", which is the framing line 39 exists to correct: nothing garbage-collects it by default (ttlSecondsAfterFinished is unset, and a CronJob's failedJobsHistoryLimit replaces the retained Job rather than removing it).
The mismatch matters because the callout is the sentence a reader skims and acts on: as written it says "wait and it clears itself", when the bullet above says "it stays until you delete it". The callout should carry the stronger claim.
| If the job's image was never attested to a Kosli flow, it is reported as an artifact with no provenance. Under an [environment policy](/policy-reference/environment_policy) that requires provenance, snapshots taken while a job was running are non-compliant and snapshots taken between runs are compliant — so compliance appears to flicker. A failed run is worse than a flicker: the `Failed` pod is reported in every snapshot until it is garbage-collected, holding the environment non-compliant for as long as it survives. | |
| If the job's image was never attested to a Kosli flow, it is reported as an artifact with no provenance. Under an [environment policy](/policy-reference/environment_policy) that requires provenance, snapshots taken while a job was running are non-compliant and snapshots taken between runs are compliant — so compliance appears to flicker. A failed run is worse than a flicker: nothing deletes a `Failed` pod by default, so it is reported in every snapshot until you remove it — holding the environment non-compliant until then. |
| <Note> | ||
| The reporter deployed by the Helm chart is itself a `CronJob`, and at the default whole-cluster scope it is running whenever it takes a snapshot — so it appears in its own snapshots. Kosli ignores the reporter's image (`ghcr.io/kosli-dev/cli`) when deciding whether a snapshot is worth saving, so the reporter's own pods never create snapshots or start and exit events. The match is on the image name, so if you override `image.repository` to mirror the reporter image into your own registry, its pods stop being ignored and each run shows up as an artifact starting and exiting. | ||
|
|
||
| The reporter is *not* exempt from compliance evaluation, though: under a policy that requires provenance it counts as an artifact without provenance — persistently, not intermittently. Either waive provenance for its image, or install the reporter into a namespace of its own and exclude that namespace — subject to the same include/exclude constraint as the [first option below](#handling-job-workloads). | ||
|
|
||
| The API token secret is namespace-scoped, so a dedicated namespace needs its own copy of it. Create both before installing: | ||
|
|
||
| ```shell | ||
| kubectl create namespace kosli | ||
| kubectl create secret generic kosli-api-token -n kosli --from-literal=apikey=<your-kosli-api-token> | ||
| helm install kosli-reporter kosli/k8s-reporter -n kosli -f values.yaml | ||
| ``` | ||
|
|
||
| The install steps in the [tutorial](/tutorials/report_k8s_envs) pass no `-n`, so they put the secret and the reporter in whichever namespace you are currently in — usually alongside the workloads you came there to report. Excluding *that* namespace would drop those workloads from the environment too. | ||
| </Note> |
There was a problem hiding this comment.
Improvement — the remedy in this <Note> still doesn't do what the prose promises, and now that the content lives on its own page there's an easy fix for both halves.
- Line 54 says "install the reporter into a namespace of its own and exclude that namespace", but the three commands only do the first half: they create
kosli, put the secret there, and install with-n kosliusing an unchangedvalues.yaml. Nothing tells the reader to addexcludeNamespaces: ["kosli"]to thereporterConfig.environmentsentry — and at the whole-cluster scope this page assumes (line 52), the reporter is then in a new namespace and still reporting itself, so the provenance failure the Note exists to explain is untouched. The include/exclude cross-reference explains the constraint but never says to make the edit. - This is now four paragraphs and a three-command procedure inside a callout, on a page whose other callout is one paragraph. CLAUDE.md asks for
<Tip>/<Note>/<Warning>"sparingly", and a callout this size stops reading as an aside — it reads as the section it actually is. A## The reporter's own podsheading between "Jobs and CronJobs" and "Handling job workloads" would also make it linkable, which the Note's own#handling-job-workloadsback-reference suggests you want.
Suggested shape: promote to a section, keep paragraph 1 as-is, and name the values edit before the commands — e.g. "Add excludeNamespaces: ["kosli"] to the environment entry, then create the namespace and secret before installing:".
|
|
||
| A snapshot lists the **pods** running in scope. The reporter is blind to workload kind — a `Job` pod is reported exactly like a `Deployment` pod — and only pods in the `Running` and `Failed` phases are reported; `Succeeded` and `Pending` pods are not. | ||
|
|
||
| That matters most for short-lived workloads. A `Job` or `CronJob` pod is captured only while it is running, so a job that starts and finishes between two snapshots never appears, and a successful run leaves no trace in the environment. A pod that *fails* is the exception — it keeps being reported until Kubernetes cleans it up. |
There was a problem hiding this comment.
Suggestion — "until Kubernetes cleans it up" carries the same implication of automatic cleanup that the explanation page corrects (understand_kosli/kubernetes_snapshots.md:39: nothing removes a Failed pod by default). A tutorial reader who only reads this summary will wait for a cleanup that never comes.
| That matters most for short-lived workloads. A `Job` or `CronJob` pod is captured only while it is running, so a job that starts and finishes between two snapshots never appears, and a successful run leaves no trace in the environment. A pod that *fails* is the exception — it keeps being reported until Kubernetes cleans it up. | |
| That matters most for short-lived workloads. A `Job` or `CronJob` pod is captured only while it is running, so a job that starts and finishes between two snapshots never appears, and a successful run leaves no trace in the environment. A pod that *fails* is the exception — nothing deletes it by default, so it keeps being reported until you remove it. |
| "group": "Environments", | ||
| "pages": [ | ||
| "administration/managing_environments/overview" | ||
| "administration/managing_environments/overview", | ||
| "administration/managing_environments/kubernetes_reporting" | ||
| ] |
There was a problem hiding this comment.
Suggestion — worth re-checking this placement now that it has moved. The page is explanation: its title starts "Understand", its description is "What the Kosli Kubernetes reporter captures…", and four of its five sections are mechanics and consequences rather than steps. doc-write's placement table maps "a concept, or the reasoning behind a design" to Documentation ▸ Understand Kosli, and Administration to "a task an org admin performs (users, roles, auth, org-wide settings)" — which is what its three sibling groups (Authentication & access, Users & roles, Managing Custom Attestation Types) all are.
The group it landed in also now holds two unrelated things: a Terraform how-to (which needed its title changed to Managing environments with Terraform to make room) and reporter mechanics. A reader who wonders why their CronJob never appears will reach this page from the tutorial link, but won't find it browsing Understand Kosli next to how_kosli_works, which is where the shallower version of the same model already lives.
If the intent was to keep it near environment operations, Documentation ▸ Tutorials ▸ Reporting environments — beside tutorials/report_k8s_envs itself — is closer to that than Administration, and doesn't require the sibling page to be retitled.
| description: 'What the Kosli Kubernetes reporter captures, which pod phases it reports, and what that means for Job and CronJob workloads.' | ||
| --- | ||
|
|
||
| A Kubernetes environment snapshot records what was running in your cluster at a moment in time. This page explains what the reporter actually collects, which pods it skips, and why short-lived workloads such as `Job` and `CronJob` behave differently from long-running ones. To set reporting up in the first place, see [Report Kubernetes environments to Kosli](/tutorials/report_k8s_envs). |
There was a problem hiding this comment.
Suggestion — this push moved ## Running multiple reporters onto the page, but neither this intro nor the description frontmatter (line 3) mentions it. Both enumerate the page's contents — "what the reporter actually collects, which pods it skips, and why short-lived workloads such as Job and CronJob behave differently" — so the page now under-describes itself by a whole section.
It matters more than a stale summary usually would, because tutorials/report_k8s_envs.md:184 now sends readers here specifically for reporter topologies. That arriving intent is the one thing the opening paragraph doesn't acknowledge, and the description is what shows in search results.
| A Kubernetes environment snapshot records what was running in your cluster at a moment in time. This page explains what the reporter actually collects, which pods it skips, and why short-lived workloads such as `Job` and `CronJob` behave differently from long-running ones. To set reporting up in the first place, see [Report Kubernetes environments to Kosli](/tutorials/report_k8s_envs). | |
| A Kubernetes environment snapshot records what was running in your cluster at a moment in time. This page explains what the reporter actually collects, which pods it skips, why short-lived workloads such as `Job` and `CronJob` behave differently from long-running ones, and which multi-reporter setups produce meaningful snapshots. To set reporting up in the first place, see [Report Kubernetes environments to Kosli](/tutorials/report_k8s_envs). |
The description needs the same extension — e.g. "…what that means for Job and CronJob workloads, and which multi-reporter setups are supported."
1 participant
Closes #387
Adds a What gets reported section to
tutorials/report_k8s_envs.md, answering the customer question "how are K8s Jobs and CronJobs handled — is it detected by the reporter?".What the section covers
Running,Failed) and which are not (Succeeded,Pending,Unknown).kosli get snapshot --output json, not in the table output.Job/CronJobpods: captured only while running, so short-lived jobs are sampled at best and completed runs leave no trace.Verified against source
Against
kosli-dev/cliinternal/kube/kube.goandkosli-dev/server:Running/Failedpods reportedprocessPodsininternal/kube/kube.goFailedpod skipped with a warning when a container has no image IDNewPodDataPodData.Owners→K8SArtifact.owners→pods[name]["owners"]inenvironment_snapshots.pySnapshotArtifact.pods,printSnapshotAsTable_annotate_common_artifact/changed_except_kosli_clisnapshotK8S.goflags andk8sEnvironmentConfigChecks
mint broken-links— no new broken links. The one reported failure (/getting_started/service-accountsintutorials/working_with_controls.mdx) is pre-existing and unrelated.pytest tests/— 44 passed.config/navigation.json:100.Follow-up, not in this PR
client_reference/kosli_snapshot_k8s.mdis generated, and its long description still says only "The reported data includes pod container images digests and creation timestamps" with no mention of pod phases. Adding the phase constraint there needs a change tosnapshotK8SLongDescinkosli-dev/cli'scmd/kosli/snapshotK8S.go.