From ab73c3f4803600f4e0e544ca4f39447167c326cc Mon Sep 17 00:00:00 2001 From: Matthew Thomas <25988124+Ma11hewThomas@users.noreply.github.com> Date: Fri, 27 Mar 2026 22:46:58 +0000 Subject: [PATCH 1/4] spec: add immutability principle (Section 2.11) - Add Section 2.11 'Immutable Report Artifacts' to Design Principles - Add immutability note to Section 5.3 (reportId) - Add immutability requirements to Section 19.1 (Producer Conformance) - Add immutability expectations to Section 19.2 (Consumer Conformance) --- CHANGELOG.md | 1 + spec/ctrf.md | 26 ++++++++++++++++++++++++++ 2 files changed, 27 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 11cb5a6..6108e54 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,5 +9,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Changed +- Added immutability guidance for emitted CTRF report artifacts. - Clarified `tags` as simple keyless classifications and `labels` as structured key-value test metadata ([#54](https://github.com/ctrf-io/ctrf/pull/54)). - Allowed non-empty arrays of strings, numbers, and booleans as label values ([#54](https://github.com/ctrf-io/ctrf/pull/54)). diff --git a/spec/ctrf.md b/spec/ctrf.md index 3a821ea..c7c834b 100644 --- a/spec/ctrf.md +++ b/spec/ctrf.md @@ -238,6 +238,26 @@ As long as the document structure and semantics are respected, CTRF does not con --- +### 2.11. Immutable Report Artifacts + +CTRF treats an emitted report as an immutable artifact. + +A CTRF document represents a snapshot of test results at the time it is produced. Implementations are free to build or transform documents internally before emission, but once a CTRF document has been written, published, or otherwise made available as an artifact, it is treated as a fixed record. + +Immutability supports: + +- auditability of test results +- reproducibility of analysis +- stable caching and artifact storage +- compatibility with hashing, signing, and deduplication +- clear separation between raw results and derived results + +When a tool or system performs additional processing after initial report generation — such as merging shard reports, adding metadata, or computing insights — the design intent is that this produces a new CTRF document with a new `reportId`, rather than modifying the original. + +This distinction between the act of constructing a report and the resulting artifact as an immutable record is central to how CTRF is designed to be used in pipelines, artifact stores, and analysis systems. + +--- + ## 3. Terminology The key words **MUST**, **MUST NOT**, **REQUIRED**, **SHALL**, **SHALL NOT**, @@ -405,6 +425,8 @@ A unique identifier for this report instance. `reportId` is OPTIONAL. If present, it MUST be a valid UUID as defined in [RFC4122]. +Each emitted CTRF document represents a distinct report instance. If a new CTRF document is produced through post-processing, it SHOULD receive a new `reportId`. + --- ### 5.4. `timestamp` @@ -1764,6 +1786,8 @@ Producers: - MUST NOT introduce fields outside `extra` objects - MUST NOT emit invalid enum values (status, etc.) - MAY include `insights` if historical or aggregate data is available +- SHOULD treat emitted CTRF documents as immutable artifacts +- SHOULD emit a new CTRF document with a new `reportId` when performing post-processing after initial report generation, rather than modifying an emitted CTRF document Producers SHOULD: @@ -1784,6 +1808,8 @@ Consumers: - MUST NOT require optional fields to be present - MUST NOT assume the presence of `insights` - SHOULD compute insights when sufficient historical data is available +- SHOULD assume that CTRF documents are immutable once emitted +- MUST NOT rely on in-place modification of existing reports --- From a0c84f9cf9b5ea9df9ca9b66215cab8e916a162f Mon Sep 17 00:00:00 2001 From: Matthew Thomas <25988124+Ma11hewThomas@users.noreply.github.com> Date: Sun, 26 Jul 2026 14:54:09 +0100 Subject: [PATCH 2/4] Link immutability changelog entry to PR --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 6108e54..cf5389a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Changed -- Added immutability guidance for emitted CTRF report artifacts. +- Added immutability guidance for emitted CTRF report artifacts ([#55](https://github.com/ctrf-io/ctrf/pull/55)). - Clarified `tags` as simple keyless classifications and `labels` as structured key-value test metadata ([#54](https://github.com/ctrf-io/ctrf/pull/54)). - Allowed non-empty arrays of strings, numbers, and booleans as label values ([#54](https://github.com/ctrf-io/ctrf/pull/54)). From f2e2f518ff7295daa2d9fd5e846da65650eebf90 Mon Sep 17 00:00:00 2001 From: Matthew Thomas <25988124+Ma11hewThomas@users.noreply.github.com> Date: Sun, 26 Jul 2026 15:08:28 +0100 Subject: [PATCH 3/4] Clarify optional reportId immutability guidance --- CHANGELOG.md | 2 +- spec/ctrf.md | 6 +++--- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index cf5389a..173c0ea 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Changed -- Added immutability guidance for emitted CTRF report artifacts ([#55](https://github.com/ctrf-io/ctrf/pull/55)). +- Clarified immutability guidance for emitted CTRF report artifacts ([#55](https://github.com/ctrf-io/ctrf/pull/55)). - Clarified `tags` as simple keyless classifications and `labels` as structured key-value test metadata ([#54](https://github.com/ctrf-io/ctrf/pull/54)). - Allowed non-empty arrays of strings, numbers, and booleans as label values ([#54](https://github.com/ctrf-io/ctrf/pull/54)). diff --git a/spec/ctrf.md b/spec/ctrf.md index c7c834b..66355f7 100644 --- a/spec/ctrf.md +++ b/spec/ctrf.md @@ -252,7 +252,7 @@ Immutability supports: - compatibility with hashing, signing, and deduplication - clear separation between raw results and derived results -When a tool or system performs additional processing after initial report generation — such as merging shard reports, adding metadata, or computing insights — the design intent is that this produces a new CTRF document with a new `reportId`, rather than modifying the original. +When a tool or system performs additional processing after initial report generation — such as merging shard reports, adding metadata, or computing insights — the design intent is that this produces a new CTRF document and, when `reportId` is used, assigns a different `reportId` value, rather than modifying the original. This distinction between the act of constructing a report and the resulting artifact as an immutable record is central to how CTRF is designed to be used in pipelines, artifact stores, and analysis systems. @@ -425,7 +425,7 @@ A unique identifier for this report instance. `reportId` is OPTIONAL. If present, it MUST be a valid UUID as defined in [RFC4122]. -Each emitted CTRF document represents a distinct report instance. If a new CTRF document is produced through post-processing, it SHOULD receive a new `reportId`. +Each emitted CTRF document represents a distinct report instance. If a new CTRF document is produced through post-processing, it SHOULD use a different `reportId` value than the source document when `reportId` is used. --- @@ -1787,7 +1787,7 @@ Producers: - MUST NOT emit invalid enum values (status, etc.) - MAY include `insights` if historical or aggregate data is available - SHOULD treat emitted CTRF documents as immutable artifacts -- SHOULD emit a new CTRF document with a new `reportId` when performing post-processing after initial report generation, rather than modifying an emitted CTRF document +- SHOULD emit a new CTRF document when performing post-processing after initial report generation and, when `reportId` is used, assign a different `reportId` value rather than modifying an emitted CTRF document Producers SHOULD: From 46c204edec413aa4d964301a247a571880d82f89 Mon Sep 17 00:00:00 2001 From: Matthew Thomas <25988124+Ma11hewThomas@users.noreply.github.com> Date: Sun, 26 Jul 2026 15:16:21 +0100 Subject: [PATCH 4/4] Move immutability producer guidance under SHOULD --- spec/ctrf.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/spec/ctrf.md b/spec/ctrf.md index 66355f7..21f123b 100644 --- a/spec/ctrf.md +++ b/spec/ctrf.md @@ -1786,14 +1786,14 @@ Producers: - MUST NOT introduce fields outside `extra` objects - MUST NOT emit invalid enum values (status, etc.) - MAY include `insights` if historical or aggregate data is available -- SHOULD treat emitted CTRF documents as immutable artifacts -- SHOULD emit a new CTRF document when performing post-processing after initial report generation and, when `reportId` is used, assign a different `reportId` value rather than modifying an emitted CTRF document Producers SHOULD: - provide timestamps - provide environment/build metadata - provide diagnostics when available +- treat emitted CTRF documents as immutable artifacts +- emit a new CTRF document when performing post-processing after initial report generation and, when `reportId` is used, assign a different `reportId` value rather than modifying an emitted CTRF document ---