From 41bb2582dd383ae8fb81a19a10d9c5b6bdd4ba1d Mon Sep 17 00:00:00 2001 From: Simon Novak <25969+snovak7@users.noreply.github.com> Date: Mon, 19 Oct 2026 20:53:23 +0200 Subject: [PATCH 1/2] story 10-1-releases-with-one-version-and-firmware-binaries: implemented and reviewed via bmad-loop (awaiting operator) --- ...ith-one-version-and-firmware-binaries.json | 11 + .github/workflows/release.yml | 525 +++++++++++++++++- .gitignore | 4 + .gitleaks.toml | 11 + .../implementation-artifacts/deferred-work.md | 16 + .../epic-10-context.md | 42 ++ ...-with-one-version-and-firmware-binaries.md | 210 +++++++ .../sprint-status.yaml | 2 +- apps/cs/server/Devices/DeviceGrain.cs | 7 +- apps/cs/server/Devices/NodeFrameReader.cs | 4 +- aspire/keycloak/realms/coldframe-realm.json | 1 + deploy/README.md | 1 + deploy/charts/set-version.sh | 52 ++ deploy/charts/test.sh | 33 +- deploy/images/README.md | 5 +- deploy/release/check-wire-compat.sh | 64 +++ deploy/release/checksums.sh | 60 ++ deploy/release/release-notes.sh | 156 ++++++ deploy/release/test.sh | 238 ++++++++ docs/build/apps.md | 36 ++ docs/operations/install.md | 8 +- docs/operations/release.md | 140 +++++ docs/quickstart.md | 8 +- packages/crypto-spec/crypto-spec.json | 1 + packages/crypto-spec/scripts/lib/spec.ts | 2 + packages/cs/crypto/Generated/CryptoSpec.g.cs | 5 + .../coldframe/core/crypto/CryptoSpec.kt | 3 + packages/proto/README.md | 8 +- packages/rs/crypto/src/spec.rs | 3 + .../server.integration/Devices/IngestTests.cs | 16 +- .../server.integration/Edge/EdgeApiFixture.cs | 130 ++++- .../Edge/InvitationTests.cs | 59 +- .../Identity/IdentityCluster.cs | 9 +- tests/cs/server.integration/KeycloakTests.cs | 3 + .../Devices/NodeFrameReaderTests.cs | 32 +- 35 files changed, 1835 insertions(+), 70 deletions(-) create mode 100644 .bmad-loop/operator/10-1-releases-with-one-version-and-firmware-binaries.json create mode 100644 _bmad-output/implementation-artifacts/epic-10-context.md create mode 100644 _bmad-output/implementation-artifacts/spec-10-1-releases-with-one-version-and-firmware-binaries.md create mode 100755 deploy/charts/set-version.sh create mode 100755 deploy/release/check-wire-compat.sh create mode 100755 deploy/release/checksums.sh create mode 100755 deploy/release/release-notes.sh create mode 100755 deploy/release/test.sh create mode 100644 docs/build/apps.md create mode 100644 docs/operations/release.md diff --git a/.bmad-loop/operator/10-1-releases-with-one-version-and-firmware-binaries.json b/.bmad-loop/operator/10-1-releases-with-one-version-and-firmware-binaries.json new file mode 100644 index 00000000..a8cd23bf --- /dev/null +++ b/.bmad-loop/operator/10-1-releases-with-one-version-and-firmware-binaries.json @@ -0,0 +1,11 @@ +{ + "actions": [ + "After this PR merges, cut the first release from main: keep the charts at 0.1.0 (or run deploy/charts/set-version.sh in a chore(release) PR and merge it), then push the tag: git tag -a v0.1.0 -m 'Coldframe 0.1.0' && git push origin v0.1.0.", + "Watch the Release workflow run for that tag and confirm it is green and that the GitHub Release v0.1.0 has coldframe-hub/node-0.1.0.bin and .elf, every -0.1.0.tgz, SHA256SUMS and the grouped notes with the docs/build/apps.md link.", + "Confirm ghcr.io/escendit/coldframe/{server,web,migrations,keycloak}:0.1.0 exist, then download the Release assets and run sha256sum --check --strict SHA256SUMS." + ], + "parked_at": "2026-10-19", + "run_id": "20261012-230921-d74a", + "spec_file": "_bmad-output/implementation-artifacts/spec-10-1-releases-with-one-version-and-firmware-binaries.md", + "story_key": "10-1-releases-with-one-version-and-firmware-binaries" +} \ No newline at end of file diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 693b385a..5463e310 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -1,73 +1,467 @@ -# Publishes the container images of a release (Stories 2.1 and 2.2, AD-23). A tag vX.Y.Z (or -# vX.Y.Z-) pushes ghcr.io/escendit/coldframe/{server,web,migrations,keycloak}:X.Y.Z -# as linux/amd64 and linux/arm64 manifest lists, only after the same verification CI runs. The -# tag must equal the version of every Helm chart in deploy/charts (and the appVersion of the -# Coldframe charts), or nothing is pushed. -# Only the exact version tag is pushed, never "latest". +# Publishes a release (Stories 2.1, 2.2 and 10.1, AD-23). One tag vX.Y.Z (or vX.Y.Z-) +# produces everything that belongs to the version X.Y.Z: # -# A component whose final base image lacks a platform is pushed for the platforms it has; the -# run warns and the step summary names the missing architecture. +# - the container images ghcr.io/escendit/coldframe/{server,web,migrations,keycloak}:X.Y.Z as +# linux/amd64 and linux/arm64 manifest lists, only after the same verification CI runs. Only the +# exact version tag is pushed, never "latest". A component whose final base image lacks a +# platform is pushed for the platforms it has; the run warns and the step summary names it; +# - a GitHub Release vX.Y.Z (a pre-release for vX.Y.Z-) with the release builds of the +# Hub and Node firmware (coldframe-{hub,node}-X.Y.Z.bin, a merged flash image for offset 0, and +# coldframe-{hub,node}-X.Y.Z.elf), every Helm chart of deploy/charts packaged as +# -X.Y.Z.tgz, a SHA256SUMS over those assets, and notes grouped by Conventional Commit +# type (deploy/release/release-notes.sh). No mobile store binaries are published. # -# Actions are pinned to a commit; the comment names the release. Registry auth uses only -# GITHUB_TOKEN, and only the publish job may write packages. +# Gates, before anything is pushed: the tag must equal the version of every Helm chart (and the +# appVersion of the Coldframe charts; deploy/charts/check-version.sh); the wire gate +# (deploy/release/check-wire-compat.sh) fails a protocolMajor bump over the previous release tag +# unless the Server still accepts the previous major; a release firmware build must refuse dev mode; +# and the assets must be exactly the expected ones with a verified SHA256SUMS (the assets job). No +# image is pushed and no Release is created unless the mode, checks, firmware, charts, notes and +# assets jobs all succeed; the Release is created last, after the images are pushed. +# +# Dry run: on a pull request that touches the release paths, and on a manual dispatch, the same +# jobs run with the version of deploy/charts/server/Chart.yaml: every asset is built and checked, +# the images are built without being pushed, and the assets, SHA256SUMS and notes are uploaded as +# one workflow artifact. Nothing is pushed or released. docs/operations/release.md is the runbook. +# +# Actions are pinned to a commit; the comment names the release. The helm download is checked +# against a sha256; espup and espflash are pinned crates.io versions installed with --locked, as in +# ci.yml, and espup installs the pinned esp toolchain version. Registry auth uses only GITHUB_TOKEN; only the publish job may write packages, +# and only the release job may write contents. name: Release on: push: tags: - "v*" + pull_request: + paths: + - .github/workflows/release.yml + - deploy/release/** + - deploy/charts/*/Chart.yaml + - deploy/charts/check-version.sh + - deploy/charts/set-version.sh + - deploy/images/release-version.sh + - packages/crypto-spec/crypto-spec.json + - packages/crypto-spec/scripts/** + - apps/rs/hub/check-image.sh + - apps/rs/hub/Cargo.toml + - apps/rs/hub/Cargo.lock + - apps/rs/hub/partitions.csv + - apps/rs/node/Cargo.toml + - apps/rs/node/Cargo.lock + - apps/rs/node/partitions.csv + workflow_dispatch: permissions: contents: read concurrency: group: ${{ github.workflow }}-${{ github.ref }} - cancel-in-progress: false + cancel-in-progress: ${{ github.event_name == 'pull_request' }} jobs: - version: - name: Version + mode: + name: Mode runs-on: ubuntu-24.04 timeout-minutes: 5 outputs: - version: ${{ steps.version.outputs.version }} + dry_run: ${{ steps.mode.outputs.dry_run }} + version: ${{ steps.mode.outputs.version }} + prerelease: ${{ steps.mode.outputs.prerelease }} steps: - name: Check out uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: persist-credentials: false - - name: Validate the tag - id: version + # A tag push releases the tag; anything else is a dry run at the version of the charts. + - name: Resolve the mode and the version + id: mode shell: bash env: - TAG: ${{ github.ref_name }} + EVENT: ${{ github.event_name }} + REF_NAME: ${{ github.ref_name }} run: | - if ! version=$(deploy/images/release-version.sh "${TAG}"); then - echo "::error::Tag '${TAG}' is not a release tag: expected vX.Y.Z or vX.Y.Z-. Nothing is pushed." + if [[ ${EVENT} == "push" ]]; then + dry_run=false + tag=${REF_NAME} + else + dry_run=true + tag="v$(sed -n -E 's/^version:[[:space:]]*"?([^"[:space:]]*)"?[[:space:]]*$/\1/p' deploy/charts/server/Chart.yaml)" + fi + if ! version=$(deploy/images/release-version.sh "${tag}"); then + echo "::error::Tag '${tag}' is not a release tag: expected vX.Y.Z or vX.Y.Z-. Nothing is pushed." exit 1 fi - echo "Release version: ${version}" - echo "version=${version}" >> "${GITHUB_OUTPUT}" + prerelease=false + [[ ${version} == *-* ]] && prerelease=true + echo "Dry run: ${dry_run}; version: ${version}; pre-release: ${prerelease}" + { + echo "dry_run=${dry_run}" + echo "version=${version}" + echo "prerelease=${prerelease}" + } >> "${GITHUB_OUTPUT}" + + checks: + name: Checks + needs: mode + runs-on: ubuntu-24.04 + timeout-minutes: 10 + outputs: + previous_tag: ${{ steps.previous.outputs.tag }} + env: + VERSION: ${{ needs.mode.outputs.version }} + steps: + - name: Check out + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + fetch-depth: 0 + persist-credentials: false + + - name: Test the release scripts + run: deploy/release/test.sh - name: Check the chart versions shell: bash - env: - VERSION: ${{ steps.version.outputs.version }} run: | if ! deploy/charts/check-version.sh "${VERSION}"; then echo "::error::The Helm charts in deploy/charts are not at ${VERSION}; see the output above. Nothing is pushed." exit 1 fi + # The newest release tag before the release commit; a final release skips pre-release tags. + # "-" when this is the first release. + - name: Find the previous release + id: previous + shell: bash + run: | + exclude=() + if [[ ${VERSION} != *-* ]]; then + exclude=(--exclude 'v*-*') + fi + if previous=$(git describe --tags --abbrev=0 --match 'v[0-9]*' "${exclude[@]}" "${GITHUB_SHA}^" 2>/dev/null); then + echo "Previous release: ${previous}" + else + previous=- + echo "::notice::No earlier v* tag: this is the first release." + fi + echo "tag=${previous}" >> "${GITHUB_OUTPUT}" + + - name: Wire-compatibility gate + shell: bash + env: + PREVIOUS: ${{ steps.previous.outputs.tag }} + run: | + spec=packages/crypto-spec/crypto-spec.json + previous_spec=- + if [[ ${PREVIOUS} != "-" ]]; then + if git cat-file -e "${PREVIOUS}:${spec}" 2>/dev/null; then + previous_spec="${RUNNER_TEMP}/previous-crypto-spec.json" + git show "${PREVIOUS}:${spec}" > "${previous_spec}" + else + echo "::notice::${PREVIOUS} has no ${spec}; the wire gate has nothing to compare against." + fi + fi + if ! output=$(deploy/release/check-wire-compat.sh "${previous_spec}" "${spec}" 2>&1); then + echo "${output}" + echo "::error::Wire gate: ${output} Nothing is pushed." + exit 1 + fi + echo "${output}" + if [[ ${previous_spec} == "-" ]]; then + echo "::notice::${output}" + fi + + firmware: + name: Firmware ${{ matrix.crate }} + needs: [mode, checks] + runs-on: ubuntu-24.04 + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + crate: [hub, node] + env: + # The pins of the firmware job of ci.yml. + ESPUP_VERSION: "0.17.1" + ESP_TOOLCHAIN_VERSION: "1.95.0.0" + ESPFLASH_VERSION: "4.5.0" + CRATE: ${{ matrix.crate }} + VERSION: ${{ needs.mode.outputs.version }} + defaults: + run: + working-directory: apps/rs/${{ matrix.crate }} + steps: + - name: Check out + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + + # cmake and ninja for the mbedTLS rebuild of the Hub; libudev for espflash. + - name: Install build tools + run: sudo apt-get update && sudo apt-get install --yes --no-install-recommends cmake ninja-build libudev-dev pkg-config + + # Dry runs only: a release build starts clean, with no cached tools, toolchain or target. + - name: Cache + if: needs.mode.outputs.dry_run == 'true' + uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2.9.2 + with: + workspaces: apps/rs/${{ matrix.crate }} + key: release-${{ matrix.crate }}-espup-${{ env.ESPUP_VERSION }}-esp-${{ env.ESP_TOOLCHAIN_VERSION }}-espflash-${{ env.ESPFLASH_VERSION }} + cache-bin: "true" + cache-directories: | + ~/.rustup/toolchains/esp + ~/export-esp.sh + + # On the runner's stable toolchain: the crate's rust-toolchain.toml names `esp`, which is not + # installed until the next step. + - name: Install espup and espflash + run: | + if [[ "$(espup --version 2>/dev/null)" != "espup ${ESPUP_VERSION}" ]]; then + rustup run stable cargo install espup --version "${ESPUP_VERSION}" --locked --force + fi + if [[ "$(espflash --version 2>/dev/null)" != "espflash ${ESPFLASH_VERSION}" ]]; then + rustup run stable cargo install espflash --version "${ESPFLASH_VERSION}" --locked --force + fi + + - name: Install the esp toolchain + env: + GITHUB_TOKEN: ${{ github.token }} + run: | + if [[ ! -f "${HOME}/export-esp.sh" ]] || ! rustup run esp rustc --version | grep -qF "(${ESP_TOOLCHAIN_VERSION})"; then + espup install --toolchain-version "${ESP_TOOLCHAIN_VERSION}" --targets esp32s3 --export-file "${HOME}/export-esp.sh" + fi + rustup run esp rustc --version + + # AD-12: a release build with dev mode must fail with the compile_error! message. + - name: Assert the release build refuses dev-mode + run: | + source "${HOME}/export-esp.sh" + if cargo build --release --features dev-mode --locked > dev-mode-release.log 2>&1; then + cat dev-mode-release.log + echo "::error::${CRATE}: a release build with dev-mode succeeded" + exit 1 + fi + # shellcheck disable=SC2016 # the backticks are part of the message + if ! grep -qF 'the `dev-mode` feature is refused in release builds' dev-mode-release.log; then + cat dev-mode-release.log + echo "::error::${CRATE}: the release + dev-mode build failed without the compile_error! message" + exit 1 + fi + + # FR-1: no credential canary reaches the Hub ELF or its flash image. It builds with canary + # values in the environment, so it runs before the release build whose ELF is shipped. + - name: Check the image + if: matrix.crate == 'hub' + run: | + source "${HOME}/export-esp.sh" + ./check-image.sh + + - name: Release build + run: | + source "${HOME}/export-esp.sh" + cargo build --release --locked + + # One merged image (bootloader, partition table, app) that flashes at offset 0. + - name: Save the flash image + run: | + elf="target/xtensa-esp32s3-none-elf/release/coldframe-${CRATE}" + out="${RUNNER_TEMP}/firmware" + mkdir -p "${out}" + espflash save-image --chip esp32s3 --merge --partition-table partitions.csv \ + "${elf}" "${out}/coldframe-${CRATE}-${VERSION}.bin" + cp "${elf}" "${out}/coldframe-${CRATE}-${VERSION}.elf" + # An ESP image starts with the magic byte 0xE9; the merged one with the bootloader's. + magic=$(head -c 1 "${out}/coldframe-${CRATE}-${VERSION}.bin" | od -An -tx1 | tr -d ' ') + if [[ ${magic} != "e9" ]]; then + echo "::error::${CRATE}: the flash image does not start with an ESP image header (got '${magic}')" + exit 1 + fi + ls -l "${out}" + + - name: Upload the firmware + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 + with: + name: firmware-${{ matrix.crate }} + path: ${{ runner.temp }}/firmware/ + if-no-files-found: error + retention-days: 7 + + charts: + name: Charts + needs: [mode, checks] + runs-on: ubuntu-24.04 + timeout-minutes: 10 + env: + HELM_VERSION: v4.2.2 + # sha256 of helm-v4.2.2-linux-amd64.tar.gz (get.helm.sh/.sha256sum). + HELM_SHA256: 9adafecab4d406853bba163a70e9f104f47dbbf65ce24b7653bae7e36150bcb6 + VERSION: ${{ needs.mode.outputs.version }} + steps: + - name: Check out + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + + - name: Install helm + shell: bash + run: | + bin="${RUNNER_TEMP}/bin" + downloads="${RUNNER_TEMP}/downloads" + mkdir -p "${bin}" "${downloads}" + curl --fail --silent --show-error --location --output "${downloads}/helm.tar.gz" \ + "https://get.helm.sh/helm-${HELM_VERSION}-linux-amd64.tar.gz" + echo "${HELM_SHA256} ${downloads}/helm.tar.gz" | sha256sum --check --strict + tar --extract --gzip --file "${downloads}/helm.tar.gz" --directory "${downloads}" linux-amd64/helm + install -m 0755 "${downloads}/linux-amd64/helm" "${bin}/helm" + echo "${bin}" >> "${GITHUB_PATH}" + + - name: Package the charts + shell: bash + run: | + out="${RUNNER_TEMP}/charts" + mkdir -p "${out}" + for chart_file in deploy/charts/*/Chart.yaml; do + helm package "$(dirname "${chart_file}")" --destination "${out}" + done + for package in "${out}"/*.tgz; do + if [[ ${package} != *"-${VERSION}.tgz" ]]; then + echo "::error::$(basename "${package}") does not carry the version ${VERSION}" + exit 1 + fi + done + ls -l "${out}" + + - name: Upload the charts + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 + with: + name: charts + path: ${{ runner.temp }}/charts/ + if-no-files-found: error + retention-days: 7 + + notes: + name: Notes + needs: [mode, checks] + runs-on: ubuntu-24.04 + timeout-minutes: 5 + env: + VERSION: ${{ needs.mode.outputs.version }} + PREVIOUS: ${{ needs.checks.outputs.previous_tag }} + steps: + - name: Check out + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + fetch-depth: 0 + persist-credentials: false + + - name: Write the release notes + shell: bash + env: + RELEASE_REPOSITORY: ${{ github.repository }} + run: | + mkdir -p "${RUNNER_TEMP}/notes" + deploy/release/release-notes.sh "${VERSION}" "${PREVIOUS}" "${GITHUB_SHA}" > "${RUNNER_TEMP}/notes/release-notes.md" + cat "${RUNNER_TEMP}/notes/release-notes.md" >> "${GITHUB_STEP_SUMMARY}" + + - name: Upload the notes + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 + with: + name: release-notes + path: ${{ runner.temp }}/notes/release-notes.md + if-no-files-found: error + retention-days: 7 + + # Every Release asset, checked before any image is pushed: exactly the expected files and a + # verified SHA256SUMS. The verified bundle (assets, SHA256SUMS, notes) is what the release job + # publishes. + assets: + name: Assets + needs: [mode, checks, firmware, charts, notes] + runs-on: ubuntu-24.04 + timeout-minutes: 10 + env: + VERSION: ${{ needs.mode.outputs.version }} + steps: + - name: Check out + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + + - name: Download the firmware + uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 + with: + pattern: firmware-* + path: ${{ runner.temp }}/release/assets + merge-multiple: true + + - name: Download the charts + uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 + with: + name: charts + path: ${{ runner.temp }}/release/assets + + - name: Download the notes + uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 + with: + name: release-notes + path: ${{ runner.temp }}/release + + # Exactly the expected assets: the four firmware files and one package per chart. + - name: Check the assets + shell: bash + run: | + assets="${RUNNER_TEMP}/release/assets" + expected=() + for crate in hub node; do + expected+=("coldframe-${crate}-${VERSION}.bin" "coldframe-${crate}-${VERSION}.elf") + done + for chart_file in deploy/charts/*/Chart.yaml; do + name=$(sed -n -E 's/^name:[[:space:]]*"?([^"[:space:]]*)"?[[:space:]]*$/\1/p' "${chart_file}" | head -n 1) + expected+=("${name}-${VERSION}.tgz") + done + want=$(printf '%s\n' "${expected[@]}" | LC_ALL=C sort) + got=$(find "${assets}" -mindepth 1 -maxdepth 1 -printf '%f\n' | LC_ALL=C sort) + if [[ ${want} != "${got}" ]]; then + echo "Expected:"; echo "${want}" + echo "Found:"; echo "${got}" + echo "::error::The release assets are not exactly the expected ones; see the output above." + exit 1 + fi + echo "${got}" + + - name: Write and verify SHA256SUMS + run: deploy/release/checksums.sh "${RUNNER_TEMP}/release/assets" + + - name: Upload the verified bundle + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 + with: + name: release-bundle + path: ${{ runner.temp }}/release/ + if-no-files-found: error + retention-days: 7 + verify: name: Verify - needs: version + needs: [mode, checks] + if: needs.mode.outputs.dry_run == 'false' uses: ./.github/workflows/images-verify.yml publish: name: Publish ${{ matrix.component }} - needs: [version, verify] + needs: [mode, checks, firmware, charts, notes, assets, verify] + # verify is skipped on a dry run; every other job must have succeeded. + if: >- + always() + && needs.mode.result == 'success' + && needs.checks.result == 'success' + && needs.firmware.result == 'success' + && needs.charts.result == 'success' + && needs.notes.result == 'success' + && needs.assets.result == 'success' + && (needs.verify.result == 'success' + || (needs.mode.outputs.dry_run == 'true' && needs.verify.result == 'skipped')) runs-on: ubuntu-24.04 timeout-minutes: 45 permissions: @@ -88,7 +482,8 @@ jobs: env: COMPONENT: ${{ matrix.component }} DOCKERFILE: ${{ matrix.dockerfile }} - VERSION: ${{ needs.version.outputs.version }} + VERSION: ${{ needs.mode.outputs.version }} + DRY_RUN: ${{ needs.mode.outputs.dry_run }} steps: - name: Check out uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 @@ -101,6 +496,7 @@ jobs: driver: docker-container - name: Log in to ghcr.io + if: needs.mode.outputs.dry_run == 'false' uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0 with: registry: ghcr.io @@ -123,6 +519,7 @@ jobs: echo "platforms=${platforms}" >> "${GITHUB_OUTPUT}" echo "missing=${missing}" >> "${GITHUB_OUTPUT}" + # A dry run builds every platform the release would push, without --push. - name: Build and push id: push shell: bash @@ -132,6 +529,11 @@ jobs: SOURCE: ${{ github.server_url }}/${{ github.repository }} run: | image="ghcr.io/${GITHUB_REPOSITORY,,}/${COMPONENT}:${VERSION}" + push=(--push) + if [[ ${DRY_RUN} == "true" ]]; then + push=() + echo "Dry run: ${image} is built, not pushed." + fi docker buildx build \ --platform "${PLATFORMS}" \ --file "${DOCKERFILE}" \ @@ -140,7 +542,7 @@ jobs: --label "org.opencontainers.image.version=${VERSION}" \ --label "org.opencontainers.image.revision=${REVISION}" \ --label "org.opencontainers.image.licenses=Apache-2.0" \ - --push \ + "${push[@]}" \ . echo "image=${image}" >> "${GITHUB_OUTPUT}" @@ -149,12 +551,75 @@ jobs: env: IMAGE: ${{ steps.push.outputs.image }} MISSING: ${{ steps.platforms.outputs.missing }} + PLATFORMS: ${{ steps.platforms.outputs.platforms }} run: | - pushed=$(docker buildx imagetools inspect --raw "${IMAGE}" \ - | jq -r '.manifests[] | select(.platform.os != "unknown") | "\(.platform.os)/\(.platform.architecture)"' \ - | sort -u | paste -sd, -) + if [[ ${DRY_RUN} == "true" ]]; then + pushed="${PLATFORMS} (built, not pushed)" + else + pushed=$(docker buildx imagetools inspect --raw "${IMAGE}" \ + | jq -r '.manifests[] | select(.platform.os != "unknown") | "\(.platform.os)/\(.platform.architecture)"' \ + | sort -u | paste -sd, -) + fi { echo "| Component | Image | Platforms | Missing |" echo "| --- | --- | --- | --- |" echo "| ${COMPONENT} | \`${IMAGE}\` | ${pushed} | ${MISSING:-none} |" } >> "${GITHUB_STEP_SUMMARY}" + + release: + name: Release + needs: [mode, assets, publish] + if: >- + always() + && needs.mode.result == 'success' + && needs.assets.result == 'success' + && needs.publish.result == 'success' + runs-on: ubuntu-24.04 + timeout-minutes: 10 + permissions: + contents: write + env: + VERSION: ${{ needs.mode.outputs.version }} + DRY_RUN: ${{ needs.mode.outputs.dry_run }} + PRERELEASE: ${{ needs.mode.outputs.prerelease }} + steps: + - name: Check out + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + + - name: Download the verified bundle + uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 + with: + name: release-bundle + path: ${{ runner.temp }}/release + + - name: Verify SHA256SUMS + run: deploy/release/checksums.sh --check "${RUNNER_TEMP}/release/assets" + + - name: Upload the dry-run release + if: needs.mode.outputs.dry_run == 'true' + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 + with: + name: release-${{ needs.mode.outputs.version }}-dry-run + path: ${{ runner.temp }}/release/ + if-no-files-found: error + retention-days: 7 + + - name: Create the GitHub Release + if: needs.mode.outputs.dry_run == 'false' + shell: bash + env: + GH_TOKEN: ${{ github.token }} + GH_REPO: ${{ github.repository }} + run: | + flags=() + if [[ ${PRERELEASE} == "true" ]]; then + flags+=(--prerelease) + fi + gh release create "v${VERSION}" \ + --verify-tag \ + --title "v${VERSION}" \ + --notes-file "${RUNNER_TEMP}/release/release-notes.md" \ + "${flags[@]}" \ + "${RUNNER_TEMP}/release/assets/"* diff --git a/.gitignore b/.gitignore index 4b25842c..cf398efe 100644 --- a/.gitignore +++ b/.gitignore @@ -26,6 +26,8 @@ mono_crash.* !apps/kt/android/src/debug/** [Dd]ebugPublic/ [Rr]elease/ +# ...except the release scripts (Story 10.1) +!deploy/release/ [Rr]eleases/ [Dd]ebug/x64/ @@ -460,6 +462,8 @@ local.properties .kotlin/ build/ !gradle/wrapper/gradle-wrapper.jar +# ...except the adopter build guide +!docs/build/ # SwiftPM, Xcode .build/ diff --git a/.gitleaks.toml b/.gitleaks.toml index 3fc5222a..89e86f69 100644 --- a/.gitleaks.toml +++ b/.gitleaks.toml @@ -29,3 +29,14 @@ commits = ["b515f4c91cd5d75c24d4e244b9671efc7e475f26"] paths = [ '''^_bmad-output/implementation-artifacts/spec-2-4-tls-with-public-certificates-on-my-home-network\.md$''', ] + +# The Epic 10 context lists the firmware reference topics ("the eFuse root and HKDF purpose-key +# hierarchy, ..."); the generic API key rule reads the prose as a key. Only that commit and file are +# allowed. +[[allowlists]] +description = "Epic 10 context names a key hierarchy, not a key" +condition = "AND" +commits = ["ee32e273dbf061a62ab15117f464144ea5bce61c"] +paths = [ + '''^_bmad-output/implementation-artifacts/epic-10-context\.md$''', +] diff --git a/_bmad-output/implementation-artifacts/deferred-work.md b/_bmad-output/implementation-artifacts/deferred-work.md index bbe09ead..80bd2b1a 100644 --- a/_bmad-output/implementation-artifacts/deferred-work.md +++ b/_bmad-output/implementation-artifacts/deferred-work.md @@ -930,3 +930,19 @@ location: tests/cs/server.integration/Edge/InvitationTests.cs source_spec: `spec-9-1-invite-someone-to-my-site.md` reason: InvitationTests asserts the link equals the configured redirect URI and accepts by signing in on Keycloak directly; the web app is not part of the AppHost, so link -> web sign-in -> invitation required action is unverified end to end. Settle by running the web app against the AppHost and following the Mailpit link. status: open + +### DW-120: A v* tag on any commit publishes; the documented "tag only from main" rule is not enforced by the workflow. +origin: spec-deferred 7aa938ff9d90 +location: .github/workflows/release.yml (mode job) +source_spec: `spec-10-1-releases-with-one-version-and-firmware-binaries.md` +severity: low +reason: Pre-existing since Story 2.1 (deploy/images/README.md already asks for a tag ruleset); the mode job accepts any v* push. +status: open + +### DW-121: The tag-only branches of release.yml (ghcr login, --push, imagetools summary, verify, gh release create, --prerelease) never run in the dry run. +origin: spec-deferred bce640467f2d +location: .github/workflows/release.yml (publish, release jobs) +source_spec: `spec-10-1-releases-with-one-version-and-firmware-binaries.md` +severity: medium +reason: dry_run is true for every non-push event, so these steps are skipped on PRs; only a real v* tag exercises them. Owed as an operator action (cut the first release and check it). +status: open diff --git a/_bmad-output/implementation-artifacts/epic-10-context.md b/_bmad-output/implementation-artifacts/epic-10-context.md new file mode 100644 index 00000000..b0836399 --- /dev/null +++ b/_bmad-output/implementation-artifacts/epic-10-context.md @@ -0,0 +1,42 @@ +# Epic 10 Context: Let others rebuild Coldframe + + + +## Goal + +Make Coldframe reproducible by someone other than the author, using only the public repository and docs. A maker should be able to build a Node and a Hub, flash them, deploy the full stack on their own home server, build their own mobile apps, and get a Threshold Alert, with every piece tied to one released version. This epic delivers the release pipeline (one SemVer plus firmware binaries), the hardware designs, the adopter build and deploy guides, the Compose reference example, and the firmware learning reference. It closes out the open-source and reproducibility promises and is validated by a real reproduction run. + +## Stories + +- Story 10.1: Releases with one version and firmware binaries +- Story 10.2: Build a Node and a Hub from the docs +- Story 10.3: Deploy Coldframe at home and build my own apps +- Story 10.4: Firmware learning reference +- Story 10.5: Someone else reproduces Coldframe + +## Requirements & Constraints + +- **Reproducibility:** another person can build the Node and Hub and run the full stack from the public docs alone. Container images are provided. The supported deployment is single-node RKE2 with Fleet. Compose is a reference example only and must be labelled that way. +- **Open source:** firmware, Server, both apps and the Node/Hub hardware design are public under Apache-2.0. No proprietary vendor libraries are used (for example no Bosch BSEC; the air Sensor reports raw gas resistance). Every dependency must be available from a public registry or repository. +- **Firmware as a learning reference:** the Rust firmware is documented well enough to learn ESP-NOW, BLE provisioning and deep sleep from it. +- **Success criterion:** at least one person other than the author (or the author on a clean machine with fresh hardware and no prior state) reproduces the setup end to end. Every point where they get stuck is filed as an issue and fixed in the docs before the epic closes, and the checklist results are committed to the repo. +- **Test-first where it applies:** link checks, the doc build and documented command snippets are exercised in CI (firmware build only, never flashing). The full reproduction is verified manually against a checklist. +- **Facts the docs must state correctly:** the eFuse key burn is irreversible (with a prominent warning and dev-mode guidance for test boards). Sleep-current budget is about 100 µA average or less. Soil moisture % and battery % are approximate. WPA3-only Wi-Fi is unsupported. A short button press sends a Reading now and a long press enters BLE setup. + +## Technical Decisions + +- **One version:** one SemVer per release tag `vX.Y.Z` across the monorepo. Every Helm chart's `appVersion` equals that tag. Images go to `ghcr.io/escendit/coldframe/:X.Y.Z`, multi-arch where supported (the image publishing itself comes from Epic 2). +- **Firmware release:** Hub and Node binaries are attached to each release with checksums. They are always release builds. Release builds refuse to compile with dev mode on. Firmware is flashed over USB with `espflash` in V1; there is no OTA. +- **Release notes** are grouped by Conventional Commit type (commit scopes are per app or package). +- **Wire compatibility gate:** a release that bumps a wire-protocol major must fail unless the Server still accepts the previous major. Every frame and envelope carries `protocol_version`. +- **Mobile:** no store binaries are published. Adopters build the apps from source with their own APNs key and FCM project. The Server URL and Keycloak issuer are build-time configuration, and the apps have no field to change them. +- **Deployment envelope the guide must follow:** single-node RKE2 with Fleet pulling the charts in `deploy/`. TLS uses public Let's Encrypt certificates via DNS-01 for a real domain, with split DNS to the LAN ingress (a private CA is only a documented fallback). Secrets are created out of band from the fixed list in `deploy/SECRETS.md`. Keycloak runs on Phase Two, and its realm setup includes the invitation email template with the home-network hint. CNPG backups go to off-node S3. +- **Repo layout:** `hardware/` holds schematics, BOM, wiring, enclosure and sealing notes. `docs/` holds the adopter build guide (`docs/build/`), the deploy guide (`docs/deploy/`), the firmware reference (`docs/firmware/`) and the reproduction checklist. `deploy/compose/` holds the Compose example. +- **Firmware reference topics, with pointers to code:** ESP-NOW transport and channel following, the custom BLE setup protocol with its proof-of-possession code, the eFuse root and HKDF purpose-key hierarchy, ChaCha20-Poly1305 frame sealing with counters and the replay window, deep sleep and the power budget, button presses, and the findings from the Hub radio-coexistence spike (`docs/spikes/hub-radio-coexistence.md`). +- **Rust docs:** `cargo doc` over `packages/rs/*` must document every public item and finish with no warnings in CI. + +## Cross-Story Dependencies + +- Epic 10 comes last. It documents and packages what Epics 1–9 built. It depends especially on Epic 2 (images, charts, Fleet bundles, `deploy/SECRETS.md`, ops runbooks, which the deploy guide links to), Epics 3–4 (Hub/Node firmware, dev mode, eFuse enrolment) and Epic 9 (the Phase Two invitation template). +- 10.1's release notes point to the build-your-own-apps guide in 10.3. 10.2 flashes the release binaries produced by 10.1. +- 10.5 requires 10.1–10.4 to be complete. It follows 10.2 and 10.3 end to end, and it may feed doc fixes back into them. diff --git a/_bmad-output/implementation-artifacts/spec-10-1-releases-with-one-version-and-firmware-binaries.md b/_bmad-output/implementation-artifacts/spec-10-1-releases-with-one-version-and-firmware-binaries.md new file mode 100644 index 00000000..7b2c73c6 --- /dev/null +++ b/_bmad-output/implementation-artifacts/spec-10-1-releases-with-one-version-and-firmware-binaries.md @@ -0,0 +1,210 @@ +--- +title: 'Releases with one version and firmware binaries' +type: 'feature' +created: '2026-10-19' +status: 'awaiting-operator' +baseline_revision: 'bb7c4a5eafcee3619348fb90361f17857cfad84f' +review_loop_iteration: 0 +followup_review_recommended: true +context: + - '{project-root}/_bmad-output/implementation-artifacts/epic-10-context.md' +warnings: [oversized] +deferred: + - summary: >- + A v* tag on any commit publishes; the documented "tag only from main" rule is not enforced by the workflow. + evidence: |- + Pre-existing since Story 2.1 (deploy/images/README.md already asks for a tag ruleset); the mode job accepts any v* push. + location: >- + .github/workflows/release.yml (mode job) + severity: low + - summary: >- + The tag-only branches of release.yml (ghcr login, --push, imagetools summary, verify, gh release create, --prerelease) never run in the dry run. + evidence: |- + dry_run is true for every non-push event, so these steps are skipped on PRs; only a real v* tag exercises them. Owed as an operator action (cut the first release and check it). + location: >- + .github/workflows/release.yml (publish, release jobs) + severity: medium +operator_actions: + - "After this PR merges, cut the first release from main: keep the charts at 0.1.0 (or run deploy/charts/set-version.sh in a chore(release) PR and merge it), then push the tag: git tag -a v0.1.0 -m 'Coldframe 0.1.0' && git push origin v0.1.0." + - "Watch the Release workflow run for that tag and confirm it is green and that the GitHub Release v0.1.0 has coldframe-hub/node-0.1.0.bin and .elf, every -0.1.0.tgz, SHA256SUMS and the grouped notes with the docs/build/apps.md link." + - "Confirm ghcr.io/escendit/coldframe/{server,web,migrations,keycloak}:0.1.0 exist, then download the Release assets and run sha256sum --check --strict SHA256SUMS." +--- + + + +## Intent + +**Problem:** A `vX.Y.Z` tag publishes only the container images today. Nothing attaches Hub and Node firmware or the charts to a release, no release notes are written, nothing stops a wire-major bump the Server cannot take, and the release path has never run outside a real tag. An adopter cannot tell which pieces belong together. + +**Approach:** Extend `.github/workflows/release.yml` so that one tag produces everything. That means the images (unchanged), Helm chart packages carrying `X.Y.Z`, release-build Hub and Node firmware, a `SHA256SUMS` file, and release notes grouped by conventional-commit type, all on a GitHub Release. A wire-compatibility gate runs before anything is pushed, and the same workflow runs in a dry-run mode on pull requests: it builds every artifact, verifies the checksums and publishes nothing. + +## Boundaries & Constraints + +**Always:** +- **One version.** `X.Y.Z` comes from `deploy/images/release-version.sh`, and `deploy/charts/check-version.sh` still gates the run. Every asset name carries the version: `coldframe-hub-X.Y.Z.bin`, `coldframe-hub-X.Y.Z.elf`, and the same for node and for `-X.Y.Z.tgz`. +- **Firmware is a release build.** Build with `cargo build --release --locked` and no `--features`. Before any firmware is attached, check for both crates that `cargo build --release --features dev-mode` fails with the existing `compile_error!` message. Flash images come from `espflash save-image --chip esp32s3 --merge --partition-table partitions.csv`, so a single file flashes at offset 0. +- **Checksums.** `SHA256SUMS` lists every non-image asset in `sha256sum` format. It is checked with `sha256sum --check --strict` before the Release is created, and in a dry run. +- **Ordering.** No image is pushed and no Release is created unless the version, chart, wire-compat and firmware jobs all succeed. Creating the Release is the last job, and it is the only job with `contents: write`. Pre-release versions (`X.Y.Z-…`) produce a GitHub pre-release. +- **Wire gate.** The Server accepts every major from `oldestAcceptedProtocolMajor` to `protocolMajor` in `packages/crypto-spec/crypto-spec.json`; both are generated constants. If the current `protocolMajor` is greater than the one at the previous `v*` tag, the gate fails unless `oldestAcceptedProtocolMajor` is at most the previous major. When there is no previous tag, the gate passes with a notice. +- **Release notes.** Group the commits from the previous `v*` tag (or from the root commit when there is none) to the release commit by conventional-commit type: Breaking changes (`!` or `BREAKING CHANGE:`), Features, Fixes, Performance, then the remaining types, then Other. Exclude merge commits. The notes name each image `ghcr.io/escendit/coldframe/:X.Y.Z` and the flashing assets. They also state that no mobile store binaries are published, with a link to `docs/build/apps.md` at the tag. +- Pin actions to a commit SHA with the release named in a comment, and verify tool downloads by sha256, as the existing workflows do. + +**Never:** +- Don't publish mobile store binaries. Don't add OTA. Don't push or release anything in dry-run mode. +- Don't edit `.github/workflows/ci.yml`: editing it reruns every job. The dry run uses release.yml's own `pull_request` and `workflow_dispatch` triggers. +- Don't touch the heartbeat major (`HeartbeatProtocolVersion`). It is out of scope. +- Don't edit `sprint-status.yaml`. + +## I/O & Edge-Case Matrix + +| Scenario | Input / State | Expected Output / Behavior | Error Handling | +|----------|--------------|---------------------------|----------------| +| Release | tag `v0.2.0`, charts at 0.2.0 | Images pushed. Release `v0.2.0` with 2×.bin, 2×.elf, chart .tgz files, SHA256SUMS and grouped notes | No error expected | +| Pre-release | tag `v0.2.0-rc.1` | As above, marked pre-release | No error expected | +| Chart mismatch | tag `v0.3.0`, charts at 0.2.0 | Nothing pushed or released | `check-version.sh` error names the charts | +| Major bump, still accepted | previous tag major 1 → now 2, oldest 1 | Gate passes | No error expected | +| Major bump, previous dropped | previous tag major 1 → now 2, oldest 2 | Run fails before any push | `::error::` names both majors | +| First release | no earlier `v*` tag | Gate passes; notes cover the full history | Notice only | +| Dry run | PR touching release paths | Every asset built, checksums verified, uploaded as a workflow artifact; no push and no Release | Fails on any missing asset or bad checksum | + + + +## Code Map + +- `.github/workflows/release.yml` -- the current jobs are version, verify (calls `images-verify.yml`) and a 4-component publish matrix. Extend this file. `on: push: tags: v*` stays; add `pull_request` (with paths) and `workflow_dispatch`. +- `.github/workflows/ci.yml:208-314` -- the `firmware` job. Copy its pinned versions (ESPUP 0.17.1, esp toolchain 1.95.0.0, espflash 4.5.0), its apt packages, its `espup install … --export-file ~/export-esp.sh` step and its rust-cache setup. The Node dev-mode refusal check is at `:286-299` and is the pattern to reuse. Read-only. +- `.github/workflows/images-verify.yml` -- the helm v4.2.2 download and its sha256 pin are the pattern for installing helm. Read-only. +- `apps/rs/{hub,node}/` -- the firmware crates, each with its own toolchain (`channel = "esp"`), `.cargo/config.toml` (target `xtensa-esp32s3-none-elf`), `partitions.csv` and `Cargo.lock`. The ELF is at `target/xtensa-esp32s3-none-elf/release/coldframe-{hub,node}`. The dev-mode guard is at `hub/src/main.rs:37-40` and `node/src/main.rs:62-65`. +- `apps/rs/hub/check-image.sh` -- the existing `save-image` call and the credential canary check. Run it in the release firmware job. +- `deploy/images/release-version.sh` and `deploy/images/test.sh` -- tag parsing and the bash `expect` test style to follow. +- `deploy/charts/check-version.sh` and `deploy/charts/test.sh:236-243` -- the version gate. Coldframe charts: server, web, keycloak. All charts are at 0.1.0. +- `packages/crypto-spec/crypto-spec.json:3` -- `protocolMajor: 1`. `scripts/lib/spec.ts:7,123` holds the interface and the `Protocol` constant group; add `oldestAcceptedProtocolMajor` (kind `u8`) there and regenerate the outputs: `packages/rs/crypto/src/spec.rs`, `packages/cs/crypto/Generated/CryptoSpec.g.cs` and `packages/kt/core/generated/.../CryptoSpec.kt`. +- `apps/cs/server/Devices/NodeFrameReader.cs:101` and `apps/cs/server/Devices/DeviceGrain.cs:641` -- the strict `!= CryptoSpec.ProtocolMajor` checks that become a range check. +- `tests/cs/server.tests/Edge/DeviceIngestRequestTests.cs` and `tests/cs/server.integration/Devices/IngestTests.cs` -- the existing tests for major rejection. +- `docs/operations/install.md:4-8` -- says Fleet installs at a release tag. Link from it to the new release doc. + +## Tasks & Acceptance + +**Execution:** +- `packages/crypto-spec/crypto-spec.json`, `scripts/lib/spec.ts` and the generated outputs -- add `oldestAcceptedProtocolMajor: 1` with the doc "The oldest wire major the Server still accepts.", then run `pnpm --filter @coldframe/crypto-spec run generate` -- this gives the gate one machine-readable source of truth. +- `apps/cs/server/Devices/NodeFrameReader.cs` and `DeviceGrain.cs` -- accept `OldestAcceptedProtocolMajor <= v <= ProtocolMajor` -- the Server now does what the gate checks. Extend the existing rejection tests to cover a major below and a major above the range. +- `deploy/release/check-wire-compat.sh` -- usage ` `. Use `jq` and exit 1 with a message naming the majors. Also fail when `oldestAcceptedProtocolMajor > protocolMajor`. +- `deploy/release/release-notes.sh` -- usage ` `, writing Markdown to stdout. It covers the grouping, the image list, the firmware and flashing note, and the mobile and apps-guide note. +- `deploy/release/checksums.sh` -- usage ``. It writes `SHA256SUMS` over every file except itself, then verifies it with `--check --strict`. +- `deploy/release/test.sh` -- tests in the style of `deploy/images/test.sh`. Cover every I/O row of the wire gate and notes grouping, using a throwaway git repo in `mktemp -d` that has feat, fix, `feat!` and merge commits. Also cover checksum creation and detection of a tampered file. +- `.github/workflows/release.yml` -- update the header comment, then add or change these jobs: + - **mode**: `dry_run` is true unless the event is `push`. A dry run takes its version from `deploy/charts/server/Chart.yaml` (`v`). + - **checks**: run `deploy/release/test.sh`, check-version, and the wire gate. Use fetch-depth 0, and find the previous tag with `git describe --tags --abbrev=0 --match 'v*' ^`. + - **firmware matrix**: hub and node. Run the dev-mode refusal, the release build, `check-image.sh` (hub only) and save-image `--merge`, then upload the `.bin` and `.elf`. + - **charts**: `helm package` each chart in `deploy/charts/*`. + - **notes**. + - **verify**: real tag runs only. + - **publish**: on a dry run, build without `--push`. + - **release**: download all assets and run `checksums.sh`. On a dry run, upload them as one workflow artifact. Otherwise run `gh release create "v$VERSION" --verify-tag --notes-file … [--prerelease]` with all assets. + - The `pull_request` paths are `.github/workflows/release.yml`, `deploy/release/**`, `deploy/charts/*/Chart.yaml`, `packages/crypto-spec/crypto-spec.json` and `apps/rs/{hub,node}/{Cargo.toml,Cargo.lock,partitions.csv}`. +- `deploy/charts/set-version.sh` and `deploy/charts/test.sh` -- usage ``. It sets `version` on every chart and `appVersion` on server, web and keycloak, and is tested by round-tripping it through `check-version.sh`. This is how a maintainer sets the charts to `X.Y.Z` before tagging. +- `docs/operations/release.md` -- how to cut a release: set-version, PR, tag, what gets published, the dry run, the wire-gate rule and how to verify `SHA256SUMS`. Link it from `docs/operations/install.md` and `deploy/README.md`. +- `docs/build/apps.md` -- a short guide. It explains that the apps are built from source with your own APNs key and FCM project, that the Server URL and issuer are build-time configuration, and points to the existing app READMEs. Story 10.3 expands it. + +**Acceptance Criteria:** +- Given a `vX.Y.Z` tag with matching charts, when the workflow runs, then the GitHub Release `vX.Y.Z` has these assets: Hub and Node `.bin` and `.elf`, every chart `.tgz` (Coldframe charts with `appVersion` X.Y.Z), and `SHA256SUMS`. Its notes are grouped by conventional-commit type and link to `docs/build/apps.md`, and the four images are pushed at `X.Y.Z`. +- Given a pull request that changes `release.yml`, when CI runs, then the `Release` dry run produces every asset, `sha256sum --check --strict` passes, and nothing is pushed or released. + +## Spec Change Log + +## Review Triage Log + +### 2026-10-19 — Review pass +- verdicts: 38 findings — high 0, medium 15, low 20, false 3, maybe-false 0 +- findings: + - `[medium]` `[patch]` (blind) Range check declares older majors accepted but Frames.Aad and downlink sealing use the current major — added tripwire test AFrameAtTheOldestAcceptedMajorIsDecodedAndItsDownlinkOpens in NodeFrameReaderTests and documented the requirement in release.md and packages/proto/README.md. + - `[medium]` `[patch]` (blind) Wire gate passes a protocolMajor decrease — check-wire-compat.sh now fails on a decrease; test row added. + - `[medium]` `[patch]` (blind) Previous-tag lookup matches pre-release and non-version v* tags — `--match 'v[0-9]*'`, plus `--exclude 'v*-*'` for a final release. + - `[medium]` `[patch]` (blind) Images pushed before the asset check and checksums, and the runbook claims nothing is pushed — new `assets` job (exact-asset check + SHA256SUMS) that publish needs; the runbook covers a failed final Release job (re-run it, do not re-tag). + - `[low]` `[patch]` (blind) firmware/charts do not wait for checks, contradicting the runbook — they now need [mode, checks]. + - `[low]` `[patch]` (blind) check-image.sh rebuilds after the --locked release build — reordered so the release build runs last. + - `[low]` `[patch]` (blind) Dry-run paths miss scripts that can break a tag run — added check-version.sh, set-version.sh, release-version.sh, check-image.sh and crypto-spec/scripts/**. + - `[low]` `[reject]` (blind) Firmware crates and app versions stay 0.0.0 — Release asset names carry X.Y.Z; changing crate versions needs Cargo.lock edits and new checks, and users rarely read the app descriptor; listed as a residual risk. + - `[low]` `[patch]` (blind) ELF flashing needs partitions.csv, which is not shipped — notes link both partitions.csv at the tag; test asserts it. + - `[low]` `[patch]` (blind) Header claims every tool download is sha256-checked — corrected: only helm is; espup/espflash are pinned crates installed with --locked. + - `[low]` `[patch]` (blind) Tag runs restore rust-cache into shipped builds — the cache step runs only on a dry run. + - `[low]` `[defer]` (blind) Tag not checked against main — pre-existing since Story 2.1; deferred. + - `[low]` `[reject]` (blind) No image digests or provenance on the Release — not required by the story; needs matrix outputs passed into the notes, which is more than a direct fix. + - `[low]` `[reject]` (blind) A backport release takes the Latest badge — no backport branches exist; a conditional --latest needs a version comparison. + - `[low]` `[patch]` (blind) Below-range test repeats the 0 case and would not compile if oldest were 0 — the gate now requires both majors >= 1. + - `[false]` `[reject]` (blind) Coldframe chart list duplicated in set-version/check-version — the round trip in deploy/charts/test.sh fails if check-version gains a chart set-version lacks; the other direction checks nothing extra, so no appVersion drift goes unseen. + - `[low]` `[patch]` (blind) Runbook says Verify runs on every PR — reworded: it runs on a tag; a dry run skips it. + - `[low]` `[reject]` (blind) Dry-run version can equal an existing tag — cosmetic; the artifact is named `-dry-run` and nothing is published. + - `[medium]` `[patch]` (edge) Older-major frames fail AEAD — same root cause and fix as the first row (tripwire test + docs). + - `[medium]` `[patch]` (edge) Major decrease passes — same fix as the decrease row. + - `[medium]` `[patch]` (edge) Pre-release previous tag shrinks the final notes — same fix as the tag-match row. + - `[low]` `[patch]` (edge) Non-release v* tag picked as previous — same fix as the tag-match row. + - `[medium]` `[patch]` (edge) Notes/publish/release failure after images pushed — same fix as the assets-job row. + - `[medium]` `[patch]` (edge) Runbook claims nothing pushed on any failure — same fix as the assets-job row. + - `[low]` `[patch]` (edge) Shipped Hub ELF may come from the check-image build — same fix as the reorder row. + - `[low]` `[patch]` (edge) Docs say gates stop everything — same fix as the needs: checks row. + - `[low]` `[reject]` (edge) Below-range test duplicates ProtocolVersion = 0 — harmless redundancy; the gate's >= 1 rule keeps it compiling. + - `[low]` `[patch]` (edge) oldestAcceptedProtocolMajor 0 breaks the test build — the gate rejects majors below 1. + - `[low]` `[reject]` (edge) set-version.sh can leave charts half-updated — needs a Chart.yaml without appVersion, which none has; the charts are in git and easy to restore. + - `[medium]` `[patch]` (verification-gap) Acceptance of an older in-range major is never tested — tripwire test added, as in the first row. + - `[medium]` `[defer]` (verification-gap) Tag-only branches of release.yml never run in the dry run — they can only run on a real tag; deferred and owed as operator_actions. + - `[medium]` `[patch]` (verification-gap) Old-major frames fail authentication (AAD) — grouped with the first row. + - `[medium]` `[patch]` (verification-gap) Old-major Nodes drop downlinks — grouped with the first row; the tripwire test also opens a downlink under the old AAD. + - `[false]` `[reject]` (intent) The workflow does not itself write appVersion — the charts at the tag must carry X.Y.Z because Fleet installs from git (Story 2.2 contract), check-version.sh enforces it, and the packaged .tgz files carry appVersion X.Y.Z; set-version.sh is the setter. + - `[medium]` `[patch]` (intent) Wire gate trusts a declared constant — grouped with the first row; the tripwire test ties the constant to the decode path. + - `[false]` `[reject]` (intent) "Dry-run tag" reading as a pushed throwaway tag — a pushed v* tag would publish, which contradicts a dry run; the PR/dispatch dry run runs the same jobs at the chart version and publishes nothing. + - `[medium]` `[patch]` (intent) Proving AC1 needs a real tag push, which only a maintainer can do — spec finalized at awaiting-operator with operator_actions. + - `[low]` `[reject]` (intent) The heartbeat major is not gated — no heartbeat bump is planned; gating it means moving HeartbeatProtocolVersion into crypto-spec (more than a direct fix). + +## Design Notes + +A range (`oldest..current`) needs no new generator kind, unlike a list of majors. Bumping `protocolMajor` while keeping `oldestAcceptedProtocolMajor` at the old value is the explicit promise that the Server still decodes the previous major. The gate holds the author to that promise across tags. + +## Verification + +**Commands:** +- `deploy/release/test.sh && deploy/charts/test.sh && deploy/images/test.sh` -- expected: all pass +- `pnpm --filter @coldframe/crypto-spec run check` -- expected: the outputs are fresh +- `dotnet test tests/cs/server.tests` -- expected: green, including the new range cases +- `cargo test --workspace --locked` -- expected: green (the generated Rust constant compiles) +- `actionlint .github/workflows/release.yml`, if available -- expected: no findings + +**Manual checks (if no CLI):** +- On the PR, the `Release` workflow's dry run is green and its artifact contains the 4 firmware files, the chart .tgz files and `SHA256SUMS`. + +## Auto Run Result + +Status: awaiting-operator + +**Summary.** A `vX.Y.Z` tag now produces one release. The four images are pushed (unchanged). A GitHub Release gets release-build Hub and Node firmware (a merged `.bin` that flashes at offset 0, plus the `.elf`), every chart as `-X.Y.Z.tgz`, `SHA256SUMS`, and notes grouped by conventional-commit type that point to `docs/build/apps.md` (no store binaries). Gates run before anything is pushed: the chart versions, the wire gate (`oldestAcceptedProtocolMajor..protocolMajor`, which the Server now enforces as a range), a check that dev-mode is refused, and the asset and checksum checks. Pull requests on the release paths, and manual runs, run the whole workflow as a dry run that pushes and releases nothing. + +**Files changed** +- `.github/workflows/release.yml`: these jobs: mode, checks, firmware matrix, charts, notes, assets, verify (tag only), publish (no push on a dry run), release (`gh release create` or the dry-run artifact). +- `deploy/release/{check-wire-compat.sh,release-notes.sh,checksums.sh,test.sh}`: the wire gate, the notes, SHA256SUMS, and 62 tests. +- `deploy/charts/set-version.sh`, `deploy/charts/test.sh`: set the charts to a version before tagging, tested by a round trip through check-version.sh. +- `packages/crypto-spec/crypto-spec.json`, `scripts/lib/spec.ts` and the generated Rust/C#/Kotlin: the `oldestAcceptedProtocolMajor` constant. +- `apps/cs/server/Devices/{NodeFrameReader,DeviceGrain}.cs`: accept a range of majors. +- `tests/cs/server.tests/Devices/NodeFrameReaderTests.cs`, `tests/cs/server.integration/Devices/IngestTests.cs`: cases above and below the range, and the tripwire test for the oldest accepted major. +- `docs/operations/release.md` (new runbook), `docs/build/apps.md` (new, short apps guide), plus links in `docs/operations/install.md`, `deploy/README.md`, `deploy/images/README.md` and `packages/proto/README.md`. +- `.gitignore`: un-ignore `deploy/release/` and `docs/build/`. + +**Review findings.** 38 findings: medium 15, low 20, false 3. Patched: 12 root causes (6 medium and 6 low entries; rows grouped as in the triage log). Deferred: 2 (tag-on-main enforcement, which predates this story; the tag-only workflow branches, owed as operator actions). Rejected, with the reasons recorded in the triage log: the firmware/app embedded version (low), image digests/provenance (low), the Latest badge on backports (low), the dry-run version colliding with a tag (low), the duplicate below-range test (low), half-updated charts in set-version (low), the heartbeat major (low), and three false findings (the chart-list duplication, the workflow not writing appVersion itself, the literal reading of "dry-run tag"). + +**Follow-up review: recommended (true).** Six medium entries were patched. The risk not yet verified: the restructured `release.yml` job graph (the new `assets` job, the `needs`/`if` conditions, the bundle hand-off to `release`) has passed actionlint and a YAML parse, but has not run on GitHub yet. The PR's dry run is its first run. + +**Verification.** +- `deploy/release/test.sh`: 62 passed. +- `deploy/images/test.sh`: 33 passed. +- `deploy/charts/test.sh`: the set-version and check-version cases pass. The Fleet/kubeconform cases fail locally only because those tools are missing. +- `pnpm --filter @coldframe/crypto-spec run check`: fresh. +- `dotnet test --project tests/cs/server.tests`: 1139 passed. +- `tests/cs/server.integration`: builds. It was not run, because Keycloak would not start under podman. +- `cargo test --workspace --locked`: 350 passed. +- actionlint 1.7.12 with shellcheck, run by the implementer: clean. `release.yml` parses as YAML. +- Firmware steps run locally by the implementer: the dev-mode refusal for both crates, the release builds, `check-image.sh`, and `save-image --merge` (the image starts with 0xE9). + +**Residual risks.** +- The real-tag path (ghcr push, `gh release create`, pre-release flag) runs only on a tag; see operator_actions. +- The firmware crates still report version 0.0.0 inside the binary. +- A dry run rebuilds the images and both firmware crates, so pull requests on the release paths take longer. +- The extended ingest integration test is unverified locally and needs CI. diff --git a/_bmad-output/implementation-artifacts/sprint-status.yaml b/_bmad-output/implementation-artifacts/sprint-status.yaml index d3010ecb..4f0ff21b 100644 --- a/_bmad-output/implementation-artifacts/sprint-status.yaml +++ b/_bmad-output/implementation-artifacts/sprint-status.yaml @@ -112,7 +112,7 @@ development_status: epic-9-retrospective: optional epic-10: backlog - 10-1-releases-with-one-version-and-firmware-binaries: backlog + 10-1-releases-with-one-version-and-firmware-binaries: awaiting-operator 10-2-build-a-node-and-a-hub-from-the-docs: backlog 10-3-deploy-coldframe-at-home-and-build-my-own-apps: backlog 10-4-firmware-learning-reference: backlog diff --git a/apps/cs/server/Devices/DeviceGrain.cs b/apps/cs/server/Devices/DeviceGrain.cs index 5d1eef43..55bc45d6 100644 --- a/apps/cs/server/Devices/DeviceGrain.cs +++ b/apps/cs/server/Devices/DeviceGrain.cs @@ -632,13 +632,16 @@ public async Task Ingest(DeviceIngest request, CancellationT ArgumentNullException.ThrowIfNull(request); ArgumentException.ThrowIfNullOrEmpty(request.RelayHubId); - // 1. Only an enrolled Node, and only the supported protocol version. + // 1. Only an enrolled Node, and only a protocol major the Server accepts + // (CryptoSpec.OldestAcceptedProtocolMajor to CryptoSpec.ProtocolMajor). if (State.SiteId is null || State.Kind != DeviceKind.Node || State.WrappedKey is not { } wrapped) { return new DeviceIngestResult(DeviceIngestStatus.UnknownDevice); } - if (request.ProtocolVersion != CryptoSpec.ProtocolMajor || request.Ciphertext is null) + if (request.ProtocolVersion < CryptoSpec.OldestAcceptedProtocolMajor + || request.ProtocolVersion > CryptoSpec.ProtocolMajor + || request.Ciphertext is null) { return new DeviceIngestResult(DeviceIngestStatus.RejectedAuth); } diff --git a/apps/cs/server/Devices/NodeFrameReader.cs b/apps/cs/server/Devices/NodeFrameReader.cs index cdfad3f4..2e444fdf 100644 --- a/apps/cs/server/Devices/NodeFrameReader.cs +++ b/apps/cs/server/Devices/NodeFrameReader.cs @@ -98,7 +98,9 @@ public static NodeFrameRead Read(ReadOnlySpan plaintext, DeviceId deviceId return new NodeFrameRead(NodeFrameVerdict.Invalid); } - if (frame.ProtocolVersion != CryptoSpec.ProtocolMajor + // Every major from the oldest accepted one to the current one is decoded (Story 10.1). + if (frame.ProtocolVersion < CryptoSpec.OldestAcceptedProtocolMajor + || frame.ProtocolVersion > CryptoSpec.ProtocolMajor || frame.Readings.Count > MaxReadings || (frame.HasBatteryPercent && frame.BatteryPercent > 100)) { diff --git a/aspire/keycloak/realms/coldframe-realm.json b/aspire/keycloak/realms/coldframe-realm.json index 68cc239e..2139b2b9 100644 --- a/aspire/keycloak/realms/coldframe-realm.json +++ b/aspire/keycloak/realms/coldframe-realm.json @@ -4,6 +4,7 @@ "displayName": "Coldframe", "enabled": true, "registrationAllowed": true, + "verifyEmail": true, "organizationsEnabled": true, "sslRequired": "external", "eventsListeners": ["jboss-logging", "temporal"], diff --git a/deploy/README.md b/deploy/README.md index 6edd0ecf..9040ee3c 100644 --- a/deploy/README.md +++ b/deploy/README.md @@ -10,6 +10,7 @@ Everything needed to run Coldframe outside a developer machine. | Epic 2, Story 2.4 | The `ingress` chart: Let's Encrypt certificates through DNS-01 (cert-manager, Cloudflare) and a Traefik Ingress on 443 only; RKE2's Traefik without port 80: see [`rke2/`](rke2/README.md); split DNS runbook [`docs/operations/split-dns.md`](../docs/operations/split-dns.md) | | Epic 2, Story 2.5a | The Fleet bundles: a `fleet.yaml` in every chart folder, the operator bundles and the reference GitRepo in [`fleet/`](fleet), and the `rke2-traefik` bundle in [`rke2/`](rke2/README.md); the GitOps install runbook [`docs/operations/install.md`](../docs/operations/install.md). | | Epic 2, Story 2.5b | The Fleet smoke [`fleet/smoke.sh`](fleet/smoke.sh): Fleet installs the bundles on a disposable cluster in dependency order, then an upgrade and three restarts lose no committed data | +| Epic 10, Story 10.1 | One release per tag `vX.Y.Z`: images, Hub and Node firmware, chart packages and `SHA256SUMS` on a GitHub Release, the wire gate and the dry run: see [`release/`](release) and the runbook [`docs/operations/release.md`](../docs/operations/release.md). [`charts/set-version.sh`](charts/set-version.sh) sets the charts to the version before tagging | | Epic 10 | The Compose file, as a reference example only | No secret value is ever committed here. Charts reference Secrets by fixed names and keys, listed in diff --git a/deploy/charts/set-version.sh b/deploy/charts/set-version.sh new file mode 100755 index 00000000..9d57f2ae --- /dev/null +++ b/deploy/charts/set-version.sh @@ -0,0 +1,52 @@ +#!/usr/bin/env bash +# Sets the charts to a release version before the release is tagged (Story 10.1). +# +# deploy/charts/set-version.sh [charts-dir] +# +# Sets `version` of every chart, and `appVersion` of the Coldframe charts (server, web, keycloak), +# to (the release tag without its "v", for example 0.2.0 or 0.2.0-rc.1). temporal and +# nats keep the upstream appVersion. deploy/charts/check-version.sh passes afterwards. +# charts-dir defaults to the directory of this script. +set -euo pipefail + +if [[ $# -lt 1 || $# -gt 2 ]]; then + echo "usage: $0 [charts-dir]" >&2 + exit 2 +fi + +version=$1 +here=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +charts=${2:-${here}} +coldframe_charts=(server web keycloak) + +if ! "${here}/../images/release-version.sh" "v${version}" >/dev/null; then + echo "'${version}' is not a release version: expected X.Y.Z or X.Y.Z-." >&2 + exit 1 +fi + +shopt -s nullglob +found=0 +for chart_file in "${charts}"/*/Chart.yaml; do + found=$((found + 1)) + chart=$(basename "$(dirname "${chart_file}")") + if ! grep -qE '^version:' "${chart_file}"; then + echo "${chart_file}: no top-level version" >&2 + exit 1 + fi + sed -i -E "s/^version:[[:space:]]*.*$/version: ${version}/" "${chart_file}" + for coldframe in "${coldframe_charts[@]}"; do + if [[ ${chart} == "${coldframe}" ]]; then + if ! grep -qE '^appVersion:' "${chart_file}"; then + echo "${chart_file}: no top-level appVersion" >&2 + exit 1 + fi + sed -i -E "s/^appVersion:[[:space:]]*.*$/appVersion: \"${version}\"/" "${chart_file}" + fi + done +done + +if [[ ${found} -eq 0 ]]; then + echo "No Chart.yaml under ${charts}" >&2 + exit 2 +fi +echo "Set ${found} charts to ${version}." diff --git a/deploy/charts/test.sh b/deploy/charts/test.sh index 1559994e..2eb19a39 100755 --- a/deploy/charts/test.sh +++ b/deploy/charts/test.sh @@ -12,8 +12,8 @@ # not list) and check-ingress.py (TLS on every Ingress and IngressRoute, DNS-01 on every ACME # issuer, nothing on port 80). The pinned rke2-traefik chart is rendered too: with its default # values it must fail check-ingress.py (port 80), with deploy/rke2/rke2-traefik-config.yaml pass. -# The checks are proven able to fail on fixtures in testdata/, and check-version.sh is tested with a -# matching and a mismatching version. +# The checks are proven able to fail on fixtures in testdata/, check-version.sh is tested with a +# matching and a mismatching version, and set-version.sh by a round trip through check-version.sh. # # Fleet bundles: `fleet apply --output -` renders the paths of deploy/fleet/gitrepo.yaml as gitjob # would, and deploy/fleet/check-fleet.py checks them (labels, the dependsOn order, the pins of @@ -242,6 +242,35 @@ expect_failure "mismatching version names the chart" "server/Chart.yaml: version expect_failure "mismatching appVersion names the chart" "keycloak/Chart.yaml: appVersion is '${current}', expected '${other}'" \ "${here}/check-version.sh" "${other}" +echo "# set-version.sh" +# On a copy of every Chart.yaml: set-version.sh then check-version.sh round-trips, and the upstream +# appVersion of temporal and nats is kept. +version_work=$(mktemp -d) +for chart_file in "${here}"/*/Chart.yaml; do + mkdir -p "${version_work}/$(basename "$(dirname "${chart_file}")")" + cp "${chart_file}" "${version_work}/$(basename "$(dirname "${chart_file}")")/Chart.yaml" +done +check "set-version.sh sets a release version" "${here}/set-version.sh" 9.8.7 "${version_work}" +check "check-version.sh passes after set-version.sh" "${here}/check-version.sh" 9.8.7 "${version_work}" +check "set-version.sh sets a pre-release version" "${here}/set-version.sh" 9.8.8-rc.1 "${version_work}" +check "check-version.sh passes on the pre-release" "${here}/check-version.sh" 9.8.8-rc.1 "${version_work}" +expect_failure "the previous version no longer passes" "server/Chart.yaml: version is '9.8.8-rc.1', expected '9.8.7'" \ + "${here}/check-version.sh" 9.8.7 "${version_work}" +for upstream in temporal nats; do + want=$(sed -n -E 's/^appVersion:[[:space:]]*"?([^"[:space:]]*)"?[[:space:]]*$/\1/p' "${here}/${upstream}/Chart.yaml") + got=$(sed -n -E 's/^appVersion:[[:space:]]*"?([^"[:space:]]*)"?[[:space:]]*$/\1/p' "${version_work}/${upstream}/Chart.yaml") + if [[ ${got} == "${want}" ]]; then + pass "set-version.sh keeps the upstream appVersion of ${upstream}" + else + failed "set-version.sh keeps the upstream appVersion of ${upstream}" "appVersion '${got}', expected '${want}'" + fi +done +expect_failure "set-version.sh refuses a version that is not SemVer" "'9.8' is not a release version" \ + "${here}/set-version.sh" 9.8 "${version_work}" +expect_failure "set-version.sh refuses a version with a v" "'v9.8.7' is not a release version" \ + "${here}/set-version.sh" v9.8.7 "${version_work}" +rm -rf "${version_work}" + for chart in "${charts[@]}"; do echo "# ${chart}" dir=${here}/${chart} diff --git a/deploy/images/README.md b/deploy/images/README.md index 5405c333..d1d0f4cf 100644 --- a/deploy/images/README.md +++ b/deploy/images/README.md @@ -144,6 +144,9 @@ deploy/images/test.sh - [`release.yml`](../../.github/workflows/release.yml) runs on a `v*` tag: it validates the tag, runs the same verification, then pushes each image with the version tag and the OCI labels. The tag must equal the version of every Helm chart (`deploy/charts/check-version.sh`), or nothing is pushed. Only the - push job has `packages: write`, and it authenticates with `GITHUB_TOKEN` alone. + push job has `packages: write`, and it authenticates with `GITHUB_TOKEN` alone. The same run + attaches the firmware, the chart packages and `SHA256SUMS` to a GitHub Release, and on a pull + request that touches the release paths it builds the images without pushing them (a dry run): + [`docs/operations/release.md`](../../docs/operations/release.md). - Any `v*` tag publishes, whatever commit it points to: push release tags only from `main`, and protect `v*` tags with a repository tag ruleset so that only maintainers can create them. diff --git a/deploy/release/check-wire-compat.sh b/deploy/release/check-wire-compat.sh new file mode 100755 index 00000000..08b1bba4 --- /dev/null +++ b/deploy/release/check-wire-compat.sh @@ -0,0 +1,64 @@ +#!/usr/bin/env bash +# The wire-compatibility gate of a release (Story 10.1). +# +# deploy/release/check-wire-compat.sh +# +# The Server accepts every wire major from `oldestAcceptedProtocolMajor` to `protocolMajor` of +# packages/crypto-spec/crypto-spec.json. A release whose `protocolMajor` is greater than the one of +# the previous release (, the file at the previous v* tag) fails unless +# `oldestAcceptedProtocolMajor` is still at most the previous major: the Server must keep decoding +# what Devices in the field speak. "-" stands for no previous release; the gate then passes with a +# notice. `oldestAcceptedProtocolMajor` greater than `protocolMajor`, a `protocolMajor` lower than +# at the previous release, and a major below 1 always fail. +# +# Exit 0 on pass, 1 on a gate failure (message on stderr), 2 on bad usage or an unreadable spec. +set -euo pipefail + +if [[ $# -ne 2 ]]; then + echo "usage: $0 " >&2 + exit 2 +fi + +previous=$1 +current=$2 + +# major : the integer field, or exit 2 naming the file. Wire majors start at 1. +major() { + local value + if ! value=$(jq -er --arg field "$2" '.[$field] | select(type == "number" and . == floor)' "$1" 2>/dev/null); then + echo "$1: '$2' is missing or not an integer" >&2 + exit 2 + fi + if (( value < 1 )); then + echo "$1: '$2' is ${value}; wire majors start at 1" >&2 + exit 1 + fi + echo "${value}" +} + +current_major=$(major "${current}" protocolMajor) +oldest=$(major "${current}" oldestAcceptedProtocolMajor) + +if (( oldest > current_major )); then + echo "oldestAcceptedProtocolMajor ${oldest} is greater than protocolMajor ${current_major} in ${current}" >&2 + exit 1 +fi + +if [[ ${previous} == "-" ]]; then + echo "No previous release: the wire gate passes (protocolMajor ${current_major}, oldest accepted ${oldest})." + exit 0 +fi + +previous_major=$(major "${previous}" protocolMajor) + +if (( current_major < previous_major )); then + echo "protocolMajor goes down from ${previous_major} (previous release) to ${current_major}; a wire major never decreases." >&2 + exit 1 +fi + +if (( current_major > previous_major && oldest > previous_major )); then + echo "protocolMajor goes from ${previous_major} (previous release) to ${current_major}, but the Server no longer accepts major ${previous_major}: oldestAcceptedProtocolMajor is ${oldest}. Keep oldestAcceptedProtocolMajor at ${previous_major} or lower until no Device speaks major ${previous_major}." >&2 + exit 1 +fi + +echo "Wire gate passes: protocolMajor ${previous_major} -> ${current_major}, the Server accepts ${oldest} to ${current_major}." diff --git a/deploy/release/checksums.sh b/deploy/release/checksums.sh new file mode 100755 index 00000000..d05ec2d8 --- /dev/null +++ b/deploy/release/checksums.sh @@ -0,0 +1,60 @@ +#!/usr/bin/env bash +# The SHA256SUMS of the release assets (Story 10.1). +# +# deploy/release/checksums.sh writes /SHA256SUMS, then verifies it +# deploy/release/checksums.sh --check only verifies an existing /SHA256SUMS +# +# SHA256SUMS lists every regular file in except itself, in `sha256sum` format and sorted by +# name, so `sha256sum --check --strict SHA256SUMS` verifies a download. Verifying also fails when +# a file of is not listed. must hold only files. +set -euo pipefail + +check_only=false +if [[ $# -eq 2 && $1 == "--check" ]]; then + check_only=true + shift +fi +if [[ $# -ne 1 ]]; then + echo "usage: $0 [--check] " >&2 + exit 2 +fi + +dir=$1 +if [[ ! -d ${dir} ]]; then + echo "${dir}: no such directory" >&2 + exit 2 +fi +cd "${dir}" + +shopt -s nullglob dotglob +assets=() +for entry in *; do + [[ ${entry} == "SHA256SUMS" ]] && continue + if [[ ! -f ${entry} || -L ${entry} ]]; then + echo "${dir}/${entry}: not a regular file; the asset directory holds only files" >&2 + exit 1 + fi + assets+=("${entry}") +done +if [[ ${#assets[@]} -eq 0 ]]; then + echo "${dir}: no assets to checksum" >&2 + exit 1 +fi + +if [[ ${check_only} == false ]]; then + LC_ALL=C sha256sum -- "${assets[@]}" | LC_ALL=C sort -k 2 > SHA256SUMS +elif [[ ! -f SHA256SUMS ]]; then + echo "${dir}/SHA256SUMS: missing" >&2 + exit 1 +fi + +# Every asset is listed: a file added after SHA256SUMS was written would not be checked. +listed=$(sed -E 's/^[0-9a-f]{64} //' SHA256SUMS | LC_ALL=C sort) +present=$(printf '%s\n' "${assets[@]}" | LC_ALL=C sort) +if [[ ${listed} != "${present}" ]]; then + echo "${dir}/SHA256SUMS does not list exactly the assets:" >&2 + diff <(echo "${listed}") <(echo "${present}") >&2 || true + exit 1 +fi + +sha256sum --check --strict SHA256SUMS diff --git a/deploy/release/release-notes.sh b/deploy/release/release-notes.sh new file mode 100755 index 00000000..4fe8f359 --- /dev/null +++ b/deploy/release/release-notes.sh @@ -0,0 +1,156 @@ +#!/usr/bin/env bash +# Writes the Markdown notes of a GitHub Release to stdout (Story 10.1). +# +# deploy/release/release-notes.sh +# +# The commits from (the previous v* tag, excluded) to (the release commit) are +# grouped by Conventional Commit type: Breaking changes (`type!:` or a `BREAKING CHANGE:` footer), +# Features, Fixes, Performance, then the other types, then Other (subjects that are not +# Conventional Commits). "-" for means no previous release: the notes cover the whole +# history up to . Merge commits are left out. The notes then name the container images, +# the firmware and how to flash it, the charts, and say that no mobile store binaries are published. +# +# RELEASE_REPOSITORY (default escendit/coldframe) names the GitHub repository and the image path. +# Run inside the git repository. +set -euo pipefail + +if [[ $# -ne 3 ]]; then + echo "usage: $0 " >&2 + exit 2 +fi + +version=$1 +from=$2 +to=$3 +repository=${RELEASE_REPOSITORY:-escendit/coldframe} +repository=${repository,,} +tag="v${version}" + +if [[ ${from} == "-" ]]; then + range=${to} +else + range="${from}..${to}" +fi + +# The sections in order: the Conventional Commit types with a heading of their own, then any other +# type (by name), then Other. +known_types=(feat fix perf refactor docs test build ci style chore revert) +declare -A headings=( + [feat]="Features" + [fix]="Fixes" + [perf]="Performance" + [refactor]="Refactoring" + [docs]="Documentation" + [test]="Tests" + [build]="Build" + [ci]="CI" + [style]="Style" + [chore]="Chores" + [revert]="Reverts" +) +declare -A entries=() +breaking="" +other="" +extra_types=() + +pattern='^([A-Za-z]+)(\(([^)]*)\))?(!)?: (.+)$' + +# One record per commit, oldest first: hash, subject, body, separated by \x1f and ended by \x1e. +log=$(git log --no-merges --reverse --format='%h%x1f%s%x1f%b%x1e' "${range}" --) + +while IFS= read -r -d $'\x1e' record; do + record=${record#$'\n'} + [[ -z ${record} ]] && continue + hash=${record%%$'\x1f'*} + rest=${record#*$'\x1f'} + subject=${rest%%$'\x1f'*} + body=${rest#*$'\x1f'} + + if [[ ${subject} =~ ${pattern} ]]; then + type=${BASH_REMATCH[1],,} + scope=${BASH_REMATCH[3]} + bang=${BASH_REMATCH[4]} + description=${BASH_REMATCH[5]} + if [[ -n ${scope} ]]; then + line="- **${scope}:** ${description} (${hash})" + else + line="- ${description} (${hash})" + fi + if [[ -n ${bang} ]] || grep -qE '^BREAKING[ -]CHANGE: ' <<<"${body}"; then + breaking+="${line}"$'\n' + else + if [[ -z ${entries[${type}]+set} && -z ${headings[${type}]+set} ]]; then + extra_types+=("${type}") + fi + entries[${type}]+="${line}"$'\n' + fi + else + other+="- ${subject} (${hash})"$'\n' + fi +done <<<"${log}" + +# section +section() { + [[ -z $2 ]] && return 0 + printf '### %s\n\n%s\n' "$1" "$2" +} + +echo "## Coldframe ${version}" +echo +echo "One version for the whole monorepo: the images, the firmware and the charts below all belong to \`${tag}\`." +echo + +echo "## Changes" +echo +section "Breaking changes" "${breaking}" +for type in "${known_types[@]}"; do + section "${headings[${type}]}" "${entries[${type}]:-}" +done +if [[ ${#extra_types[@]} -gt 0 ]]; then + mapfile -t sorted < <(printf '%s\n' "${extra_types[@]}" | sort -u) + for type in "${sorted[@]}"; do + section "${type}" "${entries[${type}]:-}" + done +fi +section "Other" "${other}" +if [[ -z ${breaking}${other} && ${#entries[@]} -eq 0 ]]; then + echo "No commits since the previous release." + echo +fi + +cat <-${version}.tgz\`. Fleet installs the +same charts from the repository at \`${tag}\` +([install guide](https://github.com/${repository}/blob/${tag}/docs/operations/install.md)). + +## Mobile apps + +No mobile store binaries are published. Build the iOS and Android apps from source with your own +APNs key and FCM project: +[docs/build/apps.md](https://github.com/${repository}/blob/${tag}/docs/build/apps.md). +NOTES diff --git a/deploy/release/test.sh b/deploy/release/test.sh new file mode 100755 index 00000000..c69fc08c --- /dev/null +++ b/deploy/release/test.sh @@ -0,0 +1,238 @@ +#!/usr/bin/env bash +# Plain-bash tests for the release scripts (Story 10.1): the wire gate, the release notes on a +# throwaway git repository, and the checksums. Needs git, jq and sha256sum. Run from anywhere: +# +# deploy/release/test.sh +set -uo pipefail + +here=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +repo=$(cd "${here}/../.." && pwd) +work=$(mktemp -d) +trap 'rm -rf "${work}"' EXIT +failures=0 +passes=0 + +pass() { + passes=$((passes + 1)) + echo "ok - $1" +} + +failed() { + failures=$((failures + 1)) + echo "FAIL - $1" >&2 + shift + for line in "$@"; do + echo " ${line}" >&2 + done +} + +# expect -- +# The expected stdout and stderr are substrings, "" for none at all, or - for anything. +expect() { + local name=$1 want_status=$2 want_out=$3 want_err=$4 + shift 5 + local out err status + err=$(mktemp) + out=$("$@" 2>"${err}") + status=$? + local err_text + err_text=$(cat "${err}") + rm -f "${err}" + + if [[ ${status} -ne ${want_status} ]]; then + failed "${name}" "exit ${status}, expected ${want_status}" "stdout: ${out}" "stderr: ${err_text}" + elif [[ -z ${want_out} && -n ${out} ]]; then + failed "${name}" "stdout '${out}', expected none" + elif [[ -n ${want_out} && ${want_out} != "-" && ${out} != *"${want_out}"* ]]; then + failed "${name}" "stdout '${out}' does not contain '${want_out}'" + elif [[ -z ${want_err} && -n ${err_text} ]]; then + failed "${name}" "stderr '${err_text}', expected none" + elif [[ -n ${want_err} && ${want_err} != "-" && ${err_text} != *"${want_err}"* ]]; then + failed "${name}" "stderr '${err_text}' does not contain '${want_err}'" + else + pass "${name}" + fi +} + +wire=${here}/check-wire-compat.sh +notes=${here}/release-notes.sh +checksums=${here}/checksums.sh + +# spec [oldestAcceptedProtocolMajor]: a spec file holding only the majors. +spec() { + local file=${work}/$1.json + if [[ $# -eq 3 ]]; then + printf '{"protocolMajor": %s, "oldestAcceptedProtocolMajor": %s}\n' "$2" "$3" > "${file}" + else + printf '{"protocolMajor": %s}\n' "$2" > "${file}" + fi + echo "${file}" +} + +echo "# check-wire-compat.sh" +expect "same major passes" 0 "Wire gate passes" "" \ + -- "${wire}" "$(spec prev-1 1 1)" "$(spec cur-1-1 1 1)" +expect "major bump, previous still accepted, passes" 0 "1 -> 2" "" \ + -- "${wire}" "$(spec prev-1 1 1)" "$(spec cur-2-1 2 1)" +expect "major bump from a spec without oldestAcceptedProtocolMajor passes" 0 "1 -> 2" "" \ + -- "${wire}" "$(spec prev-legacy 1)" "$(spec cur-2-1 2 1)" +expect "major bump, previous dropped, fails naming both majors" 1 "" \ + "protocolMajor goes from 1 (previous release) to 2, but the Server no longer accepts major 1" \ + -- "${wire}" "$(spec prev-1 1 1)" "$(spec cur-2-2 2 2)" +expect "first release passes with a notice" 0 "No previous release" "" \ + -- "${wire}" - "$(spec cur-1-1 1 1)" +expect "oldest above current fails" 1 "" "oldestAcceptedProtocolMajor 3 is greater than protocolMajor 2" \ + -- "${wire}" - "$(spec cur-2-3 2 3)" +expect "missing oldestAcceptedProtocolMajor fails" 2 "" "'oldestAcceptedProtocolMajor' is missing" \ + -- "${wire}" - "$(spec cur-legacy 1)" +expect "major decrease fails" 1 "" "protocolMajor goes down from 2 (previous release) to 1" \ + -- "${wire}" "$(spec prev-2 2 2)" "$(spec cur-1-1 1 1)" +expect "oldestAcceptedProtocolMajor 0 fails" 1 "" "'oldestAcceptedProtocolMajor' is 0; wire majors start at 1" \ + -- "${wire}" - "$(spec cur-1-0 1 0)" +expect "protocolMajor 0 fails" 1 "" "'protocolMajor' is 0; wire majors start at 1" \ + -- "${wire}" - "$(spec cur-0-0 0 0)" +expect "bad usage" 2 "" "usage:" -- "${wire}" - +expect "the repository spec passes as a first release" 0 "No previous release" "" \ + -- "${wire}" - "${repo}/packages/crypto-spec/crypto-spec.json" +expect "the repository spec passes against itself" 0 "Wire gate passes" "" \ + -- "${wire}" "${repo}/packages/crypto-spec/crypto-spec.json" "${repo}/packages/crypto-spec/crypto-spec.json" + +echo "# release-notes.sh" +history=${work}/history +# The throwaway repository ignores the user's git configuration (signing, hooks, default branch). +export GIT_CONFIG_GLOBAL=/dev/null GIT_CONFIG_NOSYSTEM=1 +git init --quiet --initial-branch=main "${history}" +commit() { # [body]: an empty commit + git -C "${history}" -c user.name=test -c user.email=test@example.org commit --quiet --allow-empty \ + --message "$1" ${2:+--message "$2"} +} +commit "chore(repo): initial commit" +commit "feat(server): first feature before the tag" +git -C "${history}" tag v0.1.0 +commit "feat(hub): add the release workflow" +commit "fix(node): stop dropping frames" +commit "feat(api)!: rename the ingest route" +commit "refactor(server): change the ingest store" "BREAKING CHANGE: the store schema changes" +commit "perf(web): cache the sites list" +commit "docs: explain the release" +commit "wip: something unusual" +commit "Update the README" +git -C "${history}" switch --quiet --create topic +commit "fix(web): a fix on a branch" +git -C "${history}" switch --quiet main +git -C "${history}" -c user.name=test -c user.email=test@example.org merge --quiet --no-ff \ + --message "Merge pull request #1 from topic" topic + +in_history() { (cd "${history}" && "$@"); } +release_notes=$(in_history "${notes}" 0.2.0 v0.1.0 HEAD) +notes_status=$? + +# has ; lacks +has() { + if [[ ${release_notes} == *"$2"* ]]; then pass "$1"; else failed "$1" "notes do not contain '$2'" "${release_notes}"; fi +} +lacks() { + if [[ ${release_notes} != *"$2"* ]]; then pass "$1"; else failed "$1" "notes contain '$2'" "${release_notes}"; fi +} +# in_section : is in the section under "### " +in_section() { + local section=${release_notes#*"### $2"$'\n'} + section=${section%%$'\n'"#"*} + if [[ ${release_notes} == *"### $2"$'\n'* && ${section} == *"$3"* ]]; then + pass "$1" + else + failed "$1" "'$3' is not under '### $2'" "${release_notes}" + fi +} +# before : appears before +before() { + local head=${release_notes%%"$3"*} + if [[ ${release_notes} == *"$2"* && ${release_notes} == *"$3"* && ${head} == *"$2"* ]]; then + pass "$1" + else + failed "$1" "'$2' does not come before '$3'" "${release_notes}" + fi +} + +if [[ ${notes_status} -eq 0 ]]; then pass "notes from a tag exit 0"; else failed "notes from a tag exit 0" "exit ${notes_status}"; fi +has "a feature is listed with its scope" "- **hub:** add the release workflow" +in_section "a feature is under Features" "Features" "- **hub:** add the release workflow" +in_section "a fix is under Fixes" "Fixes" "- **node:** stop dropping frames" +in_section "a bang commit is breaking" "Breaking changes" "- **api:** rename the ingest route" +in_section "a BREAKING CHANGE footer is breaking" "Breaking changes" "- **server:** change the ingest store" +in_section "a perf commit is under Performance" "Performance" "- **web:** cache the sites list" +lacks "a breaking commit is not repeated under Refactoring" "### Refactoring" +has "a scopeless commit is listed without a scope" "- explain the release" +in_section "an unknown type has its own section" "wip" "- something unusual" +in_section "a non-conventional subject is under Other" "Other" "- Update the README" +lacks "a merge commit is left out" "Merge pull request" +in_section "a commit merged from a branch is kept" "Fixes" "- **web:** a fix on a branch" +lacks "a commit before the previous tag is left out" "first feature before the tag" +before "Breaking changes come first" "### Breaking changes" "### Features" +before "Features come before Fixes" "### Features" "### Fixes" +before "Fixes come before Performance" "### Fixes" "### Performance" +before "Performance comes before the other types" "### Performance" "### Documentation" +before "the other types come before unknown ones" "### Documentation" "### wip" +before "Other comes last" "### wip" "### Other" +has "the server image is named" "ghcr.io/escendit/coldframe/server:0.2.0" +has "the web image is named" "ghcr.io/escendit/coldframe/web:0.2.0" +has "the migrations image is named" "ghcr.io/escendit/coldframe/migrations:0.2.0" +has "the keycloak image is named" "ghcr.io/escendit/coldframe/keycloak:0.2.0" +has "the Hub flash image is named" "coldframe-hub-0.2.0.bin" +has "the Node ELF is named" "coldframe-node-0.2.0.elf" +has "flashing at offset 0 is explained" "espflash write-bin 0x0 coldframe-node-0.2.0.bin" +has "the Hub partition table is linked at the tag" "https://github.com/escendit/coldframe/blob/v0.2.0/apps/rs/hub/partitions.csv" +has "the Node partition table is linked at the tag" "https://github.com/escendit/coldframe/blob/v0.2.0/apps/rs/node/partitions.csv" +has "the checksums are explained" "sha256sum --check --strict SHA256SUMS" +has "no mobile store binaries are published" "No mobile store binaries are published" +has "the apps guide is linked at the tag" "https://github.com/escendit/coldframe/blob/v0.2.0/docs/build/apps.md" + +release_notes=$(in_history "${notes}" 0.1.0 - v0.1.0) +has "a first release covers the root commit" "- **repo:** initial commit" +has "a first release covers the whole history" "- **server:** first feature before the tag" +lacks "a first release stops at its commit" "add the release workflow" + +release_notes=$(cd "${history}" && RELEASE_REPOSITORY=Example/Fork "${notes}" 0.2.0 v0.1.0 HEAD) +has "RELEASE_REPOSITORY names the images in lower case" "ghcr.io/example/fork/server:0.2.0" + +git -C "${history}" tag v0.2.0 +release_notes=$(in_history "${notes}" 0.2.1 v0.2.0 HEAD) +has "a release without commits says so" "No commits since the previous release." + +expect "bad usage" 2 "" "usage:" -- "${notes}" 0.2.0 v0.1.0 +expect "an unknown ref fails" 128 "" - -- in_history "${notes}" 0.2.0 v9.9.9 HEAD + +echo "# checksums.sh" +assets=${work}/assets +mkdir -p "${assets}" +printf 'hub firmware\n' > "${assets}/coldframe-hub-0.2.0.bin" +printf 'node firmware\n' > "${assets}/coldframe-node-0.2.0.bin" +printf 'chart\n' > "${assets}/server-0.2.0.tgz" +expect "SHA256SUMS is written and verified" 0 "coldframe-hub-0.2.0.bin: OK" "" -- "${checksums}" "${assets}" +listed=$(awk '{print $2}' "${assets}/SHA256SUMS" | paste -sd, -) +if [[ ${listed} == "coldframe-hub-0.2.0.bin,coldframe-node-0.2.0.bin,server-0.2.0.tgz" ]]; then + pass "SHA256SUMS lists every asset but itself, sorted" +else + failed "SHA256SUMS lists every asset but itself, sorted" "listed: ${listed}" +fi +if (cd "${assets}" && sha256sum --check --strict --quiet SHA256SUMS); then + pass "SHA256SUMS is in sha256sum format" +else + failed "SHA256SUMS is in sha256sum format" "$(cat "${assets}/SHA256SUMS")" +fi +expect "an untouched directory verifies" 0 "server-0.2.0.tgz: OK" "" -- "${checksums}" --check "${assets}" +printf 'tampered\n' >> "${assets}/coldframe-node-0.2.0.bin" +expect "a tampered file is detected" 1 "coldframe-node-0.2.0.bin: FAILED" - -- "${checksums}" --check "${assets}" +printf 'node firmware\n' > "${assets}/coldframe-node-0.2.0.bin" +printf 'unlisted\n' > "${assets}/extra.tgz" +expect "an unlisted file is detected" 1 "" "does not list exactly the assets" -- "${checksums}" --check "${assets}" +rm -f "${assets}/extra.tgz" "${assets}/server-0.2.0.tgz" +expect "a missing file is detected" 1 "" "does not list exactly the assets" -- "${checksums}" --check "${assets}" +mkdir -p "${work}/empty" +expect "an empty directory fails" 1 "" "no assets" -- "${checksums}" "${work}/empty" +mkdir -p "${assets}/nested" +expect "a subdirectory fails" 1 "" "not a regular file" -- "${checksums}" "${assets}" +expect "bad usage" 2 "" "usage:" -- "${checksums}" + +echo "# ${passes} passed, ${failures} failed" +[[ ${failures} -eq 0 ]] diff --git a/docs/build/apps.md b/docs/build/apps.md new file mode 100644 index 00000000..34526357 --- /dev/null +++ b/docs/build/apps.md @@ -0,0 +1,36 @@ +# Build your own apps + +Coldframe publishes no mobile store binaries. Each adopter builds the iOS and Android apps from +source for their own Server, with their own push credentials. This page is a short guide; Story 10.3 +expands it. + +## What you need + +- **Your own push credentials.** iOS pushes go through the Apple Push Notification service with + your APNs key: the Apple Developer Program, an App ID with the Push Notifications capability, and + the same key on the Server. Android pushes go through your own Firebase Cloud Messaging project. + No key, team ID, provisioning profile or `google-services.json` belongs in this repository. +- **The URLs of your deployment.** The Server URL and the Keycloak issuer + (`https:///realms/coldframe`) are build-time configuration. The apps have no field to + enter or change them, so every build is for one deployment. An unconfigured build starts and shows + the Unreachable notice. + +## iOS + +Put your values in `apps/swift/ios/Config/Coldframe.local.xcconfig` (not committed), next to +[`Coldframe.xcconfig`](../../apps/swift/ios/Config/Coldframe.xcconfig), which documents every key: +`COLDFRAME_SERVER_URL`, `COLDFRAME_KEYCLOAK_ISSUER`, and for pushes `COLDFRAME_PUSH = YES` and your +`DEVELOPMENT_TEAM`. Building, signing and push setup: [`apps/swift/README.md`](../../apps/swift/README.md). + +## Android + +Pass the configuration as Gradle properties, on the command line or in `~/.gradle/gradle.properties`: +`coldframe.serverUrl`, `coldframe.keycloakIssuer`, and for pushes `coldframe.firebaseProjectId`, +`coldframe.firebaseApplicationId`, `coldframe.firebaseApiKey` and `coldframe.firebaseSenderId` +(see [`apps/kt/android/build.gradle.kts`](../../apps/kt/android/build.gradle.kts)). Building and the +Firebase setup: [`apps/kt/README.md`](../../apps/kt/README.md). + +## Which version + +Build the apps from the same release tag as the Server you run +([`docs/operations/release.md`](../operations/release.md)). diff --git a/docs/operations/install.md b/docs/operations/install.md index c14a4d0e..aecb5713 100644 --- a/docs/operations/install.md +++ b/docs/operations/install.md @@ -10,6 +10,9 @@ or a later one; no earlier tag has the bundles, and the tag in the file is a pla (step 4). Nothing secret is in Git: the Secrets, the realm and the site values are created once, out of band. +What a release tag publishes (images, firmware, charts, `SHA256SUMS`) and how a maintainer cuts +one: [`docs/operations/release.md`](release.md). + | Bundle | Folder | Release (namespace) | Waits for | | --- | --- | --- | --- | | `cert-manager` | [`deploy/fleet/cert-manager`](../../deploy/fleet/cert-manager/fleet.yaml) | `cert-manager` (`cert-manager`) | nothing | @@ -108,7 +111,10 @@ it is imported, on the first start. **A realm that exists already is never re-im upgrading an existing installation set both by hand once, in the Keycloak admin console of the `coldframe` realm: Realm settings → Themes → Email theme `coldframe`; Realm settings → Email → the SMTP host, port, from address, authentication and encryption; and Authentication → Required actions → enable -**Invitation** (`invitation-required-action`), which shows the invitee the invitation after signing in. Also set +**Invitation** (`invitation-required-action`), which shows the invitee the invitation after signing in; +and Realm settings → Login → turn on **Verify email**. Registration needs a verified email: Phase Two offers +an invitation only to a verified address, so without it an invitee who registers is never shown the +invitation. Users who registered before are asked to verify their email at their next sign-in. Also set `keycloak.invitationRedirectUri` under the `server` key of your site values to `https:///garden` (the example values do), so that someone who accepts lands in the web app. Without SMTP, Phase Two still creates the invitation but cannot email it: it only logs `Unable to send invitation email`, and the Owner sees the invitation as diff --git a/docs/operations/release.md b/docs/operations/release.md new file mode 100644 index 00000000..5f6c9b10 --- /dev/null +++ b/docs/operations/release.md @@ -0,0 +1,140 @@ +# Cut a release + +One SemVer per release tag `vX.Y.Z` (or `vX.Y.Z-`) covers the whole monorepo: the +container images, the Hub and Node firmware and the Helm charts all carry `X.Y.Z` (Story 10.1, +AD-23). The tag triggers [`.github/workflows/release.yml`](../../.github/workflows/release.yml), +which builds and checks everything first and publishes only when every check passed. + +## Steps + +1. **Set the charts to the version.** On a branch from `main`: + + ```sh + deploy/charts/set-version.sh 0.2.0 + deploy/charts/check-version.sh 0.2.0 + ``` + + `set-version.sh` sets `version` of every chart in `deploy/charts` and `appVersion` of the + Coldframe charts (server, web, keycloak); temporal and nats keep the upstream `appVersion`. +2. **Open a pull request** with the change, for example `chore(release): 0.2.0`. Changing a + `Chart.yaml` runs the release workflow as a [dry run](#dry-run) at the new version; check that + it is green and look at its artifact. +3. **Merge, then tag the merge commit** on `main` and push the tag: + + ```sh + git switch main && git pull + git tag -a v0.2.0 -m "Coldframe 0.2.0" + git push origin v0.2.0 + ``` + + A tag with a pre-release part (`v0.2.0-rc.1`) produces a GitHub pre-release. The charts then + have to be at `0.2.0-rc.1`. +4. **Watch the `Release` run.** If a gate fails, nothing is pushed or released: fix the cause, + delete the tag (`git push --delete origin v0.2.0`, `git tag -d v0.2.0`) and tag again. The + exception is the final **Release** job (`gh release create`): it runs after the images were + pushed at `X.Y.Z`. If it fails, re-run the failed job from the run page (**Re-run failed jobs**); + do not delete or move the tag, because the pushed images already carry the version. + +## What gets published + +| What | Where | Name | +| --- | --- | --- | +| Container images (linux/amd64 and linux/arm64 where the base image has both) | ghcr.io | `ghcr.io/escendit/coldframe/{server,web,migrations,keycloak}:X.Y.Z` | +| Hub and Node flash images: bootloader, partition table and app merged, flashed at offset 0 | GitHub Release | `coldframe-hub-X.Y.Z.bin`, `coldframe-node-X.Y.Z.bin` | +| Hub and Node firmware ELF | GitHub Release | `coldframe-hub-X.Y.Z.elf`, `coldframe-node-X.Y.Z.elf` | +| Every Helm chart of `deploy/charts` | GitHub Release | `-X.Y.Z.tgz` | +| Checksums of every Release asset | GitHub Release | `SHA256SUMS` | +| Release notes, grouped by Conventional Commit type | GitHub Release | the Release body | + +Only the exact version is tagged, never `latest`. The firmware is a release build +(`cargo build --release --locked`, no features). Before it is attached, the workflow checks that +a release build with `--features dev-mode` fails with the `compile_error!` message, and that the +Hub image holds no credential canary ([`apps/rs/hub/check-image.sh`](../../apps/rs/hub/check-image.sh)). +There is no OTA: the firmware is flashed over USB with `espflash`. + +No mobile store binaries are published. Adopters build the apps themselves: +[`docs/build/apps.md`](../build/apps.md). + +The release notes ([`deploy/release/release-notes.sh`](../../deploy/release/release-notes.sh)) cover +the commits since the previous release tag (`v[0-9]*`; for a final release, pre-release tags are +skipped), or the whole history for the first release, without merge +commits. They are grouped as Breaking changes (`type!:` or a `BREAKING CHANGE:` footer), Features, +Fixes, Performance, the other Conventional Commit types, and Other. + +## Order and gates + +The jobs run in this order, and each gate stops everything after it: + +1. **Mode**: a tag push is a release; the tag must be `vX.Y.Z` or `vX.Y.Z-`. +2. **Checks**: the tests of the release scripts + ([`deploy/release/test.sh`](../../deploy/release/test.sh)), the chart versions + ([`deploy/charts/check-version.sh`](../../deploy/charts/check-version.sh)) and the + [wire gate](#wire-gate). +3. **Firmware** (Hub and Node), **Charts** and **Notes**, side by side. +4. **Assets**: the assets must be exactly the four firmware files and one package per chart; + `SHA256SUMS` is written and verified with `sha256sum --check --strict`. +5. **Verify**: on a tag, the image verification of CI runs again + ([`.github/workflows/images-verify.yml`](../../.github/workflows/images-verify.yml)); a dry run + skips it, because CI runs it on every pull request. +6. **Publish**: the images are pushed. +7. **Release**: the verified assets are checked again and the GitHub Release is created with them. + It is the last job and the only one that may write to the repository. It runs after the images + were pushed; see step 4 of [Steps](#steps) for a failure here. + +## Wire gate + +The Server accepts every wire major from `oldestAcceptedProtocolMajor` to `protocolMajor` of +[`packages/crypto-spec/crypto-spec.json`](../../packages/crypto-spec/crypto-spec.json). Both are +generated into Rust, C# and Kotlin (`pnpm --filter @coldframe/crypto-spec run generate`). +[`deploy/release/check-wire-compat.sh`](../../deploy/release/check-wire-compat.sh) compares the spec +at the release with the spec at the previous release tag: + +- When `protocolMajor` is greater than at the previous tag, `oldestAcceptedProtocolMajor` must still + be at most the previous major, so the Server keeps decoding what Devices in the field send. + Otherwise the run fails before anything is pushed and names both majors. +- `oldestAcceptedProtocolMajor` greater than `protocolMajor`, a `protocolMajor` lower than at the + previous tag, and a major below 1 always fail. +- With no previous tag (the first release), the gate passes with a notice. + +To bump the major, raise `protocolMajor`, leave `oldestAcceptedProtocolMajor` at the old major and +make the Server decode both. Accepting the old major means more than the range check: the frame AAD +(`Frames.Aad` in `packages/cs/crypto`) and the downlink sealing in `DeviceGrain` must use the major +the Node speaks. The test `AFrameAtTheOldestAcceptedMajorIsDecodedAndItsDownlinkOpens` +(`tests/cs/server.tests/Devices/NodeFrameReaderTests.cs`) guards this: it fails as soon as +`protocolMajor` is bumped while those paths still handle only the current major. Raise `oldestAcceptedProtocolMajor` in a later release, once no Device +speaks the old major any more. + +## Dry run + +A pull request that changes the workflow, `deploy/release/`, a `Chart.yaml`, the crypto spec or the +firmware's `Cargo.toml`, `Cargo.lock` or `partitions.csv` runs the same workflow as a dry run. A +manual run (**Actions → Release → Run workflow**) does the same on any branch. The version comes +from `deploy/charts/server/Chart.yaml`. A dry run builds every asset, writes and verifies +`SHA256SUMS`, builds the images without pushing them and skips the image verification job (CI runs +it on every pull request). It uploads the assets, `SHA256SUMS` and the notes as the workflow +artifact `release-X.Y.Z-dry-run`. It pushes nothing and creates no Release. + +The release scripts can also run locally: + +```sh +deploy/release/test.sh +deploy/release/release-notes.sh 0.2.0 v0.1.0 HEAD +deploy/release/check-wire-compat.sh <(git show v0.1.0:packages/crypto-spec/crypto-spec.json) packages/crypto-spec/crypto-spec.json +``` + +## Verify a download + +Download the assets and `SHA256SUMS` of the Release into one directory, then: + +```sh +sha256sum --check --strict SHA256SUMS +``` + +Every line must say `OK`. With the GitHub CLI: + +```sh +gh release download v0.2.0 --repo escendit/coldframe --dir coldframe-0.2.0 +cd coldframe-0.2.0 && sha256sum --check --strict SHA256SUMS +``` + +Flash a merged image at offset 0, for example `espflash write-bin 0x0 coldframe-node-0.2.0.bin`. diff --git a/docs/quickstart.md b/docs/quickstart.md index 49038ca9..8be72dbc 100644 --- a/docs/quickstart.md +++ b/docs/quickstart.md @@ -67,7 +67,7 @@ Things to know: never written to the repository. - **Keycloak imports the realm `coldframe` on start** from [`aspire/keycloak/realms/coldframe-realm.json`](../aspire/keycloak/realms/coldframe-realm.json): - registration and Organizations on, and the confidential client `coldframe-web` for the web app + registration (with email verification) and Organizations on, and the confidential client `coldframe-web` for the web app (standard flow only, PKCE S256, redirect `http://localhost:5173/.oidc/signin/callback`). Its client secret is generated on every start; read it in the dashboard from the parameter `coldframe-web-client-secret`. It also holds the public client `coldframe-mobile` for the iOS @@ -139,7 +139,8 @@ pnpm --filter @coldframe/web dev Open `http://localhost:5173`. Take the Keycloak port from the `http` endpoint of `keycloak` in the dashboard and the secret from the parameter `coldframe-web-client-secret`; both change on every start. The port 5173 must stay as it is, because it is the client's registered redirect. Create a -user through *Register* on the Keycloak sign-in page. The variables can also go in +user through *Register* on the Keycloak sign-in page, then open the verification link from the email +that Mailpit caught (the `http` endpoint of `mailpit` in the dashboard). The variables can also go in `apps/ts/web/.env`, which git ignores. [`apps/ts/web/README.md`](../apps/ts/web/README.md) lists every variable. @@ -171,7 +172,8 @@ them as Gradle properties, or put the same keys in `~/.gradle/gradle.properties` never-resolvable `.invalid` addresses, so an unconfigured build shows "Can't reach your Coldframe Server". `10.0.2.2` is the host as seen from the Android emulator; only the debug build allows plain HTTP, and only to `10.0.2.2` and `localhost`. Release builds trust public certificates only. -Sign-in opens Keycloak in a Custom Tab; create a user through *Register* there. +Sign-in opens Keycloak in a Custom Tab; create a user through *Register* there and verify the email +through Mailpit, as for the web app. Push notifications need your own Firebase project; the repository holds no `google-services.json`. Add `-Pcoldframe.firebaseProjectId=…`, `-Pcoldframe.firebaseApplicationId=…`, diff --git a/packages/crypto-spec/crypto-spec.json b/packages/crypto-spec/crypto-spec.json index b458f6e8..7371d9d6 100644 --- a/packages/crypto-spec/crypto-spec.json +++ b/packages/crypto-spec/crypto-spec.json @@ -1,6 +1,7 @@ { "$comment": "The normative Device crypto contract (AD-12, AD-17, AD-19, AD-25). Every label, size and header name here is generated into Rust, C# and Kotlin by scripts/generate.ts; no language types one by hand. Changing a value is a wire break: bump the label version instead.", "protocolMajor": 1, + "oldestAcceptedProtocolMajor": 1, "keyHierarchy": { "rootKeyLength": 32, "deviceKey": { diff --git a/packages/crypto-spec/scripts/lib/spec.ts b/packages/crypto-spec/scripts/lib/spec.ts index 351ce1f2..2437edc9 100644 --- a/packages/crypto-spec/scripts/lib/spec.ts +++ b/packages/crypto-spec/scripts/lib/spec.ts @@ -5,6 +5,7 @@ import { fromRoot, repoRoot, specPath } from './paths.ts'; /** The shape of `crypto-spec.json`. */ export interface CryptoSpec { protocolMajor: number; + oldestAcceptedProtocolMajor: number; keyHierarchy: { rootKeyLength: number; deviceKey: { algorithm: string; construction: string; label: string; length: number }; @@ -121,6 +122,7 @@ export function constantsOf(spec: CryptoSpec): ConstantGroup[] { title: 'Protocol', constants: [ { name: 'protocolMajor', kind: 'u8', value: spec.protocolMajor, doc: 'The current wire major; every frame and envelope carries it.' }, + { name: 'oldestAcceptedProtocolMajor', kind: 'u8', value: spec.oldestAcceptedProtocolMajor, doc: 'The oldest wire major the Server still accepts.' }, ], }, { diff --git a/packages/cs/crypto/Generated/CryptoSpec.g.cs b/packages/cs/crypto/Generated/CryptoSpec.g.cs index 2d88ae60..b759acfa 100644 --- a/packages/cs/crypto/Generated/CryptoSpec.g.cs +++ b/packages/cs/crypto/Generated/CryptoSpec.g.cs @@ -18,6 +18,11 @@ public static class CryptoSpec /// public const byte ProtocolMajor = 1; + /// + /// The oldest wire major the Server still accepts. + /// + public const byte OldestAcceptedProtocolMajor = 1; + // Key hierarchy (AD-12) /// diff --git a/packages/kt/core/generated/kotlin/com/escendit/coldframe/core/crypto/CryptoSpec.kt b/packages/kt/core/generated/kotlin/com/escendit/coldframe/core/crypto/CryptoSpec.kt index c64f0024..ad6e27d3 100644 --- a/packages/kt/core/generated/kotlin/com/escendit/coldframe/core/crypto/CryptoSpec.kt +++ b/packages/kt/core/generated/kotlin/com/escendit/coldframe/core/crypto/CryptoSpec.kt @@ -13,6 +13,9 @@ public object CryptoSpec { /** The current wire major; every frame and envelope carries it. */ public const val PROTOCOL_MAJOR: Int = 1 + /** The oldest wire major the Server still accepts. */ + public const val OLDEST_ACCEPTED_PROTOCOL_MAJOR: Int = 1 + // Key hierarchy (AD-12) /** Length of the eFuse root key in bytes. */ diff --git a/packages/proto/README.md b/packages/proto/README.md index 08e82f8a..4140c093 100644 --- a/packages/proto/README.md +++ b/packages/proto/README.md @@ -30,8 +30,12 @@ Sensor, otherwise in `unit` and within the range. ## Versioning - Every top-level frame and envelope carries `uint32 protocol_version = 1`; the current major is `1` - (`PROTOCOL_MAJOR` in [`packages/crypto-spec`](../crypto-spec)). The Server accepts the current and the - previous major. + (`PROTOCOL_MAJOR` in [`packages/crypto-spec`](../crypto-spec)). The Server accepts every major from + `OLDEST_ACCEPTED_PROTOCOL_MAJOR` to `PROTOCOL_MAJOR`; the release wire gate fails a major bump that + drops the previous one ([`docs/operations/release.md`](../../docs/operations/release.md#wire-gate)). + Keeping an old major accepted needs the frame AAD and the downlink sealing to handle that major + too, not only the range check; `AFrameAtTheOldestAcceptedMajorIsDecodedAndItsDownlinkOpens` in + `tests/cs/server.tests/Devices/NodeFrameReaderTests.cs` fails until they do. - Within a major, changes are additive only. A field number is never reused: when a field goes, reserve its number and name (`reserved 4; reserved "old_name";`). - A wire break is a new major in a new package (`coldframe.device.v2`), shipped in a release whose diff --git a/packages/rs/crypto/src/spec.rs b/packages/rs/crypto/src/spec.rs index 2b9bc6f7..919ffd28 100644 --- a/packages/rs/crypto/src/spec.rs +++ b/packages/rs/crypto/src/spec.rs @@ -9,6 +9,9 @@ /// The current wire major; every frame and envelope carries it. pub const PROTOCOL_MAJOR: u8 = 1; +/// The oldest wire major the Server still accepts. +pub const OLDEST_ACCEPTED_PROTOCOL_MAJOR: u8 = 1; + // Key hierarchy (AD-12) /// Length of the eFuse root key in bytes. diff --git a/tests/cs/server.integration/Devices/IngestTests.cs b/tests/cs/server.integration/Devices/IngestTests.cs index f77844a6..ba3810cd 100644 --- a/tests/cs/server.integration/Devices/IngestTests.cs +++ b/tests/cs/server.integration/Devices/IngestTests.cs @@ -203,8 +203,11 @@ public async Task WhatDoesNotAuthenticateIsRejectedAuthAndDoesNotTouchTheOtherFr otherCounter.Counter += 100; var otherDevice = template.Clone(); otherDevice.DeviceId = ByteString.CopyFrom(other.DeviceId.ToBytes()); - var otherVersion = template.Clone(); - otherVersion.ProtocolVersion = 2; + // A major above the current one and one below the oldest the Server accepts. + var newerVersion = template.Clone(); + newerVersion.ProtocolVersion = CryptoSpec.ProtocolMajor + 1u; + var olderVersion = template.Clone(); + olderVersion.ProtocolVersion = CryptoSpec.OldestAcceptedProtocolMajor - 1u; var undecodable = node.SealFrame([0x0A, 0xFF]); string[] frames = @@ -213,7 +216,8 @@ public async Task WhatDoesNotAuthenticateIsRejectedAuthAndDoesNotTouchTheOtherFr Convert.ToBase64String(Encoding.UTF8.GetBytes("not a SealedEnvelope")), SimulatedDevice.EncodeFrame(otherCounter), SimulatedDevice.EncodeFrame(otherDevice), - SimulatedDevice.EncodeFrame(otherVersion), + SimulatedDevice.EncodeFrame(newerVersion), + SimulatedDevice.EncodeFrame(olderVersion), SimulatedDevice.EncodeFrame(undecodable), SimulatedDevice.EncodeFrame(good), ]; @@ -221,10 +225,10 @@ public async Task WhatDoesNotAuthenticateIsRejectedAuthAndDoesNotTouchTheOtherFr var results = await IngestAsync(stage.Hub, SimulatedDevice.IngestBody(frames), cancellationToken); Assert.Equal( - ["rejected_auth", "rejected_auth", "rejected_auth", "rejected_auth", "rejected_auth", "rejected_auth", "stored"], + ["rejected_auth", "rejected_auth", "rejected_auth", "rejected_auth", "rejected_auth", "rejected_auth", "rejected_auth", "stored"], results.Select(result => result.Status)); - Assert.All(results.Take(6), result => Assert.Null(result.Downlink)); - Assert.Equal(good.Counter, node.OpenDownlink(results[6].Downlink!).AckedCounter); + Assert.All(results.Take(7), result => Assert.Null(result.Downlink)); + Assert.Equal(good.Counter, node.OpenDownlink(results[7].Downlink!).AckedCounter); Assert.Equal((4L, 1L), await CountsAsync(node, cancellationToken)); Assert.Equal((0L, 0L), await CountsAsync(other, cancellationToken)); } diff --git a/tests/cs/server.integration/Edge/EdgeApiFixture.cs b/tests/cs/server.integration/Edge/EdgeApiFixture.cs index c7afe825..a53d570a 100644 --- a/tests/cs/server.integration/Edge/EdgeApiFixture.cs +++ b/tests/cs/server.integration/Edge/EdgeApiFixture.cs @@ -160,38 +160,140 @@ public async Task AcceptInvitationAsync(TestUser invitee, string organizationId, { ArgumentNullException.ThrowIfNull(invitee); - var clientId = await EnsureBrowserClientAsync(cancellationToken); var cookies = new Dictionary(StringComparer.Ordinal); + using var keycloak = CreateBrowser(); + using var loginPage = await OpenSignInAsync(keycloak, cookies, cancellationToken); - using var addressed = fixture.App.CreateHttpClient(KeycloakResource, "http"); - using var handler = new HttpClientHandler { AllowAutoRedirect = false, UseCookies = false }; - using var keycloak = new HttpClient(handler) { BaseAddress = addressed.BaseAddress }; + var page = await BrowseAsync( + keycloak, + cookies, + HttpMethod.Post, + FormAction(await loginPage.Content.ReadAsStringAsync(cancellationToken)), + [new("username", invitee.Username), new("password", invitee.Password)], + cancellationToken); - var authorize = new Uri( - $"/realms/{Realm}/protocol/openid-connect/auth?client_id={clientId}&response_type=code&scope=openid" - + $"&redirect_uri={Uri.EscapeDataString(BrowserRedirectUri)}", - UriKind.Relative); + await AcceptOnInvitationPageAsync(keycloak, cookies, page, organizationId, cancellationToken); + } - using var loginPage = await BrowseAsync(keycloak, cookies, HttpMethod.Get, authorize, null, cancellationToken); - Assert.Equal(System.Net.HttpStatusCode.OK, loginPage.StatusCode); + /// + /// Registers through Keycloak's own registration form, as an invitee without an + /// account does, and accepts the Phase Two invitation to . Phase Two offers + /// an invitation only to a verified email, so the realm verifies it first: registration ends on the email + /// verification page, with no invitation offered, until the link from the verification email (read through + /// ) is opened in the same browser. Returns the new User's ID. + /// + public async Task RegisterAndAcceptInvitationAsync( + string email, + string password, + string organizationId, + Func> verificationLinkAsync, + CancellationToken cancellationToken) + { + ArgumentNullException.ThrowIfNull(verificationLinkAsync); + + var cookies = new Dictionary(StringComparer.Ordinal); + using var keycloak = CreateBrowser(); + + string registration; + using (var loginPage = await OpenSignInAsync(keycloak, cookies, cancellationToken)) + { + var link = System.Text.RegularExpressions.Regex.Match( + await loginPage.Content.ReadAsStringAsync(cancellationToken), + "href=\"([^\"]*/login-actions/registration[^\"]*)\"", + System.Text.RegularExpressions.RegexOptions.None, + TimeSpan.FromSeconds(1)); + Assert.True(link.Success, "The Keycloak sign-in page has no Register link."); + registration = System.Net.WebUtility.HtmlDecode(link.Groups[1].Value); + } + + using var form = await BrowseAsync(keycloak, cookies, HttpMethod.Get, new Uri(registration, UriKind.RelativeOrAbsolute), null, cancellationToken); + Assert.Equal(System.Net.HttpStatusCode.OK, form.StatusCode); var page = await BrowseAsync( keycloak, cookies, HttpMethod.Post, - FormAction(await loginPage.Content.ReadAsStringAsync(cancellationToken)), - [new("username", invitee.Username), new("password", invitee.Password)], + FormAction(await form.Content.ReadAsStringAsync(cancellationToken)), + [ + new("username", email), + new("email", email), + new("firstName", "neighbour"), + new("lastName", "Test"), + new("password", password), + new("password-confirm", password), + ], cancellationToken); + page = await FollowAsync(keycloak, cookies, page, cancellationToken); + + using (page) + { + Assert.Equal(System.Net.HttpStatusCode.OK, page.StatusCode); + Assert.Contains("execution=VERIFY_EMAIL", page.RequestMessage?.RequestUri?.Query ?? string.Empty, StringComparison.Ordinal); + Assert.DoesNotContain($"value=\"{organizationId}\"", await page.Content.ReadAsStringAsync(cancellationToken), StringComparison.Ordinal); + } - // The required action redirects to its own page first. + var verification = await BrowseAsync(keycloak, cookies, HttpMethod.Get, await verificationLinkAsync(cancellationToken), null, cancellationToken); + await AcceptOnInvitationPageAsync(keycloak, cookies, verification, organizationId, cancellationToken); + + using var admin = await CreateAdminClientAsync(cancellationToken); + var users = await admin.GetFromJsonAsync( + new Uri($"/admin/realms/{Realm}/users?email={Uri.EscapeDataString(email)}&exact=true", UriKind.Relative), + cancellationToken); + var user = Assert.Single(users.EnumerateArray()); + Assert.True(user.GetProperty("emailVerified").GetBoolean()); + return user.GetProperty("id").GetString()!; + } + + // A browser on Keycloak's own address: redirects are followed by hand and cookies kept in a dictionary. + private HttpClient CreateBrowser() + { + using var addressed = fixture.App.CreateHttpClient(KeycloakResource, "http"); + return new HttpClient(new HttpClientHandler { AllowAutoRedirect = false, UseCookies = false }, disposeHandler: true) + { + BaseAddress = addressed.BaseAddress, + }; + } + + private async Task OpenSignInAsync(HttpClient keycloak, Dictionary cookies, CancellationToken cancellationToken) + { + var clientId = await EnsureBrowserClientAsync(cancellationToken); + var authorize = new Uri( + $"/realms/{Realm}/protocol/openid-connect/auth?client_id={clientId}&response_type=code&scope=openid" + + $"&redirect_uri={Uri.EscapeDataString(BrowserRedirectUri)}", + UriKind.Relative); + + var loginPage = await BrowseAsync(keycloak, cookies, HttpMethod.Get, authorize, null, cancellationToken); + Assert.Equal(System.Net.HttpStatusCode.OK, loginPage.StatusCode); + return loginPage; + } + + // Follows Keycloak's redirects (a required action redirects to its own page first) until a page or the client. + private static async Task FollowAsync( + HttpClient keycloak, + Dictionary cookies, + HttpResponseMessage page, + CancellationToken cancellationToken) + { while (page.StatusCode == System.Net.HttpStatusCode.Found && page.Headers.Location is { } next - && !next.AbsoluteUri.StartsWith(BrowserRedirectUri, StringComparison.Ordinal)) + && !next.ToString().StartsWith(BrowserRedirectUri, StringComparison.Ordinal)) { page.Dispose(); page = await BrowseAsync(keycloak, cookies, HttpMethod.Get, next, null, cancellationToken); } + return page; + } + + private static async Task AcceptOnInvitationPageAsync( + HttpClient keycloak, + Dictionary cookies, + HttpResponseMessage page, + string organizationId, + CancellationToken cancellationToken) + { + page = await FollowAsync(keycloak, cookies, page, cancellationToken); + using (page) { var html = await page.Content.ReadAsStringAsync(cancellationToken); diff --git a/tests/cs/server.integration/Edge/InvitationTests.cs b/tests/cs/server.integration/Edge/InvitationTests.cs index 5249c13d..85dd3487 100644 --- a/tests/cs/server.integration/Edge/InvitationTests.cs +++ b/tests/cs/server.integration/Edge/InvitationTests.cs @@ -95,6 +95,54 @@ public async Task AnInvitationIsEmailedAndOnceAcceptedTheInviteeIsAMemberWithThe } } + [Fact] + public async Task AnInviteeWhoRegistersVerifiesTheirEmailBeforeAcceptingAndBecomesAMember() + { + var cancellationToken = TestContext.Current.CancellationToken; + var owner = await edge.CreateUserAsync("inviter-register", cancellationToken); + using var server = edge.CreateServerClient(owner.AccessToken); + var siteId = await CreateSiteAsync(server, "Home", cancellationToken); + var email = $"newcomer-{Guid.NewGuid():N}@example.org"; + + using (var invited = await InviteAsync(server, siteId, email, "Administrator", cancellationToken)) + { + Assert.Equal(HttpStatusCode.Accepted, invited.StatusCode); + } + + await WaitForEmailAsync(email, cancellationToken); + + // Someone without an account registers on Keycloak's page. Phase Two offers an invitation only to a + // verified email, so the realm verifies it first; the invitation follows the link in that email. + var userId = await edge.RegisterAndAcceptInvitationAsync( + email, + Guid.NewGuid().ToString("N"), + siteId, + async ct => + { + var verification = await WaitForEmailAsync(email, ct, subject: "Verify email"); + var link = System.Text.RegularExpressions.Regex.Match( + verification.GetProperty("Text").GetString()!, + "https?://\\S+/login-actions/action-token\\S+", + System.Text.RegularExpressions.RegexOptions.None, + TimeSpan.FromSeconds(1)); + Assert.True(link.Success, "The verification email has no link."); + return new Uri(link.Value, UriKind.Absolute); + }, + cancellationToken); + + using var arrival = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken); + arrival.CancelAfter(ArrivalTimeout); + IReadOnlyList<(string UserId, string Role)> members; + do + { + await Task.Delay(TimeSpan.FromMilliseconds(500), arrival.Token); + members = await MembersAsync(server, siteId, cancellationToken); + } + while (members.Count < 2); + + Assert.Equal([(owner.UserId, "Owner"), (userId, "Administrator")], members); + } + [Fact] public async Task AnInvalidEmailOrRoleIsRefusedAndNothingIsSent() { @@ -119,7 +167,7 @@ public async Task AnInvalidEmailOrRoleIsRefusedAndNothingIsSent() } using var mailpit = edge.AppHost.App.CreateHttpClient(MailpitResource, "http"); - Assert.Empty(await SearchAsync(mailpit, email, cancellationToken)); + Assert.Empty(await SearchAsync(mailpit, email, subject: null, cancellationToken)); } private static Task InviteAsync(HttpClient server, string siteId, string email, string role, CancellationToken cancellationToken) => @@ -138,7 +186,7 @@ .. body.RootElement.GetProperty("members").EnumerateArray() ]; } - private async Task WaitForEmailAsync(string to, CancellationToken cancellationToken) + private async Task WaitForEmailAsync(string to, CancellationToken cancellationToken, string? subject = null) { using var mailpit = edge.AppHost.App.CreateHttpClient(MailpitResource, "http"); using var arrival = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken); @@ -146,7 +194,7 @@ private async Task WaitForEmailAsync(string to, CancellationToken c while (true) { - var found = await SearchAsync(mailpit, to, cancellationToken); + var found = await SearchAsync(mailpit, to, subject, cancellationToken); if (found.Count > 0) { @@ -160,10 +208,11 @@ private async Task WaitForEmailAsync(string to, CancellationToken c } } - private static async Task> SearchAsync(HttpClient mailpit, string to, CancellationToken cancellationToken) + private static async Task> SearchAsync(HttpClient mailpit, string to, string? subject, CancellationToken cancellationToken) { + var query = subject is null ? $"to:\"{to}\"" : $"to:\"{to}\" subject:\"{subject}\""; using var response = await mailpit.GetAsync( - new Uri($"/api/v1/search?query={Uri.EscapeDataString($"to:\"{to}\"")}", UriKind.Relative), + new Uri($"/api/v1/search?query={Uri.EscapeDataString(query)}", UriKind.Relative), cancellationToken); response.EnsureSuccessStatusCode(); using var body = await EdgeApiFixture.ReadJsonAsync(response, cancellationToken); diff --git a/tests/cs/server.integration/Identity/IdentityCluster.cs b/tests/cs/server.integration/Identity/IdentityCluster.cs index bf112272..bdd52634 100644 --- a/tests/cs/server.integration/Identity/IdentityCluster.cs +++ b/tests/cs/server.integration/Identity/IdentityCluster.cs @@ -371,10 +371,17 @@ public void Configure(ISiloBuilder siloBuilder) // Reminders live in the class's own database, in the table the migration job creates, as on the Server // (AD-22). The in-memory table was a grain of the silo, and right after a restart its placement could // stall for 30 s at a time, so every grain that registered or read a reminder meanwhile timed out on CI. + // Its connections are not pooled: a pool per test class kept idle connections open, and with the suite's + // classes side by side the AppHost's PostgreSQL ran out of them ("too many clients"). Reminder reads and + // writes are rare enough to open a connection each. siloBuilder.UseAdoNetReminderService(options => { options.Invariant = "Npgsql"; - options.ConnectionString = connectionString; + options.ConnectionString = new NpgsqlConnectionStringBuilder(connectionString) + { + Pooling = false, + ApplicationName = "coldframe-tests-reminders", + }.ConnectionString; }); siloBuilder.Services.AddSingleton(); diff --git a/tests/cs/server.integration/KeycloakTests.cs b/tests/cs/server.integration/KeycloakTests.cs index 0650c8d5..26d9d3ba 100644 --- a/tests/cs/server.integration/KeycloakTests.cs +++ b/tests/cs/server.integration/KeycloakTests.cs @@ -158,6 +158,9 @@ public async Task ColdframeRealmHasTheConfidentialWebClient() Assert.True( realm.RootElement.GetProperty("registrationAllowed").GetBoolean(), "The realm does not allow registration."); + Assert.True( + realm.RootElement.GetProperty("verifyEmail").GetBoolean(), + "The realm does not verify a registered email; Phase Two offers invitations only to a verified one."); Assert.True( realm.RootElement.GetProperty("organizationsEnabled").GetBoolean(), "The realm does not have Organizations enabled."); diff --git a/tests/cs/server.tests/Devices/NodeFrameReaderTests.cs b/tests/cs/server.tests/Devices/NodeFrameReaderTests.cs index b5351503..d7b0c61a 100644 --- a/tests/cs/server.tests/Devices/NodeFrameReaderTests.cs +++ b/tests/cs/server.tests/Devices/NodeFrameReaderTests.cs @@ -113,7 +113,9 @@ void Break(Action change) broken.Add(frame); } - Break(frame => frame.ProtocolVersion = 2); + // A major above the current one and one below the oldest accepted one. + Break(frame => frame.ProtocolVersion = CryptoSpec.ProtocolMajor + 1u); + Break(frame => frame.ProtocolVersion = CryptoSpec.OldestAcceptedProtocolMajor - 1u); Break(frame => frame.ProtocolVersion = 0); Break(frame => frame.ClearMeasured()); Break(frame => frame.MeasuredAtMs = -1); @@ -375,6 +377,34 @@ public void AFrameThatIsNotStoredReadsNoSpecifications() } // The frame as the Node seals it: with the sealing boot and uptime. + /// + /// Tripwire for the release wire gate (docs/operations/release.md#wire-gate): a Node at + /// must still be decoded end to end. Today the + /// oldest major is the current one, so this passes trivially; once protocolMajor is bumped + /// with the old major still accepted, it fails until the frame AAD () and the + /// downlink sealing (DeviceGrain) handle the old major too. + /// + [Fact] + public void AFrameAtTheOldestAcceptedMajorIsDecodedAndItsDownlinkOpens() + { + const ulong counter = 7; + var node = SimulatedDevice.Create(ProtocolKind.Node); + var frame = Stamp(node, node.Wake(Now)); + frame.ProtocolVersion = CryptoSpec.OldestAcceptedProtocolMajor; + + // The frame as a Node at the oldest major seals it: its major is the first AAD byte. + var oldAad = Frames.Aad(node.DeviceId, counter); + oldAad[0] = CryptoSpec.OldestAcceptedProtocolMajor; + var sealedFrame = Aead.Seal(node.Keys.SealKey, Frames.Nonce(node.DeviceId, counter), oldAad, frame.ToByteArray()); + + var plaintext = Frames.Open(node.Keys.SealKey, node.DeviceId, counter, sealedFrame); + Assert.Equal(NodeFrameVerdict.Valid, NodeFrameReader.Read(plaintext, node.DeviceId, Now).Verdict); + + // The downlink as the Server seals it must open on that Node. + var downlink = Frames.Seal(node.Keys.AckKey, node.DeviceId, counter, [1, 2, 3]); + Assert.Equal([1, 2, 3], Aead.Open(node.Keys.AckKey, Frames.Nonce(node.DeviceId, counter), oldAad, downlink)); + } + private static NodeFrame Stamp(SimulatedDevice node, NodeFrame wake) => NodeFrame.Parser.ParseFrom(new SimulatedServer(node.Keys.DeviceKey).VerifyFrame(node.SealFrame(wake))); } From eb1326f174828a95828aa83c614f1037e00bca51 Mon Sep 17 00:00:00 2001 From: Simon Novak <25969+snovak7@users.noreply.github.com> Date: Mon, 19 Oct 2026 22:41:08 +0200 Subject: [PATCH 2/2] feat(docs): build a Node and a Hub from the docs Add a reference Node and Hub design under hardware/ (schematics as connection tables, BOM, wiring with the switched probe power and battery divider, enclosure and probe sealing) and an adopter build guide under docs/build/ (flashing with espflash from release binaries or source, first boot and the setup code, the irreversible eFuse burn and dev mode, power budget and placement). A new Docs workflow resolves every relative link and referenced path and runs the guide's build commands against the firmware crates, build only. Closes #119 Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01XJ9QyLY9aHkpAdZQsUJN3n --- .github/workflows/docs.yml | 126 +++++++ README.md | 4 +- ...-2-build-a-node-and-a-hub-from-the-docs.md | 272 ++++++++++++++ docs/build/README.md | 58 +++ docs/build/apps.md | 3 + docs/build/flash.md | 288 ++++++++++++++ docs/build/power.md | 105 ++++++ docs/check-commands.sh | 201 ++++++++++ docs/check-links.sh | 135 +++++++ docs/quickstart.md | 5 +- docs/test.sh | 351 ++++++++++++++++++ hardware/README.md | 39 +- hardware/bom.md | 47 +++ hardware/enclosure.md | 66 ++++ hardware/hub.md | 74 ++++ hardware/node.md | 215 +++++++++++ 16 files changed, 1985 insertions(+), 4 deletions(-) create mode 100644 .github/workflows/docs.yml create mode 100644 _bmad-output/implementation-artifacts/spec-10-2-build-a-node-and-a-hub-from-the-docs.md create mode 100644 docs/build/README.md create mode 100644 docs/build/flash.md create mode 100644 docs/build/power.md create mode 100755 docs/check-commands.sh create mode 100755 docs/check-links.sh create mode 100755 docs/test.sh create mode 100644 hardware/bom.md create mode 100644 hardware/enclosure.md create mode 100644 hardware/hub.md create mode 100644 hardware/node.md diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml new file mode 100644 index 00000000..f40b7dc6 --- /dev/null +++ b/.github/workflows/docs.yml @@ -0,0 +1,126 @@ +# Checks the adopter docs (Story 10.2): every relative link and referenced repository path +# resolves, and every command block of the build guide (docs/build/*.md) either runs against the +# firmware crates (``, builds only) or, when it needs a board +# (``), names only real espflash subcommands. Nothing is ever flashed, and no +# external link is fetched. Not a required status check. +# +# Actions are pinned to a commit; the comment names the release. The firmware setup of the +# `commands` job is copied from the `firmware` job of ci.yml: keep the two in step. +name: Docs + +on: + pull_request: + paths: + # Everything the docs link to or mirror (the toolchain pins of ci.yml and release.yml). + - "docs/**" + - "hardware/**" + - "README.md" + - "LICENSE" + - ".gitignore" + - "apps/**" + - "packages/**" + - "deploy/**" + - "tests/**" + - "aspire/**" + - ".github/workflows/**" + push: + branches: + - main + paths: + - "docs/**" + - "hardware/**" + - "README.md" + - "LICENSE" + - ".gitignore" + - "apps/**" + - "packages/**" + - "deploy/**" + - "tests/**" + - "aspire/**" + - ".github/workflows/**" + workflow_dispatch: + +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: ${{ github.event_name == 'pull_request' }} + +jobs: + links: + name: Links + runs-on: ubuntu-24.04 + timeout-minutes: 10 + steps: + - name: Check out + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + + - name: Test the doc checks + run: docs/test.sh + + - name: Check links and paths + run: docs/check-links.sh + + commands: + name: Commands + runs-on: ubuntu-24.04 + timeout-minutes: 60 + env: + # The same pins as the `firmware` job of ci.yml. + ESPUP_VERSION: "0.17.1" + ESP_TOOLCHAIN_VERSION: "1.95.0.0" + ESPFLASH_VERSION: "4.5.0" + steps: + - name: Check out + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + + # cmake and ninja for the mbedTLS rebuild; libudev for espflash. + - name: Install build tools + run: sudo apt-get update && sudo apt-get install --yes --no-install-recommends cmake ninja-build libudev-dev pkg-config + + - name: Cache + uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2.9.2 + with: + workspaces: | + apps/rs/hub + apps/rs/node + key: espup-${{ env.ESPUP_VERSION }}-esp-${{ env.ESP_TOOLCHAIN_VERSION }}-espflash-${{ env.ESPFLASH_VERSION }} + cache-bin: "true" + cache-directories: | + ~/.rustup/toolchains/esp + ~/export-esp.sh + + # On the runner's stable toolchain: apps/rs/hub/rust-toolchain.toml names `esp`, which is not + # installed until the next step. + - name: Install espup and espflash + working-directory: apps/rs/hub + run: | + if [[ "$(espup --version 2>/dev/null)" != "espup ${ESPUP_VERSION}" ]]; then + rustup run stable cargo install espup --version "${ESPUP_VERSION}" --locked --force + fi + if [[ "$(espflash --version 2>/dev/null)" != "espflash ${ESPFLASH_VERSION}" ]]; then + rustup run stable cargo install espflash --version "${ESPFLASH_VERSION}" --locked --force + fi + + - name: Install the esp toolchain + working-directory: apps/rs/hub + env: + GITHUB_TOKEN: ${{ github.token }} + run: | + if [[ ! -f "${HOME}/export-esp.sh" ]] || ! rustup run esp rustc --version | grep -qF "(${ESP_TOOLCHAIN_VERSION})"; then + espup install --toolchain-version "${ESP_TOOLCHAIN_VERSION}" --targets esp32s3 --export-file "${HOME}/export-esp.sh" + fi + rustup run esp rustc --version + + # The guide's own install block runs too; with the tools above already in place it reuses them. + - name: Run the documented commands + env: + GITHUB_TOKEN: ${{ github.token }} + run: | + docs/check-commands.sh --list + docs/check-commands.sh diff --git a/README.md b/README.md index df92d46a..3350de57 100644 --- a/README.md +++ b/README.md @@ -10,6 +10,8 @@ and deployment all live in this repository. ## Start here - [Developer quickstart](docs/quickstart.md): run the local stack and every test suite. +- [Build guide](docs/build/README.md): build, flash and add your own Node and Hub. +- [Hardware design](hardware/README.md): schematics, BOM, wiring, enclosure and probe sealing. - [Hub radio coexistence spike](docs/spikes/hub-radio-coexistence.md): what the Hub's radio can do at once. ## Repository layout @@ -21,7 +23,7 @@ and deployment all live in this repository. | [`tests/`](tests) | All tests, split by language | | [`aspire/`](aspire) | The local development stack, which is also the integration-test host | | [`deploy/`](deploy) | Helm charts, Fleet bundles and a Compose example | -| [`hardware/`](hardware) | Schematics, PCB and enclosure | +| [`hardware/`](hardware) | The reference Node and Hub design: schematics, BOM, wiring, enclosure | | [`docs/`](docs) | Guides and references | ## License diff --git a/_bmad-output/implementation-artifacts/spec-10-2-build-a-node-and-a-hub-from-the-docs.md b/_bmad-output/implementation-artifacts/spec-10-2-build-a-node-and-a-hub-from-the-docs.md new file mode 100644 index 00000000..a6a047fa --- /dev/null +++ b/_bmad-output/implementation-artifacts/spec-10-2-build-a-node-and-a-hub-from-the-docs.md @@ -0,0 +1,272 @@ +--- +title: 'Build a node and a hub from the docs' +type: 'feature' +created: '2026-10-19' +status: 'awaiting-operator' +baseline_revision: '21ad924cfaeb1caa249d9f8c17b682e7ea3e2c4a' +review_loop_iteration: 0 +followup_review_recommended: true +context: + - '{project-root}/_bmad-output/implementation-artifacts/epic-10-context.md' +warnings: [oversized] +deferred: [] +operator_actions: + - "Build one Node and one Hub from hardware/ (bom.md, node.md, hub.md, enclosure.md) on real parts, and correct any wiring or part error you find in those pages." + - "Check every example part number and figure in hardware/bom.md (TPS22917, RT9080-33, TP4056 input limit and R_PROG table, CN3791, the cell size) against its datasheet before anyone orders parts." + - "Flash a test board in dev mode from source, and a fresh board from the first release's binaries (after Story 10-1's v0.1.0 release), following docs/build/flash.md; confirm the identity line and the setup code appear as documented (Hub at first boot, Node on a 3 s long press), and fix any log line that differs." + - "Measure the reference Node's average sleep current with docs/bench/node-power-checklist.md, record the result in that checklist's Result table, and replace the 'not measured yet' notes in docs/build/power.md and hardware/node.md with the measured value." +--- + + + +## Intent + +**Problem:** `hardware/` is a placeholder, and `docs/build/` holds only the apps guide. A maker cannot build a Node or a Hub, wire the switched probe and divider, flash release or source firmware, or learn the irreversible eFuse burn without reading the firmware source. No CI job checks that the docs' links or build commands still work. + +**Approach:** Add a reference hardware design to `hardware/`: schematics as Markdown connection tables and ASCII block diagrams, a BOM, the wiring, enclosure notes and probe sealing. Add an adopter build guide to `docs/build/` covering flashing, first boot, eFuse and dev mode, power, and placement. Add a new path-filtered `docs.yml` workflow. It resolves every relative link and referenced repo path, and it runs every build command the guide documents against the firmware crates, with build only and no flashing. + +## Boundaries & Constraints + +**Always:** +- **Facts come from the code.** Every pin, constant and command must match the firmware. + - Pins: `apps/rs/node/src/board/pins.rs`. + - Wake and button timing: `packages/rs/sensing/src/{wake,button}.rs`. + - Partition offsets: `apps/rs/{hub,node}/partitions.csv`. + - Battery curve: `packages/rs/sensing/src/battery.rs`. + - Soil calibration: `apps/cs/server/Sensors/CalibrationMath.cs`. + - Link to the source file rather than restating a whole table where it could drift. +- **Label the design honestly.** `hardware/` is under Apache-2.0 through the root `LICENSE`, and every hardware page says so. Every part is a *reference choice* that has not been bench-verified: say so in `hardware/README.md`, and give the electrical requirement each part must meet. Examples: LDO quiescent current ≤ ~5 µA and ≥ 500 mA output; the charger has an NTC input and an open-drain active-low status pin; the load switches are active-high and need an external pull-down. +- **The eFuse warning is prominent.** It is a blockquote at the top of `docs/build/flash.md`, before any flash command. It says that the first boot of any build without `--features dev-mode`, debug or release, burns an `HMAC_UP` key block and that this cannot be undone. It says how to use dev mode on test boards, and that release binaries always burn. +- **State the setup code accurately.** The Hub prints `setup code=XXXXXXXX` on every boot while unprovisioned, which includes its first boot. The Node prints it when it enters setup mode, on a long press (3 s). Document how to watch the serial output, and the `erase-region` recovery. +- **Mark every command block.** Every fenced `bash`/`sh` block in `docs/build/*.md` is preceded by a marker: + - ``: CI runs the block in that directory, after `source ~/export-esp.sh`. + - ``: the block needs a board. CI checks that every `espflash ` in it is a real subcommand, with `espflash --help`. + - An unmarked block fails the check. +- Copy the firmware pins (espup 0.17.1, esp toolchain 1.95.0.0, espflash 4.5.0), the apt packages, the install steps and the rust-cache key from the `firmware` job in `ci.yml`. Pin actions by SHA with a release comment. + +**Never:** +- Don't edit `.github/workflows/ci.yml`, because that reruns the slow Images job. Don't make the docs workflow a required check. +- Don't change firmware behaviour or the Server. +- Don't check external `http(s)` links: they need the network and they flake. +- Don't add a docs site generator. +- Don't edit `sprint-status.yaml`. +- Don't claim a measured sleep current. The measurement is still pending. + +## I/O & Edge-Case Matrix + +| Scenario | Input / State | Expected Output / Behavior | Error Handling | +|----------|--------------|---------------------------|----------------| +| Good docs | every relative link and referenced path exists | `check-links.sh` exits 0 | No error expected | +| Broken link | `[x](../nope.md)` in `docs/build/flash.md` | exits 1 | names the file, line and target | +| Anchor and query | `[x](power.md#budget)`, ``, `mailto:` | only the file part is checked; external links are skipped | No error expected | +| Backtick path | `` `apps/rs/node/src/board/pins.rs` `` | checked to exist | missing → names file:line | +| Placeholder path | `` `coldframe-node-X.Y.Z.bin` ``, `` `` ``, `` `apps/rs/*/` `` | not checked | No error expected | +| Run block | `ci:run apps/rs/node` + `cargo build --release --locked` | `check-commands.sh` runs it there | non-zero status → names the block's file:line | +| Hardware block | `ci:hardware` + `espflash write-bin 0x0 …` | only `espflash write-bin --help` runs | unknown subcommand → fails | +| Unmarked block | a ```` ```bash ```` block with no marker | fails | names file:line | + + + +## Code Map + +- `apps/rs/node/src/board/pins.rs:7-17,22-31,45-54,78-91`: the pin table. + - GPIO1 soil (ADC1); GPIO2 divider midpoint; GPIO4 probe switch; GPIO5 divider switch; GPIO6 CHRG (active low, pull-up); GPIO7 button to GND; GPIO8/9 I²C SDA/SCL; BME680 at 0x77. + - The divider is 100 kΩ/100 kΩ with a midpoint capacitor of at most 10 nF, and it is high-side switched; a low-side NMOS is ruled out. The pads float in deep sleep, so add external pull-downs. +- `apps/rs/node/README.md:140-172,204-238,282-329` and `apps/rs/hub/README.md:62-114,133-221`: existing prose to link to rather than duplicate. It covers buttons, build commands, the eFuse WARNING wording, `erase-region` recipes (Hub cf_setup 0xC000/0x2000, cf_ident 0xE000/0x1000; Node cf_setup 0x12000/0x2000, cf_ident 0xD000/0x1000) and WPA3-only being unsupported. The Hub also needs cmake and ninja. +- `packages/rs/sensing/src/button.rs:21-27`: debounce 50 ms, long press 3 s. +- `packages/rs/sensing/src/wake.rs:35`: wakes every 15 min. +- `packages/rs/setup/src/node.rs:30`: the setup window is 180 s. +- `packages/rs/sensing/src/battery.rs:1-5,32-54`: battery % comes from a 21-point curve and is always approximate. +- `apps/cs/server/Sensors/CalibrationMath.cs`: soil % is two-point linear, rounded to 5 %, with no temperature compensation. +- `docs/bench/node-power-checklist.md`: the measurement method (a meter in series with the battery, ≥ 3 periods, release build, pass at ≤ ~100 µA). Link to it from the power page. Read-only. +- `_bmad-output/planning-artifacts/prds/prd-coldframe-2026-09-25/addendum.md:35-44,119-120` and `_bmad-output/specs/spec-coldframe/device-hardware.md:52-75`: the solar requirements (~1 W panel, no charging below 0 °C or above ~45 °C), the capacitive-probe caveats (NE555 "v1.2" boards, unsealed edges), and the board sleep-current table. +- `docs/operations/release.md:43,140` and `.github/workflows/release.yml:266-273`: the release assets `coldframe-{hub,node}-X.Y.Z.bin`, a merged image written with `espflash write-bin 0x0`, and `SHA256SUMS`. Local equivalent: `espflash save-image --chip esp32s3 --merge --partition-table partitions.csv `. +- `.github/workflows/ci.yml:203-314`: the `firmware` job to copy setup from. Read-only. +- `deploy/release/test.sh:1-40`: the `expect`/`pass`/`failed` test style. +- `README.md:13-14`, `docs/quickstart.md:318-338,467`, `docs/build/apps.md`: add links to the new guide. +- `.gitignore:463-466`: `docs/build/` is already un-ignored, and `hardware/` is not ignored. Check new paths with `git check-ignore`. + +## Tasks & Acceptance + +**Execution:** +- `hardware/README.md`: rewrite it as an index: + - status ("reference design, not yet bench-verified"); + - the Apache-2.0 notice; + - links to every page below and to `docs/build/`. +- `hardware/bom.md`: the Node and Hub BOM, each line with a part, an example part number and the requirement it must meet. Reference choices: + - ESP32-S3-WROOM-1 (N8) module. The Hub uses an ESP32-S3 DevKitC-1 (N8) on USB 5 V. + - A capacitive soil probe, with the NE555 caveat. + - A BME680 breakout with SDO high. + - A 1S LiPo, 800 mAh, with a 10 kΩ NTC. + - A ~1 W 5–6 V solar panel. + - A TP4056 charger with TEMP/NTC and an open-drain `CHRG`; CN3791 as a solar alternative. + - A low-Iq 3.3 V LDO. + - Two high-side load switches. + - The 100 kΩ/100 kΩ divider with a ≤ 10 nF capacitor. + - Pull-downs, the button, and the enclosure parts. +- `hardware/node.md`: the Node schematic, as an ASCII block diagram and per-block connection tables for: + - the power path (panel → charger with NTC → LiPo → LDO → module); + - the switched probe power on GPIO4; + - the switched battery divider on GPIO5 → GPIO2; + - CHRG → GPIO6; + - the button GPIO7 → GND; + - I²C to the BME680; + - the USB/UART flashing header with EN/BOOT. + + Add a wiring table that is a full pin-to-pin list, and link to `pins.rs` as the source of truth. +- `hardware/hub.md`: the Hub schematic and wiring. A dev board on USB 5 V uses no GPIO, so cover power, antenna clearance and placement on the home network. +- `hardware/enclosure.md`: enclosure notes (IP rating, venting for the BME680, cable glands, keeping the panel and antenna clear), probe sealing (conformal coat or epoxy on the electronics and cut edges, heat-shrink, the insertion line), and the NFR-5 outdoor requirement. +- `docs/build/README.md`: the adopter build-guide index. It sets the order (parts → assemble → toolchain → flash Hub → flash Node → add them from the app) and links to `apps.md`. +- `docs/build/flash.md`: + - the eFuse and dev-mode blockquote at the top; + - installing espup and espflash at the pinned versions; + - flashing from release binaries: download, `sha256sum --check --ignore-missing SHA256SUMS`, `espflash write-bin 0x0 …`; + - flashing from source: `cargo build --release --locked`, then `save-image --merge` or `cargo run --release`; + - the first-boot serial output, with the identity line and the setup code (Hub at first boot, Node on a long press); + - recovery with `erase-region`; + - the WPA2 / mixed WPA2/WPA3 requirement; + - short press versus long press. + + Mark every block. +- `docs/build/power.md`: + - panel placement above the canopy, angled to the sun, not shaded by the plant; + - the ~1 W panel and 14 days without sun; + - the sleep-current budget of ≤ ~100 µA, why a stock dev board draws ~1 mA, and how to measure it (link the bench checklist); + - why soil % (two-point calibration, temperature and salinity, settling for 2–3 days) and battery % (curve, worse while charging) are approximate. +- `docs/check-links.sh`: usage `[root…]`; default `docs hardware README.md`. + - For every `.md` file, resolve inline `[text](target)` links that are not external, relative to the file. + - Resolve backtick paths that start with `apps/ packages/ deploy/ docs/ hardware/ tests/ .github/` from the repo root, skipping any that contain `<`, `*`, `{`, `X.Y.Z` or a space. + - Exit 1 and list every failure. +- `docs/check-commands.sh`: usage `[file…]`; default `docs/build/*.md`. It extracts the marked blocks and runs them as described in Always. `--list` prints the blocks without running them. +- `docs/test.sh`: tests the I/O matrix for both scripts against fixtures in `mktemp -d`, in the `deploy/release/test.sh` style. `check-commands.sh` is tested with a stub `espflash` and with blocks that run `true`/`false`. +- `.github/workflows/docs.yml`: + - Triggers: `pull_request` and `push` on main, with paths `docs/**`, `hardware/**`, `README.md`, `apps/rs/{hub,node}/**`, `packages/rs/**` and the workflow itself, plus `workflow_dispatch`. + - `links` job: `docs/test.sh`, then `docs/check-links.sh`. + - `commands` job: firmware setup copied from ci.yml, then `docs/check-commands.sh`. + - Use `permissions: contents: read`. +- `README.md`, `docs/quickstart.md`, `docs/build/apps.md`: link to `docs/build/README.md` and `hardware/README.md`. + +**Acceptance Criteria:** +- Given `hardware/`, when a maker opens it, then they find: + - the Node and Hub schematics; + - a BOM with every listed part (ESP32-S3, capacitive probe, BME680, 800 mAh LiPo, solar panel, a charger with NTC and a status pin); + - the wiring, including the switched probe power and the switched battery divider; + - enclosure notes and probe sealing; + - the Apache-2.0 notice. +- Given `docs/build/`, when a maker follows it, then it shows how to flash both Devices over USB with `espflash`, from release binaries and from source, and where the setup code appears. A blockquote before the first flash command warns that the eFuse burn is irreversible and explains dev mode. +- Given the power guidance, when read, then it covers panel placement above the canopy, the ≤ ~100 µA budget and how to measure it, and why soil % and battery % are approximate. +- Given a pull request touching `docs/build/`, when the `Docs` workflow runs, then it fails on any broken relative link or referenced path. It runs every `ci:run` block against the firmware crates, and only `--help` for `ci:hardware` blocks. + +## Spec Change Log + +## Review Triage Log + +### 2026-10-19 — Review pass +- verdicts: 33 findings — high 0, medium 8, low 22, false 2, maybe-false 1 +- findings: + - `[low]` `[patch]` (blind) `check-commands.sh --help` printed `set -uo pipefail` — the help range now ends at the last header line. + - `[low]` `[reject]` (blind) Anchors and reference-style links are not checked — no doc uses reference-style links, and a broken anchor still lands on the right page; checking anchors needs heading-slug generation, which is more than a direct fix. + - `[medium]` `[patch]` (blind) The docs.yml path filter misses files the docs link to (apps/cs, deploy, tests, …) — both path lists now cover apps/**, packages/**, deploy/**, tests/**, aspire/**, .github/workflows/**, LICENSE and .gitignore. + - `[low]` `[patch]` (blind) power.md listed the 20 µA row as ~1.1 mAh a day (inconsistent with the wake overhead), and the 100 µA budget had no row — corrected to ~1.6 mAh, and a Budget row was added (~3.5 mAh, about 180 days). + - `[medium]` `[patch]` (blind) BOM N3's example, Adafruit 3660, has a regulator and level shifter, which breaks its own requirement — now requires a bare breakout and says such boards do not qualify unless modified. + - `[medium]` `[patch]` (blind) No documented way to connect USB to the Node's header — added BOM N16a and a "Connecting USB" section in node.md (GND/D−/D+, no VBUS), linked from flash.md. + - `[medium]` `[patch]` (blind) No download-mode recovery for a sleeping or crash-looping Node — flash.md now has the BOOT+RESET steps and notes that the enclosure must be opened. + - `[maybe-false]` `[reject]` (blind) The CHRG internal pull-up may draw ~70 µA in deep sleep while charging — whether the pad's pull-up survives deep sleep is unverified. If it does, it only draws while the panel is charging, so it would be low; it is firmware behaviour from Story 4.1 either way. Settled by a bench measurement while charging. + - `[low]` `[reject]` (blind) Deep-discharge and brownout behaviour is not covered — rarely met (the low-battery Alert from Story 7.3 fires first); it needs new content, not a direct fix. + - `[low]` `[reject]` (blind) ci:hardware checks only subcommand names, not flags or other tools — every documented flag is valid today (`espflash write-bin --help` lists `--monitor`); checking flags means parsing per tool, which is more than a direct fix. + - `[low]` `[patch]` (blind) The default roots skipped apps/rs/hub and apps/rs/node READMEs — added; both pass. + - `[low]` `[patch]` (edge) On a fresh machine the install block failed before it could create the export file — the export file is now sourced only if it exists; the test was updated. + - `[low]` `[reject]` (edge) An unclosed fence or a 4-backtick fence swallows the rest of the file — no doc has either; the fix adds fence-length tracking. + - `[low]` `[reject]` (edge) A global option before the espflash subcommand goes unchecked — no doc writes it that way; the fix adds option parsing. + - `[low]` `[reject]` (edge) A missing espflash gives a misleading message — the run still fails; CI always installs espflash first. + - `[low]` `[reject]` (edge) An empty docs/build passes with 0 files — unlikely (the guide is the subject of the check, and docs/test.sh asserts the repository guide's blocks). + - `[low]` `[reject]` (edge) Reference-style links, HTML href and targets with `)` are not checked — none are used in the checked docs; same root cause as the blind anchor row. + - `[low]` `[reject]` (edge) A heading-anchor rename passes silently — same as the blind anchor row. + - `[low]` `[reject]` (edge) Outside a git checkout, the gitignored-path skip fails — CI and contributors use git; adding a work-tree probe is a guard for an unshown case. + - `[low]` `[patch]` (edge) An angle-bracket target containing a space was cut at the space — bracketed targets now keep spaces; two tests were added. + - `[medium]` `[patch]` (edge) Path filter misses referenced paths — same root cause and fix as the blind path-filter row. + - `[low]` `[patch]` (edge) check-commands `--help` extra line — same fix as the first blind row. + - `[false]` `[reject]` (edge) check-links `--help` omits the DOCS_REPO_ROOT line — the header ends at line 18 and `2,18p` prints it; the `set` line is 19. + - `[false]` `[reject]` (edge) The guide's install block re-downloads the toolchain on every run — measured locally: with the toolchain in place, the whole `check-commands.sh` run took 2.4 s, so espup reuses the existing install. + - `[low]` `[patch]` (edge) Inconsistent per-day figures in power.md — same fix as the blind power row. + - `[low]` `[reject]` (edge) The AC claims the workflow fails on any broken link — inline links, code-span paths and bracketed targets are checked; anchors and reference-style links are covered by the blind anchor row. + - `[medium]` `[patch]` (verification-gap) docs.yml's path filter lets link-breaking changes through — same fix as the blind path-filter row. + - `[medium]` `[patch]` (verification-gap) Nothing checks that the documented toolchain pins match ci.yml — docs/test.sh now reads the pins from ci.yml and fails when release.yml, docs.yml or flash.md (prose and install block) differ; ci.yml and release.yml are in the trigger paths. Drift was confirmed to fail the test. + - `[medium]` `[reject]` (intent) Human-owed work (bench build, real flash, sleep-current measurement) is not in operator_actions — the fix is a spec edit; finalization records it under operator_actions with status awaiting-operator. + - `[low]` `[reject]` (intent) No CI check compares the documented pins and constants with the firmware — the current values match (checked by hand against pins.rs, wake.rs, button.rs and partitions.csv); an automated fact comparison is new tooling, not a direct fix. + - `[low]` `[reject]` (intent) The release-binary path is only checked by subcommand name — it needs a real Release and a board; owed as an operator action. + - `[low]` `[reject]` (intent) The workflow has not run on GitHub yet — the PR run is that verification; it is listed under residual risks. + - `[low]` `[reject]` (intent) The spec restates the AC at document level — the fix is a spec edit. + +## Design Notes + +**Why Markdown schematics.** A KiCad project written without an EDA tool or a bench could not be verified in this repo. Connection tables can be reviewed against `pins.rs` and checked by CI. A KiCad capture and PCB can follow once the bench verification is done (an operator action). + +**Why markers.** The AC asks that *documented* commands run. With markers, every block is either executed or explicitly marked as needing hardware, so an unmarked block fails the check instead of going untested. + +## Verification + +**Commands:** +- `docs/test.sh`: expected all pass. +- `docs/check-links.sh`: expected exit 0. +- `docs/check-commands.sh --list`: lists every block with its marker. +- `docs/check-commands.sh`: expected exit 0, with the esp toolchain installed locally. +- `actionlint .github/workflows/docs.yml`, if available: expected no findings. + +**Manual checks (if no CLI):** +- `hardware/node.md` against `pins.rs`: every GPIO and divider value matches. + +## Auto Run Result + +Status: awaiting-operator + +**Summary.** `hardware/` now holds a reference Node and Hub design under Apache-2.0: schematics as ASCII block diagrams and connection tables, a BOM with the requirement each part must meet, a full pin-to-pin wiring list tied to `pins.rs` (including the switched probe power and the high-side switched battery divider), enclosure notes and probe sealing. It is labelled "not yet bench-verified". `docs/build/` gains an adopter build guide: +- `flash.md` opens with the eFuse and dev-mode warning. It covers flashing over USB with `espflash` from release binaries (with a checksum check) or from source, the first-boot output and the setup code (Hub at first boot, Node on a 3 s press), download mode, the buttons, WPA2, and `erase-region` recovery. +- `power.md` covers panel placement, the ≤ ~100 µA budget, how to measure it, and why soil % and battery % are approximate. + +A new `Docs` workflow checks every relative link and referenced repo path. It runs every `ci:run` command block against the firmware crates (build only), checks `ci:hardware` blocks by espflash subcommand, and keeps the documented toolchain pins in step with ci.yml. + +**Files changed** +- `hardware/README.md`, `bom.md`, `node.md`, `hub.md`, `enclosure.md`: the reference design. +- `docs/build/README.md`, `flash.md`, `power.md`: the adopter build guide. `docs/build/apps.md`, `README.md` and `docs/quickstart.md` link to it. +- `docs/check-links.sh`: the link and path resolver. +- `docs/check-commands.sh`: the runner for marked command blocks. +- `docs/test.sh`: 44 fixture tests, including the pin-drift tests. +- `.github/workflows/docs.yml`: the Links and Commands jobs. They are path-filtered and not required; `ci.yml` is untouched. + +**Review findings.** There were 33 findings: high 0, medium 8, low 22, false 2, maybe-false 1. +- **Patched (10 root causes):** 5 medium (the path filter, the BOM N3 example, Node USB connection, download mode, the pin-drift check) and 5 low (`--help` range, power table, default roots, the fresh-machine export, angle-bracket targets). +- **Deferred:** none. +- **Rejected:** each with its reason in the triage log: + - anchors and reference-style links; + - the CHRG pull-up in sleep (maybe-false, firmware); + - deep discharge; + - flag checks in `ci:hardware` blocks; + - unclosed fences; + - a global espflash option before the subcommand; + - a missing espflash; + - an empty `docs/build/`; + - a non-git checkout; + - the AC "any broken link" claim; + - the operator-action bookkeeping (handled here); + - an automated fact comparison with the firmware; + - the release path checked by name only; + - the not-yet-run workflow; + - the spec restatement; + - two false findings (`check-links --help`, toolchain re-download). + +**Follow-up review: recommended (true).** Five medium entries were patched. The risk not yet verified is the widened `docs.yml` triggers: the Commands job now runs on most PRs, and neither job has run on GitHub yet. + +**Verification.** +- `docs/test.sh`: 44 passed. +- `docs/check-links.sh`: 419 links and paths in 25 files resolve. +- `docs/check-commands.sh --list`: 14 blocks, all marked. +- `docs/check-commands.sh`: exit 0 locally with the esp toolchain. It ran the Hub and Node release and dev-mode builds and `save-image --merge`, and checked every espflash subcommand. +- The implementer ran actionlint and shellcheck: clean. Gitleaks on `docs/`, `hardware/` and `.github/`: no leaks. +- `hardware/node.md` was checked by hand against `pins.rs`. + +**Residual risks.** +- The design is a reference design that nobody has built yet. Part numbers and some figures (TPS22917, RT9080-33, the TP4056 input limit and R_PROG table, CN3791) come from memory and have not been checked against datasheets. +- The Node's sleep current has not been measured. +- The documented log lines come from the firmware source, not from a captured boot. +- The `Docs` workflow has not run on GitHub yet. diff --git a/docs/build/README.md b/docs/build/README.md new file mode 100644 index 00000000..ffbcd4af --- /dev/null +++ b/docs/build/README.md @@ -0,0 +1,58 @@ +# Build guide + +How to build your own Coldframe Node and Hub from this repository, flash them, and add them to your +garden. Work through the steps in order. + +The hardware is a reference design that has not been bench-verified yet; see +[`hardware/README.md`](../../hardware/README.md) for its status and license. + +## 1. Get the parts + +[`hardware/bom.md`](../../hardware/bom.md) lists every part for one Node and one Hub, with an +example part and the requirement a substitute must meet. One Hub serves a garden; add as many Nodes +as you like. + +## 2. Assemble + +- **Node:** [`hardware/node.md`](../../hardware/node.md): the schematic, every connection, and the + full pin-to-pin wiring. Then the enclosure and the probe sealing: + [`hardware/enclosure.md`](../../hardware/enclosure.md). +- **Hub:** [`hardware/hub.md`](../../hardware/hub.md): a dev board on USB power. Nothing to wire. +- **Placement and power:** [power.md](power.md): where the panel goes, the sleep-current budget and + how to measure it, and why soil % and battery % are approximate. + +## 3. Install the toolchain + +[flash.md: Toolchain](flash.md#toolchain): `espflash`, and for building from source, `espup` and the +esp Rust toolchain at the versions CI pins. + +## 4. Flash the Hub + +[flash.md](flash.md): from the release binaries or from source. **Read the eFuse warning at the top +first**: the first boot of a build without dev mode burns the chip's identity, and that cannot be +undone. Note the Hub's setup code from its serial output. + +## 5. Flash the Node + +The same page, for the Node. Its setup code appears when you long-press its button (3 s). + +## 6. Add them from the app + +Build the iOS or Android app for your own Server: [apps.md](apps.md). The Server must be running +first (the developer [quickstart](../quickstart.md) runs one locally). Then, in the app: + +1. **Add a Hub.** The app finds the unprovisioned Hub over BLE, asks for its setup code, and sends it + your Wi-Fi network (2.4 GHz, WPA2 or mixed WPA2/WPA3) and your Server's address. +2. **Add a Node.** Long-press the Node's setup button (3 s), pick the Lot it measures, and enter + its setup code within the 180 s setup window. +3. Calibrate the soil probe once it has settled in the soil for 2–3 days + ([why](power.md#soil-moisture)). + +## Checking this guide + +CI checks these pages on every change (`.github/workflows/docs.yml`): + +- `docs/check-links.sh` fails on any broken relative link or referenced repository path. +- `docs/check-commands.sh` runs every command block marked `` against the + firmware crates (builds only, never flashing), and for blocks marked `` checks + only that their `espflash` subcommands exist. A command block without a marker fails. diff --git a/docs/build/apps.md b/docs/build/apps.md index 34526357..f0ee08ed 100644 --- a/docs/build/apps.md +++ b/docs/build/apps.md @@ -4,6 +4,9 @@ Coldframe publishes no mobile store binaries. Each adopter builds the iOS and An source for their own Server, with their own push credentials. This page is a short guide; Story 10.3 expands it. +This page is step 6 of the [build guide](README.md). The Node and Hub hardware is in +[`hardware/`](../../hardware/README.md). + ## What you need - **Your own push credentials.** iOS pushes go through the Apple Push Notification service with diff --git a/docs/build/flash.md b/docs/build/flash.md new file mode 100644 index 00000000..394a287c --- /dev/null +++ b/docs/build/flash.md @@ -0,0 +1,288 @@ +# Flash the Hub and the Node + +> [!WARNING] +> **The first boot of any build without `--features dev-mode`, debug or release, burns an eFuse +> key block, and that cannot be undone.** One of the chip's six key blocks becomes a +> read-protected, write-protected HMAC key with purpose `HMAC_UP`: the Device's identity for life. +> A plain `cargo build` or `cargo run` burns too, and **the release binaries always burn**: they +> are release builds without dev mode. +> +> - Flash such a build only onto a board meant to be a Coldframe Hub or Node, and +> [check its key purposes first](#before-you-flash-check-the-key-purposes). +> - **On a test board, use dev mode:** build from source with `--features dev-mode`. A dev-mode +> build compiles no eFuse code at all and keeps a software identity in the `cf_ident` partition +> instead, which you can erase. Dev mode is for debug builds only: +> `cargo build --release --features dev-mode` refuses to compile. + +Both Devices are flashed over USB with `espflash`. There is no over-the-air update. You can flash +the **release binaries** of a GitHub Release, or build **from source**. Do the Hub first: a Node +needs a Hub to send to. + +On Linux, add yourself to the group that owns the serial port (`dialout` or `uucp`) and log in +again, or `espflash` cannot open the board. + +## Toolchain + +You need the `esp` Rust toolchain from `espup`, and `espflash`. These are the versions CI pins: +espup 0.17.1, the esp toolchain 1.95.0.0 and espflash 4.5.0. First install Rust with +[rustup](https://rustup.rs), and the system packages: + +- To build `espflash` on Linux: `libudev-dev` and `pkg-config` (Debian and Ubuntu: + `sudo apt install libudev-dev pkg-config`). +- To build the **Hub** from source: **cmake and ninja** (Debian and Ubuntu: + `sudo apt install cmake ninja-build`, or `uv tool install cmake ninja`). The Hub rebuilds the + mbedTLS C sources. The Node needs neither. + +Then install the tools at the pinned versions: + + +```bash +cargo install espup --version 0.17.1 --locked +espup install --toolchain-version 1.95.0.0 --targets esp32s3 +cargo install espflash --version 4.5.0 --locked +``` + +`espup install` writes `~/export-esp.sh`. Source it in **every** shell before you build, and check +the versions: + + +```bash +source ~/export-esp.sh +espup --version +espflash --version +rustup run esp rustc --version +``` + +To flash only the release binaries you need just `espflash`, not the esp toolchain. + +## Before you flash: check the key purposes + +A board can become a Hub or a Node only if one of its eFuse key blocks is still unused. A board that +already has an `HMAC_UP` key keeps that identity (it reports `existing KEYn`). `espflash 4.5.0` +does not show key purposes, so use `espefuse` from esptool (`pipx install esptool`): + + +```bash +espflash list-ports +espflash board-info +espefuse --port summary +``` + +In the `espefuse` output, look at `KEY_PURPOSE_0` to `KEY_PURPOSE_5`: at least one must be `USER` +and unused. The full procedure is Part A of the +[Hub identity checklist](../bench/hub-identity-checklist.md), which applies to the Node unchanged. + +## Flash from the release binaries + +Every GitHub Release carries a merged flash image per Device (bootloader, partition table and +firmware in one file, written at offset 0) and a `SHA256SUMS` file +([what a release holds](../operations/release.md)). Download `coldframe-hub-X.Y.Z.bin`, +`coldframe-node-X.Y.Z.bin` and `SHA256SUMS` from the Release page, or with the GitHub CLI, then +check them and flash. Replace `X.Y.Z` with the version, and use the same version as your Server. + + +```bash +gh release download vX.Y.Z --repo escendit/coldframe --pattern 'coldframe-*.bin' --pattern SHA256SUMS +sha256sum --check --ignore-missing SHA256SUMS +espflash write-bin --monitor 0x0 coldframe-hub-X.Y.Z.bin +``` + +`sha256sum` must print `OK` for both images. Then connect the Node's flashing header to USB +([Connecting USB](../../hardware/node.md#connecting-usb)) and flash its image the same way: + + +```bash +espflash write-bin --monitor 0x0 coldframe-node-X.Y.Z.bin +``` + +If no port appears for the Node, put it in download mode. The Node sleeps almost all the time, and +its USB Serial/JTAG port disappears while it sleeps, so a sleeping or crash-looping Node may never +show a port. Hold `BOOT` (GPIO0), press and release `RESET` (EN), release `BOOT`, then flash. The +chip stays in the bootloader until the flash finishes and resets it. The buttons are on the board, +inside the enclosure ([BOM N14](../../hardware/bom.md#node)), so a deployed Node has to be opened. + +`--monitor` keeps the serial output open after flashing, so you see the +[first boot](#first-boot-and-the-setup-code). Press `Ctrl+C` to leave it. + +## Flash from source + +Build from the release tag you run (`git checkout vX.Y.Z`), or from `main` for development. Every +crate has a committed `Cargo.lock`; `--locked` keeps to it. + +### Hub + +Build the release firmware and turn it into the same merged image a Release carries: + + +```bash +source ~/export-esp.sh +cargo build --release --locked +espflash save-image --chip esp32s3 --merge --partition-table partitions.csv \ + target/xtensa-esp32s3-none-elf/release/coldframe-hub target/coldframe-hub.bin +``` + +Flash the image, or let `cargo run` flash the ELF with the partition table and open the monitor +(the runner is `espflash flash --monitor --partition-table partitions.csv`): + + +```bash +espflash write-bin --monitor 0x0 target/coldframe-hub.bin +# or, the same in one step: +cargo run --release +``` + +### Node + +Connect the Node over its flashing header ([Connecting USB](../../hardware/node.md#connecting-usb)). + + +```bash +source ~/export-esp.sh +cargo build --release --locked +espflash save-image --chip esp32s3 --merge --partition-table partitions.csv \ + target/xtensa-esp32s3-none-elf/release/coldframe-node target/coldframe-node.bin +``` + + +```bash +espflash write-bin --monitor 0x0 target/coldframe-node.bin +# or: +cargo run --release +``` + +### Dev mode, for test boards + +A dev-mode build burns nothing. It is a debug build, keeps its identity root in the `cf_ident` +partition, and the Node also logs its Reading values. Build both: + + +```bash +source ~/export-esp.sh +cargo build --features dev-mode --locked +``` + + +```bash +source ~/export-esp.sh +cargo build --features dev-mode --locked +``` + +and flash with `cargo run` from the crate's folder: + + +```bash +cargo run --features dev-mode +``` + +Never flash a build without `--features dev-mode` onto a board you want to keep as a test board. +A dev-mode Device is for the bench, not the garden: its identity is not bound to the chip. + +## First boot and the setup code + +Watch the serial output with `espflash monitor`. It resets the board when it connects, so you see +the boot from the start. + + +```bash +espflash monitor +``` + +The **Hub** logs its version and identity, then, while it is unprovisioned, prints its setup code +and advertises: + +```text +coldframe-hub X.Y.Z starting, identity mode=efuse +identity mode=efuse source=burned KEY0 device_id=<16 hex> +setup code=XXXXXXXX +ble advertising setup service as Coldframe Hub XXXX +``` + +- `source=burned KEYn` appears once, on the first boot: the key was burned just now. Every later + boot says `source=existing KEYn` with the same Device ID. A dev-mode build says + `identity mode=dev source=generated` and then `source=stored`. +- `setup code=XXXXXXXX` is printed **on every boot while the Hub is unprovisioned**, which includes + its first boot. Once the app has set it up, it is never shown again. To see it again before setup, + press the board's `RST` button with the monitor open. + +The **Node** logs its identity on every wake, then measures and goes to sleep: + +```text +coldframe-node X.Y.Z wake cause=cold_boot reset=… identity mode=efuse +identity mode=efuse source=burned KEY0 device_id=<16 hex> +boot id=1 +wake plan=measure +``` + +The Node prints its setup code **only when it enters setup mode**, on a **long press (3 s)** of the +setup button. Keep the monitor open: the USB serial link drops while the Node sleeps and +`espflash monitor` reconnects when it wakes. Then: + +```text +coldframe-node X.Y.Z wake cause=button reset=… identity mode=efuse +identity mode=efuse source=existing KEY0 device_id=<16 hex> +boot id=1 +wake plan=setup +setup code=XXXXXXXX +setup mode window_ms=180000 +``` + +The code is drawn on the first long press and stored; every later long press prints the same code. +If you missed the line, long-press again. + +**Write each code on a sticker on its Device**, never in the repository or a chat. The app asks for +it when you add the Device: it proves that you hold the Device. Every log line is explained in the +[Hub README](../../apps/rs/hub/README.md#log-lines) and the +[Node README](../../apps/rs/node/README.md#log-lines). + +## The buttons + +The Hub has no button of its own: an unprovisioned Hub is always ready for setup. + +The Node has one setup button: + +| Press | What it does | +| --- | --- | +| **Short** (50 ms to under 3 s) | Takes and sends one Reading now, then sleeps a full period | +| **Long** (held 3 s) | Opens the BLE setup window for 180 s and prints the setup code; then measures once and sleeps | + +A press shorter than 50 ms is a bounce and does nothing. BLE runs only in the setup window; the +Node never advertises otherwise. + +## Wi-Fi + +The Hub joins a **2.4 GHz WPA2-Personal** network, or a **mixed WPA2/WPA3** (transition mode) +network over WPA2. **WPA3-only networks are not supported**: the app lists them, and the Hub refuses +them with "unsupported security". If your router runs WPA3 only, switch it to WPA2/WPA3 mixed mode +for the 2.4 GHz band, or add a WPA2 network for the Hub. The Node never uses Wi-Fi: it talks to the +Hub over ESP-NOW. + +## Recovery: erase a region + +`espflash erase-region ` erases one partition. It never undoes an eFuse burn: a +release-build Device keeps its identity whatever you erase. + +| Device | Partition | Command | Effect | +| --- | --- | --- | --- | +| Hub | `cf_setup` | `espflash erase-region 0xC000 0x2000` | A new setup code on the next boot, and the Hub is unprovisioned: set it up again from the app | +| Hub | `cf_ident` | `espflash erase-region 0xE000 0x1000` | Dev mode only: a new identity on the next boot | +| Node | `cf_setup` | `espflash erase-region 0x12000 0x2000` | A new setup code on the next long press | +| Node | `cf_ident` | `espflash erase-region 0xD000 0x1000` | Dev mode only: a new identity on the next boot | + +For example, to give an unprovisioned Hub a new setup code: + + +```bash +espflash erase-region 0xC000 0x2000 +espflash monitor +``` + +The offsets come from each crate's `partitions.csv` +([Hub](../../apps/rs/hub/partitions.csv), [Node](../../apps/rs/node/partitions.csv)). + +> [!CAUTION] +> Do not run `espflash erase-flash` on a Node that has ever sent a Reading. It also erases the frame +> counter, and the Server then refuses the Node's frames as replays. The Node's other partitions, +> and when erasing them is safe, are in the +> [Node README](../../apps/rs/node/README.md#partitions). + +Next: [add the Hub and the Node from the app](README.md#6-add-them-from-the-app). diff --git a/docs/build/power.md b/docs/build/power.md new file mode 100644 index 00000000..13bd6419 --- /dev/null +++ b/docs/build/power.md @@ -0,0 +1,105 @@ +# Power and placement + +The Node runs on an 800 mAh LiPo cell charged by a small solar panel. This page covers where to put +the panel, the sleep-current budget the Node is designed for, and why the soil and battery +percentages are approximate. The parts are in [`hardware/bom.md`](../../hardware/bom.md). + +## Panel placement + +- **Above the canopy.** Plants grow over the panel during summer, and shading is the real risk, not + cloud. Mount the panel on a stake higher than the plants will grow by the end of the season. +- **Angled to the sun.** Face it south in the northern hemisphere (north in the southern), tilted + at about your latitude. A steeper tilt sheds rain and dirt better. +- **Not shaded by the plant**, a fence or the house at any time of day. Even a partly shaded panel + delivers little. +- Keep it clean, and keep the Node's antenna end free of the panel's metal frame + ([enclosure.md](../../hardware/enclosure.md#node-enclosure)). + +A shaded panel shows up in the app as falling battery % while the Node reports *not charging*, +until the low-battery Alert fires. If you see that in midsummer, look for leaves on the panel. + +## Solar and autonomy + +- **About 1 W** is enough. A ~1 W panel yields roughly 500 mAh a day into the cell in full sun, and + 25–50 mAh a day in heavy overcast. Both are far above what a well-built Node uses. +- So the question is not runtime but **autonomy**: how many days without sun the cell bridges. The + design target is **at least 14 days without sun**, on top of a season on solar. +- The charger stops charging below 0 °C and above about 45 °C, through the cell's NTC. In spring + frost or a sun-heated box the Node runs on the cell alone for a while; that is expected. + +## Sleep-current budget + +The Node wakes every 15 minutes (`WAKE_PERIOD_MS` in +[`packages/rs/sensing/src/wake.rs`](../../packages/rs/sensing/src/wake.rs)), measures for a fraction +of a second, sends over ESP-NOW and sleeps again. Between wakes it draws only its sleep current, so +the sleep current decides the autonomy. + +**The budget is an average sleep current of at most about 100 µA**, with short wakes (about +0.3–0.5 s). That covers a 6-month season even with no sun at all (about 3.5 mAh a day from about +640 mAh usable), and far more than 14 dark days on solar. + +| Board | Sleep current | Use per day | Days without sun | +| --- | --- | --- | --- | +| Optimized custom board | ~20 µA | ~1.6 mAh | about a year (self-discharge limits it) | +| Reasonable custom board | ~50 µA | ~2.5 mAh | about 250 | +| Budget | ~100 µA | ~3.5 mAh | about 180 | +| Stock dev board | ~1 mA | ~25 mAh | about 25 | + +These are estimates from the design, **not measurements**. The sleep current of the reference +design has **not been measured yet**. + +**Why a stock dev board draws about 1 mA asleep:** its USB-UART bridge, its power LED and its +always-on regulator draw current all the time, whatever the ESP32-S3 does. The chip itself sleeps +at a few microamps. A dev board works for a Node on the bench and bridges about 25 dark days on +solar, but it misses the budget by design. The [reference Node](../../hardware/node.md) avoids all +three: a bare module, no USB-UART bridge (it flashes over the chip's own USB), no LED, and a +low-quiescent LDO. The probe and the battery divider are switched off while asleep, and the BME680 +sleeps between forced-mode measurements. + +### Measure it + +Follow [`docs/bench/node-power-checklist.md`](../bench/node-power-checklist.md). In short: + +- Flash a **release build** (a dev-mode build logs more and is not representative). +- Put a power profiler or a µA meter **in series with the battery**, not on USB. +- Record **at least three full periods** (45 minutes or more), so the capture holds at least three + wakes. +- Read the average current while asleep and the average over whole periods. **Pass: about + 100 µA or less.** +- Check that the probe and divider switches are off while asleep; if either draws current, check + the pull-downs on their enable pins. + +## Why the readings are approximate + +### Soil moisture + +Soil moisture % is an estimate, not a measurement of water content: + +- **Two-point calibration.** The Server maps the probe's raw count linearly between two points you + record in the app, a dry one (0 %) and a wet one (100 %), clamps it to 0–100 and rounds it to the + nearest 5 % ([`CalibrationMath.cs`](../../apps/cs/server/Sensors/CalibrationMath.cs)). Real + probes are not linear between the two points. +- **Temperature and salinity** shift the raw reading. There is no temperature compensation in V1. + The Node records the air temperature with every Reading, so the effect can be analysed later. +- **Settling.** A freshly inserted probe needs **2–3 days** to settle before its readings are + stable. Calibrate it in the soil it will stay in, after it has settled. +- **Drift.** Water creeping into an unsealed probe makes it drift over the season. Seal it + ([enclosure.md](../../hardware/enclosure.md#sealing-the-soil-probe)). + +Use the Thresholds as "drier than usual" and "wetter than usual" for that one probe, not as +absolute values that compare across Nodes. + +### Battery + +Battery % is approximate too: + +- It comes from the cell voltage, read through the switched divider, mapped with a typical + single-cell LiPo **discharge curve** of 21 points, linear between them + ([`battery.rs`](../../packages/rs/sensing/src/battery.rs)). Every cell, and every cell's age and + temperature, gives a slightly different curve. +- The curve is flat in the middle, so a few millivolts of ADC error move the percentage by several + points. +- **It is worse while charging:** the charger lifts the cell voltage, so the percentage reads high + while the panel is in sun, and drops back once charging stops. + +Watch the trend over days, and the *charging* / *not charging* status, rather than a single value. diff --git a/docs/check-commands.sh b/docs/check-commands.sh new file mode 100755 index 00000000..09175422 --- /dev/null +++ b/docs/check-commands.sh @@ -0,0 +1,201 @@ +#!/usr/bin/env bash +# Runs the commands the build guide documents (Story 10.2). Firmware builds only, never flashing. +# +# docs/check-commands.sh [--list] [file...] +# +# The default files are docs/build/*.md, relative to the repository root unless absolute. Every +# fenced bash/sh block must be preceded by a marker (blank lines between them are allowed): +# +# the block runs in that directory, after +# `source ~/export-esp.sh` when it exists (the guide's +# install block creates it), with errexit and pipefail; +# the block needs a board: only `espflash --help` +# runs, for every espflash subcommand in it. +# +# An unmarked block, a marker that no bash/sh block follows, an unknown ci: marker or a ci:run +# directory that does not exist fails. --list prints every block with its marker and runs nothing. +# +# Exit 0 when every block passes, 1 naming each failing block as file:line, 2 on bad usage. +# DOCS_REPO_ROOT overrides the repository root and ESP_EXPORT the export file (the tests use both). +set -uo pipefail + +here=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +repo=${DOCS_REPO_ROOT:-$(cd "${here}/.." && pwd)} +esp_export=${ESP_EXPORT:-${HOME}/export-esp.sh} + +list=false +files=() +for arg in "$@"; do + case ${arg} in + --list) list=true ;; + -h | --help) + sed -n '2,19p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//' + exit 0 + ;; + -*) + echo "usage: check-commands.sh [--list] [file...]: unknown option '${arg}'" >&2 + exit 2 + ;; + /*) files+=("${arg}") ;; + *) files+=("${repo}/${arg}") ;; + esac +done +if [[ ${#files[@]} -eq 0 ]]; then + for file in "${repo}"/docs/build/*.md; do + [[ -f ${file} ]] && files+=("${file}") + done +fi +for file in "${files[@]}"; do + if [[ ! -f ${file} ]]; then + echo "usage: check-commands.sh [--list] [file...]: '${file}' is no file" >&2 + exit 2 + fi +done + +work=$(mktemp -d) +trap 'rm -rf "${work}"' EXIT + +# Splits a Markdown file into ${work}/.sh blocks and prints one manifest line per block or +# problem: " ", separated by +# the unit separator 0x1F (a tab would collapse an empty argument). +# Plain POSIX awk: the CI runner's awk is mawk. +extract() { + awk -v work="${work}" -v prefix="$2" ' + function emit(kind, arg, path) { print FNR "\037" kind "\037" arg "\037" path } + function trim(s) { sub(/^[ \t]+/, "", s); sub(/[ \t]+$/, "", s); return s } + infence { + if ($0 ~ /^[ \t]*(```|~~~)[ \t]*$/) { infence = 0; if (out != "") close(out); out = ""; next } + if (out != "") print > out + next + } + /^[ \t]*[ \t]*$/, "", m) + if (m == "ci:hardware") { pending = "hardware"; arg = "" } + else if (m ~ /^ci:run[ \t]+[^ \t]+$/) { pending = "run"; arg = trim(substr(m, 7)) } + else { emit("unknown", m, ""); pending = "" } + next + } + /^[ \t]*(```|~~~)/ { + info = $0 + sub(/^[ \t]*(```|~~~)/, "", info) + info = trim(info) + lang = info + sub(/[ \t{].*$/, "", lang) + infence = 1 + out = "" + if (lang == "bash" || lang == "sh" || lang == "shell") { + blocks++ + out = work "/" prefix "-" blocks ".sh" + printf "" > out + if (pending == "") emit("unmarked", "", out) + else emit(pending, arg, out) + } else if (pending != "") { + emit("orphan", pending, "") + } + pending = "" + next + } + /^[ \t]*$/ { next } + { + if (pending != "") emit("orphan", pending, "") + pending = "" + } + END { if (pending != "") emit("orphan", pending, "") } + ' "$1" +} + +failures=0 +blocks=0 +fail() { + failures=$((failures + 1)) + echo "FAIL $1" >&2 +} + +indent() { sed 's/^/ /' "$1"; } + +# The espflash subcommands a block calls, one per line, comment lines left out. +espflash_subcommands() { + grep -v '^[[:space:]]*#' "$1" | + grep -oE '(^|[^[:alnum:]_./-])espflash[[:space:]]+[a-z][a-z0-9-]*' | + awk '{ print $NF }' | sort -u +} + +run_block() { # + local where=$1 dir=$2 block=$3 + echo "==> ${where}: ci:run ${dir}" + indent "${block}" + { + if [[ -f ${esp_export} ]]; then + printf 'source %q\n' "${esp_export}" + fi + printf 'set -euo pipefail\n' + cat "${block}" + } > "${block}.run" + if ! (cd "${repo}/${dir}" && bash "${block}.run" < /dev/null); then + fail "${where}: the ci:run ${dir} block failed" + fi +} + +check_hardware() { # + local where=$1 block=$2 sub any=false + echo "==> ${where}: ci:hardware" + indent "${block}" + while IFS= read -r sub; do + any=true + if espflash "${sub}" --help > /dev/null 2>&1; then + echo " ok: espflash ${sub} is a subcommand" + else + fail "${where}: 'espflash ${sub}' is not an espflash subcommand" + fi + done < <(espflash_subcommands "${block}") + if [[ ${any} == false ]]; then + echo " (no espflash command to check)" + fi +} + +index=0 +for file in "${files[@]}"; do + index=$((index + 1)) + rel=${file#"${repo}"/} + while IFS=$'\x1f' read -r line kind arg block; do + where="${rel}:${line}" + case ${kind} in + unmarked) + blocks=$((blocks + 1)) + fail "${where}: unmarked bash/sh block; precede it with or " + ${list} && indent "${block}" + ;; + orphan) fail "${where}: a ci:${arg} marker is not followed by a bash/sh block" ;; + unknown) fail "${where}: unknown marker '${arg}'" ;; + run) + blocks=$((blocks + 1)) + if [[ ! -d ${repo}/${arg} ]]; then + fail "${where}: ci:run directory '${arg}' does not exist" + elif ${list}; then + echo "${where}: ci:run ${arg}" + indent "${block}" + else + run_block "${where}" "${arg}" "${block}" + fi + ;; + hardware) + blocks=$((blocks + 1)) + if ${list}; then + echo "${where}: ci:hardware" + indent "${block}" + else + check_hardware "${where}" "${block}" + fi + ;; + esac + done < <(extract "${file}" "${index}") +done + +if [[ ${failures} -gt 0 ]]; then + echo "check-commands: ${failures} failed (${blocks} blocks in ${#files[@]} files)" >&2 + exit 1 +fi +echo "check-commands: ${blocks} blocks in ${#files[@]} files pass" diff --git a/docs/check-links.sh b/docs/check-links.sh new file mode 100755 index 00000000..e3738d24 --- /dev/null +++ b/docs/check-links.sh @@ -0,0 +1,135 @@ +#!/usr/bin/env bash +# Checks the Markdown docs (Story 10.2): every relative link and every referenced repository path +# must exist. External links (http, https, mailto, any scheme) are never fetched. +# +# docs/check-links.sh [root...] +# +# Each root is a directory (every *.md below it) or a Markdown file, relative to the repository +# root unless absolute. The default is `docs hardware README.md apps/rs/hub/README.md +# apps/rs/node/README.md` (the build guide sends readers to the firmware READMEs). Outside fenced +# code blocks, two things are checked: +# +# - inline links `[text](target)`: the file part of the target (no #anchor, no ?query), relative +# to the Markdown file, or to the repository root when it starts with `/`; +# - code spans that start with apps/ packages/ deploy/ docs/ hardware/ tests/ .github/, from the +# repository root. A span holding `<`, `*`, `{`, `X.Y.Z`, `...` or a space is a placeholder and +# is skipped; a trailing `:` or `:-` is dropped. A missing path that git +# ignores is a local file the reader creates (an `.env`, a `*.local.xcconfig`) and passes. +# +# Exit 0 when everything resolves, 1 listing every failure as file:line, 2 on bad usage. +# DOCS_REPO_ROOT overrides the repository root (the tests use it). +set -euo pipefail + +here=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +repo=${DOCS_REPO_ROOT:-$(cd "${here}/.." && pwd)} + +if [[ ${1:-} == -h || ${1:-} == --help ]]; then + sed -n '2,20p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//' + exit 0 +fi + +roots=("$@") +if [[ ${#roots[@]} -eq 0 ]]; then + roots=(docs hardware README.md apps/rs/hub/README.md apps/rs/node/README.md) +fi + +files=() +for root in "${roots[@]}"; do + if [[ ${root} == /* ]]; then path=${root}; else path=${repo}/${root}; fi + if [[ -d ${path} ]]; then + while IFS= read -r -d '' file; do + files+=("${file}") + done < <(find "${path}" -type f -name '*.md' -print0 | sort -z) + elif [[ -f ${path} ]]; then + files+=("${path}") + else + echo "usage: check-links.sh [root...]: '${root}' is no file or directory under ${repo}" >&2 + exit 2 + fi +done + +# Prints "\t\t" for every inline link and code span outside fences. +extract() { + awk ' + /^[[:space:]]*(```|~~~)/ { fenced = !fenced; next } + fenced { next } + { + line = $0 + rest = line + while (match(rest, /`[^`]+`/)) { + print FNR "\tcode\t" substr(rest, RSTART + 1, RLENGTH - 2) + rest = substr(rest, RSTART + RLENGTH) + } + gsub(/`[^`]+`/, "", line) + rest = line + while (match(rest, /\]\([^)]*\)/)) { + print FNR "\tlink\t" substr(rest, RSTART + 2, RLENGTH - 3) + rest = substr(rest, RSTART + RLENGTH) + } + } + ' "$1" +} + +failures=0 +checked=0 +fail() { + failures=$((failures + 1)) + echo "$1" >&2 +} + +for file in "${files[@]}"; do + rel=${file#"${repo}"/} + dir=$(dirname "${file}") + while IFS=$'\t' read -r line kind target; do + case ${kind} in + link) + # `` keeps its spaces; otherwise drop an optional "title". + if [[ ${target} == '<'* ]]; then + target=${target#<} + target=${target%%>*} + else + target=${target%% *} + fi + # External: any URI scheme (https:, mailto:, ...). + if [[ ${target} =~ ^[A-Za-z][A-Za-z0-9+.-]*: ]]; then + continue + fi + target=${target%%#*} + target=${target%%\?*} + [[ -z ${target} ]] && continue + if [[ ${target} == /* ]]; then + resolved=${repo}${target} + else + resolved=${dir}/${target} + fi + checked=$((checked + 1)) + if [[ ! -e ${resolved} ]]; then + fail "${rel}:${line}: broken link '${target}'" + fi + ;; + code) + case ${target} in + apps/* | packages/* | deploy/* | docs/* | hardware/* | tests/* | .github/*) ;; + *) continue ;; + esac + case ${target} in + *'<'* | *'*'* | *'{'* | *X.Y.Z* | *...* | *' '*) continue ;; + esac + target=${target%#*} + if [[ ${target} =~ ^(.*):[0-9]+(-[0-9]+)?(,[0-9]+(-[0-9]+)?)*$ ]]; then + target=${BASH_REMATCH[1]} + fi + checked=$((checked + 1)) + if [[ ! -e ${repo}/${target} ]] && ! git -C "${repo}" check-ignore --quiet --no-index -- "${target}" 2>/dev/null; then + fail "${rel}:${line}: missing path '${target}'" + fi + ;; + esac + done < <(extract "${file}") +done + +if [[ ${failures} -gt 0 ]]; then + echo "check-links: ${failures} broken of ${checked} in ${#files[@]} files" >&2 + exit 1 +fi +echo "check-links: ${checked} links and paths in ${#files[@]} files resolve" diff --git a/docs/quickstart.md b/docs/quickstart.md index 8be72dbc..ad6c82c6 100644 --- a/docs/quickstart.md +++ b/docs/quickstart.md @@ -337,6 +337,9 @@ hardware. The Hub's on-device behaviour is checked by hand with [`docs/bench/hub-setup-checklist.md`](bench/hub-setup-checklist.md) and [`docs/bench/hub-uplink-checklist.md`](bench/hub-uplink-checklist.md). +To build a real Node and Hub, flash them and add them from the app, follow the adopter +[build guide](build/README.md) and the [hardware design](../hardware/README.md). + ### TypeScript ```sh @@ -464,7 +467,7 @@ packages/ what the runtimes reference: contracts, shared libraries, generated tests/ all tests, by language: cs/ rs/ kt/ swift/ ts/ aspire/ the local stack and integration-test host deploy/ Helm charts, Fleet bundles, Compose example -hardware/ schematics, PCB, enclosure +hardware/ reference design: schematics, BOM, wiring, enclosure docs/ guides and references ``` diff --git a/docs/test.sh b/docs/test.sh new file mode 100755 index 00000000..99520d7a --- /dev/null +++ b/docs/test.sh @@ -0,0 +1,351 @@ +#!/usr/bin/env bash +# Plain-bash tests for the doc checks (Story 10.2): check-links.sh and check-commands.sh against +# throwaway fixtures, with a stub espflash. Needs only bash, awk and coreutils. Run from anywhere: +# +# docs/test.sh +# The fixtures are Markdown in single quotes, backticks and all. +# shellcheck disable=SC2016 +set -uo pipefail + +here=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +work=$(mktemp -d) +trap 'rm -rf "${work}"' EXIT +failures=0 +passes=0 + +pass() { + passes=$((passes + 1)) + echo "ok - $1" +} + +failed() { + failures=$((failures + 1)) + echo "FAIL - $1" >&2 + shift + for line in "$@"; do + echo " ${line}" >&2 + done +} + +# expect -- +# The expected stdout and stderr are substrings, "" for none at all, or - for anything. +expect() { + local name=$1 want_status=$2 want_out=$3 want_err=$4 + shift 5 + local out err status + err=$(mktemp) + out=$("$@" 2>"${err}") + status=$? + local err_text + err_text=$(cat "${err}") + rm -f "${err}" + + if [[ ${status} -ne ${want_status} ]]; then + failed "${name}" "exit ${status}, expected ${want_status}" "stdout: ${out}" "stderr: ${err_text}" + elif [[ -z ${want_out} && -n ${out} ]]; then + failed "${name}" "stdout '${out}', expected none" + elif [[ -n ${want_out} && ${want_out} != "-" && ${out} != *"${want_out}"* ]]; then + failed "${name}" "stdout '${out}' does not contain '${want_out}'" + elif [[ -z ${want_err} && -n ${err_text} ]]; then + failed "${name}" "stderr '${err_text}', expected none" + elif [[ -n ${want_err} && ${want_err} != "-" && ${err_text} != *"${want_err}"* ]]; then + failed "${name}" "stderr '${err_text}' does not contain '${want_err}'" + else + pass "${name}" + fi +} + +links=${here}/check-links.sh +commands=${here}/check-commands.sh + +# A fresh fixture repository: -> its path. It holds docs/build/{flash,power}.md stubs and +# apps/rs/node/src/board/pins.rs, so the good links have something to resolve to. +fixture() { + local root=${work}/$1 + mkdir -p "${root}/docs/build" "${root}/apps/rs/node/src/board" + : > "${root}/apps/rs/node/src/board/pins.rs" + : > "${root}/apps/rs/node/Cargo.toml" + printf '# Power\n' > "${root}/docs/build/power.md" + printf '# Flash\n' > "${root}/docs/build/flash.md" + echo "${root}" +} + +# page : writes a Markdown page. +page() { + mkdir -p "$(dirname "$1/$2")" + printf '%s\n' "$3" > "$1/$2" +} + +echo "# check-links.sh" +good=$(fixture good) +page "${good}" docs/build/flash.md '# Flash + +See [power](power.md) and [the pins](../../apps/rs/node/src/board/pins.rs).' +expect "good docs pass" 0 "resolve" "" -- env DOCS_REPO_ROOT="${good}" "${links}" docs + +broken=$(fixture broken) +page "${broken}" docs/build/flash.md '# Flash + +A [broken link](../nope.md).' +expect "a broken link fails naming file, line and target" 1 "" "docs/build/flash.md:3: broken link '../nope.md'" \ + -- env DOCS_REPO_ROOT="${broken}" "${links}" docs + +anchors=$(fixture anchors) +page "${anchors}" docs/build/flash.md '# Flash + +[budget](power.md#budget), [query](power.md?plain=1), [here](#flash), , +[external](https://example.invalid/nope.md), [mail](mailto:someone@example.invalid), +[titled](power.md "Power"), [angled]().' +expect "anchors and queries are dropped, external links skipped" 0 "resolve" "" \ + -- env DOCS_REPO_ROOT="${anchors}" "${links}" docs + +spaced=$(fixture spaced) +page "${spaced}" docs/build/my\ file.md '# Mine' +page "${spaced}" docs/build/flash.md '[mine]() [titled]( "Mine") [gone]()' +expect "an angle-bracket target keeps its spaces" 1 "" "docs/build/flash.md:1: broken link 'no file.md'" \ + -- env DOCS_REPO_ROOT="${spaced}" "${links}" docs +expect "only the missing angle-bracket target fails" 1 "" "check-links: 1 broken" \ + -- env DOCS_REPO_ROOT="${spaced}" "${links}" docs + +anchor_missing=$(fixture anchor-missing) +page "${anchor_missing}" docs/build/flash.md '[gone](gone.md#budget)' +expect "the file part of an anchored link is checked" 1 "" "docs/build/flash.md:1: broken link 'gone.md'" \ + -- env DOCS_REPO_ROOT="${anchor_missing}" "${links}" docs + +backtick=$(fixture backtick) +page "${backtick}" docs/build/flash.md '# Flash + +The pins are in `apps/rs/node/src/board/pins.rs` and `apps/rs/node/src/board/pins.rs:7-17`.' +expect "an existing backtick path passes, with a line suffix" 0 "resolve" "" \ + -- env DOCS_REPO_ROOT="${backtick}" "${links}" docs + +backtick_missing=$(fixture backtick-missing) +page "${backtick_missing}" docs/build/flash.md '# Flash + +The pins are in `apps/rs/node/src/board/gone.rs`.' +expect "a missing backtick path fails naming file and line" 1 "" \ + "docs/build/flash.md:3: missing path 'apps/rs/node/src/board/gone.rs'" \ + -- env DOCS_REPO_ROOT="${backtick_missing}" "${links}" docs + +placeholders=$(fixture placeholders) +page "${placeholders}" docs/build/flash.md '# Flash + +`docs/coldframe-node-X.Y.Z.bin`, `apps/`, `apps/rs/*/`, `apps/rs/{hub,node}/`, +`apps/rs/node/src/.../x`, `apps/rs/node and more`, `cargo build`, `src/main.rs`.' +expect "placeholder and non-repository spans are skipped" 0 "resolve" "" \ + -- env DOCS_REPO_ROOT="${placeholders}" "${links}" docs + +fenced=$(fixture fenced) +page "${fenced}" docs/build/flash.md '# Flash + +```text +[not a link](nope.md) `apps/nope` +```' +expect "fenced blocks are not checked" 0 "resolve" "" -- env DOCS_REPO_ROOT="${fenced}" "${links}" docs + +rooted=$(fixture rooted) +page "${rooted}" README.md '[pins](/apps/rs/node/src/board/pins.rs) [gone](/apps/gone.md)' +expect "a root-relative link resolves from the repository root" 1 "" "README.md:1: broken link '/apps/gone.md'" \ + -- env DOCS_REPO_ROOT="${rooted}" "${links}" README.md + +several=$(fixture several) +page "${several}" docs/build/flash.md '[a](a.md) +[b](b.md) `docs/c.md`' +expect "every failure is listed" 1 "" "check-links: 3 broken" -- env DOCS_REPO_ROOT="${several}" "${links}" docs + +expect "a missing root is a usage error" 2 "" "usage:" -- env DOCS_REPO_ROOT="${good}" "${links}" nope +expect "the repository docs pass" 0 "resolve" "" -- "${links}" + +echo "# check-commands.sh" +# A stub espflash: logs its arguments, knows the real subcommand names, fails on any other. +stubs=${work}/bin +mkdir -p "${stubs}" +calls=${work}/espflash-calls +cat > "${stubs}/espflash" <> "${calls}" +case \$1 in + board-info|checksum-md5|completions|erase-flash|erase-parts|erase-region|flash|hold-in-reset|\\ + list-ports|monitor|partition-table|read-flash|reset|save-image|write-bin|help) exit 0 ;; + *) echo "error: unrecognized subcommand '\$1'" >&2; exit 2 ;; +esac +EOF +chmod +x "${stubs}/espflash" +esp_export=${work}/export-esp.sh +echo 'export STUB_ESP=sourced' > "${esp_export}" + +check() { # [args...] + local root=$1 + shift + env PATH="${stubs}:${PATH}" DOCS_REPO_ROOT="${root}" ESP_EXPORT="${esp_export}" "${commands}" "$@" +} + +run_ok=$(fixture run-ok) +page "${run_ok}" docs/build/flash.md '# Flash + + + +```bash +test -f Cargo.toml +test "${STUB_ESP}" = sourced +true +```' +expect "a ci:run block runs in its directory after the export file" 0 "1 blocks in 2 files pass" - \ + -- check "${run_ok}" + +run_fail=$(fixture run-fail) +page "${run_fail}" docs/build/flash.md '# Flash + + +```sh +false +true +```' +expect "a failing ci:run block names its file and line" 1 - "docs/build/flash.md:4: the ci:run apps/rs/node block failed" \ + -- check "${run_fail}" + +: > "${calls}" +hardware=$(fixture hardware) +page "${hardware}" docs/build/flash.md '# Flash + + +```bash +# espflash bogus in a comment is not checked +espflash write-bin 0x0 coldframe-node-X.Y.Z.bin +cargo run --release +```' +expect "a ci:hardware block only checks its espflash subcommands" 0 "espflash write-bin is a subcommand" "" \ + -- check "${hardware}" +if [[ $(cat "${calls}") == "write-bin --help" ]]; then + pass "a ci:hardware block runs only espflash --help" +else + failed "a ci:hardware block runs only espflash --help" "calls: $(cat "${calls}")" +fi + +hardware_bad=$(fixture hardware-bad) +page "${hardware_bad}" docs/build/flash.md ' +```bash +espflash write-binary 0x0 x.bin +```' +expect "an unknown espflash subcommand fails" 1 - "docs/build/flash.md:2: 'espflash write-binary' is not an espflash subcommand" \ + -- check "${hardware_bad}" + +unmarked=$(fixture unmarked) +page "${unmarked}" docs/build/flash.md '# Flash + +```bash +cargo build --release --locked +```' +expect "an unmarked block fails naming its file and line" 1 - "docs/build/flash.md:3: unmarked bash/sh block" \ + -- check "${unmarked}" +expect "--list reports an unmarked block too" 1 - "docs/build/flash.md:3: unmarked bash/sh block" \ + -- check "${unmarked}" --list + +orphan=$(fixture orphan) +page "${orphan}" docs/build/flash.md ' +Some prose between the marker and the block. + +```bash +true +```' +expect "a marker followed by prose is an orphan" 1 - "docs/build/flash.md:2: a ci:run marker is not followed" \ + -- check "${orphan}" + +unknown=$(fixture unknown) +page "${unknown}" docs/build/flash.md ' +```bash +true +```' +expect "an unknown marker fails" 1 - "docs/build/flash.md:1: unknown marker 'ci:skip'" -- check "${unknown}" + +nodir=$(fixture nodir) +page "${nodir}" docs/build/flash.md ' +```bash +true +```' +expect "a ci:run directory that does not exist fails" 1 - "ci:run directory 'apps/rs/gone' does not exist" \ + -- check "${nodir}" + +listed=$(fixture listed) +page "${listed}" docs/build/flash.md ' +```bash +touch ran +``` + +```text +an unmarked text block is not a command block +``` + + +```bash +espflash monitor +```' +: > "${calls}" +expect "--list prints every block with its marker" 0 "docs/build/flash.md:2: ci:run apps/rs/node" "" \ + -- check "${listed}" --list +expect "--list names the hardware block" 0 "docs/build/flash.md:11: ci:hardware" "" -- check "${listed}" --list +if [[ ! -e ${listed}/apps/rs/node/ran && ! -s ${calls} ]]; then + pass "--list runs nothing" +else + failed "--list runs nothing" "ran: $(ls "${listed}/apps/rs/node")" "calls: $(cat "${calls}")" +fi + +no_export=$(fixture no-export) +page "${no_export}" docs/build/flash.md ' +```bash +test -z "${STUB_ESP:-}" +```' +expect "without the export file a ci:run block still runs, unsourced" 0 "1 blocks in 2 files pass" - \ + -- env -u STUB_ESP PATH="${stubs}:${PATH}" DOCS_REPO_ROOT="${no_export}" ESP_EXPORT="${work}/none.sh" "${commands}" + +expect "a file argument that is no file is a usage error" 2 "" "usage:" -- check "${good}" docs/build/nope.md +expect "the repository guide marks every block" 0 "pass" "" -- "${commands}" --list + +echo "# toolchain pins" +# The pins of ci.yml's firmware job are the reference; release.yml, docs.yml and the guide follow. +pin() { # : the quoted value of `: ""` + sed -n "s/^[[:space:]]*$2:[[:space:]]*\"\([^\"]*\)\".*/\1/p" "$1" | head -n 1 +} +workflows=${here}/../.github/workflows +flash=${here}/build/flash.md +espup_pin=$(pin "${workflows}/ci.yml" ESPUP_VERSION) +toolchain_pin=$(pin "${workflows}/ci.yml" ESP_TOOLCHAIN_VERSION) +espflash_pin=$(pin "${workflows}/ci.yml" ESPFLASH_VERSION) +if [[ -n ${espup_pin} && -n ${toolchain_pin} && -n ${espflash_pin} ]]; then + pass "ci.yml names the three pins" +else + failed "ci.yml names the three pins" "espup '${espup_pin}' toolchain '${toolchain_pin}' espflash '${espflash_pin}'" +fi +for workflow in release.yml docs.yml; do + for variable in ESPUP_VERSION ESP_TOOLCHAIN_VERSION ESPFLASH_VERSION; do + want=$(pin "${workflows}/ci.yml" "${variable}") + got=$(pin "${workflows}/${workflow}" "${variable}") + if [[ ${got} == "${want}" ]]; then + pass "${workflow} pins ${variable} as ci.yml does" + else + failed "${workflow} pins ${variable} as ci.yml does" "${workflow}: '${got}', ci.yml: '${want}'" + fi + done +done +# guide : flash.md contains it. +guide() { + if grep -qF -- "$2" "${flash}"; then pass "$1"; else failed "$1" "flash.md does not contain '$2'"; fi +} +guide "the guide's prose names the espup pin" "espup ${espup_pin}" +guide "the guide's prose names the toolchain pin" "esp toolchain ${toolchain_pin}" +guide "the guide's prose names the espflash pin" "espflash ${espflash_pin}" +guide "the install block installs the espup pin" "cargo install espup --version ${espup_pin} --locked" +guide "the install block installs the toolchain pin" "espup install --toolchain-version ${toolchain_pin} " +guide "the install block installs the espflash pin" "cargo install espflash --version ${espflash_pin} --locked" +# No other version of the three tools appears in the guide. +stray=$(grep -noE "(espup|espflash) [0-9]+\.[0-9]+\.[0-9]+|--version [0-9.]+|--toolchain-version [0-9.]+|toolchain [0-9]+\.[0-9]+\.[0-9]+\.[0-9]+" "${flash}" | + grep -vE "(espup|--version) ${espup_pin//./\\.}\$|(espflash|--version) ${espflash_pin//./\\.}\$|toolchain(-version)? ${toolchain_pin//./\\.}\$") +if [[ -z ${stray} ]]; then + pass "the guide names no other tool version" +else + failed "the guide names no other tool version" "${stray}" +fi + +echo +echo "${passes} passed, ${failures} failed" +[[ ${failures} -eq 0 ]] diff --git a/hardware/README.md b/hardware/README.md index d1040ea8..cf0b3345 100644 --- a/hardware/README.md +++ b/hardware/README.md @@ -1,5 +1,40 @@ # hardware -Schematics, PCB layouts and enclosure designs for the Node and the Hub. Nothing is here yet. +The reference hardware design for the Coldframe Node and Hub: schematics, a bill of materials, the +wiring, and notes on the enclosure and on sealing the soil probe. -The designs and the build guide for adopters arrive in Epic 10. +> **License.** Everything in `hardware/` is part of Coldframe and is licensed under the +> [Apache License 2.0](../LICENSE), like the rest of the repository. + +## Status: reference design, not yet bench-verified + +This is a **reference design**. It has not been built and measured on the bench yet. + +- Every part is a *reference choice*. Each BOM line also gives the **electrical requirement** the + part must meet, so you can use a different part that meets it. +- The schematics are Markdown connection tables and ASCII block diagrams, not an EDA project. You + can review them line by line against the firmware. A KiCad capture and PCB can follow once the + design has been verified on the bench. +- The Node's sleep current has **not been measured** yet. The budget is an average of about + 100 µA or less, and [`docs/bench/node-power-checklist.md`](../docs/bench/node-power-checklist.md) + is how to measure it. + +The firmware is the source of truth for every pin. If a page here disagrees with +[`apps/rs/node/src/board/pins.rs`](../apps/rs/node/src/board/pins.rs), the firmware is right and the +page has a bug. + +## Pages + +| Page | What | +| --- | --- | +| [`bom.md`](bom.md) | The bill of materials for the Node and the Hub, with the requirement for each part | +| [`node.md`](node.md) | The Node: block diagram, connection tables per block, and the full pin-to-pin wiring | +| [`hub.md`](hub.md) | The Hub: a dev board on USB power, antenna clearance and placement | +| [`enclosure.md`](enclosure.md) | The Node enclosure (IP rating, venting, glands) and sealing the soil probe | + +## Building one + +Follow the adopter build guide in [`docs/build/`](../docs/build/README.md). It goes from parts to +assembly, the toolchain, flashing the Hub and the Node, and adding both from the app. The power +guidance (panel placement, the sleep-current budget, and why soil % and battery % are approximate) +is in [`docs/build/power.md`](../docs/build/power.md). diff --git a/hardware/bom.md b/hardware/bom.md new file mode 100644 index 00000000..aee58fcb --- /dev/null +++ b/hardware/bom.md @@ -0,0 +1,47 @@ +# Bill of materials + +The parts for one Node and one Hub. A site needs one Hub and any number of Nodes. + +> **License.** This design is part of Coldframe and is licensed under the +> [Apache License 2.0](../LICENSE). + +> **Reference choices, not yet bench-verified.** Each line names an example part, and the +> requirement a substitute must meet. The example parts have not been built and measured together +> yet (see the [status](README.md#status-reference-design-not-yet-bench-verified)). Check every +> datasheet before you buy. Part numbers are examples, not endorsements. + +## Node + +| # | Qty | Part | Example | Requirement | +| --- | --- | --- | --- | --- | +| N1 | 1 | ESP32-S3 module | Espressif ESP32-S3-WROOM-1-N8 (8 MB flash, PCB antenna) | ESP32-S3; at least 4 MB flash (the [partition table](../apps/rs/node/partitions.csv) ends at 4 MB); PCB antenna; 3.0–3.6 V supply. A bare module, not a dev board: a stock dev board draws about 1 mA asleep (see [power](../docs/build/power.md#sleep-current-budget)) | +| N2 | 1 | Capacitive soil probe | "Capacitive soil moisture sensor v2.0" with a TLC555 (CMOS 555) | Capacitive, not resistive. Runs at 3.3 V. Analogue output stays at or below about 3.1 V, the range of the ADC at 11 dB. **NE555 caveat:** many cheap "v1.2" boards carry an NE555, which works poorly at 3.3 V; read the chip marking and prefer a TLC555 or another CMOS timer. Their edges are unsealed: seal the probe as in [enclosure.md](enclosure.md#sealing-the-soil-probe) | +| N3 | 1 | BME680 breakout | A bare BME680 breakout (sensor, decoupling and I²C pull-ups only) with SDO pulled high, or the bare sensor on the Node board | Bosch BME680 on I²C at **0x77** (SDO to VCC). Powered from 3.3 V directly: no always-on regulator, no level shifter and no power LED, which would break the sleep budget. Boards like Adafruit 3660 carry an on-board 3.3 V regulator and level shifter (and the STEMMA QT revision a power LED), so they do **not** qualify unless modified. I²C pull-ups to 3.3 V | +| N4 | 1 | 1S LiPo cell | A 3.7 V, 800 mAh pouch (size code about 603040) with a protection circuit (PCM) and a 10 kΩ NTC lead | 1S LiPo, 4.2 V full, about 800 mAh. Built-in protection (over-discharge, over-current). A **10 kΩ NTC** (B about 3435 K) in the pack, wired to the charger's NTC input | +| N5 | 1 | Solar panel | A ~1 W, 5–6 V epoxy or glass panel (about 110 × 60 mm) | About 1 W. The open-circuit voltage must stay below the charger's maximum input: a TP4056 is rated to 8 V at most, and a "6 V" panel often reaches 7 V or more open-circuit, so pair a TP4056 with a 5 V panel, or use the CN3791 | +| N6 | 1 | Charger | TP4056 (linear, 1S LiPo, TEMP pin and open-drain `CHRG`). Solar alternative: CN3791 (MPPT, NTC input, open-drain `CHRG` and `DONE`) | 1S LiPo CC/CV charge to 4.2 V. An **NTC input** that stops charging below 0 °C and above about 45 °C. An **open-drain, active-low** charge status pin (`CHRG`), read by GPIO6. Charge current at most 0.5 C (400 mA for 800 mAh); a ~1 W panel delivers about 200 mA, so set it near that (TP4056: RPROG 5 kΩ gives about 250 mA, per its datasheet table). Many TP4056 modules tie TEMP to ground, which disables the NTC: use one that exposes TEMP, or build the charger from the IC | +| N7 | 1 | 3.3 V LDO | Richtek RT9080-33GJ5 (600 mA, about 2 µA quiescent) | 3.3 V out. **Quiescent current at most about 5 µA**, it flows all the time. **At least 500 mA** output for the radio peaks. Low dropout. Stable with ceramic capacitors | +| N8 | 2 | High-side load switch | TI TPS22917DBV (SOT-23-6, quick output discharge) | Input range covering 3.3 V (probe feed) and 4.2 V (cell, divider feed). **Active-high enable**, with an **external pull-down** on the enable pin (N11): the ESP32-S3 pads float in deep sleep. Off-state leakage and quiescent current at most about 1 µA. Output at 0 V when off (output discharge helps). A discrete P-MOSFET with a level-shifting driver also meets it. **Not** a low-side N-MOSFET (see [node.md](node.md#switched-battery-divider)) | +| N9 | 2 | Divider resistors | 100 kΩ, 1 %, 0603 | Exactly 100 kΩ / 100 kΩ: the firmware's `DIVIDER` in [`pins.rs`](../apps/rs/node/src/board/pins.rs) assumes it. 1 % or better | +| N10 | 1 | Divider midpoint capacitor | 4.7 nF ceramic, 0603 | **At most 10 nF.** With 100 kΩ / 100 kΩ that is RC = 0.5 ms, well inside the firmware's 10 ms settle time; a larger capacitor needs the firmware changed | +| N11 | 2 | Pull-downs | 100 kΩ, 0603 | One on each load-switch enable (GPIO4, GPIO5), to ground, so both switches stay off in deep sleep | +| N12 | 1 | Setup button | An IP67 panel-mount momentary push button (normally open) | Momentary, normally open, from GPIO7 to ground. No external pull-up: the firmware uses the internal one, kept in deep sleep | +| N13 | 1 | EN reset network | 10 kΩ pull-up and 1 µF to ground on EN | As in the ESP32-S3-WROOM-1 datasheet, so the module starts cleanly when power comes up | +| N14 | 2 | Bench buttons | 6 × 6 mm tactile switches, on the board | `RESET` (EN to ground) and `BOOT` (GPIO0 to ground, with a 10 kΩ pull-up), for flashing. Not reachable from outside the enclosure | +| N15 | 1 | Decoupling | 22 µF and 100 nF ceramic at the module's 3V3 pin; the LDO's input and output capacitors per its datasheet | As in the module and LDO datasheets | +| N16 | 1 | Flashing header | 1 × 7 pin header, 2.54 mm | GND, USB D− (GPIO19), USB D+ (GPIO20), EN, GPIO0, UART0 TX (GPIO43) and RX (GPIO44). The ESP32-S3's built-in USB Serial/JTAG needs no USB-UART bridge, which would draw current in sleep. See [node.md](node.md#flashing-header) | +| N16a | 1 | USB lead for flashing | A USB breakout board (USB-C or micro-USB) or a cut USB cable, wired to the header | GND, D− and D+ to header pins 1–3; **VBUS left unconnected**. Short (under about 20 cm). See [node.md](node.md#connecting-usb) | +| N17 | 1 | Enclosure and parts | An IP65 (or better) ABS or polycarbonate box, cable glands, a vent membrane, conformal coating and epoxy, heat-shrink | See [enclosure.md](enclosure.md) | + +## Hub + +The Hub uses no GPIO: it is a dev board on USB power, indoors. + +| # | Qty | Part | Example | Requirement | +| --- | --- | --- | --- | --- | +| H1 | 1 | ESP32-S3 dev board | Espressif ESP32-S3-DevKitC-1-N8 | ESP32-S3 on a WROOM-1 module (PCB antenna, not the -1U external-antenna variant) with at least 4 MB flash (the [partition table](../apps/rs/hub/partitions.csv) ends at 4 MB), and a USB port | +| H2 | 1 | USB power supply | Any 5 V USB charger | 5 V, at least 500 mA, always on. A good supply: brownouts reboot the Hub | +| H3 | 1 | USB cable | USB-A or USB-C to micro-USB or USB-C, to match the board | A data cable (for flashing), not a charge-only cable | +| H4 | 1 | Case | A small plastic case, or none | Not metal: the antenna needs clearance (see [hub.md](hub.md#antenna-clearance)) | + +Sleep current does not matter for the Hub: it is always on and mains powered. diff --git a/hardware/enclosure.md b/hardware/enclosure.md new file mode 100644 index 00000000..218ed7f1 --- /dev/null +++ b/hardware/enclosure.md @@ -0,0 +1,66 @@ +# Enclosure and probe sealing + +The Node lives outdoors for a whole season, with its probe in wet soil. The Hub lives indoors and +needs only a plastic case (see [hub.md](hub.md#antenna-clearance)). + +> **License.** This design is part of Coldframe and is licensed under the +> [Apache License 2.0](../LICENSE). + +> **Reference design, not yet bench-verified.** These are the reference choices; none of them has +> been through a season outdoors yet. + +## The outdoor requirement (NFR-5) + +A Node, its solar panel and its soil probe survive a season outdoors in wet soil, **without +corrosion-driven drift making Alerts unreliable**. Everything below serves that: keep water out of +the electronics, and keep the probe's reading stable over months. + +## Node enclosure + +- **IP rating:** IP65 or better (dust-tight, protected against water jets). An IP67 box tolerates + standing water, for example when the Node falls over. ABS or polycarbonate with UV stabilisation; + plain ABS goes brittle in sun. +- **Colour:** light grey or white. A dark box in full sun heats up, and the charger must stop + charging above about 45 °C anyway. +- **Venting for the BME680:** the BME680 measures air temperature, humidity and gas resistance, so + it needs outside air. Fit a breathable **ePTFE vent membrane** (an M12 vent plug) and mount the + BME680 right behind it, away from the LDO and the module, which warm up a little. A sealed box + with no vent reads its own air and collects condensation. +- **Cable glands:** one gland per cable: the probe cable, and the panel cable if the panel is + separate. Size the gland to the cable (for example M12 / PG7 for 3–6 mm cables). Leave a drip + loop below each gland so water runs off before it reaches it. Glands on the bottom face are + safest. +- **Button:** an IP67 panel-mount push button for the setup button + ([BOM N12](bom.md#node)), so you can press it without opening the box. +- **Mounting:** on a stake or post, with the box's glands facing down. +- **Panel and antenna clear:** mount the panel **above the canopy**, angled to the sun and not + shaded by the plant as it grows ([power.md](../docs/build/power.md#panel-placement)). Keep the + module's antenna end free of metal (no metal box, no metal mounting plate behind the antenna), + and keep wet foliage away from it. +- **Inside:** a small silica-gel bag absorbs the moisture trapped when you close the box. Replace + it when you open the box. + +## Sealing the soil probe + +Cheap capacitive probes have bare electronics at the top and **unsealed cut edges** along the +board. Water creeps into the board's edges and under the components, and the reading drifts as the +board absorbs it. Seal every probe before its first use: + +1. **Mark the insertion line.** The probe has a line (or a printed limit) above which it must not + go into the soil. Mark it with a pen. Only the part below the line goes into the soil; the + electronics stay above it. +2. **Coat the cut edges.** Paint every cut edge of the board, along its full length and around the + tip, with conformal coating or a thin layer of epoxy. The flat sensing faces are already covered + by solder mask; coat the edges so water cannot creep in. +3. **Pot the electronics.** Cover the components at the top (the timer chip, the regulator, the + connector) with conformal coating, or pot them in epoxy. Let it cure fully. +4. **Heat-shrink.** Slide adhesive-lined heat-shrink over the top of the probe, from the cable down + to just above the insertion line, and shrink it. It covers the electronics and the cable entry. +5. **Cable.** Use an outdoor-rated cable from the probe to the Node, and run it through a cable + gland. + +A sealed probe still needs **2–3 days to settle** after you insert it, and a calibration in your +soil; see [power.md](../docs/build/power.md#soil-moisture) for why soil % is approximate. + +Inspect the probe at the end of the season. Cracked coating, a lifted heat-shrink or a corroded +edge means the Readings of the last weeks may have drifted. diff --git a/hardware/hub.md b/hardware/hub.md new file mode 100644 index 00000000..c9a6621d --- /dev/null +++ b/hardware/hub.md @@ -0,0 +1,74 @@ +# Hub + +The Hub relays the Nodes' Readings to the Server. It talks ESP-NOW to the Nodes in the garden and +Wi-Fi with TLS to the Server on the home network, and it stores no Readings. It is always on. + +> **License.** This design is part of Coldframe and is licensed under the +> [Apache License 2.0](../LICENSE). + +> **Reference design, not yet bench-verified.** The parts are the reference choices of the +> [BOM](bom.md#hub), each with the requirement a substitute must meet. + +## Schematic + +The Hub is an unmodified ESP32-S3 dev board on USB power. The firmware uses **no GPIO**: no button, +no sensor, no LED. Setup is automatic: an unprovisioned Hub advertises over BLE on every boot. + +```text + +----------------+ USB 5 V, >= 500 mA +------------------------------+ + | USB power |----------------------->| ESP32-S3-DevKitC-1-N8 | + | supply (mains) | data cable | on-board 3.3 V regulator | + +----------------+ | ESP32-S3-WROOM-1 (PCB ant.) | + | RESET (EN) and BOOT buttons | + +------------------------------+ + ^ 2.4 GHz: Wi-Fi to the access point, + | ESP-NOW to the Nodes, + | BLE during setup only +``` + +## Wiring + +| From | To | Notes | +| --- | --- | --- | +| USB power supply | Dev board USB port | Either port of the DevKitC-1 powers the board. For flashing, use the same cable from your computer | + +That is all. Nothing connects to the pin headers. + +The DevKitC-1 has two USB ports, `UART` (through a USB-UART bridge) and `USB` (the ESP32-S3's +built-in USB Serial/JTAG). `espflash` flashes and monitors through either. + +The DevKitC-1's on-board buttons are the only controls: `RST` restarts the Hub (an unprovisioned +Hub then prints its setup code again, see [flashing](../docs/build/flash.md#first-boot-and-the-setup-code)), +and `BOOT` is only for the download mode while flashing. + +## Power + +- 5 V over USB from a mains USB supply, at least 500 mA, always on. The radio's peaks need it. +- The Hub never sleeps, so a battery or a small panel is not an option. +- Unplugging it is safe: it keeps no Readings, and the Nodes keep and resend theirs. + +## Antenna clearance + +The WROOM-1 module has a PCB antenna at the end of the board, beyond the module's metal shield. + +- Keep at least about 15 mm around the antenna end clear of metal, cables and the wall. +- Do not put the Hub in a metal box, behind a radiator, or on a metal shelf. +- A plastic case is fine. Mount the board upright, with the antenna end up, if you can. +- Use the WROOM-1 (PCB antenna), not the WROOM-1U, unless you also fit an external 2.4 GHz antenna. + +## Placement on the home network + +The Hub has to reach two things at once: + +1. **The access point**, over 2.4 GHz Wi-Fi. The ESP32-S3 has no 5 GHz radio. The network must be + WPA2-Personal or mixed WPA2/WPA3 (transition mode); **WPA3-only networks are not supported** + (see [`apps/rs/hub/README.md`](../apps/rs/hub/README.md#ble-setup)). The Hub joins the strongest + access point of the network you choose in the app. +2. **The Nodes**, over ESP-NOW, which uses the same radio and follows the access point's channel. + +So place the Hub indoors, on the garden side of the house, for example on a window sill facing the +garden, and still in good Wi-Fi range. Thick walls, metal-coated glass and wet foliage all cut the +range. The Hub is not weatherproof: keep it indoors or in a dry, sheltered place. + +The Hub reaches the Server by the Server's host name over HTTPS, so the home network's DNS must +resolve that name to the Server ([split DNS](../docs/operations/split-dns.md)). diff --git a/hardware/node.md b/hardware/node.md new file mode 100644 index 00000000..c4210df6 --- /dev/null +++ b/hardware/node.md @@ -0,0 +1,215 @@ +# Node + +The Node is a battery-powered ESP32-S3 on solar power. Every 15 minutes it wakes, powers the soil +probe and the battery divider for a moment, reads the BME680, sends the Readings to the Hub over +ESP-NOW, and goes back to deep sleep. + +> **License.** This design is part of Coldframe and is licensed under the +> [Apache License 2.0](../LICENSE). + +> **Reference design, not yet bench-verified.** The parts are the reference choices of the +> [BOM](bom.md), each with the requirement a substitute must meet. + +**The source of truth for every pin is +[`apps/rs/node/src/board/pins.rs`](../apps/rs/node/src/board/pins.rs).** This page follows it. If +you change a pin, change it there; the firmware then reads the new pin, and this page needs the +same change. + +## Block diagram + +```text + VBAT (3.3-4.2 V) + +-----------+ +-----------+ +----------------------+------------------------+ + | Solar |--->| Charger |-BAT-+ | | + | panel | | (TP4056) | | v v + | ~1 W 5-6 V| | | +-+-----------+ +-------------+ +----------------+ + +-----------+ | TEMP |<--| 1S LiPo | | LDO 3.3 V | | U2 load switch |<-- GPIO5 + | CHRG |-+ | 800 mAh | | Iq <= ~5 uA | | (divider) | + 100 k to GND + +-----------+ | | PCM, 10k NTC| +------+------+ +-------+--------+ + | +-------------+ | 3V3 | + | | R1 100 k + | | +-----------> GPIO2 (ADC1_CH1) + | | R2 100 k + C <= 10 nF + | | GND GND + | | + | +--> U1 load switch (probe) <-- GPIO4 + 100 k to GND + | | | + | | soil probe --AOUT------------> GPIO1 (ADC1_CH0) + | | + | +--> BME680, 0x77 (SDO high) <--I2C--> GPIO8 SDA, GPIO9 SCL + | | + | +--> ESP32-S3-WROOM-1 3V3 (EN: 10 k up, 1 uF down) + | + +---------------------------------- CHRG, active low --------> GPIO6 + + Setup button: GPIO7 ---[ push ]--- GND + Flashing header: GND, D- (GPIO19), D+ (GPIO20), EN, GPIO0, TXD0 (GPIO43), RXD0 (GPIO44) +``` + +The battery feeds the LDO directly. The charger has no power path, which is fine at the Node's +microamp load. + +## Power path + +| From | To | Notes | +| --- | --- | --- | +| Panel + | Charger VCC (input) | The panel's open-circuit voltage must stay below the charger's maximum input ([BOM N5](bom.md#node)) | +| Panel − | GND | | +| Charger BAT | Cell + (protected terminal) | Charge to 4.2 V, at most 0.5 C | +| Cell − (protected terminal) | GND | | +| Cell NTC | Charger TEMP / NTC | Through the network the charger's datasheet gives for a 0 °C to about 45 °C charging window. Never tie TEMP to ground: that disables the temperature cut-off | +| Cell + (VBAT) | LDO IN | Plus the input capacitor of the LDO datasheet | +| LDO OUT (3V3) | Module 3V3 | 22 µF and 100 nF at the module pin | +| LDO GND, module GND | GND | | +| 3V3 | EN, through 10 kΩ; EN to GND through 1 µF | The reset network | + +The battery is read through its own switched divider (below); the firmware maps the voltage to a +percentage with the curve in +[`packages/rs/sensing/src/battery.rs`](../packages/rs/sensing/src/battery.rs). The curve reads +0 % at 3.27 V. Below about 3.4 V the LDO is in dropout and the module runs a little under 3.3 V, +which the ESP32-S3 tolerates down to 3.0 V. + +## Switched probe power + +The probe is powered only while it is read: the firmware switches it on, waits 100 ms +(`PROBE_SETTLE_MS` in [`packages/rs/sensing/src/wake.rs`](../packages/rs/sensing/src/wake.rs)), and +takes 8 samples. + +| From | To | Notes | +| --- | --- | --- | +| GPIO4 | U1 ON (enable) | High = on | +| U1 ON | GND, through 100 kΩ | The pull-down: the pad floats in deep sleep, so the switch must stay off on its own | +| 3V3 | U1 VIN | | +| U1 VOUT | Probe VCC | 0 V while off | +| Probe AOUT | GPIO1 (ADC1_CH0) | Must stay at or below about 3.1 V (11 dB attenuation) | +| Probe GND | GND | | + +## Switched battery divider + +The divider is switched on the **high side**, between the cell and the top resistor. A low-side +N-MOSFET is ruled out: while off it would leave the midpoint, and GPIO2, at up to 4.2 V through the +top resistor, beyond the pad's rating, and current would leak through the pad. Switched on the high +side, GPIO2 sees 0 V while off. + +| From | To | Notes | +| --- | --- | --- | +| GPIO5 | U2 ON (enable) | High = on | +| U2 ON | GND, through 100 kΩ | The pull-down, as for U1 | +| VBAT | U2 VIN | | +| U2 VOUT | R1 100 kΩ (top) | | +| R1 | Midpoint | | +| Midpoint | R2 100 kΩ (bottom) to GND | 4.2 V at the cell reads 2.1 V | +| Midpoint | GPIO2 (ADC1_CH1) | Calibrated millivolts, 11 dB | +| Midpoint | GND, through at most 10 nF | RC = 50 kΩ × 10 nF = 0.5 ms, inside the 10 ms settle time. A larger capacitor needs `DIVIDER_SETTLE_MS` raised to at least 5·RC | + +The divider values are `DIVIDER` in [`pins.rs`](../apps/rs/node/src/board/pins.rs). Only ADC1 is +used, because ADC2 conflicts with the radio. + +## Charger status + +| From | To | Notes | +| --- | --- | --- | +| Charger `CHRG` | GPIO6 | Open-drain, active low: low while charging. The firmware enables the internal pull-up, so no external pull-up is needed. Read on each wake as *charging* or *not charging* | + +## Setup button + +| From | To | Notes | +| --- | --- | --- | +| GPIO7 | Button | GPIO7 is a low-power (RTC) pad, so a press wakes the Node from deep sleep | +| Button | GND | Active low. The internal pull-up is kept in deep sleep: no external resistor | + +A short press (50 ms to under 3 s) sends a Reading now. A long press (3 s) opens the BLE setup +window for 180 s. The timing is in +[`packages/rs/sensing/src/button.rs`](../packages/rs/sensing/src/button.rs). + +## I²C to the BME680 + +| From | To | Notes | +| --- | --- | --- | +| GPIO8 | BME680 SDA | | +| GPIO9 | BME680 SCL (SCK) | | +| 3V3 | BME680 VCC | Not switched: forced mode puts it back to sleep (about 0.15 µA) | +| 3V3 | BME680 SDO | Address **0x77** (`BME680_SECONDARY_ADDRESS` in `pins.rs`) | +| 3V3 | BME680 CS | Selects I²C; breakouts usually do this already | +| GND | BME680 GND | | +| 3V3 | SDA, SCL, through the breakout's pull-ups | Pull-ups to 3.3 V only | + +## Flashing header + +The ESP32-S3 has a built-in USB Serial/JTAG controller, so the Node needs no USB-UART bridge, which +would draw current while asleep. `espflash` flashes and monitors through it. + +| Header pin | To | Notes | +| --- | --- | --- | +| 1 GND | GND | | +| 2 D− | GPIO19 | USB D− | +| 3 D+ | GPIO20 | USB D+ | +| 4 EN | EN | Reset (the RESET button pulls it to ground) | +| 5 IO0 | GPIO0 | Strapping pin: held low at reset, the chip enters the download mode. 10 kΩ pull-up and the BOOT button to ground | +| 6 TXD0 | GPIO43 | UART0, for a USB-UART adapter instead of USB | +| 7 RXD0 | GPIO44 | UART0 | + +Power the board from the battery while you flash; do not connect USB VBUS to the 3V3 rail. The USB +serial link drops while the chip sleeps, and `espflash monitor` reconnects on the next wake. + +### Connecting USB + +The Node has no USB connector. To flash it, wire a USB breakout or a cut USB cable +([BOM N16a](bom.md#node)) to the header: + +- USB **GND** to header pin 1 (GND); +- USB **D−** to header pin 2 (GPIO19); +- USB **D+** to header pin 3 (GPIO20); +- leave USB **VBUS unconnected**: the Node runs from its battery while you flash. + +Keep the lead short (under about 20 cm) and the D−/D+ wires twisted together or side by side: USB +does not tolerate long, loose data wires. + +## Wiring: full pin-to-pin list + +Every connection on the module, checked against +[`pins.rs`](../apps/rs/node/src/board/pins.rs). Look up the module pad of each GPIO in the +ESP32-S3-WROOM-1 datasheet. + +| Module pin | Signal | Connects to | Direction | Notes | +| --- | --- | --- | --- | --- | +| 3V3 | Supply | LDO OUT; 22 µF + 100 nF to GND | in | | +| GND | Ground | GND | | | +| EN | Reset | 10 kΩ to 3V3, 1 µF to GND, RESET button to GND, header pin 4 | in | | +| GPIO0 | Boot strap | 10 kΩ to 3V3, BOOT button to GND, header pin 5 | in | Strapping pin, only for flashing | +| GPIO1 | Soil probe analogue out | Probe AOUT | in (ADC1_CH0) | Raw count, 11 dB | +| GPIO2 | Battery divider midpoint | R1/R2 midpoint, ≤ 10 nF to GND | in (ADC1_CH1) | Calibrated mV, 11 dB, 100 kΩ / 100 kΩ | +| GPIO4 | Probe power switch | U1 ON, 100 kΩ to GND | out | High-side switch, high = on | +| GPIO5 | Divider switch | U2 ON, 100 kΩ to GND | out | High-side switch, high = on | +| GPIO6 | Charger status (`CHRG`) | Charger `CHRG` | in | Active low, internal pull-up | +| GPIO7 | Setup button | Button to GND | in | Active low, internal pull-up kept in deep sleep; wakes the Node | +| GPIO8 | I²C SDA | BME680 SDA | in/out | | +| GPIO9 | I²C SCL | BME680 SCL | out | | +| GPIO19 | USB D− | Header pin 2 | in/out | Flashing and logs | +| GPIO20 | USB D+ | Header pin 3 | in/out | Flashing and logs | +| GPIO43 | UART0 TX | Header pin 6 | out | Optional | +| GPIO44 | UART0 RX | Header pin 7 | in | Optional | + +Every other GPIO is unconnected. Off-module parts: + +| Part | Pin | Connects to | +| --- | --- | --- | +| Panel | + / − | Charger VCC / GND | +| Charger | BAT / TEMP / `CHRG` / GND | Cell + / cell NTC network / GPIO6 / GND | +| Cell | + / − / NTC | VBAT (charger BAT, LDO IN, U2 VIN) / GND / charger TEMP | +| LDO | IN / OUT / GND | VBAT / 3V3 / GND | +| U1 (probe switch) | VIN / VOUT / ON / GND | 3V3 / probe VCC / GPIO4 + 100 kΩ to GND / GND | +| U2 (divider switch) | VIN / VOUT / ON / GND | VBAT / R1 / GPIO5 + 100 kΩ to GND / GND | +| R1 100 kΩ | 1 / 2 | U2 VOUT / midpoint | +| R2 100 kΩ | 1 / 2 | Midpoint / GND | +| C ≤ 10 nF | 1 / 2 | Midpoint / GND | +| Soil probe | VCC / AOUT / GND | U1 VOUT / GPIO1 / GND | +| BME680 | VCC / GND / SDA / SCL / SDO / CS | 3V3 / GND / GPIO8 / GPIO9 / 3V3 / 3V3 | +| Setup button | 1 / 2 | GPIO7 / GND | + +## Sleep current + +The design aims at an average sleep current of at most about 100 µA: the LDO's quiescent current, +the module in deep sleep, the BME680 asleep, the switches' leakage and the charger's drain on the +cell. **It has not been measured yet.** See [power.md](../docs/build/power.md#sleep-current-budget) +for the budget and how to measure it.