fix: mark the pkg custom domain noindex, not just *.pages.dev - #15
Merged
Conversation
The default `_headers` covered two of the three addresses a deployment answers on. `<package>.<domain-suffix>` — created by the custom-domain and DNS steps in this same workflow — was left fully indexable, verified live on all six packages. Adds a third rule, built from `inputs.domain-suffix` so the default names no consumer address, and records why `X-Robots-Tag` rather than a canonical link and what the router's strip does with it.
toastygm
added a commit
to HeroicLands/Song-of-Heroic-Lands-FoundryVTT
that referenced
this pull request
Aug 30, 2026
…loy (#1770) Four of the six packages deploy through the shared reusable workflow in `HeroicLands/.github`, which since HeroicLands/.github#15 writes a third `_headers` rule marking `<package>.pkg.heroiclands.org` `noindex`. This repository deploys from its own workflow and generates its own two-rule `_headers`, so `sohl.pkg.heroiclands.org` stayed indexable. This adds the third rule where this repository actually writes one — `utils/build-site.mjs` — and records, in the deploy workflow, why the workflow itself was not migrated. ## Measured before the change ``` https://sohl-kb.pages.dev/sohl/ 200 x-robots-tag: noindex https://8a6cf436.sohl-kb.pages.dev/sohl/ 200 x-robots-tag: noindex https://sohl.pkg.heroiclands.org/sohl/ 200 (absent) https://www.heroiclands.org/sohl/ 200 (absent) ``` All three bodies hash identically (`6653c89dc38ba18255e7d46e728f631e1a6a33276f8da3856269461749d35a6e`), so this is one deployment answering on three addresses and setting the header on only two of them. The `pkg` address is the newest of the three and the only one a reader is plausibly handed. Incidentally established while measuring: the hosting project's `pages.dev` subdomain is **`sohl-kb`**, not `sohl-site`. `--project-name=sohl-site` is correct — the deploy log ends "Deployment complete! Take a peek over at `https://8a6cf436.sohl-kb.pages.dev`" — the subdomain is simply fixed at project creation and survived the later rename. Nothing depends on it: `:project` is a placeholder. ## The decision: this repository keeps its own workflow Migrating to the shared workflow was the alternative, and it was rejected on two independent grounds. **It would not have delivered this fix.** The shared workflow writes its `_headers` payload only when the build produced none — _"A package that emits its own keeps it, byte for byte — nothing here rewrites, merges into, or appends to an existing file."_ `utils/build-site.mjs` writes one unconditionally, so this repository would have gone on serving its own two-rule file from inside the shared workflow. Its completeness guard says as much in the same words: _"A package that emits its OWN `_headers` passes this check with whatever it wrote, so it is that package's job to cover the custom domain too."_ The rule had to be added here either way. **And it would have changed what is published.** The shared workflow runs one npm script and knows nothing of its steps. The existing `build:site` (`docs:prepare docs:html build:kb site:assemble`) builds the API documentation from the working tree, i.e. from `main` — but `/sohl/api/` documents the newest **release** (#1452), which is why `deploy-sohl.yml` resolves the newest release tag, checks it out separately, and runs `npm ci` there against that tag's own lockfile. Handing the shared workflow today's `build:site` would silently republish `/sohl/api/` from `main`. Migration is possible — that workflow's checkout is unshallow with tags precisely so a package's build can check out another ref — but it is a rewrite of the build script, not a change of caller, and not worth carrying on the back of a three-line header rule. The reasoning is written into `.github/workflows/deploy-sohl.yml`, which is the issue's third acceptance criterion. ## The generated `_headers`, verbatim `npm run build:site` was run in full before and after. Page count is unchanged at 1704, and `_headers` is the only file that differs. Before: ``` https://:project.pages.dev/* X-Robots-Tag: noindex https://:version.:project.pages.dev/* X-Robots-Tag: noindex ``` After: ``` https://:project.pages.dev/* X-Robots-Tag: noindex https://:version.:project.pages.dev/* X-Robots-Tag: noindex https://:package.pkg.heroiclands.org/* X-Robots-Tag: noindex ``` ## Why the new rule cannot de-index the canonical site This is the failure mode to design against, and there are two independent guards. **The pattern cannot match `www.heroiclands.org`.** Cloudflare's `:name` placeholders are single-label wildcards — _"Placeholders match all characters apart from the delimiter, which when part of the host, is a period"_ — so `:package.pkg.heroiclands.org` requires **four** labels with a literal `pkg` third from the end. `www.heroiclands.org` has three, and no binding of `:package` can span the dot that would be needed to fold it in. The other two rules require a literal `pages.dev` suffix. The safety holds only while the namespace is a dedicated one; a consumer whose site were `www.example.net` must not point this at `example.net`, which is recorded next to the constant. That a four-label pattern binds at all is not assumed: `:version.:project.pages.dev` is four labels with two placeholders, and `8a6cf436.sohl-kb.pages.dev` answers with the header today. **And the router strips it anyway.** `canonicalHeaders` in `heroiclands-site` (`worker/src/router.js`) deletes `x-robots-tag` and nothing else, and is called on the proxy path at `worker/src/index.js:91`. That repository's suite was run here at its `main`: **38/38 green**, including one assertion on the pure function and one end to end through `handler.fetch` with a stubbed origin returning `x-robots-tag: noindex`. This also restores an arrangement that ran in production: until heroiclands-site#26 the router's origin *was* `<project>.pages.dev`, so the first rule already set `noindex` on everything it fetched and `www` never carried it. ## Tests `tests/build/site-noindex.test.ts` now models Cloudflare's single-label placeholder semantics rather than pattern-matching the rule text, so two things become assertions instead of claims: every host-assigned address is covered by exactly one rule, and `www.heroiclands.org` (with `heroiclands.org`, `api.`, `kb.` and the bare `pkg.` host) is matched by none. The "cannot match the canonical host" case passes against the old payload too — it is a standing regression guard, not a test of this change. ## Verify after the next deploy ``` curl -sI https://sohl.pkg.heroiclands.org/sohl/ # expect: X-Robots-Tag: noindex curl -sI https://www.heroiclands.org/sohl/ # expect: NO X-Robots-Tag ``` The second is the one that matters. If it carries the header, revert this rule immediately — the router's strip is not doing its job and the canonical site is being de-indexed. If only the first is absent, `:package` did not bind and the fallback is the literal `sohl.pkg.heroiclands.org`. ## Wrong in the issue Option 1 is stated as _"Migrate this repository's deploy to the shared reusable workflow. Then the rule arrives for free."_ **It would not have.** The shared workflow writes its `_headers` only when the build produced none, and this build always produces one, so migrating would have left `sohl.pkg.heroiclands.org` exactly as indexable as it is today while looking like a fix. That is the substance of the decision, and it is the opposite of how the issue frames the trade-off — option 1 is not "the rule for free plus a migration", it is "a migration, and then this change as well". Everything else in the issue held up on re-measurement. Not done here: nothing was deployed, no Cloudflare or DNS setting was touched, and the four packages on the shared workflow were left alone — none had redeployed since HeroicLands/.github#15 merged, so the third rule is still unproven at the edge for any package. Closes #1765 Co-authored-by: Tom Rodriguez <tmrodrig@gmail.com>
Closed
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The default
_headersthis workflow writes covered two of the threefamilies of address a deployment answers on.
<project>.pages.devand<deployment>.<project>.pages.devwere markednoindex;<package>.<domain-suffix>— the custom domain the two steps further down thissame workflow create — was not, and it is the newest of the three and the only
one a reader is plausibly handed.
Adds the third rule.
Verified live, before the change
X-Robots-TagonGET /<package>/, measured 2026-08-30:X-Robots-Tagsohl-kb.pages.devnoindexsohl-thalorna.pages.devnoindexsohl-kethira-basic.pages.devnoindexharn-ensemble.pages.devnoindexharn-adventures.pages.devnoindexhm3-site.pages.devnoindexsohl.pkg.heroiclands.orgthalorna.pkg.heroiclands.orgkethira.pkg.heroiclands.orgharnensemble.pkg.heroiclands.orgharnadventures.pkg.heroiclands.orghm3.pkg.heroiclands.orgwww.heroiclands.org/<package>/(all six)That is the bug, and it is also the proof that the placeholder is one label:
the same deployment, the same
_headers, sets the header at<project>.pages.devand not at<package>.pkg.heroiclands.org. So:projectdid not swallow
sohl.pkg.heroiclands, and Cloudflare matches the rule againstthe real request host rather than rewriting a custom-domain request to the
project's own name. Both are things the third rule depends on, and neither had
been checked before — #7 said as much. Cloudflare's own
documentation agrees: "Placeholders match all characters apart from the
delimiter, which when part of the host, is a period."
The
:version.:project.pages.devrule could not be exercised: previewaliases exist only for non-production branches and every package deploys from
main. It shares its mechanism with the rule above, which is now confirmed.The header does reach the canonical address — that is the whole risk
<package>.pkg.heroiclands.orgis not merely a second address, it is theorigin
heroiclands-site's router fetches. So anoindexthere is on everyresponse
www.heroiclands.org/<package>/is built from, and if it survived theproxy this change would de-index the canonical site.
Headers otherwise survive it completely. Comparing the two responses for all six
packages, the set at
www.heroiclands.org/<package>/is the origin's, verbatim—
content-type,cache-control,access-control-allow-origin,referrer-policy,x-content-type-options,cf-cache-status— differing onlyin order. Bodies hash identically, which is also what proves the Worker is in
the path at all.
What removes it is
canonicalHeadersinheroiclands-site/worker/src/router.js,which deletes
x-robots-tagand nothing else. Its suite was run here at thatrepository's
main: 28/28 green, including one assertion on the purefunction and one end to end through the handler with a stubbed origin returning
x-robots-tag: noindex. heroiclands-site#30 gates every pull request there onthat suite, and its last worker deploy succeeded on the current
main.And it has been live in production. Until heroiclands-site#26 — merged
2026-08-29, hours before #9 was filed — the router's origins
were
sohl-kb.pages.devandsohl-thalorna.pages.dev. The existing first rulealready set
noindexon every response the router fetched, andwww.heroiclands.orghas never carried it. #26 moved the origin to<package>.<domain-suffix>, which in one change both opened this hole and leftthe strip with nothing to strip. This restores an arrangement that ran for
eleven days, rather than introducing a dependency.
The residual failure mode is stated in the workflow, next to the payload.
Why
X-Robots-Tagand not a canonical link headernoindexis a directive;rel=canonicalis a consolidation hint a crawler maydecline, and it does not stop the duplicate host being crawled. More decisively,
a canonical link would have to name the canonical address — and this
workflow deliberately knows no consumer addresses.
projectanddomain-suffixare inputs; the payload names no package and no site. A canonical URL would be
the first consumer address in it, and wrong for anyone who took the repository
elsewhere. One directive across all three host families keeps the file a single
statement.
(Worth noting separately, not fixed here: none of the six packages emits a
<link rel="canonical">in its HTML either. That is the theme's, in anotherrepository.)
The rule, and why it is built from the input
${SUFFIX}isinputs.domain-suffix, resolved exactly as the custom-domain andDNS steps already resolve it, so the default stays package-agnostic and
address-agnostic. Hardcoding
pkg.heroiclands.orgwould have put a consumeraddress in a shared default for the first time. This is the one heredoc in the
workflow with an unquoted delimiter;
$appears nowhere else in the payload,since Cloudflare's placeholders are
:name.www.heroiclands.orgcannot match it: the pattern needs four labels and aliteral
pkg. That safety holds only whiledomain-suffixnames a dedicatednamespace and not the domain the canonical site is served from — a consumer
whose site is
www.example.netmust not setdomain-suffix: example.net, whichwould equally give the router
/www/as a package prefix. Recorded in thecomment.
Verified by execution
actionlint1.7.12 withshellcheck0.11.0 over every workflow: clean.Two negative controls proving the linter reaches the edited step. Renaming
its
inputs.domain-suffixtodomain-suffizis reported at437:31, the newenv:line, so actionlint type-checks it against the declaredworkflow_callinterface. Injecting
probe=$(echo $SUFFIX)above the heredoc yieldsSC2086/SC2034/SC2116— shellcheck resolving into the new script body.The step's own text, extracted with
yqso the block-scalar dedent isapplied, and run for real:
_headers, default suffixhttps://:package.pkg.heroiclands.org/*_headers,domain-suffix: origins.example.nethttps://:package.origins.example.net/*; first two unchanged_headers_headers::notice::, exit 0, nothing conjuredWhat cannot be verified without a deploy
That Cloudflare binds
:packagein a four-label pattern. The live evidenceabove shows placeholders are one label and that the matcher sees the custom
domain, which is the whole mechanism, but the rule itself has never been served.
I have no Cloudflare credentials and did not deploy.
On the first deploy of any package using this workflow, confirm:
The second is the one that matters. If the first is absent,
:packagedid notbind and the fallback is a literal per-package hostname, at the cost of the
default no longer being package-agnostic. If the second is present, stop and
revert this rule — the router's strip is not doing its job and the canonical
site is being de-indexed.
Scope
Four of the six packages call this workflow —
kethira,harnensemble,harnadventuresandhm3— and none of them emits its own_headers, so allfour get the third rule on their next deploy.
sohlandthalornastill deployfrom hand-written workflows and write their own two-rule
_headers(
utils/build-site.mjsandutils/build-site-root.mjs), so this change doesnot reach them; both remain indexable at their
pkg.heroiclands.orgaddressuntil they migrate or add the rule. That is a separate change in two other
repositories and is not made here.
Stale in the issue
Its "Confirmed working today:
https://www.heroiclands.org/sohl/returns noX-Robots-Tagwhile its origin sets one" was already untrue when written — theorigin had become
sohl.pkg.heroiclands.organ hour and a half earlier, and itsets no such header. For the same reason "it is the same mechanism already
proven for the
.pages.devrules" is not right either: the.pages.devrulesnever reach the router, so this is the first rule that depends on the strip
now, even though the strip was proven in production before #26. The direction
of the fix is unaffected.
Closes #9