Skip to content

fix: mark the pkg custom domain noindex, not just *.pages.dev - #15

Merged
toastygm merged 1 commit into
mainfrom
bug/9_pkg-domains-indexable
Aug 30, 2026
Merged

fix: mark the pkg custom domain noindex, not just *.pages.dev#15
toastygm merged 1 commit into
mainfrom
bug/9_pkg-domains-indexable

Conversation

@toastygm

Copy link
Copy Markdown
Contributor

The default _headers this workflow writes covered two of the three
families of address a deployment answers on. <project>.pages.dev and
<deployment>.<project>.pages.dev were marked noindex;
<package>.<domain-suffix> — the custom domain the two steps further down this
same workflow create — was not, and it is the newest of the three and the only
one a reader is plausibly handed.

Adds the third rule.

Verified live, before the change

X-Robots-Tag on GET /<package>/, measured 2026-08-30:

Address Status X-Robots-Tag
sohl-kb.pages.dev 200 noindex
sohl-thalorna.pages.dev 200 noindex
sohl-kethira-basic.pages.dev 200 noindex
harn-ensemble.pages.dev 200 noindex
harn-adventures.pages.dev 200 noindex
hm3-site.pages.dev 200 noindex
sohl.pkg.heroiclands.org 200 absent
thalorna.pkg.heroiclands.org 200 absent
kethira.pkg.heroiclands.org 200 absent
harnensemble.pkg.heroiclands.org 200 absent
harnadventures.pkg.heroiclands.org 200 absent
hm3.pkg.heroiclands.org 200 absent
www.heroiclands.org/<package>/ (all six) 200 absent — correct

That is the bug, and it is also the proof that the placeholder is one label:
the same deployment, the same _headers, sets the header at
<project>.pages.dev and not at <package>.pkg.heroiclands.org. So :project
did not swallow sohl.pkg.heroiclands, and Cloudflare matches the rule against
the real request host rather than rewriting a custom-domain request to the
project's own name. Both are things the third rule depends on, and neither had
been checked before — #7 said as much. Cloudflare's own
documentation agrees: "Placeholders match all characters apart from the
delimiter, which when part of the host, is a period."

The :version.:project.pages.dev rule could not be exercised: preview
aliases exist only for non-production branches and every package deploys from
main. It shares its mechanism with the rule above, which is now confirmed.

The header does reach the canonical address — that is the whole risk

<package>.pkg.heroiclands.org is not merely a second address, it is the
origin heroiclands-site's router fetches
. So a noindex there is on every
response www.heroiclands.org/<package>/ is built from, and if it survived the
proxy this change would de-index the canonical site.

Headers otherwise survive it completely. Comparing the two responses for all six
packages, the set at www.heroiclands.org/<package>/ is the origin's, verbatim
content-type, cache-control, access-control-allow-origin,
referrer-policy, x-content-type-options, cf-cache-status — differing only
in order. Bodies hash identically, which is also what proves the Worker is in
the path at all.

What removes it is canonicalHeaders in heroiclands-site/worker/src/router.js,
which deletes x-robots-tag and nothing else. Its suite was run here at that
repository's main: 28/28 green, including one assertion on the pure
function and one end to end through the handler with a stubbed origin returning
x-robots-tag: noindex. heroiclands-site#30 gates every pull request there on
that suite, and its last worker deploy succeeded on the current main.

And it has been live in production. Until heroiclands-site#26 — merged
2026-08-29, hours before #9 was filed — the router's origins
were sohl-kb.pages.dev and sohl-thalorna.pages.dev. The existing first rule
already set noindex on every response the router fetched, and
www.heroiclands.org has never carried it. #26 moved the origin to
<package>.<domain-suffix>, which in one change both opened this hole and left
the strip with nothing to strip. This restores an arrangement that ran for
eleven days, rather than introducing a dependency.

The residual failure mode is stated in the workflow, next to the payload.

Why X-Robots-Tag and not a canonical link header

noindex is a directive; rel=canonical is a consolidation hint a crawler may
decline, and it does not stop the duplicate host being crawled. More decisively,
a canonical link would have to name the canonical address — and this
workflow deliberately knows no consumer addresses. project and domain-suffix
are inputs; the payload names no package and no site. A canonical URL would be
the first consumer address in it, and wrong for anyone who took the repository
elsewhere. One directive across all three host families keeps the file a single
statement.

(Worth noting separately, not fixed here: none of the six packages emits a
<link rel="canonical"> in its HTML either. That is the theme's, in another
repository.)

The rule, and why it is built from the input

https://:package.${SUFFIX}/*
  X-Robots-Tag: noindex

${SUFFIX} is inputs.domain-suffix, resolved exactly as the custom-domain and
DNS steps already resolve it, so the default stays package-agnostic and
address-agnostic. Hardcoding pkg.heroiclands.org would have put a consumer
address in a shared default for the first time. This is the one heredoc in the
workflow with an unquoted delimiter; $ appears nowhere else in the payload,
since Cloudflare's placeholders are :name.

www.heroiclands.org cannot match it: the pattern needs four labels and a
literal pkg. That safety holds only while domain-suffix names a dedicated
namespace and not the domain the canonical site is served from — a consumer
whose site is www.example.net must not set domain-suffix: example.net, which
would equally give the router /www/ as a package prefix. Recorded in the
comment.

Verified by execution

  • actionlint 1.7.12 with shellcheck 0.11.0 over every workflow: clean.

  • Two negative controls proving the linter reaches the edited step. Renaming
    its inputs.domain-suffix to domain-suffiz is reported at 437:31, the new
    env: line, so actionlint type-checks it against the declared workflow_call
    interface. Injecting probe=$(echo $SUFFIX) above the heredoc yields
    SC2086/SC2034/SC2116 — shellcheck resolving into the new script body.

  • The step's own text, extracted with yq so the block-scalar dedent is
    applied, and run for real:

    Case Result
    no _headers, default suffix writes the three rules, third one https://:package.pkg.heroiclands.org/*
    no _headers, domain-suffix: origins.example.net third rule is https://:package.origins.example.net/*; first two unchanged
    build emitted its own _headers untouched, byte for byte
    build emitted an empty _headers untouched at 0 bytes, so the guard below still fails it
    no tree at all ::notice::, exit 0, nothing conjured

What cannot be verified without a deploy

That Cloudflare binds :package in a four-label pattern. The live evidence
above shows placeholders are one label and that the matcher sees the custom
domain, which is the whole mechanism, but the rule itself has never been served.
I have no Cloudflare credentials and did not deploy.

On the first deploy of any package using this workflow, confirm:

curl -sI https://kethira.pkg.heroiclands.org/kethira/   # expect X-Robots-Tag: noindex
curl -sI https://www.heroiclands.org/kethira/           # expect NO X-Robots-Tag

The second is the one that matters. If the first is absent, :package did not
bind and the fallback is a literal per-package hostname, at the cost of the
default no longer being package-agnostic. If the second is present, stop and
revert this rule — the router's strip is not doing its job and the canonical
site is being de-indexed.

Scope

Four of the six packages call this workflow — kethira, harnensemble,
harnadventures and hm3 — and none of them emits its own _headers, so all
four get the third rule on their next deploy. sohl and thalorna still deploy
from hand-written workflows and write their own two-rule _headers
(utils/build-site.mjs and utils/build-site-root.mjs), so this change does
not reach them
; both remain indexable at their pkg.heroiclands.org address
until they migrate or add the rule. That is a separate change in two other
repositories and is not made here.

Stale in the issue

Its "Confirmed working today: https://www.heroiclands.org/sohl/ returns no
X-Robots-Tag while its origin sets one" was already untrue when written — the
origin had become sohl.pkg.heroiclands.org an hour and a half earlier, and it
sets no such header. For the same reason "it is the same mechanism already
proven for the .pages.dev rules" is not right either: the .pages.dev rules
never reach the router, so this is the first rule that depends on the strip
now, even though the strip was proven in production before #26. The direction
of the fix is unaffected.

Closes #9

The default `_headers` covered two of the three addresses a deployment
answers on. `<package>.<domain-suffix>` — created by the custom-domain and
DNS steps in this same workflow — was left fully indexable, verified live
on all six packages.

Adds a third rule, built from `inputs.domain-suffix` so the default names
no consumer address, and records why `X-Robots-Tag` rather than a canonical
link and what the router's strip does with it.
@toastygm
toastygm merged commit a614e14 into main Aug 30, 2026
1 check passed
@toastygm
toastygm deleted the bug/9_pkg-domains-indexable branch August 30, 2026 01:02
toastygm added a commit to HeroicLands/Song-of-Heroic-Lands-FoundryVTT that referenced this pull request Aug 30, 2026
…loy (#1770)

Four of the six packages deploy through the shared reusable workflow in
`HeroicLands/.github`, which since HeroicLands/.github#15 writes a third
`_headers` rule marking `<package>.pkg.heroiclands.org` `noindex`. This
repository deploys from its own workflow and generates its own two-rule
`_headers`, so `sohl.pkg.heroiclands.org` stayed indexable. This adds
the third
rule where this repository actually writes one — `utils/build-site.mjs`
— and
records, in the deploy workflow, why the workflow itself was not
migrated.

## Measured before the change

```
https://sohl-kb.pages.dev/sohl/                 200   x-robots-tag: noindex
https://8a6cf436.sohl-kb.pages.dev/sohl/        200   x-robots-tag: noindex
https://sohl.pkg.heroiclands.org/sohl/          200   (absent)
https://www.heroiclands.org/sohl/               200   (absent)
```

All three bodies hash identically
(`6653c89dc38ba18255e7d46e728f631e1a6a33276f8da3856269461749d35a6e`), so
this is
one deployment answering on three addresses and setting the header on
only two of
them. The `pkg` address is the newest of the three and the only one a
reader is
plausibly handed.

Incidentally established while measuring: the hosting project's
`pages.dev`
subdomain is **`sohl-kb`**, not `sohl-site`. `--project-name=sohl-site`
is
correct — the deploy log ends "Deployment complete! Take a peek over at
`https://8a6cf436.sohl-kb.pages.dev`" — the subdomain is simply fixed at
project
creation and survived the later rename. Nothing depends on it:
`:project` is a
placeholder.

## The decision: this repository keeps its own workflow

Migrating to the shared workflow was the alternative, and it was
rejected on two
independent grounds.

**It would not have delivered this fix.** The shared workflow writes its
`_headers` payload only when the build produced none — _"A package that
emits its
own keeps it, byte for byte — nothing here rewrites, merges into, or
appends to
an existing file."_ `utils/build-site.mjs` writes one unconditionally,
so this
repository would have gone on serving its own two-rule file from inside
the
shared workflow. Its completeness guard says as much in the same words:
_"A
package that emits its OWN `_headers` passes this check with whatever it
wrote,
so it is that package's job to cover the custom domain too."_ The rule
had to be
added here either way.

**And it would have changed what is published.** The shared workflow
runs one npm
script and knows nothing of its steps. The existing `build:site`
(`docs:prepare docs:html build:kb site:assemble`) builds the API
documentation
from the working tree, i.e. from `main` — but `/sohl/api/` documents the
newest
**release** (#1452), which is why `deploy-sohl.yml` resolves the newest
release
tag, checks it out separately, and runs `npm ci` there against that
tag's own
lockfile. Handing the shared workflow today's `build:site` would
silently
republish `/sohl/api/` from `main`. Migration is possible — that
workflow's
checkout is unshallow with tags precisely so a package's build can check
out
another ref — but it is a rewrite of the build script, not a change of
caller,
and not worth carrying on the back of a three-line header rule.

The reasoning is written into `.github/workflows/deploy-sohl.yml`, which
is the
issue's third acceptance criterion.

## The generated `_headers`, verbatim

`npm run build:site` was run in full before and after. Page count is
unchanged at
1704, and `_headers` is the only file that differs.

Before:

```
https://:project.pages.dev/*
  X-Robots-Tag: noindex

https://:version.:project.pages.dev/*
  X-Robots-Tag: noindex
```

After:

```
https://:project.pages.dev/*
  X-Robots-Tag: noindex

https://:version.:project.pages.dev/*
  X-Robots-Tag: noindex

https://:package.pkg.heroiclands.org/*
  X-Robots-Tag: noindex
```

## Why the new rule cannot de-index the canonical site

This is the failure mode to design against, and there are two
independent guards.

**The pattern cannot match `www.heroiclands.org`.** Cloudflare's `:name`
placeholders are single-label wildcards — _"Placeholders match all
characters
apart from the delimiter, which when part of the host, is a period"_ —
so
`:package.pkg.heroiclands.org` requires **four** labels with a literal
`pkg`
third from the end. `www.heroiclands.org` has three, and no binding of
`:package`
can span the dot that would be needed to fold it in. The other two rules
require
a literal `pages.dev` suffix. The safety holds only while the namespace
is a
dedicated one; a consumer whose site were `www.example.net` must not
point this
at `example.net`, which is recorded next to the constant.

That a four-label pattern binds at all is not assumed:
`:version.:project.pages.dev`
is four labels with two placeholders, and `8a6cf436.sohl-kb.pages.dev`
answers
with the header today.

**And the router strips it anyway.** `canonicalHeaders` in
`heroiclands-site`
(`worker/src/router.js`) deletes `x-robots-tag` and nothing else, and is
called on
the proxy path at `worker/src/index.js:91`. That repository's suite was
run here
at its `main`: **38/38 green**, including one assertion on the pure
function and
one end to end through `handler.fetch` with a stubbed origin returning
`x-robots-tag: noindex`.

This also restores an arrangement that ran in production: until
heroiclands-site#26
the router's origin *was* `<project>.pages.dev`, so the first rule
already set
`noindex` on everything it fetched and `www` never carried it.

## Tests

`tests/build/site-noindex.test.ts` now models Cloudflare's single-label
placeholder semantics rather than pattern-matching the rule text, so two
things
become assertions instead of claims: every host-assigned address is
covered by
exactly one rule, and `www.heroiclands.org` (with `heroiclands.org`,
`api.`, `kb.` and the bare `pkg.` host) is matched by none. The
"cannot match the canonical host" case passes against the old payload
too — it is
a standing regression guard, not a test of this change.

## Verify after the next deploy

```
curl -sI https://sohl.pkg.heroiclands.org/sohl/   # expect: X-Robots-Tag: noindex
curl -sI https://www.heroiclands.org/sohl/        # expect: NO X-Robots-Tag
```

The second is the one that matters. If it carries the header, revert
this rule
immediately — the router's strip is not doing its job and the canonical
site is
being de-indexed. If only the first is absent, `:package` did not bind
and the
fallback is the literal `sohl.pkg.heroiclands.org`.

## Wrong in the issue

Option 1 is stated as _"Migrate this repository's deploy to the shared
reusable
workflow. Then the rule arrives for free."_ **It would not have.** The
shared
workflow writes its `_headers` only when the build produced none, and
this build
always produces one, so migrating would have left
`sohl.pkg.heroiclands.org`
exactly as indexable as it is today while looking like a fix. That is
the
substance of the decision, and it is the opposite of how the issue
frames the
trade-off — option 1 is not "the rule for free plus a migration", it is
"a
migration, and then this change as well".

Everything else in the issue held up on re-measurement.

Not done here: nothing was deployed, no Cloudflare or DNS setting was
touched,
and the four packages on the shared workflow were left alone — none had
redeployed since HeroicLands/.github#15 merged, so the third rule is
still
unproven at the edge for any package.

Closes #1765

Co-authored-by: Tom Rodriguez <tmrodrig@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

The pkg.heroiclands.org custom domains are indexable — _headers only covers *.pages.dev

1 participant