[tracking202]

Stacked on #273 (QSB2, 1.9.79-ltv-webhooks-scheduled, at 272f792): review and merge this after #273. Until #273 merges, this diff includes #273's commits; the changes of its own are commits a78e59a and 7ac4aa9 (the fix after review, below). It is not stacked on #285 (9BA0): the two share only changelog and doc files, in different sections.

Backlog: 8YRG

What was wrong

• A dead webhook stayed dead. A webhook is marked dead when one delivery fails 6 times. Now that 1.9.79: the minutely cron sends LTV webhooks, within a budget #273 sends deliveries, that is about an hour of retries. Nothing in the API, the UI or either CLI could make it active again. LTV Settings' tooltip said "fix the endpoint and re-register". The Landing Page Optimizer's pairing stores the webhook's id and the t202ctx key derived from its secret, so re-registering meant pairing again.
• A dead webhook's events were lost. enqueue() queued nothing for a dead webhook, so every event during the outage was gone. The deliveries it had pending were spent: endpoint not active used up their remaining attempts, so even a database edit back to active brought nothing back.

What changed

• While dead, events are held. MysqlWebhookRepository::enqueue() queues for active and dead webhooks. When the dispatcher reaches a dead webhook's delivery, it holds it (holdDelivery()): the delivery becomes failed, keeps the attempts it had, gets held: its webhook is dead, so it was not sent; … as its last response, and is never tried.
  • The hold asks the webhook's status again in its own UPDATE … JOIN (w.status = 'dead'). So a delivery the dispatcher read while the webhook was dead, but which was revived since, is not held: the dispatcher sends it. There is a test for that interleaving.
  • Queueing these as ordinary pending rows, rather than writing them as held, means enqueue() needs no lock. An event queued while a revival commits is never held for a webhook that is active by then.
• MysqlWebhookRepository::revive($userId, $webhookId, $requeue, $now) first takes the dispatcher's lock (WebhookDispatcher::betweenRuns()), waiting up to 20 s for a run in flight; a run still sending then is an LtvConflictException (409) and nothing changes. It then runs in one transaction that locks the webhook row (FOR UPDATE):
  • the webhook becomes active with the same id, URL and secret;
  • with $requeue (the default), every delivery of it that is failed and was queued within WebhookDispatcher::MAX_AGE_SECONDS (1.9.79: the minutely cron sends LTV webhooks, within a budget #273's 7 days) becomes pending again, with 0 attempts, due at its queueing. That covers the delivery that killed it and the held ones. Its pending deliveries are sent too: their attempts go back to 0, so the next failure does not kill it again at once ("reviving resets the failure count": a webhook dies on the 6th failure of one delivery, and there is no per-webhook counter), and each keeps its due time. The dispatcher sends them oldest first, each with its own X-P202-Delivery. A delivered row is never touched, so nothing that was delivered is sent twice. Older rows stay failed;
  • without $requeue, every delivery of it still pending is held as the dead path holds them: failed, its attempts as they were, held: its webhook was revived for new events only, so it was not sent;
  • an already active webhook is left as it is (revived: false). Another account's webhook is a RecordNotFoundException.
• PUT /ltv/webhooks/{id} (ltv:write) takes {"status": "active", "requeue": true|false}.
  • Any other key is refused (PayloadKeys::refuseUnknown). Any other status is a 422 on status, because the server alone marks a webhook dead and DELETE removes one. requeue is read with RequestFlag, so "maybe" is a 422 on requeue.
  • It answers the webhook (never its secret) with revived, requeued, attempts_reset, held and requeue. Another account's webhook is a 404, and a revival behind a run still sending after 20 s a 409 (the Go CLI's hint: run it again in a minute).
  • Like every LTV write, it cannot be staged and has no dry_run; the dispatcher refuses both as before.
• LTV Settings. A dead webhook's row has a Revive button. Its confirm says how many deliveries from the last 7 days it will send again (missedCounts()). It posts the page's token like the Delete button, and the page answers "Webhook #N is active again; K delivery(ies) it missed in the last 7 days will be sent on the next cron run." The dead pill's tooltip now says to fix the endpoint, then Revive.
• Go CLI. p202 ltv webhooks revive <webhook-id> [--requeue=false], with a search task ("revive webhook", "reactivate webhook", …).
• PHP CLI. bin/p202 ltv:webhook:revive <id> [--no-requeue]. The PHP CLI had no webhook command before.
• Docs. documentation/api/20-ltv.md ("A dead webhook"), docs/openapi.yaml (the PUT, and "held" on the delivery log), docs/cli-agent.md, documentation/cli/10-go-cli.md, and both changelogs.

Decision: re-queueing is the default

requeue defaults to true:

• A webhook dies because its receiver was down. The events it missed are what keep a subscriber consistent: the optimizer's conversions, a CRM's revenue.
• The 7 days are the window the dispatcher already applies to anything unsent (1.9.79: the minutely cron sends LTV webhooks, within a budget #273's reasoning: a week-old event is still worth having, a months-old one is not).
• Only rows that were never answered 2xx go again, each under its own delivery id. A receiver that did take one without answering 2xx de-duplicates it, as the docs already ask.

"requeue": false (--requeue=false, --no-requeue) is for a receiver that was replaced and wants new events only.

Fix after review: requeue=false and the deliveries still pending (7ac4aa9)

Codex found this on #296, which is stacked on this PR (thread). With requeue=false, revive() only reset the pending rows' attempts. The next run then sent every delivery queued while the webhook was dead that no run had held yet, and every one waiting out a retry's backoff: "new events only" sent old ones. Against a78e59a the new test's revival answered attempts_reset: 1 and left both rows pending.

• requeue=false now holds every pending delivery of the webhook inside the revival's transaction, as above. The answer's held says how many.
• requeue=true sends the pending ones. Each keeps its due time, so one in backoff waits at most 32 minutes. A claim (claimDelivery(), next_attempt_at = now + 300) reads the same as a backoff, so a due time is never shortened. The failed ones it sends again are now due at their queueing rather than at the revival. Before, a delivery queued while dead that no run had held yet went ahead of older ones it sent again: the new test received them as 2, 1.
• A delivery a run has claimed. recordAttempt() has no status condition. A hold made while the run was sending would have been overwritten by the run's record, so a held delivery was sent anyway. And a requeued row made due could be claimed again during the first send. The revival therefore takes the dispatcher's own lock and waits for the run. The dispatcher sends only under that lock (runExclusive(), from index.php and ltv_webhooks.php), so no claim is in flight while the revival changes rows. A run still sending after 20 s is a 409 and nothing changes. A revival on the run's own session re-enters the lock (testARevivalBetweenTheDispatchersReadAndItsHoldIsSent does that, standing in for a run the lock cannot see).

The Landing Page Optimizer

The pairing in 202-account/api-integrations.php creates a wildcard webhook. It stores webhook_id and ctx_key = HMAC(secret, 't202ctx-v1') in lpo_bridge_config, and hands the secret to the SaaS once. After that, three paths read the webhook by id regardless of its status, and none of them changes on a revival:

• DimensionSync::pushForUser() (the secret, by id and user);
• bridge_config.php (the secret checks the config's signature);
• the redirect path's ctx tokens (the stored ctx_key).

The only thing the death stopped was the events, and a revival restores them. testARevivedLpoPairingWebhookGoesOnWithoutPairingAgain tests this:

• a pairing state as the page writes it, with the webhook dead;
• EventBridge::emit() while dead is held;
• the revival through LtvController::updateWebhook() sends it again;
• the receiver gets it, and a new event, signed with the secret the pairing handed over;
• lpo_bridge_config still names the same webhook, and its ctx_key still equals the key derived from the webhook's stored secret.

Tests

• tests/Ltv/WebhookReviveIntegrationTest.php (@group integration, scratch database, strict sql_mode, 1.9.79: the minutely cron sends LTV webhooks, within a budget #273's loopback receiver), 11 tests:
  • killed the real way (the last attempt fails); events queued while dead are held, not tried; a revival sends the killer and the two held ones once each, oldest first, and not the delivery made before; the run after sends nothing;
  • requeue false: only new events are sent;
  • a pending row at revival time, requeue false: held (failed, its attempts, the held: reason), and only the event queued after the revival is sent (7ac4aa9);
  • the same, requeue true: sent once, oldest first with those sent again, and the one in backoff at its time, with 0 attempts (7ac4aa9);
  • a revival from a second session while a run has claimed a delivery and is sending it: refused (LtvConflictException), nothing held, the delivery recorded once; after the run the revival goes through (7ac4aa9);
  • the failure count reset: one more failure does not kill it again;
  • only the last 7 days are sent again;
  • an active webhook is left as it is, and another account's webhook is not found;
  • a revival landing between the dispatcher's read and its hold (the clock callback revives it mid-run): it is sent, not held;
  • PUT through LtvController: the answer, no secret, the refusals (status missing, dead, disabled, requeue: maybe, an unknown key), another account's 404;
  • the LPO pairing (above).
• go-cli/cmd/system_admin_test.go TestLtvWebhookReviveSendsTheRevival: the PUT and its body with and without --requeue=false; a bad id and --requeue=maybe refused before anything is sent (exit 1); a 404's hint names p202 ltv webhooks list; a 409's hint says to run it again in a minute, not "update it instead". The existing TestEveryLtvWriteRefusesStagedBeforeSendingAnything picks the new command up: 26 LTV writes refuse --staged.
• tests/Cli/Commands/LtvWebhookReviveCommandTest.php: the PUT with and without --no-requeue; a bad id refused before any request.
• Updated tests: WebhookDispatcherTest (the summary's held), and WebhookDispatchIntegrationTest's report type. Its disabled case keeps the old endpoint not active path, which disabled still takes.
• Plants: 19 on a78e59a and 6 on 7ac4aa9, each with the file backed up, compared with cmp and restored. All were caught by a named test.
  • 7ac4aa9: pending rows reset instead of held (…StillPendingAreHeld); sent-again rows due at the revival (…StillPendingAreSentOnce, order); pending rows made due now (the same test, "keeps its time"); the revival not waiting for the lock (testARevivalWaitsForARunThatHasClaimedADelivery); the lock never released (the same test, which errors with the revival's own LtvConflictException); and the Go 409 hint dropped (TestLtvWebhookReviveSendsTheRevival).

The 19 plants

plant	caught by
a dead webhook is queued nothing	testADeadWebhookHolds…, testOnlyTheLastSevenDays…, testARevivedLpoPairingWebhook…
the dispatcher tries a dead webhook's delivery	same three
the old path: a dead webhook's delivery spends an attempt	testARevivalBetweenTheDispatchersReadAndItsHoldIsSent
the hold does not ask the webhook again	testARevivalBetweenTheDispatchersReadAndItsHoldIsSent
revive does not reset attempts	testARevivalStartsTheFailureCountAgain
revive requeues without the 7-day window	testOnlyTheLastSevenDaysAreSentAgain
revive requeues delivered ones too	testADeadWebhookHolds…
revive does not requeue	testADeadWebhookHolds…, testOnlyTheLastSevenDays…, testARevivedLpo…, testPutRevives…
revive ignores the requeue flag	testPutRevivesThroughTheController, testWithoutRequeueOnlyNewEventsAreSent
revive not tied to the account	testAnActiveWebhookIsLeftAsItIsAndAnotherAccountsIsNotFound
revive of an active webhook requeues	same, testPutRevives…
revive leaves the webhook dead	six of the eight
the API takes any status	testPutRevivesThroughTheController
the API takes any key	testPutRevivesThroughTheController
the API always requeues	testPutRevivesThroughTheController
the API answers the secret	testPutRevivesThroughTheController
PHP CLI ignores --no-requeue	LtvWebhookReviveCommandTest::testItRevivesWithTheMissedDeliveriesUnlessToldNot
PHP CLI sends a bad id	LtvWebhookReviveCommandTest::testABadIdIsRefusedBeforeAnyRequest
Go CLI always sends requeue: true	TestLtvWebhookReviveSendsTheRevival

Live pass

tests/live/ltv-webhook-revive.sh (new) ran on an instance from this branch: PHP 8.3.6 php -S, MariaDB 10.11, port 8240. The receiver setup is #273's: a Python https receiver behind socat at an address put on lo (6.7.8.42:8443), with a CA made for the pass and trusted through -d curl.cainfo. On a78e59a, 29 passed. On 7ac4aa9, 36 passed, 0 failed, on an instance installed by the installer's own steps without its licence form (see "Licence"):

1. Killed. A revenue event was delivered. With the receiver stopped, the next event's delivery failed 6 times: failed/6, the curl error, webhook dead. Between ltv_webhooks.php runs the pass sets that delivery due again, which is what the backoff's minutes would do.
2. Held. Two events queued while dead: the run printed 2 held for a dead webhook, each delivery is failed/0, and GET /ltv/webhooks/{id}/deliveries shows held: …. The receiver still had only the first delivery.
3. Revived through the API.
   • A third event was queued and still pending at the revival (7ac4aa9).
   • PUT /ltv/webhooks/{id} {"status":"active"} answered revived: true, requeued: 3, held: 0, with the same webhook id, the webhook active, and no secret in the answer.
   • The next php 202-cronjobs/index.php run (the installer's line, with this minute's record removed) delivered all four. Each was received exactly once, oldest first (killer, the two held, the pending one), recorded delivered, and signed with the webhook's original secret.
   • The delivery made before the outage was not sent again, and the run after sent nothing.
4. Revived through the PHP CLI. Killed again, an event still pending, then bin/p202 ltv:webhook:revive <id> --no-requeue --json: revived: true, requeued: 0, held: 1, active. The pending one is failed/0 with held: its webhook was revived for new events only… (7ac4aa9). A new event was sent; neither the missed one nor the pending one was, and both stay failed.

The 409 ran over HTTP on 7ac4aa9: with another MySQL session holding the dispatcher's lock, PUT /ltv/webhooks/{id} answered 409 after 20 s ("LTV webhook deliveries are being sent right now, … nothing was changed. Try again in a minute."), and the webhook stayed dead. Once the lock was released, the same PUT revived it.

The LTV Settings Revive button ran separately, on a78e59a only: step 5 of the script needs P202_PASS, and see "Licence" below. 7ac4aa9 changes nothing on that page; the button always revives with requeue true. On a dead webhook with two failed deliveries in the last 7 days, 10 passed, 0 failed:

• the row has the Revive button, its confirm names the 2 missed deliveries, and the pill says "Fix the endpoint, then Revive.";
• a POST without the token is refused in the page's own sentence, and the webhook stays dead;
• the POST the button's script makes (action=revive_webhook, the page's token) answered "Webhook no setup menu bar in clickserver screen  #3 is active again; 2 delivery(ies) it missed in the last 7 days will be sent on the next cron run.";
• the webhook is active, both deliveries are pending/0, and the row no longer has the button;
• webhook_id=12abc is refused ("Pick the webhook to revive from the list.").

Licence (the stub is down)

Since the 10/28 restart the licence stub is down. The made-up key is refused by the production licence service.

• Go CLI. Its requests get a 402 from the instance, so p202 ltv webhooks revive did not run end to end. The API path it calls (the PUT) ran in the live pass, and the command is covered by its unit test against a recording server.
• The web UI. Every signed-in page redirects to api-key-required.php. For the button pass I appended valid_key|b:1; to that one scratch session's file, which is what the licence answer sets on a valid key. The pass then ran through the real page; no code was changed for it. The button was not clicked in a browser. What ran is the POST its script makes, with the page's own token.
• The PHP CLI has no licence gate and ran.
• The instance for 7ac4aa9's pass was installed with install.php's own steps (schema, account, REST key) run from a scratch script, without its licence check, because the installer's POST refuses the made-up key. The REST API and the PHP CLI need no licence answer.

Verification

.claude/skills/p202-verify/scripts/verify.sh --since origin/1.9.79-ltv-webhooks-scheduled --changed, on 7ac4aa9's tree (and on a78e59a before it, with the same result), PHP 8.3.6, with a scratch MariaDB 10.11 database for the schema tier. The unit tier ran 6,116 tests with 5 skipped: the 4 documented "User class requires DB singleton" skips, and CronEntryPointsFailLoudlyTest's executed case, which skips while the worktree holds the live instance's 202-config.php.

Scope report
------------
  go         PASS  ran under CI's go1.25.14 via GOTOOLCHAIN (local go is 1.24)
  golangci   PASS  ran under CI's go1.25.14 via GOTOOLCHAIN (local go is 1.24)
  openapi    PASS
  patterns   PASS
  phpcs      PASS
  phpstan    PASS
  schema     PASS
  syntax     PASS
  unit       PASS

Not covered by this script:
  live      stand up an instance, drive the path a user would take
  agent     add and run a case under tests/fixtures/agent-eval/cases/
  ci        a workflow still in_progress has not passed; read the run

Report the lines above verbatim. Do not write "tests pass".

Beyond the script, against fresh scratch MariaDB 10.11 databases, one suite each:

suite	result
WebhookReviveIntegrationTest	11 tests OK (7ac4aa9)
WebhookDispatchIntegrationTest	15 tests OK (7ac4aa9)
LtvDatabaseIntegrationTest	43 tests OK
ConversionBridgeIntegrationTest	5 tests OK
AccountAdministrationIntegrationTest	13 tests OK

shellcheck --severity=warning on the new live script: clean.

Not run:

• the Go CLI against the instance (licence, above);
• the Revive button in a browser (the POST ran on a78e59a, under the planted valid_key), and step 5 of the live script on 7ac4aa9;
• an agent-eval case;
• CI's MySQL 8.0;
• the full tests/run-integration-suites.sh;
• CI, until it runs on this PR.

For the reviewer to check

• Held rows are failed. No schema change: a held delivery is failed with a held: … reason and its attempts as they were. A pending status with a parking condition would have made every dispatcher read scan past a dead webhook's backlog every minute.
• A dead webhook now accumulates rows. It gets one delivery row per event, as an active one does. Nothing prunes 202_ltv_webhook_deliveries (1.9.79: the minutely cron sends LTV webhooks, within a budget #273 found that). A webhook left dead for good keeps adding held rows; only the last 7 days of them can ever be sent.
• disabled is unchanged. No code writes it. It keeps the old endpoint not active path, and revive() accepts it like dead, but the API offers only active.
• Two held: reasons. A delivery held for a dead webhook says reviving sends it. One a requeue=false revival set aside says the webhook was revived for new events only. For the second, the webhook is already active, so the first reason's "reviving … sends it" would not be true. Both start with held:.
• A revival waits for the dispatcher (7ac4aa9). It takes the dispatcher's lock, so a web request can wait up to 20 s behind a run that is sending (a run is at most RUN_SECONDS plus one delivery's timeout from index.php, or --budget from ltv_webhooks.php), and then answers 409. The guarantee that no claimed delivery is held or made due rests on that lock, as the dispatcher's one-run-at-a-time already does. A run that cannot see the lock (a server where GET_LOCK is not shared) is protected only as far as the claim's compare-and-set protects it.
• Pending rows keep their due time under requeue. Their attempts go back to 0, but next_attempt_at stays, since a claim cannot be told from a backoff on the row. A row in backoff when the webhook died is sent at most 32 minutes later rather than on the next run.
• Merging with 1.9.79: LTV webhooks go through the export webhooks' guard and sender #278 (J0F7). This PR does not call OutboundUrlGuard, and it changes neither post(), requestOptions(), headers(), create() nor assertUrlAllowed(). In WebhookDispatcher it changes the class docblock, run()'s report and its not-active branch, summary() and didWork(). 1.9.79: LTV webhooks go through the export webhooks' guard and sender #278's resolution replaces post() and drops requestOptions()/headers(). In MysqlWebhookRepository it adds HELD_REASON under MAX_ATTEMPTS, changes one line of enqueue()'s SELECT, and adds three methods at the end of the class. 1.9.79: LTV webhooks go through the export webhooks' guard and sender #278 changes create(), removes assertUrlAllowed() and adds urlRefusal()/PORTS. Expect at most adjacent-line conflicts near the constants, and the changelogs.
• The LPO card does not say its webhook is dead. My Account › API Integrations still shows the pairing as "Connected" while its webhook is dead; the revive is in LTV Settings. I left that card alone.

Found outside this item (not fixed here)

• The LPO pairing card (202-account/api-integrations.php) shows "Connected" whatever its webhook's status, so an operator is not told the optimizer has stopped receiving events.
• LTV Settings' Delete button posts webhook_id, which the page casts with (int), so 12abc deletes webhook 12 (CLAUDE.md subid missing in pixel url #4). The new Revive action reads the id strictly.

https://claude.ai/code/session_01VRoS2SViuxaucFaKWsFnLr

---

Generated by Claude Code

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	⚠️ Failed 2026-10-28T05:57:33.920842Z We couldn’t complete this review. Comment @codex review to try again.	a78e59a	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

[github-actions]

Code review: CLEAN

For a78e59a: 0 issue(s). The review is encrypted for the maintainers in the review-findings artifact of this run.

[github-actions]

Adversarial review: NON-BLOCKING

For 4a56260: 3 finding(s). The report is encrypted for the maintainers in the review-findings artifact of this run.

Rewritten on every push; the check fails on BLOCKING.