[tracking202]

Backlog: BR3T

What was wrong (measured)

PendingClickSettler::run() selected the 500 oldest pending_click installs (ORDER BY received_at, install_row_id LIMIT 500) and left an install whose settle threw where it was, "retried next run". Some throws give the same answer on every run: a stored token the install-token key no longer verifies (the key was replaced after the install arrived), or a stored body that no longer reads. 500 of them were the whole batch every run, so no other app's install settled again. Installs still waiting for a click that never came held the batch the same way, for up to the 24-hour deadline.

Measured on a live instance from origin/1.9.79's settler: 501 such installs and one settleable install (its click recorded after it arrived), three runs of 202-cronjobs/app-installs.php. Every run printed pending clicks: 500 examined, settled none, 0 still pending, 500 failed and exited 1. The settleable install stayed pending_click with no conversion.

What changed

The decision is (c), both. The cursor is per row rather than one global position, and the failure count ends only a failure the row itself causes.

• Turns (the cursor). 202_app_installs gains settle_next_at (0 until tried) and settle_failures (tries in a row that failed), plus KEY settle_due (match_state, settle_next_at).

  • A run takes the due installs in index order: those never tried first, then the one whose turn came first.
  • Every install a run takes gets its next turn. One still waiting for its click is due again now, behind the others. One that failed waits backoff(): the next run, then 60 s doubling to an hour.
  • So the next run starts with the installs this one did not reach. This is the shape CLAUDE.md Remove Intercom integration #37 (in 1.9.79: company linking walks past names it cannot link #304) describes.
  • The cursor lives on the row, not in one stored position. A run that dies part way loses nothing, and the integrity worker already uses the same pattern (integrity_next_at).

• Retirement (the count), only for failures the code knows are the row's own. stored() reads the install the way the intake read it. It answers with a reason, never a throw, in exactly three cases, and each is a pure function of the stored row and a key that did load:

  • the body is not JSON;
  • the body no longer validates (it names the missing fields);
  • the token no longer verifies.

  Such an install is retried until 24 hours after it arrived, the deadline a missing click already has, so a replaced key can be put back in that time. After that it is settled as bad_token through InstallIntake::settle(), still the one writer of a match state, with match_reason saying why. Everything else is retried for as long as it lasts and never ends an install: a missing key, a key that cannot be read, a database error, a lock timeout, or any throw from the classification or the goal engine (CLAUDE.md Fix newline at end of index #11, Fix undefined mysql array keys #34).

  • Why not "after N failures": N failed runs of an unknown cause is a guess. A deploy-before-upgrade window, or a database outage, longer than N minutes would refute real installs.

• Visible where installs are read:

  • settle_failures and settle_next_at on GET /apps/{id}/installs and /installs/{uuid}, which the Go CLI prints as the API gives them;
  • pending_failing beside pending in GET /apps/report (groups and totals);
  • a warning on Analyze › Mobile Apps (Android);
  • a line under the reason in Setup › Mobile Apps' recent installs;
  • the job's summary line, which ends … N failed, M retired as unreadable, with each retirement logged.

• Cost of the order. The batch is read from settle_due: 500 rows read and no sort, by ANALYZE on MariaDB 10.11 with 2,000 pending installs among 60,000. Ordering by received_at as well sorted every pending install (2,000 read, filesort).

• Schema. The columns and the index are in AppTables::appInstalls() and in the 1.9.78 → 1.9.79 rung (self-contained: AppTables::appInstalls() added, with a one-line comment, to the rung's reconcile list). The settler cannot read the new columns before the upgrade has run, because it never runs then: connect.php stops both cron entry points while an upgrade is needed (read in the code, not executed here). The API answers 503 naming the upgrade.

• No open 1.9.79 branch touches the settler. Three touch InstallEventsIntake.php and none touch PendingClickSettler.php or InstallIntake.php, so this is not stacked.

Tests

• tests/Apps/Android/PendingClickQueueIntegrationTest.php (new, @group integration):

  • It seeds the cron's batch plus one (501, the batch read from AndroidIntakeJob::runExclusive()'s own default) of installs whose tokens another key signed. These sit ahead of an install whose click arrives. That install settles on the second run.
  • The same with 501 installs still waiting for clicks that never come.
  • Retirement at the deadline, for a token that no longer verifies, a body cut off part way and a body that is no install. Each reason fits the column whole.
  • A missing key and a database error (a trigger) are retried past the deadline and never retired.
  • The counts in the report and the install list.

• Plants, each run against a named test, each restored and verified with cmp:

  Plant	Test that failed
  The old selection (ORDER BY received_at)	both starvation tests (this is the old behaviour planted back)
  A key failure treated as unreadable	testAMissingKeyIsRetriedAndNeverRetires
  Retiring any failure past the deadline	the key test and the database-error test
  Never retiring	the deadline test
  No turn written for a waiting install	the waiting test
  No failure recorded	the batch test and the deadline test
  The settler writing match_state directly	AttributedInstallHasConversionTest
  The report metric miscounting	the visibility test

• AttributedInstallHasConversionTest now lists the settler as a writer of 202_app_installs that sets only settle_* columns directly. InstallIntakeIntegrationTest's shape assertions gain retired.

Live pass

The instance was served from a copy of this branch's tree and installed with install.php's own steps but not its licence check, because the stub is down. php -S and the cron ran with every proxy variable at 127.0.0.1:9.

• Registration, campaign and install. The app registration and the campaign were made through the API. The settleable install went through the real intake (POST /api/v3/apps/installs answered pending_click), and its click row was then written by SQL. The 501 unverifiable installs were seeded by SQL.
• Before (origin/1.9.79's settler and job in the same tree): three runs, each 500 examined … 500 failed. The install stayed pending.
• After:
  • Run 1: 500 examined, settled none, 0 still pending, 500 failed, 0 retired as unreadable.
  • Run 2: 500 examined, settled attributed=1, … 499 failed. The install was attributed with its conversion.
  • GET /apps/1/installs?match_state=pending_click showed settle_failures: 1, and GET /apps/report?platform=android showed pending_failing: 501.
  • Signed in, Analyze › Mobile Apps showed "501 of the installs waiting for their click failed to settle on their last try…", and Setup › Mobile Apps showed the note under 9 of the 10 recent installs.
• Deadline. The failing installs were aged 25 hours by SQL. The next run printed 500 examined, settled none, 0 still pending, 0 failed, 500 retired as unreadable and exited 0. The run after it retired the last one. Each install is bad_token, trusted 0, with the reason in match_reason.
• 202-cronjobs/index.php (the default scheduler's job) printed the new summary line too.

Verification

.claude/skills/p202-verify/scripts/verify.sh --since origin/1.9.79 --changed (schema tier on a scratch MariaDB 10.11 database), on the committed branch:

Scope report
------------
  openapi    PASS
  patterns   PASS
  phpcs      PASS
  phpstan    PASS
  schema     PASS
  syntax     PASS
  unit       PASS

Not covered by this script:
  live      stand up an instance, drive the path a user would take
  agent     add and run a case under tests/fixtures/agent-eval/cases/
  ci        a workflow still in_progress has not passed; read the run

The unit tier ran 6104 tests with 4 skipped (the documented "User class requires DB singleton" skips). Beyond the script:

• Every @group integration suite (tests/run-integration-suites.sh, each on a fresh scratch database): 146 suites and 1371 tests passed, and 1 suite failed. The failure was tests/Repository/MysqlLookupRepositoriesIntegrationTest.php::testTheFactoryCachesThroughTheBootstrapHelpers, with Class "Memcached" not found, because this sandbox's PHP has no memcached extension. That run overlapped edits to the settler.
• --group integration tests/Apps/Android, rerun on the final code: 185 tests, OK. This includes the new suite and the Integrity suites.
• tests/live/upgrade-equals-install.sh (1.9.55 on PHP 7.4, upgraded through the upgrade page, compared with a fresh install): 25 passed, 0 failed. Its notable lines:
  • "the upgrade logged no error or warning";
  • "compared 162 tables after the upgrade with 162 in a fresh install: identical up to index order, table comment, partition boundaries and AUTO_INCREMENT";
  • "the rows of 155 tables are identical".
    The run used the copy described below.
• The live pass above.

What could not run, and what ran without the licence

• install-instance.sh and the Go CLI could not run against the live instance: the licence stub is down, and an instance without the licence answers the CLI 402. The live pass drove the API with curl and the cron jobs on the CLI. tests/live/android-intake.sh was not run because it uses the Go CLI.
• tests/live/upgrade-equals-install.sh ran through a copy whose fresh install (step 2) used install.php's steps without the licence check, the same scratch installer as the live pass, instead of install-instance.sh. It also used one port for both phases. Steps 1 (1.9.55 installed on PHP 7.4), 3 (the upgrade page) and 4–5 (schema and rows) are the script's own. Result: 25 passed, 0 failed.
• MySQL 8.0 was not available here: the plan measurement and every database run were on MariaDB 10.11. CI's integration job runs MySQL 8.0.

For the reviewer

• Retiring to bad_token (refuted, never paid) follows OrphanedPendingClicks, which ends a pending click that can no longer be settled the same way. No new match state was added.
• The retirement waits for the 24-hour deadline rather than for N failures, so a replaced key can be put back in time. It applies only to the three causes stored() names. Every other failure is retried indefinitely, at most once an hour, and is visible as pending_failing.
• settle_failures is left as it was when an install settles, so a retired install shows how many tries failed. pending_failing counts only installs still pending_click.
• CLAUDE.md is not edited here: 1.9.79: company linking walks past names it cannot link #304 adds Remove Intercom integration #37, which is this shape, and an edit beside it would conflict. A second paragraph for Remove Intercom integration #37 could say "end only what the row determines; retry what the server does".

---

Generated by Claude Code

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	⚠️ Failed 2026-10-28T10:28:59.941079Z We couldn’t complete this review. Comment @codex review to try again.	53cfd02	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

[github-actions]

Code review: CLEAN

For 53cfd02: 0 issue(s). The review is encrypted for the maintainers in the review-findings artifact of this run.

[github-actions]

Adversarial review: NON-BLOCKING

For 2e82fbf: 4 finding(s). The report is encrypted for the maintainers in the review-findings artifact of this run.

Rewritten on every push; the check fails on BLOCKING.