fix: share authentication and payment state across replicas

README.md

@@ -97,6 +97,12 @@ as local scratch space. Each processed publication version is written to a new
 object prefix before its database rows are updated, so a failed replacement
 cannot overwrite the previously published files.
 
+Replicas also share protocol state in the application database: authentication
+sessions and one-time nonces use the published toolbox `KnexSessionManager`,
+and both page purchases and admin funding share durable payment transaction
+claims. Startup migrations create the additive tables before requests are served.
+See [the protocol-state rollout requirements](docs/devops.md#shared-authentication-and-payment-state).
+
 ## Project Layout
 
 ```text

docs/devops.md

@@ -143,13 +143,15 @@ The production deploy remains a separate, explicitly dispatched workflow.
 Production builds must pass the runtime policy on their exact immutable image
 digest. `deploy-local.sh` requires `IMAGE_TAG` and `IMAGE_DIGEST`; the deployment
 workflow supplies both from its build output and retains the scan and rollout
-evidence, including on failure. The SDK upgrade includes no database migration.
-Any future schema change needs compatibility review before using this procedure.
+evidence, including on failure. The shared protocol-state upgrade has the
+additive schema migration and compatibility requirements described below.
 
 `promote-guarded.py` creates two candidate replicas on distinct nodes behind a
 private Service, with a PDB and a copy of the existing production egress boundary.
 Both replicas must serve health, status, catalog and a real stored free-page PNG;
-an anonymous paid-page request must still be denied. The candidate's ten-second
+an anonymous paid-page request must still be denied, and a synthetic authenticated
+client must receive the configured signed 402 challenge with spending disabled.
+The candidate's ten-second
 preStop hook is exercised by withdrawing one candidate while another node serves
 100 consecutive requests. Two exact-image Ready replicas must return before
 promotion. Public root, health, catalog and stored-page probes run throughout.
@@ -169,3 +171,37 @@ delete that pool while the public Service selects it. A failure before cutover
 removes only the isolated candidate resources and preserves the old public pool.
 Network-ops fleet gates and independent public probes remain required around
 the workflow.
+
+## Shared authentication and payment state
+
+Migration `202609240001_shared_protocol_state.cjs` creates `auth_sessions`,
+`auth_message_nonces` and `payment_replays` before serving requests. Existing
+wallet, content, purchase and payout rows are unchanged. MySQL protocol tables
+use ASCII binary collation so distinct case-sensitive base64 nonces cannot
+collapse onto the same primary key. Every replica must use the same application
+database and server wallet identity. Session/nonce handling uses the published
+`KnexSessionManager`; initial-request claims are capped at 256 per identity.
+Expired sessions and orphaned nonces are pruned hourly with no overlapping prune
+in one process. Cleanup failure is logged without granting authentication.
+
+Both page and admin-funding payment middleware use the same atomic transaction-ID
+claim table. Claims have no expiration: pruning accepted transaction IDs could
+reopen replay of old payments. A duplicate returns false; database failures
+propagate so the middleware fails closed. Keep these rows through restarts,
+upgrades, restores and application rollback. The migration deliberately refuses
+`down`; rolling back code must not delete protocol state.
+
+Before promotion, take and verify an encrypted application-database backup.
+Rehearse the additive migration and check the MySQL table collations/primary keys.
+Old code ignores the extra tables, but old process-local sessions do not become
+shared: finish the guarded cutover before accepting replicated authentication.
+Do not remove the tables or roll back replay state while serving payments.
+
+Candidate acceptance must include a non-spending authenticated paid-page request
+that obtains a signed 402 challenge with the configured amount, with handshake
+and requests deliberately sent to different replicas. The synthetic client's
+`createAction` must throw before spending. Anonymous paid-page denial and free
+rendered routes remain required. A real bounded paid test is separate operator
+authorization and must verify the purchase ledger and entitlement, not merely an
+HTTP status. On any failure, preserve the evidence and halt the release wave;
+green free routes do not establish payment acceptance.

migrations/202609240001_shared_protocol_state.cjs

@@ -0,0 +1,48 @@
+// Additive application tables; no wallet/content rows are changed.
+exports.up = async function (knex) {
+  const caseSensitive = table => {
+    if (['mysql', 'mysql2'].includes(knex.client.config.client)) {
+      table.charset('ascii')
+      table.collate('ascii_bin')
+    }
+  }
+  if (!await knex.schema.hasTable('auth_sessions')) {
+    await knex.schema.createTable('auth_sessions', table => {
+      caseSensitive(table)
+      table.string('sessionNonce', 64).primary()
+      table.string('peerNonce', 64).nullable()
+      table.string('peerIdentityKey', 130).nullable()
+      table.boolean('isAuthenticated').notNullable()
+      table.bigInteger('lastUpdate').notNullable()
+      table.boolean('certificatesRequired').nullable()
+      table.boolean('certificatesValidated').nullable()
+      table.bigInteger('expiresAt').notNullable()
+      table.index(['peerIdentityKey', 'lastUpdate'])
+      table.index('expiresAt')
+    })
+  }
+  if (!await knex.schema.hasTable('auth_message_nonces')) {
+    await knex.schema.createTable('auth_message_nonces', table => {
+      caseSensitive(table)
+      // Also holds initial:<identity> scopes, which are not session-table keys.
+      table.string('sessionNonce', 130).notNullable()
+      table.string('messageNonce', 64).notNullable()
+      table.bigInteger('expiresAt').notNullable()
+      table.primary(['sessionNonce', 'messageNonce'])
+      table.index('expiresAt')
+    })
+  }
+  if (!await knex.schema.hasTable('payment_replays')) {
+    await knex.schema.createTable('payment_replays', table => {
+      caseSensitive(table)
+      table.string('transactionId', 64).primary()
+      table.timestamp('createdAt').notNullable()
+    })
+  }
+}
+
+exports.down = async function () {
+  // Reverting application code is safe with these additive tables retained.
+  // Forgetting payment claims would reopen replay of already accepted payments.
+  throw new Error('Retain shared protocol state when rolling back application code')
+}

package.json

@@ -37,7 +37,7 @@
   },
   "dependencies": {
     "@aws-sdk/client-s3": "^3.1130.0",
-    "@bsv/auth-express-middleware": "2.2.6",
+    "@bsv/auth-express-middleware": "2.2.7",
     "@bsv/identity-react": "^1.1.14",
     "@bsv/payment-express-middleware": "2.1.7",
     "@bsv/sdk": "2.8.4",

scripts/k8s/promote-guarded.py

@@ -121,6 +121,7 @@ def endpoints(service, expected_pods=None, timeout=60):
 
 SMOKE = r"""
 const base=process.argv[1]||'http://127.0.0.1:8080';
+const peer=process.argv[2];
 async function get(path) { const r=await fetch(base+path,{signal:AbortSignal.timeout(10000)}); if(r.status!==200)throw Error(path+': '+r.status);return r; }
 const health=await (await get('/healthz')).json();if(!health.ok)throw Error('health not OK');
 await get('/'); const status=await (await get('/api/status')).json();if(status.status!=='success')throw Error('status not success');
@@ -132,15 +133,46 @@ def endpoints(service, expected_pods=None, timeout=60):
 const rendered=Buffer.from(await (await get(view.imageUrl)).arrayBuffer());if(rendered.subarray(0,8).toString('hex')!=='89504e470d0a1a0a')throw Error('rendered PNG failed');
 const paid=await fetch(base+`/api/publications/${id}/pages/2`,{signal:AbortSignal.timeout(10000)});
 if(paid.status!==401)throw Error('anonymous paid page did not reject');
-console.log(JSON.stringify({health:true,catalog:true,freePNG:true,freeJSON:true,renderedPNG:true,paidAccessDenied:true}));
+const {AuthFetch,PrivateKey,ProtoWallet}=await import('@bsv/sdk');
+const wallet=new ProtoWallet(PrivateKey.fromRandom());let challenge=false,spendBlocked=false;
+wallet.createAction=async args=>{
+  if(args.outputs?.length!==1||args.outputs[0].satoshis!==status.pricePerPageSats)throw Error('Unexpected payment amount');
+  spendBlocked=true;throw Error('TEST_PAYMENT_DISABLED');
+};
+let sequence=0;const destinations=new Set();
+const observe=async (url,init)=>{
+  const parsed=new URL(String(url));
+  const destination=peer&&sequence++%2===1?peer:base;
+  destinations.add(destination);
+  const response=await fetch(destination+parsed.pathname+parsed.search,{...init,signal:AbortSignal.timeout(10000)});
+  if(String(url).includes('/pages/2')){
+    if(response.status!==402||Number(response.headers.get('x-bsv-payment-satoshis-required'))!==status.pricePerPageSats||!response.headers.get('x-bsv-auth-signature'))throw Error('Authenticated payment challenge failed');
+    challenge=true;
+  }
+  return response;
+};
+const auth=new AuthFetch(wallet,undefined,undefined,undefined,{},observe);
+try{await auth.fetch(base+`/api/publications/${id}/pages/2?format=json`);throw Error('Expected blocked synthetic payment');}
+catch(error){if(error.message!=='TEST_PAYMENT_DISABLED')throw error;}
+if(!challenge||!spendBlocked)throw Error('Authenticated payment probe incomplete');
+if(peer&&destinations.size!==2)throw Error('Cross-replica authentication was not exercised');
+console.log(JSON.stringify({health:true,catalog:true,freePNG:true,freeJSON:true,renderedPNG:true,paidAccessDenied:true,authenticatedPaymentChallenge:true,spendingDisabled:true,crossReplica:destinations.size===2}));
 """
 
 
 def smoke(pods):
-    for pod in pods:
+    if len(pods) != 2:
+        raise RuntimeError('Smoke acceptance requires two replicas')
+    for index, pod in enumerate(pods):
         guard()
-        command('exec', pod['metadata']['name'], '--', 'node', '--input-type=module', '-e', SMOKE)
-        record('pod-smoke-passed', pod=pod['metadata']['name'])
+        peer = pods[1-index]
+        address = peer['status']['podIP']
+        if ':' in address:
+            address = '['+address+']'
+        command('exec', pod['metadata']['name'], '--', 'node', '--input-type=module', '-e', SMOKE,
+                'http://127.0.0.1:8080', 'http://'+address+':8080')
+        record('pod-smoke-passed', pod=pod['metadata']['name'],
+               peer=peer['metadata']['name'], crossReplica=True, spendingDisabled=True)
 
 
 def main(manifest, image):