Diagnose the wrong Expired status and the concurrent refresh defects

[spa5k]

Summary

Connections show the health status Expired. The status is sometimes wrong. The status is sometimes permanent. This pull request gives the analysis and the tests that show each defect. It does not change runtime behavior.

This pull request is a draft. It gives a diagnosis for review. It does not give a fix. The plan document proposes the fixes in five phases. Each phase has an acceptance test in this pull request.

Files

File	Content
plans/oauth-refresh-and-expired-status.md	The analysis. It gives eight causes in rank order. It gives evidence with file and line numbers. It gives the host scope, the fix phases, the invariants to preserve, and the pull request boundaries.
packages/core/sdk/src/oauth-expired-status-repro.test.ts	The tests for causes 1, 2, and 3.
packages/plugins/mcp/src/sdk/mcp-liveness-second-spawn.test.ts	The tests for cause 4.
packages/plugins/mcp/src/sdk/stdio-single-instance-test-server.ts	A fixture. It is a stdio MCP server. It does not start when a live process holds its lock. Chrome DevTools MCP has this shape.

Test convention

Each cause has two tests.

The first test shows the behavior on main today. This test passes.

The second test gives the required behavior after the fix. This test fails on main. Therefore the test suite skips it. The pull request that makes the fix removes the skip. The test must then pass without changes.

Cause 1: two refreshes at the same time make a valid connection permanently Expired

Test file: oauth-expired-status-repro.test.ts.

Two executor instances use one database and one credential store. Each instance has its own database handle. The cloud app has this shape.

The sequence is:

1. Instance A reads the stored refresh token.
2. Instance B sends a refresh grant with the same token.
3. The authorization server rotates the token. Instance B stores the new token.
4. Instance A sends a refresh grant with the consumed token.
5. The server answers invalid_grant.

The result on main:

• The system writes the permanent rejection record oauthReauthRequiredAt. The stored refresh token is valid. The record is wrong.
• All surfaces answer expired. No surface probes the upstream.
• The access token expires later. Instance B then cannot refresh. The system sends no more grant requests. The AS receives zero requests.

Cause 2: one temporary HTTP 429 response ends the refresh permanently

Test file: oauth-expired-status-repro.test.ts.

A fixture endpoint answers the first refresh grant with 429 Too Many Requests. The fixture forwards all later grants to the healthy authorization server.

The result on main:

• The health status is expired.
• The next call sends no grant. The endpoint is healthy at that time.

Cause 3: the health probe does not refresh before it answers Expired

Test file: oauth-expired-status-repro.test.ts.

The integration has a declared health check. The access token has a long advertised lifetime. The upstream revokes the token.

The result on main:

• The probe receives a 401 response. It writes expired. It sends zero refresh grants.
• The next tool call refreshes the token. The call succeeds. The system writes healthy to the same row.
• The status changes from Expired to healthy in seconds. The user does nothing. This is the reported symptom.

Cause 4: the MCP health probe makes a second connection

Test files: mcp-liveness-second-spawn.test.ts and stdio-single-instance-test-server.ts.

checkHealth builds a new connector. It does not use the pooled connection that tool calls use. For a stdio server, the probe starts a second child process.

The result on main:

• One server instance runs and holds the lock. Tool calls work.
• The probe starts a second child process. The spawn log shows it.
• The second process does not start. The probe answers degraded for a live server.
• The UI probes again on every mount for a non-healthy status. Each probe starts another child process.

Other causes in the plan

The plan gives four more causes. This pull request has no tests for them.

• Cause 5. No health check spec exists. The system answers healthy after it reads the token from the store. It does not contact the upstream.
• Cause 6. A refresh response without expires_in sets expires_at to null. Proactive refresh then never runs again.
• Cause 7. A 403 response for an insufficient scope reads as Expired. A GraphQL text match on the word permission also reads as Expired.
• Cause 8. The refresh skew is 60 seconds. No background refresh exists. The probe gate is per request in the cloud app.

Host scope

• The local app makes one executor with one database handle. The in-process refresh lock works there. Cause 1 does not occur in the local app.
• Cause 1 occurs in the cloud app and in multi-process self-hosting.
• Causes 3 and 4 occur in a single-process local app. These two causes match the reported symptoms.
• Cause 2 occurs in all hosts.

Linked issue

None. The plan proposes one issue per phase.

Verification

• bun run format:check — all files have the correct format.
• bun run lint — 0 warnings and 0 errors in 1869 files. The changelog stub check passes.
• bun run typecheck — I ran the command for the two changed packages. No errors.
• bun run test — I ran the command for the two changed packages. sdk: 923 passed, 3 skipped, 65 files. plugin-mcp: 311 passed, 30 skipped, 41 files. No existing test changed.
• e2e — I did not run an e2e scenario. This change does not affect user-visible behavior. The plan names the two scenarios to add with the fixes.

Run the tests:

cd packages/core/sdk && npx vitest run src/oauth-expired-status-repro.test.ts      # 3 passed | 3 skipped
cd packages/plugins/mcp && npx vitest run src/sdk/mcp-liveness-second-spawn.test.ts # 1 passed | 1 skipped

Remove one skip to see that test fail on main.

Checklist

• No changeset is necessary. This change adds tests and one document. It does not change published behavior.
• Tests are added. The fix pull requests use the skipped tests.
• The diff has no secrets, no credentials, and no private data. The fixtures make their own tokens.

Notes for review

• oauth-flow.test.ts already makes the two-instance race. That test examines the credential store only. It does not examine the connection row. This is the reason nobody found cause 1 before.
• The permanent rejection record is a deliberate design. It stopped a real incident with unbounded rejected grant requests. This plan does not remove it. Phase 1 requires more evidence before the system writes it. Phase 2 adds coordination in the database. The refreshGateFor documentation already recommends that coordination.
• Please review two decisions. First, is the skipped-test convention acceptable? The alternative is to keep the acceptance tests out of the repository until each fix pull request. Second, should cause 4 go first? It is the only fix that addresses the reported local symptom on its own. It does not change OAuth code.

Stack

A fix pull request is stacked on this branch:

• This pull request (Diagnose the wrong Expired status and the concurrent refresh defects #2036) — the diagnosis: the analysis, the reproductions, and the plan.
• spa5k/executor#1 — the fix for causes R1, R2, and R3. It un-skips the three acceptance tests and they pass.

The fix pull request is open in the fork, because a stacked pull request needs
its base branch in the same repository, and the base branch exists only in the
fork. After this pull request merges, retarget that one to
UsefulSoftwareCo/executor:main.