Skip to content

feat: actor deregistration with safety checks, CLI, admin UI (#56) - #63

Open
rcbevans wants to merge 20 commits into
mainfrom
spec/actor-deregistration
Open

feat: actor deregistration with safety checks, CLI, admin UI (#56)#63
rcbevans wants to merge 20 commits into
mainfrom
spec/actor-deregistration

Conversation

@rcbevans

@rcbevans rcbevans commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds client.actors.deregister() — a first-class actor deregistration API with defined safety semantics — so ephemeral per-run actor deployments can clean up their actor_config and orphaned queues rows without hand-rolled SQL. Also adds the taskq actor-config deregister CLI command and an /admin/actors page with a deregister button. The default path refuses deregistration while non-terminal jobs or enabled cron schedules reference the actor; force=True cancels pending/scheduled jobs, disables schedules, and proceeds (running jobs always block).

Closes #56.

Implementation

deregister_actor() — ops layer (src/taskq/actor_config_ops.py)

Transaction-scoped function (conn.transaction()) with two modes:

  • force=False (default): refuses if any non-terminal jobs (pending/scheduled/running) reference the actor (ActorHasActiveJobsError with per-status counts), or if any enabled cron schedules reference it (ActorHasEnabledSchedulesError with schedule IDs). If both checks pass, deletes the actor_config row.
  • force=True: refuses only if running jobs exist (running jobs are actively executing; the dispatch query inner-joins actor_config, so a running job that retries would be stranded, and the terminal-write path's COALESCE for result_ttl would silently fall back to the @actor(...) literal). Cancels pending/scheduled jobs (marks as cancelled with error_class='ActorDeregistered'), disables enabled cron schedules (sets enabled=false — not delete, so the operator can re-enable if the actor is re-registered), then deletes the actor_config row.

Terminal job history (succeeded/failed/cancelled/crashed/abandoned) is never deleted or modified — jobs.actor is plain text, not an FK, so terminal rows remain queryable by actor name after deregistration. DeregisterResult.terminal_jobs_remaining reports the count.

purge_queue=True deletes the orphaned queues row only when no other actor_config row references the same queue name (shared queues are never purged).

ActorNotFoundError is raised if no actor_config row exists.

The TOCTOU race under READ COMMITTED is documented: callers must quiesce the actor (stop enqueuing, disable cron, wait for running jobs to terminate) before calling deregister.

Exception hierarchy (src/taskq/exceptions.py)

TaskQError
  └─ ActorDeregistrationError
       ├─ ActorHasActiveJobsError      (carries active_count, status_counts)
       ├─ ActorHasEnabledSchedulesError (carries schedule_ids)
       └─ ActorNotFoundError

ActorsClient — pool-wrapping facade (src/taskq/client/_actors.py)

Acquires a connection from the injected asyncpg.Pool per call, delegates to actor_config_ops. Methods: list(), get(), set_capacity(), deregister().

TaskQ.actors property (src/taskq/client/_taskq.py)

Returns the ActorsClient instance (created during open()). Raises RuntimeError if called before open().

CLI (src/taskq/cli.py)

taskq actor-config deregister <ACTOR> [--force] [--purge-queue]

Exit code 0 on success (prints DeregisterResult summary), 1 on refusal or not found.

Admin UI (src/taskq/web/admin/actors.py, actors.html)

  • GET /admin/actors — lists all actor_config rows with active job counts and enabled schedule counts
  • POST /admin/actors/{actor}/deregister — deregister with force and purge_queue form params; CSRF-protected; gated on admin_actions_enabled; returns 404 for unknown actor, 409 for refusal, 303 on success

Public API exports (src/taskq/__init__.py)

ActorsClient, DeregisterResult, ActorDeregistrationError, ActorHasActiveJobsError, ActorHasEnabledSchedulesError, ActorNotFoundError are importable from taskq.

Breaking changes

  • actor_config.py and actor_config_ops.py moved from taskq.worker. to the top-level taskq. package. Update imports: from taskq.actor_config_ops import deregister_actor (was from taskq.worker.actor_config_ops import ...).
  • worker/_transient.py removed. TRANSIENT_PG_ERRORS and UnexpectedLoopErrorGuard no longer exist. Call sites in leader.py, _leader_sweeps.py, heartbeat.py, run.py now inline the specific exception tuples they need.
  • dispatcher_command_timeout setting removed from WorkerSettings. Leader loops no longer wrap iterations in asyncio.timeout(); dedicated connections no longer set command_timeout.
  • watchdog_dump_after_fraction setting removed from WorkerSettings. Straggler dumps now fire immediately during shutdown countdown.
  • test_watchdog_safety.py deleted (1969 lines) — tested the removed UnexpectedLoopErrorGuard / dispatcher_command_timeout / dump_after_fraction surfaces.

Test coverage

File Tier Coverage
tests/test_exceptions.py Unit Exception constructors, message format, inheritance hierarchy (TaskQErrorActorDeregistrationError → specific errors)
tests/test_actor_deregistration.py PG integration force=False refusal paths (pending/running jobs, enabled schedules), force=True (cancels pending/scheduled, disables schedules, refuses running, keeps terminal history), purge_queue (orphaned, shared, noop when queue row absent), idempotency (double deregister), combined force+purge, concurrent deregistration
tests/test_actors_client.py Unit ActorsClient delegation wiring (list/get/set_capacity/deregister) with fake pool; error propagation
tests/test_actor_deregistration_client.py PG integration Full client path through TaskQ.actors — deregister clean, not found, active jobs refusal, force cancels, purge queue, double deregister, list, get
tests/test_cli_actor_deregister.py Unit (CLI) Default args, --force, --purge-queue, not found exit 1, active jobs error exit 1, schedules error exit 1, output format, double deregister
tests/test_web_admin_actors.py PG integration Actors page lists rows, deregister form present, 403 when admin actions disabled, deregister succeeds (303), 409 with active jobs, force cancels pending, 404 for unknown actor, notice banner
tests/test_taskq_actors_property.py Unit + PG Public API imports, tq.actors raises before open(), returns ActorsClient after open()
tests/e2e/test_actor_deregistration.py E2E (real worker) Deregister after jobs complete, refuses with running jobs, force+purge after completion

Concurrency safety

test_concurrent_deregister_one_succeeds_one_raises — two concurrent deregister_actor calls on the same actor via separate asyncpg connections. Under READ COMMITTED, both pass the safety checks, but only one DELETE ... RETURNING returns a row; the other gets 0 rows and raises ActorNotFoundError. The transaction rolls back cleanly on the losing side.


Closes #56

@rcbevans
rcbevans requested review from XBeg9, clinzy and kjw-azx July 30, 2026 01:36
@rcbevans rcbevans self-assigned this Jul 30, 2026
Base automatically changed from feat/e2e-test-suite to main July 30, 2026 04:32
rcbevans added 20 commits July 29, 2026 21:54
Adds GET /actors (lists actor_config rows with active job counts and
schedule counts) and POST /actors/{actor}/deregister (deregister with
force + purge_queue form params, gated on admin_actions_enabled).

- src/taskq/web/admin/actors.py: route module following the workers.py
  pattern with CSRF validation and admin_actions_enabled gate
- src/taskq/web/templates/actors.html: table view with per-row
  deregister form (force/purge_queue checkboxes, confirm dialog)
- src/taskq/web/templates/_base.html: Actors nav link after Workers
- tests/test_web_admin_actors.py: 4 integration tests covering page
  listing, form rendering, 403 on disabled actions, and successful
  deregister redirect + row deletion
- pyproject.toml: S608 per-file ignore for the new test file
These modules are shared by client, CLI, admin UI, testing, and __init__ —
not worker-internal. Relocating them from taskq.worker.* to taskq.* fixes
the layering violation where client code reached into the worker package.
- M1: Admin UI now consumes notice query param and shows success banner
- L1: Remove unused structlog loggers from _actors.py and admin/actors.py
- Add missing 409 test for deregister with active jobs
- Add full-stack integration tests via TaskQ.actors client path
…es, schema assertion, edge-case tests

L1: Document admin UI limitation for actor names containing '/'
L2: Whitelist notice query param to fixed messages (XSS prevention)
L4: Fix warning wording — sweep transitions to terminal, not heals
L5: Add cron fire/re-enable race warning to docstring
L6: Add concurrent deregistration orphaned-queue warning to docstring
L7: Type-annotate status_counts as dict[str, int] with explicit casts
L10: Document ActorsClient as Postgres-only in class docstring
L11: Clarify ActorNotFoundError scope in docstring
L12: Distinct CLI exit codes: 2=refusal, 3=not-found (was: 1 for all)
L15: Add set_capacity None-vs-UNSET forwarding test
L16: Add refusal gap tests: scheduled-only, jobs+schedules precedence, invalid schema
L17: Assert WARNING message and exception prefix in CLI tests
L18: Add negative CSRF test for deregister route
L19: Strengthen property test schema assertion, replace admin_pool with module_pg_pool, tighten e2e >=1 to ==1
@rcbevans
rcbevans force-pushed the spec/actor-deregistration branch from 57b45a3 to 048da63 Compare July 30, 2026 04:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant