Skip to content

整合 dev 应用更新与后端数据库可靠性优化 - #18

Open
nextroad-dev wants to merge 41 commits into
mainfrom
dev
Open

nextroad-dev wants to merge 41 commits into
mainfrom
dev

Conversation

@nextroad-dev

@nextroad-dev nextroad-dev commented Oct 17, 2026 •

Copy link
Copy Markdown
Owner

修复后端重复查询、微秒分页漏行、账户删除事务失败、缓存查询放大、连接故障等待,以及多 Worker 的 Outbox/Provider 所有权问题。原有 3 个前端测试失败也已修复,包括窄屏记录卡片和弹层焦点回退。

本 PR 从现有 dev 合入 main,还包含此前未合并的前端、画布 Agent、媒体默认模型与配置更新。本轮后端优化共 13 个独立提交(首批 4 个,后续 9 个),完整实现与部署说明见 deploy/BACKEND_TUNING.md;此前 dev 变更没有在本轮重写。

实现结果:

  • 任务 outputs 批量查询;输入 URL 复用;列表使用请求内签名配置/客户端;图库和管理列表保留 PG 微秒游标并兼容旧游标。
  • API/Worker 设置共享并发刷新,最多 5 秒配置传播;本地 invalidate 使用 epoch 隔离旧读取。
  • Redis 错误监听、连接/命令预算及离线失败降级;PostgreSQL 池、锁/语句/闲置事务超时;OTP 独立邮箱/IP 预算。
  • Outbox 短 SQL 原子认领/token 确认,网络发布不持事务;默认消费并发 2;独立阻塞 Redis 连接;Provider 自动续租、失租 HTTP/S3 中止及 SQL fencing。
  • Stream 默认保留 7 天,仅清理受管唯一消费组中已确认且早于安全边界的消息;pending/未读/未知消费组保护。数据库历史工具默认 dry-run,只有 --execute 单批清理已派发且无租约历史。
  • 调度与对象清理分离;清理限批且账户完成等待旧资产删除任务;Worker 本地健康探针与持续失去进展自退出;四份 Compose 为长驻服务增加自动重启和资源配置透传。

数据库迁移和回滚:

  • 0035 增加 Outbox token/expiry;0036/0037 并发添加所有者及未派发索引;0038/0039 并发移除由唯一约束覆盖的冗余排序索引。
  • Down SQL 独立放在 packages/database/rollback/,全部已在独占 PG fixture 中逆序执行并重新构建验证。
  • 迁移专用会话覆盖请求超时并销毁连接;并发索引前后校验有效性,INVALID 时 fail-fast。
  • 回滚 0035 前须停新 Worker、回退代码;Stream/对象/显式历史删除无法由代码回滚恢复,执行前需备份。未对部署环境执行迁移或清理。

本地验证:

  • 全仓 typecheck、build、lint、token guard 通过(既有 advisory warnings)。
  • 工作区测试:contracts 67、domain 23、database 77、providers 240(另 1 跳过)、web 165、api 295、worker 85 通过,共 952 通过,无失败。
  • 全部普通集成入口 39 通过、4 个需显式服务环境用例跳过;包含真实本地 production Next 鉴权/Origin 烟雾测试。
  • 另行显式启用隔离 PostgreSQL 17.11:9 项真实迁移/事务/并发认领/过期租约/微秒分页/历史保护/回滚及计划测试通过。不使用继承的业务连接配置,每次只创建并删除自己的随机 fixture 数据库。
  • 50 任务 SQL mock 记录 52→3;暖配置缓存 100 并发读取无额外 SQL。
  • 独占合成 PG 暖缓存三次中位执行时间:10 万资产查询 13.188→0.066ms,20 万 Outbox 查询 52.132→0.063ms,计划使用新索引;不代表生产/API 性能承诺。

新增 backend-reliability.yml 使用 disposable PostgreSQL 17 / Redis 7.4 / Node 22 验证真实 Lua 保留边界、数据库并发和四份 Compose 配置,并上传完整 EXPLAIN/buffer artifact。该任务 10/10 通过、无跳过;最终提交 50a2ed7 的全部 8 项远端检查均成功,包括综合质量门禁、Web 门禁、密码轮换和四个镜像构建。CI 不能代替部署健康检查或生产场景验收;部署环境尚未核实,本 PR 保持草稿,未合并或部署。

- rename apps/web-next to apps/web (package, Dockerfile, compose, CI, nginx, docs)
- design baseline: spacing/caption/content-width tokens, 3 durations + 2 easings,
  runtime-derived exit durations, page-enter motion, new token guard rules
- shared UI: ghost-icon variant, unified press/hover/focus states, rounded-pill
- redesign public, auth, setup, workspace and admin pages
- add error, not-found, global-error and loading states
Batch A-E from the 2026-10-16 repository audit.

- deps: next 15.5.28, sharp 0.35.x, nodemailer 10.x; pin patched
  sharp/postcss/source-map-js via pnpm-workspace.yaml overrides
  (pnpm audit --prod: 35 -> 0)
- gates: add root eslint flat config and per-package lint, web test
  script, typecheck for database/domain/config; wire lint into CI
- boundaries: worker and api import @musecanvas/* by package name
  instead of relative packages/*/src paths; api enables
  transpilePackages; database/providers gain subpath exports
- hardening: fail-closed rate limiting, non-root containers, unified
  SSRF validation, security headers, break api runtime cycles
- cleanup: drop dead exports, move web session state to TanStack Query,
  ignore .pi/, prune redundant .gitkeep

Verified: pnpm typecheck | lint | test | check:tokens | build all exit 0;
pnpm audit --prod reports no known vulnerabilities.
- Redesign /login with bento-style showcase gallery
- Redesign /generate as three-column AI console (conversation rail, timeline, jobs rail)
- Redesign /library grid and asset lightbox dual-pane modal
- Redesign /admin dashboard with metric cards, stacked bar chart and recent jobs table
- Redesign /admin/media-models with tabs, defaults card and model table
- Add chart-positive/chart-negative tokens; tweak muted-foreground for contrast
- Update ui-redesign e2e probes for the new UI
@nextroad-dev nextroad-dev changed the title 整合 dev 应用更新与后端查询、分页及删除事务修复 整合 dev 应用更新与后端数据库可靠性优化 Oct 18, 2026
Reject null in UpdateMediaModelDefaultsRequest (no clearing) and document
GET/PATCH /api/admin/media-model-defaults in openapi.yml.
…th design mocks

- landing: hero, works, platform strip, capability accordion, workflow, dark CTA,
  three-column footer, mobile sticky CTA (features/landing)
- generate: stage stepper transition, slow emphasized result fade, model picker
  exit, robust finished-job exit timers, reveal only for streamed entries,
  reduced-motion shimmer, white segmented thumb
- library/login: 6-column default, short tile captions, lightbox stage and copy,
  OAuth on email step only, fieldset legend for the code cells
- admin: overview 1fr/1fr/2fr layout, breadcrumb, chart a11y, sr-only actions column
The API returns { providers: [...] } but the login form required data to be an
array, so the third-party sign-in block always showed an error and kept
GitHub/Google disabled. Align the client type and the e2e fixture with the API.
…d database status

- web: 7-step /setup wizard (claim, site, SMTP, admin, database, storage, finish) with
  shared settings forms reused by the admin settings page
- api: claim-code delivery modes (file/log/off, opt-in in production), database status and
  test endpoints, driver-aware storage save/test, keep stored driver when the request omits it
- storage: StorageDriver abstraction with a local-disk driver (atomic writes, key and symlink
  safety), HMAC-signed /api/media/object and /api/media/upload routes, worker local driver
- database: migration 0040 (storage_driver, local_root) and a shared resolveStorageConfig
  used by both the API and the worker, failing closed on undecryptable secrets
- contracts: storage driver and database status DTOs, setup.database endpoints, media endpoints
- deploy: media_data volume for api and worker, /data/media ownership, nginx upload limit,
  SETUP_CLAIM_DELIVERY knob (off by default for image and prod stacks)
…IM_REQUIRED

SETUP_CLAIM_REQUIRED=false opens /setup without a claim code, but only while onboarding is
incomplete and no active admin exists; the window closes by itself once an admin is created.
Unset or any other value keeps today's behaviour. Database errors fail closed.

- api: resolveClaimRequired, open-mode access rule, no claim minted in open mode,
  setup/claim answers 409 SETUP_CLAIM_NOT_REQUIRED, status exposes claimRequired,
  per-IP rate limits on setup/smtp/test and setup/storage/test
- web: wizard builds its step list from claimRequired (7 steps, or 6 without the claim step),
  deep-link fallback, open-setup notice
- contracts: SetupStatusResponse.claimRequired, SetupErrorCode.SETUP_CLAIM_NOT_REQUIRED
- deploy/.env.example: SETUP_CLAIM_REQUIRED passthrough and documentation with a warning
)

Node.js 22.23.4 fixes a Brotli decoder out-of-bounds write (CVE-2026-111497, RCE) and TLS hostname verification bypasses (CVE-2026-106069, CVE-2026-106072). The api, web and worker images run Node at runtime and make outbound HTTPS calls via fetch (OAuth, provider APIs).

- Pin api/web/worker Dockerfiles to node:22.23.4-alpine so floating tags and build cache cannot keep an old runtime
- Pin setup-node to 22.23.4 in CI workflows
@nextroad-dev
nextroad-dev marked this pull request as ready for review October 22, 2026 03:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant