feat(identities): sync and clean the events collection on identity webhooks - #13
Merged
Merged
Conversation
…bhooks - bump @data-fair/lib-express to 1.26.0 (secret in the x-secret-key header) - move the webhook effects to identities/service.ts - rename: sender and originator names (departmentName included) are rewritten on events with their search texts, notifications follow the sender name like subscriptions do - delete: the events of the identity's own feed are removed, the events a deleted user triggered on other feeds only keep the user id (pseudonymized, no name nor email left), pending webhooks and the notifications pointer of the identity are removed too - sparse indexes on originator.user.id and originator.organization.id
Notifications are snapshots taken at delivery and the collection has no index on sender: rewriting it would scan the whole collection on every user webhook.
The shared identities router refuses the calls that come through the proxy, and the other services of the dev environment do not need the webhooks.
…t in the embed A department missing from the complete list sent by simple-directory was deleted: subscriptions, push and webhook subscriptions and events keep the department id but lose departmentName. The events embed shows "Former department - <id>" on originators. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017hqsZKwijEBoB3m3v9srVD
…n messages The production build uses the runtime-only vue-i18n: inline messages given to useI18n are not compiled and the key was displayed instead of the label. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017hqsZKwijEBoB3m3v9srVD
…vent An organization can own tens of thousands of events and simple-directory waits for the webhook response: rewriting them one by one with their search texts (and the text index behind) took more than ten minutes on staging, delaying every service notified after events. - names are no longer part of the search texts (ids, topic, title and body only), so a rename never touches _search nor the text index - rename and delete use updateMany on sender, originator.user and originator.organization (department names included, which were not propagated on the originator)
# Conflicts: # api/package.json # api/src/mongo.ts # package-lock.json # package.json
Names are back in the events search texts: they are what makes a user findable in an organization feed. The identity webhook no longer rewrites them: it only bulk updates the names and flags the events (_needsSearch), and a search worker running in the api server rebuilds their texts by batches of 1000. - the webhook only touches the events whose name actually changes (simple-directory also posts on every membership change) - a deleted user's name leaves the search texts too - the worker skips an event flagged again by a newer rename in the meantime, the next batch picks it up - partial index on _needsSearch, hidden from the API responses
config 5 no longer resolves NODE_CONFIG_DIR=api/config (Cannot find module 'api/config/default.cjs'): the quality check fails since the dependencies refresh (#15).
- a fresh _needsSearch value per update: an event flagged by two updates of the same webhook could otherwise get the texts of the first one and lose its flag - an event whose texts cannot be built has its flag cleared (like drainSearchIndex in data-fair) instead of coming back first in every batch and blocking the others - read only the fields buildSearchTexts needs - buildSearchTexts unit tests back to names being searchable
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Per-service pass of the identities contract work, guided by the
data-fair-identitiesskill of the lib.@data-fair/lib-expressto 1.27.0 (secret accepted in thex-secret-keyheader)identities/service.tsdepartmentNameincluded, onoriginator.organizationtoo) are bulk updated on events; notifications follow the recipient name like subscriptions do (the sender name of a delivered notification is a snapshot and is not rewritten: no index on sender)_search(they are what makes a user findable in an organization feed), but the webhook no longer rewrites them: it bulk updates the names of the events whose name actually changes and flags them (_needsSearch); a search worker running in the api server (events/search-worker.ts, same loop and lock pattern as the webhooks worker) rebuilds their texts by batches of 1000 withbuildSearchTexts, and skips an event flagged again by a newer rename in the meantime (each update writes a fresh_needsSearchvalue and the worker only clears the one it read). LikedrainSearchIndexin data-fair, an event whose texts cannot be built keeps its previous texts and has its flag cleared, so it never blocks the queue. The worker reads only the fieldsbuildSearchTextsneeds. A first version rewrote every event inside the webhook: on staging an organization with ~55k events kept it busy for more than ten minutes, and since simple-directory notifies its services one after the other, every service after events received the rename that late. Local bench on 55k events: the webhook answers in ~2.4s, the worker catches up in ~25s in the background, a post without a name change (every membership change) costs ~0.2sdepartmentslist sent by simple-directory was deleted; its subscriptions, push and webhook subscriptions and events (sender and originator) keep the department id but losedepartmentName; the embed labels it "Ancien département (id)" / "Former department (id)" throughowner-avatarof lib-vuetify 2.5.1 (the localuse-display-ownercomposable is gone, feat(vue): useDisplayOwner and owner-avatar label a deleted department lib#52)originator.user.idandoriginator.organization.id, partial index on_needsSearchNODE_CONFIG_DIR=./api/configin the quality workflow: config 5 (chore(deps): refresh dependencies and take verified majors #15) no longer resolvesapi/config, the check fails on main since thentests/identities-webhook.api.spec.ts: renames (searchable once the worker went through, consecutive renames end on the last name), private subscriptions, department deletion, user and organization deletion asserted right after the webhookWhy: the events collection was never touched by the identity webhooks: a deleted user's name and email stayed in every event they triggered.