Skip to content

fix(generator): stop losing links and misfiling categories silently - #129

Merged
kdroidFilter merged 3 commits into
masterfrom
fix/silent-link-and-category-loss
Oct 12, 2026
Merged

kdroidFilter merged 3 commits into
masterfrom
fix/silent-link-and-category-loss

Conversation

@kdroidFilter

@kdroidFilter kdroidFilter commented Oct 12, 2026 •

Copy link
Copy Markdown
Owner

Problem

In the 20261011223929 release (Zayit 0.6.0), the links between חברותא and הערות על חברותא are missing for most tractates. Berakhot has 0 of its 3417 links, Bava Batra has 412 of 3698. The Otzaria link files are complete upstream, and their line indexes match the DB.

Cause: Otzaria links (appendOtzariaLinks) were inserted through repository.insertLink, which uses SQLite auto-increment ids outside the id allocator. generateHavroutaLinks runs in its own JVM and reloads the allocator from seforim.db.buildstate, whose LINK counter was behind those rows. It then handed out ids that were already taken, and INSERT OR IGNORE dropped every colliding link without an error.

The same audit found a second silent loss. Otzaria categories were keyed by their bare folder name, so every folder with the same name (סדר מועד, חלק ב, …) got the id of the first one created. About 200 books ended up filed under a same-named folder elsewhere in the tree. Examples:

  • תלמוד בבלי / ראשונים / קובץ שיטות קמאי / סדר מועד → הלכה / שולחן ערוך / מפרשים / בית מאיר / סדר מועד
  • מדרש / אגדה / ילקוט מדרשים / חלק ב → … / בתי מדרשות / חלק ב

This is the cause of #130 (קרן אורה, שפת אמת, בית מאיר, … filed under אור הישר).

A third problem (#131): Otzaria often lists a commentary's links only in the commentary's own file, typed source. For example, קרן אורה על זבחים_links.json has 755 links to זבחים, and זבחים_links.json has none back. They were stored as they are: commentary → base, type SOURCE. Stored links are always base → dependant, though, and SOURCE is a read-time virtual type (see the Link KDoc). The commentary panel reads sourceLineId, and the SOURCE view only reads dependant types, so neither ever reads these rows. In the current DB, 184,079 links over 401 books never show: קובץ שיטות קמאי, קרן אורה, שפת אמת, אוצר ההלכה, …

Changes

  • Link ids: Otzaria links take their id from the allocator, and insertLink honours a caller-provided id.
  • No silent drop: every link insert with an explicit id (insertLink, insertLinksBatch, insertLinkWithId) now fails when the id is held by a different link. Re-inserting the same link is still a no-op. The Otzaria link loops rethrow that error instead of swallowing it.
  • Categories: Otzaria categories are keyed by their full path from the root. Root keys are unchanged, so the per-root-category user setting (diacritics) keeps its id.
  • Otzaria source links: they are flipped into a COMMENTARY link from the base line (canonicalOtzariaLink). If Otzaria also has the base-side link, both map to the same allocator id, so the second insert is a no-op.
  • Fail fast: a failed build_state snapshot, or a failed in-memory seed in GenerateLines / GenerateLinks / BuildLuceneIndex, now fails the step. Before, it logged and carried on with stale ids, or wrote a partial DB over seforim.db.
  • Category name (ניסוח לא נכון וראוי בתפריטים Zayit#666): RenameCategoriesPostProcess renames Sefaria's כתבי האר״י וחיים ויטאל to כתבי האר״י ורבי חיים ויטאל, matching the references. Otzaria's folder כתבי הארי ורבי חיים ויטאל was a second category until now (3 books). It now resolves to the renamed one, so the two are merged.

Impact on the next delta

  • Otzaria links and Otzaria sub-categories get new ids once (the flipped links included), so the next patch is larger than usual.
  • Books whose category changes are re-indexed by LuceneUpdater (metadata-only upsert_book).
  • User data is unaffected: notes and bookmarks reference books and lines, and the diacritics setting is stored on root categories only.

Not covered here

The audit also found these, left for follow-ups:

  • INSERT OR IGNORE with allocator ids on the other tables (book, line, tocEntry, lookups) has no collision check yet.
  • The transitive Talmud→Hearot links and Sefaria alt_toc_entry still use auto-increment ids. This causes delta churn but no loss within one run.
  • sourceId == 1L is hardcoded in GenerateHavroutaLinks / SefariaLinksImporter.

Testing

  • :dao:jvmTest passes, including a new test: re-inserting the same link is a no-op, and a different link with the same id fails.
  • :generator-common:jvmTest passes.
  • :otzariasqlite:jvmTest passes: a source link is flipped into a commentary from the base, and other links keep their orientation.
  • Not run: a full generateSeforimDb. It is needed to confirm the Havrouta link counts, the category tree, that קרן אורה / שפת אמת now show next to זבחים, and that no other collision now fails the build.

Fixes #130
Fixes #131

kdroidFilter and others added 3 commits October 12, 2026 11:53
Otzaria links were inserted with SQLite auto-increment ids, outside the id
allocator. The next step (generateHavroutaLinks) reloaded the allocator,
handed out ids those links already held, and INSERT OR IGNORE dropped the
new rows: in the 20261011223929 release, 0 of 3417 Havrouta -> Hearot links
for Berakhot survived, and most other tractates lost part of theirs.

- Otzaria links take their id from the allocator; insertLink honours it.
- Every link insert with an explicit id fails when the id is held by a
  different link (re-inserting the same link stays a no-op), and the
  Otzaria link loops no longer swallow that error.
- Otzaria categories are keyed by their full path, not the bare folder
  name: about 200 books were filed under a same-named folder elsewhere
  (e.g. Kovetz Shitot Kamai / Seder Moed under Shulchan Arukh / Beit Meir).
- A failed build_state snapshot or in-memory seed now fails the step
  instead of logging and continuing with stale ids or a partial DB.

Signed-off-by: Elie Gambache <elyahou.hadass@gmail.com>
Sefaria's "כתבי האר״י וחיים ויטאל" becomes "כתבי האר״י ורבי חיים ויטאל", as in the
references. Otzaria's folder of that name then lands in it instead of a second category.

Fixes kdroidFilter/Zayit#666

Signed-off-by: Elie Gambache <elyahou.hadass@gmail.com>
… base

Otzaria often lists a commentary's links only in the commentary's own file,
typed "source" (e.g. Keren Orah on Zevachim -> Zevachim, nothing in
Zevachim_links.json). They were stored as is, commentary -> base with type
SOURCE, although stored links are always base -> dependant and SOURCE is a
read-time virtual type. Neither the commentary panel nor the SOURCE view
reads such rows: 184,079 links over 401 books (Kovetz Shitot Kamai, Keren
Orah, Sfat Emet, Otzar HaHalacha, ...) never showed in the app.

They are now flipped into a COMMENTARY link from the base line.

Fixes #131

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Elie Gambache <elyahou.hadass@gmail.com>
@kdroidFilter
kdroidFilter marked this pull request as ready for review October 12, 2026 17:24
@kdroidFilter
kdroidFilter merged commit 9bcac65 into master Oct 12, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

להציג יותר מפרשים על הש"ס הרבה אחרונים נכנסו שלא במקומם לתוך ספר אור הישר

1 participant