Repository navigation
fix(generator): stop losing links and misfiling categories silently - #129
Merged
Merged
Conversation
Otzaria links were inserted with SQLite auto-increment ids, outside the id allocator. The next step (generateHavroutaLinks) reloaded the allocator, handed out ids those links already held, and INSERT OR IGNORE dropped the new rows: in the 20261011223929 release, 0 of 3417 Havrouta -> Hearot links for Berakhot survived, and most other tractates lost part of theirs. - Otzaria links take their id from the allocator; insertLink honours it. - Every link insert with an explicit id fails when the id is held by a different link (re-inserting the same link stays a no-op), and the Otzaria link loops no longer swallow that error. - Otzaria categories are keyed by their full path, not the bare folder name: about 200 books were filed under a same-named folder elsewhere (e.g. Kovetz Shitot Kamai / Seder Moed under Shulchan Arukh / Beit Meir). - A failed build_state snapshot or in-memory seed now fails the step instead of logging and continuing with stale ids or a partial DB. Signed-off-by: Elie Gambache <elyahou.hadass@gmail.com>
Sefaria's "כתבי האר״י וחיים ויטאל" becomes "כתבי האר״י ורבי חיים ויטאל", as in the references. Otzaria's folder of that name then lands in it instead of a second category. Fixes kdroidFilter/Zayit#666 Signed-off-by: Elie Gambache <elyahou.hadass@gmail.com>
… base Otzaria often lists a commentary's links only in the commentary's own file, typed "source" (e.g. Keren Orah on Zevachim -> Zevachim, nothing in Zevachim_links.json). They were stored as is, commentary -> base with type SOURCE, although stored links are always base -> dependant and SOURCE is a read-time virtual type. Neither the commentary panel nor the SOURCE view reads such rows: 184,079 links over 401 books (Kovetz Shitot Kamai, Keren Orah, Sfat Emet, Otzar HaHalacha, ...) never showed in the app. They are now flipped into a COMMENTARY link from the base line. Fixes #131 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Elie Gambache <elyahou.hadass@gmail.com>
kdroidFilter
marked this pull request as ready for review
October 12, 2026 17:24
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
In the
20261011223929release (Zayit 0.6.0), the links between חברותא and הערות על חברותא are missing for most tractates. Berakhot has 0 of its 3417 links, Bava Batra has 412 of 3698. The Otzaria link files are complete upstream, and their line indexes match the DB.Cause: Otzaria links (
appendOtzariaLinks) were inserted throughrepository.insertLink, which uses SQLite auto-increment ids outside the id allocator.generateHavroutaLinksruns in its own JVM and reloads the allocator fromseforim.db.buildstate, whose LINK counter was behind those rows. It then handed out ids that were already taken, andINSERT OR IGNOREdropped every colliding link without an error.The same audit found a second silent loss. Otzaria categories were keyed by their bare folder name, so every folder with the same name (
סדר מועד,חלק ב, …) got the id of the first one created. About 200 books ended up filed under a same-named folder elsewhere in the tree. Examples:תלמוד בבלי / ראשונים / קובץ שיטות קמאי / סדר מועד→הלכה / שולחן ערוך / מפרשים / בית מאיר / סדר מועדמדרש / אגדה / ילקוט מדרשים / חלק ב→… / בתי מדרשות / חלק בThis is the cause of #130 (קרן אורה, שפת אמת, בית מאיר, … filed under אור הישר).
A third problem (#131): Otzaria often lists a commentary's links only in the commentary's own file, typed
source. For example,קרן אורה על זבחים_links.jsonhas 755 links to זבחים, andזבחים_links.jsonhas none back. They were stored as they are: commentary → base, typeSOURCE. Stored links are always base → dependant, though, andSOURCEis a read-time virtual type (see theLinkKDoc). The commentary panel readssourceLineId, and the SOURCE view only reads dependant types, so neither ever reads these rows. In the current DB, 184,079 links over 401 books never show: קובץ שיטות קמאי, קרן אורה, שפת אמת, אוצר ההלכה, …Changes
insertLinkhonours a caller-provided id.insertLink,insertLinksBatch,insertLinkWithId) now fails when the id is held by a different link. Re-inserting the same link is still a no-op. The Otzaria link loops rethrow that error instead of swallowing it.sourcelinks: they are flipped into aCOMMENTARYlink from the base line (canonicalOtzariaLink). If Otzaria also has the base-side link, both map to the same allocator id, so the second insert is a no-op.build_statesnapshot, or a failed in-memory seed inGenerateLines/GenerateLinks/BuildLuceneIndex, now fails the step. Before, it logged and carried on with stale ids, or wrote a partial DB overseforim.db.RenameCategoriesPostProcessrenames Sefaria'sכתבי האר״י וחיים ויטאלtoכתבי האר״י ורבי חיים ויטאל, matching the references. Otzaria's folderכתבי הארי ורבי חיים ויטאלwas a second category until now (3 books). It now resolves to the renamed one, so the two are merged.Impact on the next delta
LuceneUpdater(metadata-onlyupsert_book).Not covered here
The audit also found these, left for follow-ups:
INSERT OR IGNOREwith allocator ids on the other tables (book, line, tocEntry, lookups) has no collision check yet.alt_toc_entrystill use auto-increment ids. This causes delta churn but no loss within one run.sourceId == 1Lis hardcoded inGenerateHavroutaLinks/SefariaLinksImporter.Testing
:dao:jvmTestpasses, including a new test: re-inserting the same link is a no-op, and a different link with the same id fails.:generator-common:jvmTestpasses.:otzariasqlite:jvmTestpasses: asourcelink is flipped into a commentary from the base, and other links keep their orientation.generateSeforimDb. It is needed to confirm the Havrouta link counts, the category tree, that קרן אורה / שפת אמת now show next to זבחים, and that no other collision now fails the build.Fixes #130
Fixes #131