feat: link page numbers to the paper dictionaries' scans on archive.org

CLAUDE.md

@@ -55,7 +55,7 @@ The FULLTEXT index is the *reason* MariaDB exists in this project: SQLite has no
 
 ### Data validation
 
-[arabterm/scripts/validate_db.py](arabterm/scripts/validate_db.py) (`make validate`) checks the invariants of `arabterm.db`: every term has an Arabic and an English value, no orphan terms, `nbr_entries` equals the real row count, every `wikidata_id` is a well-formed and unique QID. It reads only `arabterm.db` and never the network, so that a CI result depends on the pull request alone and not on somebody editing Wikidata. The consequence: it cannot tell whether a QID is the *right* item (e.g. the item of another edition of the same work) — that stays a manual check, step 2 of the new-dictionary checklist. The script is stdlib-only on purpose, so CI runs it with plain `python3`.
+[arabterm/scripts/validate_db.py](arabterm/scripts/validate_db.py) (`make validate`) checks the invariants of `arabterm.db`: every term has an Arabic and an English value, no orphan terms, `nbr_entries` equals the real row count, every `wikidata_id` is a well-formed and unique QID, every `archive_org_url` is a bare `archive.org/details/` URL, and a dictionary with a scan and page numbers has a `page_offset` (and no `page_offset` without a scan). It reads only `arabterm.db` and never the network, so that a CI result depends on the pull request alone and not on somebody editing Wikidata. The consequence: it cannot tell whether a QID is the *right* item (e.g. the item of another edition of the same work) — that stays a manual check, step 2 of the new-dictionary checklist. The script is stdlib-only on purpose, so CI runs it with plain `python3`.
 
 Existing violations are recorded in [arabterm/scripts/validation_baseline.json](arabterm/scripts/validation_baseline.json) (term ids / `name_tech` slugs per check, plus `exempt` for whole dictionaries such as `al_mawrid_al_hadeeth`, which has no Arabic column). The baseline is a ratchet: a violation not in it fails, and so does a baseline entry that is no longer a violation — after fixing data, run `make validate_update_baseline` and commit the shrunken file. Never use `validate_update_baseline` to silence a new violation that is a real data error; fix the data instead.
 
@@ -78,6 +78,7 @@ Legacy unprefixed URLs from the original Angular site (e.g. `/water_engineering/
 - Adding a new dictionary — full checklist:
   1. **Back up first**: `cp arabterm.db arabterm.db.bak` (gitignored). A botched insert into the canonical SQLite is the only real failure mode here.
   2. **Resolve the Wikidata labels** if the dictionary has a QID — fetch `https://www.wikidata.org/wiki/Special:EntityData/<QID>.json` and use the official `ar`/`en`/`fr` labels. Append the publication year in parentheses to `name_arabic` to match the existing series (e.g. `... (2020)`). Wikidata sometimes uses ALL CAPS or typos in the English/French labels — match the style of existing entries (sentence case for French, no typos).
+     For a paper dictionary, also take its archive.org scan from the "full work available at URL" (P953) statement and the offset from that statement's "file page offset" (P8330) qualifier, into `archive_org_url` (bare item URL: drop any `/mode/…`, keep the file path of a multi-volume item) and `page_offset`. Printed page `p` is then `<archive_org_url>/page/n<p + page_offset - 1>/mode/1up`, which is the link wikitermbase and the Cite Q module on ar.wikipedia build. Without P8330, infer the offset from the scan's OCR (`<book>_hocr_searchtext.txt.gz` + `_hocr_pageindex.json.gz`): for each term, the 0-based scan page `n` whose text contains its English form votes for `n - page + 1`; take the mode, and add it to Wikidata as the P8330 qualifier. A scan that drifts by a few pages (missing or extra sheets) gets the dominant offset — near the right page is good enough. `page` holds the *printed* page: a multi-volume work whose volumes each start at page 1 must not be numbered continuously.
   3. **Pick a `name_tech` slug**: follow the existing series naming. The 2020 educational series uses `topic_NN` (e.g. `educational_supervision_47`); older ArabTerm-website entries use the `at_` prefix (e.g. `at_water_engineering`). `samples/NN_topic_…/` folder names flip to `topic_NN` slugs.
   4. **Insert** the `dictionary` row (set `nbr_entries` to the actual row count) **and** the `term` rows in a single SQLite transaction. Pre-check that the slug and `wikidata_id` aren't already taken. Use the `Dictionary.id` returned by `lastrowid` as the `term.dictionary_id`.
   5. **Run `make regenerate_dumps`** — it starts with `make validate`, so a term without Arabic/English, a wrong `nbr_entries` or a duplicate QID stops it before any dump is written. That single target now also rewrites the Dictionaries table in `README.md`. Commit the resulting `arabterm.db.gz`, `db/sqlite/arabterm.sql.gz`, `db/mariadb/arabterm.sql.gz`, and `README.md` together (plus `validation_baseline.json` if it changed). `arabterm.db` is gitignored and is never committed. The website will pick up the new dictionary automatically on the next deploy.

README.md

@@ -30,6 +30,8 @@ It contains 2 tables: `dictionary` and `term`.
 | `nbr_entries` | INTEGER | Number of terms in the dictionary |
 | `name_tech` | VARCHAR | Technical/slug name (unique) |
 | `wikidata_id` | VARCHAR | Wikidata item ID for the dictionary |
+| `archive_org_url` | VARCHAR | Scan of the paper dictionary on archive.org (`https://archive.org/details/<item>`), as in Wikidata's P953 |
+| `page_offset` | INTEGER | Printed page `p` is page `p + page_offset` of the scan (Wikidata's P8330). Its archive.org link is `<archive_org_url>/page/n<p + page_offset - 1>/mode/1up` |
 | `created_at` | DATETIME | Record creation timestamp |
 | `updated_at` | DATETIME | Record last update timestamp |
 
@@ -43,7 +45,7 @@ It contains 2 tables: `dictionary` and `term`.
 | `french` | VARCHAR | Term in French |
 | `description` | VARCHAR | Term description (usually in Arabic) |
 | `dictionary_id` | INTEGER | Foreign key to `dictionary.id` |
-| `page` | INTEGER | Page number in source document (if applicable) |
+| `page` | INTEGER | Printed page number in the source document (if applicable) |
 | `uri` | VARCHAR | Source URI/link |
 | `created_at` | DATETIME | Record creation timestamp |
 | `updated_at` | DATETIME | Record last update timestamp |

arabterm/mariadb_models.py

@@ -32,6 +32,10 @@ class Dictionary(Base):
     wikidata_id: Mapped[Optional[str]] = mapped_column(String(255))
     dict_type: Mapped[Optional[str]] = mapped_column(String(255))
     tier: Mapped[Optional[int]] = mapped_column(Integer)
+    # Text, not String(255): a URL into a multi-volume item names the volume's
+    # file, percent-encoded (dstt_1's is 292 characters).
+    archive_org_url: Mapped[Optional[str]] = mapped_column(Text)
+    page_offset: Mapped[Optional[int]] = mapped_column(Integer)
     created_at: Mapped[datetime.datetime] = mapped_column(
         DateTime, server_default=text("CURRENT_TIMESTAMP")
     )

arabterm/scripts/migrate_to_mariadb.py

@@ -30,6 +30,8 @@ def migrate_dictionary(sqlite_dict: DictionarySQLite) -> Dict[str, Any]:
         "wikidata_id": sqlite_dict.wikidata_id,
         "dict_type": sqlite_dict.dict_type,
         "tier": sqlite_dict.tier,
+        "archive_org_url": sqlite_dict.archive_org_url,
+        "page_offset": sqlite_dict.page_offset,
         "created_at": sqlite_dict.created_at or datetime.datetime.now(),
         # A plain `None` here is indistinguishable to SQLAlchemy from "no value
         # given" and gets silently dropped from the INSERT, letting MariaDB's

arabterm/scripts/validate_db.py

@@ -35,6 +35,11 @@
 BASELINE = Path(__file__).resolve().with_name("validation_baseline.json")
 
 QID_RE = re.compile(r"Q[1-9]\d*")
+# An archive.org item, or one file of a multi-volume item, as the page links
+# need it: no /page/ or /mode/ of its own, no query, no trailing slash.
+ARCHIVE_ORG_URL_RE = re.compile(
+    r"https://archive\.org/details/[A-Za-z0-9][A-Za-z0-9._-]*(/[^/?#\s]+)?"
+)
 
 # Tab, LF, CR, space, no-break space: a value made only of these is empty.
 BLANK = "char(9, 10, 13, 32, 160)"
@@ -117,6 +122,45 @@ def check_wikidata_id_duplicate(con: sqlite3.Connection) -> Violations:
     return {slug: f"{qid} is used by more than one dictionary" for slug, qid in rows}
 
 
+def check_archive_org_url_invalid(con: sqlite3.Connection) -> Violations:
+    rows = con.execute(
+        "SELECT name_tech, archive_org_url FROM dictionary"
+        " WHERE archive_org_url IS NOT NULL"
+    )
+    return {
+        slug: f"archive_org_url={url!r} is not an archive.org/details/ URL"
+        for slug, url in rows
+        if not ARCHIVE_ORG_URL_RE.fullmatch(url)
+    }
+
+
+def check_page_offset_without_url(con: sqlite3.Connection) -> Violations:
+    rows = con.execute(
+        "SELECT name_tech, page_offset FROM dictionary"
+        " WHERE page_offset IS NOT NULL AND archive_org_url IS NULL"
+    )
+    return {
+        slug: f"page_offset={offset} but no archive_org_url" for slug, offset in rows
+    }
+
+
+def check_page_offset_missing(con: sqlite3.Connection) -> Violations:
+    """A scan and page numbers without the offset: no page could be linked."""
+    rows = con.execute(
+        """
+        SELECT name_tech FROM dictionary d
+        WHERE archive_org_url IS NOT NULL AND page_offset IS NULL
+          AND EXISTS (
+              SELECT 1 FROM term t WHERE t.dictionary_id = d.id AND t.page IS NOT NULL
+          )
+        """
+    )
+    return {
+        slug: "archive_org_url and term pages are set, but not page_offset"
+        for (slug,) in rows
+    }
+
+
 def run_checks(
     con: sqlite3.Connection, exempt: dict[str, list[str]]
 ) -> dict[str, Violations]:
@@ -131,6 +175,9 @@ def run_checks(
         "dictionary_nbr_entries_mismatch": check_nbr_entries(con),
         "dictionary_wikidata_id_invalid": check_wikidata_id_invalid(con),
         "dictionary_wikidata_id_duplicate": check_wikidata_id_duplicate(con),
+        "dictionary_archive_org_url_invalid": check_archive_org_url_invalid(con),
+        "dictionary_page_offset_without_url": check_page_offset_without_url(con),
+        "dictionary_page_offset_missing": check_page_offset_missing(con),
     }
 
 

arabterm/scripts/validation_baseline.json

@@ -168,6 +168,9 @@
     "term_orphan": [],
     "dictionary_nbr_entries_mismatch": [],
     "dictionary_wikidata_id_invalid": [],
-    "dictionary_wikidata_id_duplicate": []
+    "dictionary_wikidata_id_duplicate": [],
+    "dictionary_archive_org_url_invalid": [],
+    "dictionary_page_offset_without_url": [],
+    "dictionary_page_offset_missing": []
   }
 }