diff --git a/CLAUDE.md b/CLAUDE.md index 645fe2d..f8de90d 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -55,7 +55,7 @@ The FULLTEXT index is the *reason* MariaDB exists in this project: SQLite has no ### Data validation -[arabterm/scripts/validate_db.py](arabterm/scripts/validate_db.py) (`make validate`) checks the invariants of `arabterm.db`: every term has an Arabic and an English value, no orphan terms, `nbr_entries` equals the real row count, every `wikidata_id` is a well-formed and unique QID. It reads only `arabterm.db` and never the network, so that a CI result depends on the pull request alone and not on somebody editing Wikidata. The consequence: it cannot tell whether a QID is the *right* item (e.g. the item of another edition of the same work) — that stays a manual check, step 2 of the new-dictionary checklist. The script is stdlib-only on purpose, so CI runs it with plain `python3`. +[arabterm/scripts/validate_db.py](arabterm/scripts/validate_db.py) (`make validate`) checks the invariants of `arabterm.db`: every term has an Arabic and an English value, no orphan terms, `nbr_entries` equals the real row count, every `wikidata_id` is a well-formed and unique QID, every `archive_org_url` is a bare `archive.org/details/` URL, and a dictionary with a scan and page numbers has a `page_offset` (and no `page_offset` without a scan). It reads only `arabterm.db` and never the network, so that a CI result depends on the pull request alone and not on somebody editing Wikidata. The consequence: it cannot tell whether a QID is the *right* item (e.g. the item of another edition of the same work) — that stays a manual check, step 2 of the new-dictionary checklist. The script is stdlib-only on purpose, so CI runs it with plain `python3`. Existing violations are recorded in [arabterm/scripts/validation_baseline.json](arabterm/scripts/validation_baseline.json) (term ids / `name_tech` slugs per check, plus `exempt` for whole dictionaries such as `al_mawrid_al_hadeeth`, which has no Arabic column). The baseline is a ratchet: a violation not in it fails, and so does a baseline entry that is no longer a violation — after fixing data, run `make validate_update_baseline` and commit the shrunken file. Never use `validate_update_baseline` to silence a new violation that is a real data error; fix the data instead. @@ -78,6 +78,7 @@ Legacy unprefixed URLs from the original Angular site (e.g. `/water_engineering/ - Adding a new dictionary — full checklist: 1. **Back up first**: `cp arabterm.db arabterm.db.bak` (gitignored). A botched insert into the canonical SQLite is the only real failure mode here. 2. **Resolve the Wikidata labels** if the dictionary has a QID — fetch `https://www.wikidata.org/wiki/Special:EntityData/.json` and use the official `ar`/`en`/`fr` labels. Append the publication year in parentheses to `name_arabic` to match the existing series (e.g. `... (2020)`). Wikidata sometimes uses ALL CAPS or typos in the English/French labels — match the style of existing entries (sentence case for French, no typos). + For a paper dictionary, also take its archive.org scan from the "full work available at URL" (P953) statement and the offset from that statement's "file page offset" (P8330) qualifier, into `archive_org_url` (bare item URL: drop any `/mode/…`, keep the file path of a multi-volume item) and `page_offset`. Printed page `p` is then `/page/n

/mode/1up`, which is the link wikitermbase and the Cite Q module on ar.wikipedia build. Without P8330, infer the offset from the scan's OCR (`_hocr_searchtext.txt.gz` + `_hocr_pageindex.json.gz`): for each term, the 0-based scan page `n` whose text contains its English form votes for `n - page + 1`; take the mode, and add it to Wikidata as the P8330 qualifier. A scan that drifts by a few pages (missing or extra sheets) gets the dominant offset — near the right page is good enough. `page` holds the *printed* page: a multi-volume work whose volumes each start at page 1 must not be numbered continuously. 3. **Pick a `name_tech` slug**: follow the existing series naming. The 2020 educational series uses `topic_NN` (e.g. `educational_supervision_47`); older ArabTerm-website entries use the `at_` prefix (e.g. `at_water_engineering`). `samples/NN_topic_…/` folder names flip to `topic_NN` slugs. 4. **Insert** the `dictionary` row (set `nbr_entries` to the actual row count) **and** the `term` rows in a single SQLite transaction. Pre-check that the slug and `wikidata_id` aren't already taken. Use the `Dictionary.id` returned by `lastrowid` as the `term.dictionary_id`. 5. **Run `make regenerate_dumps`** — it starts with `make validate`, so a term without Arabic/English, a wrong `nbr_entries` or a duplicate QID stops it before any dump is written. That single target now also rewrites the Dictionaries table in `README.md`. Commit the resulting `arabterm.db.gz`, `db/sqlite/arabterm.sql.gz`, `db/mariadb/arabterm.sql.gz`, and `README.md` together (plus `validation_baseline.json` if it changed). `arabterm.db` is gitignored and is never committed. The website will pick up the new dictionary automatically on the next deploy. diff --git a/README.md b/README.md index edb82ee..b2f8987 100644 --- a/README.md +++ b/README.md @@ -30,6 +30,8 @@ It contains 2 tables: `dictionary` and `term`. | `nbr_entries` | INTEGER | Number of terms in the dictionary | | `name_tech` | VARCHAR | Technical/slug name (unique) | | `wikidata_id` | VARCHAR | Wikidata item ID for the dictionary | +| `archive_org_url` | VARCHAR | Scan of the paper dictionary on archive.org (`https://archive.org/details/`), as in Wikidata's P953 | +| `page_offset` | INTEGER | Printed page `p` is page `p + page_offset` of the scan (Wikidata's P8330). Its archive.org link is `/page/n

/mode/1up` | | `created_at` | DATETIME | Record creation timestamp | | `updated_at` | DATETIME | Record last update timestamp | @@ -43,7 +45,7 @@ It contains 2 tables: `dictionary` and `term`. | `french` | VARCHAR | Term in French | | `description` | VARCHAR | Term description (usually in Arabic) | | `dictionary_id` | INTEGER | Foreign key to `dictionary.id` | -| `page` | INTEGER | Page number in source document (if applicable) | +| `page` | INTEGER | Printed page number in the source document (if applicable) | | `uri` | VARCHAR | Source URI/link | | `created_at` | DATETIME | Record creation timestamp | | `updated_at` | DATETIME | Record last update timestamp | diff --git a/arabterm.db.gz b/arabterm.db.gz index 1ced417..6f5495a 100644 Binary files a/arabterm.db.gz and b/arabterm.db.gz differ diff --git a/arabterm/mariadb_models.py b/arabterm/mariadb_models.py index d1af77e..ad6de54 100644 --- a/arabterm/mariadb_models.py +++ b/arabterm/mariadb_models.py @@ -32,6 +32,10 @@ class Dictionary(Base): wikidata_id: Mapped[Optional[str]] = mapped_column(String(255)) dict_type: Mapped[Optional[str]] = mapped_column(String(255)) tier: Mapped[Optional[int]] = mapped_column(Integer) + # Text, not String(255): a URL into a multi-volume item names the volume's + # file, percent-encoded (dstt_1's is 292 characters). + archive_org_url: Mapped[Optional[str]] = mapped_column(Text) + page_offset: Mapped[Optional[int]] = mapped_column(Integer) created_at: Mapped[datetime.datetime] = mapped_column( DateTime, server_default=text("CURRENT_TIMESTAMP") ) diff --git a/arabterm/scripts/migrate_to_mariadb.py b/arabterm/scripts/migrate_to_mariadb.py index daca480..d27bcc5 100644 --- a/arabterm/scripts/migrate_to_mariadb.py +++ b/arabterm/scripts/migrate_to_mariadb.py @@ -30,6 +30,8 @@ def migrate_dictionary(sqlite_dict: DictionarySQLite) -> Dict[str, Any]: "wikidata_id": sqlite_dict.wikidata_id, "dict_type": sqlite_dict.dict_type, "tier": sqlite_dict.tier, + "archive_org_url": sqlite_dict.archive_org_url, + "page_offset": sqlite_dict.page_offset, "created_at": sqlite_dict.created_at or datetime.datetime.now(), # A plain `None` here is indistinguishable to SQLAlchemy from "no value # given" and gets silently dropped from the INSERT, letting MariaDB's diff --git a/arabterm/scripts/validate_db.py b/arabterm/scripts/validate_db.py index bd93e47..997c28b 100644 --- a/arabterm/scripts/validate_db.py +++ b/arabterm/scripts/validate_db.py @@ -35,6 +35,11 @@ BASELINE = Path(__file__).resolve().with_name("validation_baseline.json") QID_RE = re.compile(r"Q[1-9]\d*") +# An archive.org item, or one file of a multi-volume item, as the page links +# need it: no /page/ or /mode/ of its own, no query, no trailing slash. +ARCHIVE_ORG_URL_RE = re.compile( + r"https://archive\.org/details/[A-Za-z0-9][A-Za-z0-9._-]*(/[^/?#\s]+)?" +) # Tab, LF, CR, space, no-break space: a value made only of these is empty. BLANK = "char(9, 10, 13, 32, 160)" @@ -117,6 +122,45 @@ def check_wikidata_id_duplicate(con: sqlite3.Connection) -> Violations: return {slug: f"{qid} is used by more than one dictionary" for slug, qid in rows} +def check_archive_org_url_invalid(con: sqlite3.Connection) -> Violations: + rows = con.execute( + "SELECT name_tech, archive_org_url FROM dictionary" + " WHERE archive_org_url IS NOT NULL" + ) + return { + slug: f"archive_org_url={url!r} is not an archive.org/details/ URL" + for slug, url in rows + if not ARCHIVE_ORG_URL_RE.fullmatch(url) + } + + +def check_page_offset_without_url(con: sqlite3.Connection) -> Violations: + rows = con.execute( + "SELECT name_tech, page_offset FROM dictionary" + " WHERE page_offset IS NOT NULL AND archive_org_url IS NULL" + ) + return { + slug: f"page_offset={offset} but no archive_org_url" for slug, offset in rows + } + + +def check_page_offset_missing(con: sqlite3.Connection) -> Violations: + """A scan and page numbers without the offset: no page could be linked.""" + rows = con.execute( + """ + SELECT name_tech FROM dictionary d + WHERE archive_org_url IS NOT NULL AND page_offset IS NULL + AND EXISTS ( + SELECT 1 FROM term t WHERE t.dictionary_id = d.id AND t.page IS NOT NULL + ) + """ + ) + return { + slug: "archive_org_url and term pages are set, but not page_offset" + for (slug,) in rows + } + + def run_checks( con: sqlite3.Connection, exempt: dict[str, list[str]] ) -> dict[str, Violations]: @@ -131,6 +175,9 @@ def run_checks( "dictionary_nbr_entries_mismatch": check_nbr_entries(con), "dictionary_wikidata_id_invalid": check_wikidata_id_invalid(con), "dictionary_wikidata_id_duplicate": check_wikidata_id_duplicate(con), + "dictionary_archive_org_url_invalid": check_archive_org_url_invalid(con), + "dictionary_page_offset_without_url": check_page_offset_without_url(con), + "dictionary_page_offset_missing": check_page_offset_missing(con), } diff --git a/arabterm/scripts/validation_baseline.json b/arabterm/scripts/validation_baseline.json index b78cd0a..5ae13e9 100644 --- a/arabterm/scripts/validation_baseline.json +++ b/arabterm/scripts/validation_baseline.json @@ -168,6 +168,9 @@ "term_orphan": [], "dictionary_nbr_entries_mismatch": [], "dictionary_wikidata_id_invalid": [], - "dictionary_wikidata_id_duplicate": [] + "dictionary_wikidata_id_duplicate": [], + "dictionary_archive_org_url_invalid": [], + "dictionary_page_offset_without_url": [], + "dictionary_page_offset_missing": [] } } diff --git a/arabterm/sqlite_models.py b/arabterm/sqlite_models.py index 8553225..b74a4b4 100644 --- a/arabterm/sqlite_models.py +++ b/arabterm/sqlite_models.py @@ -26,6 +26,10 @@ class Dictionary(Base): wikidata_id: Mapped[Optional[str]] = mapped_column(String) dict_type: Mapped[Optional[str]] = mapped_column(String) tier: Mapped[Optional[int]] = mapped_column(Integer) + # Scan of a paper dictionary on archive.org, and where its printed pages are + # in it: printed page p is page p + page_offset of the file (Wikidata P8330). + archive_org_url: Mapped[Optional[str]] = mapped_column(String) + page_offset: Mapped[Optional[int]] = mapped_column(Integer) created_at: Mapped[datetime.datetime] = mapped_column( DateTime, server_default=text("CURRENT_TIMESTAMP") ) diff --git a/db/mariadb/arabterm.sql.gz b/db/mariadb/arabterm.sql.gz index e1376e2..1fe0acf 100644 Binary files a/db/mariadb/arabterm.sql.gz and b/db/mariadb/arabterm.sql.gz differ diff --git a/db/sqlite/arabterm.sql.gz b/db/sqlite/arabterm.sql.gz index f1d3a83..4dcbe0b 100644 Binary files a/db/sqlite/arabterm.sql.gz and b/db/sqlite/arabterm.sql.gz differ diff --git a/website/src/components/TermTable.astro b/website/src/components/TermTable.astro index 1bbddd4..a8f18f1 100644 --- a/website/src/components/TermTable.astro +++ b/website/src/components/TermTable.astro @@ -1,13 +1,15 @@ --- -import type { Term, TermColumns } from '../lib/db'; +import { scanPageUrl, type Dictionary, type Term, type TermColumns } from '../lib/db'; interface Props { terms: Term[]; startIndex: number; // 1-based index of first row on this page cols: TermColumns; + // Page numbers link to the dictionary's archive.org scan, when it has one. + scan: Pick; } -const { terms, startIndex, cols } = Astro.props; +const { terms, startIndex, cols, scan } = Astro.props; ---

@@ -23,14 +25,22 @@ const { terms, startIndex, cols } = Astro.props; - {terms.map((t, i) => ( + {terms.map((t, i) => { + const pageUrl = scanPageUrl(scan, t.page); + return ( {cols.arabic && } {cols.english && } {cols.french && } {cols.description && } - {cols.page && } + {cols.page && ( + + )} {cols.uri && ( )} - ))} + ); + })}
{startIndex + i}{t.arabic ?? ''}{t.english ?? ''}{t.french ?? ''}{t.description ?? ''}{t.page ?? ''} + {pageUrl ? ( + {t.page} + ) : (t.page ?? '')} + {t.uri && ( @@ -39,7 +49,8 @@ const { terms, startIndex, cols } = Astro.props;
diff --git a/website/src/lib/db.ts b/website/src/lib/db.ts index 7652719..5e04891 100644 --- a/website/src/lib/db.ts +++ b/website/src/lib/db.ts @@ -29,6 +29,8 @@ export interface Dictionary { name_french: string | null; nbr_entries: number | null; wikidata_id: string | null; + archive_org_url: string | null; + page_offset: number | null; term_count: number; cols: TermColumns; } @@ -51,6 +53,8 @@ interface DictionaryRow { name_french: string | null; nbr_entries: number | null; wikidata_id: string | null; + archive_org_url: string | null; + page_offset: number | null; term_count: number; has_arabic: number | null; has_english: number | null; @@ -73,7 +77,7 @@ function db(): Database.Database { // per dictionary per column. const DICT_SELECT = ` SELECT d.id, d.name_tech, d.name_arabic, d.name_english, d.name_french, - d.nbr_entries, d.wikidata_id, + d.nbr_entries, d.wikidata_id, d.archive_org_url, d.page_offset, COUNT(t.id) AS term_count, MAX(t.arabic IS NOT NULL AND t.arabic <> '') AS has_arabic, MAX(t.english IS NOT NULL AND t.english <> '') AS has_english, @@ -99,6 +103,21 @@ function toDictionary(row: DictionaryRow): Dictionary { }; } +/** + * The page of the dictionary's archive.org scan that shows printed page + * `page`, or null when there is no scan or no page. archive.org numbers the + * pages of the file from 0 (`n0` is the first page), and the file page of + * printed page p is p + page_offset (Wikidata's P8330). + */ +export function scanPageUrl( + dict: Pick, + page: number | null +): string | null { + if (!dict.archive_org_url || dict.page_offset === null || page === null) return null; + const n = page + dict.page_offset - 1; + return n < 0 ? dict.archive_org_url : `${dict.archive_org_url}/page/n${n}/mode/1up`; +} + export function getAllDictionaries(): Dictionary[] { const rows = db() .prepare(`${DICT_SELECT} GROUP BY d.id ORDER BY d.created_at DESC, d.id DESC`) diff --git a/website/src/pages/[dict]/index.astro b/website/src/pages/[dict]/index.astro index ff584e5..6982ea4 100644 --- a/website/src/pages/[dict]/index.astro +++ b/website/src/pages/[dict]/index.astro @@ -95,6 +95,12 @@ if (props.kind === 'redirect') {
{dict!.wikidata_id}
)} + {dict!.archive_org_url && ( +
+
النسخة المصوّرة
+
archive.org
+
+ )}
@@ -109,7 +115,7 @@ if (props.kind === 'redirect') { من أصل {dict!.term_count}.

- + )} diff --git a/website/src/pages/[dict]/page/[page].astro b/website/src/pages/[dict]/page/[page].astro index 32c700d..1b2194c 100644 --- a/website/src/pages/[dict]/page/[page].astro +++ b/website/src/pages/[dict]/page/[page].astro @@ -43,6 +43,6 @@ const description = `قاموس ${dict.name_arabic} — الصفحة ${page} م من أصل {dict.term_count}.

- +