Skip to content

Prepare wikipedia 1.5.0 - #356

Draft
goldsmith wants to merge 3 commits into
masterfrom
codex/wikipedia-public-review
Draft

goldsmith wants to merge 3 commits into
masterfrom
codex/wikipedia-public-review

Conversation

@goldsmith

Copy link
Copy Markdown
Owner

Fixes incorrect page matches, hanging requests, and errors when loading page
data. Adds regression tests, wheels, and CI for Python 3.10–3.14.

Changes

  • Check exact titles before using search suggestions.
  • Add HTTPS, connection reuse, configurable timeouts and bounded retries.
  • Fix redirects, disambiguation, sections, HTML and missing page properties.
  • Keep page objects tied to their original language. Limit module caches to
    256 entries each for five minutes and protect cached lists from mutation.
  • Update packaging, dependencies and documentation.

Compatibility

Python 3.10 is the new minimum. Existing function names and positional arguments
are preserved. auto_suggest=True remains the default, but exact disambiguation
pages now raise instead of being replaced by a suggestion.

Requests default to a 5-second connect timeout and a 15-second read timeout,
with at most two retries. RequestError covers HTTP/API failures;
HTTPTimeoutError remains available. Missing lists return [] and absent
coordinates return None.

Tests

143 offline tests pass on Python 3.10–3.14 on macOS, including all 32 legacy
tests. Clean wheel/source installs and four LangChain integration tests pass.
Live checks passed on English, French and Japanese Wikipedia. Linux and Windows
results will be checked in CI.

Remaining work

This overlaps with Akber's packaging and HTTP PRs, listed below. Trusted
Publishing and Dependabot can be added on top. The remaining differences need
review before combining the changes.

Backlog

All 163 issues and 45 PRs open on September 28 are listed below. Related issues
are grouped. Some reports still need their original examples checked; this is
not a list of issues to close automatically.

45 pull requests
PR This branch Notes
#355 To compare Akber’s HTTP and title-lookup PR overlaps with this branch. Compare timeout, retry, title-resolution and User-Agent changes before combining them.
#354 To combine Akber’s packaging PR overlaps with this branch. Trusted Publishing and Dependabot are still to add; Python and dependency versions need agreement.
#351 Included unchanged Niki’s equality fix, with original authorship preserved.
#343 Handled here Exact titles are tried first; auto_suggest=True remains the default.
#336 Left for later Formatting and type hints can be a separate PR.
#333 Left for later Proxy environment variables are documented. A new setter can follow separately.
#310 Handled here Exact lookup now happens before search, including when search returns nothing.
#305 Handled here Missing titles get one suggestion fallback. Unrelated formatting is left out.
#296 Handled here Credits are retained in the README and changelog.
#287 Patch not used Indexes the first result before checking for an empty response. Both cases are tested here.
#277 Left for later Including subsections would change section()’s documented behavior.
#261 Adapted Tim Gates’s spelling correction is included.
#253 Patch not used Uses comparison instead of assignment. Title lookup is fixed separately here.
#236 Patch not used maxlag errors get bounded retries here. A configurable maxlag parameter is still deferred.
#235 Left for later Error construction is fixed. Input trimming needs a separate decision.
#230 Patch not used Language validation should not require CSV files or numpy.
#219 Partly adapted Anna Katrine Jørgensen’s geosearch and coordinate fixes; revision history is deferred.
#215 Patch not used Whitespace-only license change. The license is unchanged.
#206 Left for later Existing proxy environment variables work. A new proxy API is deferred.
#205 Partly adapted rowan08’s missing-property handling, with additional response checks. Registry changes are left out.
#195 Handled here Dependency bounds are updated and tested.
#187 Left for later New revision-timestamp property deferred.
#182 Left for later Needs a signature that preserves existing positional calls.
#179 Left for later Language-link lookup needs its own API and pagination tests.
#173 Left for later Category traversal needs pagination and namespace handling.
#172 Handled here Old examples are replaced.
#170 Handled here Dead badges are removed.
#160 Adapted Christopher Beland’s HTTPS proposal is included.
#159 Left for later Custom endpoints need a separate support decision. Tests can replace the internal transport.
#157 Adapted Michael E. Rose’s absent-coordinate handling; malformed data still raises an error.
#147 Left for later Exposing arbitrary pageprops is deferred.
#137 Adapted Emmanuel Bizieau’s timeout proposal, applied to every request.
#132 Adapted mkasprz’s handling of unavailable image URLs, with response validation.
#131 Handled here One fallback replaces recursive suggestion attempts.
#128 Already on master Explicit html.parser selection is retained.
#121 Handled here Tests now use isolated fixtures and run against installed packages.
#115 Already on master The page-parameter fix is retained; TOC handling is updated.
#112 Already on master Explicit html.parser selection is retained.
#110 Left for later Pagination needs to preserve existing positional calls.
#109 Handled here Documentation and examples are updated.
#101 Already on master The section-request parameter fix is retained.
#100 Left for later Translation lookup needs a separate API and tests.
#92 Left for later Extra disambiguation context would change the return type.
#84 Already on master The section-request parameter fix is retained.
#64 Adapted David Lutz’s connection-reuse proposal, without a new public session API.
163 issues, grouped by problem
Reports What this PR does / what remains Issues
Wrong page or missing page (29) Exact titles are checked before search. Tests cover normalization, typos and missing pages; the original examples still need confirmation. #346, #344, #342, #340, #321, #320, #316, #314, #307, #304, #295, #291, #273, #266, #258, #256, #227, #217, #204, #199, #192, #190, #186, #108, #104, #102, #85, #58, #54
Suggestion ranking still needs work (2) Partly addressed. When the exact title is absent, suggestions can still outrank better search results. #284, #259
Connection failures, busy servers and invalid responses (20) HTTPS, timeouts, retries and clearer errors are added. Outages, blocked requests and server settings still need investigation. #353, #334, #303, #294, #292, #281, #271, #242, #220, #202, #197, #175, #155, #154, #152, #142, #130, #122, #98, #96
Missing links, images and other page data (8) Missing properties and pagination are tested, including repeated tokens and failed requests. The original hanging and page-size examples need confirmation. #290, #285, #133, #93, #87, #78, #71, #65
Disambiguation options (2) Options now use unique article titles instead of display labels. #244, #79
Language and geosearch (1) Geosearch clears its cache after a language change. Existing pages keep their original language. #218
Sections (5) TOC parsing, empty sections and heading matching are fixed. section() still stops at the next heading. #168, #119, #105, #95, #77
Redirects and page IDs (2) Page-ID redirects follow MediaWiki’s response and respect redirect=False. #164, #118
Empty queries (10) Error construction is fixed. Trimming or rejecting empty input consistently is still deferred. #262, #233, #223, #213, #203, #200, #185, #162, #135, #89
User agents (3) Application names and contact details are documented. set_user_agent() remains optional. #350, #322, #126
Text formatting and collection order (8) Current limits are documented. Article layout, rich markup and citation extraction are unchanged. #330, #288, #237, #183, #145, #140, #114, #106
Parser warnings (5) The parser fix was already on master and is retained. #323, #279, #264, #225, #107
Search length limit (1) The API error is exposed. The upstream search limit is unchanged. #318
Packaging and old badges (3) Packaging and CI are updated; stale badges are removed. Old installation failures may still need reproduction. #270, #254, #167
A new PyPI release (1) Builds and release notes are prepared. Nothing has been published to PyPI. #251
Old Python or copied-install syntax errors (2) Python 3.10+ installs are tested. Old interpreters and manually copied installations remain unverified. #81, #67
Async and independent client configuration (3) Deferred. Configuration remains global; async support needs a separate design. #338, #337, #146
Infoboxes, tables and link labels (5) Deferred. Existing text and page properties are retained. #315, #212, #178, #111, #221
Categories and category members (3) Category names are supported. Category traversal is deferred. #265, #180, #134
Other languages and MediaWiki sites (6) Translation lookup and custom endpoints are deferred. Existing pages keep their original language. #188, #156, #144, #127, #117, #88
Proxy configuration (2) Requests’ proxy environment variables are documented. A new setter is deferred. #313, #184
More ways to identify or inspect pages (6) Linked IDs, URL lookup, page status, history and redirect lists are deferred. #232, #231, #181, #166, #113, #90
Other new query features (5) Random-page filtering, more search results, user information and batch queries are deferred. #228, #139, #125, #97, #70
A page with a disambiguation companion (1) An article can have a separate disambiguation page without being one itself. Detection uses pageprops. #306
Search rankings (1) Search ordering comes from MediaWiki. Exact-title lookup does not change search() rankings. #289
ODD_ERROR_MESSAGE (1) The constant remains available. Redirect handling no longer relies on it. #272
Handling missing or ambiguous pages (3) Callers should handle PageError and DisambiguationError and choose from the returned options. #224, #151, #136
Unicode input (1) Pass a Unicode title, not an already URL-encoded string. #189
Project-list inclusion (1) Requires a maintainer reply; no code change. #174
Geosearch radius (1) MediaWiki’s radius constraint is unchanged. Invalid parameters raise an API error. #169
Summary URL (1) Use page(title).url and page(title).summary. #161
Simple English Wikipedia (1) Use set_lang("simple"). #158
Reports needing a reproducible example (8) Need a small example, package/Python versions and the actual error response. #329, #328, #325, #324, #319, #302, #257, #226
Authorization-header error (1) Need response and environment details. This package does not add access-token authentication. #348
Missing exception attributes (1) Exception attributes are tested. The reported failure still needs a reproducible example. #250
Missing fullurl (1) Malformed metadata raises RequestError. The original cause remains unconfirmed. #239
French disambiguation report (1) Need the pageprops response to check whether the page was actually a disambiguation page. #201
Console encoding (1) Unicode repr() is fixed and tested. Output-stream encoding and the original screenshot remain unverified. #83
Reports outside this package (7) No package change proposed. These reports concern websites or unrelated content. #345, #341, #332, #327, #326, #283, #275

Niki and others added 3 commits September 28, 2026 13:07
Bare except: catches SystemExit and KeyboardInterrupt.

(cherry picked from commit 8ccc0f2)
Resolve exact pages before a bounded suggestion fallback. Add configurable timeouts, typed transport failures and bounded retries; bind lazy properties to their originating language and bound module caches. Handle empty and malformed metadata, continuation and supported parsing responses consistently.

Retain the legacy suite with isolated fixtures and add deterministic transport, page, cache, API contract and framework tests. The offline suite passes 143 tests; the optional LangChain tests run separately.
Add modern package metadata, CI, clean-install checks and opt-in live probes. Document behavior changes, installation from the development branch, configuration and limits; credit the earlier contributions incorporated or adapted in this work.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant