Repository navigation
Conversation
wikiHow moved step images out of their `div.step` and dropped the `data-src` attribute, so `_parse_pictures` always produced empty picture URLs and, on pages with more images than steps, could raise an uncaught IndexError (masked as a generic ParseError with the original cause discarded). Pictures are now matched to their step via the `Step-<n>` token in the image href, which is the article-global step number and lines up with the flat, DOM-ordered steps list. Also fixes non-ASCII article titles being left percent-encoded (e.g. "C%C3%A9lulas" instead of "Células"), missing URL-encoding of special characters in search queries, and preserves the original exception as __cause__ when a page fails to parse. Adds pytest coverage against recorded real wikiHow fixtures (no network access during tests), an example for the picture field, a minimal pyproject.toml build-system section, and a [test] extra in setup.py. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bugs fixed (with evidence)
Step pictures were always empty, and could crash the parser —
pywikihow/__init__.py::_parse_pictures. wikiHow no longer nests<a class="image">insidediv.step, and no longer setsdata-srcon the<img>(it now usessrc). The old code assumed both:data-src=nowhere in the<img>tag, soHowToStep.picturewasNonefor every step, on every page. Verified live againsthttps://www.wikihow.com/Tie-a-Tie.https://www.wikihow.com/Fold-a-Fitted-Sheet(a multi-part article with more<a class="image">links than steps), the old sequential-index mapping raisedIndexError: list index out of rangeinside_parse_pictures, which was then masked as a genericParseErrorby the bareexcept Exception: raise ParseError— i.e. a real page that should parse fine crashed instead. Fixed by matching each image to its step via theStep-<n>token in itshref(the article-global 1-based step number, which lines up with the flat, DOM-orderedself._stepslist), and readingimg.get("src")(falling back todata-srcfor forward-compat).Non-ASCII titles left percent-encoded —
_parse_titlebuilt the title from the URL slug without decoding it. Verified live:https://pt.wikihow.com/Contar-C%C3%A9lulas-no-Google-Planilhas-no-Windows-ou-Macproducedtitle == "Contar C%C3%A9lulas no Google Planilhas..."instead of"Contar Células no Google Planilhas...". Fixed withurllib.parse.unquote.Search queries weren't URL-encoded —
WikiHow.searchonly replaced spaces with+; any other special character (&,#, non-ASCII, etc.) went into the URL raw, silently truncating or corrupting the query. Fixed withurllib.parse.quote.ParseErrordiscarded the real cause —except Exception as e: raise ParseErrordroppedeentirely, which is exactly what hid bug Suggestion: iterate over pages until max_results satisfied #1'sIndexErroras an unhelpful, undebuggableParseError. Now usesraise ParseError(str(e)) from e.Tests added (
tests/test_pywikihow.py, 13 tests)as_dict(), lazy parsing.src, correlate to the correct step (not sequential-index order), and the multi-part fixture that used to raiseIndexErrornow parses cleanly.%C3%A9decodes toé.max_results, raisesUnsupportedLanguageforHowTo/RandomHowTo/search.ParseErrorwith the original exception preserved as__cause__.No network access during tests —
pywikihow.get_htmlis monkeypatched to replay recorded fixtures.Fixtures recorded (source URLs + when, 2026-08-02/03)
All fetched live, once, via the package's own
get_html/requeststransport, saved as raw HTML undertests/fixtures/:tie_a_tie.html—https://www.wikihow.com/Tie-a-Tiefold_fitted_sheet.html—https://www.wikihow.com/Fold-a-Fitted-Sheetcontar_celulas_pt.html—https://pt.wikihow.com/Contar-C%C3%A9lulas-no-Google-Planilhas-no-Windows-ou-Macsearch_bitcoin.html—https://www.wikihow.com/wikiHowTo?search=buy%20bitcoinnot_found.html—https://www.wikihow.com/Special:ThisPageDoesNotExist123456(404, used for the malformed-input test)Docs/examples added
HowToStep.picturefield.examples/step_pictures.pyshowing how to read step pictures.Packaging
Added a minimal
pyproject.toml[build-system]section and a[test]extra insetup.py(pip install -e ".[test]"). Full pyproject-only migration was not done:.github/workflows/release_workflow.ymlandpublish_stable.ymlcall the sharedTigreGotico/gh-automationsreusable workflows with asetup_py: 'setup.py'input and runpython setup.py --version/setup.py sdist bdist_wheeldirectly — removingsetup.pywould break the release pipeline without also verifying/updating that external reusable workflow, sosetup.pystays as the CI-facing entry point.Model usage
Claude Sonnet 5.