Skip to content

deep: bugs, real-fixture tests, docs/examples - #13

Merged
JarbasAl merged 1 commit into
devfrom
deep/qa
Aug 3, 2026
Merged

JarbasAl merged 1 commit into
devfrom
deep/qa

Conversation

@JarbasAl

@JarbasAl JarbasAl commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Bugs fixed (with evidence)

  1. Step pictures were always empty, and could crash the parser — pywikihow/__init__.py::_parse_pictures. wikiHow no longer nests <a class="image"> inside div.step, and no longer sets data-src on the <img> (it now uses src). The old code assumed both:

    • It always found data-src= nowhere in the <img> tag, so HowToStep.picture was None for every step, on every page. Verified live against https://www.wikihow.com/Tie-a-Tie.
    • Worse, on https://www.wikihow.com/Fold-a-Fitted-Sheet (a multi-part article with more <a class="image"> links than steps), the old sequential-index mapping raised IndexError: list index out of range inside _parse_pictures, which was then masked as a generic ParseError by the bare except Exception: raise ParseError — i.e. a real page that should parse fine crashed instead. Fixed by matching each image to its step via the Step-<n> token in its href (the article-global 1-based step number, which lines up with the flat, DOM-ordered self._steps list), and reading img.get("src") (falling back to data-src for forward-compat).
  2. Non-ASCII titles left percent-encoded — _parse_title built the title from the URL slug without decoding it. Verified live: https://pt.wikihow.com/Contar-C%C3%A9lulas-no-Google-Planilhas-no-Windows-ou-Mac produced title == "Contar C%C3%A9lulas no Google Planilhas..." instead of "Contar Células no Google Planilhas...". Fixed with urllib.parse.unquote.

  3. Search queries weren't URL-encoded — WikiHow.search only replaced spaces with +; any other special character (&, #, non-ASCII, etc.) went into the URL raw, silently truncating or corrupting the query. Fixed with urllib.parse.quote.

  4. ParseError discarded the real cause — except Exception as e: raise ParseError dropped e entirely, which is exactly what hid bug Suggestion: iterate over pages until max_results satisfied #1's IndexError as an unhelpful, undebuggable ParseError. Now uses raise ParseError(str(e)) from e.

Tests added (tests/test_pywikihow.py, 13 tests)

  • Happy path: title/url/intro/steps parsing, as_dict(), lazy parsing.
  • Picture bug regressions: pictures are populated from src, correlate to the correct step (not sequential-index order), and the multi-part fixture that used to raise IndexError now parses cleanly.
  • Unicode title regression: %C3%A9 decodes to é.
  • Search: encodes special characters, respects max_results, raises UnsupportedLanguage for HowTo/RandomHowTo/search.
  • Malformed input: a real "page not found" fixture raises ParseError with the original exception preserved as __cause__.

No network access during tests — pywikihow.get_html is monkeypatched to replay recorded fixtures.

Fixtures recorded (source URLs + when, 2026-08-02/03)

All fetched live, once, via the package's own get_html/requests transport, saved as raw HTML under tests/fixtures/:

  • tie_a_tie.html — https://www.wikihow.com/Tie-a-Tie
  • fold_fitted_sheet.html — https://www.wikihow.com/Fold-a-Fitted-Sheet
  • contar_celulas_pt.html — https://pt.wikihow.com/Contar-C%C3%A9lulas-no-Google-Planilhas-no-Windows-ou-Mac
  • search_bitcoin.html — https://www.wikihow.com/wikiHowTo?search=buy%20bitcoin
  • not_found.html — https://www.wikihow.com/Special:ThisPageDoesNotExist123456 (404, used for the malformed-input test)

Docs/examples added

  • README: documented the previously-undocumented HowToStep.picture field.
  • New examples/step_pictures.py showing how to read step pictures.

Packaging

Added a minimal pyproject.toml [build-system] section and a [test] extra in setup.py (pip install -e ".[test]"). Full pyproject-only migration was not done: .github/workflows/release_workflow.yml and publish_stable.yml call the shared TigreGotico/gh-automations reusable workflows with a setup_py: 'setup.py' input and run python setup.py --version / setup.py sdist bdist_wheel directly — removing setup.py would break the release pipeline without also verifying/updating that external reusable workflow, so setup.py stays as the CI-facing entry point.

Model usage

Claude Sonnet 5.

wikiHow moved step images out of their `div.step` and dropped the
`data-src` attribute, so `_parse_pictures` always produced empty
picture URLs and, on pages with more images than steps, could raise
an uncaught IndexError (masked as a generic ParseError with the
original cause discarded). Pictures are now matched to their step via
the `Step-<n>` token in the image href, which is the article-global
step number and lines up with the flat, DOM-ordered steps list.

Also fixes non-ASCII article titles being left percent-encoded
(e.g. "C%C3%A9lulas" instead of "Células"), missing URL-encoding of
special characters in search queries, and preserves the original
exception as __cause__ when a page fails to parse.

Adds pytest coverage against recorded real wikiHow fixtures (no
network access during tests), an example for the picture field, a
minimal pyproject.toml build-system section, and a [test] extra in
setup.py.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@JarbasAl
JarbasAl marked this pull request as ready for review August 3, 2026 10:29
@JarbasAl
JarbasAl merged commit 4222d40 into dev Aug 3, 2026
1 of 2 checks passed
@JarbasAl
JarbasAl deleted the deep/qa branch August 3, 2026 10:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant