feat(markdown): convert crawled HTML and documents to Markdown via convert-markdown.ifad.org

[lleirborras]

Summary

Adds an opt-in markdown_conversion feature: when enabled, every crawled HTML page and PDF/DOCX/XLSX/PPTX is uploaded to the shared IFAD service convert-markdown.ifad.org and the returned Markdown becomes the Elasticsearch body, so all IFAD apps index identical conversions. Plain-text extraction (HTML) and the _attachment Tika path (binaries) remain the fallback.

Design spec: docs/superpowers/specs/2026-09-04-markdown-conversion-design.md. Feature doc: docs/features/MARKDOWN_CONVERSION.md.

How it works

• Coordinator#process_crawl_result calls config.markdown_converter.convert! on the crawl-task thread after the rule engine and before the sink, so no HTTP call ever runs under the Elasticsearch sink's queue lock. Coordinator delta is one call site plus one small helper.
• New Crawler::MarkdownConverter (lib/crawler/markdown_converter.rb) owns everything: MIME map, Jsoup pre-processing (honours exclude_tags / data-elastic-exclude, UTF-8 charset, clone so link extraction is untouched), hand-built multipart upload via net/http, submit + poll with backoff and a hard deadline, retry-once rules, SSRF guard on the server-supplied status_url, health check, circuit breaker, atomic stats.
• DocumentMapper: body = markdown when present, plus body_format (markdown|text) and content_hash on every doc; _attachment omitted only when markdown is present.
• Elasticsearch sink defaults _reduce_whitespace to false when the feature is on (the ingest pipeline would otherwise flatten markdown); an explicit true is honoured with a warning.
• Crawl#start! and start_url_test! fail fast with Errors::MarkdownConverterUnavailableError when the converter is unhealthy, so an outage cannot silently produce a text-only index. Mid-crawl outages trip a circuit breaker (20 consecutive failures, 60 s cooldown, single-thread re-probe).

Config

markdown_conversion:
  enabled: true
  base_url: https://convert-markdown.ifad.org
  wait_seconds: 10
  poll_interval: 2
  timeout: 900
  on_failure: text   # or skip — see doc for the purge interaction
  ca_file:           # optional PEM

Note: changes that apply even when disabled

body_format and content_hash are added to every document (HTML and binary) regardless of enabled, and both names are reserved for extraction rules. Deliberate: content_hash enables a later skip-unchanged re-crawl. Documented in the feature doc.

Verification

• Full suite in the crawler-ci Docker image: 791 examples, 0 failures, 16 pre-existing pending. Rubocop project-wide clean.
• New coverage: unit specs for config validation, converter core/transport (WebMock, every retry/poll/deadline/SSRF branch), coordinator hook incl. strict-double disabled path, mapper, sinks; integration spec crawling a Faux site (HTML + PDF) against a stubbed converter for success, 5xx fallback and unhealthy-abort.
• Manual: with bin/crawler's JVM properties (https.protocols=SSLv3, force.http.jre.executor=true) set inside the container, Net::HTTP reached https://convert-markdown.ifad.org/api/v1/health and got HTTP 200, proving jruby-openssl is unaffected by that property.

Follow-ups (not in this PR)

• Service side: replica affinity or shared job store (the in-process job store 404s polls that land on another replica), light/heavy executors, content-hash result cache.
• Crawler phase 3: skip-unchanged re-crawl using content_hash.

🤖 Generated with Claude Code

https://claude.ai/code/session_01XMpa3bomxH6WW3GBBfmoVZ