Index HTML documents under canonical URLs and populate additional_urls - #495
Open
Jan-Kazlouski-elastic wants to merge 4 commits into
Open
Index HTML documents under canonical URLs and populate additional_urls#495Jan-Kazlouski-elastic wants to merge 4 commits into
Jan-Kazlouski-elastic wants to merge 4 commits into
Conversation
When a crawled page declares a different canonical URL, store the canonical in url/id and record the fetched URL in additional_urls instead of enqueueing a separate crawl for the canonical page.
Keep alternate URLs discovered for the same canonical page in memory so later writes include every variant, including when the canonical URL itself is crawled directly.
Use a self-referential canonical URL in the matching-url example and expect additional_urls in sorted order when accumulating variants.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of https://github.com/elastic/sdh-search/issues/1986
Part of #261
When a crawled HTML page declares a canonical URL that differs from the fetched URL, index the document under the canonical URL and record the fetched URL in
additional_urls. This matches the behavior described indocs/features/CRAWLER_DIRECTIVES.mdand prevents duplicate documents for URL variants such as tracking query parameters.Changes:
DocumentMapperuses the canonical URL forurland documentidwhen a valid canonical tag points elsewhereadditional_urlsrecords alternate URLs discovered during the crawl, including multiple variants of the same canonical pageCoordinatorno longer enqueues a separate crawl for the canonical URL, since the page content is already available from the variantRelated context: search-team#3139 discusses canonical tags and duplicate-content handling in Enterprise Crawler. That issue is closed and out of scope here — Open Crawler does not implement duplicate-content detection, and this PR does not add content fingerprinting for pages whose canonical URL differs in body or links.
Checklists
Pre-Review Checklist
crawler.yml.exampleandelasticsearch.yml.example)v0.1.0)make noticeif any dependencies have been addedChanges Requiring Extra Attention
Related Pull Requests
Release Note
HTML pages with a canonical URL link tag are now indexed under the canonical URL. Alternate crawled URLs for the same page are stored in
additional_urls, reducing duplicate documents for URL variants.