fix(pipelines): cite published docs URLs and align the incremental cleaner with the full run

[Neilblaze]

Fixes #243

Why

Two things in the docs ingestion path, both in logic the full and incremental pipelines carry as separate copies.

Citations are not the URLs the site publishes. chunk_and_embed cites <base_url>/<path minus extension>. Hugo publishes a page at that path with a trailing slash, and publishes a section's _index.md at the section directory. Of the 182 pages the pipeline indexes from kubeflow/website@master, none of the URLs we emit resolve directly: 169 redirect, and 13 — every _index.md section landing page, including the KFP installation guide — return 404. Since #237 those URLs are the source pills the widget renders under tool-grounded answers.

The incremental cleaner diverged from the full one. Both write into kubeflow_docs, but chunk_and_embed_incremental still runs the pre-#234 cleaner. The two no longer agree on which pages to index (182 vs 183 out of the same 220 files), and of the pages both keep, 101 lose text under the incremental one — 181,491 characters, 23.8% of the corpus, up to 96% of a single page. Nothing automates that pipeline today, so this is latent, but pipelines/README.md recommends it for updates after the first full build.

What

File	Change
pipelines/utils.py	adds build_docs_citation_url(); clean_content() updated to the cleaner chunk_and_embed actually runs
pipelines/kubeflow-pipeline.py	chunk_and_embed calls the two helpers instead of inlining the logic
pipelines/incremental-pipeline.py	same helpers; max_tei_chars becomes a parameter; chunk defaults come from DEFAULT_DOCS_*
pipelines/README.md	incremental parameter table: chunk defaults 1200/100 → 600/60, plus the new max_tei_chars row
.github/workflows/oke-cicd.yaml	KFP-compile incremental-pipeline.py, which the step was skipping while compiling its three siblings
tests/	8 URL mapping cases, 3 cleaner regressions, and two component tests that pin both inline copies to utils

A KFP component runs in its own container and cannot import utils at runtime, so the helpers are mirrored inline under the same names and the tests pin the copies to the shared versions. A real shared import is not available here, so this does not close #81.

Verification

Requested every citation URL the pipeline builds for the 182 pages it indexes, without following redirects:

status	before	after
200	0	182
301	169	0
404	13	0

Content is unchanged for the full pipeline: over all 220 files under content/en/docs, the new utils.clean_content output is byte-identical to the inline cleaner chunk_and_embed ran before this change, and the indexed set is still the same 182 pages. Only citation_url moves.

• pytest: 156 passed locally with the KFP SDK installed; 148 passed / 8 skipped with only requirements-test.txt, against 137 / 6 on main
• both pipelines compile: python kubeflow-pipeline.py, python incremental-pipeline.py
• ruff check docs-agent-mcp/session-issuer tests docs-agent-mcp/pipelines: clean
• the cross-component test does bite: dropping a cleaning step from either inline copy, or breaking its _index case, fails it

---

Some notes for reviewers

• Existing red CI. main fails both steps of the Lint and format job today. ruff check . reports F811 for the duplicate import json in mcp-server/server.py, and ruff format --check reports 3 files. This branch leaves both to fix(ci): satisfy ruff lint and format checks on main #241 and adds no findings; it does take the format count from 3 to 2, because appending tests to tests/test_docs_pipeline.py overwrites the trailing blank line that issue Bug: PR Safety workflow failing on main — ruff format check fails for tests/test_docs_pipeline.py #236 and PR Fix: Format tests/test_docs_pipeline.py to pass ruff checks #235 are about.

• Where the new tests run. The two component tests need the KFP SDK. PR Safety installs only requirements-test.txt, so it skips them, the same way it already skips the existing component tests. Build, Test, and Deploy to OKE installs kfp kfp-kubernetes and runs on every pull request, so they do execute there. Happy to add kfp to requirements-test.txt if you would rather both jobs cover it.

• Rollout. No migration, but the fix lands page by page. store_milvus deletes and re-inserts every (repo_name, file_path) it processes, so one full run rewrites all 182 citations. The incremental pipeline deletes by file_unique_id in delete_old_vectors and only touches changed files, so running it alone leaves the rest of the collection on the old URLs.

• Likely conflicts. Fix: issues pipeline chunk_size (1500) exceeds TEI embedding truncation limit (1000) #212 rewraps the same from utils import ... statement in incremental-pipeline.py. Fix: Format tests/test_docs_pipeline.py to pass ruff checks #235/fix(ci): satisfy ruff lint and format checks on main #241/feat: improve RAG retrieval and adopt Helm upgrades #240 each touch the tail of tests/test_docs_pipeline.py, and feat: improve RAG retrieval and adopt Helm upgrades #240 also touches oke-cicd.yaml. Preserve newlines in markdown cleaning for Langchain chunking #147 does the \s+ → [ \t]+ half of the cleaner fix on a path that has since moved.

• Left alone. The ZeroDivisionError in the incremental component's chunk-count print (bug(pipelines): ZeroDivisionError in chunk_and_embed_incremental when text splitter returns empty chunks #163) is two lines from this diff; leaving it to fix(pipelines): guard against ZeroDivisionError in chunk_and_embed_incremental when chunk list is empty #164. store_milvus_incremental still hardcodes dim=768 and has no schema check, unlike store_milvus — out of scope here.

[google-oss-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign franciscojavierarceo for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment