Make v2 search case- and accent-insensitive

[jrhoads]

Purpose

Make v2 advanced (and bare-term) search case- and accent-insensitive, fixing ror-community/ror-roadmap#175 and ror-community/ror-roadmap#398 in one change.

Today query.advanced is a query_string over default_field="*". Most v2 keyword fields have no normalizer (exact-byte matching), and names.value / relationships.label use the standard analyzer (lowercase only, no accent folding). That makes queries like denver vs Denver, relationships.type:child vs Child, and Hue vs Huế / Universite de Montreal vs Université de Montréal miss.

Approach

Normalize keyword fields at index time with a shared folding_normalizer (lowercase + asciifolding), switch the main analyzers on names.value and relationships.label to the existing string_lowercase analyzer, NFC-normalize query/filter input in Python, and aggregate on un-normalized .raw subfields so facet keys keep their original casing.

Key Modifications

• rorapi/v2/index_template_es7.json
  • Add analysis.normalizer.folding_normalizer (lowercase + asciifolding).
  • Apply it to: locations.geonames_details.{name,country_name,country_code,continent_name,continent_code,country_subdivision_name,country_subdivision_code}, types, status, relationships.type, names.lang, names.types, links.type, external_ids.type. Leave id, external_ids.all, external_ids.preferred, dates, and numbers untouched.
  • Give each aggregated field (types, status, country_code, continent_code) a fields.raw: {type: keyword} subfield.
  • Set names.value and relationships.label main analyzer to string_lowercase (keep .keyword / .norm subfields).
• rorapi/common/queries.py
  • NFC-normalize query, query.advanced, and filter values before building the ES query.
  • Point aggregations at types.raw, locations.geonames_details.country_code.raw, locations.geonames_details.continent_code.raw, and status.raw.
  • Keep the existing Python filter casing logic (harmless with the normalizer).
• Tests
  • Unit: assert .raw aggregations and NFC normalization of query/filter input.
  • Integration: case/diacritic/NFD–NFC equivalence for keyword and text fields, plus facet titles keeping original casing.

Important Technical Details

• Index recreate / reindex required. Template changes only take effect on a newly created index. After deploying to staging/prod, recreate and reindex the v2 index (createindex, then indexror / indexrordump) together with the deploy. Until then, the new *.raw aggregation fields will not exist (facets empty on the old index) and search behavior will not change.
• ES 7.10 in CI has no ICU plugin; asciifolding does not reliably collapse NFD combining marks, so Python NFC normalization of query/filter input is required.
• A normalizer on a keyword field also normalizes aggregation bucket keys, which is why facets move to .raw.
• Integration tests are not run in CI (run_tests.yml only runs tests_unit); they need an index built from the current template.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)

Reviewer, please remember our guidelines:

• Be humble in the language and feedback you give, ask don't tell.
• Consider using positive language as opposed to neutral when offering feedback. This is to avoid the negative bias that can occur with neutral language appearing negative.
• Offer suggestions on how to improve code e.g. simplification or expanding clarity.
• Ensure you give reasons for the changes you are proposing.

Test plan

• CI (test-pr / test and flow-guard pass; ruff soft-fails repo-wide with continue-on-error, unrelated to this change)
• Unit tests (rorapi.tests.tests_unit) pass locally aside from pre-existing test_heartbeat_success without MySQL
• Verified against Elasticsearch 7.10.0 with an index from the new template: denver/Denver, relationships.type:child/Child, stellenbosch/Stellenbosch, Hue/Huế, Montreal/Montréal, NFD vs NFC Huế, and names.value:"Universite de Montreal" return matching results; facet titles keep original casing. The new integration tests fail against the old template and pass against the new one.
• After deploy: recreate and reindex the v2 index on staging, re-run the integration tests in tests_search_v2.py, then repeat on prod

  

[Copilot]

Copilot review overview

🔵 Needs a closer look

The mapping migration requires a coordinated production reindex, and its integration coverage is not exercised by CI.

Review effort: Balanced
Findings: None

What changed in this PR

Makes v2 search case-, accent-, and Unicode-normalization-insensitive while preserving facet casing.

Changes:

• Adds folding normalizers and raw aggregation fields.
• NFC-normalizes query and filter input.
• Adds unit and integration coverage.

File	Description
rorapi/​v2/​index_template_es7.json	Configures normalized search fields and raw facet fields.
rorapi/​common/​queries.py	Normalizes input and targets raw aggregations.
rorapi/​tests/​tests_unit/​tests_queries_v2.py	Tests normalization and aggregation fields.
rorapi/​tests/​tests_integration/​tests_search_v2.py	Tests case/accent equivalence and facet casing.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.