diff --git a/docs/evaluation/triage/portugal.md b/docs/evaluation/triage/portugal.md new file mode 100644 index 0000000..cc38ef9 --- /dev/null +++ b/docs/evaluation/triage/portugal.md @@ -0,0 +1,37 @@ +# Portugal address search and autocomplete + +Triage of the Portugal rows in [#222](https://github.com/Mokto/opengeocoding/issues/222), on the 100 `global_osm_pt` cases (50 groups, 11 held out). Inspection (`evaluate inspect`) and probes used only the 39 tuning groups; held-out groups were scored, not inspected. + +## Cause + +Every Portuguese house in the index comes from one source, Overture's `pt/countrywide` addresses (5.6 million rows), which name streets as the national register (CTT) writes them: the leading street type abbreviated (`R`, `AV`, `TV`, `PCT`, `BR`, `CAM`, `ESTR`, `LG`, `URB`, `SIT`, `PC`, …), most titles abbreviated (`TEN`, `COMEND`, `CMDT`, `FR`, `COR`, `D` for Dom/Dona, `NSA` for Nossa, `S` for São, Santa and Santo), and usually without the particle after the type (`R IGREJA` 16,143 rows, `R DA IGREJA` 3,547; inner particles mostly stay, `R 25 DE ABRIL` 27,101 against `R 25 ABRIL` 2,611). The map's streets keep the full name (`Rua dos Farias`). Every query word must be in a street's name, so `12 Rua dos Farias` found the map's street, which has no houses, and answered at street level. The existing alternative spellings ran only when no street matched at all. + +A second, parser defect: libpostal reads a number written before `Avenida` as part of the road (`6 avenida da noruega`, no house number); `Rua`, `Av.`, `Travessa` and the other types split correctly. + +## Fix + +- `matching::road_expansions` gives a Portuguese road its register form: the type at the start (after a typed house number) and the titles the register abbreviates in the large majority of its rows, Santa/Santo as `S`, and the particle after the type dropped. `Marquês` (also a surname), `Monsenhor`, `Major` and `Padre` stay whole, as most register rows write them. +- `Index::streets` tries those spellings when the stated road has no such house, not only when no street matched. The existing guards hold: a stated town or a known postcode, a house that exists, a postcode not contradicted, and no answer when two spellings give different streets. +- SuggestAddress tries the register form when a typed Portuguese address with a number finds no house. Only for Portugal: South Africa's `straat`/`street` expansion turns back on itself, which made a general version recurse until it timed out. +- With Portugal stated, a number before `avenida` in the road becomes the house number. Not elsewhere: `6 Avenida` is a whole street name in Guatemala. + +The builder and index are unchanged. + +## Results + +Local full index `74a1eec034f5` (builder `9d4006a4eec15abb`), same evaluator, models and reference data on both sides, base `7a02601`: + +| `global_osm_pt` | Base | Candidate | +|---|---:|---:| +| SearchAddress | 22/100 | 46/100 | +| tuning groups | 9/39 | 19/39 | +| held-out groups | 2/11 | 4/11 | +| SuggestAddress (full) | 22/100 | 46/100 | +| ReverseGeocode country | 100/100 | 100/100 | + +The full 21,282-case gold diff has no newly failing or worse cases. All 48 newly passing checks are Portuguese; the one changed unscored address case, `4 Rua da Imprensa à Estrela, Portugal`, now resolves to house 4 on the register's `R IMPRENSA A ESTRELA S BENTO` (1200-873) about 100 m from the street point it gave before. Local timings were taken on a shared host under heavy load and are not a speed measurement. + +## Remaining tuning failures (20 groups) + +- **Number absent from the index (18):** no street of the name within 150 m holds the number: Rua da República 15 (Mirandela) and 18 (5200-233), Rua do Corredouro 39, Praceta Frei Rodrigo Egídio 8, Av. Gonçalo Rodrigues Caldeira 19, Rua São Nicolau 18, Rua das Amendoeiras 107, Avenida dos Combatente do Ultramar 1, Avenida Nacional 100, Rua Teófilo Carvalho dos Santos 28, Rua Senhora da Abadia 473, Rua da Ceifeira 3, Largo Doutor Augusto de Oliveira Coimbra 8, Rua Marquês de Pombal 51, Rua da Valinha 90, Rua Comendador Joaquim Mendes Almeida 15, Rua do Queiró 9, and Rua Principal 79 (Santalha), whose nearby street has no houses, so a same-named street 2.4 km away answers. The register street is present in most of these with other numbers; this is source coverage, not a name mismatch. +- **Names this fix does not bridge (2):** `Avenida Virgílio Ferreira 2` is indexed 3 m away as `Avenida Vergílio Ferreira` (a spelling difference between sources; the misspelling repair runs only when no street matches), and `Rua da GNR 23` as `R DA GUARDA NACIONAL REPUBLICANA` 84 m away (an acronym). Neither is generalized here. diff --git a/src/index/mod.rs b/src/index/mod.rs index 6eb3130..40109ac 100644 --- a/src/index/mod.rs +++ b/src/index/mod.rs @@ -1481,6 +1481,31 @@ fn unit_from_road(parsed: &mut Vec<(String, String)>) { } } +/// "6 Avenida da Noruega, Ribeira de Pena, Portugal": libpostal reads a number written before a +/// Portuguese avenida as part of the road (it splits "6 Rua …" and "6 Av. …"). Only with the +/// country stated as Portugal: "6 Avenida" is a whole street name in Guatemala. +fn number_from_avenida(parsed: &mut Vec<(String, String)>) { + if parsed.iter().any(|(k, _)| k == "house_number") + || crate::matching::expected_country(&parsed.iter().cloned().collect()) != "pt" + { + return; + } + let Some(at) = parsed.iter().position(|(k, _)| k == "road") else { + return; + }; + let Some((number, road)) = parsed[at].1.split_once(' ') else { + return; + }; + if !number.is_empty() + && number.bytes().all(|b| b.is_ascii_digit()) + && road.starts_with("avenida ") + { + let (number, road) = (number.to_string(), road.to_string()); + parsed[at].1 = road; + parsed.insert(at, ("house_number".into(), number)); + } +} + /// The process's one libpostal parser. pub struct Parser { postal: Mutex, @@ -1653,6 +1678,7 @@ impl Parser { None => whole, }; unit_from_road(&mut parsed); + number_from_avenida(&mut parsed); parsed.into_iter().collect() }; if self.capacity > 0 { @@ -2487,7 +2513,7 @@ impl Index { }) .collect(); let hits = self.streets_by(road, number, city, &named, postcode, cc, exact); - if !hits.iter().any(|h| { + let housed = hits.iter().any(|h| { matches!( h, Hit::Street { @@ -2495,8 +2521,8 @@ impl Index { .. } ) - }) && let Some(compact) = number.and_then(compact_house_suffix) - { + }); + if !housed && let Some(compact) = number.and_then(compact_house_suffix) { let found = self.streets(road, Some(&compact), city, postcode, cc, exact); if found.iter().any(|h| { matches!( @@ -2510,7 +2536,10 @@ impl Index { return found; } } - if hits.is_empty() && number.is_some_and(|n| !n.trim().is_empty()) { + // Another spelling of the road can hold the house even when the stated one names a + // street without it: Portugal's register writes "R FARIAS" beside the map's "Rua dos + // Farias". + if !housed && number.is_some_and(|n| !n.trim().is_empty()) { let mut alternatives = cc .map(|cc| crate::matching::road_expansions(road, cc)) .unwrap_or_default(); diff --git a/src/index/suggest.rs b/src/index/suggest.rs index ee7b52b..ad5e217 100644 --- a/src/index/suggest.rs +++ b/src/index/suggest.rs @@ -1813,6 +1813,29 @@ impl Index { return found; } } + // "12 Rua dos Farias": Portugal's register names it "R FARIAS". Only Portugal's form, + // which expands no further: other countries' turn back ("straat" <-> "street"). + if !out.iter().any(|s| s.kind == "address") + && typed.parts[0].split(' ').any(has_digit) + && cc.is_some_and(|c| c.eq_ignore_ascii_case("pt")) + { + for road in crate::matching::road_expansions(&typed.parts[0], "pt") { + let mut parts = typed.parts.clone(); + parts[0] = road; + let found = self.suggest_streets( + &Typed { + parts, + open: typed.open, + }, + cc, + focus, + want, + ); + if found.iter().any(|s| s.kind == "address") { + return found; + } + } + } out.truncate(want); out } diff --git a/src/matching.rs b/src/matching.rs index a382827..6cdd97e 100644 --- a/src/matching.rs +++ b/src/matching.rs @@ -605,6 +605,9 @@ pub fn road_agrees(address: &ParsedAddress, details: Option<&AddressDetails>) -> } pub(crate) fn road_expansions(road: &str, cc: &str) -> Vec { + if cc.eq_ignore_ascii_case("pt") { + return ctt_road(road); + } if cc.eq_ignore_ascii_case("za") { let road = normalize_place(road); return [(" straat", " street"), (" street", " straat")] @@ -675,6 +678,99 @@ pub(crate) fn road_expansions(road: &str, cc: &str) -> Vec { .collect() } +/// A Portuguese road as the national address register (CTT) writes it, which is how the index +/// stores most Portuguese houses: "Rua dos Farias" is "R FARIAS", "Bairro de Santa Bárbara" is +/// "BR S BARBARA", "Rua Tenente Valadim" is "R TEN VALADIM". The register abbreviates the +/// leading street type, most titles and Santa/Santo, and usually drops the particle after the +/// type. Only the abbreviations it uses for the large majority of its names are applied ("Marquês" is also a surname, "Monsenhor" mostly stays whole). +fn ctt_road(road: &str) -> Vec { + const TYPES: &[(&str, &str)] = &[ + ("praceta", "pct"), + ("bairro", "br"), + ("caminho", "cam"), + ("urbanizacao", "urb"), + ("sitio", "sit"), + ("praca", "pc"), + ("beco", "bc"), + ("quinta", "qta"), + ("calcada", "cc"), + ("calcadinha", "ccnh"), + ("monte", "mte"), + ("casal", "csl"), + ("vereda", "vr"), + ("canada", "can"), + ("quelha", "qlha"), + ("rotunda", "rot"), + ("escadinhas", "escnh"), + ("escadas", "esc"), + ("parque", "pq"), + ("azinhaga", "az"), + ("ladeira", "lad"), + ("loteamento", "loteam"), + ("jardim", "jrd"), + ("conjunto", "conj"), + ("zona", "zn"), + ("terreiro", "terr"), + ("rampa", "ramp"), + ("aldeamento", "aldt"), + ("herdade", "hrd"), + ("viela", "vla"), + ("patio", "pto"), + ]; + const TITLES: &[(&str, &str)] = &[ + ("tenente", "ten"), + ("comendador", "comend"), + ("comandante", "cmdt"), + ("almirante", "alm"), + ("conselheiro", "cons"), + ("visconde", "visc"), + ("brigadeiro", "brig"), + ("sargento", "sarg"), + ("infante", "inf"), + ("frei", "fr"), + ("coronel", "cor"), + ("arquitecto", "arq"), + ("arquiteto", "arq"), + ("duque", "dq"), + ("embaixador", "emb"), + ("doutora", "dra"), + ("dom", "d"), + ("dona", "d"), + ("nossa", "nsa"), + ("santa", "s"), + ("santo", "s"), + ]; + const PARTICLES: &[&str] = &["da", "do", "das", "dos", "de"]; + let road = normalize_place(road); + let short = |table: &[(&str, &'static str)], w: &str| { + table.iter().find(|(full, _)| *full == w).map(|(_, s)| *s) + }; + // the street type, after a house number typed first ("12 rua dos farias") + let at = road + .split(' ') + .position(|w| !w.starts_with(|c: char| c.is_ascii_digit())) + .unwrap_or(0); + let form = road + .split(' ') + .enumerate() + .filter(|&(i, w)| i != at + 1 || !PARTICLES.contains(&w)) + .map(|(i, w)| { + (i == at) + .then(|| short(TYPES, w)) + .flatten() + .or_else(|| short(TITLES, w)) + .unwrap_or(w) + }) + .collect::>() + .join(" "); + // a bare type ("Rua da") names no street + if form.contains(' ') && form != road { + vec![form] + } else { + vec![] + } +} + /// A Polish road as the index may name it: without "ul." (ulica), which most sources leave /// out, and without its given names' initials ("ul. M. Kopernika" is Mikołaja Kopernika's /// street). Without a stated country, only a road written with "ul." is taken to be Polish. diff --git a/tests/matching.rs b/tests/matching.rs index 642b6e8..9f38cd6 100644 --- a/tests/matching.rs +++ b/tests/matching.rs @@ -994,6 +994,14 @@ fn road_abbreviations_are_country_scoped_and_preserve_house_numbers() { ("nl", "Koeln", "Koelaan", false), ("nl", "Willowln.", "Willowstraat", false), ("pt", "R. du Bosquet", "Rua du Bosquet", true), + ("pt", "Rua dos Farias", "R FARIAS", true), + ("pt", "Rua Tenente Valadim", "R TEN VALADIM", true), + ("pt", "Bairro de Santa Bárbara", "BR S BARBARA", true), + ("pt", "Caminho Figueirinhas", "CAM FIGUEIRINHAS", true), + ("pt", "Rua dos Farias", "R FARIA", false), + ("pt", "Rua da Cruz", "TV CRUZ", false), + ("pt", "Rua Manuel da Fonseca", "R MANUEL FONSECA", false), + ("es", "Rua dos Farias", "R FARIAS", false), ("pt", "R. du Bosquet", "Rue du Bosquet", false), ("gb", "R. du Bosquet", "Rue du Bosquet", false), ("se", "Granvn.", "Granveien", false), diff --git a/tests/synthetic/main.rs b/tests/synthetic/main.rs index 4ae6905..f2d26cc 100644 --- a/tests/synthetic/main.rs +++ b/tests/synthetic/main.rs @@ -316,6 +316,33 @@ fn suggestions_find_houses_postcodes_and_names_in_other_languages() { assert!(delhi.contains(&("ca".into(), "town"))); } +#[test] +fn portuguese_houses_are_suggested_on_the_registers_abbreviated_street() { + let want = house(street("R FARIAS"), "12"); + for typed in [ + "12 Rua dos Farias, Pinhal, Portugal", + "Rua dos Farias 12, 2955-243 Pinhal, Portugal", + ] { + let found = index().suggest(typed, 5, None, None, false); + assert!(!found.is_empty(), "{typed}"); + assert_eq!(found[0].kind, "address", "{typed}"); + assert!(metres((found[0].x, found[0].y), want) < 5.0, "{typed}"); + } + // another number is not borrowed, nor the house from another country's same-named road + for (typed, cc) in [ + ("99 Rua dos Farias, Pinhal, Portugal", None), + ("12 Rua dos Farias, Pinhal", Some("ES")), + ] { + assert!( + index() + .suggest(typed, 5, cc, None, false) + .iter() + .all(|s| s.kind != "address"), + "{typed}" + ); + } +} + #[test] fn joined_city_and_region_suggestions_keep_both_constraints() { let index = index(); @@ -1119,6 +1146,22 @@ fn cases() -> Vec { "24 Birkensrasse, 50667 Köln, Germany", at_house("Birkenstrasse", "24"), ), + case( + "12 Rua dos Farias, 2955-243 Pinhal, Portugal", + at_house("R FARIAS", "12"), + ), + case( + "Rua dos Farias 12, Pinhal, Portugal", + at_house("R FARIAS", "12"), + ), + case( + "6 Avenida da Noruega, Pinhal, Portugal", + at_house("AV NORUEGA", "6"), + ), + case( + "99 Rua dos Farias, Pinhal, Portugal", + Want::At((-8.009, 39.5), 300.0, false), + ), case("Paris, France", town("fr-paris", true)), case("Missing Road, Paris, France", town("fr-paris", false)), case("Île-de-France, Haut-Rhin", town("fr-haut-rhin", true)), diff --git a/tests/synthetic/world.rs b/tests/synthetic/world.rs index 88781dd..5d99d9e 100644 --- a/tests/synthetic/world.rs +++ b/tests/synthetic/world.rs @@ -125,6 +125,30 @@ pub const PLACES: &[Place] = &[ "Veldhaven", &[("en", "Veldhaven")] ), + place!( + "pt", + "country", + "", + "PT", + "", + 10_000_000, + (-8.0, 39.5), + "", + "Portugal", + &[("en", "Portugal")] + ), + place!( + "pt-pinhal", + "locality", + "town", + "PT", + "", + 20_000, + (-8.0, 39.5), + "pt", + "Pinhal", + &[("en", "Pinhal")] + ), place!( "no", "country", @@ -1458,6 +1482,40 @@ pub const STREETS: &[Street] = &[ lat: -30.0, numbers: &["7"], }, + // Portugal's address register writes "R FARIAS" for the map's "Rua dos Farias" + Street { + class: "residential", + length: 0.002, + name: "Rua dos Farias", + postcode: "", + city: "Pinhal", + cc: "PT", + lon: -8.01, + lat: 39.5, + numbers: &[], + }, + Street { + class: "residential", + length: 0.0, + name: "R FARIAS", + postcode: "2955-243", + city: "Pinhal", + cc: "PT", + lon: -8.0095, + lat: 39.5, + numbers: &["10", "12"], + }, + Street { + class: "residential", + length: 0.0, + name: "AV NORUEGA", + postcode: "2955-244", + city: "Pinhal", + cc: "PT", + lon: -8.005, + lat: 39.5, + numbers: &["6"], + }, Street { class: "residential", length: 0.0,