Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 20 additions & 1 deletion docs/evaluation/global-addresses.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,20 @@ An early admission pilot found phone numbers and Plus Codes misfiled as house nu

Keep the authoritative registry-first corpora and curated global place tests. This benchmark complements them by revealing missing index coverage, number-role failures and country/format problems across more markets. Do not tune against the hold-out, relabel failures or relax expectations to produce green country scores.

## Registry adjudications

Where a national register places the same address elsewhere, the case also accepts the register's point (`expect.any_of`, the OSM expectation first) and records it under `adjudication`. Number, postcode, house level, `goodMatch` and the 150 m radius stay; only the point differs. Such cases have two expected points, so ReverseGeocode no longer checks them.

So far only Australia, against the August 2026 G-NAF release (`G-NAF` in [the notice](../../tests/global/NOTICE)). `adjudicate-gnaf.py find` reads the manifest's records and the official PSV zip, never a service answer, and admits a record when exactly one current principal G-NAF address without unit or level has its complete number, street and postcode, with an address-level default geocode more than 150 m from the OSM node. Of 50 Australian groups, 43 match one G-NAF address; 27 of those lie within 150 m and 16 are adjudicated, at 0.19 to 13.7 km (mostly rural property and building centroids). Leading zeros, ranges and addresses absent from G-NAF are not adjudicated. See the [Australian triage](triage/global-au.md).

```sh
python3 tests/global/adjudicate-gnaf.py find --manifest tests/global/manifest.json \
--gnaf SOURCE/g-naf_aug26_allstates_gda2020_psv_110.zip --release "August 2026" \
--out tests/global/adjudications.json
```

The zip's SHA256 is recorded in `adjudications.json`. A new G-NAF release is a data update: rerun `find`, then the replay and `apply` steps below, and review the corpus diff.

## Run

The default `evaluate run --suite gold --index SOURCE/index --out eval-results/gold` includes this panel alongside the existing corpora. To run only the panel:
Expand All @@ -60,9 +74,14 @@ To reproduce the case file byte-for-byte from the checked-in source-record snaps
```sh
python3 tests/global/build-addresses.py \
--replay tests/global/manifest.json --out /tmp/global-address-replay
cmp tests/global/global-addresses.jsonl /tmp/global-address-replay/global-addresses.jsonl
python3 tests/global/adjudicate-gnaf.py apply \
--corpus /tmp/global-address-replay/global-addresses.jsonl \
--adjudications tests/global/adjudications.json --out /tmp/global-address-replay/adjudicated.jsonl
cmp tests/global/global-addresses.jsonl /tmp/global-address-replay/adjudicated.jsonl
```

The second step adds the [registry adjudications](#registry-adjudications); it is separate because the manifest binds the generator's hash to the capture.

To resample the original data, retain every PBF named by the manifest, its exact SHA256, the countryInfo.txt snapshot, and the pinned Natural Earth file. A `-latest` URL changes over time; downloading its current contents is a **new data experiment**, not a reproduction. `urls.txt` beside the PBFs records their public Geofabrik URLs, one per line.

```sh
Expand Down
8 changes: 7 additions & 1 deletion docs/evaluation/triage/global-au.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,4 +19,10 @@ Each was checked against the August 2026 G-NAF release. Held-out groups were adj
| Range in the source | 1 | 2 | 155 Horace Street, Sea Lake: Vicmap holds 151-157; G-NAF's 155 has confidence -1 (in no contributor dataset). Matching a number inside a range is not done. |
| Leading-zero source number | 1 | 2 | OSM "01692" East West Road: G-NAF's 1692 is 4.1 km from the OSM node, so even an exact match would miss the 150 m radius. |

The 16 point disagreements are source conflicts, not resolver defects: the panel's 150 m radius around an OSM node does not hold for rural Australian addresses whose authoritative point is the parcel. One held-out case, 1 Manuka Road, Logan Village, answers the point G-NAF gives its 200 units (192 m) rather than the building record without a unit (71 m); it was not tuned.
The 16 point disagreements are source conflicts, not resolver defects: the panel's 150 m radius around an OSM node does not hold for rural Australian addresses whose authoritative point is the parcel. For 1 Manuka Road, Logan Village, G-NAF's record without a unit shares its 200 units' point (192 m); the record 71 m away is the range 1-15, with confidence -1.

## Registry adjudication

These 16 groups now also accept G-NAF's point; see [registry adjudications](../global-addresses.md#registry-adjudications). The rule reads only the OSM records and G-NAF, and selects exactly the 16 groups above. Re-scoring the same local run (binaries of #223, same index) against the adjudicated corpus moves 31 queries from near_miss to pass; Sturt Highway 14382 written street first still answers the street. The panel passes **91/100 searches and 92/100 suggestions**. ReverseGeocode now checks the 68 queries with one expected point, all passing.

The remaining 9 failing queries are the 4 groups the rule leaves out (numbers missing from G-NAF, the Horace Street range, the leading zero) and that one street answer.
10 changes: 10 additions & 0 deletions tests/global/NOTICE
Original file line number Diff line number Diff line change
Expand Up @@ -26,3 +26,13 @@ English country labels come from GeoNames countryInfo.txt, CC BY 4.0:

This is mapped-source consistency evidence, not independent national accuracy,
mail deliverability, survey-grade coordinates or endorsement by these providers.

Alternative points for some Australian cases, and tests/global/adjudications.json,
come from G-NAF (August 2026 release) © Geoscape Australia, licensed by the
Commonwealth of Australia under the Open G-NAF End User Licence Agreement:
https://data.gov.au/data/dataset/geocoded-national-address-file-g-naf
The Open G-NAF EULA incorporates CC BY 4.0 with an additional mailing restriction:
G-NAF must not be used to generate addresses or address compilations for sending
mail unless each address's mail capability is verified using a secondary source.
These points are a reformatted adaptation, not an endorsed product, supplied as-is
without warranties. See tests/registry/au.NOTICE for the full terms.
140 changes: 140 additions & 0 deletions tests/global/adjudicate-gnaf.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,140 @@
#!/usr/bin/env python3
"""G-NAF points for the panel's Australian addresses whose OSM node is over 150 m from them.

`find` reads only the manifest's source records and the official G-NAF PSV release, never a
service answer: a record is adjudicated when exactly one current principal G-NAF address, without
unit or level, has its complete number, street and postcode, and an address-level default geocode.
`apply` adds each such point to the replayed cases as an alternative expectation. It is a separate
step because the manifest binds build-addresses.py's hash to the capture.
"""
import argparse
import hashlib
import io
import json
import math
from pathlib import Path
import zipfile

RADIUS_KM = 0.15
# street- and locality-level, gap and unknown geocodes do not place one address
NOT_ADDRESS_LEVEL = {"STL", "LOC", "GG", "UNK"}
STATES = ["ACT", "NSW", "NT", "OT", "QLD", "SA", "TAS", "VIC", "WA"]


def km(a, b):
lat = math.radians((a[0] + b[0]) / 2)
return math.hypot((a[0] - b[0]) * 111.195, (a[1] - b[1]) * 111.195 * math.cos(lat))


def norm(text):
return " ".join(text.upper().split())


def apply(lines, adjudications):
alternatives = {(a["node"], a["version"]): a for a in adjudications["au"]}
out, seen = [], set()
for line in lines:
case = json.loads(line)
key = (int(case["source"]["ref"]), case["source"]["version"])
if case["cohort"] == "global_osm_au" and key in alternatives:
a = alternatives[key]
seen.add(key)
case["expect"] = {"any_of": [case["expect"], {**case["expect"], "point": a["point"]}]}
case["adjudication"] = {**adjudications["source"], "ref": a["pid"], "geocode_type": a["geocode_type"], "distance_km": a["distance_km"], "role": adjudications["role"]}
out.append(json.dumps(case, ensure_ascii=False) + "\n")
if set(alternatives) - seen:
raise ValueError("adjudication names no case")
return "".join(out)


def main():
parser = argparse.ArgumentParser()
commands = parser.add_subparsers(dest="command", required=True)
find = commands.add_parser("find")
find.add_argument("--manifest", type=Path, required=True)
find.add_argument("--gnaf", type=Path, required=True, help="the G-NAF PSV release zip")
find.add_argument("--release", required=True, help='for example "August 2026"')
find.add_argument("--out", type=Path, required=True)
applied = commands.add_parser("apply")
applied.add_argument("--corpus", type=Path, required=True, help="build-addresses.py --replay output")
applied.add_argument("--adjudications", type=Path, required=True)
applied.add_argument("--out", type=Path, required=True)
args = parser.parse_args()
if args.command == "apply":
adjudications = json.loads(args.adjudications.read_bytes())
args.out.write_text(apply(args.corpus.read_text().splitlines(), adjudications))
return
manifest = json.loads(args.manifest.read_bytes())
records = next(s for s in manifest["snapshots"] if s["country"] == "AU")["records"]
h = hashlib.sha256()
with args.gnaf.open("rb") as stream:
for chunk in iter(lambda: stream.read(8 * 1024 * 1024), b""):
h.update(chunk)
archive = zipfile.ZipFile(args.gnaf)
names = archive.namelist()

def rows(suffix):
name = next(n for n in names if n.endswith("/" + suffix))
with archive.open(name) as raw:
lines = io.TextIOWrapper(raw, encoding="utf-8")
header = next(lines).rstrip("\n").split("|")
for line in lines:
yield dict(zip(header, line.rstrip("\n").split("|")))

suffixes = {r["CODE"]: r["NAME"] for r in rows("Authority_Code_STREET_SUFFIX_AUT_psv.psv")}
wanted = {norm(r["tags"]["addr:street"]) for r in records}
streets = {}
for state in STATES:
for r in rows(f"{state}_STREET_LOCALITY_psv.psv"):
full = norm(" ".join(x for x in [r["STREET_NAME"], r["STREET_TYPE_CODE"], suffixes.get(r["STREET_SUFFIX_CODE"], "")] if x))
if full in wanted and not r["DATE_RETIRED"]:
streets[r["STREET_LOCALITY_PID"]] = full
by_key = {}
for state in STATES:
for r in rows(f"{state}_ADDRESS_DETAIL_psv.psv"):
if r["STREET_LOCALITY_PID"] not in streets or r["DATE_RETIRED"] or r["ALIAS_PRINCIPAL"] != "P":
continue
if any(r[k] for k in ["FLAT_NUMBER", "FLAT_TYPE_CODE", "LEVEL_NUMBER", "LEVEL_TYPE_CODE", "NUMBER_FIRST_PREFIX", "NUMBER_LAST_PREFIX"]) or not r["NUMBER_FIRST"]:
continue
number = r["NUMBER_FIRST"] + r["NUMBER_FIRST_SUFFIX"]
if r["NUMBER_LAST"]:
number += "-" + r["NUMBER_LAST"] + r["NUMBER_LAST_SUFFIX"]
key = (number.upper(), streets[r["STREET_LOCALITY_PID"]], r["POSTCODE"])
by_key.setdefault(key, []).append(r)
pids = {r["ADDRESS_DETAIL_PID"] for found in by_key.values() for r in found}
geocodes = {}
for state in STATES:
for r in rows(f"{state}_ADDRESS_DEFAULT_GEOCODE_psv.psv"):
if r["ADDRESS_DETAIL_PID"] in pids and not r["DATE_RETIRED"]:
geocodes[r["ADDRESS_DETAIL_PID"]] = r
adjudicated, counts = [], {"records": len(records), "unique_match": 0, "within_radius": 0, "adjudicated": 0}
for record in records:
tags = record["tags"]
found = by_key.get((tags["addr:housenumber"].upper(), norm(tags["addr:street"]), tags.get("addr:postcode", "")), [])
if len(found) != 1:
continue
detail = found[0]
geocode = geocodes.get(detail["ADDRESS_DETAIL_PID"])
if geocode is None or geocode["GEOCODE_TYPE_CODE"] in NOT_ADDRESS_LEVEL:
continue
counts["unique_match"] += 1
point = [round(float(geocode["LATITUDE"]), 6), round(float(geocode["LONGITUDE"]), 6)]
distance = km(record["point"], point)
if distance <= RADIUS_KM:
counts["within_radius"] += 1
continue
adjudicated.append({"node": record["id"], "version": record["version"], "pid": detail["ADDRESS_DETAIL_PID"], "geocode_type": geocode["GEOCODE_TYPE_CODE"], "point": point, "distance_km": round(distance, 3)})
counts["adjudicated"] = len(adjudicated)
out = {
"role": "authoritative-alternative-point",
"rule": "Exactly one current principal G-NAF address without unit or level has the source record's complete number, street and postcode, with an address-level default geocode over 0.15 km from the OSM node; its point is accepted as an alternative.",
"source": {"name": "G-NAF", "release": args.release, "license": "Open G-NAF EULA (CC BY 4.0 with a mailing restriction)", "url": "https://data.gov.au/data/dataset/geocoded-national-address-file-g-naf", "zip": args.gnaf.name, "zip_sha256": h.hexdigest(), "datum": "GDA2020"},
"counts": counts,
"au": adjudicated,
}
args.out.write_text(json.dumps(out, ensure_ascii=False, indent=2) + "\n")
print(json.dumps(counts))


if __name__ == "__main__":
main()
Loading