Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
103 changes: 103 additions & 0 deletions ARGO/oceancurrent/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -219,3 +219,106 @@ Merging activates these configurations on the next scheduled run because the
production job downloads the scanner from `master`. Gliders runs after existing
products and writes up to six indexes under `gliders/`; check the existing runtime
and memory logs after deployment. No cron or deployment changes are required.

## Gliders mission catalogue

`gliders_catalogue.py` independently converts the four mission tables in the
legacy catalogue into `gliders/gliders-catalogue.json`, relative to the website
root. This contains mission metadata, **not** the generic image inventory schema;
do not add it to `FILE_PATH_CONFIG` or the generic API image-index configuration.
API integration is a separate task.

The source is `/mnt/oceancurrent/website/gliders/index.html`, published at
<https://oceancurrent.aodn.org.au/gliders/index.html>. A search of this repository
and its documented OceanCurrent pipeline found no maintained structured source
for this catalogue. Existing Gliders GIF/MP4 indexes describe assets, not mission
metadata. The parser uses the existing `beautifulsoup4` dependency with Python's
HTML parser; it does not fetch remote resources or inspect scientific files.

### JSON schema (version 1)

The top-level object contains `schema_version`, `source_url`, `missions` and
`orders`. `missions` is an array sorted by `id`, with one record per exact
published mission name and processing-version string. Optional scalar fields are
`null`; unavailable visualisations are empty arrays.

| Field | Meaning |
| --- | --- |
| `id` | URL-quoted exact `mission_name`, then `@`, then URL-quoted `processing_version` (or `unknown` when absent). Stable for that published name/version; independent of catalogue row numbering. |
| `mission_name` | Exact catalogue name, including historical case, leg and `_nrt` suffixes. |
| `mission_id` | Name with only the terminal `_nrt` removed, to associate NRT and non-NRT publications. Other suffixes remain significant. This is a catalogue identity, not a scientific deployment identifier. |
| `catalogue_numbers` | All original `N` values for this record, as numerically sorted strings. Identical rows sometimes have different numbers. |
| `processing_version` | Exact processing text with surrounding whitespace removed, e.g. `5.00`, `3.0.` or `5.`. Never converted to a number. |
| `is_nrt` | Boolean based on the catalogue's terminal `_nrt` naming convention. False means a non-NRT catalogue publication; it does not independently certify scientific quality or delayed-mode status. |
| `dates_raw` | Original deployment date text with whitespace normalised. Always retained, even when parsing fails. |
| `start_date`, `end_date` | ISO dates for recognised `DD Mon - DD Mon YYYY` ranges. The trailing year is the end year; a wrapped month uses the previous start year. Unrecognised or invalid ranges leave both null. No dates are inferred from mission names. |
| `latitude`, `longitude` | Signed decimal degrees: the catalogue's degrees south become negative latitude; degrees east retain their sign. Header units are checked before conversion. |
| `duration_days`, `distance_travelled_km` | Nonnegative numeric values from `days` and `km`. The latter is travelled distance, **not** distance alongshore to Darwin. Nonfinite and out-of-range numbers become null. |
| `glider_id` | Published glider identifier. |
| `visualisations` | `details` and `animations` arrays of `{ "label": "4", "url": "https://..." }`. Labels and linked targets are preserved, including available 4/12/30-day pages and MP4s; no missing periods are invented. |
| `temperature_salinity_url` | Published T-S reference, which can be an HTML page rather than an image. |
| `scientific_data_url` | Published mission scientific data link only; null if absent. |

`orders` maps `deployment_date`, `mission_name`,
`distance_alongshore_to_darwin` and `event_based_sampling` to ordered arrays of
record IDs. These retain the first occurrence in each respective source table,
including its membership omissions. Event membership and alongshore ordering
must use these lists; no numeric distance-to-Darwin is inferred. Year headings
and totals are excluded. Repeated occurrences do not duplicate a record or its
position within a view. These orders may have different lengths.

### Local execution and tests

Use an environment with the repository's `beautifulsoup4` dependency installed.
Both paths are required to prevent an accidental production run; the output's
parent directory must already exist. For example, using a local source copy:

```bash
python3 ARGO/oceancurrent/gliders_catalogue.py \
--input /tmp/gliders-index.html \
--output /tmp/gliders-catalogue.json
python3 -m pytest -q ARGO/oceancurrent/test_gliders_catalogue.py
```

The standard library runner is also supported:

```bash
cd ARGO/oceancurrent
python3 -m unittest test_gliders_catalogue
```

`--source-url` optionally changes the base used to resolve relative links; local
input paths never become public URLs. Output is deterministic for identical
input and base URL, with no generated timestamp. The generator parses and
validates before writing, uses a temporary file alongside the output, then
atomically replaces the destination. Parsing, validation or replacement failures
leave the previous catalogue intact, remove the temporary file and exit nonzero.

### Refresh and limitations

No cron, deployment, scanner or scientific processing changes are included.
A future separately reviewed scheduling change should run this generator after
the legacy catalogue has finished publishing, with explicit input and output
paths, in the existing OceanCurrent job environment. It should log the record
count and failures and alert on unexpected count reductions. Avoid reading a
partially written HTML file: use the publisher's completion signal or a completed
snapshot. Start with a manual comparison before enabling recurring refreshes;
refresh at the legacy catalogue's publication cadence rather than running a new
service.

The four table anchors, column headers and coordinate units are deliberately
validated. Missing tables, empty main views and conflicting metadata for the
same name/version fail the run. The event table may be empty. Year/summary rows
and incomplete or unrecognised mission rows are skipped. This is a structural
sanity check, not proof that a partially published catalogue is complete: there
is no fixed mission-count threshold. Historical letter suffixes and `_1`/`_2`
legs are supported; new naming/header conventions need parser review. Scientific
and visualisation links are preserved but their availability is not checked.
The source's precision and errors remain authoritative; no scientific metadata,
annual-map hotspots or missing links are inferred.

Future `ocean-current-api` work should define a dedicated catalogue response and
loader for this schema, retain the four order lists and mission/version identity,
and associate visualisation references with the existing Gliders asset indexes.
The catalogue must not be interpreted as an image index. Endpoints, frontend
browsing and caching/refresh behaviour remain separate integration work.
Loading
Loading