refactor(gliders): simplify catalogue publication

ARGO/oceancurrent/README.md

@@ -269,7 +269,7 @@ position within a view. These orders may have different lengths.
 
 ### Local execution and tests
 
-The generator requires only the Python 3 standard library.
+The generator, publisher and their tests require only the Python 3 standard library.
 Both paths are required to prevent an accidental production run; the output's
 parent directory must already exist. For example, using a local source copy:
 
@@ -281,14 +281,16 @@ python3 -m pytest -q ARGO/oceancurrent/test_gliders_catalogue.py
 python3 -m pytest -q ARGO/oceancurrent/test_publish_gliders_catalogue.py
 ```
 
+`-S` excludes site-packages, so both suites also run without BeautifulSoup.
+
 The JSON fixture records the previous BeautifulSoup implementation's output; tests
 compare both the complete parsed catalogue and generated bytes against it.
 
 The standard library runner is also supported:
 
 ```bash
 cd ARGO/oceancurrent
-python3 -m unittest test_gliders_catalogue
+python3 -S -m unittest test_gliders_catalogue test_publish_gliders_catalogue
 ```
 
 `--source-url` optionally changes the base used to resolve relative links; local
@@ -306,30 +308,28 @@ historically connecting from `140.79.40.143`. Publication intervals vary (roughl
 two-hourly or daily); no fixed completion schedule or signal is confirmed. The image
 scanner runs every two hours independently; legacy `index.php` reads the HTML.
 
-**Proposed initial schedule: check every two hours, independently of the scanner.**
+**Agreed schedule: twice daily at 02:00 and 14:00, independently of the scanner.**
 Manage a separate `po_gliders_catalogue` job through
 `chef-private/data_bags/cronjobs`, registered in the webapp node's `cronjobs` list.
-Use `minute: "0"`, `hour: "*/2"`, `shell: "/bin/bash"`, the verified execution user
+Use `minute: "0"`, `hour: "2,14"`, `shell: "/bin/bash"`, the verified execution user
 (proposed `projectofficer`), existing operations `mailto` and `monitored: true`.
 Its command should source the verified job environment and run the command below.
-This is a polling interval, not an assumed upstream publication time. No production
-cron configuration or automatic publication is installed by this change.
+Configure a 15-minute timeout separately in `chef-private`. These check times do
+not imply upstream completion. No production cron configuration is changed here.
 
 ### Publication
 
 Install both Python modules together. Set `PYTHON` to the verified Python 3 interpreter
-with `beautifulsoup4`, and `DATA_SERVICES_DIR` to the installed checkout. The dependency
-is declared in `requirements.txt`, but production availability is unverified.
+and `DATA_SERVICES_DIR` to the installed checkout. No additional Python packages
+are required for Gliders; other repository components still use `beautifulsoup4`.
 `projectofficer` is the proposed user; confirm source readability and permissions to
-write temporary files and replace JSON in the destination directory. Set
-`GLIDERS_LOCK_FILE` to an operations-provisioned writable lock outside the rsync tree;
-use the same lock for scheduled and manual runs.
+write temporary files and replace JSON in the destination directory. No file lock
+is required or created.
 
 ```bash
 "${PYTHON:?}" "${DATA_SERVICES_DIR:?}/ARGO/oceancurrent/publish_gliders_catalogue.py" \
   --input /mnt/oceancurrent/website/gliders/index.html \
   --output /mnt/oceancurrent/website/gliders/gliders-catalogue.json \
-  --lock-file "${GLIDERS_LOCK_FILE:?}" \
   --stability-seconds 30
 ```
 
@@ -344,22 +344,45 @@ incomplete transfer can pause longer, or restart after the final check. Coordina
 with the upstream publisher for stronger serialization/completion guarantees.
 `--source-complete` skips the wait only for an explicitly confirmed manual run.
 
-The publisher validates schema version 1, mission records and all four order lists.
-Every mission-count or per-order count reduction is blocked by default;
-`--max-reduction-percent P` permits an explicitly reviewed percentage (`0 <= P < 100`).
-Unchanged JSON is not rewritten. The existing generator's atomic writer preserves
-the previous output on failure. The lock serializes publishers, not upstream writes;
-never unlink it while in use. Logs on stderr include start/completion, paths, duration,
-counts, changes and failures. Preserve stderr/status in existing mail/Nagios reporting:
-success returns 0, failure 1, and invalid CLI syntax 2.
+The publisher validates schema version 1, mission identifiers, metadata and all
+four order lists, including unique IDs, valid ordering references and nonempty
+main views. Missing tables, unexpected columns/units and conflicting metadata fail
+parsing; the event view may be empty. As in the generator, incomplete or unrecognised
+individual rows are skipped. Structural checks cannot prove that all upstream
+missions have arrived, even when the HTML is stable.
+
+If no JSON exists, the first catalogue is published automatically after the same
+validation, without manual bootstrapping. A valid existing catalogue is validated
+before comparison. Valid increases and decreases in mission counts or individual
+ordering lists are accepted, including upstream corrections or reorganisation.
+Logs report previous/new mission counts and previous/new sizes and change status
+for each ordering list, including reordering without a count change.
+
+Unchanged JSON content is not rewritten, even if the existing formatting differs.
+Changed content is written to a temporary file in the destination directory,
+flushed and atomically replaced using the existing writer. Parsing, validation or
+replacement failures preserve the previous output and clean up temporary files.
+Corrupt existing JSON fails the run and must be recovered; it is never treated as
+an ordinary first publication.
+
+There is no file locking or prevention of overlapping executions. The team accepts
+the small overlap risk with twice-daily checks and a normal runtime of about 30
+seconds. Atomic replacement prevents partial JSON reads; it does not serialize
+publishers or prevent all concurrent-update races. An overlapping run can overwrite
+another run's newer result.
+
+Logs on stderr include start/completion, paths, duration, counts, changes and
+failures. Preserve stderr/status in existing mail/Nagios reporting: success
+returns 0, failure 1, and invalid CLI syntax 2.
 
 ### Rsync deletion protection (required before deployment)
 
 The writable module's local configuration has no explicit deletion/filter protection;
 the upstream command and effective daemon settings remain unverified. Rsync deletion
 could remove locally generated JSON absent from the sender. A local dry run confirmed
-this for the JSON, default lock and atomic temporary files. Missing JSON fails closed
-on the next check without a reviewed initial baseline; polling is not a mitigation.
+this for the JSON and atomic temporary files. Missing JSON is regenerated after
+validation on the next successful check, but that does not prevent deletion or
+protect an in-progress publication; polling is not deletion protection.
 
 Coordinate receiver-side deletion protection **and** sender-side exclusion to prevent
 overwrite. For a transfer rooted at the website directory, proposed rules are:
@@ -380,24 +403,17 @@ no upstream rsync configuration is changed here.
 
 ### First publication and recovery
 
-1. Coordinate with operations and the upstream owner; confirm HTML generation and
-   transfer completed and prevent another update during review/publication.
-2. Verify `"$PYTHON" -c 'import bs4'` and directory permissions as the execution user.
-   Copy the completed HTML to a temporary directory and run the local generator above.
-3. Inspect the candidate with `"$PYTHON" -m json.tool`. Compare mission/version counts,
-   metadata and all four orders with the legacy HTML; back up any existing JSON.
-4. Confirm the live HTML still matches the reviewed copy. Run the publication command,
-   adding `--initial-mission-count N` with the reviewed count when no output exists.
-   Compare the published JSON with the candidate, then verify remote access below.
-5. Confirm rsync protection and failure reporting, then approve the separate Chef job.
-   Scheduled runs omit both `--source-complete` and `--initial-mission-count`.
+The scheduled command above handles the first publication automatically. Before
+enabling the separately managed Chef job, verify the standard library Python 3
+interpreter, source readability, destination directory permissions, rsync deletion
+protection and failure reporting. Optional manual inspection can use a completed
+HTML copy and the local generator; no approved mission count is required.
 
 On failure, inspect stderr and fix the source/environment/permissions. A changing
 source is retried on the next scheduled check; manual retries use the same command.
-Review reductions before allowing a one-time tolerance; do not delete the baseline
-to bypass the guard. If existing JSON is invalid, preserve a copy and restore the
-last known valid catalogue before retrying. Count guards cannot detect every
-incomplete source, so upstream coordination remains necessary.
+If existing JSON is corrupt, preserve a copy and restore the last known valid
+catalogue before retrying. Structural validation and a quiet observation window
+cannot detect every incomplete source, so upstream coordination remains necessary.
 
 ### Remote and API verification
 

ARGO/oceancurrent/publish_gliders_catalogue.py

@@ -1,6 +1,5 @@
 """Check stable Gliders HTML and atomically publish validated mission metadata."""
 import argparse
-import fcntl
 import json
 import logging
 import math
@@ -13,7 +12,7 @@
 
 
 def validate_catalogue(catalogue):
-    """Validate generated JSON and the previous baseline before comparing counts."""
+    """Validate generated JSON and any previously published catalogue."""
     from gliders_catalogue import MISSION, VIEWS
 
     def require(condition, message):
@@ -79,66 +78,45 @@ def _read_source(path):
     return content, signature(after)
 
 
-def publish_catalogue(source, destination, *, source_complete=False,
-                      stability_seconds=30, lock_file=None,
-                      max_reduction_percent=0, initial_mission_count=None):
-    """Observe source stability, or accept explicitly confirmed manual completion.
+def publish_catalogue(source, destination, *, source_complete=False, stability_seconds=30):
+    """Validate a stable source and publish changed content, including the first run.
 
-    Quiet snapshots cannot prove upstream completion. The lock serializes catalogue
-    publishers only; stronger guarantees require cooperation from the HTML producer.
+    Quiet snapshots cannot prove upstream completion. Executions may overlap;
+    atomic replacement prevents partial JSON reads but does not serialize updates.
     """
-    # Lazy import lets the CLI report missing BeautifulSoup through normal logging.
     from gliders_catalogue import parse_catalogue, write_catalogue
 
     source, destination = Path(source), Path(destination)
     if not math.isfinite(stability_seconds) or stability_seconds <= 0:
         raise ValueError("stability-seconds must be finite and positive")
-    if not math.isfinite(max_reduction_percent) or not 0 <= max_reduction_percent < 100:
-        raise ValueError("max-reduction-percent must be finite and in [0, 100)")
-    if initial_mission_count is not None and initial_mission_count <= 0:
-        raise ValueError("initial-mission-count must be positive")
-    lock_path = Path(lock_file) if lock_file is not None else Path(str(destination) + ".lock")
-    if len({source.resolve(), destination.resolve(), lock_path.resolve()}) != 3:
-        raise ValueError("Source must differ from destination and lock; destination must differ from lock")
-    # Never unlink the lock: removing it would allow two processes to lock different inodes.
-    with lock_path.open("a") as lock:
-        try:
-            fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB)
-        except BlockingIOError as error:
-            raise ValueError("Another catalogue publication holds the output lock") from error
-        html, signature = _read_source(source)
-        if not source_complete:
-            LOG.info("Observing HTML stability for %s seconds (not an upstream completion signal)", stability_seconds)
-            time.sleep(stability_seconds)
-            observed_html, observed_signature = _read_source(source)
-            if observed_html != html or observed_signature != signature:
-                raise ValueError("Source changed during stability observation; retry on the next check")
-        catalogue = parse_catalogue(html.decode("utf-8"))
-        new_count = validate_catalogue(catalogue)
-        previous_bytes = destination.read_bytes() if destination.exists() else None
-        previous = json.loads(previous_bytes) if previous_bytes is not None else None
-        previous_count = validate_catalogue(previous) if previous_bytes is not None else None
-        LOG.info("Mission counts previous=%s new=%d", previous_count, new_count)
-        if previous is None:
-            if initial_mission_count != new_count:
-                raise ValueError("First publication requires --initial-mission-count matching the reviewed candidate (%d)" % new_count)
-        else:
-            counts = [("missions", previous_count, new_count)]
-            counts.extend((view, len(old), len(catalogue["orders"][view]))
-                          for view, old in previous["orders"].items() if view in catalogue["orders"])
-            for label, old, new in counts:
-                if old and (old - new) * 100 > old * max_reduction_percent:
-                    raise ValueError("Reduction requires manual review: %s previous=%d new=%d threshold=%s%%"
-                                     % (label, old, new, max_reduction_percent))
-        current_html, current_signature = _read_source(source)
-        if current_html != html or current_signature != signature:
-            raise ValueError("Source changed during generation; publication aborted")
-        # Semantic equality avoids rewriting even a differently formatted valid baseline.
-        changed = previous != catalogue
-        if changed:
-            write_catalogue(catalogue, destination)
-        LOG.info("Catalogue publication changed=%s missions=%d destination=%s", changed, new_count, destination)
-        return catalogue
+    if source.resolve() == destination.resolve():
+        raise ValueError("Source must differ from destination")
+    html, signature = _read_source(source)
+    if not source_complete:
+        LOG.info("Observing HTML stability for %s seconds (not an upstream completion signal)", stability_seconds)
+        time.sleep(stability_seconds)
+        observed_html, observed_signature = _read_source(source)
+        if observed_html != html or observed_signature != signature:
+            raise ValueError("Source changed during stability observation; retry on the next check")
+    catalogue = parse_catalogue(html.decode("utf-8"))
+    new_count = validate_catalogue(catalogue)
+    previous_bytes = destination.read_bytes() if destination.exists() else None
+    previous = json.loads(previous_bytes) if previous_bytes is not None else None
+    previous_count = validate_catalogue(previous) if previous_bytes is not None else None
+    LOG.info("Mission counts previous=%s new=%d", previous_count, new_count)
+    for view, order in catalogue["orders"].items():
+        old = previous["orders"][view] if previous is not None else None
+        LOG.info("Catalogue order %s previous=%s new=%d changed=%s", view,
+                 len(old) if old is not None else None, len(order), old != order)
+    current_html, current_signature = _read_source(source)
+    if current_html != html or current_signature != signature:
+        raise ValueError("Source changed during generation; publication aborted")
+    # Semantic equality avoids rewriting even a differently formatted valid baseline.
+    changed = previous != catalogue
+    if changed:
+        write_catalogue(catalogue, destination)
+    LOG.info("Catalogue publication changed=%s missions=%d destination=%s", changed, new_count, destination)
+    return catalogue
 
 
 def main(argv=None):
@@ -149,22 +127,14 @@ def main(argv=None):
                         help="Manual confirmation of upstream completion; skips the stability wait")
     parser.add_argument("--stability-seconds", type=float, default=30,
                         help="Quiet observation window for periodic checks; default 30, not proof of completion")
-    parser.add_argument("--lock-file", type=Path,
-                        help="Shared lock outside incoming rsync tree (default: <output>.lock)")
-    parser.add_argument("--max-reduction-percent", type=float, default=0,
-                        help="Maximum reviewed reduction in missions or any order; default 0 blocks every decrease")
-    parser.add_argument("--initial-mission-count", type=int,
-                        help="Reviewed exact mission count, required only when output does not exist")
     args = parser.parse_args(argv)
     logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(name)s %(message)s")
     started = time.monotonic()
     LOG.info("Publication started source=%s destination=%s", args.input, args.output)
     status = 1
     try:
         publish_catalogue(args.input, args.output, source_complete=args.source_complete,
-                          stability_seconds=args.stability_seconds, lock_file=args.lock_file,
-                          max_reduction_percent=args.max_reduction_percent,
-                          initial_mission_count=args.initial_mission_count)
+                          stability_seconds=args.stability_seconds)
         status = 0
     except (OSError, ValueError, ImportError) as error:
         LOG.error("Validation/publication failed; previous catalogue preserved: %s", error)