diff --git a/ARGO/oceancurrent/README.md b/ARGO/oceancurrent/README.md index f2556385..1bb8915e 100644 --- a/ARGO/oceancurrent/README.md +++ b/ARGO/oceancurrent/README.md @@ -269,7 +269,7 @@ position within a view. These orders may have different lengths. ### Local execution and tests -The generator requires only the Python 3 standard library. +The generator, publisher and their tests require only the Python 3 standard library. Both paths are required to prevent an accidental production run; the output's parent directory must already exist. For example, using a local source copy: @@ -281,6 +281,8 @@ python3 -m pytest -q ARGO/oceancurrent/test_gliders_catalogue.py python3 -m pytest -q ARGO/oceancurrent/test_publish_gliders_catalogue.py ``` +`-S` excludes site-packages, so both suites also run without BeautifulSoup. + The JSON fixture records the previous BeautifulSoup implementation's output; tests compare both the complete parsed catalogue and generated bytes against it. @@ -288,7 +290,7 @@ The standard library runner is also supported: ```bash cd ARGO/oceancurrent -python3 -m unittest test_gliders_catalogue +python3 -S -m unittest test_gliders_catalogue test_publish_gliders_catalogue ``` `--source-url` optionally changes the base used to resolve relative links; local @@ -306,30 +308,28 @@ historically connecting from `140.79.40.143`. Publication intervals vary (roughl two-hourly or daily); no fixed completion schedule or signal is confirmed. The image scanner runs every two hours independently; legacy `index.php` reads the HTML. -**Proposed initial schedule: check every two hours, independently of the scanner.** +**Agreed schedule: twice daily at 02:00 and 14:00, independently of the scanner.** Manage a separate `po_gliders_catalogue` job through `chef-private/data_bags/cronjobs`, registered in the webapp node's `cronjobs` list. -Use `minute: "0"`, `hour: "*/2"`, `shell: "/bin/bash"`, the verified execution user +Use `minute: "0"`, `hour: "2,14"`, `shell: "/bin/bash"`, the verified execution user (proposed `projectofficer`), existing operations `mailto` and `monitored: true`. Its command should source the verified job environment and run the command below. -This is a polling interval, not an assumed upstream publication time. No production -cron configuration or automatic publication is installed by this change. +Configure a 15-minute timeout separately in `chef-private`. These check times do +not imply upstream completion. No production cron configuration is changed here. ### Publication Install both Python modules together. Set `PYTHON` to the verified Python 3 interpreter -with `beautifulsoup4`, and `DATA_SERVICES_DIR` to the installed checkout. The dependency -is declared in `requirements.txt`, but production availability is unverified. +and `DATA_SERVICES_DIR` to the installed checkout. No additional Python packages +are required for Gliders; other repository components still use `beautifulsoup4`. `projectofficer` is the proposed user; confirm source readability and permissions to -write temporary files and replace JSON in the destination directory. Set -`GLIDERS_LOCK_FILE` to an operations-provisioned writable lock outside the rsync tree; -use the same lock for scheduled and manual runs. +write temporary files and replace JSON in the destination directory. No file lock +is required or created. ```bash "${PYTHON:?}" "${DATA_SERVICES_DIR:?}/ARGO/oceancurrent/publish_gliders_catalogue.py" \ --input /mnt/oceancurrent/website/gliders/index.html \ --output /mnt/oceancurrent/website/gliders/gliders-catalogue.json \ - --lock-file "${GLIDERS_LOCK_FILE:?}" \ --stability-seconds 30 ``` @@ -344,22 +344,45 @@ incomplete transfer can pause longer, or restart after the final check. Coordina with the upstream publisher for stronger serialization/completion guarantees. `--source-complete` skips the wait only for an explicitly confirmed manual run. -The publisher validates schema version 1, mission records and all four order lists. -Every mission-count or per-order count reduction is blocked by default; -`--max-reduction-percent P` permits an explicitly reviewed percentage (`0 <= P < 100`). -Unchanged JSON is not rewritten. The existing generator's atomic writer preserves -the previous output on failure. The lock serializes publishers, not upstream writes; -never unlink it while in use. Logs on stderr include start/completion, paths, duration, -counts, changes and failures. Preserve stderr/status in existing mail/Nagios reporting: -success returns 0, failure 1, and invalid CLI syntax 2. +The publisher validates schema version 1, mission identifiers, metadata and all +four order lists, including unique IDs, valid ordering references and nonempty +main views. Missing tables, unexpected columns/units and conflicting metadata fail +parsing; the event view may be empty. As in the generator, incomplete or unrecognised +individual rows are skipped. Structural checks cannot prove that all upstream +missions have arrived, even when the HTML is stable. + +If no JSON exists, the first catalogue is published automatically after the same +validation, without manual bootstrapping. A valid existing catalogue is validated +before comparison. Valid increases and decreases in mission counts or individual +ordering lists are accepted, including upstream corrections or reorganisation. +Logs report previous/new mission counts and previous/new sizes and change status +for each ordering list, including reordering without a count change. + +Unchanged JSON content is not rewritten, even if the existing formatting differs. +Changed content is written to a temporary file in the destination directory, +flushed and atomically replaced using the existing writer. Parsing, validation or +replacement failures preserve the previous output and clean up temporary files. +Corrupt existing JSON fails the run and must be recovered; it is never treated as +an ordinary first publication. + +There is no file locking or prevention of overlapping executions. The team accepts +the small overlap risk with twice-daily checks and a normal runtime of about 30 +seconds. Atomic replacement prevents partial JSON reads; it does not serialize +publishers or prevent all concurrent-update races. An overlapping run can overwrite +another run's newer result. + +Logs on stderr include start/completion, paths, duration, counts, changes and +failures. Preserve stderr/status in existing mail/Nagios reporting: success +returns 0, failure 1, and invalid CLI syntax 2. ### Rsync deletion protection (required before deployment) The writable module's local configuration has no explicit deletion/filter protection; the upstream command and effective daemon settings remain unverified. Rsync deletion could remove locally generated JSON absent from the sender. A local dry run confirmed -this for the JSON, default lock and atomic temporary files. Missing JSON fails closed -on the next check without a reviewed initial baseline; polling is not a mitigation. +this for the JSON and atomic temporary files. Missing JSON is regenerated after +validation on the next successful check, but that does not prevent deletion or +protect an in-progress publication; polling is not deletion protection. Coordinate receiver-side deletion protection **and** sender-side exclusion to prevent overwrite. For a transfer rooted at the website directory, proposed rules are: @@ -380,24 +403,17 @@ no upstream rsync configuration is changed here. ### First publication and recovery -1. Coordinate with operations and the upstream owner; confirm HTML generation and - transfer completed and prevent another update during review/publication. -2. Verify `"$PYTHON" -c 'import bs4'` and directory permissions as the execution user. - Copy the completed HTML to a temporary directory and run the local generator above. -3. Inspect the candidate with `"$PYTHON" -m json.tool`. Compare mission/version counts, - metadata and all four orders with the legacy HTML; back up any existing JSON. -4. Confirm the live HTML still matches the reviewed copy. Run the publication command, - adding `--initial-mission-count N` with the reviewed count when no output exists. - Compare the published JSON with the candidate, then verify remote access below. -5. Confirm rsync protection and failure reporting, then approve the separate Chef job. - Scheduled runs omit both `--source-complete` and `--initial-mission-count`. +The scheduled command above handles the first publication automatically. Before +enabling the separately managed Chef job, verify the standard library Python 3 +interpreter, source readability, destination directory permissions, rsync deletion +protection and failure reporting. Optional manual inspection can use a completed +HTML copy and the local generator; no approved mission count is required. On failure, inspect stderr and fix the source/environment/permissions. A changing source is retried on the next scheduled check; manual retries use the same command. -Review reductions before allowing a one-time tolerance; do not delete the baseline -to bypass the guard. If existing JSON is invalid, preserve a copy and restore the -last known valid catalogue before retrying. Count guards cannot detect every -incomplete source, so upstream coordination remains necessary. +If existing JSON is corrupt, preserve a copy and restore the last known valid +catalogue before retrying. Structural validation and a quiet observation window +cannot detect every incomplete source, so upstream coordination remains necessary. ### Remote and API verification diff --git a/ARGO/oceancurrent/publish_gliders_catalogue.py b/ARGO/oceancurrent/publish_gliders_catalogue.py index f2b68ed2..9f97d197 100644 --- a/ARGO/oceancurrent/publish_gliders_catalogue.py +++ b/ARGO/oceancurrent/publish_gliders_catalogue.py @@ -1,6 +1,5 @@ """Check stable Gliders HTML and atomically publish validated mission metadata.""" import argparse -import fcntl import json import logging import math @@ -13,7 +12,7 @@ def validate_catalogue(catalogue): - """Validate generated JSON and the previous baseline before comparing counts.""" + """Validate generated JSON and any previously published catalogue.""" from gliders_catalogue import MISSION, VIEWS def require(condition, message): @@ -79,66 +78,45 @@ def _read_source(path): return content, signature(after) -def publish_catalogue(source, destination, *, source_complete=False, - stability_seconds=30, lock_file=None, - max_reduction_percent=0, initial_mission_count=None): - """Observe source stability, or accept explicitly confirmed manual completion. +def publish_catalogue(source, destination, *, source_complete=False, stability_seconds=30): + """Validate a stable source and publish changed content, including the first run. - Quiet snapshots cannot prove upstream completion. The lock serializes catalogue - publishers only; stronger guarantees require cooperation from the HTML producer. + Quiet snapshots cannot prove upstream completion. Executions may overlap; + atomic replacement prevents partial JSON reads but does not serialize updates. """ - # Lazy import lets the CLI report missing BeautifulSoup through normal logging. from gliders_catalogue import parse_catalogue, write_catalogue source, destination = Path(source), Path(destination) if not math.isfinite(stability_seconds) or stability_seconds <= 0: raise ValueError("stability-seconds must be finite and positive") - if not math.isfinite(max_reduction_percent) or not 0 <= max_reduction_percent < 100: - raise ValueError("max-reduction-percent must be finite and in [0, 100)") - if initial_mission_count is not None and initial_mission_count <= 0: - raise ValueError("initial-mission-count must be positive") - lock_path = Path(lock_file) if lock_file is not None else Path(str(destination) + ".lock") - if len({source.resolve(), destination.resolve(), lock_path.resolve()}) != 3: - raise ValueError("Source must differ from destination and lock; destination must differ from lock") - # Never unlink the lock: removing it would allow two processes to lock different inodes. - with lock_path.open("a") as lock: - try: - fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB) - except BlockingIOError as error: - raise ValueError("Another catalogue publication holds the output lock") from error - html, signature = _read_source(source) - if not source_complete: - LOG.info("Observing HTML stability for %s seconds (not an upstream completion signal)", stability_seconds) - time.sleep(stability_seconds) - observed_html, observed_signature = _read_source(source) - if observed_html != html or observed_signature != signature: - raise ValueError("Source changed during stability observation; retry on the next check") - catalogue = parse_catalogue(html.decode("utf-8")) - new_count = validate_catalogue(catalogue) - previous_bytes = destination.read_bytes() if destination.exists() else None - previous = json.loads(previous_bytes) if previous_bytes is not None else None - previous_count = validate_catalogue(previous) if previous_bytes is not None else None - LOG.info("Mission counts previous=%s new=%d", previous_count, new_count) - if previous is None: - if initial_mission_count != new_count: - raise ValueError("First publication requires --initial-mission-count matching the reviewed candidate (%d)" % new_count) - else: - counts = [("missions", previous_count, new_count)] - counts.extend((view, len(old), len(catalogue["orders"][view])) - for view, old in previous["orders"].items() if view in catalogue["orders"]) - for label, old, new in counts: - if old and (old - new) * 100 > old * max_reduction_percent: - raise ValueError("Reduction requires manual review: %s previous=%d new=%d threshold=%s%%" - % (label, old, new, max_reduction_percent)) - current_html, current_signature = _read_source(source) - if current_html != html or current_signature != signature: - raise ValueError("Source changed during generation; publication aborted") - # Semantic equality avoids rewriting even a differently formatted valid baseline. - changed = previous != catalogue - if changed: - write_catalogue(catalogue, destination) - LOG.info("Catalogue publication changed=%s missions=%d destination=%s", changed, new_count, destination) - return catalogue + if source.resolve() == destination.resolve(): + raise ValueError("Source must differ from destination") + html, signature = _read_source(source) + if not source_complete: + LOG.info("Observing HTML stability for %s seconds (not an upstream completion signal)", stability_seconds) + time.sleep(stability_seconds) + observed_html, observed_signature = _read_source(source) + if observed_html != html or observed_signature != signature: + raise ValueError("Source changed during stability observation; retry on the next check") + catalogue = parse_catalogue(html.decode("utf-8")) + new_count = validate_catalogue(catalogue) + previous_bytes = destination.read_bytes() if destination.exists() else None + previous = json.loads(previous_bytes) if previous_bytes is not None else None + previous_count = validate_catalogue(previous) if previous_bytes is not None else None + LOG.info("Mission counts previous=%s new=%d", previous_count, new_count) + for view, order in catalogue["orders"].items(): + old = previous["orders"][view] if previous is not None else None + LOG.info("Catalogue order %s previous=%s new=%d changed=%s", view, + len(old) if old is not None else None, len(order), old != order) + current_html, current_signature = _read_source(source) + if current_html != html or current_signature != signature: + raise ValueError("Source changed during generation; publication aborted") + # Semantic equality avoids rewriting even a differently formatted valid baseline. + changed = previous != catalogue + if changed: + write_catalogue(catalogue, destination) + LOG.info("Catalogue publication changed=%s missions=%d destination=%s", changed, new_count, destination) + return catalogue def main(argv=None): @@ -149,12 +127,6 @@ def main(argv=None): help="Manual confirmation of upstream completion; skips the stability wait") parser.add_argument("--stability-seconds", type=float, default=30, help="Quiet observation window for periodic checks; default 30, not proof of completion") - parser.add_argument("--lock-file", type=Path, - help="Shared lock outside incoming rsync tree (default: .lock)") - parser.add_argument("--max-reduction-percent", type=float, default=0, - help="Maximum reviewed reduction in missions or any order; default 0 blocks every decrease") - parser.add_argument("--initial-mission-count", type=int, - help="Reviewed exact mission count, required only when output does not exist") args = parser.parse_args(argv) logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(name)s %(message)s") started = time.monotonic() @@ -162,9 +134,7 @@ def main(argv=None): status = 1 try: publish_catalogue(args.input, args.output, source_complete=args.source_complete, - stability_seconds=args.stability_seconds, lock_file=args.lock_file, - max_reduction_percent=args.max_reduction_percent, - initial_mission_count=args.initial_mission_count) + stability_seconds=args.stability_seconds) status = 0 except (OSError, ValueError, ImportError) as error: LOG.error("Validation/publication failed; previous catalogue preserved: %s", error) diff --git a/ARGO/oceancurrent/test_publish_gliders_catalogue.py b/ARGO/oceancurrent/test_publish_gliders_catalogue.py index de3ea276..a9e2c7df 100644 --- a/ARGO/oceancurrent/test_publish_gliders_catalogue.py +++ b/ARGO/oceancurrent/test_publish_gliders_catalogue.py @@ -1,5 +1,4 @@ import copy -import fcntl import json import os from pathlib import Path @@ -9,7 +8,7 @@ import unittest from unittest.mock import patch -from bs4 import BeautifulSoup +from gliders_catalogue_test_utils import edit_rows, has_mission import publish_gliders_catalogue as publisher from gliders_catalogue import parse_catalogue @@ -29,18 +28,14 @@ def setUp(self): self.count = len(parse_catalogue(self.html)["missions"]) def publish(self, **kwargs): - options = dict(source_complete=True, initial_mission_count=self.count) + options = dict(source_complete=True) options.update(kwargs) return publisher.publish_catalogue(self.source, self.destination, **options) def remove_mission(self, name, only_view=None): - soup = BeautifulSoup(self.html, "html.parser") - container = soup.find("a", id=only_view).find_next("table") if only_view else soup - for row in container.find_all("tr"): - cells = row.find_all("td", recursive=False) - if len(cells) == 12 and cells[-1].get_text(strip=True) == name: - row.decompose() - self.source.write_text(str(soup), encoding="utf-8") + html = edit_rows(self.html, lambda row: "" if has_mission(row, name) else row, + view=only_view) + self.source.write_text(html, encoding="utf-8") def test_initial_publication_and_unchanged_output(self): with self.assertLogs(publisher.LOG, level="INFO") as logs: @@ -74,24 +69,108 @@ def test_invalid_or_incomplete_source_preserves_output(self): self.publish() self.assertEqual(self.destination.read_bytes(), original) - def test_reduction_and_explicit_threshold(self): + def test_mission_increase_and_decrease_are_published(self): self.publish() - original = self.destination.read_bytes() self.remove_mission("Brisbane20180306A_nrt") - for threshold in (0, 10): - with self.assertRaisesRegex(ValueError, "Reduction requires manual review"): - self.publish(max_reduction_percent=threshold) - self.assertEqual(self.destination.read_bytes(), original) - result = self.publish(max_reduction_percent=12.5) + with self.assertLogs(publisher.LOG, level="INFO") as logs: + result = self.publish() self.assertEqual(len(result["missions"]), self.count - 1) + self.assertEqual(json.loads(self.destination.read_text()), result) + self.assertIn("previous=8 new=7", "\n".join(logs.output)) + self.source.write_text(self.html, encoding="utf-8") + with self.assertLogs(publisher.LOG, level="INFO") as logs: + result = self.publish() + self.assertEqual(len(result["missions"]), self.count) + self.assertEqual(json.loads(self.destination.read_text()), result) + self.assertIn("previous=7 new=8", "\n".join(logs.output)) + + def test_individual_order_reductions_are_published(self): + from gliders_catalogue import VIEWS + + for anchor, view in VIEWS.items(): + with self.subTest(view=view): + self.source.write_text(self.html, encoding="utf-8") + previous = self.publish() + self.remove_mission("TwoRocks20260327", only_view=anchor) + with self.assertLogs(publisher.LOG, level="INFO") as logs: + result = self.publish() + self.assertEqual(len(result["missions"]), self.count) + self.assertEqual(len(result["orders"][view]), len(previous["orders"][view]) - 1) + self.assertEqual(json.loads(self.destination.read_text()), result) + expected = "Catalogue order %s previous=%d new=%d changed=True" % ( + view, len(previous["orders"][view]), len(result["orders"][view])) + self.assertIn(expected, "\n".join(logs.output)) + + def test_order_changes_logged_even_when_counts_unchanged(self): + previous = self.publish() + reordered = copy.deepcopy(previous) + for order in reordered["orders"].values(): + order.reverse() + with patch("gliders_catalogue.parse_catalogue", return_value=reordered): + with self.assertLogs(publisher.LOG, level="INFO") as logs: + result = self.publish() + self.assertEqual(json.loads(self.destination.read_text()), result) + for view, order in previous["orders"].items(): + self.assertIn("Catalogue order %s previous=%d new=%d changed=True" % ( + view, len(order), len(order)), "\n".join(logs.output)) + + def test_invalid_initial_html_does_not_publish(self): + for invalid in ("", "Maintenance", self.html[:self.html.index("")]): + with self.subTest(invalid=invalid[:30]): + self.source.write_text(invalid) + with self.assertRaises(ValueError): + self.publish() + self.assertFalse(self.destination.exists()) + + def test_invalid_generated_catalogue_rejected_before_any_write(self): + valid = parse_catalogue(self.html) + changes = [lambda c: c.update(schema_version=2), + lambda c: c["missions"][0].update(mission_name="invalid"), + lambda c: c["missions"][0].pop("latitude"), + lambda c: c["missions"][0].update(longitude=999), + lambda c: c["orders"].pop("mission_name"), + lambda c: c["orders"].update(deployment_date=["unknown"])] + for existing in (False, True): + if existing: + self.publish() + original = self.destination.read_bytes() if existing else None + for change in changes: + with self.subTest(existing=existing, change=change): + candidate = copy.deepcopy(valid) + change(candidate) + with patch("gliders_catalogue.parse_catalogue", return_value=candidate): + with patch("gliders_catalogue.write_catalogue") as write: + with self.assertRaises(ValueError): + self.publish() + write.assert_not_called() + if existing: + self.assertEqual(self.destination.read_bytes(), original) + else: + self.assertFalse(self.destination.exists()) + + def test_atomic_publication_and_reference_output(self): + from gliders_catalogue import os as writer_os - def test_partial_view_reduction_even_when_total_unchanged(self): self.publish() original = self.destination.read_bytes() - self.remove_mission("TwoRocks20260327", only_view="byebs") - with self.assertRaisesRegex(ValueError, "event_based_sampling"): - self.publish() - self.assertEqual(self.destination.read_bytes(), original) + self.assertEqual(original, FIXTURE.with_suffix(".json").read_bytes()) + self.source.write_text(self.html.replace("SG540", "SG541")) + replace = writer_os.replace + snapshots = [] + + def inspect_replace(temporary, destination): + temporary = Path(temporary) + self.assertEqual(temporary.parent, destination.parent) + self.assertEqual(destination.read_bytes(), original) + snapshots.append(json.loads(temporary.read_text())) + replace(temporary, destination) + + with patch("gliders_catalogue.os.replace", side_effect=inspect_replace) as replaced: + result = self.publish() + replaced.assert_called_once() + self.assertEqual(snapshots, [result]) + self.assertEqual(json.loads(self.destination.read_text()), result) + self.assertEqual(set(Path(self.folder.name).iterdir()), {self.source, self.destination}) def test_missing_unreadable_and_changing_source(self): self.publish() @@ -130,15 +209,10 @@ def changing_read(path): self.publish() self.assertFalse(self.destination.exists()) - def test_configuration_and_first_publication_review_required(self): - for options in (dict(initial_mission_count=None), - dict(initial_mission_count=1), dict(initial_mission_count=0), - dict(stability_seconds=0), dict(stability_seconds=-1), - dict(stability_seconds=float("nan")), dict(stability_seconds=float("inf")), - dict(max_reduction_percent=-1), dict(max_reduction_percent=100), - dict(max_reduction_percent=float("nan")), dict(max_reduction_percent=float("inf"))): - with self.subTest(options=options), self.assertRaises(ValueError): - self.publish(**options) + def test_invalid_configuration(self): + for seconds in (0, -1, float("nan"), float("inf")): + with self.subTest(seconds=seconds), self.assertRaises(ValueError): + self.publish(stability_seconds=seconds) self.assertFalse(self.destination.exists()) with self.assertRaisesRegex(ValueError, "Source must differ"): publisher.publish_catalogue(self.source, self.source, source_complete=True) @@ -171,19 +245,11 @@ def test_schema_records_and_orders_validation(self): with self.assertRaises(ValueError): publisher.validate_catalogue(candidate) - def test_concurrent_publication_fails(self): - self.publish() - original = self.destination.read_bytes() - lock_path = Path(str(self.destination) + ".lock") - with lock_path.open("a") as lock: - fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB) - result = subprocess.run([sys.executable, str(SCRIPT), "--input", str(self.source), - "--output", str(self.destination), "--source-complete"], - capture_output=True, text=True) - self.assertEqual(result.returncode, 1) - self.assertIn("holds the output lock", result.stderr) - self.assertEqual(self.destination.read_bytes(), original) - self.assertTrue(lock_path.exists()) + def test_no_lock_created_and_manual_confirmation_skips_wait(self): + with patch.object(publisher.time, "sleep") as sleep: + self.publish() + sleep.assert_not_called() + self.assertEqual(set(Path(self.folder.name).iterdir()), {self.source, self.destination}) def test_periodic_check_waits_and_publishes_only_stable_source(self): with patch.object(publisher.time, "sleep") as sleep: @@ -191,8 +257,9 @@ def test_periodic_check_waits_and_publishes_only_stable_source(self): sleep.assert_called_once_with(15) self.assertEqual(json.loads(self.destination.read_text()), result) original = self.destination.stat().st_mtime_ns - with patch.object(publisher.time, "sleep"), patch("gliders_catalogue.write_catalogue") as write: + with patch.object(publisher.time, "sleep") as sleep, patch("gliders_catalogue.write_catalogue") as write: self.publish(source_complete=False) + sleep.assert_called_once_with(30) write.assert_not_called() self.assertEqual(self.destination.stat().st_mtime_ns, original) @@ -230,24 +297,9 @@ def test_observation_compares_content_even_if_stat_signature_matches(self): self.publish(source_complete=False) self.assertEqual(self.destination.read_bytes(), original) - def test_external_lock_and_manual_confirmation(self): - lock_path = Path(self.folder.name) / "outside-transfer.lock" - with patch.object(publisher.time, "sleep") as sleep: - self.publish(lock_file=lock_path) - sleep.assert_not_called() - self.assertTrue(lock_path.exists()) - self.assertFalse(Path(str(self.destination) + ".lock").exists()) - with lock_path.open("a") as lock: - fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB) - with self.assertRaisesRegex(ValueError, "holds the output lock"): - self.publish(lock_file=lock_path) - with self.assertRaisesRegex(ValueError, "destination must differ"): - self.publish(lock_file=self.destination) - def test_periodic_cli_failure_exit_and_logging(self): self.publish() original = self.destination.read_bytes() - lock_path = Path(self.folder.name) / "outside-transfer.lock" def update_source(seconds): self.source.write_text(self.html.replace("SG540", "SG541")) @@ -255,10 +307,9 @@ def update_source(seconds): with patch.object(publisher.time, "sleep", side_effect=update_source) as sleep: with self.assertLogs(publisher.LOG, level="INFO") as logs: status = publisher.main(["--input", str(self.source), "--output", str(self.destination), - "--stability-seconds", "5", "--lock-file", str(lock_path)]) + "--stability-seconds", "5"]) self.assertEqual(status, 1) sleep.assert_called_once_with(5) - self.assertTrue(lock_path.exists()) self.assertEqual(self.destination.read_bytes(), original) messages = "\n".join(logs.output) for expected in ("Observing HTML stability", "stability observation", "status=1", "duration_seconds="): @@ -273,11 +324,11 @@ def test_publication_failure_preserves_previous_and_cleans_temp(self): self.publish() self.assertEqual(self.destination.read_bytes(), original) self.assertEqual({p.name for p in Path(self.folder.name).iterdir()}, - {self.source.name, self.destination.name, self.destination.name + ".lock"}) + {self.source.name, self.destination.name}) def test_cli_exit_codes_and_logging(self): args = ["--input", str(self.source), "--output", str(self.destination), - "--source-complete", "--initial-mission-count", str(self.count)] + "--source-complete"] with self.assertLogs(publisher.LOG, level="INFO") as logs: self.assertEqual(publisher.main(args), 0) messages = "\n".join(logs.output) @@ -290,22 +341,20 @@ def test_cli_exit_codes_and_logging(self): self.assertIn("Validation/publication failed", "\n".join(logs.output)) self.assertIn("status=1", "\n".join(logs.output)) result = subprocess.run([sys.executable, str(SCRIPT), "--input", str(self.source), - "--output", str(self.destination), "--max-reduction-percent", "invalid"], + "--output", str(self.destination), "--stability-seconds", "invalid"], capture_output=True, text=True) self.assertEqual(result.returncode, 2) self.assertIn("invalid float value", result.stderr) - def test_cli_missing_beautifulsoup_and_source_log_failures(self): - self.publish() - original = self.destination.read_bytes() + def test_cli_without_site_packages_and_source_log_failures(self): args = [str(SCRIPT), "--input", str(self.source), "--output", str(self.destination), "--source-complete"] - # -S removes site-packages, simulating the production dependency being absent. + # -S excludes site-packages, including BeautifulSoup, for a first publication. result = subprocess.run([sys.executable, "-S"] + args, capture_output=True, text=True) - self.assertEqual(result.returncode, 1) - self.assertIn("No module named 'bs4'", result.stderr) - self.assertIn("Validation/publication failed", result.stderr) - self.assertIn("status=1", result.stderr) + self.assertEqual(result.returncode, 0, result.stderr) + self.assertIn("status=0", result.stderr) + self.assertEqual(json.loads(self.destination.read_text()), parse_catalogue(self.html)) + original = self.destination.read_bytes() self.source.unlink() result = subprocess.run([sys.executable] + args, capture_output=True, text=True) self.assertEqual(result.returncode, 1)