Skip to content

test: REST bbbc on the rebuilt image; drop convention tests - #534

Merged
vjaganat90 merged 6 commits into
PolusAI:masterfrom
vjaganat90:fix/vjaganat90-images
Oct 15, 2026
Merged

vjaganat90 merged 6 commits into
PolusAI:masterfrom
vjaganat90:fix/vjaganat90-images

Conversation

@vjaganat90

@vjaganat90 vjaganat90 commented Oct 14, 2026 •

Copy link
Copy Markdown
Member

What

  • The REST bbbc payloads (bbbc_download_wfb.json, single_node_bbbc_download.json) run vjaganat90/bbbc-download-plugin:0.1.1-dev1.
  • Dispatched Lint And Test and Run Workflows runs are titled by the commit subject, plus a paired mm-workflows or image-workflows branch when it is not main.
  • 50 tests and the "PyTest Benchmark Harness" step are deleted. Each pinned how the code or the CI is built, not what a user sees. No src/ change.
  • The weekly benchmark lane goes red when a case raises or its corpus preparation fails, after the full table is written. No timing fails it. The || true, the continue-on-errors and the dispatch step that failed on purpose to show the lane stays green are gone.
  • test_realtime.py's cache helper writes each file in one step. The watcher thread could scan a log between its creation and its text, count a change no test made, and do an extra final run, which made test_a_failed_analysis_is_a_line intermittently report 2 runs.

Why

  • REST bbbc: the old image's 2022 CA bundle cannot verify data.broadinstitute.org's chain, so both REST bbbc runs fail with CERTIFICATE_VERIFY_FAILED. This is the REST half of fix: pull the rebuilt tool images from vjaganat90 image-workflows#28.
  • Run titles: push and cross-repo dispatches passed the whole commit message, and the title also listed every repo and ref, so the run list was paragraphs. The run's own branch is already shown by GitHub. The "Name what this run tests" step still writes all three checked-out commits to the summary.
  • Tests: every deleted test fails on a rename, a refactor or a new enum member while users see no change. A failure there costs a fix and catches nothing. The groups and their reasons follow.

Deleted tests

  • Removed names and module layout. The API simply has none of these names, and reaching for one is Python's own AttributeError/ImportError.
    • test_python_api_workflow.py: string indexing on ports, write_wic as the only writer, the top-level module list, the legacy module path, the string-keyed workflow and step methods.
    • test_python_api_tool_builder.py:
      • the old module name, no from_input on outputs, identity comparison of spec classes;
      • two of Python's own TypeErrors (positional inputs and outputs, a single FieldSpec);
      • that validation goes through cwltool in-process, which is how it runs, not what it reports.
  • Release and CI config read back as text. test_release.py (7) asserted the release-please config and the YAML of the publish jobs.
  • Tripwires on a single version. Sophios writes one CWL version and has one language version. Adding a second is a deliberate change that has to touch these tables anyway.
    • test_cwl_version.py: v1.2 declared, every supported version admitted.
    • test_lang_version.py: exactly one version.
    • test_inline_flags.py: the per-version table.
    • test_reference_compatibility.py: dispatch over versions.
  • Inventories of the tests themselves.
    • test_cookbook.py: every construct has a cookbook pair. The pairs themselves are still tested.
    • test_manufactured_documents.py: the hand-kept list of sites no driver reaches. The checks on reached sites stay.
  • Mechanism pins. The behaviour these protect is tested through parse, compile and emit.
    • test_lower.py: __post_init__ invariants, no mutable container in the graph, how often a name is printed, the output_step helper.
    • test_resolve.py: phases typed against each other, test-setup side effects.
    • test_emit.py: the compiler emits through its graph.
    • test_lang_schema.py: surface declarations, an undeclared field raising, schema objects not shared.
    • test_namespacing.py: read-only and frozen tables, membership sets computed, the exported schema not shared.
    • test_lang_parser.py: Diagnostics slicing.
    • test_lang_render.py: a literal keeping its source text.
    • test_inline_flags.py: an identity return.
    • test_diagnostics.py: a private coercion helper, which the wic020 tests cover.
    • test_settings_boundary.py: the converter's required arguments.
  • Benchmark harness. test_benchmarks.py (4) and its CI step tested the weekly timing script, not Sophios. benchmarks_weekly.yml still runs the script and publishes the table, and a broken harness now turns that run red.

Known gaps

  • The release workflow is no longer checked on each PR, only when release-please cuts a release.

@vjaganat90 vjaganat90 changed the title fix(tests): run the REST bbbc downloads on vjaganat90/bbbc-download-plugin:0.1.1-dev1 fix(tests): REST bbbc runs use the rebuilt image Oct 15, 2026
@vjaganat90
vjaganat90 force-pushed the fix/vjaganat90-images branch from bd97a4b to 9f0f6bd Compare October 15, 2026 00:24
The old bbbc-download image's CA bundle cannot verify
data.broadinstitute.org, so both REST bbbc downloads failed.
@vjaganat90
vjaganat90 force-pushed the fix/vjaganat90-images branch 2 times, most recently from 31c4ca1 to d2f3ce7 Compare October 15, 2026 00:48
The title is the subject line, plus a paired mm-workflows or
image-workflows branch when it is not main.
@vjaganat90
vjaganat90 force-pushed the fix/vjaganat90-images branch from d2f3ce7 to 679de13 Compare October 15, 2026 01:08
@vjaganat90 vjaganat90 changed the title fix(tests): REST bbbc runs use the rebuilt image test: REST bbbc on the rebuilt image; drop convention tests Oct 15, 2026
@vjaganat90
vjaganat90 force-pushed the fix/vjaganat90-images branch 2 times, most recently from b67a2a0 to e57c3ed Compare October 15, 2026 02:02
We drop removed-name checks, CI and doc text read back, censuses no user
depends on, and tests that pin a mechanism rather than a meaning.
@vjaganat90
vjaganat90 force-pushed the fix/vjaganat90-images branch from e57c3ed to 5980604 Compare October 15, 2026 02:10
Timings still gate nothing. A case, or the corpus preparation, that fails now turns the weekly run red instead of passing silently.
Design §6.3 said benchmarks never fail a build; a case that raises now does.
The watcher could scan a log between its creation and its text, count a change no test made, and do an extra final run ("2 runs").

@darkclad darkclad left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@vjaganat90
vjaganat90 merged commit 378aff1 into PolusAI:master Oct 15, 2026
3 checks passed
@vjaganat90
vjaganat90 deleted the fix/vjaganat90-images branch October 15, 2026 15:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants