Progress and handover

Current state of the Python app, what is still open, and the decisions and lessons a new session needs before changing anything. For how to set up, build, test and release, see README.md and packaging/README.md.

The full session-by-session history (seven rounds of curator feedback, the performance investigation, the packaging saga) was trimmed from this file on 2026-09-26 -- it is in git history (git log -- python/PROGRESS.md) if a similar bug resurfaces and you want to see how it was fixed last time.

Round 10 curator feedback (2026-10-18) -- done

1 Guide: use the window's width (it stopped at 900px) -- two columns (.bc-guide-grid, 3:1): the steps, and the grades and the BAGS paper in a box beside them; one column below 1100px. Found on the way: the list's padding cut "10." to "0.".
2 Default search: "Sialidae", not "Nymphalidae" (the Search tab's taxa box).

3 Speed: the first table after a large search, and large Excel downloads, are slow, and the download toast disappears long before the save dialog. Options to discuss before changing anything -- measured, options put to the owner, awaiting a choice. On a 200,000-record fake-package snapshot (make_fake_package.py --rows 200000; its Nymphalidae is 85,067 records, about the real family's size), this sandbox:

Step	Pieridae (28,329)	Nymphalidae (85,067)
Search (plan)	0.02 s	0.03 s
First summary screen (`SearchState.analysis`): fetch 0.7, process 0.6, score 1.6, BAGS 0.75, auto-select 0.5, BINs 0.3	1.7 s	5.1 s
Checklist, groups, one specimen page	< 0.4 s	< 0.4 s
Download All, xlsx (openpyxl)	15 s	50 s
Download All, TSV	0.7 s	2.0 s
Prototype hand-written xlsx (inline strings, streamed into the zip; reopens in openpyxl)	--	5.2 s

The owner chose A1 (faster writer) + A2 (prepare in the background, then save) + B1 (analyse in the background). A TSV-then-convert route was considered and dropped: the conversion is the Excel writing.

A1 write_xlsx writes the spreadsheet XML itself, streamed into the zip 2,000 rows at a time, with inline strings (never formulas), a bold frozen header and the About sheet: Nymphalidae's 85,067 x 83 in 5.5 s (was 50 s). Checked by reopening in openpyxl and converting with LibreOffice (=1+1 stays text). It reports progress per chunk.
B1 The analysis starts on a background thread as soon as there is a search (also after loading a session or reconnecting): _start_analysis in ui/app.py. SearchState.analysis holds a lock, so anything else asking waits for that run; the result is published only once representatives are chosen, from a copy of the picks, swapped in as a new dict. Every summary screen shows the stage ("Fetching 85,067 records...", scoring, BAGS, BINs, representatives) with a spinner, and the Specimens tab works meanwhile: on the 85,067-record search, the progress line at 0.1 s, a specimen page at 1.8 s with the analysis still running, the checklist at 8 s. Autosave skips while it runs (saving needs the result).
A2 Download All / Selected / Annotated are now buttons that prepare the workbook on a background thread (_start_prep), from a copy of the annotations taken at the click (Annotations.snapshot). A panel in the corner, on every tab, shows "Preparing all_specimens_....xlsx" with a bar and "40,000 of 85,067 rows". When it's written, the server asks the page (custom message bc_click) to click the hidden dl_prepared link, and the save starts by itself; the panel says it's ready, offers "Save it again", and goes after 20 s. With nothing to export it says why instead of saving an empty workbook (replacing round 9's one-cell workbook). The hidden link needs suspend_when_hidden=False: Shiny never gives a hidden download link its address. Measured, Nymphalidae, clicked the moment the search ran: waiting for the analysis 4 s, then writing, saved at 10.4 s (was over 55 s, frozen), and the table paged in 0.6 s meanwhile. drive_ui.py checks the save starts by itself. The curation report (TSV), FASTA and the process-ID list are still direct downloads. To confirm on a real machine: the native window starts a download the page clicks for it (WebView2, WKWebView).
Groundwork: SnapshotStore.connection gives each thread its own cursor (same read-only, locked-down database, its own registrations).

Also found: Shiny runs a download's generator, and every render, on its one event loop (shiny/session/_session.py, wrap_content_sync), so while either runs the whole app is frozen -- every button, every tab. That, plus a 4-second toast, is why a curator is left wondering. The real 7.95 GB file will fetch more slowly than this one; boldcurator benchmark --snapshot FILE --taxon Nymphalidae gives the real numbers.

Round 9 curator feedback (2026-10-17) -- done

All items closed; 9c deliberately not done (owner's call). Still to confirm on a real machine: the native window keeps the launch cookie (9b), and the macOS one-command install on the curator's Mac (8).

Round 8 curator feedback (2026-10-05) -- done

All 17 items closed; each note says what changed. Still to confirm on a real machine: the reconnect after an actual sleep (simulated here by closing the websocket), and the installer's licence page (built by CI only).

State (2026-10-17)

Feature-complete and in curators' hands. Nine rounds of curator-reported issues are closed; there are no open curator bugs.

The app: Guide, Data, Search, Gap analysis, Species, BINs, BAGS A-E (E and C marked "work here first"), Phylogeny and Specimens tabs; per-record annotations (flag, note, corrected ID; clearable) and representative picks; Find records on Specimens and every group; session save/load with 60-second autosave; xlsx/CSV/FASTA/Newick downloads, a process-ID list and the BOLD curation report (TSV), all stamped with the CC BY-SA 4.0 attribution. The local server only serves the window it opened (localguard.py). See the README's "What the app does".
Snapshots: built from the BOLD public data package by build/snapshot_builder.py, published on Zenodo (concept DOI 10.5281/zenodo.22849515, always the latest version). The app downloads it on first run or from the Data tab -- in parallel byte ranges (4 connections), with a BOLDcurator/<version> User-Agent and Retry-After handling, as Zenodo asks of automated clients -- and can check for a newer one.
Distribution: every published GitHub release builds Windows (zip and Inno Setup installer), macOS (Intel and Apple Silicon .app) and Linux executables, and publishes to PyPI for the uv/pip route with its shortcut command. The version comes from the release tag everywhere (packaging/stamp_version.py); the frozen build's --version is checked in CI. Unsigned, deliberately.
Tests: about 645 pytest tests, the R parity gate (PASS, nine recorded divergences) and CI on Linux, macOS and Windows. tools/drive_ui.py makes 56 checks against the running UI (it needs the URL gui prints).

Open items

Nothing here blocks curators. Roughly in priority order:

Confirm on a real Windows/WebView2 machine -- this sandbox can't run pywebview, so these were reasoned from pywebview's source, not observed:
- a native window's Save As keeps the file extension (desktop._patch_edgechromium_download_extension);
- native-window downloads work at all (desktop._enable_webview_downloads, ALLOW_DOWNLOADS);
- the native window (WebView2 here, WKWebView on macOS) keeps the launch cookie through the first redirect and sends it with downloads (round 9, item 9b). If it doesn't, the window shows "can't connect" straight away.
Confirm by hand, on real machines: the native Browse… file picker (needs a display); the macOS .app opened from a real, quarantined browser download; a clean-VM install of each platform's download. CI's smoke tests are a proxy for these, not a replacement.
PyPI: confirm the trusted publisher is set up on pypi.org (project boldcurator, workflow python-pypi.yml, environment pypi) and that a release has published. The website's one-command install depends on it.
Republish the Zenodo snapshot if it predates the "keep every column" builder fix. The builder used to keep only columns named in schema.py; it now keeps every source column except identifier_email. A snapshot built before that change lacks the extra columns until it is rebuilt from a raw BOLD package and republished.
Optional: say in the UI that BAGS covers about a third of records. 67% of COI-5P records have no species-level name, so they carry no grade. Normal for BOLD, but a curator may assume otherwise.
R app issues found while porting (the R app isn't this port's code, so none of these were changed):
- the dataset-code check ^DS-[A-Z0-9]+$ (R/modules/data_import/mod_data_import_utils.R:50) rejects 544 of 13,706 real DS- codes -- though the R UI currently has no dataset or project input, so it can't be reached;
- dead code: the dataset/project search path, the BIN-analysis Excel download and ExportManager have no UI; R/modules/export_history/ is never sourced;
- R/modules/export/mod_export.R:228 still lists an institution column;
- rsconnect/shinyapps.io/.../BOLDcuratoR.dcf lists about.md under ignoredFiles, but the Data Input tab renders it.
To list real dataset codes the R pattern rejects:
```
SELECT DISTINCT recordset_code FROM specimen_recordset
WHERE recordset_code LIKE 'DS-%'
  AND NOT regexp_matches(recordset_code, '^DS-[A-Z0-9]+$') LIMIT 30;
```

Decisions already taken -- do not relitigate

Decision	Choice
Repository	R and Python apps stay together in this repo. R stays at the root (its shinyapps/rsconnect deployment, `.Rprofile` and renv rely on those paths); `python/` is self-contained. Release workflows are Python-only.
Snapshot	COI-5P only, sequences included, one file (7.95 GB, 1.9 GB gzipped). Every source column kept except `identifier_email` (privacy).
Species-name rule	One unified rule replacing R's five divergent regexes
`HAS_IMAGE`	Removed; image requirement dropped from `RANK_2` so rank 2 stays reachable. Max score 15, not 16
Dataset/project codes	Implemented properly via `specimen_recordset`
BAGS grade E	Evaluated against the whole snapshot, not just downloaded records
BAGS C+E membership	A C+E group shows the species' own records in every BIN, plus the other species in its shared BINs, plus higher-rank riders (round 8, item 8.1, reversing the earlier "own records only" choice): the problem is the BIN, so the whole BIN is shown.
Flag dropdown	Blank means "no change"; "Clear flag" and "Clear annotations" undo (round 9, item 5). A blank flag used to clear, which silently wiped flags when only a note was applied.
Local server access	Only the window that opened the one-time launch URL, plus a loopback `Host` check (`localguard.py`, round 9, item 9b). Don't add an unguarded route or serve the app without `LocalGuard`.
Record search	One "Find records" row (column or any, contains, case-insensitive) per screen, not a filter per column (round 9, item 2).
Connection loss	The loopback server never pings its window (`desktop.WEBSOCKET_KEEPALIVE`). If the socket drops anyway, the page reloads itself and the server restores the session (round 8, item 1.1). Don't reintroduce uvicorn's default pings.
BIN-less records	Excluded from BAGS entirely (project owner's instruction, reversing an earlier match to R): a species-level record with no BIN is not counted, and a species with no BIN-assigned records gets no grade. They still appear in the specimen table.
GUI	Shiny for Python. No widget ever receives a whole result: the specimen table is server-side paged from `core/table.py`
Desktop window	Best effort: `auto` tries a native pywebview window, then a Chromium `--app` window, then a browser tab. A pywebview failure never crashes the app
Signing	Not signed (project owner's call)
Versions	From the release tag only; `pyproject.toml` keeps `0.0.0.dev0` and is never edited by hand
Performance	Closed. Reopen only with a measurement showing a real user waiting (round 10 did). Slow work goes on a background thread with visible progress, never in a render or a download handler: Shiny runs those on its one event loop, so they freeze the whole app. A background thread uses `store.connection` (its own cursor) and reads a copy of the annotations.

Known divergences from the R app -- all deliberate, all tested

parity/REPORT.md (generated by parity/compare.py) is the authority. Nine categories, each with a registered explanation:

UNIFIED_SPECIES_RULE -- one species-name rule instead of R's five, and wider than any of them: it also rejects interim names R let through (cf/aff/sp without a full stop, nr., gr., agg., complex, indet., ?) and stops mistaking ssp. for sp..
RANK2_IMAGE_REMOVED -- no image criterion; max score 15.
CF_AFF_CONCORDANCE -- R's cf\.|aff\.<name> pattern accepts any cf. record by alternation precedence; here such records aren't species-level.
R_ROW_ERROR_ZEROES_SCORE -- R scores a whole row 0 when one criterion errors.
BIN_LESS_EXCLUDED_FROM_BAGS -- see the decisions table.
SPECIES_KEPT_VERBATIM -- the species field is never blanked; name_status records what kind of name it is.
INTERIM_SPECIES_GRADED -- interim species names at species rank are graded, and strictly make a shared BIN grade E.
RANK_DECIDES_SPECIES_LEVEL -- a binomial at a non-species rank is not a species, in the BIN list too.
GENUS_CONFLICT_DISCORDANT -- more than one genus, family or order in a BIN makes it discordant (grade E), whatever rank the records are at.

Adding one means adding it to EXPLANATIONS in compare.py and recording it here.

Rules this project cost the most to learn

Measure before optimising. Both real performance causes -- a projection DuckDB couldn't push down, and the sequence table stored in the wrong physical order -- were invisible to reasoning and obvious to boldcurator benchmark.
Drive the UI in a browser before believing it. Every UI bug so far has been invisible to the unit tests and obvious on the first click. Run tools/drive_ui.py; it exits non-zero on any failure.
In a Shiny effect, read every input you react to outside reactive.isolate(), and isolate only the writes. Two controls rendered, accepted clicks and did nothing because of this.
When checking the UI, match against the table, not the panel. The annotation toolbar's flag <select> lists every flag name, so "synonym" in panel.inner_text() passes for an annotation that never rendered.
ui.navset_pill_list keeps every tab mounted, just hidden. Scope Playwright selectors to the active panel's output id (#specimens_body .bc-row-select), not a class alone. A <select> change is a websocket round-trip: wait before clicking Apply, or it applies the previous value.
Scripts inside ui.HTML(...) never run (browsers don't execute innerHTML scripts). Per-row controls in a table that re-renders use one document-level delegated listener -- see ROW_CHECK_CLASS, ROW_REP_CLASS, SORT_HEADER_CLASS and SCROLL_CLASS in ui/app.py.
value != value catches NaN but raises on pd.NA, which is what BIN-less rows carry. Use pd.isna.
Verify a snapshot from a fresh process. A leftover DuckDB write-ahead log makes the file unopenable read-only, and the process that wrote it can't see that.
A frozen build fails in ways a source install doesn't. Missing package data (--collect-all shinychat, Bio), missing metadata (--copy-metadata boldcurator), a CA path that only exists on the CI runner. boldcurator selftest and the release workflow's smoke tests exist for exactly this; extend them when a new one bites.

Process notes

Verify from a clean clone in a fresh virtualenv before pushing. Setup failures (a .gitignore pattern excluding the build package, an undocumented dependency, a stale .pyc) have all come from testing where the environment already had what a new one wouldn't.
R for the parity harness: apt-get update && apt-get install -y --no-install-recommends r-base-core r-cran-r6, then use Rscript --vanilla -- without it the repo's .Rprofile bootstraps renv and tries to reach CRAN.
Release packaging lessons (why each PyInstaller flag, the macOS .app, the Windows installer) live in packaging/README.md, not here.

Navigation Menu

Repository navigation

PROGRESS.md

PROGRESS.md

Progress and handover

Round 10 curator feedback (2026-10-18) -- done

Round 9 curator feedback (2026-10-17) -- done

Round 8 curator feedback (2026-10-05) -- done

State (2026-10-17)

Open items

Decisions already taken -- do not relitigate

Known divergences from the R app -- all deliberate, all tested

Rules this project cost the most to learn

Process notes

FilesExpand file tree

PROGRESS.md

Latest commit

History

PROGRESS.md

File metadata and controls

Progress and handover

Round 10 curator feedback (2026-10-18) -- done

Round 9 curator feedback (2026-10-17) -- done

Round 8 curator feedback (2026-10-05) -- done

State (2026-10-17)

Open items

Decisions already taken -- do not relitigate

Known divergences from the R app -- all deliberate, all tested

Rules this project cost the most to learn

Process notes