Skip to content

Fix stale CSV header missing 'open' column in statistics_history.csv - #377

Open
Chessing234 wants to merge 2 commits into
teorth:mainfrom
Chessing234:fix/statistics-history-csv-header
Open

Fix stale CSV header missing 'open' column in statistics_history.csv#377
Chessing234 wants to merge 2 commits into
teorth:mainfrom
Chessing234:fix/statistics-history-csv-header

Conversation

@Chessing234

Copy link
Copy Markdown
Contributor

The open field was added to the row schema in commit 06fdb21 (scripts/plot_statistics_history.py FIELDNAMES) but the CSV header on line 1 of data/statistics_history.csv was never updated, since the header is only written by update_history() when the file doesn't yet exist. Every row appended since 2025-12-12 has 11 values while the header names only 10 columns.

Verified with csv.DictReader (what the repo's own scripts use to read this file): the 11th value on each affected row lands under the None restkey instead of open, so row.get("open", ...) in generate_charts() never finds the recorded value and silently falls back to a computed default. This happens to match numerically right now (default = total_problems - total_solved), which is why it wasn't noticed, but the CSV is malformed for any other consumer (pandas, spreadsheets, etc.) and the real recorded 'open' values are being discarded on every read.

Fix: add open to the header row so it matches the 11 columns actually written. Confirmed with DictReader before/after that the 'open' key now parses correctly for all post-06fdb21 rows, and that older rows (pre-'open' schema) correctly get None for the missing field. Ran python scripts/validate.py — passes.

The 'open' field was added to the row schema in commit 06fdb21 but the
CSV header (first line) was never updated to include it, since the
header is only written when the file is first created. Every row
since 2025-12-12 has 11 values while the header only names 10 columns,
so csv.DictReader (used by scripts/plot_statistics_history.py) puts
the 11th value under the None restkey instead of 'open', silently
discarding the recorded value on every read.
Adding `open` to the CSV header makes csv.DictReader hand back
`open: None` for the 350 rows written before that column existed, and
`int(None)` raises. The existing `row.get("open", ...)` fallback cannot
help: DictReader creates a key for every column the header declares, so
the default only fires when the header itself lacks the column.

Read the numeric cells through a helper that treats missing and blank as
the default, so both the chart generation and the last-row comparison in
update_history keep working across a column addition.
@Chessing234
Chessing234 force-pushed the fix/statistics-history-csv-header branch from 27bdd37 to 55d4c70 Compare August 13, 2026 01:12
@Chessing234

Copy link
Copy Markdown
Contributor Author

rebased onto main, and added the script side — adding open to the header alone breaks chart generation.

csv.DictReader creates a key for every column the header declares, so the existing row.get("open", ...) fallback never fires; the 350 rows written before that column get open: None and int(None) raises. running plot_statistics_history.py on this branch before the second commit: TypeError at line 134 in generate_charts, exit 1. after: exit 0.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant