Repository navigation
fix: rebuild the time-series date index after large eTUFF loads - #461
Merged
Merged
Conversation
POST/GET /ingest return 202 with a job id instead of holding the
request open while every archive member is processed, which caused
nginx 504s on large zips. Poll GET /ingest/jobs/{job_id} for status.
One worker, started by docker-entrypoint.sh beside Gunicorn, runs
queued jobs from the new ingest_job table. Clients that read the old
200 Ingest200 body must poll the job instead.
POST /ingest/jobs streams a .txt, .zip, .tar.gz, or .tgz up to 2 GiB and returns 202. One worker drains the ingest_job queue. Synchronous /ingest still finishes inside the request and rejects bodies over 100 MiB. The eTUFF migration matches histogram rows without joining every time-series row, and resolves position times with one indexed lookup. submission.filename stores the eTUFF name. Finished jobs are purged by cron. fswatch and the upload page use the job API. Archive extraction uses the Python standard library.
The job upload opened its staging file inside an async function, and repeated error strings were duplicated literals.
bash 5 treated a zero-second read as another copy of the previous path, so later drops never reached the job API. The watcher also exited on startup because its state volume was not writable.
The contract test read the pending list as soon as the second POST returned, before the watcher finished writing that job.
Maintaining data_time_series_date_time_index on every row held a large load to about 29 MB/min of heap once existing data already covered 2011–2026. Submissions of at least one million time-series rows now drop that index for the insert and create it once, in the same transaction, after positions are linked.
Merge origin/main, including queued-upload ingest from #437. Keep the deferred time-series date index rebuild in both schema files.
Member
Author
|
@renato2099 this is the most important performance improvement before we shift SQL stored procedures to Python. |
lewismc
changed the title
lewismc
force-pushed
the
fix/timeseries-date-index
branch
from
October 18, 2026 04:12
24e8f08 to
8da2ee2
Compare
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
@renato2099 this PR will be updated once #437 is merged into
main. This PR include #437 but really all that will change in this PR is the SQL files. I also want your take on refactoring the SQL files into one file rather than two. We can discuss that separately.Cross posting from Slack: I'm sharing some results from smoke testing ingest performance of the feat/async-ingest branch and some local modifications to the tagbase schema. The local changes will land in a separate PR after we finish review of feat/async-ingest branch.
Large eTUFF files now finish in about 27 minutes. The previous file of this size took 1 hour 5 minutes.
A file with at least one million time-series rows no longer updates the date index on every row. Tagbase drops that index for the load, writes the measurements, links them to positions, and builds the index once before the load is visible. Smaller files are unchanged.
I loaded five ICCAT bluefin tuna files, 715–792 MiB, one at a time into a database that already held the earlier smoke-test data. Job time includes parsing and the database write. All five succeeded, and the date index was present again after each one.
• 2024BFT07891A_264262_23P2947.txt — 714.8 MiB, 59 min, 8% shorter than 1:04:45
• 2024BFT07903A_265548_24P0087.txt — 764.4 MiB, 1 hr 4 min, 1% shorter
• 2024BFT07902A_265536_24P0021.txt — 766.5 MiB, 27 min, 59% shorter
• 2024BFT07799A_264255_23P2610.txt — 788.8 MiB, 27 min, 58% shorter
• 2024BFT07801A_264258_23P2820.txt — 791.5 MiB, 27 min, 58% shorter
The previous file was 2024BFT07901A_265012_23P2911.txt (714.0 MiB, 1:04:45).
The last three files agree. Each wrote about 18–19 million time-series rows. Rebuilding the date index took about 12 seconds.
Why the first two files were still about an hour: parsing (~7 min) and position linking (~4 min) took the same time on every file. The time-series insert did not. On the first two files it ran for 34–42 minutes at about 35 MB/min and spent most of that time reading data pages from disk. On the last three it finished in 2–7 minutes at 200–600 MB/min and stayed on CPU, with almost no data-page reads.
All five files already had the date index removed during the load, so that change does not explain the split. New rows are written in file order, while the primary key is ordered by submission, tag, variable, then time, and its pages are packed full. Those inserts were splitting index pages that did not stay in the 256 MB database cache. After the first file, autovacuum cleaned
data_time_series. The second file started during that cleanup and was the slowest insert. After the cleanup, the next three inserts touched a small set of cached primary-key pages, and the disk reads went away.Use the later three runs as the time to expect for another file of this size on this database. During a large load, the date index is unavailable until that load commits.