Document how the storage backend is chosen - #110
Merged
Conversation
tibvdm
force-pushed
the
feature/port-to-unipept-index-main
branch
from
September 4, 2026 07:50
4b9a6aa to
9fd47e8
Compare
tibvdm
force-pushed
the
docs/storage-backend-build
branch
from
September 4, 2026 07:50
e933442 to
e88f35c
Compare
tibvdm
force-pushed
the
feature/port-to-unipept-index-main
branch
from
September 4, 2026 08:57
9fd47e8 to
458ab8c
Compare
tibvdm
force-pushed
the
docs/storage-backend-build
branch
from
September 4, 2026 08:57
e88f35c to
ed906a2
Compare
tibvdm
force-pushed
the
feature/port-to-unipept-index-main
branch
from
September 4, 2026 09:12
458ab8c to
ef1418e
Compare
tibvdm
force-pushed
the
docs/storage-backend-build
branch
2 times, most recently
from
September 4, 2026 09:54
e24cef7 to
5d0f683
Compare
tibvdm
marked this pull request as ready for review
September 4, 2026 10:06
tibvdm
force-pushed
the
feature/port-to-unipept-index-main
branch
2 times, most recently
from
September 4, 2026 10:43
3c07a85 to
bc347c7
Compare
The backend is compiled in rather than passed as a flag, so someone building the server has to know the features exist before they can pick one — and picking wrongly is now a redeploy rather than a restart. Records which build production uses, what the startup line reports, and the part that is easy to get backwards: preloading only pays when the whole index is guaranteed resident, and is the wrong default anywhere the memory ceiling might move. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
tibvdm
force-pushed
the
docs/storage-backend-build
branch
from
September 4, 2026 10:46
5d0f683 to
61670ec
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Step 7 of 8. Stacked on #106. Independent of #107, #108 and #109.
After #106 the storage backend is a compile-time choice, which means someone building the
server has to know the features exist before they can pick one — and picking wrongly is a
redeploy rather than a restart. Nothing in the README said so.
Adds a section covering the nine configurations, which one production uses
(
--features mmap), the startup line that reports the choice back, and the deploymentguidance from the index's own benchmark record.
The part worth reviewing carefully
The guidance is counter-intuitive and I have quoted it from upstream rather than derived it:
preloading is worth +22% to +58% only when the whole index is guaranteed resident, and is
actively worse under a memory ceiling —
preloaded-proteinsis behind plainmmapat everyceiling, and the fully preloaded build is OOM-killed at every one. Also documents
RAYON_NUM_THREADS, which is the single largest effect in that record and costs ~10% when theindex is resident.
Someone who knows the deployment should check these against how the servers are actually
provisioned before this is marked ready. The figures are one machine's, against the 223 GB
UniProt index, and the README says so.
Verification
Prose only; no code changes.