Tier: D (cMD paper) · Type: atomic · Category: Machine Learning
What
Evaluate a classifier by holding out one entire dataset at a time — training on all remaining datasets, testing on the held-out one — and reporting performance per held-out dataset. The classifier is an input to this protocol, not part of it.
Why it matters
Split from the random forest protocol, per review: LODO applies to any classifier, and random forests are used without LODO. Keeping them separate is what makes the useful half reusable — anyone benchmarking a new model on multi-cohort data needs this validation scheme and should not have to adopt someone else's classifier to get it.
LODO is the honest evaluation for multi-cohort microbiome data. Ordinary k-fold cross-validation lets batch and study effects leak between train and test folds, inflating AUROC — sometimes dramatically — because the model can learn the study rather than the phenotype. LODO measures what people actually want to know: will this transfer to a cohort I have not seen?
Writing it down is partly a methods-standards contribution. It gives anyone benchmarking on curatedMetagenomicData a citable specification of the right cross-validation scheme, independent of what they are benchmarking.
Citation: the method predates its microbiome use
LODO is not a microbiome method, and citing a microbiome paper for it would repeat exactly the mistake
this batch is trying to avoid.
- Riester et al. (2014), J Natl Cancer Inst, 10.1093/jnci/dju048 — the earliest identified use of
the term and the procedure: "for each of the six datasets, we trained a prediction model using the
remaining five datasets only and then stratified the patients of the dataset not used for training",
described as assessing cross-study consistency and explicitly noted as not subject to optimistic
bias. Ovarian cancer gene expression, not microbiome.
- Pasolli et al. (2016), 10.1371/journal.pcbi.1004977 — the first application in microbiome data,
as lodoCSV. This is where the protocol's worked examples come from, but it is not the method's origin.
Do not confuse this with leave-one-dataset-in. Bernau et al., "Cross-study validation for the
assessment of prediction algorithms" (Bioinformatics 2014), is sometimes reached for here and is
about the opposite direction: train on one dataset, validate on the others. That is a different
question — how well does a model built from a single study travel? — and deserves its own protocol
another time. It is not this one, and it is not the citation for this one.
Stating the distinction in ## Notes is worth the two sentences: the two schemes are easy to conflate
by name, and they answer different questions about generalization.
Source material
- Riester et al. 2014 and Pasolli et al. 2016, above
waldronlab/curatedMetagenomicDataAnalyses — cMD3_paper_analyses/ML/ml_tests_on_diseases_rf.py, figure4_complete_ml.py, all_command_lines_py.sh (SECTION 3), and the external metaml dependency- Applied again in: 10.1038/s41467-025-66888-1
Scope
In: the LODO split definition; the rule that no information from the held-out dataset may enter training — including preprocessing, which rules out fitting transformations, filters or feature selection on the pooled data before splitting, and is the most common way LODO is accidentally invalidated; the minimum dataset size to serve as a valid held-out fold; how per-fold performance is summarized across folds, and why a simple mean can mislead when fold sizes differ; the comparison against within-dataset cross-validation as a leakage diagnostic.
Out: the classifier and its hyperparameters; feature importance.
Frontmatter starting point
type: "atomic"
category: "Machine Learning"
citation: "10.1093/jnci/dju048" # Riester et al. 2014 — earliest identified LODO use
tags: [cross-validation, lodo, benchmarking, generalization, batch-effects]
A clean case of the citation rule working, and a warning about how easy it is to stop one step short.
Pasolli 2016 looked like the origin until the question was pushed further back — it is the first
microbiome use, which is a different claim. Record that distinction in ## Notes so the next reader
does not have to rediscover it.
Acceptance criteria
Cite the method's origin, not its users
PROTOCOL_STANDARD.md is explicit: an atomic protocol carries "strictly 1 citation... corresponding
to the primary literature where the method was originally published." Find the paper that proposed
the method. Do not cite a paper that merely applied it — including the BugSigDB and curatedMetagenomicData
papers, which are the source of the analysis these protocols were extracted from but almost never the
source of the method.
Tracing a method back to its first publication is real work, and it is part of the task, not a
formality. Three things to expect:
- Candidate DOIs in this issue are leads, not answers. Anything marked VERIFY has not been checked.
- Some methods predate modern citation practice or have no single identifiable origin. If that is
genuinely the case, say so in the pull request rather than reaching for a convenient recent paper.
Raise it as an issue in waldronlab/agent-protocol-standard — the standard may need a way to express
"classical method, no primary source".
- If you cannot name one paper that proposed everything the protocol does, it is more than one
protocol. That test has now split four protocols out of this batch: enrichment into three methods,
filtering from transformation, LODO from random forest, and PERMANOVA from ANOSIM.
Where the lab's own paper genuinely did propose the method — the oral-to-gut score, and LODO
cross-validation in Pasolli et al. 2016 — citing it is correct. That is the exception, not the pattern.
Before you start
Read CONTRIBUTING.md and
PROTOCOL_STANDARD.md.
The format is defined in the standard repo, not this one. Protocols are prose, not code: they say what to do
and why, precisely enough that two people — or two agents, in two languages — get the same answer. The existing
independent-filtering-variance
protocol is the model to imitate for tone and level of detail.
Validate locally before opening the PR:
git clone https://github.com/waldronlab/agent-protocol-standard.git
Rscript agent-protocol-standard/scripts/validate-protocol.R protocols
Tier: D (cMD paper) · Type:
atomic· Category: Machine LearningWhat
Evaluate a classifier by holding out one entire dataset at a time — training on all remaining datasets, testing on the held-out one — and reporting performance per held-out dataset. The classifier is an input to this protocol, not part of it.
Why it matters
Split from the random forest protocol, per review: LODO applies to any classifier, and random forests are used without LODO. Keeping them separate is what makes the useful half reusable — anyone benchmarking a new model on multi-cohort data needs this validation scheme and should not have to adopt someone else's classifier to get it.
LODO is the honest evaluation for multi-cohort microbiome data. Ordinary k-fold cross-validation lets batch and study effects leak between train and test folds, inflating AUROC — sometimes dramatically — because the model can learn the study rather than the phenotype. LODO measures what people actually want to know: will this transfer to a cohort I have not seen?
Writing it down is partly a methods-standards contribution. It gives anyone benchmarking on curatedMetagenomicData a citable specification of the right cross-validation scheme, independent of what they are benchmarking.
Citation: the method predates its microbiome use
LODO is not a microbiome method, and citing a microbiome paper for it would repeat exactly the mistake
this batch is trying to avoid.
the term and the procedure: "for each of the six datasets, we trained a prediction model using the
remaining five datasets only and then stratified the patients of the dataset not used for training",
described as assessing cross-study consistency and explicitly noted as not subject to optimistic
bias. Ovarian cancer gene expression, not microbiome.
as
lodoCSV. This is where the protocol's worked examples come from, but it is not the method's origin.Do not confuse this with leave-one-dataset-in. Bernau et al., "Cross-study validation for the
assessment of prediction algorithms" (Bioinformatics 2014), is sometimes reached for here and is
about the opposite direction: train on one dataset, validate on the others. That is a different
question — how well does a model built from a single study travel? — and deserves its own protocol
another time. It is not this one, and it is not the citation for this one.
Stating the distinction in
## Notesis worth the two sentences: the two schemes are easy to conflateby name, and they answer different questions about generalization.
Source material
waldronlab/curatedMetagenomicDataAnalyses—cMD3_paper_analyses/ML/ml_tests_on_diseases_rf.py,figure4_complete_ml.py,all_command_lines_py.sh(SECTION 3), and the externalmetamldependencyScope
In: the LODO split definition; the rule that no information from the held-out dataset may enter training — including preprocessing, which rules out fitting transformations, filters or feature selection on the pooled data before splitting, and is the most common way LODO is accidentally invalidated; the minimum dataset size to serve as a valid held-out fold; how per-fold performance is summarized across folds, and why a simple mean can mislead when fold sizes differ; the comparison against within-dataset cross-validation as a leakage diagnostic.
Out: the classifier and its hyperparameters; feature importance.
Frontmatter starting point
A clean case of the citation rule working, and a warning about how easy it is to stop one step short.
Pasolli 2016 looked like the origin until the question was pushed further back — it is the first
microbiome use, which is a different claim. Record that distinction in
## Notesso the next readerdoes not have to rediscover it.
Acceptance criteria
## Notesdistinguishes the method's origin from its first microbiome application, and LODO from leave-one-dataset-inCite the method's origin, not its users
PROTOCOL_STANDARD.mdis explicit: an atomic protocol carries "strictly 1 citation... correspondingto the primary literature where the method was originally published." Find the paper that proposed
the method. Do not cite a paper that merely applied it — including the BugSigDB and curatedMetagenomicData
papers, which are the source of the analysis these protocols were extracted from but almost never the
source of the method.
Tracing a method back to its first publication is real work, and it is part of the task, not a
formality. Three things to expect:
genuinely the case, say so in the pull request rather than reaching for a convenient recent paper.
Raise it as an issue in
waldronlab/agent-protocol-standard— the standard may need a way to express"classical method, no primary source".
protocol. That test has now split four protocols out of this batch: enrichment into three methods,
filtering from transformation, LODO from random forest, and PERMANOVA from ANOSIM.
Where the lab's own paper genuinely did propose the method — the oral-to-gut score, and LODO
cross-validation in Pasolli et al. 2016 — citing it is correct. That is the exception, not the pattern.
Before you start
Read
CONTRIBUTING.mdandPROTOCOL_STANDARD.md.The format is defined in the standard repo, not this one. Protocols are prose, not code: they say what to do
and why, precisely enough that two people — or two agents, in two languages — get the same answer. The existing
independent-filtering-varianceprotocol is the model to imitate for tone and level of detail.
Validate locally before opening the PR: