Repository navigation
feat: MDR targets (yes/no and combination) and design spec - #54
Conversation
Each genome tested in a class gets its combination of resistant classes as a label (target resistant_classes). Eligibility keeps labels with at least min_label_genomes (default 25) training genomes per scope and drops genomes, training or test, with other labels.
Tasks with labels other than Resistant/Susceptible use multinom_reg (glmnet), report MCC and nMCC only, and keep coefficients per label. Adds label to importance, n_labels to performance and prob_predicted to predictions.
Resolves conflicts in fit-models.R between de13896's shared cross-drug fits and the MDR multinomial path: .fitTask() keeps both the reused fit and the label levels, and the empty results tables keep the new MDR columns.
Adds .MULTICLASS_TARGETS and .isMulticlass() next to .MODES. Eligibility and fitting read it instead of matching the MDR combination target by name or inferring it from labels; fitting now errors if a Resistant/Susceptible target has other phenotypes. Results on S. flexneri are unchanged.
|
@amcim @epbrenner #54 (MDR) is ready for review. It's up to date with Two things that touch your PRs: 1. Feature rescoring (#53) and the MDR combination target. The combination target has importance per label (a I have a fix ready if you want it: branch 2. Merge order. #52, #53 and #54 all change Also, #54 adds |
AbhirupaGhosh
left a comment
There was a problem hiding this comment.
I have done my first pass and things look ok to me. No major error or logical problem detected.
|
Overall this looks pretty good @eboyer221 Answer to what you raised Old runMDRmodels() defaulted to split = c(0.8, 0); I compared with split = c(1, 0), as in the earlier #52 comparison. Worth confirming that's the split we want to match.
Your point about PR #53
merge order
|
#52 moved fitModels() into fit-execution.R with checkpoints and resume. The MDR multi-class flag now travels in each task group's job, the empty result schemas keep label, n_labels and prob_predicted, and runModelingWorkflow() passes mdr_min_classes and min_label_genomes again and saves them in its resume settings. S. flexneri still matches old amRml exactly.
Clearer for users, since MDR combinations are the only labels it applies to (review suggestion from @AbhirupaGhosh).
Add the time and location
…asses With one class, the combination labels are that class or Susceptible, so the model repeats the class model. New eligibility rule single_class_labels, which also covers a dataset tested on one class only (where the yes/no MDR target has no model either). Suggested by @AbhirupaGhosh.
|
Merging. Noting what's still open so we can come back to it: Added after review: @AbhirupaGhosh suggested dropping MDR when a dataset has fewer than 2 classes. The yes/no target already had no model there (every genome is Not tested yet: Follow-ups in other PRs:
Not from this PR: |
MDR: multidrug-resistance targets and design spec
This PR adds multidrug resistance (MDR) as a new unit, next to drug and drug class, with two targets, and adds
design/mdr.md, the spec for both.target = "mdr"): a genome is MDR if it is resistant to 2 or more drug classes. It is an ordinary Resistant/Susceptible target, so only the profile changes.target = "resistant_classes"): a port of old amRml's MDR model. Each combination of resistant classes (e.g.MAC_PEN_TMD) is one label of a multinomial glmnet model. On S. flexneri it reproduces old amRml exactly (see Testing).Unlike old MDR, both targets go through the same stages and rules as drugs and classes, and run in the year and country modes, not only the whole dataset.
cross_drugandlodoonly take drug rows, so they leave MDR out.Up to date with
runnerthrough de13896 (cross-drug fit reuse). The merge only needed conflicts infit-models.Rresolved:.fitTask()now keeps both the reused fit and the MDR label levels. Fit reuse only applies tocross_drug, which never includes MDR. After the merge, the S. flexneri comparison below is unchanged.What's in this PR
design/mdr.mdR/registry.R.MULTICLASS_TARGETSand.isMulticlass(): the one place that declares the combination target multi-class, next to.MODES. Eligibility and fitting read itR/orb-profile.RprofileORB()gainsmdr_min_classes(default 2) and the$mdrtable, and addsunit = "mdr"rows for both targets.n_resistantnow counts anything not Susceptible, so a combination counts as resistantR/eligibility.ReligibleScopes()gainsmin_mdr_label_genomes(default 25, oldmin_n) and the per-scope label filter. Scopes gainn_labels,n_dropped,test_n_labels,test_n_droppedR/fit-models.RR/workflow.Rmin_mdr_label_genomesthroughman/fitModels.RdandrunModelingWorkflow.Rdalso pick up theBPPARAMargument from #52, which hadn't been documented yettests/test-fit-models.RHow it works
Yes/no MDR. Each genome gets one status, built from the class phenotypes the profile already computes:
mdr: Resistant in at leastmdr_min_classesclasses. PhenotypeResistant.not_mdr: tested in at leastmdr_min_classesclasses, but Resistant in fewer. PhenotypeSusceptible.undetermined: tested in fewer classes, so MDR can't be ruled out. Listed in$mdrwith no phenotype, so it never enters the model. This matters on real data: 931 of 2,907 E. faecium genomes were tested in only one class.Combination.
_, orSusceptible. That matches amRdata'sresistant_classesfor every genome in S. flexneri, S. argenteus and E. faecium.min_mdr_label_genomestraining genomes and drops every genome, training or test, with another label. On the whole dataset this gives old amRml's genome set exactly. In cross and leave-one-out modes it also drops test genomes whose label the model never trained on. The usual rules then apply, with "phenotype" meaning label: the scope needs 2 labels (single_phenotype), and its rarest label must fill the folds. Its test set needs onlymin_test_genomes..MULTICLASS_TARGETSusesmultinom_reg()with the same engine, grid and MCC selection. Folds are now assigned per label present, not only to Resistant and Susceptible rows; Resistant/Susceptible fold assignment is unchanged (tested).Results columns (shared tables, no separate MDR tables):
importancelabelNA"POS"points to itperformancen_labelspredictionsprob_predictedFor the combination target,
performancehas onlymccandnmcc, as old amRml reported;res_prop,res_prop_testandprob_resistantareNA.Testing
Old vs new, S. flexneri (amRdata main, October 2026; old amRml from #42):
.parquet2MDRMatrix(),min_n = 25) are identical to the new whole-dataset combination matrices: the same 288 genomes and 4 labels, the same features, and the same non-zero values.runMLPipeline()and newfitModels()choose the same penalty (1e-4) and mixture (0.2), test on the same 59 genomes, predict the same label for every one, and score MCC 0.85. All 521 non-zero coefficients match exactly.Other data:
metadata.parquet. The whole-dataset scope is eligible, and it built and fitted end to end on gene features with no changes to matrix building or fitting.PEN, 65 genomes). The new code recordssingle_phenotype. The yes/no target (5 MDR, 68 not MDR) fails the fold check.amRml_inputis still marked as building, so it hasn't been tested.R CMD check: 0 errors. The one warning (
tidyselectnot in Imports, from.buildRecipe()onrunner) and the notes don't come from this PR.Unit tests: 92 tests; all new ones pass. 7 tests in
test-orb-read.Randtest-orb-profile.Ralready fail onrunner(7ba0cca), without this PR. Three of them (two onmissing_features, one on moved ORBs) test checks that the ORB-structure changes commented out; I haven't looked into the other four. Four existing eligibility and profile tests were updated because the new target adds scopes or rows. Their old expectations no longer apply; nothing else changed in them.Questions for reviewers
Undetermined genomes in the combination target. The yes/no target leaves out genomes tested in fewer than 2 classes (
undetermined), but the combination target keeps them, labelled from the one class they were tested in. That's deliberate, to match old amRml exactly. Should the combination target leave them out too? It would change results compared with old amRml.The yes/no gene-feature fits scored MCC = 1.00 on 75 test genomes. I expect that's because the classes behind MDR here (penicillins, trimethoprim, macrolides) have well-known resistance genes in the gene features, but I haven't checked the top features yet.
New scope columns (
n_labels,n_dropped,test_n_labels,test_n_dropped) and results columns (label,n_labels,prob_predicted) should be included in the global rename.Old
runMDRmodels()defaulted tosplit = c(0.8, 0); I compared withsplit = c(1, 0), as in the earlier Fitting and running models through a single entry point #52 comparison. Worth confirming that's the split we want to match.