From bdc9aedb171bd20d025eec12ac4023a420878732 Mon Sep 17 00:00:00 2001 From: Eduard Kerkhoven Date: Wed, 9 Sep 2026 22:00:41 +0200 Subject: [PATCH 1/2] Resolve every function in the MATLAB vs Python "not yet mapped" list All 63 RAVEN functions with no recorded Python counterpart, checked individually against the tracked raven-toolbox source (build-validated, so a wrong name fails the build) and, where the guide pages already answered it, against their own function tables. - 12 are real counterparts under a different name (gapFillFastCore and gapFillSwiftCore both map to fill_gaps_fast_lp, which implements both algorithms behind a method= argument; replaceMets to replace_metabolite; the rest were already answered by a guide page's own function table). - 13 are covered by cobrapy directly, most already stated in a guide page (addMets, addExchangeRxns, getAllowedBounds, getMinimalMedium, and others); parseGrRule, grRuleToString, getGenesFromGrRules and getAllRxnsFromGenes are covered by cobrapy's own GPR/Gene classes, not previously recorded anywhere. - 38 are deliberately absent, each with its own reason: internal helpers already folded into an already-mapped public function (buildEquation, constructS, parseRxnEqu, the three KEGG global-model builders, the sample* rounding/warmup subroutines, writeExcel, cleanSheet), MATLAB-specific plumbing with no Python equivalent needed (parseRAVENargs, printOrange, ravenList, the two notice/warning helpers, parallelWorkersRAVEN, setRavenProgress, checkFileExistence, the two MATLAB-struct-field-order functions and their registry, runRAVENtests), and a handful raven-toolbox genuinely has no equivalent for yet (findPotentialErrors, getTransportRxns, removeBadRxns, reverseRxns, readFasta, runRobustnessAnalysis, runSimpleOptKnock), confirmed by grepping the raven-toolbox source rather than assumed absent. --- scripts/curated_pairs.yml | 164 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 164 insertions(+) diff --git a/scripts/curated_pairs.yml b/scripts/curated_pairs.yml index c1ab863..3aedf4c 100644 --- a/scripts/curated_pairs.yml +++ b/scripts/curated_pairs.yml @@ -75,6 +75,20 @@ aliases: getIndexes: parse_name_comp exportToExcelFormat: export_to_excel + # Gap-filling and analysis + getEssentialRxns: find_task_essential_reactions + deleteUnusedGenes: remove_genes + gapFillFastLP: fill_gaps_fast_lp + gapFillMILP: fill_gaps_kumar_milp + gapFillTopological: analyse_topology + makeFakeBlastStructure: make_ortholog_hits + getModelFromKEGG: build_reference_model + sampleACHR: random_sampling + sampleCHRR: random_sampling + gapFillFastCore: fill_gaps_fast_lp + gapFillSwiftCore: fill_gaps_fast_lp + replaceMets: replace_metabolite + cobrapy: # RAVEN function -> the cobrapy object or function that replaces it. importModel: cobra.io.read_sbml_model @@ -91,8 +105,29 @@ cobrapy: runPhenotypePhasePlane: cobra.flux_analysis.production_envelope modelSummary: cobra.Model.summary setRavenSolver: cobra.Configuration + addMets: cobra.Metabolite + addGenesRaven: cobra.Reaction.gene_reaction_rule + addExchangeRxns: cobra.Model.add_boundary + getAllowedBounds: cobra.flux_analysis.flux_variability_analysis + printModelStats: cobra.Model + addIdentifierPrefix: cobra.io.read_sbml_model + removeIdentifierPrefix: cobra.io.write_sbml_model + getMinimalMedium: cobra.medium.minimal_medium + optimizeProb: cobra.Model.solver + parseGrRule: cobra.core.gene.GPR + grRuleToString: cobra.core.gene.GPR + getGenesFromGrRules: cobra.core.gene.GPR + getAllRxnsFromGenes: cobra.Gene.reactions not_ported: + getINITModel: >- + The original tINIT implementation. RAVEN keeps it for models already + built with it; raven-toolbox, a new implementation with no such + installed base, carries ftINIT alone. + runINIT: >- + The original tINIT implementation. RAVEN keeps it for models already + built with it; raven-toolbox, a new implementation with no such + installed base, carries ftINIT alone. runDynamicFBA: >- Dynamic FBA is covered by maintained Python packages (dfba, reframed, mewpy). ravenCobraWrapper: >- @@ -107,3 +142,132 @@ not_ported: printFluxes: >- No equivalent yet. Use cobra's model.summary(), or the solution's fluxes Series directly. + + # Internal helpers, subsumed into an already-mapped public function + buildEquation: >- + Internal helper for building an equation string from raw metabolites and + coefficients; the equivalent is inside add_reactions_from_equations and + cobra.Reaction.build_reaction_string, both already mapped. + constructS: >- + Internal helper for building a stoichiometric matrix from equation + strings; the equivalent is inside add_reactions_from_equations, already + mapped from addRxns. + parseRxnEqu: >- + Internal helper for extracting metabolite names from equation strings; + the equivalent is inside add_reactions_from_equations, already mapped + from addRxns. + buildGlobalGPR: >- + Internal to assembling the global KEGG reference model; the equivalent + is inside build_reference_model, already mapped from getModelFromKEGG. + buildGlobalKEGGModel: >- + Internal to assembling the global KEGG reference model; the equivalent + is inside build_reference_model, already mapped from getModelFromKEGG. + keggDataVersion: >- + Internal to assembling the global KEGG reference model; the equivalent + is inside build_reference_model, already mapped from getModelFromKEGG. + parseFormulas: >- + Internal formula-to-elemental-composition parser; the equivalent is + inside get_elemental_balance, not exposed separately. + writeExcel: >- + Internal low-level .xlsx writer; the equivalent is inside + export_to_excel, already mapped from exportToExcelFormat. + cleanSheet: >- + Internal to the old Excel-import pipeline (importExcelModel), which + RAVEN 3 removed; raven-toolbox never had an Excel reader either. + sampleChebyshevCenter: >- + Internal rounding-step subroutine of RAVEN's own sampleCHRR; raven-toolbox's + random_sampling (already mapped from sampleACHR/sampleCHRR) handles + rounding and warmup internally without exposing separate functions. + sampleMaxVolEllipse: >- + Internal rounding-step subroutine of RAVEN's own sampleCHRR; raven-toolbox's + random_sampling (already mapped from sampleACHR/sampleCHRR) handles + rounding and warmup internally without exposing separate functions. + sampleWarmupPoints: >- + Internal warmup-point generator for RAVEN's own sampleACHR; + raven-toolbox's random_sampling (already mapped from + sampleACHR/sampleCHRR) handles warmup internally. + + # MATLAB-specific plumbing, with no Python equivalent needed + convertCharArray: >- + MATLAB char-array/string-array normalisation; Python has one string type. + parseRAVENargs: >- + Resolves RAVEN's positional-or-named optional arguments; Python + functions take native keyword arguments directly. + parseYAML: >- + A generic YAML parser for condition files; Python's standard YAML + libraries (PyYAML) cover this directly, with no toolbox wrapper needed. + printOrange: >- + Prints coloured text to the MATLAB Command Window; not applicable + outside the MATLAB GUI. + ravenList: >- + Formats a header and item list for a MATLAB warning()/error() call; + Python's own string formatting covers this directly. + deprecationWarning: >- + Emits RAVEN's own one-time deprecated-function notice; Python's + warnings.warn covers this directly. + legacyMethodNotice: >- + Emits RAVEN's own one-time legacy-method notice; Python's warnings.warn + covers this directly. + parallelWorkersRAVEN: >- + Returns a parfor worker count for the Parallel Computing Toolbox; + raven-toolbox's parallelism is configured directly through + cobra.Configuration().processes. + setRavenProgress: >- + Sets a preference for how RAVEN reports progress in the MATLAB GUI or + console; no equivalent concept in Python. + checkFileExistence: >- + A generic file-existence check; Python's own pathlib/os.path cover this + directly. + checkSolution: >- + Checks whether a Mosek solution is feasible and optimal; Mosek is not a + solver either toolbox supports through this site's documented setup. + splitProbForConditioning: >- + Improves LP numerical conditioning by splitting wide-ratio columns + before solving; cobrapy and the solvers it wraps (through optlang) + handle numerical scaling internally, with no equivalent public option. + permuteModel: >- + Reorders the parallel arrays of a RAVEN MATLAB struct; a cobra.Model has + no comparable field-order concept to reorder. + standardizeModelFieldOrder: >- + Orders the fields of a RAVEN MATLAB struct to a canonical layout; a + cobra.Model has no comparable field-order concept. + ravenModelFields: >- + A registry of RAVEN's own MATLAB struct fields, used internally by + permuteModel and removeReactions; a cobra.Model has no comparable + struct-field concept. + runRAVENtests: >- + Runs RAVEN's own MATLAB unit test suite. raven-toolbox's tests run + through pytest, the standard Python testing workflow, not a toolbox + function. + defaultCompartmentMap: >- + raven-toolbox exposes the same table as a module-level constant, + localization.DEFAULT_COMPARTMENT_MAP, not a function. + + # No raven-toolbox equivalent yet + findPotentialErrors: >- + Detects grRules that are not in disjunctive normal form. cobrapy's own + GPR class represents a parsed rule directly, so raven-toolbox has no + equivalent "detect an invalid legacy string" step. + getTransportRxns: >- + Classifies which reactions move a metabolite between compartments. + raven-toolbox has no equivalent classifier yet. + removeBadRxns: >- + Iteratively removes reactions that let the model make or consume + something from nothing. raven-toolbox has no equivalent yet; see + find_leak_metabolite (mapped from makeSomething/consumeSomething) for + finding the same kind of issue. + reverseRxns: >- + Negates a reaction's stoichiometry and bounds to reverse its direction. + Simple enough to do directly against a cobra.Reaction's own API; no + dedicated function exists. + readFasta: >- + raven-toolbox has no public FASTA reader; the closest relative is + write_fasta (already auto-matched from writeFasta), the read side stays + internal to the functions that need it. + runRobustnessAnalysis: >- + Robustness analysis for one reaction against an objective. + raven-toolbox has no equivalent yet. + runSimpleOptKnock: >- + A simple, brute-force OptKnock implementation. raven-toolbox has no + equivalent; strain-design algorithms like OptKnock are explicitly out of + scope for both toolboxes, see RAVEN 3 and raven-toolbox. From 812559ed32c075faa0092110cbbe8627a33083c6 Mon Sep 17 00:00:00 2001 From: Eduard Kerkhoven Date: Wed, 9 Sep 2026 22:31:24 +0200 Subject: [PATCH 2/2] Split the two long migration/comparison pages, fix install instructions for 3.0.0b1, and small home page fixes - docs/raven3-migration.md: at over 500 lines, this page mixed an orientation checklist with twelve detail sections. Kept the checklist and an "if you read nothing else" box on the landing page, moved the rest into four grouped pages under docs/raven3-migration/: what moved/was removed/was renamed, the full backward-incompatible punch list, file formats and reconstruction pipelines, and new functionality plus dependencies and error handling. Every `[section]` cross-reference, including the ones from docs/guide/io.md and docs/tutorials/index.md into specific sections, now points at the page the content actually moved to. - docs/raven3-vs-raven-toolbox.md: same problem at a smaller scale. Kept the orientation content (which one to use, naming, solvers) on the landing page, moved "what each has" and "same function, different answer" into docs/raven3-vs-raven-toolbox/. Retitled from "RAVEN and raven-toolbox" to "RAVEN vs. raven-toolbox". - docs/installation/raven.md: none of the three install methods actually installs the 3.0.0b1 pre-release this site documents. The Add-Ons manager and the GitHub releases page both only ever offer the latest stable release (RAVEN 2.x); a plain `git clone` tracks `main`, also RAVEN 2.x. Added a warning and pointed at what actually works: a direct archive download of the `3.0.0b1` tag (verified it resolves), and `git clone` plus an explicit checkout of that tag. Fixed the Upgrade tab's `git pull origin main` the same way; a tag doesn't move, so "upgrading" past it means switching to the `develop3` branch, not pulling the tag. - docs/yaml-format.md: moved from under Parameter tuning to a new "Start" section under Guide (renamed from "Installation", now installation plus this page), and retitled from "RAVEN / cobrapy YAML model format" to "YAML file format". - docs/index.md: bolded "MATLAB" and "Python" in the tagline; split the MATLAB (Add-Ons) tab's two instructions onto separate lines, they were running together as one sentence. - docs/conf.py: the navbar had one GitHub icon, linking to the MATLAB repo only; added a second, linking to raven-toolbox. --- docs/conf.py | 15 +- docs/guide/index.md | 3 +- docs/guide/io.md | 4 +- docs/index.md | 5 +- docs/installation/raven.md | 41 +- docs/parameter-tuning/index.md | 7 - docs/raven3-migration.md | 475 ++---------------- .../raven3-migration/backward-incompatible.md | 124 +++++ .../formats-and-reconstruction.md | 122 +++++ docs/raven3-migration/new-and-environment.md | 76 +++ .../raven3-migration/structure-and-renames.md | 113 +++++ docs/raven3-vs-raven-toolbox.md | 169 +------ .../same-function-different-answer.md | 84 ++++ docs/raven3-vs-raven-toolbox/what-each-has.md | 75 +++ docs/tutorials/index.md | 4 +- docs/yaml-format.md | 2 +- 16 files changed, 691 insertions(+), 628 deletions(-) create mode 100644 docs/raven3-migration/backward-incompatible.md create mode 100644 docs/raven3-migration/formats-and-reconstruction.md create mode 100644 docs/raven3-migration/new-and-environment.md create mode 100644 docs/raven3-migration/structure-and-renames.md create mode 100644 docs/raven3-vs-raven-toolbox/same-function-different-answer.md create mode 100644 docs/raven3-vs-raven-toolbox/what-each-has.md diff --git a/docs/conf.py b/docs/conf.py index a4278b2..eac4147 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -57,7 +57,20 @@ # clicked, rather than the whole tree opening at once. "collapse_navigation": True, "show_toc_level": 2, - "github_url": "https://github.com/SysBioChalmers/RAVEN", + "icon_links": [ + { + "name": "RAVEN (MATLAB) on GitHub", + "url": "https://github.com/SysBioChalmers/RAVEN", + "icon": "fa-brands fa-github", + "type": "fontawesome", + }, + { + "name": "raven-toolbox (Python) on GitHub", + "url": "https://github.com/SysBioChalmers/raven-toolbox", + "icon": "fa-brands fa-github", + "type": "fontawesome", + }, + ], "use_edit_page_button": False, "navbar_end": ["theme-switcher", "navbar-icon-links"], # The wordmark already carries the "RAVEN" text, so no separate navbar diff --git a/docs/guide/index.md b/docs/guide/index.md index 9e0d501..c81e296 100644 --- a/docs/guide/index.md +++ b/docs/guide/index.md @@ -126,9 +126,10 @@ not yet at parity between the two toolboxes. ```{toctree} :hidden: -:caption: Installation +:caption: Start ../installation/index +../yaml-format ``` ```{toctree} diff --git a/docs/guide/io.md b/docs/guide/io.md index 1db90a7..07bcadc 100644 --- a/docs/guide/io.md +++ b/docs/guide/io.md @@ -56,7 +56,7 @@ makes those annotations unusable as a key for matching metabolites across models. `importModel` accepts SBML Level 3 Version 1 with FBC version 2 and errors -on anything older; see the [RAVEN 3 migration guide](../raven3-migration.md#sbml-io) +on anything older; see the [RAVEN 3 migration guide](../raven3-migration/formats-and-reconstruction.md#sbml-io) if a file is rejected. ::: :::{tab-item} ๐Ÿ Python @@ -233,7 +233,7 @@ RXNS, METS, COMPS, GENES, MODEL `exportToExcelFormat` writes `.xlsx` and errors on any other extension, so a bare directory path is rejected rather than interpreted. The -[RAVEN 3 migration guide](../raven3-migration.md#excel-io) covers scripts +[RAVEN 3 migration guide](../raven3-migration/formats-and-reconstruction.md#excel-io) covers scripts that pass one. There is no matching importer: `curateModelFromTables` reads a curated spreadsheet back by applying tabular edits to an existing model, not by building one from scratch. diff --git a/docs/index.md b/docs/index.md index 8e30a55..4478b37 100644 --- a/docs/index.md +++ b/docs/index.md @@ -6,7 +6,7 @@ **Reconstruction, Analysis and Visualization of Metabolic Networks** A toolkit for building, curating, and simulating genome-scale metabolic -models, available as a MATLAB toolbox and a Python package built on cobrapy. +models, available as a **MATLAB** toolbox and a **Python** package built on cobrapy. ## Install @@ -20,6 +20,7 @@ pip install --pre raven-toolbox :::{tab-item} MATLAB (Add-Ons) Home โ†’ Add-Ons โ†’ Get Add-Ons โ†’ search "RAVEN Toolbox". + Then run `checkRaven` to verify. ::: :::{tab-item} Python (git) @@ -143,7 +144,7 @@ Complete function reference for both MATLAB and Python. Set up RAVEN in MATLAB or raven-toolbox in Python with a solver. ::: -:::{grid-item-card} RAVEN and raven-toolbox +:::{grid-item-card} RAVEN vs. raven-toolbox :link: raven3-vs-raven-toolbox :link-type: doc diff --git a/docs/installation/raven.md b/docs/installation/raven.md index d425ca2..eaad891 100644 --- a/docs/installation/raven.md +++ b/docs/installation/raven.md @@ -73,16 +73,25 @@ setRavenSolver('cobra'); `ravenCobraWrapper` converts a model between the RAVEN struct and the COBRA Toolbox structure; see -[RAVEN 3 and raven-toolbox](../raven3-vs-raven-toolbox.md#solvers). +[RAVEN vs. raven-toolbox](../raven3-vs-raven-toolbox.md#solvers). --- ## Install +:::{warning} Getting 3.0.0b1 specifically +This site documents RAVEN **3.0.0b1**, a pre-release. Both the **Add-Ons +manager** and the **Release download**'s normal releases page only offer the +latest stable release (RAVEN 2.x), not 3.0.0b1: neither method can install +what this site documents. Use the **Clone with git** tab, or the direct +archive link in the **Release download** tab below. +::: + ::::{tab-set} :::{tab-item} {octicon}`plug;1em` Add-Ons manager -Installs from within MATLAB, with no separate download. +Installs from within MATLAB, with no separate download. **Installs the +latest stable release, not 3.0.0b1**; see the warning above. 1. Open the **Home** tab and click **Add-Ons โ†’ Get Add-Ons**. 2. Search for **RAVEN Toolbox** and click **Add โ†’ Add to MATLAB**. @@ -93,20 +102,29 @@ If MATLAB does not pick up the toolbox after step 2, run ::: :::{tab-item} {octicon}`download;1em` Release download -Good for offline or managed environments. +Good for offline or managed environments. **The +[RAVEN releases page](https://github.com/SysBioChalmers/RAVEN/releases) +itself only lists stable releases**; 3.0.0b1 is a git tag with no packaged +release, so download it directly instead: -1. Download the latest ZIP from the - [RAVEN releases page](https://github.com/SysBioChalmers/RAVEN/releases). +1. Download + [the 3.0.0b1 archive](https://github.com/SysBioChalmers/RAVEN/archive/refs/tags/3.0.0b1.zip) + directly (this works without git). 2. Extract it to a location of your choice. 3. In MATLAB, add the RAVEN folder to the path (`pathtool`), then [verify](#verify). ::: :::{tab-item} {octicon}`git-branch;1em` Clone with git -Tracks the development branch, and upgrades with a single command. +Clones the repository, then checks out the `3.0.0b1` tag specifically: a +plain `git clone` alone tracks the `main` branch (stable RAVEN 2), not this +site's pre-release. ```bash git clone --depth=1 https://github.com/SysBioChalmers/RAVEN.git +cd RAVEN +git fetch --depth=1 origin tag 3.0.0b1 +git checkout 3.0.0b1 ``` Add the folder to the MATLAB path and [verify](#verify). @@ -185,11 +203,18 @@ release, and run `checkRaven`. ::: :::{tab-item} {octicon}`git-branch;1em` Clone with git +The `3.0.0b1` tag does not move, so there is nothing to pull while staying on +it. To move past this site's documented snapshot onto the actively-developed +branch: + ```bash -git pull origin main +git checkout develop3 +git pull origin develop3 ``` -Then run `checkRaven`. +`develop3` can be ahead of what this page describes; see +[RAVEN 2 to RAVEN 3](../raven3-migration.md) for what changed and when this +document was last checked against it. Then run `checkRaven`. ::: :::: diff --git a/docs/parameter-tuning/index.md b/docs/parameter-tuning/index.md index 098fce1..768fdd8 100644 --- a/docs/parameter-tuning/index.md +++ b/docs/parameter-tuning/index.md @@ -100,18 +100,11 @@ cross-toolbox to-do list. |---|---| | [Parameter benchmarks](benchmarks.md) | `fseof`, `remove_genes`, `connect_blocked_reactions`, `fill_gaps_*`, `check_tasks`, `find_task_essential_reactions` | -## Model file format - -[YAML model format](../yaml-format.md): the format cobrapy, raven-toolbox and -RAVEN MATLAB all read and write, for moving a model file between any of them -or into version control. - ```{toctree} :hidden: ../tuned-parameters benchmarks -../yaml-format ``` ```{toctree} diff --git a/docs/raven3-migration.md b/docs/raven3-migration.md index 4e359a7..bc1fb28 100644 --- a/docs/raven3-migration.md +++ b/docs/raven3-migration.md @@ -8,8 +8,8 @@ offers to fetch any missing on-demand binaries/data, and its checks catch several of the changes below before your scripts do. Beyond that, the two changes most likely to break silently rather than error are homology/KEGG reconstruction returning a different draft model from the same call -([ยง7](#kegg-reconstruction)), and WoLF PSORT-based localization scores having -no in-toolbox migration path ([ยง3](#backward-incompatible-changes)). +([ยง7](raven3-migration/formats-and-reconstruction.md#kegg-reconstruction)), and WoLF PSORT-based localization scores having +no in-toolbox migration path ([ยง3](raven3-migration/backward-incompatible.md#backward-incompatible-changes)). ::: This guide describes what changed between RAVEN 2 and RAVEN 3. It is written @@ -36,13 +36,13 @@ mostly the same. :::{note} This is the MATLAB-to-MATLAB axis This page is about RAVEN 2 โ†’ RAVEN 3, both MATLAB. How the Python package differs from the MATLAB one is a separate question, answered in -[RAVEN and raven-toolbox](raven3-vs-raven-toolbox.md). +[RAVEN vs. raven-toolbox](raven3-vs-raven-toolbox.md). ::: Start with the [upgrade checklist](#upgrade-checklist) below, then read -[Backward-incompatible changes](#backward-incompatible-changes); it lists -everything that can break. The sections after it go into the detail behind -each item. +[Backward-incompatible changes](raven3-migration/backward-incompatible.md#backward-incompatible-changes); it lists +everything that can break. The detail pages after it go into the detail +behind each item. --- @@ -52,466 +52,47 @@ each item. 1. Update to RAVEN 3.0.0 (or later) and run `checkRaven` (replaces `checkInstallation`); it will offer to fetch any missing on-demand binaries/data (ยง11). -2. Search your scripts for the functions listed in [ยง4](#removed-functionality) - (removed) and [ยง5](#renamed-merged-and-consolidated-functions) +2. Search your scripts for the functions listed in [ยง4](raven3-migration/structure-and-renames.md#removed-functionality) + (removed) and [ยง5](raven3-migration/structure-and-renames.md#renamed-merged-and-consolidated-functions) (renamed/merged) and update or remove those calls. 3. If you use `getModelFromKEGG`/`getKEGGModelForOrganism` or `getModelFromHomology`: expect a different draft reconstruction even from an unmodified call: default cutoffs, the KEGG data source, and the homology - best-hit criterion all changed ([ยง7](#kegg-reconstruction)). Rebuild and + best-hit criterion all changed ([ยง7](raven3-migration/formats-and-reconstruction.md#kegg-reconstruction)). Rebuild and re-validate any KEGG- or homology-derived draft models. 4. If you use `.xlsx` curation templates or `importExcelModel`: convert your curation data to the `.tsv` format `curateModelFromTables` expects; there - is no drop-in Excel importer ([ยง6](#excel-io)). + is no drop-in Excel importer ([ยง6](raven3-migration/formats-and-reconstruction.md#excel-io)). 5. If you use WoLF PSORT-based localization scores: switch to one of the modern predictors (DeepLoc2, MULocDeep, COMPARTMENTS, UniProt); there is no - in-toolbox migration path ([ยง10](#new-functionality)). + in-toolbox migration path ([ยง10](raven3-migration/new-and-environment.md#new-functionality)). 6. If your workflows include metabolic-task gap-filling (`fitTasks`, `checkTasks`, `ftINIT`): re-run and check task pass/fail results; a fixed bug means a previously "passing" task can now correctly fail - ([ยง9](#ftinit)). + ([ยง9](raven3-migration/new-and-environment.md#ftinit)). 7. If your model files are under version control (YAML/SBML): expect diffs on the next export even with no content changes (quoting style, EC-code - annotation, ฮ”G fields; [ยง6](#file-formats-and-model-io)). + annotation, ฮ”G fields; [ยง6](raven3-migration/formats-and-reconstruction.md#file-formats-and-model-io)). 8. Re-run your own test suite / validation pipeline and diff key outputs (model size, growth rate, task results) against your RAVEN 2 baseline before trusting RAVEN 3 results in production. --- -## 1. Repository reorganization +## Detail pages -RAVEN 2's top-level layout (`core/`, `external/`, `hpa/`, `io/`, `legacy/`, -`pathway/`, `plotting/`, `solver/`, `struct_conversion/`, `installation/`, -`utils/`) has been replaced with functional folders that group functions by what -they do rather than by legacy history: - -| RAVEN 3 folder | Contents | Mostly came from (RAVEN 2) | -|---|---|---| -| `INIT/` | `ftINIT`, `INIT/tINIT/` for `getINITModel`/`runINIT`, and HPA/expression parsing + gene scoring (`parseHPA`, `parseHPArna`, `scoreModel`) | `INIT/`, `hpa/` | -| `analysis/` | FBA-adjacent analyses: FVA, sampling, FSEOF, OptKnock, robustness, production envelope, flux comparison/tracing | `core/` | -| `annotation/` | MIRIAM editing, SBO term assignment, ฮ”G CSV I/O | `struct_conversion/` (+ new) | -| `biomass/` | Biomass composition fitting/scaling, GAM | `core/` (+ new) | -| `comparison/` | Multi-model comparison, structured model diffing | `core/` (+ new) | -| `conditions/` | Applying a data-driven growth/media condition to a model | *new folder* | -| `conversion/` | RAVENโ‡„COBRA conversion, identifier prefixing, field ordering | `struct_conversion/` | -| `curation/` | Genome-annotation-driven gene/model curation | *new folder* | -| `gapfilling/` | `fillGaps` and its pluggable algorithms, task fitting, leak-metabolite/exchange checks | `core/` (+ new) | -| `installation/` | Path setup, on-demand binary/data download | `installation/` | -| `io/` | Model import/export (SBML, YAML, Excel), FASTA I/O | `io/` | -| `localization/` | Subcellular localization prediction/scoring, compartment assignment | `external/` (+ new) | -| `manipulation/` | Structural edits: add/remove/change rxns, mets, genes; merge, sort, simplify | `core/` | -| `queries/` | Non-mutating lookups: indexes, exchange/transport reactions, stoichiometry construction, model checking | `core/` | -| `reconstruction/` | KEGG- and homology-based draft reconstruction | `external/` | -| `solver/` | LP/MILP solving | `solver/` | -| `tasks/` | Metabolic task list parsing/checking | `core/` | -| `testing/` | `matlab.unittest` suite (`function_tests/`) | `testing/` | -| `utils/` | Generic helpers, including the new argument-parsing/registry infrastructure | `utils/` | -| `deprecated/` | Thin backward-compatible wrappers for merged/renamed functions | *new folder* | - -Entirely removed (no successor folder): `external/metacyc/` (MetaCyc -reconstruction), `pathway/` and `plotting/` (visualization), `legacy/` -(CellDesigner import and other pre-2.0 code), `struct_conversion/`'s Excel-POI -helpers. - -**The reorganization affects fewer scripts than the change list suggests.** -If your scripts call RAVEN -functions by name (the normal usage pattern, with the whole RAVEN folder tree, -including subfolders, added to the MATLAB path via `addRavenToUserPath` or -`addpath(genpath(...))`), moving a file to a new subfolder does not break the -call: MATLAB resolves functions by name across the whole path, not by a -hard-coded relative location. The reorganization only matters if you (a) -hard-coded a path to a specific `.m` file, (b) called one of the functions that -was actually removed or merged (see below), or (c) shadowed a RAVEN function -name with your own; folder names are irrelevant otherwise. - -The full moved-file mapping (RAVEN 2 path โ†’ RAVEN 3 path) for every relocated -function is available on request / in the repository's git history -(`git diff --stat -M develop...develop3`); it is not reproduced in full here -because it is large and, as the point above explains, rarely something you -need to act on. - ---- - -## 2. Calling convention: positional or named arguments - -RAVEN 2 functions took optional arguments strictly positionally -(`f(model, a, b, c)`), which made it easy to lose track of argument order and -impossible to skip an early optional argument to set a later one without also -specifying all of the ones in between. - -RAVEN 3 introduces `utils/parseRAVENargs.m`, used by 112 functions across the -toolbox. Required leading arguments stay as explicit, positional parameters; -everything optional is collected into `varargin` and can be supplied three ways, -interchangeably: - -```matlab -removeReactions(model, rxnList, true, true, true) % positional (RAVEN 2 style, still works) -removeReactions(model, rxnList, "removeUnusedGenes", true) % named -removeReactions(model, rxnList, true, "removeUnusedComps", true) % hybrid -``` - -The rule: parsing scans for the first argument that is a string matching a known -parameter name. Everything before that point is assigned positionally, in the -order the function's `parseRAVENargs` specification declares; everything from -that point on must be valid name-value pairs. - -**This means old positional calls keep working, if the function's new -parameter order still matches the old positional order.** In every function -spot-checked for this guide (`importModel`, `exportModel`, -`exportToExcelFormat`, `removeReactions`, `ftINIT`, `fillGaps`, `fitTasks`, -`getModelFromHomology`, `addIdentifierPrefix`/`removeIdentifierPrefix`, and -`exportForGit`), new optional parameters were appended at the end of the list, -so old positional calls keep working unchanged. - -One caveat from the parser's own documentation: if a positional argument you're -passing is itself a string that happens to match one of the function's parameter -names, use the explicit name-value form for that argument to avoid ambiguity. - ---- - -(backward-incompatible-changes)= -## 3. Backward-incompatible changes - -This is the consolidated list of everything a RAVEN 2 script could plausibly -break on, or silently produce different output from, when run against RAVEN 3. -Items are grouped by how they manifest. - -### Will error / undefined function - -| RAVEN 2 usage | What happens in RAVEN 3 | +| | | |---|---| -| `importExcelModel(...)` | Removed. Use `curation/curateModelFromTables.m` (different input format: `.tsv`, not `.xlsx`; see [ยง6](#excel-io)) or convert your template to YAML/SBML and use `importModel`. | -| `importModel(...)` on SBML Level < 3, Level 3 without FBC v2, or any non-FBC SBML | Errors. Only SBML **Level 3 Version 1, FBC package version 2** is accepted (see [ยง6](#sbml-io)). | -| `exportToExcelFormat(model, fileName)` where `fileName` is a bare path (no `.xlsx`) | No longer falls back to tab-delimited text export; errors with "only export to xlsx format is supported". | -| `exportToTabDelimited(...)` | Removed outright (was a dead-code path anyway). | -| `getWoLFScores(...)` / `parseScores(..., 'wolf', ...)` | WoLF PSORT support is fully removed. `parseScores` now errors on `'predictor','wolf'` with a message pointing to `'deeploc'`, `'mulocdeep'`, `'compartments'`, or `'uniprot'`. **No in-toolbox migration path for existing WoLF PSORT output files.** | -| `qMOMA(...)`, `solveQP(...)` | Removed (RAVEN's last `quadprog`/Optimization Toolbox dependency). `findGeneDeletions`'s `analysisType`/`refModel`/`oeFactor` arguments (MOMA-based over-expression analysis) were removed with them; only FBA-based single/double gene deletion (`sgd`/`dgd`) remains. | -| `checkRxn(...)` | Removed, no direct successor. For debugging why a reaction can't carry flux, use `gapfilling/findLeakMetabolite.m` / `gapfilling/canExchange.m`, or `gapfilling/checkProduction.m`. | -| `dispEM(...)`, `followFluxes(...)`, `followChanged(...)`, `printModel(...)`, `mapCompartments(...)`, `getExpressionStructure(...)`, `getMetsInComp(...)`, `parallelPoolRAVEN(...)` | All removed (see [ยง4](#removed-functionality) for what replaces each, where anything does). | -| Anything under `external/metacyc/`, `pathway/`, `plotting/`, `legacy/` | Entire subsystems removed: MetaCyc-based reconstruction and all visualization/plotting functionality (`drawMap`, `drawPathway`, `markPathwayWithFluxes`, `setOmicDataToRxns`, etc.). No in-toolbox replacement; use COBRA Toolbox or Escher for visualization. | -| `getGenesFromKEGG`, `getMetsFromKEGG`, `getRxnsFromKEGG`, `constructMultiFasta`, `getWSLpath`, `getBlastFromExcel` | Removed along with the old local-KEGG-FTP-dump workflow (see [ยง7](#kegg-reconstruction)). | -| `getToolboxVersion(...)` | Removed as a standalone function; folded into an internal helper of `exportForGit`. | - -### Will silently produce different results - -Same call, same input model, different output. Grouped by function; see the -linked section for the full explanation of each. - -| Function | New behavior (default, unless noted) | -|---|---| -| `getModelFromHomology`, `strictness=3` | Best-hit tie-break now uses **bitscore** instead of E-value (`"scoreBy","evalue"` restores the old criterion); [ยง7](#kegg-reconstruction) | -| `getModelFromHomology`, `minLen` | Default lowered from `200` to `100`; [ยง7](#kegg-reconstruction) | -| `getModelFromHomology`, GPR merging | Gene substitution is exact-token-based instead of regex/`strrep`-based; [ยง7](#kegg-reconstruction) | -| `getKEGGModelForOrganism`, `cutOff` | Default lowered from `10^-50` to `10^-30`; [ยง7](#kegg-reconstruction) | -| `getKEGGModelForOrganism`, `minScoreRatioG` | Default raised from `0.8` to `0.9`; [ยง7](#kegg-reconstruction) | -| `removeGenes`, `expandModel`, `findPotentialErrors`, `addRxns` | grRules are now parsed correctly (exact gene-token matching, structure-aware isozyme/complex handling) instead of via substring/regex matching; [ยง8](#gpr-parsing) | -| `changeGrRules(..., 'replace', false)` onto a gene-less reaction | Produces a valid grRule instead of an unparseable one; [ยง8](#gpr-parsing) | -| `simplifyModel` (used by task-based gap-filling) | No longer drops a task's boundary constraint on a metabolite with no reactions: a task that used to silently pass may now correctly fail; [ยง9](#ftinit) | -| `ftINITInternalAlg` | `allowExcretion` constraint sign corrected | -| `ravenCobraWrapper` | Gene-ID-to-MIRIAM matching no longer breaks on gene IDs containing regex metacharacters; [ยง6](#raven-cobra-interoperability) | -| `permuteModel`, `randomSampling`, `optimizeProb`, `getFluxZ`, `mergeCompartments`, `convertToIrrev`, `replaceMets`, `setExchangeBounds`, `fitParameters`, `guessComposition` | Each had a pre-existing bug fixed; output differs only for models/inputs that triggered it | - -None of these are API changes; the same call with the same arguments now -returns a different, corrected result. - -### Will warn, but keep working - -- **Error/warning identifiers changed.** `dispEM` (RAVEN 2's universal - error/warning function) always raised errors with an **empty** identifier - (`MException('', ...)`), so no RAVEN 2 script could have legitimately caught - on `err.identifier`. RAVEN 3 replaces `dispEM` with native `error()`/ - `warning()` calls, most (not all) carrying real identifiers, - predominantly `RAVEN:badInput`, plus a handful of specific ones - (`RAVEN:modelError`, `RAVEN:badGrRule`, `RAVEN:sampling`, `RAVEN:infeasible`, - `RAVEN:grRuleTooComplex`). Message text is largely unchanged (many messages - are byte-identical to RAVEN 2's), so `try/catch` blocks that pattern-match on - message text keep working; scripts can newly (optionally) match on - `err.identifier` instead. -- **`getINITModel`/`runINIT` now emit a one-time "legacy" notice** pointing at - `ftINIT`, silenceable with `warning('off','RAVEN:legacyMethod')`. This is - explicitly *not* a deprecation; RAVEN's own documentation states these - functions "remain supported" indefinitely, to reproduce previously published - Human-GEM-derived models. See [ยง9](#ftinit). -- **`makeSomething`, `consumeSomething`, `canProduce`, `canConsume`** now live in - a new `deprecated/` folder as thin wrappers around `findLeakMetabolite`/ - `canExchange`, and emit a one-time `RAVEN:deprecated` warning stating they - "will be removed in the next major release." Unlike the legacy-notice - functions above, these are on an actual removal track; migrate call sites - before the next major release. See [ยง9](#ftinit). -- **`checkInstallation` is now `checkRaven`**, moved from `installation/` to the - repository root, with an unchanged signature - (`[currVer, installType] = checkRaven(developMode, checkBinaries)`). Existing - calls do **not** break: `checkInstallation` survives as a forwarding wrapper - that passes every argument and output straight through, after emitting a - `RAVEN:deprecated` warning naming `checkRaven`. It is on the same removal - track as the `deprecated/` functions above, so switch at the next opportunity. - -### Environment / installation changes - -KEGG HMM libraries and external command-line binaries (BLAST+, DIAMOND, HMMER, -cd-hit, MAFFT, WoLF PSORT) are no longer committed to the repository. RAVEN 3 -fetches them on demand from the `raven-data` GitHub release the first time a -function needs one, which means **internet access is required on first use**, -not just at clone/install time. See [ยง11](#toolbox-deps) for what's needed for -an air-gapped install, and the full dependency comparison. - ---- - -(removed-functionality)= -## 4. Removed functionality - -Beyond the items already listed as backward-incompatible above, RAVEN 3 removed -a number of functions and whole subsystems as part of a deliberate effort to -make the codebase smaller. None of these have call sites left in RAVEN 3 itself. - -| Removed | What it did | Replacement / notes | -|---|---|---| -| `pathway/*`, `plotting/*` (16 functions: `drawMap`, `drawPathway`, `colorPathway`, `colorSubsystem`, `markPathwayWithFluxes`, `markPathwayWithExpression`, `setOmicDataToRxns`, `plotAdditionalInfo`, etc.) | Metabolic map/pathway visualization | None in RAVEN core. Use COBRA Toolbox, Escher, or export flux/omics data for external plotting. | -| `external/metacyc/*` (7 functions: `getModelFromMetaCyc`, `getRxnsFromMetaCyc`, `getMetsFromMetaCyc`, `getEnzymesFromMetaCyc`, `addSpontaneousRxns`, `linkMetaCycKEGGRxns`, `combineMetaCycKEGGModels`) | MetaCyc-based draft reconstruction | None; KEGG- and homology-based reconstruction remain the supported paths. | -| `legacy/*` (CellDesigner import, `changeGeneAssoc`, the bundled `xml_toolbox`) | Pre-2.0 compatibility code | None; considered dead. | -| `io/importExcelModel.m`, `io/loadWorkbook.m`, `io/loadSheet.m`, `io/writeSheet.m`, `io/SBMLFromExcel.m`, `io/addJavaPaths.m`, `io/startup.m` | Apache-POI-based Excel model I/O | `io/writeExcel.m` (dependency-free OOXML writer) for export; `curation/curateModelFromTables.m` for tabular curation. No full Excel-template *import* path remains; see [ยง6](#excel-io). | -| `io/exportModelToSIF.m`, `io/exportToTabDelimited.m` | SIF graph export; tab-delimited model export | Dead/orphaned code with no callers; removed outright. | -| `io/getFullPath.m`, `io/getMD5Hash.m`, `io/getToolboxVersion.m` | Path canonicalization; file MD5 hashing; toolbox git-version lookup | Inlined where still needed (e.g. MD5 via Java's `MessageDigest` directly inside `getBlast`), or folded into `exportForGit`. | -| `core/dispEM.m` | Universal error/warning display | Native `warning()`/`error()` + new `utils/ravenList.m` for formatted item lists. See [ยง3](#backward-incompatible-changes). | -| `core/followFluxes.m`, `core/followChanged.m` | Flux-change reporting relative to a reference solution | `queries/printFluxes.m` covers the cutoff-filtering half; nothing replaces the reference-flux comparison directly. | -| `core/printModel.m` | Printed reactions to screen/file | `queries/printModelStats.m` / `queries/printFluxes.m` cover related summary/flux printing; no direct one-line-per-reaction dump remains. | -| `core/getMetsInComp.m` | Returned metabolite indices in a given compartment | Inline it as `model.metComps == compIndex`. | -| `core/mapCompartments.m` | Remapped compartment labels in a localization-score structure | Superseded by the new predictor-based localization workflow and `localization/defaultCompartmentMap.m`. | -| `core/getExpressionStructure.m` | Loaded an expression-experiment structure from an ad hoc Excel format | Superseded by `INIT/parseHPA.m`/`INIT/parseHPArna.m` for the supported expression-data workflow. | -| `core/checkRxn.m` | Per-reaction reactant/product synthesizability debugging | No direct successor; use `gapfilling/findLeakMetabolite.m`, `gapfilling/canExchange.m`, or `gapfilling/checkProduction.m`. | -| `core/parallelPoolRAVEN.m` | Parallel Computing Toolbox pool setup | Renamed/rewritten as `utils/parallelWorkersRAVEN.m`. | -| `external/getWoLFScores.m` | WoLF PSORT score parsing | None; see [ยง3](#backward-incompatible-changes). | -| `external/kegg/{getGenesFromKEGG,getMetsFromKEGG,getRxnsFromKEGG,constructMultiFasta,getWSLpath}.m`, `external/getBlastFromExcel.m` | Local-KEGG-FTP-dump parsing; per-KO multi-FASTA assembly; WSL path translation; Excel-based BLAST import | Superseded by the raven-data-artifact-based KEGG pipeline. See [ยง7](#kegg-reconstruction). | -| `solver/qMOMA.m`, `solver/solveQP.m` | MOMA-based analyses (quadratic programming) | Removed; no in-toolbox QP-based replacement. See [ยง3](#backward-incompatible-changes) for `findGeneDeletions` impact. | -| `testing/unit_tests/*.m` (script-based tests) | Ad hoc test scripts | `testing/function_tests/*.m`, a `matlab.unittest`-class-based suite. Contributor-facing only; `runRAVENtests` usage is unchanged for end users. | - ---- - -(renamed-merged-and-consolidated-functions)= -## 5. Renamed, merged, and consolidated functions - -RAVEN 3 introduces a formal **`deprecated/` folder convention**: functions that -were merged into a more general replacement stay callable under their old name, -as a thin wrapper that emits a one-time `RAVEN:deprecated` warning ("will be -removed in the next major release") and forwards to the new function. - -| Old function(s) | New function | Notes | -|---|---|---| -| `makeSomething(model, ignoreMets, isNames, minNrFluxes, allowExcretion, params, ignoreIntBounds)` | `findLeakMetabolite(model, 'produce', ...)` | Argument order maps 1:1; the wrapper forwards `varargin` unchanged. | -| `consumeSomething(model, ignoreMets, isNames, minNrFluxes, params, ignoreIntBounds)` | `findLeakMetabolite(model, 'consume', ...)` | **Not** a verbatim forward: `consumeSomething`'s old positional order has no `allowExcretion` slot, so the wrapper explicitly remaps arguments by name to avoid corrupting `params`/`ignoreIntBounds`. | -| `canProduce(model, mets)` | `canExchange(model, 'produce', 'mets', mets)` | | -| `canConsume(model, mets)` | `canExchange(model, 'consume', 'mets', mets)` | | -| `compareRxnsGenesMetsComps` | folded into `comparison/compareMultipleModels.m` | Its rxn/gene/met/comp overlap output is now one part of `compareMultipleModels`'s output; no separate function remains. | -| `INIT/scoreComplexModel.m` | `INIT/scoreModel.m` | **Merged, not renamed.** RAVEN 2 had *both* `INIT/scoreComplexModel.m` and `hpa/scoreModel.m`; RAVEN 3 folded them into one `scoreModel`, called with different settings from tINIT than from ftINIT. A RAVEN 2 call to `scoreModel` is unaffected; a call to `scoreComplexModel` has to move to `scoreModel` *and* pass the settings that reproduce the old behaviour (a single operator for both `and`/`or`, `dataPrecedence` `'reaction'`). | -| `INIT/ftINITFillGapsMILP.m` | folded into `solver/getMinNrFluxes.m` | Exposed now via `getMinNrFluxes`'s `resolveTies`/`proveAbsGap` options. | -| `INIT/ftINITFillGapsForAllTasks.m` | folded into `gapfilling/ftINITFillGaps.m` and `fitTasks`'s `'preMerged'` gap-fill mode | | -| `getGenesFromKEGG`/`getMetsFromKEGG`/`getRxnsFromKEGG` | `reconstruction/kegg/readKEGGTable.m` + the raven-data KEGG artifacts | Not a direct signature-compatible rename: the whole data source changed; see [ยง7](#kegg-reconstruction). | -| `checkInstallation` | `checkRaven` | Moved from `installation/checkInstallation.m` to repo-root `checkRaven.m`; identical calling signature. `checkInstallation` remains as a deprecated forwarding wrapper, so old calls keep working with a `RAVEN:deprecated` warning. | - -Separately, `getINITModel` and `runINIT` (now under `INIT/tINIT/`) are marked -**legacy but indefinitely supported** (not on the `deprecated/` removal track) -via `utils/legacyMethodNotice.m`, which fires a one-time, silenceable -`RAVEN:legacyMethod` warning pointing at `ftINIT`. Use `ftINIT` for new work; -keep `getINITModel`/`runINIT` only to reproduce models originally built with -them. - ---- - -(file-formats-and-model-io)= -## 6. File formats and model I/O - -(sbml-io)= -### SBML I/O - -| | RAVEN 2 | RAVEN 3 | -|---|---|---| -| `exportModel` output format | SBML Level 3 Version 1, FBC v2 | unchanged | -| `importModel` accepted input | any SBML level/version; L2-style notes-based gene associations (`GENE_ASSOCIATION`/`PROTEIN_ASSOCIATION`) also read; FBC fields optional | only SBML **Level 3 Version 1, FBC v2**; anything else errors | -| MIRIAM prefixes read on import | `urn:miriam:`, `http://identifiers.org/`, `https://identifiers.org/` | unchanged | - -Older SBML files must be converted first (e.g. re-exported from COBRApy/COBRA -Toolbox) before `importModel` will accept them. - -(excel-io)= -### Excel I/O - -| | RAVEN 2 | RAVEN 3 | -|---|---|---| -| Excel engine | Apache POI (Java), bundled | dependency-free OOXML writer (`io/writeExcel.m`) | -| Full-model `.xlsx` import | `importExcelModel.m` (`RXNS`/`METS`/`COMPS`/`GENES`/`MODEL` sheets โ†’ complete model) | removed, no equivalent importer | -| Nearest replacement for curation | no equivalent | `curation/curateModelFromTables.m`: curates/extends an *existing* model from separate `.tsv` files (mets by name+compartment, reactions by stoichiometry, genes by name), not a from-scratch importer | -| `exportToExcelFormat` on a bare path (no `.xlsx`) | fell back to tab-delimited text export | errors, `.xlsx` only | - -If your pipeline depends on the old Excel curation template, either convert it -to the `.tsv` shape `curateModelFromTables` expects, or build the model via -`importModel`/YAML and use `curateModelFromTables` for incremental edits. - -### YAML I/O - -`readYAMLmodel`/`writeYAMLmodel` are pure-MATLAB in both branches; model YAML -I/O has no Python dependency in RAVEN 2 or RAVEN 3. (`io/parseYAML.m` is an -unrelated, new, pure-MATLAB generic YAML parser used only by -`conditions/applyCondition.m` to read condition files; it doesn't affect -model I/O.) Output format changes, visible as diffs in version-controlled -model files: - -| | RAVEN 2 | RAVEN 3 | -|---|---|---| -| Quote style | single quotes | double quotes | -| Reaction EC codes | top-level `eccodes` field only | also written under `annotation: ec-code` | -| Gene-less model | `genes:` section omitted | `genes: []` written explicitly | -| Empty `gene_reaction_rule` | written as an empty string | omitted | -| `objective_coefficient` on import | ignored | honored | - -(raven-cobra-interoperability)= -### RAVENโ‡„COBRA interoperability - -`conversion/ravenCobraWrapper.m` keeps its signature -(`newModel = ravenCobraWrapper(model)`) unchanged. Two correctness fixes change -output for affected models: a stale-variable bug that could reuse the wrong -MIRIAM data during gene-annotation conversion, and gene IDs containing regex -metacharacters (e.g. `ENSG00000141510.16`) that could be mis-matched during -annotation conversion; both fixed. - ---- - -(kegg-reconstruction)= -## 7. Reconstruction: KEGG and homology pipelines - -### KEGG-based reconstruction (`getModelFromKEGG`, `getKEGGModelForOrganism`) - -| | RAVEN 2 | RAVEN 3 | -|---|---|---| -| Reaction/gene database source | local KEGG FTP dump (license required), parsed by `getGenesFromKEGG`/`getMetsFromKEGG`/`getRxnsFromKEGG` into a cached `keggModel.mat` | prebuilt artifacts (a gene-free reference model plus KO/reaction/organism tables) fetched on demand from the `raven-data` GitHub release, assembled by `buildGlobalKEGGModel`/`buildGlobalGPR`/`readKEGGTable` | -| `getModelFromKEGG`'s `keggPath` argument | required | removed, not needed | -| Ortholog HMM library | one `.hmm` file per KO (self-built via CD-HIT + MAFFT + `hmmbuild`, or downloaded); queried with one `hmmsearch` call per KO, via WSL on Windows | one concatenated per-domain library (`kegg118_eukaryotes.hmm` / `kegg118_prokaryotes.hmm`); one `hmmsearch` call against the whole proteome, no WSL | -| `dataDir` | arbitrary local folder | must be `kegg118_eukaryotes` or `kegg118_prokaryotes` | -| `nSequences`, `seqIdentity` (CD-HIT clustering controls) | configurable | removed, no local clustering step | -| `cutOff` default | `10^-50` | `10^-30` | -| `minScoreRatioG` default | `0.8` | `0.9` | -| Calling convention | positional | positional-or-named (`parseRAVENargs`); old positional order preserved | - -### Homology-based reconstruction (`getModelFromHomology`) - -| | RAVEN 2 | RAVEN 3 | -|---|---|---| -| `strictness` parameter | `1`/`2`/`3` | unchanged | -| Best-hit tie-break at `strictness=3` | E-value | bitscore by default (`"scoreBy","evalue"` restores the old criterion) | -| `minLen` default | `200` | `100` | -| GPR gene substitution when merging homology hits | regex/`strrep` substitution (could corrupt IDs that were substrings of one another, or contained regex metacharacters) | exact-token-based substitution | - -### Shared infrastructure - -External binaries (BLAST+, DIAMOND, HMMER) and the KEGG data artifacts -themselves are fetched on demand rather than bundled with the repository; see -[ยง11](#toolbox-deps). - -**New:** `curation/downloadGenomeData.m`, `curation/getGeneData.m`, and -`curation/processProteinFastaFile.m` give a new on-ramp from a bare NCBI genome -accession (`GCF_`/`GCA_`) straight to a model-ready proteome and gene ID -mapping table. - ---- - -(gpr-parsing)= -## 8. Gene-reaction-rule (GPR) parsing - -RAVEN 3 adds a proper grRule tokenizer/parser: `utils/parseGrRule.m` builds a -parse tree (`type`: `'gene'|'and'|'or'`, with `children`); `utils/grRuleToDNF.m` -converts it to disjunctive normal form (RAVEN's canonical "OR of AND-clauses" -shape for enzyme-complex/isozyme structure); `utils/isDnfGrRule.m` checks -whether a rule is already in that form; `utils/grRuleToString.m` renders a tree -back to a canonical string. These are used internally by four functions, -they are **not** applied automatically to every loaded model; a grRule keeps -its original text unless one of these functions is called on it. - -| Function | RAVEN 2 parsing | RAVEN 3 parsing | -|---|---|---| -| `removeGenes` | substring match on gene ID (`"10"` also matched `"100"`) | exact gene-token match | -| `expandModel` | naive split on `' or '` (could drop an `AND`-ed subunit inside a nested rule) | parse-tree-based isozyme expansion, preserves nested `AND`/`OR` | -| `findPotentialErrors` | string-pattern check for `") and ("` etc. (false positives on valid rules) | parse-tree-based non-DNF detection | -| `addRxns` (new gene IDs from a grRule) | split on the substrings "and"/"or" anywhere (could shred an ID like `"band1"`) | split only on space-bounded `' and '`/`' or '` tokens | -| `comparison/diffModels.m` | (function didn't exist) | grRules compared as logical DNF equality, not string equality | - -The accepted grRule *syntax* itself is unchanged: `' and '`/`' or '` -(case-insensitive) still work exactly as before, and -`manipulation/standardizeGrRules.m` (which runs on import) uses the same -string-splitting logic it always did. - ---- - -(ftinit)= -## 9. Gap-filling and context-specific model extraction - -| | RAVEN 2 | RAVEN 3 | -|---|---|---| -| `fillGaps` algorithm | one built-in MILP connectivity-maximization strategy | same strategy is still the default; `algorithm` option adds `'fastLP'`/`'swiftLP'` (FASTCORE/SWIFTCORE-style LP relaxation), `'gapfillMILP'` (growth-floor MILP with directionality repair), `'topological'` (BFS producibility pre-screen, no model modification) | -| `fillGaps` positional arguments | `allowNetProduction, useModelConstraints, supressWarnings, rxnScores` | unchanged; new options are name-value-only | -| `ftINIT` tie-breaking | depends on solver seed | optional `resolveTies` pins degenerate MILP optima to a deterministic answer | -| `ftINIT` MILP optimality proof | escalating relative-gap search only | optional `proveAbsGap` proves each stage to a fixed absolute gap in one solve | -| `ftINIT` positional arguments | preserved | unchanged; `resolveTies`/`proveAbsGap` are name-value-only, default off | -| `getINITModel`/`runINIT` | supported, no runtime notice | supported, now under `INIT/tINIT/`; one-time "legacy, use `ftINIT`" notice (see [ยง5](#renamed-merged-and-consolidated-functions)) | -| `fitTasks` gap-fill mode | one behavior (equivalent to today's `'merge'`) | `gapFillMode`: `'merge'` (default, same as before) or `'preMerged'` (faster, ftINIT-style) | -| `fitTasks` outputs | `[outModel, addedRxns]` | `[outModel, addedRxns, failedTasks]` | -| Task requiring net production of a metabolite with no reactions | boundary constraint silently dropped before gap-filling; task could incorrectly report as passing | constraint preserved; task correctly reports failing if the reference model can't produce it | - ---- - -(new-functionality)= -## 10. New functionality - -All additive; none of this replaces or changes behavior of existing RAVEN 2 -functions, except where noted. - -| Area | New function(s) | What it adds | -|---|---|---| -| Flux sampling (`analysis/`) | `sampleACHR`, `sampleCHRR`, `sampleMaxVolEllipse`, `sampleChebyshevCenter`, `sampleWarmupPoints` | `randomSampling` becomes a dispatcher (`'method'` option) across RAVEN 2's original random-objective method (now `'randomObjective'`) plus two new near-uniform MCMC samplers: ACHR (hit-and-run) and CHRR (hit-and-run with rounding, better mixing on ill-conditioned polytopes) | -| Flux analysis / QC | `compareFluxes`, `getMinimalMedium`, `traceFluxPath`, `walkFluxes`, `modelSummary` | structured flux-vector diffing; MILP-based minimal-medium search; highest-flux-fraction path tracing between two reactions; interactive flux-neighborhood navigation; model/flux summary printing | -| Model comparison (`comparison/`) | `diffModels` | structured, ID-keyed diff between two models: stoichiometry, bounds, objective, grRule (as logical DNF equality), EC codes, metabolite properties; distinct from the existing overlap-style `compareMultipleModels` | -| Manipulation | `findDuplicateRxns`, `findPotentialErrors` | standalone duplicate-stoichiometry query; parse-tree-based non-DNF grRule detection (see [ยง8](#gpr-parsing)) | -| Genome-based curation (`curation/`, new folder) | `downloadGenomeData`, `getGeneData`, `processProteinFastaFile`, `renameModelGenes`, `curateModelFromTables` | pipeline from an NCBI genome accession to a gene ID mapping table to bulk gene renaming in a model; `curateModelFromTables` bulk-curates mets/rxns/genes from `.tsv` files against an existing model | -| Biomass composition (`biomass/`) | `getBiomassFractions`, `scaleBiomassFraction`, `scaleBiomassPseudoreaction`, `setGAM` | organism-agnostic biomass-composition and growth-associated-maintenance (GAM) calibration; `tutorial/tutorial7.m` demonstrates the workflow | -| Localization (`localization/`) | `assignCompartments`, `defaultCompartmentMap`, `getUniProtScores` | deterministic MILP compartment assignment (alternative to the heuristic `predictLocalization`); `parseScores` gains `'deeploc'`/`'cello'`/`'mulocdeep'`/`'compartments'`/`'uniprot'` predictors (replacing WoLF PSORT; see [ยง3](#backward-incompatible-changes)) | -| Annotation (`annotation/`) | `assignSBOterms`, `loadDeltaGCSV`/`saveDeltaGCSV` | organism-agnostic SBO term assignment; ฮ”G thermodynamic data CSV round-trip | -| Conditions (`conditions/`, new folder) | `applyCondition` | applies a data-driven growth/media condition (a YAML file, not code) to a model in a fixed sequence: exchange bounds, optional cofactor-pseudoreaction edits, optional biomass-stoichiometry deltas, per-reaction bound overrides | -| Internal infrastructure | `ravenModelFields` | central registry of every 1-D model field (entity type, default value), backing `permuteModel`/`removeReactions` internally | - ---- - -(toolbox-deps)= -## 11. Toolbox dependencies and installation - -| | RAVEN 2 | RAVEN 3 | -|---|---|---| -| MATLAB version | no pinned/documented minimum (guidance: avoid very new MATLAB syntax, to accommodate older installs) | CI is tested against **R2024b** | -| Statistics Toolbox | required (`combnk`) | not required (`nchoosek`, base MATLAB) | -| Bioinformatics Toolbox | required (`fastaread`/`fastawrite`) | not required (`io/readFasta.m`/`io/writeFasta.m`) | -| Optimization Toolbox | required for `qMOMA`/`solveQP` | not required (both removed) | -| Excel I/O | bundled Apache POI (Java) | dependency-free OOXML writer | -| LP/MILP solver | required (bundled `glpk`, or Gurobi/SCIP/COBRA) | unchanged, still required | -| External binaries (BLAST+, DIAMOND, HMMER, cd-hit, MAFFT) | committed to the repository, available offline immediately after clone | fetched on demand from the `raven-data` GitHub release on first use (per-platform ZIPs); an offline `*-full`/`*-binaries` release bundle is available for air-gapped installs | -| KEGG HMM libraries / reference data | manual download or local build required | fetched on demand from `raven-data`, cached locally after first use | -| Installation self-check | `installation/checkInstallation.m`; ran the full test suite to populate its pass/fail table | renamed `checkRaven`, moved to the repo root; does fast, targeted checks (a small import/export round-trip, a trivial LP per solver) instead of invoking the full test classes, and proactively offers to download missing binaries | - -Net effect: a bare RAVEN 3 clone is far smaller and installs with fewer MATLAB -toolbox prerequisites. The trade-off: it needs network access the first time -KEGG- or homology-based reconstruction functions actually run. - ---- - -## 12. Error and warning handling - -RAVEN 2's `dispEM` always raised errors with an empty identifier, so no RAVEN 2 -`try/catch` could have matched on `err.identifier`; RAVEN 3's move to native -`error()`/`warning()` with real (mostly `RAVEN:*`) identifiers is additive from -that angle, not breaking. Message text is preserved for most existing checks, -so text-matching `try/catch` blocks are unaffected either way. See -[ยง3](#backward-incompatible-changes) for the details and the new -`utils/ravenList.m` message-formatting helper. - -One related, purely additive change: `queries/checkModelStruct.m`, called *with* -an output argument (`issues = checkModelStruct(model)`), now returns a -structured array of findings (`category`/`target`/`message`) instead of only -throwing or printing, useful for programmatic model QC. The no-output-argument -calling convention (throws/warns directly) is unchanged. +| [ยง1โ€“5. What moved, what's gone, and what's renamed](raven3-migration/structure-and-renames.md) | The folder reorganization, every removed function and subsystem, and every renamed or merged function. | +| [ยง2โ€“3. Backward-incompatible changes](raven3-migration/backward-incompatible.md) | The positional-or-named calling convention, and the full punch list: what will error, what will silently differ, what will warn, and what changed about installation. | +| [ยง6โ€“8. File formats, reconstruction, and GPR parsing](raven3-migration/formats-and-reconstruction.md) | SBML, Excel and YAML I/O; the KEGG and homology reconstruction pipelines; the new grRule parser. | +| [ยง9โ€“12. New functionality, dependencies, and error handling](raven3-migration/new-and-environment.md) | Gap-filling and ftINIT changes, everything purely additive, the toolbox dependency comparison, and error/warning handling. | + +```{toctree} +:hidden: + +raven3-migration/structure-and-renames +raven3-migration/backward-incompatible +raven3-migration/formats-and-reconstruction +raven3-migration/new-and-environment +``` diff --git a/docs/raven3-migration/backward-incompatible.md b/docs/raven3-migration/backward-incompatible.md new file mode 100644 index 0000000..b55bda8 --- /dev/null +++ b/docs/raven3-migration/backward-incompatible.md @@ -0,0 +1,124 @@ +# Backward-incompatible changes + +## 1. Calling convention: positional or named arguments + +RAVEN 2 functions took optional arguments strictly positionally +(`f(model, a, b, c)`), which made it easy to lose track of argument order and +impossible to skip an early optional argument to set a later one without also +specifying all of the ones in between. + +RAVEN 3 introduces `utils/parseRAVENargs.m`, used by 112 functions across the +toolbox. Required leading arguments stay as explicit, positional parameters; +everything optional is collected into `varargin` and can be supplied three ways, +interchangeably: + +```matlab +removeReactions(model, rxnList, true, true, true) % positional (RAVEN 2 style, still works) +removeReactions(model, rxnList, "removeUnusedGenes", true) % named +removeReactions(model, rxnList, true, "removeUnusedComps", true) % hybrid +``` + +The rule: parsing scans for the first argument that is a string matching a known +parameter name. Everything before that point is assigned positionally, in the +order the function's `parseRAVENargs` specification declares; everything from +that point on must be valid name-value pairs. + +**This means old positional calls keep working, if the function's new +parameter order still matches the old positional order.** In every function +spot-checked for this guide (`importModel`, `exportModel`, +`exportToExcelFormat`, `removeReactions`, `ftINIT`, `fillGaps`, `fitTasks`, +`getModelFromHomology`, `addIdentifierPrefix`/`removeIdentifierPrefix`, and +`exportForGit`), new optional parameters were appended at the end of the list, +so old positional calls keep working unchanged. + +One caveat from the parser's own documentation: if a positional argument you're +passing is itself a string that happens to match one of the function's parameter +names, use the explicit name-value form for that argument to avoid ambiguity. + +--- + +(backward-incompatible-changes)= +## 2. Backward-incompatible changes + +This is the consolidated list of everything a RAVEN 2 script could plausibly +break on, or silently produce different output from, when run against RAVEN 3. +Items are grouped by how they manifest. + +### Will error / undefined function + +| RAVEN 2 usage | What happens in RAVEN 3 | +|---|---| +| `importExcelModel(...)` | Removed. Use `curation/curateModelFromTables.m` (different input format: `.tsv`, not `.xlsx`; see [ยง6](../raven3-migration/formats-and-reconstruction.md#excel-io)) or convert your template to YAML/SBML and use `importModel`. | +| `importModel(...)` on SBML Level < 3, Level 3 without FBC v2, or any non-FBC SBML | Errors. Only SBML **Level 3 Version 1, FBC package version 2** is accepted (see [ยง6](../raven3-migration/formats-and-reconstruction.md#sbml-io)). | +| `exportToExcelFormat(model, fileName)` where `fileName` is a bare path (no `.xlsx`) | No longer falls back to tab-delimited text export; errors with "only export to xlsx format is supported". | +| `exportToTabDelimited(...)` | Removed outright (was a dead-code path anyway). | +| `getWoLFScores(...)` / `parseScores(..., 'wolf', ...)` | WoLF PSORT support is fully removed. `parseScores` now errors on `'predictor','wolf'` with a message pointing to `'deeploc'`, `'mulocdeep'`, `'compartments'`, or `'uniprot'`. **No in-toolbox migration path for existing WoLF PSORT output files.** | +| `qMOMA(...)`, `solveQP(...)` | Removed (RAVEN's last `quadprog`/Optimization Toolbox dependency). `findGeneDeletions`'s `analysisType`/`refModel`/`oeFactor` arguments (MOMA-based over-expression analysis) were removed with them; only FBA-based single/double gene deletion (`sgd`/`dgd`) remains. | +| `checkRxn(...)` | Removed, no direct successor. For debugging why a reaction can't carry flux, use `gapfilling/findLeakMetabolite.m` / `gapfilling/canExchange.m`, or `gapfilling/checkProduction.m`. | +| `dispEM(...)`, `followFluxes(...)`, `followChanged(...)`, `printModel(...)`, `mapCompartments(...)`, `getExpressionStructure(...)`, `getMetsInComp(...)`, `parallelPoolRAVEN(...)` | All removed (see [ยง4](../raven3-migration/structure-and-renames.md#removed-functionality) for what replaces each, where anything does). | +| Anything under `external/metacyc/`, `pathway/`, `plotting/`, `legacy/` | Entire subsystems removed: MetaCyc-based reconstruction and all visualization/plotting functionality (`drawMap`, `drawPathway`, `markPathwayWithFluxes`, `setOmicDataToRxns`, etc.). No in-toolbox replacement; use COBRA Toolbox or Escher for visualization. | +| `getGenesFromKEGG`, `getMetsFromKEGG`, `getRxnsFromKEGG`, `constructMultiFasta`, `getWSLpath`, `getBlastFromExcel` | Removed along with the old local-KEGG-FTP-dump workflow (see [ยง7](../raven3-migration/formats-and-reconstruction.md#kegg-reconstruction)). | +| `getToolboxVersion(...)` | Removed as a standalone function; folded into an internal helper of `exportForGit`. | + +### Will silently produce different results + +Same call, same input model, different output. Grouped by function; see the +linked section for the full explanation of each. + +| Function | New behavior (default, unless noted) | +|---|---| +| `getModelFromHomology`, `strictness=3` | Best-hit tie-break now uses **bitscore** instead of E-value (`"scoreBy","evalue"` restores the old criterion); [ยง7](../raven3-migration/formats-and-reconstruction.md#kegg-reconstruction) | +| `getModelFromHomology`, `minLen` | Default lowered from `200` to `100`; [ยง7](../raven3-migration/formats-and-reconstruction.md#kegg-reconstruction) | +| `getModelFromHomology`, GPR merging | Gene substitution is exact-token-based instead of regex/`strrep`-based; [ยง7](../raven3-migration/formats-and-reconstruction.md#kegg-reconstruction) | +| `getKEGGModelForOrganism`, `cutOff` | Default lowered from `10^-50` to `10^-30`; [ยง7](../raven3-migration/formats-and-reconstruction.md#kegg-reconstruction) | +| `getKEGGModelForOrganism`, `minScoreRatioG` | Default raised from `0.8` to `0.9`; [ยง7](../raven3-migration/formats-and-reconstruction.md#kegg-reconstruction) | +| `removeGenes`, `expandModel`, `findPotentialErrors`, `addRxns` | grRules are now parsed correctly (exact gene-token matching, structure-aware isozyme/complex handling) instead of via substring/regex matching; [ยง8](../raven3-migration/formats-and-reconstruction.md#gpr-parsing) | +| `changeGrRules(..., 'replace', false)` onto a gene-less reaction | Produces a valid grRule instead of an unparseable one; [ยง8](../raven3-migration/formats-and-reconstruction.md#gpr-parsing) | +| `simplifyModel` (used by task-based gap-filling) | No longer drops a task's boundary constraint on a metabolite with no reactions: a task that used to silently pass may now correctly fail; [ยง9](../raven3-migration/new-and-environment.md#ftinit) | +| `ftINITInternalAlg` | `allowExcretion` constraint sign corrected | +| `ravenCobraWrapper` | Gene-ID-to-MIRIAM matching no longer breaks on gene IDs containing regex metacharacters; [ยง6](../raven3-migration/formats-and-reconstruction.md#raven-cobra-interoperability) | +| `permuteModel`, `randomSampling`, `optimizeProb`, `getFluxZ`, `mergeCompartments`, `convertToIrrev`, `replaceMets`, `setExchangeBounds`, `fitParameters`, `guessComposition` | Each had a pre-existing bug fixed; output differs only for models/inputs that triggered it | + +None of these are API changes; the same call with the same arguments now +returns a different, corrected result. + +### Will warn, but keep working + +- **Error/warning identifiers changed.** `dispEM` (RAVEN 2's universal + error/warning function) always raised errors with an **empty** identifier + (`MException('', ...)`), so no RAVEN 2 script could have legitimately caught + on `err.identifier`. RAVEN 3 replaces `dispEM` with native `error()`/ + `warning()` calls, most (not all) carrying real identifiers, + predominantly `RAVEN:badInput`, plus a handful of specific ones + (`RAVEN:modelError`, `RAVEN:badGrRule`, `RAVEN:sampling`, `RAVEN:infeasible`, + `RAVEN:grRuleTooComplex`). Message text is largely unchanged (many messages + are byte-identical to RAVEN 2's), so `try/catch` blocks that pattern-match on + message text keep working; scripts can newly (optionally) match on + `err.identifier` instead. +- **`getINITModel`/`runINIT` now emit a one-time "legacy" notice** pointing at + `ftINIT`, silenceable with `warning('off','RAVEN:legacyMethod')`. This is + explicitly *not* a deprecation; RAVEN's own documentation states these + functions "remain supported" indefinitely, to reproduce previously published + Human-GEM-derived models. See [ยง9](../raven3-migration/new-and-environment.md#ftinit). +- **`makeSomething`, `consumeSomething`, `canProduce`, `canConsume`** now live in + a new `deprecated/` folder as thin wrappers around `findLeakMetabolite`/ + `canExchange`, and emit a one-time `RAVEN:deprecated` warning stating they + "will be removed in the next major release." Unlike the legacy-notice + functions above, these are on an actual removal track; migrate call sites + before the next major release. See [ยง9](../raven3-migration/new-and-environment.md#ftinit). +- **`checkInstallation` is now `checkRaven`**, moved from `installation/` to the + repository root, with an unchanged signature + (`[currVer, installType] = checkRaven(developMode, checkBinaries)`). Existing + calls do **not** break: `checkInstallation` survives as a forwarding wrapper + that passes every argument and output straight through, after emitting a + `RAVEN:deprecated` warning naming `checkRaven`. It is on the same removal + track as the `deprecated/` functions above, so switch at the next opportunity. + +### Environment / installation changes + +KEGG HMM libraries and external command-line binaries (BLAST+, DIAMOND, HMMER, +cd-hit, MAFFT, WoLF PSORT) are no longer committed to the repository. RAVEN 3 +fetches them on demand from the `raven-data` GitHub release the first time a +function needs one, which means **internet access is required on first use**, +not just at clone/install time. See [ยง11](../raven3-migration/new-and-environment.md#toolbox-deps) for what's needed for +an air-gapped install, and the full dependency comparison. diff --git a/docs/raven3-migration/formats-and-reconstruction.md b/docs/raven3-migration/formats-and-reconstruction.md new file mode 100644 index 0000000..3041aa8 --- /dev/null +++ b/docs/raven3-migration/formats-and-reconstruction.md @@ -0,0 +1,122 @@ +# File formats, reconstruction, and GPR parsing + +(file-formats-and-model-io)= +## 1. File formats and model I/O + +(sbml-io)= +### SBML I/O + +| | RAVEN 2 | RAVEN 3 | +|---|---|---| +| `exportModel` output format | SBML Level 3 Version 1, FBC v2 | unchanged | +| `importModel` accepted input | any SBML level/version; L2-style notes-based gene associations (`GENE_ASSOCIATION`/`PROTEIN_ASSOCIATION`) also read; FBC fields optional | only SBML **Level 3 Version 1, FBC v2**; anything else errors | +| MIRIAM prefixes read on import | `urn:miriam:`, `http://identifiers.org/`, `https://identifiers.org/` | unchanged | + +Older SBML files must be converted first (e.g. re-exported from COBRApy/COBRA +Toolbox) before `importModel` will accept them. + +(excel-io)= +### Excel I/O + +| | RAVEN 2 | RAVEN 3 | +|---|---|---| +| Excel engine | Apache POI (Java), bundled | dependency-free OOXML writer (`io/writeExcel.m`) | +| Full-model `.xlsx` import | `importExcelModel.m` (`RXNS`/`METS`/`COMPS`/`GENES`/`MODEL` sheets โ†’ complete model) | removed, no equivalent importer | +| Nearest replacement for curation | no equivalent | `curation/curateModelFromTables.m`: curates/extends an *existing* model from separate `.tsv` files (mets by name+compartment, reactions by stoichiometry, genes by name), not a from-scratch importer | +| `exportToExcelFormat` on a bare path (no `.xlsx`) | fell back to tab-delimited text export | errors, `.xlsx` only | + +If your pipeline depends on the old Excel curation template, either convert it +to the `.tsv` shape `curateModelFromTables` expects, or build the model via +`importModel`/YAML and use `curateModelFromTables` for incremental edits. + +### YAML I/O + +`readYAMLmodel`/`writeYAMLmodel` are pure-MATLAB in both branches; model YAML +I/O has no Python dependency in RAVEN 2 or RAVEN 3. (`io/parseYAML.m` is an +unrelated, new, pure-MATLAB generic YAML parser used only by +`conditions/applyCondition.m` to read condition files; it doesn't affect +model I/O.) Output format changes, visible as diffs in version-controlled +model files: + +| | RAVEN 2 | RAVEN 3 | +|---|---|---| +| Quote style | single quotes | double quotes | +| Reaction EC codes | top-level `eccodes` field only | also written under `annotation: ec-code` | +| Gene-less model | `genes:` section omitted | `genes: []` written explicitly | +| Empty `gene_reaction_rule` | written as an empty string | omitted | +| `objective_coefficient` on import | ignored | honored | + +(raven-cobra-interoperability)= +### RAVENโ‡„COBRA interoperability + +`conversion/ravenCobraWrapper.m` keeps its signature +(`newModel = ravenCobraWrapper(model)`) unchanged. Two correctness fixes change +output for affected models: a stale-variable bug that could reuse the wrong +MIRIAM data during gene-annotation conversion, and gene IDs containing regex +metacharacters (e.g. `ENSG00000141510.16`) that could be mis-matched during +annotation conversion; both fixed. + +--- + +(kegg-reconstruction)= +## 2. Reconstruction: KEGG and homology pipelines + +### KEGG-based reconstruction (`getModelFromKEGG`, `getKEGGModelForOrganism`) + +| | RAVEN 2 | RAVEN 3 | +|---|---|---| +| Reaction/gene database source | local KEGG FTP dump (license required), parsed by `getGenesFromKEGG`/`getMetsFromKEGG`/`getRxnsFromKEGG` into a cached `keggModel.mat` | prebuilt artifacts (a gene-free reference model plus KO/reaction/organism tables) fetched on demand from the `raven-data` GitHub release, assembled by `buildGlobalKEGGModel`/`buildGlobalGPR`/`readKEGGTable` | +| `getModelFromKEGG`'s `keggPath` argument | required | removed, not needed | +| Ortholog HMM library | one `.hmm` file per KO (self-built via CD-HIT + MAFFT + `hmmbuild`, or downloaded); queried with one `hmmsearch` call per KO, via WSL on Windows | one concatenated per-domain library (`kegg118_eukaryotes.hmm` / `kegg118_prokaryotes.hmm`); one `hmmsearch` call against the whole proteome, no WSL | +| `dataDir` | arbitrary local folder | must be `kegg118_eukaryotes` or `kegg118_prokaryotes` | +| `nSequences`, `seqIdentity` (CD-HIT clustering controls) | configurable | removed, no local clustering step | +| `cutOff` default | `10^-50` | `10^-30` | +| `minScoreRatioG` default | `0.8` | `0.9` | +| Calling convention | positional | positional-or-named (`parseRAVENargs`); old positional order preserved | + +### Homology-based reconstruction (`getModelFromHomology`) + +| | RAVEN 2 | RAVEN 3 | +|---|---|---| +| `strictness` parameter | `1`/`2`/`3` | unchanged | +| Best-hit tie-break at `strictness=3` | E-value | bitscore by default (`"scoreBy","evalue"` restores the old criterion) | +| `minLen` default | `200` | `100` | +| GPR gene substitution when merging homology hits | regex/`strrep` substitution (could corrupt IDs that were substrings of one another, or contained regex metacharacters) | exact-token-based substitution | + +### Shared infrastructure + +External binaries (BLAST+, DIAMOND, HMMER) and the KEGG data artifacts +themselves are fetched on demand rather than bundled with the repository; see +[ยง11](../raven3-migration/new-and-environment.md#toolbox-deps). + +**New:** `curation/downloadGenomeData.m`, `curation/getGeneData.m`, and +`curation/processProteinFastaFile.m` give a new on-ramp from a bare NCBI genome +accession (`GCF_`/`GCA_`) straight to a model-ready proteome and gene ID +mapping table. + +--- + +(gpr-parsing)= +## 3. Gene-reaction-rule (GPR) parsing + +RAVEN 3 adds a proper grRule tokenizer/parser: `utils/parseGrRule.m` builds a +parse tree (`type`: `'gene'|'and'|'or'`, with `children`); `utils/grRuleToDNF.m` +converts it to disjunctive normal form (RAVEN's canonical "OR of AND-clauses" +shape for enzyme-complex/isozyme structure); `utils/isDnfGrRule.m` checks +whether a rule is already in that form; `utils/grRuleToString.m` renders a tree +back to a canonical string. These are used internally by four functions, +they are **not** applied automatically to every loaded model; a grRule keeps +its original text unless one of these functions is called on it. + +| Function | RAVEN 2 parsing | RAVEN 3 parsing | +|---|---|---| +| `removeGenes` | substring match on gene ID (`"10"` also matched `"100"`) | exact gene-token match | +| `expandModel` | naive split on `' or '` (could drop an `AND`-ed subunit inside a nested rule) | parse-tree-based isozyme expansion, preserves nested `AND`/`OR` | +| `findPotentialErrors` | string-pattern check for `") and ("` etc. (false positives on valid rules) | parse-tree-based non-DNF detection | +| `addRxns` (new gene IDs from a grRule) | split on the substrings "and"/"or" anywhere (could shred an ID like `"band1"`) | split only on space-bounded `' and '`/`' or '` tokens | +| `comparison/diffModels.m` | (function didn't exist) | grRules compared as logical DNF equality, not string equality | + +The accepted grRule *syntax* itself is unchanged: `' and '`/`' or '` +(case-insensitive) still work exactly as before, and +`manipulation/standardizeGrRules.m` (which runs on import) uses the same +string-splitting logic it always did. diff --git a/docs/raven3-migration/new-and-environment.md b/docs/raven3-migration/new-and-environment.md new file mode 100644 index 0000000..78d31ae --- /dev/null +++ b/docs/raven3-migration/new-and-environment.md @@ -0,0 +1,76 @@ +# New functionality, dependencies, and error handling + +(ftinit)= +## 1. Gap-filling and context-specific model extraction + +| | RAVEN 2 | RAVEN 3 | +|---|---|---| +| `fillGaps` algorithm | one built-in MILP connectivity-maximization strategy | same strategy is still the default; `algorithm` option adds `'fastLP'`/`'swiftLP'` (FASTCORE/SWIFTCORE-style LP relaxation), `'gapfillMILP'` (growth-floor MILP with directionality repair), `'topological'` (BFS producibility pre-screen, no model modification) | +| `fillGaps` positional arguments | `allowNetProduction, useModelConstraints, supressWarnings, rxnScores` | unchanged; new options are name-value-only | +| `ftINIT` tie-breaking | depends on solver seed | optional `resolveTies` pins degenerate MILP optima to a deterministic answer | +| `ftINIT` MILP optimality proof | escalating relative-gap search only | optional `proveAbsGap` proves each stage to a fixed absolute gap in one solve | +| `ftINIT` positional arguments | preserved | unchanged; `resolveTies`/`proveAbsGap` are name-value-only, default off | +| `getINITModel`/`runINIT` | supported, no runtime notice | supported, now under `INIT/tINIT/`; one-time "legacy, use `ftINIT`" notice (see [ยง5](../raven3-migration/structure-and-renames.md#renamed-merged-and-consolidated-functions)) | +| `fitTasks` gap-fill mode | one behavior (equivalent to today's `'merge'`) | `gapFillMode`: `'merge'` (default, same as before) or `'preMerged'` (faster, ftINIT-style) | +| `fitTasks` outputs | `[outModel, addedRxns]` | `[outModel, addedRxns, failedTasks]` | +| Task requiring net production of a metabolite with no reactions | boundary constraint silently dropped before gap-filling; task could incorrectly report as passing | constraint preserved; task correctly reports failing if the reference model can't produce it | + +--- + +(new-functionality)= +## 2. New functionality + +All additive; none of this replaces or changes behavior of existing RAVEN 2 +functions, except where noted. + +| Area | New function(s) | What it adds | +|---|---|---| +| Flux sampling (`analysis/`) | `sampleACHR`, `sampleCHRR`, `sampleMaxVolEllipse`, `sampleChebyshevCenter`, `sampleWarmupPoints` | `randomSampling` becomes a dispatcher (`'method'` option) across RAVEN 2's original random-objective method (now `'randomObjective'`) plus two new near-uniform MCMC samplers: ACHR (hit-and-run) and CHRR (hit-and-run with rounding, better mixing on ill-conditioned polytopes) | +| Flux analysis / QC | `compareFluxes`, `getMinimalMedium`, `traceFluxPath`, `walkFluxes`, `modelSummary` | structured flux-vector diffing; MILP-based minimal-medium search; highest-flux-fraction path tracing between two reactions; interactive flux-neighborhood navigation; model/flux summary printing | +| Model comparison (`comparison/`) | `diffModels` | structured, ID-keyed diff between two models: stoichiometry, bounds, objective, grRule (as logical DNF equality), EC codes, metabolite properties; distinct from the existing overlap-style `compareMultipleModels` | +| Manipulation | `findDuplicateRxns`, `findPotentialErrors` | standalone duplicate-stoichiometry query; parse-tree-based non-DNF grRule detection (see [ยง8](../raven3-migration/formats-and-reconstruction.md#gpr-parsing)) | +| Genome-based curation (`curation/`, new folder) | `downloadGenomeData`, `getGeneData`, `processProteinFastaFile`, `renameModelGenes`, `curateModelFromTables` | pipeline from an NCBI genome accession to a gene ID mapping table to bulk gene renaming in a model; `curateModelFromTables` bulk-curates mets/rxns/genes from `.tsv` files against an existing model | +| Biomass composition (`biomass/`) | `getBiomassFractions`, `scaleBiomassFraction`, `scaleBiomassPseudoreaction`, `setGAM` | organism-agnostic biomass-composition and growth-associated-maintenance (GAM) calibration; `tutorial/tutorial7.m` demonstrates the workflow | +| Localization (`localization/`) | `assignCompartments`, `defaultCompartmentMap`, `getUniProtScores` | deterministic MILP compartment assignment (alternative to the heuristic `predictLocalization`); `parseScores` gains `'deeploc'`/`'cello'`/`'mulocdeep'`/`'compartments'`/`'uniprot'` predictors (replacing WoLF PSORT; see [ยง3](../raven3-migration/backward-incompatible.md#backward-incompatible-changes)) | +| Annotation (`annotation/`) | `assignSBOterms`, `loadDeltaGCSV`/`saveDeltaGCSV` | organism-agnostic SBO term assignment; ฮ”G thermodynamic data CSV round-trip | +| Conditions (`conditions/`, new folder) | `applyCondition` | applies a data-driven growth/media condition (a YAML file, not code) to a model in a fixed sequence: exchange bounds, optional cofactor-pseudoreaction edits, optional biomass-stoichiometry deltas, per-reaction bound overrides | +| Internal infrastructure | `ravenModelFields` | central registry of every 1-D model field (entity type, default value), backing `permuteModel`/`removeReactions` internally | + +--- + +(toolbox-deps)= +## 3. Toolbox dependencies and installation + +| | RAVEN 2 | RAVEN 3 | +|---|---|---| +| MATLAB version | no pinned/documented minimum (guidance: avoid very new MATLAB syntax, to accommodate older installs) | CI is tested against **R2024b** | +| Statistics Toolbox | required (`combnk`) | not required (`nchoosek`, base MATLAB) | +| Bioinformatics Toolbox | required (`fastaread`/`fastawrite`) | not required (`io/readFasta.m`/`io/writeFasta.m`) | +| Optimization Toolbox | required for `qMOMA`/`solveQP` | not required (both removed) | +| Excel I/O | bundled Apache POI (Java) | dependency-free OOXML writer | +| LP/MILP solver | required (bundled `glpk`, or Gurobi/SCIP/COBRA) | unchanged, still required | +| External binaries (BLAST+, DIAMOND, HMMER, cd-hit, MAFFT) | committed to the repository, available offline immediately after clone | fetched on demand from the `raven-data` GitHub release on first use (per-platform ZIPs); an offline `*-full`/`*-binaries` release bundle is available for air-gapped installs | +| KEGG HMM libraries / reference data | manual download or local build required | fetched on demand from `raven-data`, cached locally after first use | +| Installation self-check | `installation/checkInstallation.m`; ran the full test suite to populate its pass/fail table | renamed `checkRaven`, moved to the repo root; does fast, targeted checks (a small import/export round-trip, a trivial LP per solver) instead of invoking the full test classes, and proactively offers to download missing binaries | + +Net effect: a bare RAVEN 3 clone is far smaller and installs with fewer MATLAB +toolbox prerequisites. The trade-off: it needs network access the first time +KEGG- or homology-based reconstruction functions actually run. + +--- + +## 4. Error and warning handling + +RAVEN 2's `dispEM` always raised errors with an empty identifier, so no RAVEN 2 +`try/catch` could have matched on `err.identifier`; RAVEN 3's move to native +`error()`/`warning()` with real (mostly `RAVEN:*`) identifiers is additive from +that angle, not breaking. Message text is preserved for most existing checks, +so text-matching `try/catch` blocks are unaffected either way. See +[ยง3](../raven3-migration/backward-incompatible.md#backward-incompatible-changes) for the details and the new +`utils/ravenList.m` message-formatting helper. + +One related, purely additive change: `queries/checkModelStruct.m`, called *with* +an output argument (`issues = checkModelStruct(model)`), now returns a +structured array of findings (`category`/`target`/`message`) instead of only +throwing or printing, useful for programmatic model QC. The no-output-argument +calling convention (throws/warns directly) is unchanged. diff --git a/docs/raven3-migration/structure-and-renames.md b/docs/raven3-migration/structure-and-renames.md new file mode 100644 index 0000000..6a3b87e --- /dev/null +++ b/docs/raven3-migration/structure-and-renames.md @@ -0,0 +1,113 @@ +# What moved, what's gone, and what's renamed + +## 1. Repository reorganization + +RAVEN 2's top-level layout (`core/`, `external/`, `hpa/`, `io/`, `legacy/`, +`pathway/`, `plotting/`, `solver/`, `struct_conversion/`, `installation/`, +`utils/`) has been replaced with functional folders that group functions by what +they do rather than by legacy history: + +| RAVEN 3 folder | Contents | Mostly came from (RAVEN 2) | +|---|---|---| +| `INIT/` | `ftINIT`, `INIT/tINIT/` for `getINITModel`/`runINIT`, and HPA/expression parsing + gene scoring (`parseHPA`, `parseHPArna`, `scoreModel`) | `INIT/`, `hpa/` | +| `analysis/` | FBA-adjacent analyses: FVA, sampling, FSEOF, OptKnock, robustness, production envelope, flux comparison/tracing | `core/` | +| `annotation/` | MIRIAM editing, SBO term assignment, ฮ”G CSV I/O | `struct_conversion/` (+ new) | +| `biomass/` | Biomass composition fitting/scaling, GAM | `core/` (+ new) | +| `comparison/` | Multi-model comparison, structured model diffing | `core/` (+ new) | +| `conditions/` | Applying a data-driven growth/media condition to a model | *new folder* | +| `conversion/` | RAVENโ‡„COBRA conversion, identifier prefixing, field ordering | `struct_conversion/` | +| `curation/` | Genome-annotation-driven gene/model curation | *new folder* | +| `gapfilling/` | `fillGaps` and its pluggable algorithms, task fitting, leak-metabolite/exchange checks | `core/` (+ new) | +| `installation/` | Path setup, on-demand binary/data download | `installation/` | +| `io/` | Model import/export (SBML, YAML, Excel), FASTA I/O | `io/` | +| `localization/` | Subcellular localization prediction/scoring, compartment assignment | `external/` (+ new) | +| `manipulation/` | Structural edits: add/remove/change rxns, mets, genes; merge, sort, simplify | `core/` | +| `queries/` | Non-mutating lookups: indexes, exchange/transport reactions, stoichiometry construction, model checking | `core/` | +| `reconstruction/` | KEGG- and homology-based draft reconstruction | `external/` | +| `solver/` | LP/MILP solving | `solver/` | +| `tasks/` | Metabolic task list parsing/checking | `core/` | +| `testing/` | `matlab.unittest` suite (`function_tests/`) | `testing/` | +| `utils/` | Generic helpers, including the new argument-parsing/registry infrastructure | `utils/` | +| `deprecated/` | Thin backward-compatible wrappers for merged/renamed functions | *new folder* | + +Entirely removed (no successor folder): `external/metacyc/` (MetaCyc +reconstruction), `pathway/` and `plotting/` (visualization), `legacy/` +(CellDesigner import and other pre-2.0 code), `struct_conversion/`'s Excel-POI +helpers. + +**The reorganization affects fewer scripts than the change list suggests.** +If your scripts call RAVEN +functions by name (the normal usage pattern, with the whole RAVEN folder tree, +including subfolders, added to the MATLAB path via `addRavenToUserPath` or +`addpath(genpath(...))`), moving a file to a new subfolder does not break the +call: MATLAB resolves functions by name across the whole path, not by a +hard-coded relative location. The reorganization only matters if you (a) +hard-coded a path to a specific `.m` file, (b) called one of the functions that +was actually removed or merged (see below), or (c) shadowed a RAVEN function +name with your own; folder names are irrelevant otherwise. + +The full moved-file mapping (RAVEN 2 path โ†’ RAVEN 3 path) for every relocated +function is available on request / in the repository's git history +(`git diff --stat -M develop...develop3`); it is not reproduced in full here +because it is large and, as the point above explains, rarely something you +need to act on. + +--- + +(removed-functionality)= +## 2. Removed functionality + +Beyond the items already listed as backward-incompatible above, RAVEN 3 removed +a number of functions and whole subsystems as part of a deliberate effort to +make the codebase smaller. None of these have call sites left in RAVEN 3 itself. + +| Removed | What it did | Replacement / notes | +|---|---|---| +| `pathway/*`, `plotting/*` (16 functions: `drawMap`, `drawPathway`, `colorPathway`, `colorSubsystem`, `markPathwayWithFluxes`, `markPathwayWithExpression`, `setOmicDataToRxns`, `plotAdditionalInfo`, etc.) | Metabolic map/pathway visualization | None in RAVEN core. Use COBRA Toolbox, Escher, or export flux/omics data for external plotting. | +| `external/metacyc/*` (7 functions: `getModelFromMetaCyc`, `getRxnsFromMetaCyc`, `getMetsFromMetaCyc`, `getEnzymesFromMetaCyc`, `addSpontaneousRxns`, `linkMetaCycKEGGRxns`, `combineMetaCycKEGGModels`) | MetaCyc-based draft reconstruction | None; KEGG- and homology-based reconstruction remain the supported paths. | +| `legacy/*` (CellDesigner import, `changeGeneAssoc`, the bundled `xml_toolbox`) | Pre-2.0 compatibility code | None; considered dead. | +| `io/importExcelModel.m`, `io/loadWorkbook.m`, `io/loadSheet.m`, `io/writeSheet.m`, `io/SBMLFromExcel.m`, `io/addJavaPaths.m`, `io/startup.m` | Apache-POI-based Excel model I/O | `io/writeExcel.m` (dependency-free OOXML writer) for export; `curation/curateModelFromTables.m` for tabular curation. No full Excel-template *import* path remains; see [ยง6](../raven3-migration/formats-and-reconstruction.md#excel-io). | +| `io/exportModelToSIF.m`, `io/exportToTabDelimited.m` | SIF graph export; tab-delimited model export | Dead/orphaned code with no callers; removed outright. | +| `io/getFullPath.m`, `io/getMD5Hash.m`, `io/getToolboxVersion.m` | Path canonicalization; file MD5 hashing; toolbox git-version lookup | Inlined where still needed (e.g. MD5 via Java's `MessageDigest` directly inside `getBlast`), or folded into `exportForGit`. | +| `core/dispEM.m` | Universal error/warning display | Native `warning()`/`error()` + new `utils/ravenList.m` for formatted item lists. See [ยง3](../raven3-migration/backward-incompatible.md#backward-incompatible-changes). | +| `core/followFluxes.m`, `core/followChanged.m` | Flux-change reporting relative to a reference solution | `queries/printFluxes.m` covers the cutoff-filtering half; nothing replaces the reference-flux comparison directly. | +| `core/printModel.m` | Printed reactions to screen/file | `queries/printModelStats.m` / `queries/printFluxes.m` cover related summary/flux printing; no direct one-line-per-reaction dump remains. | +| `core/getMetsInComp.m` | Returned metabolite indices in a given compartment | Inline it as `model.metComps == compIndex`. | +| `core/mapCompartments.m` | Remapped compartment labels in a localization-score structure | Superseded by the new predictor-based localization workflow and `localization/defaultCompartmentMap.m`. | +| `core/getExpressionStructure.m` | Loaded an expression-experiment structure from an ad hoc Excel format | Superseded by `INIT/parseHPA.m`/`INIT/parseHPArna.m` for the supported expression-data workflow. | +| `core/checkRxn.m` | Per-reaction reactant/product synthesizability debugging | No direct successor; use `gapfilling/findLeakMetabolite.m`, `gapfilling/canExchange.m`, or `gapfilling/checkProduction.m`. | +| `core/parallelPoolRAVEN.m` | Parallel Computing Toolbox pool setup | Renamed/rewritten as `utils/parallelWorkersRAVEN.m`. | +| `external/getWoLFScores.m` | WoLF PSORT score parsing | None; see [ยง3](../raven3-migration/backward-incompatible.md#backward-incompatible-changes). | +| `external/kegg/{getGenesFromKEGG,getMetsFromKEGG,getRxnsFromKEGG,constructMultiFasta,getWSLpath}.m`, `external/getBlastFromExcel.m` | Local-KEGG-FTP-dump parsing; per-KO multi-FASTA assembly; WSL path translation; Excel-based BLAST import | Superseded by the raven-data-artifact-based KEGG pipeline. See [ยง7](../raven3-migration/formats-and-reconstruction.md#kegg-reconstruction). | +| `solver/qMOMA.m`, `solver/solveQP.m` | MOMA-based analyses (quadratic programming) | Removed; no in-toolbox QP-based replacement. See [ยง3](../raven3-migration/backward-incompatible.md#backward-incompatible-changes) for `findGeneDeletions` impact. | +| `testing/unit_tests/*.m` (script-based tests) | Ad hoc test scripts | `testing/function_tests/*.m`, a `matlab.unittest`-class-based suite. Contributor-facing only; `runRAVENtests` usage is unchanged for end users. | + +--- + +(renamed-merged-and-consolidated-functions)= +## 3. Renamed, merged, and consolidated functions + +RAVEN 3 introduces a formal **`deprecated/` folder convention**: functions that +were merged into a more general replacement stay callable under their old name, +as a thin wrapper that emits a one-time `RAVEN:deprecated` warning ("will be +removed in the next major release") and forwards to the new function. + +| Old function(s) | New function | Notes | +|---|---|---| +| `makeSomething(model, ignoreMets, isNames, minNrFluxes, allowExcretion, params, ignoreIntBounds)` | `findLeakMetabolite(model, 'produce', ...)` | Argument order maps 1:1; the wrapper forwards `varargin` unchanged. | +| `consumeSomething(model, ignoreMets, isNames, minNrFluxes, params, ignoreIntBounds)` | `findLeakMetabolite(model, 'consume', ...)` | **Not** a verbatim forward: `consumeSomething`'s old positional order has no `allowExcretion` slot, so the wrapper explicitly remaps arguments by name to avoid corrupting `params`/`ignoreIntBounds`. | +| `canProduce(model, mets)` | `canExchange(model, 'produce', 'mets', mets)` | | +| `canConsume(model, mets)` | `canExchange(model, 'consume', 'mets', mets)` | | +| `compareRxnsGenesMetsComps` | folded into `comparison/compareMultipleModels.m` | Its rxn/gene/met/comp overlap output is now one part of `compareMultipleModels`'s output; no separate function remains. | +| `INIT/scoreComplexModel.m` | `INIT/scoreModel.m` | **Merged, not renamed.** RAVEN 2 had *both* `INIT/scoreComplexModel.m` and `hpa/scoreModel.m`; RAVEN 3 folded them into one `scoreModel`, called with different settings from tINIT than from ftINIT. A RAVEN 2 call to `scoreModel` is unaffected; a call to `scoreComplexModel` has to move to `scoreModel` *and* pass the settings that reproduce the old behaviour (a single operator for both `and`/`or`, `dataPrecedence` `'reaction'`). | +| `INIT/ftINITFillGapsMILP.m` | folded into `solver/getMinNrFluxes.m` | Exposed now via `getMinNrFluxes`'s `resolveTies`/`proveAbsGap` options. | +| `INIT/ftINITFillGapsForAllTasks.m` | folded into `gapfilling/ftINITFillGaps.m` and `fitTasks`'s `'preMerged'` gap-fill mode | | +| `getGenesFromKEGG`/`getMetsFromKEGG`/`getRxnsFromKEGG` | `reconstruction/kegg/readKEGGTable.m` + the raven-data KEGG artifacts | Not a direct signature-compatible rename: the whole data source changed; see [ยง7](../raven3-migration/formats-and-reconstruction.md#kegg-reconstruction). | +| `checkInstallation` | `checkRaven` | Moved from `installation/checkInstallation.m` to repo-root `checkRaven.m`; identical calling signature. `checkInstallation` remains as a deprecated forwarding wrapper, so old calls keep working with a `RAVEN:deprecated` warning. | + +Separately, `getINITModel` and `runINIT` (now under `INIT/tINIT/`) are marked +**legacy but indefinitely supported** (not on the `deprecated/` removal track) +via `utils/legacyMethodNotice.m`, which fires a one-time, silenceable +`RAVEN:legacyMethod` warning pointing at `ftINIT`. Use `ftINIT` for new work; +keep `getINITModel`/`runINIT` only to reproduce models originally built with +them. diff --git a/docs/raven3-vs-raven-toolbox.md b/docs/raven3-vs-raven-toolbox.md index c2ad747..3ee2472 100644 --- a/docs/raven3-vs-raven-toolbox.md +++ b/docs/raven3-vs-raven-toolbox.md @@ -1,4 +1,4 @@ -# RAVEN and raven-toolbox +# RAVEN vs. raven-toolbox :::{note} Looking for a specific function? The [MATLAB vs Python](matlab-vs-python.md) table pairs every function that @@ -58,164 +58,12 @@ Check the [mapping table](matlab-vs-python.md) rather than guessing. It also records what cobrapy covers instead of raven-toolbox, and what is deliberately absent. -## What only raven-toolbox has +## Detail pages -### Built on cobrapy - -The largest difference is not a feature but a foundation: a raven-toolbox model -**is** a `cobra.Model`. Every cobrapy tool, and everything in the wider COBRA -Python ecosystem, works on it without conversion. RAVEN's own model is a MATLAB -struct, and `ravenCobraWrapper` converts between that and the COBRA Toolbox -format as an explicit step. - -The practical consequence: FBA, FVA, knockouts, media handling and the rest come -from cobrapy rather than from raven-toolbox, which is why so many RAVEN -functions have a *cobrapy* row rather than a Python counterpart in the mapping -table. - -### KEGG artefact generation - -raven-toolbox can build the KEGG reference artefacts themselves, parsing a KEGG -release into reaction and compound tables, assembling the reference model, -building the per-KO FASTA sets and HMM libraries, and deriving the phylogenetic -distance matrix. RAVEN consumes pre-built artefacts; it does not produce them. - -This is what keeps the KEGG route reproducible against a stated KEGG release -rather than against whichever artefact is currently distributed. - -### Smaller additions - -- **Growth conditions**: apply a named, versioned growth condition to a model. -- **Batch curation**: apply a table of curation edits to a model in one pass. -- **ฮ”G and SBO annotation**: load and save thermodynamic data through CSV, and - assign SBO terms. -- **Biomass helpers**: sum a biomass composition, rescale a pseudoreaction, and - scale a fraction to a measured value. -- **Checksummed provisioning**: both toolboxes now fetch BLAST+, DIAMOND, - HMMER and the KEGG artefacts on demand rather than bundling them, from the - same release. What is Python-only is the baked registry that pins a given - release to the exact assets it was tested against; RAVEN resolves from the - published release each time. See - [Download data and binaries](installation/data-and-binaries.md). - -## What only RAVEN has - -### tINIT - -`getINITModel` and `runINIT` are the original tINIT implementation. RAVEN keeps -them for the models already built with them; raven-toolbox, a new -implementation with no such installed base, carries ftINIT alone. A tINIT model -has to be reproduced in MATLAB; see -[14. Context-specific models](guide/init.md). - -### COBRA Toolbox conversion - -`ravenCobraWrapper` converts between the RAVEN and COBRA Toolbox model -structures. There is nothing to convert in Python: the model is already a -`cobra.Model`, so no equivalent exists or is needed. - -### Dynamic FBA - -`runDynamicFBA` has no Python counterpart, deliberately. Several maintained -Python packages already cover dynamic FBA well -([dfba](https://pypi.org/project/dfba/), -[reframed](https://pypi.org/project/reframed/), -[mewpy](https://pypi.org/project/mewpy/)), and reimplementing it would add a -second-rate version of something that already exists. - -### MATLAB-specific plumbing - -A large share of RAVEN's function count is MATLAB housekeeping with nothing to -map to: path management (`addRavenToUserPath`, `findRAVENroot`), argument -handling (`parseRAVENargs`, `convertCharArray`), progress and printing -(`setRavenProgress`, `printOrange`), and the solver abstraction -(`optimizeProb`, `setRavenSolver`), which in Python is cobrapy's solver -interface via optlang. - -## Same function, different answer - -Where the two agree on the job but differ in what they hand back, how they order -it, or what they do to the model on the way. Deliberately short: an entry is -added only once the difference has been confirmed in both sources. - -:::{admonition} Not a complete list -:class: info -Absence from this section is not a guarantee of identical behaviour. It only -means the difference has not been confirmed and written up yet. -::: - -### Duplicate reactions: gene associations are not merged - -`contractModel` merges duplicate reactions **and their gene associations**: when -it collapses a set of duplicates it joins the distinct `grRules` with `or`, so -every gene that pointed at any of the duplicates still points at the survivor. - -`remove_duplicate_reactions` keeps one reaction of each duplicate set and removes -the rest, without merging gene associations. A gene that was associated *only* -with a removed duplicate is no longer associated with anything. - -The stoichiometric network is the same either way; the geneโ€“reaction mapping is -not. If you are contracting a draft assembled from several templates, where the -same reaction commonly arrives with different gene associations, check the GPRs -of the survivors afterwards. - -### Metabolic tasks: the same result, a different runtime - -`checkTasks` rebuilds the working model from the original for each task. -`check_tasks` instead applies each task's constraints to one model inside a -`with model:` block and reverts them afterwards, restoring by hand the one kind -of edit cobra's context manager does not track (direct mass-balance bound -changes). - -The pass and fail results are the same. The runtime is not: at genome scale, -copying the model for each task dominates the MATLAB runtime, which is why the -Python version reuses a single model. This changes runtime comparisons, not -results. - -### Gap-filling: `fillGaps` splits into three functions in raven-toolbox - -`fillGaps` does several different jobs through one function call. -raven-toolbox splits them, so porting a `fillGaps` call means choosing: - -| What you were doing | Use | +| | | |---|---| -| Connecting blocked reactions against template models | `connect_blocked_reactions` | -| Fast LP-based filling of a large candidate set | `fill_gaps_fast_lp` | -| MILP filling with explicit weights | `fill_gaps_kumar_milp` | -| Only *finding* the gaps (`canExchange`, `checkProduction`, `getAllSubGraphs`, `haveFlux`) | `analyse_topology` | - -The choice changes both the reaction set added and the runtime, because each -of the three functions uses a different algorithm; they are not the same -algorithm under three different names. - -### Anything solved by MILP - -ftINIT extraction, gap-filling, and compartment assignment all solve -mixed-integer problems that routinely have **several optima of equal objective -value**. Two runs can return different reaction sets and both be correct; across -languages, across solvers, and in some configurations across runs of the same -solver. - -Do not compare these outputs for identity. Compare them for overlap, and -expect a range of acceptable values, not one exact number. - -### Elemental balance: an unknown result is not the same as balanced - -Elemental balance checks whether a reaction has the same count of each -chemical element (carbon, hydrogen, oxygen, and so on) on both sides, the way -a correct chemical equation must. `getElementalBalance` and -`get_elemental_balance` each report one of three results per reaction, not -two: `balanced`, `unbalanced`, or `unknown`. A reaction is `unknown` when one -of its metabolites has no chemical formula recorded, so the element counts -cannot be computed at all; that is different from `unbalanced`, which means -the counts were computed and did not match. The two functions agree on this -three-way result. - -That third case is lost if you switch to plain cobrapy: its -`check_mass_balance` has no separate `unknown` result, so a reaction with a -missing formula and a reaction that is genuinely unbalanced can look the -same. raven-toolbox keeps its own function instead of using cobrapy's -directly, specifically to keep that distinction. +| [What each has that the other doesn't](raven3-vs-raven-toolbox/what-each-has.md) | Capabilities that exist on only one side: cobrapy's foundation and KEGG artefact generation on the Python side; tINIT, COBRA Toolbox conversion, dynamic FBA, and MATLAB-specific plumbing on the MATLAB side. | +| [Same function, different answer](raven3-vs-raven-toolbox/same-function-different-answer.md) | Where both sides do the same job but hand back a different result: gene-association merging, task-checking runtime, gap-filling's three-way split, MILP alternate optima, and elemental balance. | ## Solvers @@ -229,3 +77,10 @@ Genome-scale MILP work (ftINIT extraction in particular) is where the choice matters most; see [Installation](installation/index.md) for the full solver matrix. +```{toctree} +:hidden: + +raven3-vs-raven-toolbox/what-each-has +raven3-vs-raven-toolbox/same-function-different-answer +``` + diff --git a/docs/raven3-vs-raven-toolbox/same-function-different-answer.md b/docs/raven3-vs-raven-toolbox/same-function-different-answer.md new file mode 100644 index 0000000..5147d16 --- /dev/null +++ b/docs/raven3-vs-raven-toolbox/same-function-different-answer.md @@ -0,0 +1,84 @@ +# Same function, different answer + +Where the two agree on the job but differ in what they hand back, how they order +it, or what they do to the model on the way. Deliberately short: an entry is +added only once the difference has been confirmed in both sources. + +:::{admonition} Not a complete list +:class: info +Absence from this section is not a guarantee of identical behaviour. It only +means the difference has not been confirmed and written up yet. +::: + +## Duplicate reactions: gene associations are not merged + +`contractModel` merges duplicate reactions **and their gene associations**: when +it collapses a set of duplicates it joins the distinct `grRules` with `or`, so +every gene that pointed at any of the duplicates still points at the survivor. + +`remove_duplicate_reactions` keeps one reaction of each duplicate set and removes +the rest, without merging gene associations. A gene that was associated *only* +with a removed duplicate is no longer associated with anything. + +The stoichiometric network is the same either way; the geneโ€“reaction mapping is +not. If you are contracting a draft assembled from several templates, where the +same reaction commonly arrives with different gene associations, check the GPRs +of the survivors afterwards. + +## Metabolic tasks: the same result, a different runtime + +`checkTasks` rebuilds the working model from the original for each task. +`check_tasks` instead applies each task's constraints to one model inside a +`with model:` block and reverts them afterwards, restoring by hand the one kind +of edit cobra's context manager does not track (direct mass-balance bound +changes). + +The pass and fail results are the same. The runtime is not: at genome scale, +copying the model for each task dominates the MATLAB runtime, which is why the +Python version reuses a single model. This changes runtime comparisons, not +results. + +## Gap-filling: `fillGaps` splits into three functions in raven-toolbox + +`fillGaps` does several different jobs through one function call. +raven-toolbox splits them, so porting a `fillGaps` call means choosing: + +| What you were doing | Use | +|---|---| +| Connecting blocked reactions against template models | `connect_blocked_reactions` | +| Fast LP-based filling of a large candidate set | `fill_gaps_fast_lp` | +| MILP filling with explicit weights | `fill_gaps_kumar_milp` | +| Only *finding* the gaps (`canExchange`, `checkProduction`, `getAllSubGraphs`, `haveFlux`) | `analyse_topology` | + +The choice changes both the reaction set added and the runtime, because each +of the three functions uses a different algorithm; they are not the same +algorithm under three different names. + +## Anything solved by MILP + +ftINIT extraction, gap-filling, and compartment assignment all solve +mixed-integer problems that routinely have **several optima of equal objective +value**. Two runs can return different reaction sets and both be correct; across +languages, across solvers, and in some configurations across runs of the same +solver. + +Do not compare these outputs for identity. Compare them for overlap, and +expect a range of acceptable values, not one exact number. + +## Elemental balance: an unknown result is not the same as balanced + +Elemental balance checks whether a reaction has the same count of each +chemical element (carbon, hydrogen, oxygen, and so on) on both sides, the way +a correct chemical equation must. `getElementalBalance` and +`get_elemental_balance` each report one of three results per reaction, not +two: `balanced`, `unbalanced`, or `unknown`. A reaction is `unknown` when one +of its metabolites has no chemical formula recorded, so the element counts +cannot be computed at all; that is different from `unbalanced`, which means +the counts were computed and did not match. The two functions agree on this +three-way result. + +That third case is lost if you switch to plain cobrapy: its +`check_mass_balance` has no separate `unknown` result, so a reaction with a +missing formula and a reaction that is genuinely unbalanced can look the +same. raven-toolbox keeps its own function instead of using cobrapy's +directly, specifically to keep that distinction. diff --git a/docs/raven3-vs-raven-toolbox/what-each-has.md b/docs/raven3-vs-raven-toolbox/what-each-has.md new file mode 100644 index 0000000..d145a5a --- /dev/null +++ b/docs/raven3-vs-raven-toolbox/what-each-has.md @@ -0,0 +1,75 @@ +# What each has that the other doesn't + +## What only raven-toolbox has + +### Built on cobrapy + +The largest difference is not a feature but a foundation: a raven-toolbox model +**is** a `cobra.Model`. Every cobrapy tool, and everything in the wider COBRA +Python ecosystem, works on it without conversion. RAVEN's own model is a MATLAB +struct, and `ravenCobraWrapper` converts between that and the COBRA Toolbox +format as an explicit step. + +The practical consequence: FBA, FVA, knockouts, media handling and the rest come +from cobrapy rather than from raven-toolbox, which is why so many RAVEN +functions have a *cobrapy* row rather than a Python counterpart in the mapping +table. + +### KEGG artefact generation + +raven-toolbox can build the KEGG reference artefacts themselves, parsing a KEGG +release into reaction and compound tables, assembling the reference model, +building the per-KO FASTA sets and HMM libraries, and deriving the phylogenetic +distance matrix. RAVEN consumes pre-built artefacts; it does not produce them. + +This is what keeps the KEGG route reproducible against a stated KEGG release +rather than against whichever artefact is currently distributed. + +### Smaller additions + +- **Growth conditions**: apply a named, versioned growth condition to a model. +- **Batch curation**: apply a table of curation edits to a model in one pass. +- **ฮ”G and SBO annotation**: load and save thermodynamic data through CSV, and + assign SBO terms. +- **Biomass helpers**: sum a biomass composition, rescale a pseudoreaction, and + scale a fraction to a measured value. +- **Checksummed provisioning**: both toolboxes now fetch BLAST+, DIAMOND, + HMMER and the KEGG artefacts on demand rather than bundling them, from the + same release. What is Python-only is the baked registry that pins a given + release to the exact assets it was tested against; RAVEN resolves from the + published release each time. See + [Download data and binaries](../installation/data-and-binaries.md). + +## What only RAVEN has + +### tINIT + +`getINITModel` and `runINIT` are the original tINIT implementation. RAVEN keeps +them for the models already built with them; raven-toolbox, a new +implementation with no such installed base, carries ftINIT alone. A tINIT model +has to be reproduced in MATLAB; see +[14. Context-specific models](../guide/init.md). + +### COBRA Toolbox conversion + +`ravenCobraWrapper` converts between the RAVEN and COBRA Toolbox model +structures. There is nothing to convert in Python: the model is already a +`cobra.Model`, so no equivalent exists or is needed. + +### Dynamic FBA + +`runDynamicFBA` has no Python counterpart, deliberately. Several maintained +Python packages already cover dynamic FBA well +([dfba](https://pypi.org/project/dfba/), +[reframed](https://pypi.org/project/reframed/), +[mewpy](https://pypi.org/project/mewpy/)), and reimplementing it would add a +second-rate version of something that already exists. + +### MATLAB-specific plumbing + +A large share of RAVEN's function count is MATLAB housekeeping with nothing to +map to: path management (`addRavenToUserPath`, `findRAVENroot`), argument +handling (`parseRAVENargs`, `convertCharArray`), progress and printing +(`setRavenProgress`, `printOrange`), and the solver abstraction +(`optimizeProb`, `setRavenSolver`), which in Python is cobrapy's solver +interface via optlang. diff --git a/docs/tutorials/index.md b/docs/tutorials/index.md index 6a60e42..1c32655 100644 --- a/docs/tutorials/index.md +++ b/docs/tutorials/index.md @@ -27,7 +27,7 @@ folder of the RAVEN repository. `checkRaven` passes; see [Installation](../installation/raven.md). - Tutorials 2โ€“4 involve editing models in **RAVEN-compatible Excel format**, using `importExcelModel`. RAVEN 3 removed `importExcelModel`, so this step - does not run as written on RAVEN 3; see [Excel I/O](../raven3-migration.md#excel-io) + does not run as written on RAVEN 3; see [Excel I/O](../raven3-migration/formats-and-reconstruction.md#excel-io) for the replacement, `curateModelFromTables`, which curates an existing model from `.tsv` files rather than importing a full model from `.xlsx`. - To run a section of a script in MATLAB, highlight it, right-click, and choose @@ -42,7 +42,7 @@ raven-toolbox function name; some RAVEN functions map to cobrapy instead (`solveLP` โ†’ `model.optimize()`), and some have no Python counterpart at all (there is no Excel reader). Look each one up in the [API reference](../api/index.md), and see -[RAVEN 3 and raven-toolbox](../raven3-vs-raven-toolbox.md) for what maps where. +[RAVEN vs. raven-toolbox](../raven3-vs-raven-toolbox.md) for what maps where. ::: ```{toctree} diff --git a/docs/yaml-format.md b/docs/yaml-format.md index 7b21afe..db12aa4 100644 --- a/docs/yaml-format.md +++ b/docs/yaml-format.md @@ -1,4 +1,4 @@ -# RAVEN / cobrapy YAML model format +# YAML file format This document describes the YAML format produced and consumed by