Build the network model result on mikeio1d's topology layer - #702
Conversation
Reopens the question ecomodeller raised on #694, with the seam set where NetworkModelResult already draws it: modelskill keeps the comparer-facing classes, mikeio1d gets the formats, the graph and the Network class itself. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Phase 1 landed as mikeio1d #247, so the decision is no longer a proposal: the module is built upstream and the Phase 0 snapshots pass against it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A network result file identifies nodes and reaches by ID, but observations are usually recorded against real-world station names held in the MIKE+ setup database. Read that sqlite database to resolve a station to the node or reach it sits on, so observations can be placed without hand-mapping every ID. The resolver is one class in obs.py rather than a module of its own: everything MIKE+ specific -- the table names, the join, the locationtype codes, the encoding of resitemname -- sits in one class body, and its only caller is a few lines below it. It returns a list of _Station rather than a DataFrame with five agreed column names, so the contract is in the type. Dataset variables also gain a long_name attribute, so a quantity keeps its label once it reaches xarray. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A node observation records whatever the caller addressed it with, so a comparer
built from at="node_A" already carries the name. The model side disagreed: it
recorded the integer the network handed out, and load() coerced the coordinate
back with int(), which raises on a name. A breakpoint was worse off again -- it
carries reach and distance rather than node, and several places tested for
"node" alone.
- One list of the coordinates that say where a timeseries sits, so the three
places that dropped a subset of them on the way to a dataframe now drop the
same set. Fixes NodeObservation(df, at=("r1", 24.5)).to_dataframe(), which
raised because the node coordinate it dropped is not there.
- NodeModelResult takes a node name or a (reach, distance) pair, and records the
graph integer beside it as node_index. Neither identity nor load reads that
integer back.
- load() stops re-deriving the location: the coordinates travelled with the
file. A reach comparer can now be saved and loaded at all, where it used to
raise NotImplementedError.
The backend is unchanged, so extraction still hands out integers; the next
commit swaps it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The extra is renamed networks -> network, matching mikeio1d's own, so modelskill[network] and mikeio1d[network] read the same. It shipped in the 1.4.0a3 alpha only, so nothing needs a deprecation. networkx and xarray now arrive through mikeio1d's extra rather than being named here. The module is unreleased, so a uv source points at mikeio1d's main branch. Both entries carry a TODO to swap it for a version floor once mikeio1d releases; per ADR-013, modelskill 1.4.0 waits for that release. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
modelskill's own topology layer goes: the abstract types, the Res1D adapter, the per-product constructors, the EPANET companions, the extension policy table and the .inp reader. All of it now lives in mikeio1d, where the formats and the fixtures already were (ADR-013). 1503 lines of source, and the CI failure that came with the extension table -- a mikeio1d release adding a format is no longer our problem. NetworkModelResult takes a path or a mikeio1d Network. A path goes to Network.open, which picks the reader and finds the EPANET companions; refusals for .out, .resx and the fixture-less formats are upstream's to word now. The network is used as given rather than deep-copied, which was the largest cost of building a model result on a river model. Extraction reads the dataset and consults the topology only to explain a failure. Two consequences worth knowing: - A location whose column is entirely NaN now counts as having no data. It used to be selected, and produced a match with no points, so the observation quietly disappeared; another break point on the reach is used instead. - Break points with an unknown distance now take part. That is EPANET read without its .inp, where no reach has a length and the second break point of every reach is unaddressable by distance. Node lookups delegate to Network.find, so there is one chainage tolerance rather than a second copy here, and a failed lookup names the near misses instead of listing the first five aliases in the map. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A network hands out integers to label its graph, and which integer a location gets depends on the network it was built from and the mikeio1d that built it. Nothing a user writes should depend on that, so at= now takes a node name or a (reach, distance) pair, and an integer raises with the recipe for getting the name back. The signature shipped in the 1.4.0a3 alpha only, so it goes without a shim. The guard catches np.int64 as well as int: Comparer.node returns a numpy scalar, and isinstance(np.int64(5), int) is False. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The user guide loses the sections that documented the topology layer -- building a network, the companions, selective loading, inspecting the graph, find and recall -- and gains a short section saying where the network comes from and how to hand modelskill a path. What stays is the part modelskill still owns: the skill assessment workflow and the MIKE+ database lookup. 587 lines to 197. The notebook addressed its observations by graph integer and highlighted junctions by reading a node attribute that upstream moved to edges; both are fixed, and it runs end to end. It is in the notebook SKIP_LIST, so CI would not have caught either. ADR-012 is narrowed rather than superseded: the constructors and the extension tables it argues for are mikeio1d's now, and the reasoning still applies there. ADR-010's open question about version constraints is answered for this feature. The res1d mapping diagram went with the section that explained it. The fixtures for the formats whose tests moved are kept, and the testdata README now says nothing here reads them. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The alpha wrote the graph integer into the node coordinate. Nothing on the load path derives a location from it any more, so such a file still loads and skills -- but a regression there would be silent, so the fixture is a file the alpha actually wrote rather than one built to look like it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ADR-013 was written as a proposal and parts of it no longer describe what was built: mikeio1d collapsed from_mike and from_epanet into Network.open, the extra is called network, and it named a fixture this repository does not have. The snapshot paragraphs said the same thing twice, once in each tense. ADR-012's note described its own position in the document. It now sits under the status line and says what moved. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Inverted constructions, closing kickers and sentences that restate the one before them. Stacked clauses split into separate sentences. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
…nto network-phase-2
A reach-gtype comparer fell through the gtype dispatch to NotImplementedError, even though the node branch's body is already correct for it: _drop_scalar_coords drops reach and distance too.
A breakpoint that exists but holds nothing for the selected quantity surfaced as "All datetime indices must be non-empty" from matching, because every timestep was dropped as NaN. The reach path already treats an all-NaN column as no data; the node path now does too.
The example paired breakpoint 21.285 on 94l1 with a WaterLevel model result, but that breakpoint only carries Discharge, so the block could not run -- and its plain python fence kept quarto from ever executing it. Point it at 42.57, which does carry WaterLevel, and make the fence executable so the render catches this next time.
The source filter was an unanchored LIKE '%name%' on tsfilename, so it could not tell calib.dfs0 from my_calib.dfs0 -- it still reported the item as ambiguous and told the caller to pass the source they had just passed -- and an underscore in a real file name acted as a wildcard. Compare file names in pandas instead, splitting on the Windows separator the database stores.
Every row matching the requested item names was classified before any quantity or kind filter, so a station whose locationtype is not a place in the network -- a rain gauge or a catchment registered in the same file -- failed the whole resolve, and on_missing="skip" could not help. Such stations are now left out, and named in the error only when they were what the caller asked for.
Its name was fed to the station resolver as a database item filter, so supplying metadata failed unless the name happened to match the database's own -- which the docstring never said it had to. A string selects; a Quantity describes.
_coords.py is a module of coordinate types. location_from_coords was the one piece of logic in it, and what it returns is NodeObservation.at's type -- a node name or a (reach, distance) pair. Both callers already import obs.py, so the move costs no new import edge. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Matches mikeio1d, which names the along-reach offset position. Comparer.distance becomes Comparer.position. Only the 1.4.0 alphas stored 'distance', so no loading shim is kept for it. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
1.4.0 does not have to load network comparers saved before it. The fixture was written by main after the alpha tag, and its tests loaded a netcdf file without touching mikeio1d. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
ADR 013 and the roadmap still waited for a mikeio1d release and measured the at= change against the alpha. mikeio1d 1.4.0 is out and is the release modelskill 1.4.0 requires. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Observations address locations by name or (reach, position), so the notebook no longer teaches graph_node integers, and says position instead of distance. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The user guide's ReachObservation example needs a discharge series; the existing sensors all hold water level. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Sensor 1 sits at node 98 and sensor 2 at break point (117l1, 48.7); the reach example scores the discharge sensor against Discharge. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The not-found error points to network.addresses(); it lists no nearby break points. A reach observation uses the lowest-position break point with data, not an arbitrary one. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
mikeio1d 1.4.0 takes EPANET reach lengths from the .res and refuses an .inp companion. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A position then snaps onto a breakpoint that carries the item. A location lacking it still gets the 'has no data for quantity' message. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
NodeObservation(position_tol=...) is passed to Network.resolve, so a measured chainage can snap onto the nearest breakpoint carrying the item. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A reach comparer picked up the first model's breakpoint position, since the observation had no position coordinate to win the merge. The comparer now sits where the observation sits, for nodes and reaches; each model's breakpoint stays on raw_mod_data. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
ReachObservation had no .at and a reach Comparer.at returned None. Both now return the reach name, so obs.at == cmp.at holds for reaches as it does for nodes and breakpoints. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
| from collections.abc import Hashable, Iterable | ||
|
|
||
| _RESERVED_NAMES = ["Observation", "time", "x", "y", "z"] | ||
| _RESERVED_NAMES = ["Observation", "time", "x", "y", "z", "node", "reach", "position"] |
The module-level importorskip already skips the file wherever mikeio1d cannot be installed, so the four markers never changed an outcome. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Each checked only that a match produced points or a node name, on a setup TestValuesReachTheComparer already scores to zero; the model result trim test repeats the observation one on the same trim. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Both built the same reach match; the .at assert now sits with the other reach comparer location asserts. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A comparer sits where its observation sits, but its raw model results kept the breakpoint each model was read at. Save and load wrote one location per comparer, so that breakpoint turned into the comparer's location after a round trip. Raw model results now take the observation's location when matched; where a model was read is NetworkModelResult.extract(obs).at. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A comparer's raw model results now sit at the observation's location, so the guide points to NetworkModelResult.extract for the break point read. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
ecomodeller
left a comment
There was a problem hiding this comment.
Nice work. Moving the topology and format handling to mikeio1d removes about 1,500 lines here, and what is left is small and easy to follow: building a NetworkModelResult reads nothing, extract reads only the locations the observations name, and users only ever handle node names and (reach, position) pairs. Reading location from the coordinates in one place (network_gtype, NETWORK_LOCATION_COORDS) fixes the save/load bugs where they came from, and the tests now run against a real result file instead of a hand-built network.
One correction to the description: Collection_systems_network is now skipped only on Python ≥ 3.15, so the notebooks workflow (Python 3.14) will run it, either on the push to main or through a manual run of that workflow. It has not run on this PR yet.
Optional follow-ups, none of them needed for this PR:
network_locationand_at_from_coordscould become one function intimeseries/_coords.pythat coerces tostr/float. Thencomparisonandmodel/networkwould no longer import a private helper fromobs.- The node branch of
NetworkModelResult.extractcould move to_extract_node, next to_extract_reach. network_cali.res11andswmm.outare no longer read here and could be removed in a separate PR.test_init_fails_with_unsupported_typecan assertTypeErroralone now.
Phase 2 of ADR-013. mikeio1d reads network result files and builds the graph. modelskill keeps the model result, the observations and the matching. Phase 1 was mikeio1d #247, released in mikeio1d 1.4.0.
Removed
src/modelskill/network.py(1200 lines),model/adapters/_res1d.py(189) and_inp.py(109), with their tests.NetworkModelResult.nodes,.dataand.time..nodespublished graph integers, which ADR-013 says users should not handle..dataand.timeneeded the whole network read up front..periodgives the time span from the file header.at=<int>onNodeObservation. It shipped in the 1.4.0a3 alpha only. An integer now raises and points atnetwork.to_networkx().nodes[<int>]["address"].NodeModelResult(data, node=..., name=..., item=...). It takes only a dataset that already carries its location, which is whatextractreturns.epanet.inpfixture. mikeio1d reads EPANET reach lengths from the.res, so nothing reads an.inp.API
NetworkModelResulttakes a path or amikeio1d.network.Network. A path goes toNetwork.open, which picks the reader and finds companion files where the format needs them.ms.model_result()builds aNetworkModelResultfrom either, by type or by.res1d,.res11or.resextension.NodeObservationnames its location by node name or by(reach, position).position_tolsets how far a position may be from a break point; by default it absorbs rounding only (mikeio1d's 1e-3).from_multipletakes either form as a key, andposition_tol.ReachObservation.from_multipleis new.NodeObservation,ReachObservation,NodeModelResultandComparerhave.at: a node name, a(reach, position)break point, or a reach name.Comparer.atisNoneoutside a network.Compareralso has.reachand.position.mr.extract(obs).at.distance→position.node,reachandpositionare reserved names.networks→network, matching mikeio1d's. Both requiremikeio1d[network] >= 1.4.0.Reading
Building a
NetworkModelResultreads no timeseries.extractresolves the observation's location withNetwork.resolveand reads that location only.A node observation is resolved with the model's item, so a position snaps onto a break point that carries it. A location that exists but lacks the item gets a "has no data for quantity" error, not "not found".
A
ReachObservationreads the break points on its reach that carry the quantity, in one read. They must agree. The one at the lowest position is used.Chainage tolerance, location lookup and format refusal messages are mikeio1d's.
Bugs fixed
Each has a test that fails on
main.Comparer.loadcoerced the node coordinate withint(). A comparer built fromat="node_A"could not be reloaded.NotImplementedError: Unknown gtype: reach.NodeObservation(df, at=("r1", 24.5)).to_dataframe()raised. It dropped anodecoordinate a break point does not have.An observation built from data that already carries a location now raises if
atorreachnames another. Before, the argument was dropped silently.Stacked PRs
Closed unmerged. Each change landed here or in mikeio1d:
Network.open(quantities=),_EMPTY_DATAand thenode_datacache innetwork/_res1d.py. Pinned by theres1d_one_quantitysnapshot andTestQuantityFilteringboundaryremoved_build_reach_breakpoints, and the clamped zero-lengthboundary=Trueedge innetwork/_graph.py. Pinned byTestAReachThatDoesNotStartAtZeroAddress = str | tuple[str, float], and.resxreach quantities merged onto break points. Its two modelskill-side guards are superseded by_extract_reachandNetwork.resolvemikeplus-station-lookupbranch.ReachObservation.from_multipleand the address keys onfrom_multipleare here. Theto_datasetquantity fix is moot:NetworkModelResulttakes its quantities fromNetwork.quantitiesand no longer callsto_datasetNetwork.open. The extension policy, companion discovery, the cp1252/latin-1 name repair and the CI coverage test are mikeio1d's_policy.pyand_companions.pymikeio1d's code answers both review threads on #695. #679, #680 and #599 are closed separately. #682, #683 and #684 moved to mikeio1d.
Checks
791 passed, 5 skipped, 1 failed locally.
ruff check src,ruff format --check,mypyand the doctests inmetrics.pyandtypes.pyare clean. CI is green. The user guide renders. The notebook runs end to end against mikeio1d 1.4.0. CI does not run it, since it is inSKIP_LIST.The failure is
tests/regression/test_regression_rose.py, which fails onmaintoo. The lockfile pins matplotlib 3.10.9, and the baseline image came from a newer one.The network tests and the docs build run on Python 3.14. mikeio1d requires < 3.15.
Notes for review
position_toldoes.mikeplus-station-lookup, which is this branch plus that feature.from_multiple(db=...)is public API and would otherwise ship with 1.4.0.network.res1dand the EPANET.res/.resxare read here.🤖 Generated with Claude Code