Skip to content

Build the network model result on mikeio1d's topology layer - #702

Merged
jpalm3r merged 119 commits into
mainfrom
network-phase-2
Oct 2, 2026
Merged

jpalm3r merged 119 commits into
mainfrom
network-phase-2

Conversation

@jpalm3r

@jpalm3r jpalm3r commented Aug 25, 2026 •

Copy link
Copy Markdown
Collaborator

Phase 2 of ADR-013. mikeio1d reads network result files and builds the graph. modelskill keeps the model result, the observations and the matching. Phase 1 was mikeio1d #247, released in mikeio1d 1.4.0.

Removed

  • src/modelskill/network.py (1200 lines), model/adapters/_res1d.py (189) and _inp.py (109), with their tests.
  • The extension table and its CI test. That test failed whenever a mikeio1d release added a format.
  • NetworkModelResult.nodes, .data and .time. .nodes published graph integers, which ADR-013 says users should not handle. .data and .time needed the whole network read up front. .period gives the time span from the file header.
  • at=<int> on NodeObservation. It shipped in the 1.4.0a3 alpha only. An integer now raises and points at network.to_networkx().nodes[<int>]["address"].
  • NodeModelResult(data, node=..., name=..., item=...). It takes only a dataset that already carries its location, which is what extract returns.
  • The epanet.inp fixture. mikeio1d reads EPANET reach lengths from the .res, so nothing reads an .inp.

API

mr = ms.NetworkModelResult("model.res1d", item="WaterLevel")
obs = ms.NodeObservation(data, at="node_A")
cc = ms.match(obs, mr)
  • NetworkModelResult takes a path or a mikeio1d.network.Network. A path goes to Network.open, which picks the reader and finds companion files where the format needs them.
  • ms.model_result() builds a NetworkModelResult from either, by type or by .res1d, .res11 or .res extension.
  • A NodeObservation names its location by node name or by (reach, position). position_tol sets how far a position may be from a break point; by default it absorbs rounding only (mikeio1d's 1e-3). from_multiple takes either form as a key, and position_tol.
  • ReachObservation.from_multiple is new.
  • NodeObservation, ReachObservation, NodeModelResult and Comparer have .at: a node name, a (reach, position) break point, or a reach name. Comparer.at is None outside a network. Comparer also has .reach and .position.
  • A comparer sits at its observation's location, and so do its raw model results. Where a model was read is mr.extract(obs).at.
  • The break point coordinate is renamed distance → position. node, reach and position are reserved names.
  • Observation, model result and comparer reprs print the location. A quantity with no unit prints without empty brackets.
  • The extra and the dependency group are renamed networks → network, matching mikeio1d's. Both require mikeio1d[network] >= 1.4.0.

Reading

Building a NetworkModelResult reads no timeseries. extract resolves the observation's location with Network.resolve and reads that location only.

A node observation is resolved with the model's item, so a position snaps onto a break point that carries it. A location that exists but lacks the item gets a "has no data for quantity" error, not "not found".

A ReachObservation reads the break points on its reach that carry the quantity, in one read. They must agree. The one at the lowest position is used.

Chainage tolerance, location lookup and format refusal messages are mikeio1d's.

Bugs fixed

Each has a test that fails on main.

  • Comparer.load coerced the node coordinate with int(). A comparer built from at="node_A" could not be reloaded.
  • A reach comparer was saved without its raw model data. Loading it raised NotImplementedError: Unknown gtype: reach.
  • NodeObservation(df, at=("r1", 24.5)).to_dataframe() raised. It dropped a node coordinate a break point does not have.

An observation built from data that already carries a location now raises if at or reach names another. Before, the argument was dropped silently.

Stacked PRs

Closed unmerged. Each change landed here or in mikeio1d:

PR Where it is now
#685 quantities filter, shared empty frame, once-per-node read mikeio1d #247: Network.open(quantities=), _EMPTY_DATA and the node_data cache in network/_res1d.py. Pinned by the res1d_one_quantity snapshot and TestQuantityFiltering
#694 reach-end gridpoints promoted, boundary removed mikeio1d #247: _build_reach_breakpoints, and the clamped zero-length boundary=True edge in network/_graph.py. Pinned by TestAReachThatDoesNotStartAtZero
#695 EPANET link quantities mikeio1d #247: the synthetic gridpoint duplicated to both ends, Address = str | tuple[str, float], and .resx reach quantities merged onto break points. Its two modelskill-side guards are superseded by _extract_reach and Network.resolve
#698 MIKE+ lookup Split. The MIKE+ half is on the mikeplus-station-lookup branch. ReachObservation.from_multiple and the address keys on from_multiple are here. The to_dataset quantity fix is moot: NetworkModelResult takes its quantities from Network.quantities and no longer calls to_dataset
#699 path constructor Here, rewritten onto Network.open. The extension policy, companion discovery, the cp1252/latin-1 name repair and the CI coverage test are mikeio1d's _policy.py and _companions.py

mikeio1d's code answers both review threads on #695. #679, #680 and #599 are closed separately. #682, #683 and #684 moved to mikeio1d.

Checks

791 passed, 5 skipped, 1 failed locally. ruff check src, ruff format --check, mypy and the doctests in metrics.py and types.py are clean. CI is green. The user guide renders. The notebook runs end to end against mikeio1d 1.4.0. CI does not run it, since it is in SKIP_LIST.

The failure is tests/regression/test_regression_rose.py, which fails on main too. The lockfile pins matplotlib 3.10.9, and the baseline image came from a newer one.

The network tests and the docs build run on Python 3.14. mikeio1d requires < 3.15.

Notes for review

🤖 Generated with Claude Code

jpalm3r and others added 11 commits August 25, 2026 15:30
Reopens the question ecomodeller raised on #694, with the seam set where
NetworkModelResult already draws it: modelskill keeps the comparer-facing
classes, mikeio1d gets the formats, the graph and the Network class itself.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Phase 1 landed as mikeio1d #247, so the decision is no longer a proposal:
the module is built upstream and the Phase 0 snapshots pass against it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A network result file identifies nodes and reaches by ID, but observations are
usually recorded against real-world station names held in the MIKE+ setup
database. Read that sqlite database to resolve a station to the node or reach it
sits on, so observations can be placed without hand-mapping every ID.

The resolver is one class in obs.py rather than a module of its own: everything
MIKE+ specific -- the table names, the join, the locationtype codes, the
encoding of resitemname -- sits in one class body, and its only caller is a few
lines below it. It returns a list of _Station rather than a DataFrame with five
agreed column names, so the contract is in the type.

Dataset variables also gain a long_name attribute, so a quantity keeps its label
once it reaches xarray.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A node observation records whatever the caller addressed it with, so a comparer
built from at="node_A" already carries the name. The model side disagreed: it
recorded the integer the network handed out, and load() coerced the coordinate
back with int(), which raises on a name. A breakpoint was worse off again -- it
carries reach and distance rather than node, and several places tested for
"node" alone.

- One list of the coordinates that say where a timeseries sits, so the three
  places that dropped a subset of them on the way to a dataframe now drop the
  same set. Fixes NodeObservation(df, at=("r1", 24.5)).to_dataframe(), which
  raised because the node coordinate it dropped is not there.
- NodeModelResult takes a node name or a (reach, distance) pair, and records the
  graph integer beside it as node_index. Neither identity nor load reads that
  integer back.
- load() stops re-deriving the location: the coordinates travelled with the
  file. A reach comparer can now be saved and loaded at all, where it used to
  raise NotImplementedError.

The backend is unchanged, so extraction still hands out integers; the next
commit swaps it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The extra is renamed networks -> network, matching mikeio1d's own, so
modelskill[network] and mikeio1d[network] read the same. It shipped in the
1.4.0a3 alpha only, so nothing needs a deprecation. networkx and xarray now
arrive through mikeio1d's extra rather than being named here.

The module is unreleased, so a uv source points at mikeio1d's main branch. Both
entries carry a TODO to swap it for a version floor once mikeio1d releases; per
ADR-013, modelskill 1.4.0 waits for that release.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
modelskill's own topology layer goes: the abstract types, the Res1D adapter, the
per-product constructors, the EPANET companions, the extension policy table and
the .inp reader. All of it now lives in mikeio1d, where the formats and the
fixtures already were (ADR-013). 1503 lines of source, and the CI failure that
came with the extension table -- a mikeio1d release adding a format is no longer
our problem.

NetworkModelResult takes a path or a mikeio1d Network. A path goes to
Network.open, which picks the reader and finds the EPANET companions; refusals
for .out, .resx and the fixture-less formats are upstream's to word now. The
network is used as given rather than deep-copied, which was the largest cost of
building a model result on a river model.

Extraction reads the dataset and consults the topology only to explain a
failure. Two consequences worth knowing:

- A location whose column is entirely NaN now counts as having no data. It used
  to be selected, and produced a match with no points, so the observation
  quietly disappeared; another break point on the reach is used instead.
- Break points with an unknown distance now take part. That is EPANET read
  without its .inp, where no reach has a length and the second break point of
  every reach is unaddressable by distance.

Node lookups delegate to Network.find, so there is one chainage tolerance rather
than a second copy here, and a failed lookup names the near misses instead of
listing the first five aliases in the map.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A network hands out integers to label its graph, and which integer a location
gets depends on the network it was built from and the mikeio1d that built it.
Nothing a user writes should depend on that, so at= now takes a node name or a
(reach, distance) pair, and an integer raises with the recipe for getting the
name back.

The signature shipped in the 1.4.0a3 alpha only, so it goes without a shim. The
guard catches np.int64 as well as int: Comparer.node returns a numpy scalar, and
isinstance(np.int64(5), int) is False.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The user guide loses the sections that documented the topology layer -- building
a network, the companions, selective loading, inspecting the graph, find and
recall -- and gains a short section saying where the network comes from and how
to hand modelskill a path. What stays is the part modelskill still owns: the
skill assessment workflow and the MIKE+ database lookup. 587 lines to 197.

The notebook addressed its observations by graph integer and highlighted
junctions by reading a node attribute that upstream moved to edges; both are
fixed, and it runs end to end. It is in the notebook SKIP_LIST, so CI would not
have caught either.

ADR-012 is narrowed rather than superseded: the constructors and the extension
tables it argues for are mikeio1d's now, and the reasoning still applies there.
ADR-010's open question about version constraints is answered for this feature.

The res1d mapping diagram went with the section that explained it. The fixtures
for the formats whose tests moved are kept, and the testdata README now says
nothing here reads them.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The alpha wrote the graph integer into the node coordinate. Nothing on the load
path derives a location from it any more, so such a file still loads and skills
-- but a regression there would be silent, so the fixture is a file the alpha
actually wrote rather than one built to look like it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Comment thread src/modelskill/obs.py Fixed
Comment thread src/modelskill/obs.py Fixed
Comment thread src/modelskill/obs.py Fixed
jpalm3r and others added 2 commits August 25, 2026 16:59
ADR-013 was written as a proposal and parts of it no longer describe what was
built: mikeio1d collapsed from_mike and from_epanet into Network.open, the extra
is called network, and it named a fixture this repository does not have. The
snapshot paragraphs said the same thing twice, once in each tense.

ADR-012's note described its own position in the document. It now sits under the
status line and says what moved.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Inverted constructions, closing kickers and sentences that restate the one before
them. Stacked clauses split into separate sentences.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@ecomodeller ecomodeller added the enhancement New feature or request label Aug 27, 2026
jpalm3r and others added 12 commits September 3, 2026 09:17
Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
A reach-gtype comparer fell through the gtype dispatch to
NotImplementedError, even though the node branch's body is already
correct for it: _drop_scalar_coords drops reach and distance too.
A breakpoint that exists but holds nothing for the selected quantity
surfaced as "All datetime indices must be non-empty" from matching,
because every timestep was dropped as NaN. The reach path already
treats an all-NaN column as no data; the node path now does too.
The example paired breakpoint 21.285 on 94l1 with a WaterLevel model
result, but that breakpoint only carries Discharge, so the block could
not run -- and its plain python fence kept quarto from ever executing
it. Point it at 42.57, which does carry WaterLevel, and make the fence
executable so the render catches this next time.
The source filter was an unanchored LIKE '%name%' on tsfilename, so it
could not tell calib.dfs0 from my_calib.dfs0 -- it still reported the
item as ambiguous and told the caller to pass the source they had just
passed -- and an underscore in a real file name acted as a wildcard.
Compare file names in pandas instead, splitting on the Windows
separator the database stores.
Every row matching the requested item names was classified before any
quantity or kind filter, so a station whose locationtype is not a place
in the network -- a rain gauge or a catchment registered in the same
file -- failed the whole resolve, and on_missing="skip" could not help.
Such stations are now left out, and named in the error only when they
were what the caller asked for.
Its name was fed to the station resolver as a database item filter, so
supplying metadata failed unless the name happened to match the
database's own -- which the docstring never said it had to. A string
selects; a Quantity describes.
@jpalm3r
jpalm3r marked this pull request as ready for review September 3, 2026 12:44
@jpalm3r
jpalm3r requested a review from ecomodeller as a code owner September 3, 2026 12:44
_coords.py is a module of coordinate types. location_from_coords was the
one piece of logic in it, and what it returns is NodeObservation.at's type
-- a node name or a (reach, distance) pair. Both callers already import
obs.py, so the move costs no new import edge.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
jpalm3r and others added 20 commits September 30, 2026 15:03
Matches mikeio1d, which names the along-reach offset position.
Comparer.distance becomes Comparer.position. Only the 1.4.0 alphas
stored 'distance', so no loading shim is kept for it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
1.4.0 does not have to load network comparers saved before it. The fixture
was written by main after the alpha tag, and its tests loaded a netcdf file
without touching mikeio1d.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
ADR 013 and the roadmap still waited for a mikeio1d release and measured
the at= change against the alpha. mikeio1d 1.4.0 is out and is the
release modelskill 1.4.0 requires.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Observations address locations by name or (reach, position), so the
notebook no longer teaches graph_node integers, and says position
instead of distance.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The user guide's ReachObservation example needs a discharge series;
the existing sensors all hold water level.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Sensor 1 sits at node 98 and sensor 2 at break point (117l1, 48.7);
the reach example scores the discharge sensor against Discharge.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The not-found error points to network.addresses(); it lists no
nearby break points. A reach observation uses the lowest-position
break point with data, not an arbitrary one.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
mikeio1d 1.4.0 takes EPANET reach lengths from the .res and refuses an
.inp companion.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A position then snaps onto a breakpoint that carries the item. A location
lacking it still gets the 'has no data for quantity' message.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
NodeObservation(position_tol=...) is passed to Network.resolve, so a
measured chainage can snap onto the nearest breakpoint carrying the item.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A reach comparer picked up the first model's breakpoint position, since
the observation had no position coordinate to win the merge. The
comparer now sits where the observation sits, for nodes and reaches; each
model's breakpoint stays on raw_mod_data.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
ReachObservation had no .at and a reach Comparer.at returned None. Both
now return the reach name, so obs.at == cmp.at holds for reaches as it
does for nodes and breakpoints.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Comment thread src/modelskill/utils.py
from collections.abc import Hashable, Iterable

_RESERVED_NAMES = ["Observation", "time", "x", "y", "z"]
_RESERVED_NAMES = ["Observation", "time", "x", "y", "z", "node", "reach", "position"]
jpalm3r and others added 5 commits October 1, 2026 17:32
The module-level importorskip already skips the file wherever mikeio1d
cannot be installed, so the four markers never changed an outcome.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Each checked only that a match produced points or a node name, on a
setup TestValuesReachTheComparer already scores to zero; the model
result trim test repeats the observation one on the same trim.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Both built the same reach match; the .at assert now sits with the
other reach comparer location asserts.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A comparer sits where its observation sits, but its raw model results kept
the breakpoint each model was read at. Save and load wrote one location per
comparer, so that breakpoint turned into the comparer's location after a
round trip. Raw model results now take the observation's location when
matched; where a model was read is NetworkModelResult.extract(obs).at.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A comparer's raw model results now sit at the observation's location, so
the guide points to NetworkModelResult.extract for the break point read.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

@ecomodeller ecomodeller left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work. Moving the topology and format handling to mikeio1d removes about 1,500 lines here, and what is left is small and easy to follow: building a NetworkModelResult reads nothing, extract reads only the locations the observations name, and users only ever handle node names and (reach, position) pairs. Reading location from the coordinates in one place (network_gtype, NETWORK_LOCATION_COORDS) fixes the save/load bugs where they came from, and the tests now run against a real result file instead of a hand-built network.

One correction to the description: Collection_systems_network is now skipped only on Python ≥ 3.15, so the notebooks workflow (Python 3.14) will run it, either on the push to main or through a manual run of that workflow. It has not run on this PR yet.

Optional follow-ups, none of them needed for this PR:

  • network_location and _at_from_coords could become one function in timeseries/_coords.py that coerces to str/float. Then comparison and model/network would no longer import a private helper from obs.
  • The node branch of NetworkModelResult.extract could move to _extract_node, next to _extract_reach.
  • network_cali.res11 and swmm.out are no longer read here and could be removed in a separate PR.
  • test_init_fails_with_unsupported_type can assert TypeError alone now.

@jpalm3r
jpalm3r merged commit 2e160ce into main Oct 2, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Include tolerance as parameter in find Hide internal network integer IDs from public API where possible (future improvement)

2 participants