Resource-dispatched `constraints_capacity`

[espenbodal]

Problem. The capacity constraints of a node are selected by the node type only. Since 0.9.5 an extension package can attach physics to a resource instead of a node type and add constraints to every node carrying it through constraints_resource. It cannot, however, replace one of the standard constraints that way. Example from a power-flow package under development: its resources carry reactive power, and for a load the installed capacity is the apparent power the connection is rated for, while cap_use is the active power that drives the variable OPEX. The sink balance therefore has to read cap_use + sink_deficit == cap_inst · pf + sink_surplus with the power factor pf of the load, a node-specific parameter that the package provides as ExtensionData on the sink. The only way to get there today is a copy of RefSink with one additional field, and the same copying is needed for every other reference node whose capacity equation the resource changes. This defeats the purpose of the resource hooks: a RefSink from EMB, or a node from another extension package, cannot be used as a load in such a model.

Proposal. Split constraints_capacity into an entry point and resource-aware methods, selected by a boolean trait on the resource and an optional override on the node:

# structures/resource.jl
is_capacity_resource(p::Resource) = false          # "p brings its own capacity constraints"

# structures/data.jl
struct CapacityResource{T} <: ExtensionData        # explicit choice of the dispatch object
    resource::T                                    # untyped: a Resource or a composite type
end

# structures/node.jl
function capacity_resource(n::Node)
    𝒟 = filter(d -> isa(d, CapacityResource), node_data(n))
    isempty(𝒟) || return first(𝒟).resource
    𝒫 = unique(vcat(inputs(n), outputs(n)))
    𝒫ᶜᵃᵖ = filter(is_capacity_resource, 𝒫)
    return isempty(𝒫ᶜᵃᵖ) ? first(𝒫) : first(𝒫ᶜᵃᵖ)
end

# constraint_functions.jl
constraints_capacity(m, n::Node, 𝒯, modeltype) =
    constraints_capacity(m, n, 𝒯, capacity_resource(n), modeltype)
constraints_capacity(m, n::Node, 𝒯, p::Resource, modeltype)    = <current Node body>
constraints_capacity(m, n::Storage, 𝒯, p::Resource, modeltype) = <current Storage body>
constraints_capacity(m, n::Sink, 𝒯, p::Resource, modeltype)    = <current Sink body>

# checks.jl, called from check_elements for nodes with has_capacity(n)
check_capacity_resource(n::Node)   # ≤ 1 CapacityResource; flagged resources of ≤ 1 family

An extension package declares is_capacity_resource(::ElectricPower) = true and writes, for example, constraints_capacity(m, n::Sink, 𝒯, p::ElectricPower, modeltype) with the balance above, reading the power factor from the data of the node. The resource instance is passed, not its type, because the instance carries the resource-level parameters (voltage limits, the linearisation of the capacity limits); node-specific parameters such as the power factor of a load stay in the ExtensionData of the node.

Why a boolean and not a rank. A node usually carries several resources, so the entry point needs a rule for which one to pass on. The first design that comes to mind is a priority: an integer resource_rank(p) with default 0, raised by extension packages for their resources, and argmax over the resources of the node. It is one line and it also lets a package declare precedence when a node carries advanced resources from two packages. That last property is the reason to reject it. A node whose capacity is governed by two different physics (a heat pump with an apparent-power limit on the electric side and a temperature-dependent limit on the heat side) has no automatic answer, and a rank would resolve it silently and arbitrarily, depending on which number each package happened to pick. The trait therefore only separates resources that bring their own capacity constraints from ordinary carriers, and never decides between two of them. With a boolean a collision between packages is impossible by construction, and the conflict becomes an explicit error in the consistency log with an explicit way out, described next.

Conflicts and combination. A node has one cap_inst and one cap_use in one unit, so exactly one resource owns the capacity equation of a node. Every other advanced resource adds its limits formulated on its own flow variables through constraints_resource, which already runs per resource family on every node; that combination needs no code that knows both packages. The conflict case is two resources of different families that both need to replace the equation. check_capacity_resource rejects such a node unless it carries a CapacityResource, and because the payload of CapacityResource is passed through untyped, a package extension that depends on both packages (the ext/ mechanism every EMX package already uses for EnergyModelsInvestments, known as a glue package before Julia 1.9) can combine the physics in one method on a composite type it owns:

# A heat package owns the capacity equation of a heat pump (RefNetworkNode, capacity in MW heat)
is_capacity_resource(::TemperatureHeat) = true
function constraints_capacity(m, n::NetworkNode, 𝒯, p::TemperatureHeat, modeltype)
    @constraint(m, [t ∈ 𝒯], m[:cap_use][n, t] ≤ m[:cap_inst][n, t] * derating(p, t))
    constraints_capacity_installed(m, n, 𝒯, modeltype)
end
# A power-flow package adds the electric side on the flows of its resource through the hook
function constraints_resource(m, n::NetworkNode, 𝒯, 𝒫::Vector{<:ElectricPower}, modeltype)
    @constraint(m, [t ∈ 𝒯, p ∈ intersect(inputs(n), 𝒫)],
        m[:flow_in][n, t, p] / power_factor(n, p) ≤ electric_rating(n, p, t)   # both from node data
    )
end
# If both packages flag their resource, the modeller resolves the conflict with a composite
# type owned by a package extension that is loaded when both packages are present
struct HeatPumpCapacity{H,P}; heat::H; power::P; end
function constraints_capacity(m, n::NetworkNode, 𝒯, c::HeatPumpCapacity, modeltype)
    # thermal derating, electric rating and their coupling written once; installed called once
end
RefNetworkNode(..., [CapacityResource(HeatPumpCapacity(Heat, Power))])

Why non-breaking.

• create_node keeps calling the four-argument method, so every existing four-argument method on a node type (EnergyModelsInvestments, EnergyModelsRenewableProducers, EnergyModelsHydrogen, EnergyModelsHeat) keeps precedence over the new entry point, because it is more specific in the node argument.
• Without flagged resources the dispatch object is the first input or output resource and the five-argument fallbacks are the current bodies, which ignore it, so the constraints are identical.
• CapacityResource is a new data type; existing data is untouched. The investment extension dispatches on constraints_capacity_installed, which is untouched.
• A node with neither inputs nor outputs has no dispatch object; no such node type exists in the ecosystem, and check_elements would report it before the model is built.

Tests (in the accompanying pull request). With two test resource types that both declare is_capacity_resource = true: a node carrying both is rejected by the checks with a message naming the two families; a node with two CapacityResource entries is rejected; a CapacityResource with a plain resource selects the standard fallback; a CapacityResource with a composite payload reaches a method defined for that composite type. With one test resource type: a five-argument method on Sink changes the sink balance for a RefSink carrying it, and the existing suite is unchanged for resources with the default trait.

[JulStraus]

Not looked into it rather detailed yet, I think that #106 and #107 are potentially things that should be implemented first independently. Regarding the topic here I have the following questions:

1. Why did you use ExtensionData? Due to the potential for multiple resources that may impact the capacity?
2. How are we solving potential method ambiguities? Each additional method to constraints_capacity would here require additional methods.

Point 2 is probably not too relevant as we have to provide a method anyhow, but it can be confusing for a user.

[espenbodal]

1. Yes, ExtensionData is an option used to explicitly tell the model which resource formulation to use if there are multiple resources that impact the capacity, such that it cannot be inferred from the inputs and outputs through the capacity_resource-function. The checks should check the number of resources with this function for each node, and throw an error if there are more than one given it is not specified in the additional data which one to prioritize. The user is forced to choose which resource to prioritize for the capacity formulation or define a new composite type and related method that takes both resources into account.

[espenbodal]

2. Ambiguity can be a problem for new nodes in combination with new resources that override the capacity_constraint ( is_capacity_resource = true). One approach to limit confusion can be to instruct the user to not the four argument version in EMB and include comprehensive checks.

More details on how this can work:

Precedence. create_node calls the four-argument constraints_capacity(m, n, 𝒯, modeltype). A method with four arguments for a node type is more specific than the new entry point on Node and therefore takes precedence: the entry point, capacity_resource(n) and all five-argument methods are not reached for that node type. This is what makes the proposal non-breaking, and it is the right default, since a node type with its own capacity equation knows its own unit. The consequence is that a node of such a type carrying a capacity resource gets the node type's constraints, not the resource's, and a CapacityResource entry on the node has no effect either.

Convention for new node types. A node type whose capacity constraints should be combinable with capacity resources provides the five-argument form, constraints_capacity(m, n::MyNode, 𝒯, p::Resource, modeltype), and leaves the four-argument entry point to EnergyModelsBase. With ordinary resources the node type's method is called as before. With a capacity resource from another package, the node type's method and the resource's method are equally specific, which is an ambiguity error in Julia: nobody wins silently, and the resolution is a method for the combination, constraints_capacity(m, n::MyNode, 𝒯, p::TheResource, modeltype), in a package extension that depends on both packages. Existing packages can migrate to the five-argument form at their own pace; the four-argument form keeps working.

Checks. check_capacity_resource(n, 𝒯, modeltype) runs from check_elements for every node except Availability. Every finding is an entry in the consistency log, so create_model stops with AssertionError("Inconsistent case data.") after all checks have run and all problems are reported at once, before any constraint is built. The method lookups use which on the four- and five-argument signatures and type intersection against the method table (methods(f, sig) returns an empty list for ambiguous concrete signatures); they assume that no method restricts the type of the first argument, the JuMP model, which holds for all methods in EnergyModelsBase.

Situation on node n	Without the check	Reported	Resolution
More than one CapacityResource in data	the first one is used silently	"Only one CapacityResource can be added to each node."	keep one
Capacity resources of different families, no CapacityResource	the first flagged resource is used silently	the families are named; a CapacityResource is requested	add CapacityResource(p) to pick one resource, or CapacityResource(obj) with a composite type and its method
Node type has a four-argument method and n has a CapacityResource	the data is ignored silently	the node type and the ignored data are named	remove the data, or move the node type to the five-argument form so that the data takes effect
Node type has a four-argument method and n carries a capacity resource with a five-argument method that would otherwise apply	the node type's constraints are built, the resource's are missing silently	the node type, the resource and the bypassed method are named	move the node type to the five-argument form (which leads to the ambiguity row), or provide a method for the combination
Entry point reached, CapacityResource payload has no method	MethodError when the model is built	the node and the payload are named	provide constraints_capacity(m, n::<NodeType>, 𝒯, p::<PayloadType>, modeltype)
Entry point reached, a five-argument node-type method and a resource method are equally specific	MethodError: ... is ambiguous when the model is built	the two competing methods are named	provide a method for the combination of the node type and the resource, typically in a package extension that depends on both packages

[JulStraus]

I do not see an issue with concise checks to be honest. That can be done via check_node_data, i.e., only a single version is allowed. All checks related to the resources should also be from there, at least in my opinion. That would require us to define an abstract type CapacityResource <: ExtensionData where people can then create subtypes, e.g., HeatPumpCapacity or something else and we limit the number to 1 as done for EmissionsData.

We can however never move for the default to the 5 argument method in create_node. This will then definitely lead to method ambiguities if people do not explicitly adjust for that approach in additional packages. This seems to me to be quite nightmarish with respect to extensions in the individual packages.

In addition, it will not work for any Node which has a 4 argument method for constraints_capacity. In this situation, as pointed out, it will utilize the 4 argument method without failing for the user. And creating here a check is in my opinion annoying as I think that relying here on which or @which is rather unsafe.

That implies to me that while this extension is not breaking, it will however lead to silent errors in cases where a user dispatch on constraint_capacity in an extension within a 4 argument function.