feat: multi-input multi-resource type operators

[michael-johnston]

This PR introduces multi-input multi-resource type operators.

It also changes the type of the operator functions parameters parameter to be instances of the operators parameter model instead of a dictionary (answers #647)

Example new function definition.

def operator_function(my_param_one: OperatorInputType, 
                 another_resource_param: OperatorInputType, 
                 operationInfo: FunctionOperationInfo=None, 
                 parameters:GenericOperatorParameters= GenericOperatorParameters())
  ...

Note: Here we use OperatorInputType to denote the allowed input types e.g. DiscoverySpace (see more below) and GenericOperatorParameters as a placeholder for the specific parameter model of the operator

Prior to this PR operators could only work on a single input resource, which had to be of type DiscoverySpace and named discoverySpace. Further the parameters for the operator were passed as keyword args i.e.

operator_fn(discoverySpace:DiscoverySpace, operationInfo:FunctionOperationInfo=None, **kwargs)

Operator Input Types

ADO resources that can be operated on are now associated with a "rich" type that will be passed to the operator. In the PR we have two resources that can be operated on

• DiscoverySpaceResource -> DiscoverySpace (rich type) <- As it is now
• DataContainerResource -> DataContainerResource (rich type) <- New resource type and associated rich type

Note: Its a particular property of DataContainerResource that it can be its own rich type.

Setting inputs in operation.yaml

The PR removes the DiscoveryOperationConfiguration spaces field and introduce the more general input field.

before

spaces:  # A list of resource ids
    - space_id_one
    - space_id_two # Note: technically you could have more than one id, but nothing could use it

now, for an operator with two DiscoverySpace input params called space_one and space_two

inputs: 
    space_one: 
          identifier: space_id_one
          kind: DiscoverySpace
    space_two:
          identifier: space_id_two
           kind: DiscoverySpace

In practice kind can be inferred when the YAML is read so users can omit it.

For existing operators that previously had a YAML section like

spaces: 
    - space-1234-abcd

Users will now set

inputs: 
   discoverySpace:
        identifier: space-1234-abcd

Non-breaking changes

• The spaces field is kept as a computed field, so any code relying on querying it in metastore does not need to be changed
• All older operation resources are updated to have the new input field if read. Not breaking
• All existing operators resource parameters i.e. discoverySpace: DiscoverySpace, is supported out-of-box. No change needed
• If users pass the spaces field in YAML it will be auto-updated to new format. This will only be valid for operators talking a single space with a parameter called discoverySpace (all existing operators)

Breaking changes

• The parameters field of all operators has been updated to be an instance of the operator class not keyword args. Old operators (pre this change) cannot be used with this change. Scripts or operators calling operator functions directly also must be updated.

Code Changes

The code changes fall into three groups:

1. Updates to allow operators to have multiple resource inputs of multiple types and for operations to store them (new input field, models for that field, validators are parsers for operator signature etc.)
2. Updates in orchestration flow to handle multiple inputs (instead of just looking for spaces[0]), including storing relationships, and handling cases where the rich resource type does not provide a way to acces the metastore
3. Updates due to change to use Operators parameter class and introduction of GenerericOperatorConfiguration (changing signatures, changing plugin versions etc).

[michael-johnston]

@AlessandroPomponio docs to be updated once the implementation is deemed good enough.

[AlessandroPomponio]

Initial set of comments

[AlessandroPomponio]

I find this function to be a bit weird: it returns the identifier of a parameter of kind kind if:

• It's the only one matching the kind (no check on whether it was set or not)
• It's the only one unset matching the kind

What I mean with this is that this function is very specific but the name doesn't really convey it. This part of the docstring does a lot of heavy lifting:

When several inputs share kind, prefers an input not yet present in operation_data["inputs"].

This function is used only in one place for now and it's private, so I'm not sure if I'd keep this separate, given how many caveats it has to be understandable

[michael-johnston]

This exists due to the current implementation of `--with

before changing lets figure out how with will work.