Skip to content

feat(stream): propagate schemas through pipelines - #421

Open
hmgomes wants to merge 3 commits into
adaptive-machine-learning:mainfrom
hmgomes:feat/pipeline-schema
Open

hmgomes wants to merge 3 commits into
adaptive-machine-learning:mainfrom
hmgomes:feat/pipeline-schema

Conversation

@hmgomes

@hmgomes hmgomes commented Sep 19, 2026

Copy link
Copy Markdown
Collaborator

Pipelines could not describe their own data. ClassifierPipeline inherits Classifier but no
__init__ in the chain ever called Classifier.__init__, so pipeline.schema raised
AttributeError; nothing aggregated the schemas of the elements; and MOATransformer.get_schema()
returned the schema it was given rather than the one its filter emits. Users threaded schemas
by hand through every example in notebooks/07_pipelines.ipynb, and nothing checked the result.

That last point is the one with teeth. With a filter that changes the feature set, the transformer
reported the wrong shape and no error was raised:

t = MOATransformer(schema=stream.get_schema(), moa_filter=HashingTrickFilter(), CLI="-d 3")
len(t.transform_instance(instance).x)      # 3  -- what it actually emits
t.get_schema().get_num_attributes()        # 6  -- what it told the next element

What changed

  • Pipelines report their schemas. BasePipeline.get_input_schema() returns what the pipeline
    consumes, get_schema() what it emits; both walk the elements, so a nested pipeline composes.
  • schema and random_seed now exist on ClassifierPipeline/RegressorPipeline, satisfying
    the base-class contract. schema is a property, so it tracks elements added after construction,
    and it reports the input schema — the instances handed to train/predict.
  • Schema.is_compatible_with() and Schema.describe_difference() compare attribute structure
    and target while ignoring the dataset name, which normally differs downstream of a filter.
  • Adding an element whose schema does not fit now raises ValueError naming the differences,
    rather than being accepted silently. This applies to elements passed to the constructor as well
    as appended ones. validate_schema=False opts out.
  • MOATransformer distinguishes input from output schema and derives the output one, preferring
    the header the transformed instance carries over the filter's own — some filters never publish
    the latter.

Design decisions

Schema.is_compatible_with() rather than __eq__. Defining __eq__ would have made Schema
unhashable unless __hash__ were defined alongside, and "equal" is the wrong word for what
pipelines need: a filter's output legitimately carries a different relation name while remaining
interchangeable for a learner. A named predicate says what it checks, and describe_difference()
turns a rejection into a message that names the mismatch.

Output-schema derivation prefers the transformed instance's own header over filter.getHeader().
Filters that change the feature set publish a header only after an instance has passed through,
and some (HashingTrickFilter) never publish one at all while still stamping it on what they
emit. Derivation is therefore lazy and retried on the first transform_instance.

PipelineElement.get_schema()/get_input_schema() have defaults returning None rather than
being abstract, so existing third-party PipelineElement implementations keep working.

Behaviour changes

No public API is removed, but four behaviours differ. None uses a breaking-change marker; they are
described here instead.

  • MOATransformer.get_schema() returns the filter's output schema. For every feature-preserving
    filter (normalisation, standardisation, added noise) that is the same value as before.
  • BasePipeline.__str__ no longer leaves a trailing " | ".
  • Adding a schema-incompatible element raises ValueError where it was previously accepted. Code
    relying on the old silence needs validate_schema=False.
  • MOATransformer(moa_filter=...) without a schema raises a clear RuntimeError instead of a Java
    NullPointerException. That path never worked; only the error improved.

Validation

tests/test_pipeline.py is new, with 78 tests — the module previously had no pytest coverage
at all, its only exercise being 07_pipelines.ipynb, which nbmake runs without checking output.

CapyMOA has few transformers, so the filter coverage is exhaustive rather than representative:
every moa.streams.filters entry reachable through MOATransformer, parameterised across
ElectricityTiny, CovtypeTiny and FriedTiny (6 numeric attributes over 2 classes; 54
attributes of which 44 nominal, over 7 classes; and 10 numeric attributes for regression).

The central assertion, over every usable filter and stream: the schema a transformer reports
must equal the number of features the instances it emits actually carry.
That is the invariant
whose absence caused the bug above.

Mismatched wiring is checked across five stream pairings, asserting that the message names the
attribute count, the class labels, and the classification/regression split. Two further tests
assert equivalence rather than a pinned constant — a ClassifierPipeline, with and without a
transformer, must score exactly what the hand-written test-then-train loop it replaces scores,
which is what guards the schema swap inside transform_instance.

Two limits are asserted rather than left to be discovered:

  • A feature-reducing transformer cannot be detected before the first instance flows, because MOA
    publishes a filter's output header only after it has seen one. A pipeline assembled and never
    run therefore accepts a mismatched learner.
  • Four filters (RemoveDiscreteAttributeFilter, RBFFilter, ReLUFilter,
    RandomProjectionFilter) raise inside MOA before producing a usable instance, identically on
    main. They are documented in a test that fails if one starts working.

07_pipelines.ipynb gains section 7.8, explaining consumed-versus-emitted schemas, a
transformer that changes the feature set, the errors raised when schemas do not line up, and the
escape hatch. The error cells catch the ValueError and print it, so the notebook shows the real
message without failing under nbmake. Purely additive: 221 insertions, no deletions.

Two pre-existing stored outputs were updated for the __str__ change. Every stored value here,
new and updated, was produced by executing the code rather than typed — which matters because
nbsphinx will not re-execute a notebook that already carries outputs. Accuracy figures are
untouched, since nothing in this change affects learning.

All four CI jobs were reproduced locally: ruff format and lint clean, 378 pytest passed,
135 doctests passed, 22 notebooks passed under NB_FAST, and invoke docs.build succeeded with
zero warnings.

Follow-ups, deliberately not in this PR

  • SelectAttributesFilter is not a moa.streams.filters.StreamFilter, so FilteredQueueStream
    rejects it and it cannot be used in a pipeline at all. The hashing trick is currently the only
    reachable filter that changes the feature set.
  • MOATransformer builds its filter CLI as -f (<creation string>), which MOA fails to parse when
    the filter's own options contain commas.
  • Evaluators are still constructed from stream.get_schema() rather than the pipeline's schema.
  • Anomaly detection and other task types, and the scikit-learn-style _repr_html_, are the
    remaining workstreams of the issue below.
  • Drift-detector ergonomics inside pipelines, and an event mechanism so detectors can notify
    rather than be polled, are tracked separately as backlog#158 and backlog#157.

Addresses adaptive-machine-learning/backlog#154
Addresses adaptive-machine-learning/backlog#87

Assisted-by: claude-code:claude-opus-5

Pipelines could not describe their own data. ClassifierPipeline inherits
Classifier but no __init__ in the chain called Classifier.__init__, so
pipeline.schema raised AttributeError; nothing aggregated element schemas;
and MOATransformer.get_schema() returned the schema it was given rather
than the one its filter emits. With a filter that changes the feature set
the transformer reported 6 attributes while emitting 3, silently.

- BasePipeline gains get_input_schema() and get_schema(), walking elements
  so nested pipelines compose.
- ClassifierPipeline/RegressorPipeline now carry schema and random_seed,
  satisfying the base-class contract. schema is a property reporting the
  input schema, so it tracks elements added later.
- Schema.is_compatible_with() and Schema.describe_difference() compare
  attribute structure and target, ignoring the dataset name.
- Adding an element whose schema does not fit raises ValueError naming the
  differences; validate_schema=False opts out.
- MOATransformer separates input from output schema, deriving the output
  from the header the transformed instance carries, since some filters
  never publish one on the filter object.

Behaviour changes, described here rather than marked: get_schema() on a
transformer now reports the output schema (unchanged for every
feature-preserving filter); __str__ drops its trailing separator;
incompatible elements raise instead of being accepted; and constructing a
MOATransformer without a schema raises a clear RuntimeError instead of a
Java NullPointerException.

Adds tests/test_pipeline.py; the module previously had no pytest coverage.

Addresses adaptive-machine-learning/backlog#154
Addresses adaptive-machine-learning/backlog#87

Assisted-by: claude-code:claude-opus-5
CapyMOA has few transformers, so the filter inventory can be complete
rather than representative. Parameterised over every moa.streams.filters
entry reachable through MOATransformer, across ElectricityTiny,
CovtypeTiny and FriedTiny.

- The central invariant, over every usable filter and stream: the schema
  a transformer reports must match the number of features the instances
  it emits actually carry.
- Feature-preserving filters stay compatible with their input;
  HashingTrickFilter reports its new, smaller shape.
- Mismatched wiring is refused across five stream pairings, with tests
  that the message names the attribute count, the class labels and the
  classification/regression split.
- A learner placed after a feature-reducing transformer is rejected.
- No false positives: correctly wired pipelines build without complaint,
  and validate_schema=False still accepts every mismatch.

Four filters (RemoveDiscreteAttributeFilter, RBFFilter, ReLUFilter,
RandomProjectionFilter) raise inside MOA before producing a usable
instance. Verified identical on upstream/main, so they are documented in
a test rather than fixed here; if one starts working the test fails and
says where to move it.

Note RemoveDiscreteAttributeFilter fails only when the instance's `x` is
read: transform_instance() returns a headerless instance quite happily,
and Instance.x is lazy.

Addresses adaptive-machine-learning/backlog#154
Addresses adaptive-machine-learning/backlog#87

Assisted-by: claude-code:claude-opus-5
Adds section 7.8 to 07_pipelines.ipynb: the difference between what a
pipeline consumes and what it emits, a transformer that changes the
feature set, the errors raised when schemas do not line up, and the
escape hatch.

The error cells catch the ValueError and print it, so the notebook shows
the message a user would see without failing under nbmake.

Purely additive -- 221 insertions, no deletions. Stored outputs were
produced by executing the cells, not written by hand, which matters
because nbsphinx will not re-execute a notebook that already has them.

Addresses adaptive-machine-learning/backlog#154

Assisted-by: claude-code:claude-opus-5
@hmgomes
hmgomes force-pushed the feat/pipeline-schema branch from f24628b to 24edf5a Compare September 22, 2026 04:08

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant