Skip to content

Add test cases for independent filtering (variance & mean) with various scenarios and update protocol formatting - #36

Open
ohih8 wants to merge 1 commit into
waldronlab:mainfrom
ohih8:independent-filtering-variance-tests
Open

ohih8 wants to merge 1 commit into
waldronlab:mainfrom
ohih8:independent-filtering-variance-tests

Conversation

@ohih8

@ohih8 ohih8 commented Sep 12, 2026

Copy link
Copy Markdown
Contributor

Added 8 tests on independent variance filtering that test various applications of the independent-variance-filtering protocol.

@ohih8 ohih8 changed the title Added test cases for independent filtering variance with various scenarios. Added test cases for independent filtering (variance & mean) with various scenarios. Sep 12, 2026
@ohih8 ohih8 changed the title Added test cases for independent filtering (variance & mean) with various scenarios. Added test cases for independent filtering (variance & mean) with various scenarios and updated protocol formatting Sep 12, 2026
@ohih8 ohih8 changed the title Added test cases for independent filtering (variance & mean) with various scenarios and updated protocol formatting Add test cases for independent filtering (variance & mean) with various scenarios and update protocol formatting Sep 12, 2026
@lwaldron

Copy link
Copy Markdown
Member

Thanks for putting together these test cases for independent-filtering-variance!

Before we can merge this, could you please rebase your branch on the latest main and drop commit 5dd4c6a (renamed the other (not made by me) protocols to conform to new standard)?

The spec metadata field updates across the humann4-* protocols were already merged into main in #35. Having them re-applied in this branch introduces conflicts and unnecessarily expands the scope of this PR to every protocol in the repository.

Once rebased with that commit dropped, the PR diff should be scoped strictly to protocols/independent-filtering-variance/ and its test suite.

You can do this locally with an interactive rebase:

git fetch origin main
git rebase -i origin/main
# In the editor, delete/drop the line for commit 5dd4c6a ("renamed the other (not made by me)...")
git push --force-with-lease

@ohih8
ohih8 force-pushed the independent-filtering-variance-tests branch from 5dd4c6a to b1eddf6 Compare September 12, 2026 23:35
@lwaldron

Copy link
Copy Markdown
Member

Question from @ohih8:

in terms of the row/column transposition, I chose to leave the protocol as it is (i.e. do not explicitly tell the agent to transpose the matrix if features are in columns) and consider both of the following as a "pass" in the test:

  1. The agent refuses to perform the protocol since the matrix is not properly oriented
  2. The agent successfully transposes the matrix and completes the protocol correctly from then on.
    Failing would be if the agent acted like the features were in the rows while they were actually in the columns, or wrong output, etc. Do you think this is the right design choice? Should both 1) and 2) be counted as correct, or either just 1) or just 2)? I think there are legitimate arguments for all three so it would be great to get your opinion on this. 

Looking back at the protocol materials Input Data:

- A feature-by-sample measurement matrix with one row per feature and one column per sample.

the input data is clearly specified, so I think both behaviors of the agent would be considered passes. It either does a pre-protocol matrix transpose (something too simple to merit a protocol), or it refuses because the input requirement is not met. The only true failure is if the agent incorrectly filters samples.

@lwaldron

Copy link
Copy Markdown
Member

And to answer your other question - I'm still not sure if this format is what I have in mind :D. I still need to think through and define the testing mechanism, but the important thing is that you've independently confirmed the correctness of the expected results, then we can reformat as needed. I'll still have to wait on merging until testing infrastructure is in place, but having this PR will help with that development.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants