Skip to content

Real-world testing: run fixture3 on a matrix of real projects #4

Description

@tartakovsky

What

Run fixture3 against real projects (not just the self-suite and the fake-project) and capture every failure mode.

Why

The self-suite proves fixture3 can verify itself. The fake-project proves it can run an external workflow. Neither tells us how well it survives real-world fixture trees, real CI environments, real review flows. Every new real project exposes assumptions baked into the manifest schema, the diff format, or the doctor checks.

Suggested matrix

  • A Next.js project using fixture3 for snapshot testing route handlers.
  • A Python service using fixture3 for end-to-end output snapshots.
  • A Rust workspace using fixture3 alongside cargo test.
  • An ML pipeline (Python or Rust) where the suite output is large JSON.
  • A monorepo with multiple fixture3.yaml files in subdirectories.
  • A repo that wants fixture3 to gate code review without blocking.

What to track per project

  • Did the manifest schema cover the project's needs, or did we need a feature we don't have?
  • Were the JSON diffs readable in code review?
  • Did doctor catch real misconfigurations or miss them?
  • Was the approval flow ergonomic, or did it require workarounds?
  • Did the reducer help, hurt, or get ignored?

Done

A document listing every project, what worked, what didn't. Each ergonomic gap becomes a follow-up feature-request issue.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    good first issueGood for newcomershelp wantedExtra attention is neededtestingReal-world / corpus / smoke testing

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions