Skip to content

CI: a skipped integration_tests step reports the job as green #1102

Description

@randy-cro

Problem

integration_tests is diff-gated (.github/workflows/test.yml):

- uses: technote-space/get-diff-action@v6.1.2
  with:
    PATTERNS: |
      **/**.sol
      **/**.go
      go.mod
      go.sum
      tests/integration_tests/**
- name: Run integration tests
  run: make run-integration-tests
  if: env.GIT_DIFF

When GIT_DIFF is empty the step is skipped, and a skipped step still makes the job report success. As a required check, that is indistinguishable from a real pass.

Effect

Every dependency bump reports integration_tests (unmarked): success without running anything. Between 2026-07-30 and 2026-09-16 the suite did not execute on develop at all, yet the branch showed green the whole time:

success   Run technote-space/get-diff-action@v6.1.2
skipped   Run integration tests

(from run 34244306239, develop @ 2253b15)

Because the suite also pulls its fixtures from upstream ethereum/execution-apis, real breakage accumulated invisibly and then failed on the next PR that happened to touch a .go file — #1099, whose diff had nothing to do with it.

Suggested fix

Make "did not run" distinguishable from "passed". Options:

  1. A final aggregation job that the branch protection requires, which asserts each upstream job actually ran rather than treating skipped as satisfied.
  2. Drop the path filter for jobs used as required checks, and control cost with concurrency/caching instead.
  3. Keep the filter but have the job fail when it is required and GIT_DIFF is empty.

Related: #1101 (stale results satisfying the gate). The fixture pin in #1100 removes the upstream moving target but leaves this reporting hole.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions