ci: run only the integration suites a pull request can affect [APPS-37918] by Sarath1018 · Pull Request #775 · UiPath/uipath-typescript

Sarath1018 · 2026-09-24T07:10:33Z

Summary

This adds an integration-scope job to coverage.yml that decides which suites a PR needs from the files it changes (scripts/integration-scope.mjs, ~120 lines, no dependencies). The rule is name-based: suites are the folders under tests/integration/shared/, and a domain's sources live under the same name in src/services/ and src/models/.

Changed path	Runs
`src/services/<name>/`, `src/models/<name>/`, `tests/integration/shared/<name>/**` (suite folder exists)	`tests/integration/shared/<name>` plus the always-on suites
`src/services/<name>/`, `src/models/<name>/` where `src/services/<name>/` exists but no suite folder does (`integration-service`)	nothing — there is no suite for it
one of the always-on suites: `shared/smoke.integration.test.ts`, `shared/http/`, `auth-errors.integration.test.ts`	the always-on suites only
`docs/`, `samples/`, `packages/`, `tests/unit/`, `tests/utils/mocks/**`, Markdown, lint/build config	nothing
anything else — `src/core`, `src/utils`, `src/services/base.ts`, `src/models` folders that are not a service (`common`, `document-understanding`), the test harness, config, workflows, `package.json`	everything

The integration job is skipped only when the resolver explicitly reports nothing affected (run_integration=false); a missing output, a failed diff or an unrecognised path all run the full suite. sonar and summary handle the skipped case; test-and-build passes because a skipped job does not fail the called workflow. A version bump edits package.json, so it runs everything; weekly-coverage.yml never passes scope_to_changes, so it always runs everything.

Deliberately not modelled (kept simple on request): cross-domain dependencies. Two exist today, both into maestro: its case instances wrap the Action Center TaskService (source import), and its process-instance and case-instance suites seed data through Orchestrator processes.start() (test-side). So an action-center-only or orchestrator-only change does not run maestro. processes.start() itself is covered by the orchestrator suite, and the weekly run covers the rest. Adding a mapping later is a one-line table.

Escape hatch: the ci:full-integration label forces a full run. The label name lives only in FULL_RUN_LABEL in the script; the workflow passes the PR's labels through --labels and the script decides. Labels are read from the event that started the run, so push a commit after labelling. Scheduled/manual runs are always full.

Demo PRs

A demo PR into main would always resolve to the full run, because its diff would include this PR's own workflow and script changes. The demos are therefore stacked on a throwaway base, demo/integration-scope-base: this branch plus one line so pr-checks runs for pull requests into any branch. None of them is for merge.

demo(#775) 1/3: change in an existing service runs only its suite #795 change in an existing service (src/services/action-center/tasks.ts) → scope=action-center
demo(#775) 2/3: a new service with its own suite folder runs only that suite #796 new service with its own suite folder (src/services/demo-service/, tests/integration/shared/demo-service/) → scope=demo-service
demo(#775) 3/3: change under src/core runs the full suite #797 change under src/core/ → scope=all
this PR → scope=all (workflow and script changed)

Dry run against recent PRs

#720 (coded-action-app docs/types) → nothing · #709 (sample) → nothing · #745 (maestro tests) → maestro only · #736, #747, #734, #663 (config / core / workflow changes) → everything.

Docs

tests/integration/README.md → "Which suites run on a pull request": the rule table, the label escape hatch, what a new suite folder must be called, and the local dry-run command.

Notes

SonarCloud PR analysis sees integration coverage only from the suites that ran; coverage on the changed files is unaffected, other files may show less. Sonar is not a required check.
Unit tests: tests/unit/scripts/integration-scope.test.ts (behavioural cases, the label path, the suite-folder discovery, and a guard that every suite folder is named after a src/services/ folder — a mis-named suite fails npm run test:unit instead of never running on a PR).

Test plan

npm run test:unit, npm run lint
node scripts/integration-scope.mjs --base origin/main on this branch → full run (workflow + script changed)
coverage / integration-scope on this PR resolves scope=all; the integration legs run the full suite
the demo PRs resolve action-center, demo-service and all

🤖 Generated with Claude Code

claude · 2026-09-24T10:18:23Z

✅ No issues found. Checked for bugs and CLAUDE.md compliance.

PR coverage runs used to execute every integration suite for every change, so a docs- or packages-only PR spent ~17 minutes per leg creating Data Fabric entities and Maestro instances it could not influence, and every platform hiccup in ~3,000 requests failed the PR. A new `scope` job classifies the files changed against the base branch (scripts/integration-scope.mjs). Suites are the folders under tests/integration/shared; a change under src/services/<name>, src/models/<name> or tests/integration/shared/<name> runs that suite, docs, samples, packages and unit tests run nothing, and anything else (core, utils, the harness, config, workflows, a domain with no suite folder) runs everything. The cross-cutting smoke, http and auth-errors suites run whenever any suite runs. The `integration` job receives the resulting vitest path filters and is skipped only when the resolver explicitly says nothing is affected; a missing output still runs. The `ci:full-integration` label forces a full run; weekly-coverage.yml keeps running everything. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

claude · 2026-09-24T10:33:54Z

✅ No issues found. Checked for bugs and CLAUDE.md compliance.

swati354 · 2026-09-29T08:35:59Z

can you create draft throwaway PRs to show how this works and link them in the PR description?
can we name it better? integration-scope or select integration suites.

swati354 · 2026-09-29T08:41:26Z

Maestro's cases wrap the Action Center TaskService, so an action-center-only change runs action-center but not maestro

they also depend on orchestrator processes? (process.start())?

swati354 · 2026-09-29T08:46:48Z

+}
+
+/** One path → { kind: 'ignore' | 'always-on' | 'domain' | 'all' }. */
+export function classify(file, domains) {


ques - a version bump PR would run all tests right?
other than this, the weekly coverage would run all tests every week?

Yes, should we not run for version bump?

IMO we should run

swati354 · 2026-09-29T08:51:34Z

+]);
+
+/** PR label that forces the full run. */
+export const FULL_RUN_LABEL = 'ci:full-integration';


what's this used for? I think the check is hardcoding this string.

Its ununsed, now the workflow passes the PR labels through (--labels "$PR_LABELS", joined from github.event.pull_request.labels.*.name) and the script does the check, so the constant is the single source of truth

swati354 · 2026-09-29T09:01:12Z

+  if (IGNORED_PATTERNS.some(pattern => pattern.test(file))) return { kind: 'ignore' };
+  const domain = file.match(DOMAIN_PATH)?.[1];
+  if (domain && domains.includes(domain)) return { kind: 'domain', domain };
+  if (domain && ALWAYS_ON.includes(`${SHARED}/${domain}`)) return { kind: 'always-on' };


editing a file in http/ runs only the always-on suites, but editing smoke.integration.test.ts or auth-errors.integration.test.ts runs everything.
ideally the behavior should be same for all 3?
They should run all tests but ig the source they cover already triggers "all", so a test file edit alone then would only affect that file?

source under src/core/ → everything, since every service goes through it
a unit test under tests/unit/ → nothing, it cannot affect an integration suite
one of the three always-on integration tests → always-on suites only, the edit can only change that test

swati354 · 2026-09-29T09:04:57Z

 - **The `requirement` argument is `'both'` unless the service rejects one of the credentials.** `'both'` runs the suite once per configured credential — under the PAT *and* under the user token in CI — because the two cover different things: the PAT exercises the external-application OAuth scope model, the user token exercises the general API surface. Use `'user'` only for services that reject PAT and client-credentials tokens outright, and `'pat'` only for something specific to the external-application identity.
 - **Always `throw new Error()` when test preconditions are not met** — whether it's missing config (e.g., no `folderId`) or missing test data (e.g., no running jobs). Never use `console.warn()` + `return` to silently skip — silent skips hide unrunnable tests and make CI green when tests aren't actually exercised.
 - **When a service rejects PAT auth, run it under a user token — do not `describe.skip` it.** `insightsrtm_` endpoints (Agents, Agent Memory, Agent Traces, Governance) and the notification service return 401 for PAT and client-credentials tokens regardless of scopes. Those suites are declared with `describeIntegration(name, 'user', modes, body)`, which states the requirement once and derives the credential, the host and the collection-time skip guard from it, so they run wherever `UIPATH_USER_TOKEN` is configured and are reported as skipped where it isn't. Declaring the guard separately from the requirement lets the two disagree — use the helper. See "Authentication modes" in `tests/integration/README.md`. Do **not** use `describe.skip` for missing test data, missing config, or flakiness — those require a `beforeAll` guard or a `throw`. Equivalently, **NEVER** exclude integration test files via `vitest.integration.config.ts` using env vars or file exclusion patterns — that is functionally equivalent to `describe.skip` across an entire file and has the same problem: tests appear to pass but are never actually exercised. Guard with `beforeAll` + `throw` inside the test file instead.
+- **Scoping a pull-request run to the suites its changed files can affect (the `scope` job, `scripts/integration-scope.mjs`) is not such an exclusion**: no suite is disabled, unrecognised paths run everything, the `ci:full-integration` label forces the full run, and `weekly-coverage.yml` still runs everything. Name a new service's suite folder after its `src/services/` folder.


why is this needed for the agent?

better we update the integration readme with this.

swati354 · 2026-09-29T09:12:19Z

+  const domain = file.match(DOMAIN_PATH)?.[1];
+  if (domain && domains.includes(domain)) return { kind: 'domain', domain };
+  if (domain && ALWAYS_ON.includes(`${SHARED}/${domain}`)) return { kind: 'always-on' };
+  return { kind: 'all', reason: `${file} is outside the per-domain folders` };


could is outside the per-domain folders be misleading for something like src/services/integration-service/connections.ts, as the domain is set but there is no test suite.

true, now If there is no test suite it will simply don't run any

Sarath1018 · 2026-10-01T11:48:13Z

Maestro's cases wrap the Action Center TaskService, so an action-center-only change runs action-center but not maestro

they also depend on orchestrator processes? (process.start())?

We will not run cross dependant test cases, thats the limitation as it is hard to maintain the dependancy graph. We will have a daily run that posts to a slack channel so any cross dependent failures will be caught

- Rename the `scope` job to `integration-scope`, matching the script. - Treat all three always-on entries alike: a change to the smoke test, `shared/http/` or `auth-errors` runs only the always-on suites. Other loose test files still run everything. - Make the full-run reason name the missing suite folder for a domain that has none, instead of calling the path "outside the per-domain folders". - Move the full-run label check into the script (`--labels`); the workflow passes the PR's labels through and `FULL_RUN_LABEL` is the only place the name lives. - Document the scoping in tests/integration/README.md ("Which suites run on a pull request") instead of an agent_docs/rules.md bullet; keep only the suite-folder naming clause there. - Unit tests for the always-on paths, both reason messages, a version-bump change set and the label path. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

claude · 2026-10-01T12:04:57Z

✅ No issues found. Checked for bugs and CLAUDE.md compliance.

The integration-test scoping is documented in tests/integration/README.md only; the suite-folder naming clause goes too. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

claude · 2026-10-01T12:10:50Z

✅ No issues found. Checked for bugs and CLAUDE.md compliance.

A change under src/services/<name> or src/models/<name> where the service folder exists but tests/integration/shared/<name> does not (integration-service today) has no suite to run, so it no longer falls back to the full run. Only real service folders qualify; src/models folders that are not a service (common, document-understanding) are shared code and still run everything. The former safety net for a mis-named suite folder — fall back to the full run — is replaced by a unit test asserting every suite folder is named after a src/services folder. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

claude · 2026-10-01T12:47:23Z

+    expect(scope).toMatchObject({ run: true, all: true, paths: [] });
+    expect(scope.reasons[0]).toContain(FULL_RUN_LABEL);
+  });
+});


The resolveArgs describe block has no test for the git diff failure path — lines 107–110 in the script, whose comment explicitly says "A diff problem must never skip the run." That's the most safety-critical branch in the whole CLI, yet a regression there (e.g., accidentally re-throwing or calling process.exit) would be invisible.

The branch is testable by mocking node:child_process with vi.mock/vi.hoisted (ESM pattern the project already uses elsewhere):

import { vi, describe, it, expect, beforeEach } from 'vitest'; const { mockExecFileSync } = vi.hoisted(() => ({ mockExecFileSync: vi.fn() })); vi.mock('node:child_process', () => ({ execFileSync: mockExecFileSync })); describe('integration-scope resolveArgs — git diff failure', () => { beforeEach(() => mockExecFileSync.mockReset()); it('falls back to the full run when git diff throws', () => { mockExecFileSync.mockImplementation(() => { throw new Error('fatal: bad object origin/main'); }); const { scope } = resolveArgs(['--base', 'origin/main'])!; expect(scope).toMatchObject({ run: true, all: true, paths: [] }); expect(scope!.reasons[0]).toContain('could not diff against origin/main'); }); });

claude · 2026-10-01T12:47:34Z

Review findings

New inline comment at tests/unit/scripts/integration-scope.test.ts line 136: the resolveArgs git diff failure catch block (script lines 107-110, comment: 'A diff problem must never skip the run') has no test. Suggested a vi.hoisted + vi.mock pattern to cover it.

A base ref git does not know must fall back to the full run, never throw or exit. Also covers the --base success path and --files parsing. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Sarath1018 · 2026-10-01T13:02:19Z

Created 3 throwaway PRs covering 3 scenarios

change in a specific service demo(#775) 1/3: change in an existing service runs only its suite #795
New service demo(#775) 2/3: a new service with its own suite folder runs only that suite #796
changes in common files that requires to run all tests demo(#775) 3/3: change under src/core runs the full suite #797

claude · 2026-10-01T13:08:22Z

✅ No issues found. Checked for bugs and CLAUDE.md compliance.

Sarath1018 requested a review from a team September 24, 2026 07:10

claude Bot reviewed Sep 24, 2026

View reviewed changes

Comment thread scripts/integration-scope.mjs Outdated

Sarath1018 force-pushed the ci/scope-integration-tests-to-changes branch from 1df05a8 to 2630b5f Compare September 24, 2026 10:13

Sarath1018 force-pushed the ci/scope-integration-tests-to-changes branch from 2630b5f to 4189c2b Compare September 24, 2026 10:29

Sarath1018 changed the title ~~ci: run only the integration suites a pull request can affect~~ ci: run only the integration suites a pull request can affect [APPS-37918] Sep 25, 2026

swati354 reviewed Sep 29, 2026

View reviewed changes

amrit-agarwal-1 mentioned this pull request Oct 1, 2026

ci: one PR Checks run per pull request, budgets for the insightsrtm_ suites #791

Open

3 tasks done

docs: leave agent_docs/rules.md untouched

49cf7a8

The integration-test scoping is documented in tests/integration/README.md only; the suite-folder naming clause goes too. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Sarath1018 force-pushed the ci/scope-integration-tests-to-changes branch from d0ff65f to 49cf7a8 Compare October 1, 2026 12:05

claude Bot reviewed Oct 1, 2026

View reviewed changes

This was referenced Oct 1, 2026

demo(#775) 1/3: change in an existing service runs only its suite #795

Draft

demo(#775) 2/3: a new service with its own suite folder runs only that suite #796

Draft

demo(#775) 3/3: change under src/core runs the full suite #797

Draft

test: cover the git diff failure path of the scope resolver

a65bd50

A base ref git does not know must fall back to the full run, never throw or exit. Also covers the --base success path and --files parsing. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Sarath1018 requested a review from swati354 October 1, 2026 13:54

Conversation

Sarath1018 commented Sep 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Demo PRs

Dry run against recent PRs

Docs

Notes

Test plan

Uh oh!

Uh oh!

claude Bot commented Sep 24, 2026

Uh oh!

claude Bot commented Sep 24, 2026

Uh oh!

swati354 commented Sep 29, 2026

Uh oh!

swati354 commented Sep 29, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Sarath1018 commented Oct 1, 2026

Uh oh!

claude Bot commented Oct 1, 2026

Uh oh!

claude Bot commented Oct 1, 2026

Uh oh!

claude Bot Oct 1, 2026

Choose a reason for hiding this comment

Uh oh!

claude Bot commented Oct 1, 2026

Review findings

Uh oh!

Sarath1018 commented Oct 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

claude Bot commented Oct 1, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Sarath1018 commented Sep 24, 2026 •

edited

Loading

Sarath1018 commented Oct 1, 2026 •

edited

Loading