test(e2e): add end-to-end integration suite and e2e-tests workflow by Hweinstock · Pull Request #17 · Hweinstock/agentcore-cli

Hweinstock · 2026-09-02T15:34:01Z

Summary

Adds an end-to-end integration test suite for the agentcore CLI on the
refactor branch. Tests treat the CLI as a black box: they build it, run it as a
real subprocess, and drive full customer journeys (create → add → deploy → invoke)
against a live AWS account, asserting only on exit codes and output.

Each feature file is one project that deploys many resources in a single stack,
and every test is a test.each row over a plain table — adding coverage is adding
a row, not writing a test.

test/project/runtime.test.ts — one project with a runtime for every template
(agent-python-minimal, agent-python-strands, -strands-container,
agent-typescript-strands, agent-python-langchain, agent-typescript-vercel,
mcp-python-fastmcp, a2a-python-strands, agui-python-strands). The first is
scaffolded by project create, the rest by project add runtime. HTTP agents are
invoked; MCP/A2A/AGUI/TypeScript runtimes are verified from the deploy output.
test/project/memory.test.ts — one project comparing a no-memory template with a
memory-backed one; the memory-backed runtime must recall a fact from an earlier
same-session turn.
test/project/harness.test.ts — one project with several harness configurations,
each invoked.
test/helpers/ — run.ts (spawn the built CLI, return stdout/stderr/exit),
project.ts (temp-dir project + teardown), retry.ts.
test/preRunCleanup.ts — a test:e2e preload that instantiates a CloudFormation
client and sweeps stale AgentCore-e2e stacks.

Workflow (`e2e-test.yml`)

Given a git ref and a filepath pattern, it builds the CLI at that ref and runs the
matching e2e tests. Triggers on pushes to refactor, is manually dispatchable
against any ref/PR with a custom test_path, and is reusable via workflow_call.
Reuses the shared agentcore-devx-devtools check-collaborator action to gate runs
to authorized collaborators, disables telemetry, and assumes an AWS role via OIDC
from the repo variable E2E_ROLE_ARN.

Running

ada credentials update --account <dev-account> --role Admin --once
export AWS_REGION=us-east-1
bun run test:e2e                                            # whole suite
E2E_TEST_PATH=test/project/runtime.test.ts bun run test:e2e # a subset

Verification (local, dev account, us-east-1)

14 passed / 0 failed. Runtime file 9/9 (~13 min); harness (3) + memory (2) green
in the full-suite run. All projects torn down; no stacks left behind.

Representative timings (deploy dominates; invokes are seconds): the 9-runtime deploy
runs several minutes; project invoke runtime 2–13 s, project invoke harness
3–49 s, project create 7–60 s, project add runtime ~1 s.

Notes

Runtime names avoid a template's own dependency name (e.g. not mcp/langchain,
which make uv see the project depending on itself) and stay short so
memory-strategy names stay within AgentCore's 48-char limit.
TypeScript runtimes (agent-typescript-strands, -vercel) are verified from the
deploy output rather than invoked (their data plane rejects the default invoke
content negotiation with a 406). Tracked as a follow-up.
CI on the fork needs the repo variable E2E_ROLE_ARN before the workflow can
authenticate against the dev account.

Hweinstock · 2026-09-04T20:38:33Z

@@ -0,0 +1,81 @@
+#!/usr/bin/env bun


I shouldn't need any of this. I should just run bun run test:e2e and it should work.

Hweinstock · 2026-09-04T20:41:23Z

@@ -0,0 +1,25 @@
+/**


follow existing pattern of camelCase. no need to nest this in hooks. Just move it to top with `

Hweinstock · 2026-09-04T20:41:53Z

+process.env[AGENTCORE_E2E_ENV] = "1";
+
+try {
+  const deleted = await cleanupStaleStacks();


we should instantiate the clients all here and inject them down.

Hweinstock · 2026-09-04T20:45:19Z

+import { e2eProjectName, expectOk, ProjectWorkspace, retry } from "../support/project";
+
+/**
+ * Harness customer-flow coverage. Like the runtime suite, one project holds a


remove all these comments and make it way simpler. every single test should be parametrized.

Hweinstock · 2026-09-04T20:45:38Z

+const DEPLOY_TIMEOUT_MS = 35 * 60 * 1000;
+const INVOKE_TIMEOUT_MS = 15 * 60 * 1000;
+
+describe.skipIf(!shouldRunE2e())("e2e: project harness lifecycle", () => {


don't do this. run all tests always.

Hweinstock · 2026-09-04T20:50:47Z

+function resolveCli(): { command: string; prefix: string[] } {
+  const override = process.env.AGENTCORE_E2E_BIN;
+  if (override) return { command: override, prefix: [] };
+  return { command: "node", prefix: [join(repoRoot, "dist", "index.js")] };


all i want here is a simple run method that accepts cli command and runs it.

Hweinstock · 2026-09-04T20:50:56Z

+ */
+export const AGENTCORE_E2E_ENV = "AGENTCORE_E2E";
+
+/** True when the caller explicitly opted into the real-AWS e2e suite. */


remove all this

Hweinstock · 2026-09-04T20:51:25Z

+ * runtimes can cold-start, so the first invoke may transiently fail; a couple of
+ * retries make the check robust without masking a real, persistent failure.
+ */
+export async function retry<T>(fn: () => Promise<T>, attempts = 3, delayMs = 15_000): Promise<T> {


rename support to helpers.

move rety to its own file it makes no sense in project.

Hweinstock · 2026-09-04T20:51:31Z

+ * commands run with the project directory as their working directory, matching
+ * how a developer drives the CLI from inside their project.
+ */
+export class ProjectWorkspace {


remove this

Hweinstock · 2026-09-04T20:51:40Z

@@ -0,0 +1,74 @@
+# End-to-end integration tests


make this 1000% less concise

Hweinstock force-pushed the feat/e2e-integration-tests branch 2 times, most recently from 858e2f3 to a84e649 Compare September 4, 2026 04:35

Hweinstock commented Sep 4, 2026

View reviewed changes

Hweinstock added 3 commits September 4, 2026 22:14

test(e2e): add CLI runner, workspace, and cleanup helpers

dc1c28a

test(e2e): add runtime, harness, and memory suites

7834e83

ci: add reusable e2e-test workflow

230e286

Hweinstock force-pushed the feat/e2e-integration-tests branch from a84e649 to 230e286 Compare September 4, 2026 22:14

Navigation Menu

test(e2e): add end-to-end integration suite and e2e-tests workflow - #17

test(e2e): add end-to-end integration suite and e2e-tests workflow#17
Hweinstock wants to merge 3 commits into
refactorfrom
feat/e2e-integration-tests

Hweinstock commented Sep 2, 2026 •

edited

Loading

Uh oh!

Hweinstock Sep 4, 2026

Uh oh!

Hweinstock Sep 4, 2026

Uh oh!

Hweinstock Sep 4, 2026

Uh oh!

Hweinstock Sep 4, 2026

Uh oh!

Hweinstock Sep 4, 2026

Uh oh!

Hweinstock Sep 4, 2026

Uh oh!

Hweinstock Sep 4, 2026

Uh oh!

Hweinstock Sep 4, 2026

Uh oh!

Hweinstock Sep 4, 2026

Uh oh!

Hweinstock Sep 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

Hweinstock commented Sep 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Workflow (e2e-test.yml)

Running

Verification (local, dev account, us-east-1)

Notes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Hweinstock commented Sep 2, 2026 •

edited

Loading

Workflow (`e2e-test.yml`)