Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .agents/skills/headless-computer-use/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,8 @@ Chromium capabilities; macOS WebKit reports them as unsupported.
4. Start with `inspect --context summary --task "..."`, then use an outline,
scoped text, or scoped actions only when the task needs them.
5. Prefer role/name targeting; otherwise use a ref from the latest inspection.
Use `hover` when the task requires a tooltip or other hover-only state; it
does not click or focus the target. Inspect again after the state changes.
For native dropdowns, use `select` with exactly one exact `--label` or
`--value`; do not treat custom ARIA widgets as native selects.
6. After navigation or a substantial rerender, wait for the expected URL, text,
Expand All @@ -49,6 +51,7 @@ headless session create agent-qa
headless --session agent-qa visit http://localhost:3000
headless --session agent-qa inspect --context summary --task "click Continue"
headless --session agent-qa inspect --context actions --task "click Continue" --limit 10
headless --session agent-qa hover --role button --name Continue
headless --session agent-qa click --role button --name Continue
headless --session agent-qa wait --url /next --settled --timeout 10000
headless --session agent-qa inspect --context actions --task "verify next page"
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,8 @@ headless --session NAME inspect --context actions --task "TASK"
headless --session NAME inspect --context full --text
headless --session NAME click REF
headless --session NAME click --role ROLE --name NAME
headless --session NAME hover REF
headless --session NAME hover --role ROLE --name NAME
headless --session NAME fill REF "TEXT"
headless --session NAME fill REF -- "--json stays literal"
headless --session NAME select REF --label LABEL
Expand All @@ -48,7 +50,9 @@ large pages, request `outline`, select a returned `@rN` region, then use
bound the result; check `omitted` before assuming it describes the whole page.
Use `click --role ... --name ...` for unique accessible controls. Use an `@eN`
ref from the latest inspection when role/name is ambiguous. Inspect again after
navigation or a large rerender. Native single-select controls advertise
navigation or a large rerender. Use `hover` only for hover-dependent state such
as tooltips; it does not focus or click, and its output never includes internal
coordinates. Native single-select controls advertise
`select`; use exactly one exact `--label` or `--value`. File inputs advertise
`upload` for an existing private artifact-store basename. Upload never accepts
or imports a filesystem path. Ask before uploading, as in [safety.md](safety.md).
Expand Down
7 changes: 5 additions & 2 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,8 @@ Two versions travel independently, on purpose:
`HEADLESS_VERSION`, reported by the CLI, host, and MCP adapter, and used for
release assets.
- **Protocol version**: `headlessProtocolVersion` in `Protocol.swift`,
currently `0.5`. It changes only when the wire contract changes, and always
with an entry in
currently `0.5`. It changes only for incompatible wire-contract changes,
always with an entry in
[`docs/roadmap/architecture-decisions.md`](docs/roadmap/architecture-decisions.md).

## [Unreleased]
Expand All @@ -24,6 +24,9 @@ Two versions travel independently, on purpose:

### Added

- Semantic `hover` targets a fresh element ref or an exact role/name without
exposing coordinates. Chromium dispatches a trusted CDP mouse move, while
WebKit reports its fixed synthetic hover path in capabilities.
- macOS Command-, opens one Settings window built from the typed registry and
writing through the same backend as `headless config`. Linux has no Settings
GUI.
Expand Down
6 changes: 5 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,7 @@ headless --session qa inspect --context outline --limit 20
headless --session qa inspect --context actions --task "click Continue"
headless --session qa record start --fps 10
headless --session qa tour --full-page
headless --session qa hover --role button --name Continue
headless --session qa click --role button --name Continue
headless --session qa wait --url /next --settled
headless --session qa record stop --output dashboard-flow.mp4
Expand Down Expand Up @@ -128,7 +129,10 @@ structural region references such as `@r4`, then inspect only that region with
explicit broad-page escape hatch.

Each control's `actions` list contains only protocol verbs that can run (`click`,
`fill`, or `upload`); unsupported controls never advertise nonexistent commands. Full
`fill`, `select`, or `upload`); unsupported controls never advertise nonexistent
commands. Hover is available for any visible semantic target but is not
advertised because a page does not expose whether hovering has meaningful
behavior. Full
inspection retains roles, names, rendered media metadata, safety markers,
bounds, and element references such as `@e1`.
`capture-info` returns the browser surface, page state, action trace, and
Expand Down
4 changes: 3 additions & 1 deletion apps/headless-rs/src/protocol.rs
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ pub enum CommandName {
Visit,
Inspect,
Click,
Hover,
Fill,
Press,
Scroll,
Expand Down Expand Up @@ -93,6 +94,7 @@ impl CommandName {
CommandName::Visit => "visit",
CommandName::Inspect => "inspect",
CommandName::Click => "click",
CommandName::Hover => "hover",
CommandName::Fill => "fill",
CommandName::Press => "press",
CommandName::Scroll => "scroll",
Expand Down Expand Up @@ -350,7 +352,7 @@ impl CommandRequest {
}
Ok(())
}
Click => self.target(false, true, true),
Click | Hover => self.target(false, true, true),
Fill => self.target(true, true, true),
Press => {
self.allow(&["key"])?;
Expand Down
16 changes: 16 additions & 0 deletions apps/headless-rs/tests/protocol_tests.rs
Original file line number Diff line number Diff line change
Expand Up @@ -165,6 +165,22 @@ fn command_parameter_validation() {
let request = valid_request(CommandName::Click, params(&[("target", json!("@food"))]));
assert!(request.validate().is_err());

// hover follows the same exclusive target contract as click
let request = valid_request(CommandName::Hover, params(&[("target", json!("@e8"))]));
request.validate().unwrap();
let request = valid_request(
CommandName::Hover,
params(&[("role", json!("button")), ("name", json!("Account"))]),
);
request.validate().unwrap();
let request = valid_request(CommandName::Hover, params(&[]));
assert!(request.validate().is_err());
let request = valid_request(
CommandName::Hover,
params(&[("target", json!("@e8")), ("name", json!("Account"))]),
);
assert!(request.validate().is_err());

// fill requires a value
let request = valid_request(CommandName::Fill, params(&[("target", json!("@e12"))]));
assert!(request.validate().is_err());
Expand Down
8 changes: 8 additions & 0 deletions apps/headless/Host/AgentBridge.swift
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,11 @@ extension BrowserWindowController {
return try callAgent("return globalThis.__headlessAgent.click(args);", arguments: ["args": args])
}

func agentHover(parameters: [String: JSONValue]) throws -> JSONValue {
let args = try browserTargetArguments(parameters)
return try callAgent("return globalThis.__headlessAgent.hover(args);", arguments: ["args": args])
}

func agentFill(parameters: [String: JSONValue]) throws -> JSONValue {
var args = try browserTargetArguments(parameters)
guard let value = parameters["value"]?.stringValue else {
Expand Down Expand Up @@ -644,6 +649,9 @@ extension BrowserWindowController: BrowserEngineSession {
func hostClick(parameters: [String: JSONValue]) throws -> JSONValue {
try agentClick(parameters: parameters)
}
func hostHover(parameters: [String: JSONValue]) throws -> JSONValue {
try agentHover(parameters: parameters)
}
func hostFill(parameters: [String: JSONValue]) throws -> JSONValue {
try agentFill(parameters: parameters)
}
Expand Down
15 changes: 15 additions & 0 deletions apps/headless/LinuxHost/BrowserProcess.swift
Original file line number Diff line number Diff line change
Expand Up @@ -773,6 +773,21 @@ final class LinuxBrowserSession: @unchecked Sendable {
])
}

func hover(parameters: [String: JSONValue]) throws -> JSONValue {
let target = try trustedInputTarget(parameters: parameters, action: "hover")
_ = try command("Input.dispatchMouseEvent", parameters: [
"type": "mouseMoved", "x": target.x, "y": target.y,
])
// A hover handler can start a cross-document navigation before the next
// command. Pause capture so a recording does not composite that frame.
pauseRecordingCapture()
return .object([
"hovered": .string(target.reference),
"role": .string(target.role),
"name": .string(target.name),
])
}

func upload(parameters: [String: JSONValue], artifactURL: URL) throws -> JSONValue {
let args = try browserTargetArguments(parameters)
let objectId = try evaluateNode(
Expand Down
3 changes: 3 additions & 0 deletions apps/headless/LinuxHost/main.swift
Original file line number Diff line number Diff line change
Expand Up @@ -88,6 +88,9 @@ final class ChromiumBrowserEngineSession: BrowserEngineSession {
func hostClick(parameters: [String: JSONValue]) throws -> JSONValue {
try browserSession.click(parameters: parameters)
}
func hostHover(parameters: [String: JSONValue]) throws -> JSONValue {
try browserSession.hover(parameters: parameters)
}
func hostFill(parameters: [String: JSONValue]) throws -> JSONValue {
try browserSession.fill(parameters: parameters)
}
Expand Down
3 changes: 3 additions & 0 deletions apps/headless/Sources/HeadlessProtocol/CLI.swift
Original file line number Diff line number Diff line change
Expand Up @@ -151,6 +151,8 @@ public struct CLIParser {
return try parseInspect(arguments, session: session, jsonOutput: jsonOutput)
case "click":
return try parseTargeted(.click, arguments: arguments, session: session, jsonOutput: jsonOutput)
case "hover":
return try parseTargeted(.hover, arguments: arguments, session: session, jsonOutput: jsonOutput)
case "upload":
return try parseUpload(arguments, session: session, jsonOutput: jsonOutput)
case "fill":
Expand Down Expand Up @@ -884,6 +886,7 @@ Commands:
inspect [--context summary|outline|text|actions|full] [--task TEXT]
[--within @rN] [--limit N] [--budget TOKENS] [--depth N] [--text]
click REF | click --role ROLE [--name NAME]
hover REF | hover --role ROLE [--name NAME]
fill REF TEXT | fill REF -- TEXT_WITH_LITERAL_FLAGS | press KEY
select REF --label LABEL | select REF --value VALUE
select --role ROLE [--name NAME] (--label LABEL | --value VALUE)
Expand Down
4 changes: 4 additions & 0 deletions apps/headless/Sources/HeadlessProtocol/Capabilities.swift
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ public struct BrowserEngineCapabilities: Sendable {
public let qaDiagnosticSynchronization: String
public let screenshotClipboard: Bool
public let inputDispatch: String
public let hoverDispatch: String
public let selectDispatch: String
public let networkIdleWait: Bool
public let normalProfileStorage: String
Expand Down Expand Up @@ -70,6 +71,7 @@ public struct BrowserEngineCapabilities: Sendable {
"screenshotClipboard": .bool(screenshotClipboard),
"tourTimeoutMs": .number(65_000),
"inputDispatch": .string(inputDispatch),
"hoverDispatch": .string(hoverDispatch),
"selectDispatch": .string(selectDispatch),
"networkIdleWait": .bool(networkIdleWait),
"fileUpload": .bool(fileUpload),
Expand Down Expand Up @@ -117,6 +119,7 @@ public struct BrowserEngineCapabilities: Sendable {
qaDiagnosticSynchronization: "best-effort-page-world-observer",
screenshotClipboard: true,
inputDispatch: "synthetic-dom",
hoverDispatch: "synthetic-dom",
selectDispatch: "synthetic-dom",
networkIdleWait: false,
normalProfileStorage: "persistent-wkwebsite-data-store",
Expand All @@ -142,6 +145,7 @@ public struct BrowserEngineCapabilities: Sendable {
qaDiagnosticSynchronization: "runtime-round-trip-flush",
screenshotClipboard: false,
inputDispatch: "trusted-cdp",
hoverDispatch: "trusted-cdp",
selectDispatch: "synthetic-dom",
networkIdleWait: true,
normalProfileStorage: "private-xdg-data-directory",
Expand Down
6 changes: 5 additions & 1 deletion apps/headless/Sources/HeadlessProtocol/HostCore.swift
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@ public protocol BrowserEngineSession: AnyObject {
func hostVisit(_ url: URL) throws -> JSONValue
func hostInspect(parameters: [String: JSONValue]) throws -> JSONValue
func hostClick(parameters: [String: JSONValue]) throws -> JSONValue
func hostHover(parameters: [String: JSONValue]) throws -> JSONValue
func hostFill(parameters: [String: JSONValue]) throws -> JSONValue
func hostSelect(parameters: [String: JSONValue]) throws -> JSONValue
func hostPress(parameters: [String: JSONValue]) throws -> JSONValue
Expand Down Expand Up @@ -558,6 +559,7 @@ public final class HostCore<Engine: BrowserEngine>: @unchecked Sendable {
return try session.hostVisit(url)
case .inspect: return try session.hostInspect(parameters: request.parameters)
case .click: return try session.hostClick(parameters: request.parameters)
case .hover: return try session.hostHover(parameters: request.parameters)
case .fill: return try session.hostFill(parameters: request.parameters)
case .select: return try session.hostSelect(parameters: request.parameters)
case .upload:
Expand Down Expand Up @@ -675,7 +677,9 @@ public final class HostCore<Engine: BrowserEngine>: @unchecked Sendable {
after command: CommandName, request: CommandRequest, sessionName: String,
session: Engine.Session, result: JSONValue
) throws -> CommandResponse? {
guard [.visit, .inspect, .click, .wait, .back, .reload].contains(command) else { return nil }
guard [.visit, .inspect, .click, .hover, .wait, .back, .reload].contains(command) else {
return nil
}
let form = try AuthenticationForm(session.hostAuthenticationState())
guard form.detection != .none else {
authenticationChallenges.invalidate(session: sessionName)
Expand Down
3 changes: 2 additions & 1 deletion apps/headless/Sources/HeadlessProtocol/Protocol.swift
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,7 @@ public enum CommandName: String, Codable, CaseIterable, Sendable {
case visit
case inspect
case click
case hover
case fill
case select
case upload
Expand Down Expand Up @@ -262,7 +263,7 @@ public struct CommandRequest: Codable, Equatable, Sendable {
throw ProtocolValidationError.invalidParameter("Invalid integer parameter: \(key)")
}
}
case .click:
case .click, .hover:
try target(allowValue: false)
case .fill:
try target(allowValue: true)
Expand Down
9 changes: 9 additions & 0 deletions apps/headless/Sources/HeadlessProtocol/ProtocolSchema.swift
Original file line number Diff line number Diff line change
Expand Up @@ -318,6 +318,11 @@ public func protocolResultDefinition(for command: CommandName) -> ProtocolResult
resultField("clicked", .string), resultField("role", .string),
resultField("name", .string),
])
case .hover:
return result("Hover", [
resultField("hovered", .string), resultField("role", .string),
resultField("name", .string),
])
case .fill:
return result("Fill", [resultField("filled", .string), resultField("valueLength", .number)])
case .select:
Expand Down Expand Up @@ -705,6 +710,10 @@ public let protocolCommandDefinitions: [CommandName: ProtocolCommandDefinition]
.click, targetParameters, untrusted: true,
constraints: ["exactly one target reference or semantic role/name target"]
),
command(
.hover, targetParameters, untrusted: true,
constraints: ["exactly one target reference or semantic role/name target"]
),
command(
.fill,
targetParameters + [
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ if (!globalThis.__headlessAgent) {
const regionRefs = new WeakMap();
let current = new Map();
let currentRegions = new Map();
let hoveredElement = null;
// Every reference ever handed out, so a failed lookup can say whether the
// reference expired or was never issued at all.
const issuedRefs = new Set();
Expand Down Expand Up @@ -706,6 +707,58 @@ if (!globalThis.__headlessAgent) {
element.click();
return {clicked: refFor(element), role: role(element), name: name(element)};
};
// Hit-test the layout after scroll, before any pointer event.
const pointerHit = element => {
if (!visible(element)) fail('ELEMENT_NOT_FOUND', 'ELEMENT_NOT_VISIBLE');
const rect = element.getBoundingClientRect();
const left = Math.max(0, rect.left);
const right = Math.min(innerWidth, rect.right);
const top = Math.max(0, rect.top);
const bottom = Math.min(innerHeight, rect.bottom);
if (right <= left || bottom <= top) fail('ELEMENT_NOT_FOUND', 'ELEMENT_NOT_VISIBLE');
const x = left + (right - left) / 2;
const y = top + (bottom - top) / 2;
const hit = document.elementFromPoint(x, y);
if (!hit || (hit !== element && !element.contains(hit))) {
fail('ELEMENT_NOT_FOUND', 'ELEMENT_OBSCURED');
}
return {x, y};
};
const hover = args => {
const element = target(args);
element.scrollIntoView({block: 'center', inline: 'center', behavior: 'instant'});
const point = pointerHit(element);
const Pointer = typeof PointerEvent === 'function' ? PointerEvent : MouseEvent;
const pointer = {bubbles: true, clientX: point.x, clientY: point.y, pointerId: 1,
pointerType: 'mouse', isPrimary: true};
const mouse = {bubbles: true, clientX: point.x, clientY: point.y};
const previous = hoveredElement instanceof Element && hoveredElement.isConnected
? hoveredElement : null;
hoveredElement = element;
if (previous !== element) {
if (previous) {
previous.dispatchEvent(new Pointer('pointerout', {...pointer, relatedTarget: element}));
previous.dispatchEvent(new Pointer('pointerleave', {
...pointer, bubbles: false, relatedTarget: element
}));
previous.dispatchEvent(new MouseEvent('mouseout', {...mouse, relatedTarget: element}));
previous.dispatchEvent(new MouseEvent('mouseleave', {
...mouse, bubbles: false, relatedTarget: element
}));
}
element.dispatchEvent(new Pointer('pointerover', {...pointer, relatedTarget: previous}));
element.dispatchEvent(new Pointer('pointerenter', {
...pointer, bubbles: false, relatedTarget: previous
}));
element.dispatchEvent(new MouseEvent('mouseover', {...mouse, relatedTarget: previous}));
element.dispatchEvent(new MouseEvent('mouseenter', {
...mouse, bubbles: false, relatedTarget: previous
}));
}
element.dispatchEvent(new Pointer('pointermove', pointer));
element.dispatchEvent(new MouseEvent('mousemove', mouse));
return {hovered: refFor(element), role: role(element), name: name(element)};
};
// Chromium uses this isolated-world resolver only to select and validate a
// target. The host performs the action through CDP's trusted input domain.
// Page-derived coordinates remain bounded to the visible viewport.
Expand All @@ -717,6 +770,13 @@ if (!globalThis.__headlessAgent) {
if (!editable || element.disabled || element.readOnly) throw new Error('NOT_EDITABLE');
}
element.scrollIntoView({block: 'center', inline: 'center', behavior: 'instant'});
if (action === 'hover') {
const point = pointerHit(element);
return {
ref: refFor(element), role: role(element), name: name(element),
x: point.x, y: point.y,
};
}
element.focus({preventScroll: true});
const rect = element.getBoundingClientRect();
const left = Math.max(0, rect.left);
Expand Down Expand Up @@ -1199,7 +1259,7 @@ if (!globalThis.__headlessAgent) {
return {count: document.getAnimations().length, animations: all, truncated: document.getAnimations().length > all.length};
};
return {
snapshot, click, fill, select, credentialFill, finishCredentialFill, press, inputTarget, fileInput, fileInputMetadata,
snapshot, click, hover, fill, select, credentialFill, finishCredentialFill, press, inputTarget, fileInput, fileInputMetadata,
authentication, scroll, state, tour, screenshotPlan, regionSlice, scrollToCapturePoint, rectangle, styles, storage,
performance: performanceSummary, animations
};
Expand Down
Loading
Loading