Ai scriptless skills

config/runtime.py

@@ -98,6 +98,10 @@ def _hydrate_ctx_user_config(ctx: Any, user_config: dict[str, Any]) -> None:
         return
 
     config_copy = dict(user_config)
+    try:
+        setattr(ctx, "user_config", dict(config_copy))
+    except Exception:
+        pass
     request_context = getattr(ctx, "request_context", None)
     request = getattr(request_context, "request", None)
     request_state = getattr(request, "state", None)

main.py

@@ -3,6 +3,7 @@
 import logging
 import os
 import sys
+from contextlib import contextmanager
 from typing import Literal, cast
 
 # Patch MCP ArgModelBase so tools with an "arguments" param receive the full payload
@@ -22,7 +23,14 @@ def _wrap_root_as_arguments(cls, data: object) -> object:
         return data
 
 
-func_metadata.ArgModelBase = _PatchedArgModelBase
+@contextmanager
+def _patched_arg_model_base():
+    original = func_metadata.ArgModelBase
+    func_metadata.ArgModelBase = _PatchedArgModelBase
+    try:
+        yield
+    finally:
+        func_metadata.ArgModelBase = original
 
 from mcp.server.fastmcp import FastMCP, Icon
 
@@ -140,6 +148,17 @@ def build_mcp_server(
     instructions = """
 # Perfecto MCP Server
 
+## Proactive Knowledge Consultation
+
+- **ALWAYS consult Perfecto Skills and Help tools first** before answering questions about Perfecto features, authoring AI Scriptless tests, interpreting results, troubleshooting, or providing recommendations.
+- **Use `perfecto_skills`**: Access specialized knowledge about Perfecto capabilities (starting with AI Scriptless), best practices, and official getting-started guidance.
+- **Use `perfecto_help`**: Consult live Perfecto documentation (category_id='perfecto', subcategory_id_list=['ide'] for AI Scriptless).
+- **Golden rule**: If you're not 100% certain about something related to Perfecto, consult Skills or Help first, and if you need to search online, prioritize help.perfecto.io.
+
+## Important Guidelines
+- **Batch Operations**: When making multiple calls to the same tool, check if that tool supports a `batch` action and use it instead of separate calls.
+- **Don't assume / Don't invent**: If something is unclear, consult Skills/Help before responding.
+- **Provide resources**: Include markdown-formatted links to Perfecto help documentation when relevant.
 """
     mcp_kwargs: dict = {
         "instructions": instructions,
@@ -154,7 +173,8 @@ def build_mcp_server(
             stateless_http=False,
         )
     mcp = FastMCP("perfecto-mcp", **mcp_kwargs)
-    register_tools(mcp, app_runtime)
+    with _patched_arg_model_base():
+        register_tools(mcp, app_runtime)
     return mcp, wire_transport
 
 

pyproject.toml

@@ -34,7 +34,7 @@ where = ["."]
 include = ["tools*", "config", "models", "formatters", "resources"]
 
 [tool.setuptools.package-data]
-"resources" = ["*.png"]
+"resources" = ["**/*"]
 
 [tool.pytest.ini_options]
 pythonpath = ["."]

resources/skills/perfecto-ai-scriptless/SKILL.md

@@ -0,0 +1,94 @@
+---
+name: perfecto-ai-scriptless
+description: Author, run, and troubleshoot Perfecto AI Scriptless (also called Scriptless Mobile) tests — workspace UI, DUT execution, loops/conditions, AI user actions, AI validations, visual comparisons, and Perfecto MCP perfecto_ai_scriptless workflows. Use this skill whenever the user mentions AI Scriptless, Scriptless Mobile, Perfecto scriptless, natural-language test steps, AI commands, AI validation, AI user action, visual comparison, DUT, scriptless lab, or wants to create/edit/run Perfecto scriptless tests via UI or MCP — even if they do not say the exact product name.
+---
+
+# Perfecto AI Scriptless
+
+Teach agents how to help users with Perfecto AI Scriptless: product behavior, licenses,
+lab UI, and MCP authoring/execution. Prefer this skill over guessing.
+
+Official docs hub: [AI Scriptless interface](https://help.perfecto.io/perfecto-help/content/perfecto/ide/scriptless-mobile-interface.htm)
+
+## When this skill loads
+
+1. Read this `SKILL.md` first.
+2. Load only the reference files needed for the user's question (see [Reference files](#reference-files)).
+3. For live Perfecto Help pages not covered here, use `perfecto_help` with `category_id='perfecto'` and `subcategory_id='ide'`.
+4. Explain **why** a step matters when the rule prevents broken tests (licenses, `step_path`, command policy).
+
+## Licenses (check early)
+
+AI Scriptless behavior depends on cloud licenses:
+
+| Licenses present | UI label | Scope |
+| --- | --- | --- |
+| Perfecto AI + Desktop Web | **AI Scriptless** | Mobile + desktop web, natural-language commands |
+| Missing one or both | **Scriptless Mobile** | Mobile only |
+
+- AI commands need Perfecto AI **and** admin feature-toggle opt-in. Without them, AI steps stay inactive and dependent tests fail.
+- Desktop web as DUT needs Perfecto AI **and** Desktop Web.
+- If MCP AI Scriptless calls fail with license/feature errors, tell the user to contact their Perfecto admin — do not invent workarounds.
+
+## Quick start (UI)
+
+1. Open Perfecto → **Scriptless Automation** → **Build ai scriptless test** (optional: pick a real device).
+2. Lab URL: `https://{cloud}.app.perfectomobile.com/lab/scriptless-mobile/` (also from `perfecto_user` → `read_user`).
+3. Orient on the workspace — read [interface.md](references/interface.md) if the user asks about toolbar, sidebar, editor, devices, or widgets.
+4. Create/open/save tests, add AI or classic commands, then run with a DUT and open the Single Test Report.
+
+## Agent rules for MCP authoring
+
+Follow these when using `perfecto_ai_scriptless` (why in parentheses):
+
+1. **Consult skills/help before inventing product behavior** — Perfecto Scriptless rules are license- and UI-specific.
+2. **Call `list_commands` before `add_command`** and follow the selection policy in the tool `info` field. Prefer primary AI commands: `ai_user-action`, `ai_validation`, `ai_visual-comparison` for natural-language steps.
+3. **Call `get_command_definitions` before filling arguments** so parameter names/types match the repository.
+4. **Treat `step_path` as ephemeral** (e.g. `0`, `2.0`, `5.b0.1`). Perfecto does not persist paths; they shift after insert/move/delete. Always `view_test_structure` before the next structure edit — never reuse a path from an older mutation response.
+5. **Re-`view_test_structure` after every structure mutation** before the next edit.
+6. **Validate the device before `execute_test`**: resolve DUT via `perfecto_devices`, check real-device availability, then execute; monitor with `perfecto_execution`.
+7. **Do not invent per-test URLs.** Only lab entry exists; open tests in the UI by folder/name from `list_tests`.
+8. **Capabilities not in MCP yet** (DataTables, Scheduler, Embedded tests, Object Spy, AI Assistant chat, restore snapshot, download as Appium, etc.): send the user to the lab UI and the matching help/reference — do not fake support.
+
+Full action catalog: [mcp-tools.md](references/mcp-tools.md).
+
+### Author + run workflow
+
+```
+1. list_commands (+ get_command_definitions as needed)
+2. create_test OR list_tests → view_test_structure
+3. add_command / add_logical_step / add_loop / add_condition …
+4. view_test_structure again after each mutation
+5. Validate device (real / virtual / desktop)
+6. execute_test → list_live_executions / list_report_executions
+```
+
+## Phrase AI steps well
+
+When helping write natural-language steps, load the matching reference:
+
+- User actions → [ai-user-actions-best-practices.md](references/ai-user-actions-best-practices.md) — describe the **goal**, not locators; keep actions ≤ ~15 internal steps / 3 minutes.
+- Validations → [ai-validations-best-practices.md](references/ai-validations-best-practices.md) — binary PASS/FAIL only, not open questions.
+- Visual comparisons → [ai-visual-comparison-best-practices.md](references/ai-visual-comparison-best-practices.md) — avoid Device / Pixel Diff as failure categories unless intentional.
+- FAQ / limits → [ai-commands.md](references/ai-commands.md)
+
+## Reference files
+
+Load on demand (one level deep):
+
+| Need | File |
+| --- | --- |
+| What it is, licenses, access | [getting-started.md](references/getting-started.md) |
+| Workspace UI layout | [interface.md](references/interface.md) |
+| New/open/save/delete/versions | [handle-tests.md](references/handle-tests.md) |
+| Run / stop / DUT / reports | [execute-tests.md](references/execute-tests.md) |
+| Conditions, loops, logical steps | [control-constructs.md](references/control-constructs.md) |
+| MCP action map | [mcp-tools.md](references/mcp-tools.md) |
+| AI commands FAQ | [ai-commands.md](references/ai-commands.md) |
+| AI user-action phrasing | [ai-user-actions-best-practices.md](references/ai-user-actions-best-practices.md) |
+| AI validation phrasing | [ai-validations-best-practices.md](references/ai-validations-best-practices.md) |
+| AI visual comparison | [ai-visual-comparison-best-practices.md](references/ai-visual-comparison-best-practices.md) |
+
+## Official documentation
+
+Live catalog: [AI Scriptless](https://help.perfecto.io/perfecto-help/content/perfecto/ide/get-started-with-scriptless-mobile.htm). Prefer `perfecto_help` for pages under subcategory `ide` when a reference is missing or may be outdated.

resources/skills/perfecto-ai-scriptless/evals/evals.json

@@ -0,0 +1,40 @@
+{
+  "skill_name": "perfecto-ai-scriptless",
+  "evals": [
+    {
+      "id": 1,
+      "prompt": "I'm new to Perfecto — how do I open AI Scriptless and create my first test? Our cloud might only have Scriptless Mobile, not the full AI label.",
+      "expected_output": "Explains access path (Scriptless Automation → Build ai scriptless test), lab URL pattern, license labeling (AI Scriptless vs Scriptless Mobile), and a short create/save/run path. Mentions AI license/toggle for AI commands.",
+      "files": [],
+      "expectations": [
+        "Mentions Perfecto landing page or Scriptless Automation access path",
+        "Explains AI Scriptless vs Scriptless Mobile license difference",
+        "Mentions DUT or execute/run step for a first test",
+        "Does not invent a per-test deep link URL"
+      ]
+    },
+    {
+      "id": 2,
+      "prompt": "Using Perfecto MCP, add an AI validation step that checks the login button is green, then run the test on a free real device named something like Galaxy-S23.",
+      "expected_output": "Uses perfecto_ai_scriptless with list_commands / get_command_definitions, prefers ai_validation, phrases a PASS/FAIL validation (not an open question), view_test_structure around edits, validates device availability, then execute_test and monitoring via perfecto_execution/devices.",
+      "files": [],
+      "expectations": [
+        "Calls or describes list_commands before add_command",
+        "Uses or recommends ai_validation (not a vague open question)",
+        "Mentions view_test_structure or ephemeral step_path behavior",
+        "Includes device availability check before execute_test"
+      ]
+    },
+    {
+      "id": 3,
+      "prompt": "Help me phrase an AI user action for checkout: pick VISA and confirm. Also should I fail the visual comparison on Device and Pixel Diff?",
+      "expected_output": "Gives goal-based phrasing (not locator/coordinates), points at user-action best practices, and advises against Device/Pixel Diff as default failure categories for visual comparison.",
+      "files": [],
+      "expectations": [
+        "Phrasing describes logical goal rather than mouse coordinates or brittle labels alone",
+        "Warns against Device and Pixel Diff as default visual-comparison failure categories",
+        "References or applies AI user-action / visual-comparison guidance from the skill"
+      ]
+    }
+  ]
+}

resources/skills/perfecto-ai-scriptless/references/ai-commands.md

@@ -0,0 +1,47 @@
+# AI commands overview and FAQ
+
+Sources:
+- [AI commands FAQ](https://help.perfecto.io/perfecto-help/content/perfecto/ide/sm-ai-commands-faq.htm)
+- [AI commands](https://help.perfecto.io/perfecto-help/content/perfecto/sm-commands-and-checkpoints/ai-commands.htm)
+
+## What AI commands do
+
+AI commands (Scriptless) and AI functions (Appium) use natural language — in any language — like a conversation. Benefits:
+
+- Complex validations that may combine multiple traditional steps into one
+- Coverage beyond fragile locator-based scripting
+- Platform-agnostic steps that adapt better to UI changes
+
+In the Scriptless UI, AI commands live under the **AI** left-sidebar tab. Prefer Perfecto's AI Assistant in the lab when crafting prompts.
+
+Primary MCP command IDs (confirm via `list_commands`):
+
+- `ai_user-action` — perform actions on the DUT
+- `ai_validation` — PASS/FAIL assertions
+- `ai_visual-comparison` — screen comparison categories
+
+## Licensing FAQ
+
+- AI is **not** on by default for every Perfecto feature — only AI commands/functions.
+- A **separate Perfecto AI license** and admin **feature-toggle opt-in** are required.
+- If the org opts out later, AI-based commands cannot run and dependent tests fail.
+- Data use: see Perforce [Generative AI Policy](https://www.perforce.com/generative-ai-policy).
+
+## Limits and tips
+
+- One AI command can contain up to **30** internal steps; execution limit is **3 minutes** per call. For best results, target **≤ 15** steps.
+- Form fill / text extract with AI works on the **default tab** in web sessions (desktop browser, mobile web, WebView) — not native mobile/desktop apps, and not tabs opened mid-flow (Selenium stays on the default tab).
+- If answers are inconsistent: refine with the AI Assistant suggestions, then review best-practice docs. Ambiguous prompts are the usual cause.
+
+## Example validation conversation pattern
+
+User: "Is there a sign-up here option for new user?"  
+Assistant may suggest: "Is there a sign-up option for new users labeled 'Sign Up Here'?"
+
+Use assistant suggestions as the final validation text when possible.
+
+## Related best practices
+
+- [AI user actions](ai-user-actions-best-practices.md)
+- [AI validations](ai-validations-best-practices.md)
+- [AI visual comparisons](ai-visual-comparison-best-practices.md)

resources/skills/perfecto-ai-scriptless/references/ai-user-actions-best-practices.md

@@ -0,0 +1,69 @@
+# Best practices: AI user actions
+
+Source: [Best practices for working with AI user actions](https://help.perfecto.io/perfecto-help/content/perfecto/ide/sm-ai-user-action-best-practices.htm)
+
+Write the action once; run across devices/OS. Describe the **logical goal**, not locators.
+
+In Scriptless, use AI commands ([AI commands](https://help.perfecto.io/perfecto-help/content/perfecto/sm-commands-and-checkpoints/ai-commands.htm)). For Appium, see [AI functions](https://help.perfecto.io/perfecto-help/content/perfecto/automation-testing/ai-functions.htm).
+
+## Performance
+
+- Phrase each User Action so it resolves to **≤ 15** steps and finishes in **under 3 minutes**.
+- Enable **Reasoning** when counting, sorting, or ordering needs extra reasoning resources.
+
+## What works well
+
+- Common UI: hamburger, help, notifications
+- Click buttons, links, icons, FABs
+- Type into empty fields or replace text
+- Radio buttons, checkboxes, sliders
+- Tabs, lists, navbars
+- Dismiss interrupting popups/notifications
+- Browser navigation (go to, back, forward, refresh, clean)
+- Install mobile apps (see install section)
+
+Still evolving / may be unreliable: inline labels relative to fields, table cell/row actions, color pickers. Not yet fully tested: append text to existing field content. LLMs can make mistakes.
+
+## Why AI user actions help
+
+- **Write once, run everywhere** — no device-specific locators
+- **Survive UI renames/moves** — goal-based instructions
+- **Wait instead of fail** — instruct wait/retry/dismiss modals/progress
+- **Focus** — “focus on / zoom into …” for dense UIs
+
+## Do's and don'ts
+
+| Don't | Do | Why |
+| --- | --- | --- |
+| Move mouse to Cart button bottom-right | On the profile page, open the shopping cart | Logical goal, not coordinates/labels |
+| Register a new account | Fill the form: Enter "Jill A. Smith" in the name fields… | Enough concrete detail |
+| Click the red shoes | From search results, select the red sneakers that cost $30 | Disambiguate similar items |
+| Pay by card | Fill the form: In the drop down, select credit card… | Triggers form-fill tooling for dropdowns |
+
+## Specialized knowledge (`ai.txt`)
+
+For domain UIs (seat maps, dashboards, canvases):
+
+1. Write instructions in plain text (max **2,000** characters; larger files are ignored).
+2. Save as `ai.txt`.
+3. Upload to the **root** of the Perfecto Repository.
+4. Reloads on login; applied as general user instructions for AI assistants/commands.
+
+## Install mobile apps via User Action
+
+Example prompts: `Install public:/path/myapp.ipa`, `Install ${myApp} on the device, open it and log in`.
+
+Configure instrumentation in the UI widget (Sensor, WebView, Secured Screen, Resign). iOS notes:
+
+- Virtual iOS: use `appname.zip`
+- Real iOS: Resign Application on by default (mandatory unless WebView changes rules)
+
+See [Install a mobile app using AI User Actions](https://help.perfecto.io/perfecto-help/content/perfecto/ide/sm-ai-user-action-install-app.htm).
+
+## Fill forms
+
+Prefer **one** prompt that fills many fields, then a separate submit action.
+
+Supported: text/textarea, native `<select>`, `date` (YYYY-MM-DD), checkbox/radio, common combobox libraries.
+
+Not supported: masked inputs, custom date pickers, sliders, file upload, rich-text editors, chip/tag inputs, cross-origin iframes, closed Shadow DOM, canvas inputs.

resources/skills/perfecto-ai-scriptless/references/ai-validations-best-practices.md

@@ -0,0 +1,58 @@
+# Best practices: AI validations
+
+Source: [Best practices for working with AI validations](https://help.perfecto.io/perfecto-help/content/perfecto/ide/sm-ai-validations-best-practices.htm)
+
+Validations should be logical, business-oriented, and answerable from the screen (or factual public knowledge / variables).
+
+## Question vs validation
+
+| Type | Nature | Examples |
+| --- | --- | --- |
+| **Question/query** | Open-ended (what/who/where…) — **not** a validation | How much is the balance? What color is the button? |
+| **Validation** | Binary PASS/FAIL | Is the balance equal to $22? Verify the login button is green. Does user Joe appear in the header? |
+
+## Limitations
+
+- Models can err — test prompts with the AI Assistant before baking into Scriptless.
+- Not connected to your backend or “the world” beyond the screen + public facts. Time reference is environment **UTC**.
+- Enable **Reasoning** for counting/sorting/ordering.
+
+## Semantic approach
+
+AI matches meaning to what is visible (like a human tester), not DOM locators. Prefer validations that state the business meaning clearly (“available balance equals …”) rather than fragile positional guesses.
+
+## Conversation-driven assistant
+
+Use the lab AI Validation Assistant to turn ambiguous asks into objective PASS/FAIL text. Example flow:
+
+1. User: “Does the pizza costs $80?”
+2. Assistant may interpret “any pizza” and suggest a clearer sentence.
+3. User adds context (“Capsicum one?”).
+4. Assistant returns FAIL plus a precise suggestion: “Is the Capsicum pizza price equal to $80?”
+
+The assistant also fixes grammar/typos and works in many languages (best results when matching the app language).
+
+## Wait, time, variables, knowledge
+
+- Loading screens: AI validation can retry until loaded; or add a User Action “Wait for … then continue.”
+- Time: ask relative to UTC / timezone conversions.
+- Variables and factual checks are allowed if the answer is binary (`Is ${x} between 1 and 10?`).
+- Subjective questions (“tasty?”, “beautiful?”) are invalid.
+- Missing variables: Assistant may prompt and add a test variable.
+
+## Specialized knowledge
+
+Same `ai.txt` repository pattern as user actions (≤ 2,000 characters). See [ai-user-actions-best-practices.md](ai-user-actions-best-practices.md).
+
+## Do's
+
+- Be specific and objective
+- Ask about elements clearly shown on screen
+- Examples: “Do the triple black sneakers cost $601?”, “Is Nike the first in the filter list options?”
+
+## Don'ts
+
+- Subjective words (nice, clear, understood)
+- Ambiguous counts when items are partially visible or ads confuse counts
+- Vague comparatives (“much more expensive”)
+- External knowledge you cannot see (“cheaper than at Amazon”)