🤖 AI Web Action Framework

Proof of Concept — An AI agent that can perform real actions inside a user's browser.

---

💡 The Idea

What if an AI chatbot could do more than just reply with text? What if it could actually click buttons, navigate pages, fill forms, and manipulate the DOM — all from a simple chat conversation?

This project proves that's possible on the web, today, using nothing but a hidden communication protocol between a server-side AI and a lightweight JavaScript SDK.

---

✅ What This Proof of Concept Demonstrates

Capability	Status	How It Works
AI → Browser Navigation	✅ Working	AI tells the browser to window.location.href to a URL
AI → Browser Alerts	✅ Working	AI triggers native alert() dialogs with dynamic messages
AI → Console Logging	✅ Working	AI sends debug info to console.log()
AI → DOM Manipulation	✅ Working	AI updates any element's text/HTML by ID (calculator demo)
Multiple Actions in One Response	✅ Working	AI can fire several browser actions from a single response
Custom Action Registration	✅ Working	Developers can register any custom JS function as a "tool"
Hidden Protocol (User Never Sees JSON)	✅ Working	Action tokens are embedded as HTML comments — invisible to end users

---

🏗️ Architecture

┌─────────────────────────────────────────────────────────┐
│                    USER'S BROWSER                       │
│                                                         │
│  ┌───────────────┐     ┌────────────────────────────┐   │
│  │   Chat UI     │ ──▶ │  AIWebAgent (JS SDK)       │   │
│  │  (clean text) │     │  • Parses action tokens     │   │
│  └───────────────┘     │  • Executes registered      │   │
│                        │    JavaScript functions      │   │
│                        │  • Returns clean text for UI │   │
│                        └────────────────────────────┘   │
│                                   ▲                     │
└───────────────────────────────────│─────────────────────┘
                                    │  HTTP / SSE
┌───────────────────────────────────│─────────────────────┐
│               LARAVEL BACKEND (The Brain)               │
│                                   │                     │
│  ┌────────────┐   ┌───────────────┴──────────────────┐  │
│  │ LLM / AI   │──▶│  ClientActionTool (PHP)          │  │
│  │ (GPT, etc) │   │  • NavigateToPage                │  │
│  └────────────┘   │  • ShowAlert                     │  │
│                   │  • (extensible via subclasses)    │  │
│                   └──────────────────────────────────┘  │
└─────────────────────────────────────────────────────────┘

The Key Trick: Hidden Action Tokens

The AI's response looks like this to the user:

"Sure! I'm navigating you to the product page now."

But the raw response actually contains:

Sure! I'm navigating you to the product page now.
<!-- CLIENT_ACTION_START {"action":"navigate","payload":{"url":"/products/iphone-15"}} CLIENT_ACTION_END -->

The hidden HTML comment is parsed by the JS SDK, which executes the action and strips the token before displaying the text. The user never sees the JSON.

---

📂 Project Structure

experiment-001/
├── README.md                           ← You are here
├── Guideline.md                        ← Architecture spec & implementation guidelines
│
├── app/Ai/Tools/                       ← 🔧 Backend (Laravel)
│   ├── ClientActionTool.php            ← Abstract base class with dispatch() method
│   ├── NavigateToPage.php              ← Concrete tool: browser navigation
│   └── ShowAlert.php                   ← Concrete tool: browser alerts
│
└── public/                             ← 🌐 Frontend
    ├── js/ai-browser-tools.js          ← AIWebAgent SDK (vanilla JS, ~230 lines)
    └── simulate.html                   ← Interactive test page with all demos

---

🚀 Running the Demo

No backend required! The simulation page works standalone:

1. Open public/simulate.html in any browser
2. Click the preset buttons to test each action type:
   • 🧭 Navigate — Simulates browser redirect (suppressed for safety)
   • ⚠️ Alert — Triggers a browser alert via the protocol
   • 📋 Console — Logs to browser DevTools console
   • 🔗 Multiple — Fires two actions from one response
   • 🧮 Calculator — Full DOM manipulation demo (AI "calculates" and updates the page)
3. Use the Custom Simulation textarea to test your own payloads

---

🔧 How It Works — Code Examples

Backend: Creating a New Client Tool (PHP)

class AddToCart extends ClientActionTool
{
    public function actionName(): string
    {
        return 'add_to_cart';
    }

    public function description(): Stringable|string
    {
        return 'Add a product to the shopping cart in the browser.';
    }

    public function schema(JsonSchema $schema): array
    {
        return $schema->object([
            'product_id' => $schema->string('The product ID.'),
            'quantity'    => $schema->integer('Number of items to add.'),
        ])->require(['product_id', 'quantity']);
    }

    public function handle(Request $request): Stringable|string
    {
        return $this->dispatch($this->actionName(), [
            'product_id' => $request->input('product_id'),
            'quantity'    => $request->input('quantity'),
        ]);
    }
}

Frontend: Registering the Matching JS Handler

const agent = new AIWebAgent();

agent.registerTool('add_to_cart', (payload) => {
    fetch('/api/cart', {
        method: 'POST',
        body: JSON.stringify({
            product_id: payload.product_id,
            quantity: payload.quantity,
        }),
    }).then(() => {
        updateCartBadge();
        showToast('Item added to cart!');
    });
});

---

🔮 Future Roadmap

Phase 1: Core Enhancements

• Streaming support — Parse action tokens from SSE (Server-Sent Events) streams in real-time
• Action acknowledgment — Browser reports back to the server whether an action succeeded or failed
• Action queueing — Buffer multiple rapid actions and execute sequentially with optional delays
• Conditional actions — Server sends actions that only execute if certain DOM conditions are met

Phase 2: Rich Browser Tools

• Form auto-fill — AI fills forms by targeting input fields by name/ID
• Click simulation — AI triggers click events on buttons and links
• Scroll to element — AI scrolls the viewport to a specific section
• CSS class toggling — AI adds/removes classes (e.g., dark mode, highlights)
• Toast/notification system — Non-blocking UI notifications instead of alert()
• Modal dialogs — AI opens rich modal content with HTML

Phase 3: Advanced Capabilities

• Multi-step workflows — AI chains multiple browser actions into sequences (e.g., "fill form → submit → navigate to confirmation")
• Visual feedback — Highlight elements the AI is about to interact with
• Undo system — Let users revert AI-triggered browser actions
• Permission system — User approves certain actions before execution (e.g., "AI wants to navigate away, allow?")
• Sandbox mode — Preview what actions the AI would take without executing them

Phase 4: Framework & Ecosystem

• NPM package — Publish the JS SDK as an installable package
• Composer package — Publish the Laravel tools as a standalone package
• React/Vue adapters — Framework-specific wrappers for the SDK
• WordPress plugin — Bring AI browser actions to WordPress sites
• TypeScript types — Full type definitions for the SDK
• Action marketplace — Community-contributed tool packs (e-commerce, forms, analytics)

---

🛡️ Security Considerations

Measure	Implementation
Whitelist-only execution	Only registered tools can run — no eval() ever
URL validation	Navigation restricted to relative or whitelisted domains
Payload validation	Both backend schema validation and frontend type checking
No server-side effects	Client tools return tokens only — they don't modify server state
HTML comment hiding	Tokens are invisible in rendered HTML, preventing social engineering

---

🧪 Tech Stack

Layer	Technology	Purpose
AI Brain	Laravel 12+ with Laravel AI SDK	Tool definitions, LLM orchestration
Protocol	Hidden HTML comments with JSON	Communication bridge
Browser SDK	Vanilla JavaScript (AIWebAgent)	Token parsing, tool execution
Test Page	Pure HTML/CSS/JS	No build tools needed

---

🤝 Why This Matters

Traditional AI chatbots are read-only — they can only respond with text. This framework makes AI read-write on the web, enabling:

• E-commerce: "Add that jacket to my cart" → AI actually adds it
• SaaS dashboards: "Show me last month's revenue" → AI navigates to the report
• Customer support: "Reset my password" → AI triggers the reset flow in the browser
• Accessibility: Voice-driven AI that physically operates the interface for users

The web browser becomes the AI's hands. 🖐️

---

Built as a proof of concept — proving that AI-driven browser actions are possible, practical, and ready for production.