Skip to content

task.md

Latest commit

 

History

History
78 lines (68 loc) · 5.4 KB

task.md

File metadata and controls

78 lines (68 loc) · 5.4 KB
Raw

Universal Digital Accessibility Agent — Task Tracker

Phase 1: Project Scaffolding

  • Create project directory structure
  • Initialize Next.js frontend (frontend/)
  • Initialize FastAPI backend (backend/)
  • Create env config files (.env.example)
  • Create Docker & Cloud Run configs

Phase 2: Backend Core (FastAPI + Gemini)

  • config.py — Pydantic Settings, env vars
  • main.py — FastAPI app, CORS, REST endpoints
  • websocket.py — WebSocket handler /ws/{session_id}
  • browser.py — Playwright browser manager, action executor, screenshot capture
  • computer_use.py — Gemini Computer Use agent loop
  • live_stream.py — Gemini Live API streaming
  • storage.py — Cloud Storage integration
  • firestore_client.py — Firestore session/task state
  • pubsub.py — Pub/Sub action queue
  • requirements.txt & Dockerfile

Phase 3: Frontend (Next.js + TypeScript)

  • globals.css — Accessibility-first dark theme design system
  • types.ts — TypeScript types & interfaces
  • useWebSocket.ts — WebSocket hook with auto-reconnect
  • layout.tsx — App layout shell
  • page.tsx — Main page assembly
  • TaskInput.tsx — Task input with URL & example tasks
  • AgentStatus.tsx — Real-time status & step counter
  • ActionFeed.tsx — Live action feed
  • ScreenView.tsx — Live screenshot viewer with overlays
  • SessionHistory.tsx — Past sessions list
  • Safety confirmation dialog

Phase 4: Integration & Verification

  • Start backend and verify health endpoint
  • Start frontend dev server
  • TypeScript type-check (npx tsc --noEmit)
  • Visual QA of UI in browser

Phase 5: Deployment Config & Docs

  • Premium Dashboard UI Overhaul (Glassmorphism, Animations, Lucide Icons)
  • docker-compose.yml
  • README.md with setup, architecture, demo

Phase 6: Hybrid Minimal-GCP Architecture

  • Refactor config.py (remove GCS/PubSub config)
  • Refactor storage.py to save screenshots to local disk
  • Refactor pubsub.py to log actions to local file
  • Update requirements.txt (drop google-cloud-storage, google-cloud-pubsub)

Phase 7: Final Execution & Debugging

  • Configure necessary IAM roles (roles/datastore.user, roles/serviceusage.serviceUsageConsumer)
  • Integrate valid Gemini Computer Use model (gemini-2.5-computer-use-preview-10-2025)
  • Start FastAPI and Next.js reliably
  • Validate end-to-end mission execution of Universal Digital Accessibility Agent

Phase 8: Dashboard UI Redesign

  • Analyze user-provided HTML template
  • Integrate Tailwind CSS configuration into Next.js workspace
  • Restructure and rewrite page.tsx integrating UI state bindings with Tailwind designs
  • Fix Material UI rendering bugs via direct stylesheet injection
  • Validate interactions and state updates using Browser Subagentes

Phase 9: MVP Live Browser Integration

  • Implement BrowserAdapter interface layer in backend
  • Refactor browser.py into playwright_adapter.py
  • Create basic Chrome Extension structure (manifest.json, content_script.js, etc.)
  • Implement LiveBrowserAdapter and WebSocket communication
  • Modify computer_use.py to route commands through selected adapter
  • Add 'Execution Mode' toggle to Next.js dashboard UI
  • Implement Chrome Extension visual overlay and frame throttling (2-3 FPS)
  • Test end-to-end communication and browser control loop

Phase 10: Automated Live Browser Launching

  • Modify LiveBrowserAdapter to auto-launch a local Chrome instance using Playwright's launch_persistent_context
  • Inject the unpacked UDAA Extension directly into the Chrome instance during startup
  • Pass the active Session ID automatically to the extension so the user doesn't have to manually connect via the toolbar icon