A gesture-driven 3D voxel building experience powered by AI hand tracking
Build 3D structures using nothing but your hands — no mouse, no keyboard, no controller.
| Gesture | Hand | Action |
|---|---|---|
| ☝️ Index finger | Right | Aim cursor position |
| ✌️ Index + Middle | Right | Place voxels continuously |
| ✊ Fist drag | Left | Rotate the 3D scene |
| 🤏 Pinch + drag | Left | Pan the scene |
| 🤌 Pinch (scale) | Left | Zoom in/out |
Three tiers of challenges to test your skills:
- 🏗️ Shape Challenges — Replicate target shapes with precision
- ⚡ Speed Challenges — Build against the clock
- 🎯 Precision Challenges — Place voxels at exact coordinates
Each challenge features:
- 🕐 3-2-1 countdown with sound effects
- 📊 Live progress bar and timer
- ⭐ Star rating system (1-3 stars)
- 📈 Performance metrics (accuracy, stability, mistakes)
- 👶 Kids mode with easier tolerances
- Glassmorphism design with animated gradients
- Real-time webcam feed with scanning overlay
- Ghost target voxels with glow effects
- Sound effects via Web Audio API
- Node.js 18+ and npm
- A webcam (built-in or external)
- Google Chrome recommended (best MediaPipe support)
# Clone the repository
git clone https://github.com/AIabood/AI-Hand-Tracked.git
cd AI-Hand-Tracked
# Install dependencies
npm install
# Start the development server
npm run devOpen http://localhost:5173 in your browser and allow camera access.
src/
├── components/ # UI components
│ ├── UIOverlay.jsx # Main HUD overlay
│ ├── GuidedUI.jsx # Challenge selector & dashboard
│ ├── ChallengeHUD.jsx # Countdown, timer, progress bar
│ └── ChallengeResults.jsx # Star rating & results screen
├── handTracking/ # MediaPipe integration
│ ├── HandTracker.jsx # Hand detection & gesture recognition
│ └── GestureDetector.js # Gesture classification logic
├── scene/ # 3D scene components
│ ├── VoxelScene.jsx # Main R3F scene renderer
│ ├── SceneInteraction.jsx # Gesture → 3D action mapping
│ └── TrackpadInteraction.jsx # Fallback mouse controls
├── voxel/ # Voxel system
│ ├── useVoxelStore.js # Zustand state management
│ ├── Voxel.jsx # Individual voxel renderer
│ ├── GhostModel.jsx # Ghost target visualization
│ ├── GhostVoxel.jsx # Placement preview voxel
│ └── challenges.json # 8 built-in challenges
├── utils/
│ └── sounds.js # Web Audio API sound effects
└── index.css # Global styles & design system
| Technology | Purpose |
|---|---|
| React 18 | UI framework |
| React Three Fiber | Declarative 3D rendering |
| Three.js | 3D graphics engine |
| MediaPipe Hands | Real-time hand tracking |
| Zustand | Lightweight state management |
| Vite | Build tool & dev server |
| Lucide React | Icon library |
| Web Audio API | Sound effects |
| # | Challenge | Type | Difficulty | Time Limit |
|---|---|---|---|---|
| 1 | Single Voxel | 🎯 Precision | Easy | ∞ |
| 2 | Row of 3 | 🏗️ Shape | Easy | ∞ |
| 3 | L-Shape | 🏗️ Shape | Easy | 60s |
| 4 | Staircase | 🏗️ Shape | Medium | 90s |
| 5 | Cube 2×2×2 | 🏗️ Shape | Medium | 120s |
| 6 | House | 🏗️ Shape | Hard | 180s |
| 7 | Speed Row | ⚡ Speed | Medium | 30s |
| 8 | Precision Dot | 🎯 Precision | Hard | 45s |
┌──────────┐ ┌──────────────┐ ┌──────────────┐ ┌──────────┐
│ Webcam │ → │ MediaPipe │ → │ Gesture │ → │ Three.js │
│ Input │ │ Hand Model │ │ Classifier │ │ Scene │
└──────────┘ └──────────────┘ └──────────────┘ └──────────┘
↓
┌──────────────┐
│ Zustand │
│ Store │
└──────────────┘
- Webcam captures video at 640×480
- MediaPipe Hands detects 21 hand landmarks per hand
- GestureDetector classifies gestures (pinch, fist, point, etc.)
- SceneInteraction maps gestures to 3D actions
- Zustand Store manages state reactively
- React Three Fiber renders the 3D scene at 60fps
This project is open source and available under the MIT License.
Built with ❤️ using AI Hand Tracking