🤖 WALL-E — Autonomous Physical AI Companion & Robotics Agent

An indigenous physical AI robotics agent powered by Raspberry Pi 5, Google Gemini Live API, Faster-Whisper edge wake engine, and ESP32 dual-OLED expressive eyes.

🌟 Overview

WALL-E is an embodied AI robotics platform engineered to bring real-time multimodal intelligence into the physical world. Unlike pure software chatbots, WALL-E combines edge audio processing, offline wake word detection, dynamic Voice Activity Detection (VAD), physical facial expressions on dual OLED displays, and serial motor navigation with the reasoning power of Google's Gemini Live API.

🛠️ System Architecture

                               ┌──────────────────────────────────────────────┐
                               │           Raspberry Pi 5 (8GB)               │
                               │                                              │
  [ USB Microphone ] ─────────►│  AudioPipeline                               │
                               │    ├── Virtual Split Queue                   │
                               │    └── Dynamic Noise Floor VAD               │
                               │                                              │
                               │  WakePipeline (Faster-Whisper)               │
                               │    └── Wake Word: "Hey WALL-E" / "WALL-E"    │
                               │                                              │
                               │  IdentityPipeline                            │
                               │    ├── ECAPA-TDNN Voice Biometrics           │
                               │    └── OpenCV / dlib Face Engine             │
                               │                                              │
                               │  WalleSession (Gemini Live Full-Duplex)      │
                               │    ├── WebSockets Bidirectional Audio Stream │
                               │    ├── Context Memory & Behavioral State     │
                               │    └── Live Tool Invocation                  │
                               │                                              │
  [ USB / 3.5mm Speaker ] ◄────┤  ALSA / PipeWire Software Debounced Audio    │
                               └──────┬───────────────────────────────┬───────┘
                                      │ UART Serial                   │ USB Serial
                                      ▼                               ▼
                       ┌────────────────────────────┐  ┌────────────────────────────┐
                       │   ESP32 Emotion Engine     │  │   Arduino Motor Controller │
                       │                            │  │                            │
                       │ Dual 128x64 I2C OLED Eyes  │  │ Differential Drive Motors  │
                       │ (Blink, Happy, Curious,    │  │ (Forward, Turn, Spin,      │
                       │  Sleep, Surprise, Neutral) │  │  Obstacle Avoidance)       │
                       └────────────────────────────┘  └────────────────────────────┘

🚀 Key Engineering Highlights

Edge Wake Word Pipeline: Low-latency offline wake word detection using faster-whisper with dynamic noise floor calibration to prevent audio hallucinations on USB microphones.
Full-Duplex Gemini Live Streaming: Bidirectional raw PCM audio streaming over low-latency WebSockets with sub-second response times.
Hardware Noise Debounce & AEC: Custom software-level Acoustic Echo Cancellation (AEC) debouncing and ALSA audio loop suppression, eliminating audio feedback loops without hardware AEC chips.
Dual OLED Emotion Eyes: ESP32 microcontroller controlling dual 0.96" SSD1306 OLED displays rendering organic procedural eye animations (blinking, squinting, looking around, expressions) in sync with AI agent state transitions.
Persistent Behavioral Memory: Lightweight SQLite engine retaining user profiles, cross-session conversational memory, and emotional context while staying within strict token budgets.
Differential Drive Navigation: Serial interface to Arduino motor drivers supporting offline voice navigation commands (via Vosk speech recognition) and autonomous obstacle avoidance.
Systemd Production Daemon: Auto-starting Linux daemon (walle.service) with watchdog health monitoring and graceful USB hotplug recovery.

🔌 Hardware Wiring & Pinout

ESP32 Dual OLED Display (UART Control)

ESP32 Pin	OLED 1 (Left Eye)	OLED 2 (Right Eye)	Raspberry Pi 5
3.3V	VCC	VCC	—
GND	GND	GND	GND (Pin 6)
GPIO 21 (SDA)	SDA	SDA (0x3D)	—
GPIO 22 (SCL)	SCL	SCL	—
GPIO 16 (RX2)	—	—	TXD / GPIO 14 (Pin 8)
GPIO 17 (TX2)	—	—	RXD / GPIO 15 (Pin 10)

📦 Software Stack

Host OS: Raspberry Pi OS Bookworm 64-bit
Core Orchestrator: Python 3.11, asyncio, sounddevice, numpy, scipy
AI / LLM: Google Gemini Live API (WebSocket bidirectional audio)
Local Edge ML: Faster-Whisper, Vosk, OpenCV
Microcontrollers: ESP32 (C++/Arduino), Arduino Uno/Nano
Audio Routing: ALSA, PipeWire, WebRTC AEC
Database: SQLite3

⚡ Quick Start (Raspberry Pi 5)

1. Clone & Setup

git clone https://github.com/SahilKumar337/WALLE-AI-Robot.git ~/walle
cd ~/walle
python3 -m venv venv --system-site-packages
source venv/bin/activate
pip install -r requirements-pi.txt

2. Configure Environment

cp .env.example .env
nano .env  # Add GEMINI_API_KEY

3. Launch WALL-E

python3 main.py

4. Enable Background Service (Auto-boot on Pi)

sudo cp deploy/pi/walle.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now walle.service

📁 Repository Structure

├── main.py                     # Master hardware orchestrator
├── server.py                   # Optional FastAPI web visualizer
├── core/
│   ├── config.py               # Central settings & hardware pins
│   ├── logger.py               # Multi-level structured logger
│   ├── registry.py             # Async pipeline lifecycle manager
│   └── tts.py                  # Local fallback TTS & audio chimes
├── pipelines/
│   ├── audio_pipeline.py       # Dual-channel mic capture & dynamic VAD
│   ├── wake_pipeline.py        # Faster-whisper offline wake detection
│   ├── gemini_pipeline.py      # Google Gemini Live WebSocket session
│   ├── identity_pipeline.py    # Voice biometrics & face recognition
│   ├── navigation_pipeline.py  # Vosk offline commands & serial routing
│   └── tool_pipeline.py        # Function calling & external tools
├── arduino_firmware/
│   ├── walle_eyes/             # ESP32 dual OLED animation firmware
│   └── robot_control.ino       # Arduino motor control sketch
├── deploy/
│   └── pi/                     # Systemd service & deployment scripts
└── scripts/                    # Audio setup & ALSA/PipeWire configuration

👤 Author

Sahil Kumar

GitHub: @SahilKumar337
LinkedIn: linkedin.com/in/sahilkumar337
Portfolio: sahil-kumar.vercel.app

Name		Name	Last commit message	Last commit date
_archive		_archive
arduino_firmware		arduino_firmware
core		core
deploy/pi		deploy/pi
pipelines		pipelines
scripts		scripts
static		static
storage		storage
.dockerignore		.dockerignore
.env.example		.env.example
.gitignore		.gitignore
.railwayignore		.railwayignore
Dockerfile		Dockerfile
README.md		README.md
Sign_Up_Plan.md		Sign_Up_Plan.md
audio_engine.py		audio_engine.py
auth.py		auth.py
check_db.py		check_db.py
database.py		database.py
debug_http.py		debug_http.py
debug_signup.py		debug_signup.py
emotion_engine.py		emotion_engine.py
enroll_sahil.py		enroll_sahil.py
face_engine.py		face_engine.py
implementation_plan.md		implementation_plan.md
main.py		main.py
memory.py		memory.py
navigation_engine.py		navigation_engine.py
requirements-pi.txt		requirements-pi.txt
requirements-prod.txt		requirements-prod.txt
requirements.txt		requirements.txt
scratch.py		scratch.py
serial_engine.py		serial_engine.py
server.py		server.py
sifra_live.py		sifra_live.py
sifra_session.py		sifra_session.py
sifra_tools.py		sifra_tools.py
start_sifra.bat		start_sifra.bat
test_conn_error.py		test_conn_error.py
test_memory.py		test_memory.py
test_tools.py		test_tools.py
test_tools2.py		test_tools2.py
verify_models.py		verify_models.py
verify_sifra.py		verify_sifra.py
walle_memory.json.migrated		walle_memory.json.migrated
watchdog.py		watchdog.py

Navigation Menu

Repository files navigation

🤖 WALL-E — Autonomous Physical AI Companion & Robotics Agent

🌟 Overview

🛠️ System Architecture

🚀 Key Engineering Highlights

🔌 Hardware Wiring & Pinout

ESP32 Dual OLED Display (UART Control)

📦 Software Stack

⚡ Quick Start (Raspberry Pi 5)

1. Clone & Setup

2. Configure Environment

3. Launch WALL-E

4. Enable Background Service (Auto-boot on Pi)

📁 Repository Structure

👤 Author

About

Releases

Packages

Contributors

Languages

Latest commit

History

Folders and files

Repository files navigation

🤖 WALL-E — Autonomous Physical AI Companion & Robotics Agent

🌟 Overview

🛠️ System Architecture

🚀 Key Engineering Highlights

🔌 Hardware Wiring & Pinout

ESP32 Dual OLED Display (UART Control)

📦 Software Stack

⚡ Quick Start (Raspberry Pi 5)

1. Clone & Setup

2. Configure Environment

3. Launch WALL-E

4. Enable Background Service (Auto-boot on Pi)

📁 Repository Structure

👤 Author

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages