Fitment

Winner, Partly x WDCC x DEVS 2026 hackathon

A repairer talks through a walkaround of a damaged car, however long it takes. By the time they're back at their desk, the damage is transcribed and matched to real parts, each part has an AI-generated 3D model they can place in AR at true scale next to the car, and a priced, supplier-compared order is ready to send.

Identifying damage isn't the hard part, a repairer can do that in seconds by eye. What's expensive is everything after: getting the exact right part, from the right supplier, ordered correctly, the first time. Miss one and you find out weeks later once the car's already stripped down, and the job stops cold. The goal here is that the walkaround narration becomes the order itself.

Architecture

video + narration
      |
      v
  Deepgram (transcription, batch or live)
      |
      v
  OpenAI (match narration to a damaged part + timestamp)
      |
      v
  Partly API (match to a real catalogue part)
      |
      v
  phone: plays video, polls for results, renders overlays / 3D / AR

The phone never does frame-by-frame analysis. React Native has no way to read decoded video frames on-device (no <canvas>, no getImageData, and expo-gl has no video-texture path for it), so that's not a performance choice, it's the only option that works at all on this stack. All the actual detection work happens server-side or through hosted APIs, and the phone's job is to play, poll, and render.

What's actually running

Everything below is wired end to end and runs on a phone through Expo Go. One screen needs a custom dev build, see Native AR.

Voice to damaged parts, two ways

A batch pipeline sends the walkaround video straight to Deepgram for word-level-timestamped transcription, then a tool-calling OpenAI pass reads the transcript and returns each damaged part with a timestamp and the quote that supports it (backend/analysis.js). There's also a live path (backend/liveAnalysis.js): ffmpeg re-reads a video's audio at real-time pace and streams raw PCM into Deepgram's live WebSocket API, with each finalized utterance fed to OpenAI as it lands. The phone polls a growing event list instead of holding a socket open, so the Expo Go client stays on plain fetch.

Real parts data, not placeholders

The dataset is 8 real damaged vehicles with full OEM parts catalogues, hotspot-annotated diagrams, and Partly's own AI damage-to-catalogue-part predictions, served through a FastAPI service (packages/partly/). Matched parts resolve against this, not a mock catalogue.

Photo to 3D to AR, at real-world scale

Pick a damaged part and the app sends a photo to Hunyuan3D (via fal.ai) and gets back a textured 3D mesh. That mesh gets converted from GLB to USDZ and scale-normalized to the part's real dimensions, then handed to a custom native Expo Module (modules/expo-car-ar, Swift/ARKit/RealityKit) that anchors it in the world at true physical size. Hold the phone up to the actual car and the replacement part sits where it'll go, at the size it'll actually be.

Compare and track

Once a part's identified, the app compares supplier price, condition and ETA, and carries it through to an order-tracking view.

What's mocked

• Supplier price comparison is deterministic mock data (the same part always returns the same listings), no live supplier API yet.
• Order tracking is a hardcoded demo list, not a real order backend.
• The dataset ships hidden-damage predictions (what typically breaks behind a damaged part), that data exists in the packages/partly/ API but isn't surfaced in the app UI yet.
• Live-mic capture in the demo falls back to a pre-recorded walkaround video rather than an actual live recording. The real-time transcription pipeline behind it is real and does run.

Native AR

src/screens/ar/ is the one deliberate exception to everything running in Expo Go. Anchoring a 3D model to a real surface at true physical scale is an ARKit capability with no JS-only equivalent, expo-camera and react-native-vision-camera can't do world tracking, and vision-camera's native dependencies can't run in Expo Go at all. So this screen ships a small native Expo Module written in Swift against ARKit/RealityKit, guarded so importing it never crashes Expo Go on platforms that don't support it (it just falls back to a message). It needs a custom EAS dev client build and a physical iPhone, since ARKit doesn't do camera-based world tracking in the Simulator. Everything else in the app, on every platform, stays plain Expo Go.

Tech stack

• React Native (Expo SDK 57) via Expo Go: expo-video / expo-audio for playback, expo-camera for capture
• Deepgram (batch + live WebSocket) for transcription, OpenAI tool-calling for part extraction
• fal.ai / Hunyuan3D for image-to-3D generation, converted to USDZ and scale-normalized for AR
• A native Swift / ARKit / RealityKit Expo Module for scale-accurate AR placement
• FastAPI service over Partly's real vehicle/parts/diagram dataset

Getting started

npm run setup   # checks Node/npm and installs dependencies
npm start       # starts the Partly API + Expo dev server, scan the QR code with Expo Go

Full setup, troubleshooting, and the native AR build steps are in SETUP.md

Status

Built in two days for a hackathon. It's a working prototype, not a production system. CLAUDE.md has the architectural constraints and decisions behind it, for anyone picking this repo up next.