☕ Spectacles ML Example — Coffee Cup Detection

A Lens Studio project that detects a real coffee cup through your Spectacles and rises steam off it. The detector was trained in one click — no machine-learning setup required — and you can retrain it on anything you like.

This Lens is a working example: the bundled coffee cup detector (coffee.onnx) was trained exactly the way described below — real footage recorded on the Spectacles, plus synthetic frames generated with FLUX. Everything here shows how it was made, how to retrain it on anything you like, and how to tweak it.

---

1. Train a model in one click

The detector is trained on Replicate — open the link, fill in the form, hit Run. That's the whole pipeline:

⚠️ Download your result right away. Replicate only keeps the output .zip for about an hour. Refresh the run page often and grab the file as soon as it finishes — once it expires you'll have to retrain.

1. Real footage — upload a .zip of frames captured through Spectacles (images_zip_url).
2. Synthetic data (FLUX) — FLUX.1-schnell generates extra POV training frames from text prompts and mixes them in, so you don't need to capture hundreds of images yourself.
3. Auto-labeling — SAM 3 draws the bounding boxes for you against your class names.
4. Train + export — YOLOv7-tiny is fine-tuned and exported to a 224×224 SnapML ONNX.

A synthetic FLUX frame with the bounding box SAM 3 auto-labeled for training — no manual annotation:

You get back a single .zip containing best.onnx, best.pt, data.yaml, and classes.txt.

The knobs that matter on the Replicate form:

Input	What it does
images_zip_url	Your real Spectacles frames (the more, the better).
classes	What to detect, e.g. coffee cup or banana, apple.
synthetic_count	How many FLUX frames to generate (0 = real footage only).
synthetic_prompts	One POV scene per line — describe angle, object, surface, lighting.
epochs	Training length (200 is a good default).

➡️ Full pipeline details: /Users/stijnspanhove/scripts/cog-spectacles-yolov7-trainer

Use your trained model in this Lens

1. Unzip the Replicate output and rename best.onnx → coffee.onnx.
2. Replace Assets/coffee.onnx with it.
3. If you changed the class name, update the label in Assets/Scripts/CoffeeMLController.ts:209 (new CoffeeDetector(["coffee cup"], ...)).

---

2. Share it

• Share the trainer — the Replicate model is public. Anyone can open the link and train their own detector (their own objects, their own footage) with no code.
• Share the Lens — publish this project from Lens Studio to your Spectacles, or send the Lens link so others can try it on their own glasses.

---

3. Test it — grab a coffee cup

1. Open the project in Lens Studio and push it to your Spectacles (or run it in Preview).
2. Pinch to place a surface in front of you (a tabletop works best).
3. Look around to scan. Point your gaze at a real coffee cup.
4. When it's detected, steam rises off the cup and "Coffee cup found" appears briefly.

No cup handy? Toggle previewSteamOnStart ON in the CoffeeExperienceController to show the steam effect in front of the camera without placing or detecting anything.

Debug view (2D detection)

The scene includes a debug overlay that shows exactly what the model sees: a small screen-space inset of the live camera frame with the detection boxes drawn on top. It's driven by the DetectionDebugOverlay component — toggle showCameraFeed and tune the panel position/color in the Inspector. Use it to confirm the model is locking onto the cup (and where) before worrying about the 3D steam placement.

---

4. Change the score (detection sensitivity)

The "score" is the model's confidence threshold — how sure it must be before it accepts a detection. It lives on the CoffeeMLController component as the scoreThreshold slider (0–1, default 0.5) — see Assets/Scripts/CoffeeMLController.ts:31.

• Lower it (e.g. 0.3) → detects more easily, but more false positives (steam on non-cups).
• Raise it (e.g. 0.7) → only fires on confident matches, but may miss cups.

You can drag the slider live in the Inspector during Preview — it updates every frame.

Related knobs on the same component:

• iouThreshold (default 0.45) — overlap cutoff for merging duplicate boxes.
• frameSkip (default 3) — run inference every N frames; higher = better battery/perf.

---

How it fits together

File	Role
CoffeeMLController.ts	Runs coffee.onnx on the camera feed, emits detections.
CoffeeDetector.ts	Decodes YOLO output into boxes + scores (NMS).
CoffeeExperienceController.ts	Surface placement, places & smooths the steam on the cup.
Assets/coffee.onnx	The trained detector — swap this to detect anything.

---

Built for Spectacles with Lens Studio · detector trained via stspanho/spectacles-yolov7-trainer

Created by Pavlo & Stijn — pavlo-stijn.dev