▶ Watch the 2:56 demo — the live pipeline, the guardrail audit flagging our own model output, and a first-class refusal on an unusable frame.
Intelligent, spoken doorway accessibility descriptions scheduled for blind and low-vision viewers. Built for the Amazon Build, Ship, Shape Developer Hackathon 2026 — Ring Track
Doorstep transforms Ring smart doorbell and camera events into objective, spoken accessibility descriptions for blind and low-vision viewers.
When a Ring camera detects motion or a doorbell button is pressed, Doorstep:
- Normalizes the webhook event schema using official Ring Partner API metadata (
data.attributes.sub_type:human,vehicle,motion, etc.). - Reads the frame for that event. Ring serves live camera media over WebRTC (WHEP):
ops/capture-ring-browser.mjsopens a WHEP session on the Developers Playground device and saves one decoded frame, verified 2026-09-23 (see Where the demo frames come from). Ring also documents a historical image-download endpoint; on the Playground it found no stored image in the last 24 hours (HTTP 416). Event history is readable, but the Playground device logs only live-view (on_demand) events, so there is no motion or doorbell event to drive the webhook path, and the demo scenarios read frames from disk, labelled on screen. - Excises the server-side watermark overlay (top 15% band containing Ring logo, Device ID, and timestamp per the June 8, 2026 Ring API specification) so model vision is not polluted by on-screen text.
- Performs multimodal inference via Amazon Bedrock Nova Pro (
amazon.nova-pro-v1:0inus-east-1) under strict accessibility guardrails (single factual sentence, present tense, zero identity speculation, zero motive guessing). - Enforces loud and visible refusals on unusable inputs (pitch-black unlit frames, corrupted feeds, or model refusal) rather than polite or deceptive fallbacks.
- Presents an honest token failure state when the 30-minute Developers Playground sandbox token is expired or absent, linking directly to the official console (
https://developer.amazon.com/ring/console/playground) with zero fabricated live transcripts. - Speaks the caption aloud in the browser with the Web Speech API.
| Component | Status | Verification Evidence |
|---|---|---|
| Ring Partner API | Verified (authenticated reads, 2026-09-14) | Six endpoints returned HTTP 200 with a real Developers Playground token: /devices, /locations, /users/me, and a device's /capabilities, /status, /configurations (server: envoy, distinct x-request-id per call, e.g. d5791cac-8808-4824-aad8-1cf7486f1682). The sandbox device is a Doorbell Pro reporting online: true. Full output: docs/00-research/ring-live-api-evidence.md. |
| Event history | Verified readable (2026-09-24) | The documented GET /v1/history/devices/{id}/events returned 200 on a Playground token: 7 events, all event_type: on_demand (live-view sessions), each with start and end. The Playground device logs no motion or doorbell events, so there is still no Ring event to drive the webhook path. Our earlier "blocked by scope" row was wrong: it came from the undocumented GET /v1/devices/{id}/events (403; a made-up route returns 404). Probe: ops/probe-ring-events.mjs. |
| Playground media (WHEP) | Verified: a real Ring frame received and described (2026-09-23) | POST /devices/{id}/media/streaming/whep/sessions → 201; H264 (profile 64001f) negotiated; ICE connected; 15 frames decoded; one 1280×720 frame saved; session DELETE → 200. Run through the pipeline, Nova Pro said "A brown package is on the snowy steps." — 6 of 6 runs described it. It is a Playground sandbox device, not a customer camera, and it is labelled that way. Only an offer built by headless Chrome works: the werift client's offer (VP8 plus one H264 profile) got 500. |
| Historical image download | Endpoint exists; no stored image (2026-09-23) | The documented POST .../media/image/download returned 303; its signed download returned 416 — no image in the past 24 hours. Our earlier "no snapshot endpoint" conclusion was wrong: those 404s came from guessed GET paths. |
| Watermark Excision | Verified (Unit & Visual Tests) | Slices top 15% rows (134px on 896p). Unit test tests/watermark-crop.test.ts proves 0% watermark pixels remain in model payload. |
| Bedrock Nova Pro Vision | Verified (Live AWS Inference) | Multimodal inference via @aws-sdk/client-bedrock-runtime against amazon.nova-pro-v1:0 in us-east-1. Generates concise 1-sentence descriptions. |
| Loud Refusal State | Verified (Unit & Live Tests) | Pitch-black frame (< 3.0/255 luminance) triggers explicit REFUSED state, red banner, and refusal speech alert. No silent failures. |
| Token Absence Transparency | Verified (Surface & API Tests) | When token is expired or missing, UI displays warning banner with link to https://developer.amazon.com/ring/console/playground and suppresses transcripts. |
| Fire TV APK Integration | Deferred to Phase 2 | Intentionally isolated to protect frozen Project 1 and Project 3 release artifacts. |
| Cross-origin & network guard | Verified 2026-09-23 — and it was a real hole until then | The service answered every origin with Access-Control-Allow-Origin: * and listened on every interface, so any web page, or anyone on the same network, could drive /api/describe — Bedrock inference billed to the operator's AWS account. tests/origin-guard.test.ts was run against the old code first: a foreign-origin describe returned 200. Now foreign origins get 403 before any route runs, a rebound Host gets 403, and the service binds 127.0.0.1. |
Until 2026-09-23 every description ran on our own fixtures. On the first real Ring frame, Nova Pro refused twice: "No discernible human presence or activity in the frame." The frame shows a parcel on snowy steps, which is exactly what a blind user wants to hear about.
The cause was ours. With no Ring event behind a frame, the service invented
the sensor hint sub_type="human", and the refusal rule let "no person" count
as "nothing to describe". Now no hint is invented when no event exists, a real
sub_type is framed as a hint that may not match the image, and objects,
weather and light are not grounds for refusal.
| Same frame, same model | Described |
|---|---|
Invented sub_type="human" hint (before) |
0 of 2 (both refused) |
| Hint removed | 2 of 3 (one refusal: "too obscured by snow") |
| Hint removed + weather and light are not obstructions | 6 of 6: "A brown package is on the snowy steps." |
The fixture scenarios still behave as documented: both AI-generated frames are
described, and both refusal paths (the procedural frame and the pitch-black
frame) still refuse.
Regression test: services/descriptor/tests/describer-prompt.test.ts.
One frame has come from Ring: the Developers Playground sandbox device, over WHEP, on 2026-09-23. Below is exactly what the model saw: the top 15% band, which carries the Ring logo and device ID, is removed before inference.
Frame from the Ring Developers Playground sandbox stream, which the Playground credits as "Thief stealing our package" by YouTube user frollard, CC BY 4.0. One frame, top 15% removed. This image is licensed CC BY 4.0, not under the repository's MIT licence.
Reproduce it with a fresh Playground token:
powershell -NoProfile -ExecutionPolicy Bypass -File .\ops\with-ring-token.ps1 capture-ring-browser.mjs
(hidden prompt; the token never reaches the browser, a file or the log), then
choose Ring Playground WHEP sandbox capture in the surface.
No frame has come from a customer's camera, and the demo video predates this capture. Every image the pipeline runs on is one of the following, and the web surface labels which one it is on screen, next to the image, on every run.
| File / scenario | Origin | How we know |
|---|---|---|
fixtures/SYNTHETIC-ai-generated-porch-delivery.jpg |
AI-generated image | C2PA content credentials in the file: c2pa.created — "Created by Google Generative AI", digitalSourceType: trainedAlgorithmicMedia, and c2pa.edited — "Applied imperceptible SynthID watermark." |
fixtures/SYNTHETIC-ai-generated-driveway-vehicle.jpg |
AI-generated image | Same C2PA credentials, read from the file's own manifest. |
doorbell_chime_press scenario |
Procedurally drawn | Generated pixel-by-pixel by createSyntheticFrame() in src/sample-frames.ts. |
pitch_black_unusable scenario |
Procedurally drawn | Same function; a deliberately unusable frame for the refusal path. |
ring_playground_whep scenario |
Ring Developers Playground, sandbox device (WHEP) | Written by ops/capture-ring-browser.mjs to the gitignored ops/captures/. Labelled "Ring Playground WHEP sandbox — Not a customer camera." If no capture exists, it falls back to a fixture labelled AI-generated. |
| "Upload Frame" button | Whatever you supply | The app makes no claim about a file you choose yourself. |
The full evidence, including the raw C2PA manifest output, is in
docs/00-research/fixture-media-provenance.md.
One consequence, since it looks like a feature otherwise: only the two
AI-generated frames and the Playground frame produce a real description. Run doorbell_chime_press and
Nova Pro refuses it — "The image is a solid color with no discernible subjects
or actions." Our procedurally drawn frames are flat shapes, and the model is
right to decline them. So the describing path is exercised by generated photographs and one real
Ring frame, and the refusal path by our own drawings.
Reproduce both with node ops/verify-frame-origin.mjs against a running
service.
This is enforced rather than remembered. SampleScenario.frameOrigin carries
the origin with the frame, the API returns it, the UI writes its image label
from it, and services/descriptor/tests/fixture-provenance.test.ts fails the
build if an image lands in fixtures/ without a provenance entry, if a
documented AI-generated file does not say SYNTHETIC in its own filename, or
if any scenario claims ring-live while reading a fixture off disk.
We found this ourselves, after the images had already shipped into the repo labelled "camera test frames" and the surface had already rendered one under a pane reading "Raw Camera Feed". Nobody asked us to check. What that cost, and how the images got there, is written down in the provenance document above.
projects/02-ring-doorstep/
├── LICENSE
├── README.md
├── .env.example
├── apps/
│ └── surface/ # Web testing & judging UI surface (Vite, TypeScript, Hand-crafted CSS)
├── services/
│ └── descriptor/ # Core Node.js/TypeScript event-to-description service
│ ├── fixtures/ # SYNTHETIC AI-generated test frames (see "Where the demo frames come from")
│ ├── src/
│ │ ├── config.ts # Token & AWS configuration
│ │ ├── watermark-cropper.ts # Pure JS top-band watermark excision (15%)
│ │ ├── bedrock-describer.ts # Amazon Bedrock Nova Pro multimodal integration
│ │ ├── ring-client.ts # Real HTTP client for api.amazonvision.com
│ │ ├── event-pipeline.ts # Webhook normalizer & pipeline orchestrator
│ │ ├── sample-frames.ts # Test frames & synthetic scenario generators
│ │ ├── server.ts # Express API & static surface host
│ │ └── index.ts # Service launcher
│ └── tests/ # 17 unit tests (crop, refusal, schema, token, fixture provenance)
├── ops/
│ ├── test.cmd # Runs all test suites & builds (Must be 100% green)
│ ├── verify-ring-api.cmd # Real HTTP round-trip verification to Ring Partner API
│ ├── start.cmd # Launches backend service and web surface on port 3002
│ ├── dev.cmd # Launches service + Vite dev server concurrently
│ └── capture-evidence.mjs # Headless browser automated screenshot & SHA-256 tool
└── docs/
├── 00-research/ # Phase 0 feasibility analysis
├── 04-agents/ # Agent handoff documents & reviews
└── assets/screenshots/ # Verified visual evidence artifacts
Runs all 17 unit tests (watermark crop verification, loud refusal assertions, webhook schema parsing, Ring client 401 handling, fixture provenance guards) and compiles both packages:
ops\test.cmdSends a real HTTP GET to https://api.amazonvision.com/v1/devices and verifies genuine Envoy response headers:
ops\verify-ring-api.cmdLaunches the descriptor API service and hosts the Doorstep Accessibility Studio on http://localhost:3002:
ops\start.cmd| Screenshot | Description | SHA-256 |
|---|---|---|
01-token-missing-banner.png |
Honest token warning banner linking to Developers Playground (https://developer.amazon.com/ring/console/playground) with HTTP 401 badge. |
ebcfb9711e6b4241a0b8efc407797adb031d4f895557cbe67bca55ccebfaa8d9 |
02-porch-package-pipeline.png |
Courier on porch: sub_type: "human", top 15% (134 rows) excised, Bedrock Nova Pro factual description ("A man wearing a blue jacket and blue jeans stands on the front porch holding a cardboard box."), and TTS audio caption. |
6c47742ea6bb804eb40101b19e1a74971a3dc74dc30e4a7ab74c50e1c10e0d2e |
03-vehicle-driveway-pipeline.png |
Vehicle in driveway: sub_type: "vehicle", top watermark sliced off, Bedrock Nova Pro description ("A silver car is parked in the driveway of a house."). |
e40e0185afbea9f30aa68122994fbb060661ba20eac408d13283ec4cbaa16131 |
04-loud-refusal-pitch-black.png |
Unusable pitch-black frame: bright red LOUD REFUSAL TRIGGERED banner, reason ("Frame is pitch black (average luminance 0.0/255)"), no fake transcript, and refusal speech alert. |
86b0490ff833a037c50b4599760f9ad1e8063b66568df0cc08d2c8cd9991b892 |