Skip to content

feat(10): Homework vision - #52

Open
tpaulshippy wants to merge 8 commits into
feature/roadmap-09-profile-accessfrom
feature/roadmap-10-homework-vision-voice
Open

tpaulshippy wants to merge 8 commits into
feature/roadmap-09-profile-accessfrom
feature/roadmap-10-homework-vision-voice

Conversation

@tpaulshippy

@tpaulshippy tpaulshippy commented Aug 25, 2026

Copy link
Copy Markdown
Owner

Roadmap 10: Homework Vision

Implements #62 (voice portion split to #64).

What changed

  • Backend: Image processing for worksheet capture, KaTeX math rendering pipeline, homework-mode prompt injection, vision model auto-upgrade
  • Frontend: Camera integration for worksheet photos, KaTeX rendering in chat
  • Capstone feature: photograph a worksheet and talk to a tutor that renders math
  • Fixes since review: empty-string text child no longer trips the RN-web View check on send; web notification crash guarded

Evidence

Worksheet photographed in chat and sent as an image message:

Worksheet sent as image message

Camera attach affordance

pr52-vision-voice.mp4

@tpaulshippy tpaulshippy changed the title feat(10): Homework vision and voice feat(10): Homework vision Aug 26, 2026
@tpaulshippy
tpaulshippy force-pushed the feature/roadmap-10-homework-vision-voice branch from c249d9a to 28bd63b Compare September 3, 2026 14:37
@tpaulshippy
tpaulshippy changed the base branch from main to feature/roadmap-09-profile-access September 3, 2026 14:37
@tpaulshippy
tpaulshippy force-pushed the feature/roadmap-09-profile-access branch from 6613dfd to 6b0f3d4 Compare September 3, 2026 15:15
@tpaulshippy
tpaulshippy force-pushed the feature/roadmap-10-homework-vision-voice branch from 28bd63b to 7edea66 Compare September 3, 2026 15:15
@tpaulshippy
tpaulshippy force-pushed the feature/roadmap-09-profile-access branch from 0604e42 to 80ff7d7 Compare September 3, 2026 15:33
@tpaulshippy
tpaulshippy force-pushed the feature/roadmap-10-homework-vision-voice branch from 86fcba2 to cbf0321 Compare September 3, 2026 15:33
@tpaulshippy
tpaulshippy force-pushed the feature/roadmap-09-profile-access branch from 80ff7d7 to 3654b64 Compare September 3, 2026 15:45
tpaulshippy and others added 7 commits September 3, 2026 15:46
…ion evidence

{message.text && ...} evaluates to '' for the loading bubble, and
react-native-web warns on string children of View (dev overlay on
every send). Use ternary. Also guard getLastNotificationResponse on
web.

Evidence: replace duplicated placeholder with worksheet photo captured
and sent as an image message, plus attach affordance.
@tpaulshippy
tpaulshippy force-pushed the feature/roadmap-10-homework-vision-voice branch from cbf0321 to f1b70a8 Compare September 3, 2026 15:46
@tpaulshippy

Copy link
Copy Markdown
Owner Author

Automated review — backwards-compat + tests

Backwards-compat: SAFE. Backend-only diff (+51/-0): bot.enable_vision default False, message.intent default 'chat', message.meta default dict. No serializer/URL changes, so old phones neither send nor see the new fields; stored defaults keep old rows valid.
Tests: NONE in this diff. No test files; PR body leans on evidence screenshots/video and the camera/KaTeX frontend described but not present in this diff.

MUST FIX

  1. enable_vision is write-only — _upgrade_to_vision_model (chat.py) upgrades any image message regardless of the flag. Either gate on bot.enable_vision or remove the flag; as-is the per-bot vision switch in the product story does nothing.

SHOULD FIX

  1. Vision-model picker AiModel.objects.filter(supported_input_modalities__contains=['image']) (JSONField default=list): no ordering guarantee on .first() and exclude(model_id=current) can bounce between two vision models across turns. Add explicit ordering and pin per-chat once chosen.
  2. Silent fallback — no vision model → use_default_model with only a log warning; image gets sent to a text-only model path. Surface a user-facing error or disable image attach for that bot.

NICE TO HAVE

  1. message.meta/intent have no serializer exposure in this diff — fine for compat, but confirm the follow-up exposes/validates them rather than letting clients free-write meta.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant