A second architecture answers a point prompt - #599
Open
JArmandoAnaya wants to merge 7 commits into
Open
Conversation
The resolver picks an adapter from the family a model's own config declares, so registering a second point-prompt architecture is an entry in the segmenter family register rather than a new adapter. SAM 3 declares `sam3` when a repository publishes the whole model and `sam3_tracker` when it publishes only the promptable half; both are served here, and the capability each one grants is derived from the register as before. What could not be derived is the pair of `transformers` classes a family loads through, so `sam_provider` gains a table. SAM 2 keeps `AutoProcessor`, which resolves correctly for the repositories it is pointed at. SAM 3 cannot: a repository publishing the whole model declares the concept processor, which takes text and has no `input_points` argument, so resolving there would hand the adapter something no click can be expressed to and the failure would land inside a call rather than in a refusal. Everything below the loader is untouched, because the tracker agrees with SAM 2 signature for signature on the encode, the forward and the mask post-processing. Fetching weights gains one behaviour. A repository whose terms have not been accepted was reported by re-raising the hub client's own text, which opens with a status line and a request id; it now says what happened and what to do, and the transport stays out of it. The case is told from a mistyped model id by the exception's class and never by its status code, because both answer 401 and only the class separates them. cf. #595
… be accepted Adding SAM 3 to the point-prompt ladder puts an entry in the list that nobody can simply download, which is a thing the list had not had to say before. `CuratedModel` gains an optional `access` pair — one sentence and the page where it is cleared — and the form states it while the model is being chosen rather than leaving it to the refusal a download would eventually produce. A person who has not been granted access now reads that before creating anything. The entry's size is the hub's figure for the pinned revision, as every entry's is, and it is roughly twice the model it installs: the repository publishes its weights as both a checkpoint and safetensors, and fetching a revision fetches all of it. What lands on the disk is the number that belongs beside a download. The module's own note said both groups were Apache-2.0, which this makes false, so it now states the exception and why the entry is offered anyway. Leaving a gated model out of a curated list does not spare anybody its terms; it means the people who want it type the id in from somewhere else, having read nothing. The annotator's not-capable blocker named the SAM 2 family as what a click needs. It now names what the model has to do, because more than one architecture does it and prose repeating a register is the copy that goes stale without any build failing. cf. #595
…revision The access line resolved its entry through `curatedEntry`, which compares the revision as well as the model id. That is the right question for the select, where it answers "is this row showing exactly this curated entry", and the wrong one here: pinning a different commit of a gated model does not exempt anybody from its terms, so the line vanished while the requirement still applied. Found by mutation rather than by reading. Replacing the lookup with one keyed on the model id alone left every test green, which said the distinction was unverified — and looking at which of the two behaviours was correct showed the mutation was the fix. The test that now pins it drives the edit dialog of a stored connection naming the gated model at another commit. cf. #595
…ching The curation rules excluded custom-gated licenses alongside copyleft ones, and a section recorded `facebook/sam3` as blocked on two counts. Both are now decided the other way and the model ships, so the rules state the position that replaced them rather than leaving a reader to infer it from the catalog disagreeing with the prose. The reference documentation gains the two steps a gated model needs — request access, set HF_TOKEN — and says where the gate is not: reading a size needs neither, which is what lets the form price a download before anybody asks for one, and running weights already in a workspace needs neither, which is what keeps a copied workspace working. Two measurements are written down because both contradict the assumption a reader would otherwise make: the hub sizes a gated repository unauthenticated, and a gated repository answers 401 rather than 403 — the same status a repository that does not exist answers, so the two are told apart by the client's exception class and never by the code. cf. #595
…ves in Measured against the wheels rather than inferred: 4.57.6 ships neither the `sam3` nor the `sam3_tracker` module and 5.0.0 ships both, exporting `Sam3TrackerProcessor` and `Sam3TrackerModel` at the top level, which is what the adapter names. 5.0.0 is also where naming the processor rather than resolving it stops being a preference and becomes the only thing that works: `sam3_tracker` is absent from the auto processor map in that release, so `AutoProcessor` has no answer for it at all. It gained one later. The adapter's table works on both. The bump is a major one for everybody carrying the extra, including the SAM 2 and Grounding DINO users who gain nothing from it. What it mostly does is make the floor honest: `uv.lock` has resolved 5.x for as long as the extra has existed, so `>=4.44` was a compatibility claim no run has ever tested. cf. #595
… is reachable The api service passed no token, so weights that have to be asked for could not be fetched from the dev stack at all. The failure mode is the bad one: the refusal names `HF_TOKEN`, the reader sets it on the host, the container never sees it, and the identical refusal comes back with nothing about it changed. A refusal naming a remedy nobody in that position can take is the mistake `NotAWorkspace` is the standing example of. One forwarded variable, empty when the host has none, which is what every ungated model already expects. Verified through `docker compose config` in both directions: absent it renders empty, set it renders the value. cf. #595
…ctually opens The conditional is already held in jsdom. What that layer cannot say is anything about the journey: that the sentence appears in the real dialog, above the real size line, before the control that would fetch anything is reachable. This opens Inference in chromium, picks the gated entry and reads the line and its link. cf. #595
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
facebook/sam3becomes a second driver of the interactive point-segmentation capability thatalready exists, alongside SAM 2. It is not a new capability:
ModelCapability.POINT_SUGGESTisunchanged, the dashboard gains no section, the annotator gains no control, and a connection
pointed at SAM 3 becomes a candidate for the suggest tool on the same terms as a SAM 2 one. The
scope and the licence treatment are the decision recorded on the SAM 3 provider issue (#595).
openapi.json, the generated client andVERSIONare byte-identical. No migration, no new kernelservice, model, error or event, nothing added to
ERROR_RULES, and no CI job added or renamed —so the
mainruleset needs no edit.The task's premise was wrong in one load-bearing way
The work was specified against
Sam3ModelandSam3Processor. Those are the text-and-conceptpath, which this issue puts out of scope. Point prompts run through
Sam3TrackerModelandSam3TrackerProcessor, loaded from the same repository with no subfolder. The library's ownauto-maps are unambiguous:
The consequence is that
AutoProcessorcannot be used here. A repository publishing the wholemodel declares the concept processor, which takes text and has no
input_pointsargument at all,so resolving there would hand the adapter something no click can be expressed to — and the failure
would land inside a call rather than in a refusal.
Why the change is small
Resolution is family-driven:
SEGMENTER_FAMILIESis a set ofmodel_typestrings read from amodel's own downloaded config, and
CAPABILITY_BY_FAMILYis derived from it, so an adapter and itscapability declaration are the same edit. And the interface has no model-centric surface — the unit
is a connection, the dashboard groups by the wire's
capabilities, and the annotator's pickerselects a connection filtered on
point_suggest, remembering the choice per project already.So there is no new adapter class, no new capability, no registration table and no new component.
providers.pyneeded one argument threaded through and nothing else.The tracker is a straight port of SAM 2's image path, compared in the installed source rather than
taken from documentation. Every signature the adapter depends on is identical:
get_image_embeddings(pixel_values, **kwargs), the fullforwardkeyword list,post_process_masks(masks, original_sizes, …), a processor__call__that acceptsoriginal_sizeswithoutimages, and outputs carryingpred_masksandiou_scores. That is whythe embedding cache, the decode path, the fp16 guard and the mask-to-shape pipeline are untouched
and only two class names vary.
sam_provider._CLASSESholds them, one row per family, keyed by exactly the members ofSEGMENTER_FAMILIESand held to it by a test. SAM 2 keepsAutoProcessoron purpose: thecheckpoint the form suggests declares
processor_class: Sam2VideoProcessor, so that call alreadyreturns the video processor and that is shipped behaviour, not an oversight to tidy.
The one behavioural fix
Fetching weights for a repository whose terms have not been accepted re-raised the hub client's own
text, which opens with a status line and a request id:
That reached a person verbatim on a failed job. It is now two sentences — what happened, and what
to do — and the translation lives in one helper both
downloadandmeasureroute through ratherthan at the one call site that prompted it.
Two details decide how it is written, and both were measured:
401, not the403the word suggests — and so does a repositorythat does not exist. So the case is told by the client's exception class and never by the
status code; a branch on the number sends somebody with a typo'd model id to request access to a
model nobody publishes. There is a test for exactly that discrimination.
GatedRepoErrorinherits fromRepositoryNotFoundError, so a handler naming the parentfirst silently swallows the gated case — the ordering trap the media adapter records for
UnidentifiedImageErrorunderOSErrorand the persistence adapter forIntegrityErrorunderDatabaseError.No new error class:
LocalInferenceUnavailablealready carriesexpose_message, so the sentencereaches a person unchanged and nothing moves on the wire.
Measurements that changed the design
model_infofor a gated repositoryunauthenticated, so the form prices a download before anybody decides whether to go and ask for
it. An earlier draft of this work assumed the opposite.
twice, as a checkpoint and as safetensors, and fetching a revision fetches all of it. The curated
entry carries the figure that describes the disk.
transformersfloor moves>=4.44to>=5.0, probed against the wheels: 4.57.6 shipsneither the
sam3nor thesam3_trackermodule, 5.0.0 ships both. 5.0.0 is also where naming theprocessor stops being a preference —
sam3_trackeris absent from the auto processor map in thatrelease and gained an entry later. The bump is a major one for everybody carrying the extra,
including SAM 2 and Grounding DINO users who gain nothing from it; what it mostly does is make the
floor honest, since
uv.lockhas resolved 5.x for as long as the extra has existed.A defect found by mutation rather than by reading
The access line resolved its entry through
curatedEntry, which compares the revision as well asthe model id. Replacing that with a lookup keyed on the model id alone left every test green, which
said the distinction was unverified — and asking which of the two behaviours was correct showed the
mutation was the fix. An access gate belongs to the repository: pinning a different commit of a
gated model does not exempt anybody from its terms, so the line vanishing when the revision was
edited was hiding a requirement that still applied. A test now drives the edit dialog of a stored
connection naming the gated model at another commit.
The dev stack could not have taken the remedy
docker/compose.yaml's api service passed no token, so the failure mode was the bad one: therefusal names
HF_TOKEN, the reader sets it on the host, the container never sees it, and the samerefusal comes back unchanged. A refusal naming a remedy nobody in that position can take is the
mistake
NotAWorkspaceis the standing example of. One forwarded variable, empty when the host hasnone, verified through
docker compose configin both directions.Documentation that had recorded the opposite
docs/architecture/backend/inference.mdcarried a curation rule excluding custom-gated licencesalongside copyleft ones, and a section recording this model as blocked on two counts. Both are now
decided the other way, so the rules state the position that replaced them rather than leaving a
reader to find the catalog disagreeing with the prose. The new rule is that a gated model may be
curated and may never be the default, and there is a test holding the default to a model anybody
can fetch.
Found, not fixed
annotate.spec.tsscenarios fail under the local gate's ten workers — the trackedlocal-only flake, cf. Two annotate.spec.ts scenarios fail only under the local gate's ten workers #550. Evidence below.
frontend/app/cycle/cycle.spec.tsneverexercises auto-labeling, so "a suggestion works end to end against a real server" is not gated
anywhere. Out of scope here, worth its own issue.
_device.resolvedcovers a device being absent, not anallocation failing mid-forward. SAM 3 is roughly ten times the parameters of the balanced SAM 2
rung, which makes an out-of-memory during a forward pass reachable in a way it was not before.
Nothing here changes it and no refusal names it.
polarsandpolars-runtime-32are yanked inuv.lock. Pre-existing, surfaced as a warningby any
uv lock; untouched here.Test plan
New coverage: the family register's two spellings and the exclusion of nested config halves; the
class table's agreement with
SEGMENTER_FAMILIESand the per-family classes; that no load reachesthe network; the gated refusal from both
downloadandmeasure; that a repository which is notthere is not reported as a licence to accept; that a client too old to name the error still
reports the failure; the catalog's access note and its absence from the default; the access line
appearing while a model is chosen, disappearing when the choice moves back, and surviving a
different pinned revision; and one chromium scenario reading the line in the real dialog.
Twelve mutations were applied and reverted by recorded patch, each asserted present-once before and
present after, with a clean-tree check between cases. Every one turned a named test red. The
harness itself was rewritten partway through: a zsh version using heredocs inside a function
silently failed to apply its mutations while reporting the replacement absent, which is the
false-calm failure the protocol warns about, so it was rebuilt in Python where there is no shell
quoting to get wrong.
The gate was run in stages against the ~10-minute ceiling. Exit lines verbatim:
Frontend suites:
frontend/annotator 1039 passed (1039),frontend/ui-core 1013 passed (1013).Browser stage, this branch:
and the real-server cycle run beside it:
1 passed (44.7s).The two failures were red at the merge-base
Reproduced by me, in this environment, on unmodified
mainat the merge-base9bf9266, with thecommand #550 gives. It is a flake, so it takes repetition to show — three runs:
The same file on this branch, same command:
1 failed,1 flaky,90 passed (1.6m)— the sametwo scenario names.
The diff does not touch what those scenarios exercise. They walk asset navigation and a canvas
scroll position; the only annotator file in this diff is
SuggestPanel.tsx, and the change there isone sentence of copy in the blocker shown when no connection can answer a click.
annotate.spec.tsitself is untouched. Worth knowing for anybody reading a red gate on their own branch: adding any
scenario anywhere reshuffles what runs beside what across the ten workers, so a branch touching
nothing in that file can still flip it — two consecutive full browser runs on this branch were green
before a scenario was added to
inference.spec.ts, a different file. A fresh reproduction isrecorded on #550.
What is not verified
No forward pass was ever run. Two independent blockers, both stated rather than worked around:
the repository is gated and no token was available in this session, and this machine cannot install
the runtime at all —
torchpublishes no macOS x86_64 wheel at the locked version, souv sync --extra local-inferencefails outright.The consequence is one open question: what
facebook/sam3'sconfig.jsondeclares as itsmodel_type. The family is deliberately read from a model's own config rather than guessed fromits name, so rather than assume, both registered spellings are served —
sam3for a repositorypublishing the whole model and
sam3_trackerfor one publishing only the promptable half. The codeis correct either way, and the reading is still owed. #595 stays open until somebody with access has
confirmed it and run a click end to end.