Repository navigation
feat: announce stored images on NATS, stop overwriting rewritten objects - #18
Merged
Merged
Conversation
…objects Every stored object now goes out as a StoredEvent on NATS_STORED_SUBJECT (default image-stored): path, type, id, source URL, size and content type. upscaler-service consumes it. A failed announcement is logged, not fatal; the image is stored either way. Objects are written with their real content type instead of octet-stream, and with the source's byte length recorded as metadata. The skip rule now compares that recorded length with the source's Content-Length. Comparing the object's own size broke the moment anything rewrote the object: an upscaled poster is bigger than its source, looked changed, and was overwritten with the 225px original on the next event. Objects written before the metadata existed still fall back to their own size. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Contributor
Author
|
🎉 This PR is included in version 1.19.0 🎉 The release is available on:
Your semantic-release bot 📦🚀 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
image-storedannouncements. After an object is written, aStoredEvent(path,type,id,source_url,size,content_type) is published onNATS_STORED_SUBJECT, defaultimage-stored, over the same NATS connection the consumer already holds.upscaler-serviceconsumes it to upscale what needs it. A failed publish is a warning, never a failed message: the image is stored regardless, and the consumer may not be deployed yet.application/octet-stream; they now carry whathttp.DetectContentTypefinds (image/jpeg,image/png,image/webp).source-length, plussource-url), andunchangedcompares that with the source'sContent-Length. The old rule compared the object's own size, which meant any rewrite of the object looked like new artwork: an upscaled poster is bigger than its source and was overwritten with the 225px original on the next event. Objects written before the metadata existed fall back to their own size, so untouched downloads are still skipped.StoragegainsHead(size plus metadata) andPutObject(content type plus metadata);Put/Statstay.NewImageProcessortakes the publisher; the Kafka handler passes nil.Checked
New processor tests with an in-memory store and an
httptestsource: content type and metadata on store; the overwrite regression (object rewritten bigger, same source length, second event does not re-download or re-announce); a genuinely longer source is fetched again; a legacy object with no metadata and the same size is skipped; a failing announcement does not fail the store.Deploy
Needs a
image-storedstream (plus-retry/-dlq) in weeb-argocd's nats-streams, coming in the upscaler-service deploy PR. Until the stream exists each stored image logs one warning.🤖 Generated with Claude Code