diff --git a/CLAUDE.md b/CLAUDE.md index 00a44ac..6ac28ec 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -203,7 +203,7 @@ No composition file to update — each class self-instantiates its deps. `main.t ### 1. Middleware Pipeline (server.ts) ```typescript -app.use(bodyParser.json()); // Parse JSON +app.use(jsonBody); // Parse JSON, 100 KB (POST /v1/library/status parses its own 5 MB body) app.use(compress()); // Gzip compression app.use(helmet()); // Security headers app.use(authMiddleware); // JWT validation → sets req.user @@ -384,8 +384,9 @@ All routes require auth + an active subscription (`checkSubscription`); most als | POST / GET | `/upload/parts` | Presign part URLs (≤32 per request) / list the parts S3 holds, to resume | | POST | `/upload/complete` | Complete from S3's own part list and set `synced=true` (and `downloaded` on a media-server book's external resources) — the only confirmation a multipart upload gets; safe to retry | | POST | `/upload/abort` | Abort; succeeds when the upload or row is already gone | -| GET | `/keys` | Synced identifiers | +| GET | `/keys` | Synced keys (**deprecated**: still served for shipped builds' first-sync / tier-change pass; new clients use `/status`) | | POST | `/uuids` | Match client uuids to rows | +| POST | `/status` | The missing-items pass: of the client's uuids (the whole library, one body, parsed by the route with a 5 MB limit after `checkSubscription` and `requireCloudData`; `jsonBody` keeps every other route at 100 KB), which no row has, active or deleted (`unknown`: register them), and which are active books with no file in S3 (`unsynced`: upload them by uuid). Contract in `docs/multipart-uploads.md` | ### Storage Routes (`/v1/storage`) @@ -502,11 +503,11 @@ async DoSomething(): Promise { } ``` -**Exception — reads whose empty result clients treat as authoritative.** `LibraryService.getLibrary` -and `getLastItemPlayed` throw `LibraryLookupError` when a DB read fails instead of returning `null` -or `[]`: sync clients reconcile deletions (items, server links) against a listing, so a failed read -must never look like an empty library. DB classes still return `null` on error; the service turns -that `null` into the throw. `GET /` and `GET /last_played` map it to a 500 the clients retry — with +**Exception — reads whose empty result clients treat as authoritative.** `LibraryService.getLibrary`, +`getLastItemPlayed` and `getItemsStatus` throw `LibraryLookupError` when a DB read fails instead of returning `null` +or `[]`: sync clients reconcile deletions (items, server links) against a listing, and register whatever the status +read calls unknown, so a failed read must never look like an empty library. DB classes still return `null` on error; the service turns +that `null` into the throw. `GET /`, `GET /last_played` and `POST /status` map it to a 500 the clients retry — with one deliberate exception: on the root listing the resume item is best-effort, so the controller logs a `getLastItemPlayed` failure and omits the `lastItemPlayed` key (absent = unavailable this time, `null` = nothing played yet) rather than failing a listing that has already succeeded. Use the same diff --git a/docs/multipart-uploads.md b/docs/multipart-uploads.md index fec682f..2e61da1 100644 --- a/docs/multipart-uploads.md +++ b/docs/multipart-uploads.md @@ -22,9 +22,12 @@ when S3 rejected the PUT, leaving rows that claim to be backed up with nothing i book is left alone — Hardcover has no file, and its `sync_status` is the client's own marker. - **`synced` means the file is in S3, on every tier.** `PUT /` creates rows unsynced, and `POST /` ignores `synced:true` for a book with no object. So `GET /keys`, which lists synced rows, is "books whose file is in S3": - a LITE account's books are left out of it on purpose, because LITE never uploads a file. Clients use `/keys` only - for the one-off "upload what the server is missing" pass (iOS: an install's first sync; Android: a tier change), - and should run that pass only on PRO. + a LITE account's books are left out of it on purpose, because LITE never uploads a file. **`/keys` is + deprecated**: shipped builds compare their local paths against it in a one-off "upload what the server is missing" + pass (iOS: an install's first sync; Android: a tier change), and it stays served for them. A path that is stale on + that device (the item was moved or renamed on another one) reads as missing, and re-uploading it there moves the + item back: `PUT /` treats a known uuid at a new key as a move. New clients use the + [missing-items pass](#the-missing-items-pass), which asks by uuid. - **One open upload per book.** `start` aborts whatever is still open for the book before opening a new one. Uploading the same book from two devices at once is unsupported: each device's `start` would cancel the other's upload. That is deliberate — a book's file is uploaded by the device it was imported on, and every other device downloads it. @@ -73,3 +76,43 @@ and a 400 `RequestTimeout` or any 5xx means retry the part. `complete` is safe to repeat. If the upload is gone but the object exists (a previous attempt succeeded and its response was lost), it answers success and makes sure the row is synced. `start` does the same: if the object already exists, it answers `exists` instead of opening a second upload. + +## The missing-items pass + +`POST /v1/library/status` answers, for the uuids in a client's local library, what the server lacks. It exists for an +account that signs in over a library built while signed out, whose sync lapsed and came back, or that moved from LITE +to PRO: items added while sync was off (and uploads the lapse cleared from the queue) never reached the server, and +books registered on LITE have no file. +Open to PRO and LITE. + +| Body | Success | +|---|---| +| `{ uuids: [uuid…] }`: every item in the local library (books, folders, bound books), in one request | `{ unknown: [uuid…], unsynced: [uuid…] }` | + +- `unknown`: no row has the uuid, active or deleted. First send those items through `POST /uuids` + (`{ items: { "": "" } }`, at most 1,000 per request): + the server's row may already sit at that key under no uuid (a legacy row) or another one (the same file imported + on two devices), and `PUT /` at an occupied key answers with that row without storing the client's uuid, so the + item would come back `unknown` every run and its bookmarks and external resources would answer `item_not_found`. + `/uuids` sets the uuid on a legacy row and answers a conflict for the other case (adopt the server's uuid). Then + register each one like an import (`PUT /` at its local path, then its external resources and bookmarks), parents + before children. No row holds the uuid, so the `PUT` can't move anything, and a deleted item keeps its uuid, so a + book deleted on another device isn't registered again. On PRO the `PUT`'s answer then asks for the file as usual. + The one exception (accepted): an item from before the server had uuids (March 2026) that another device deleted + under its own uuid, or none, before this device's uuid was matched, reads as `unknown` and comes back. `/uuids` + only looks at active rows, so nothing tells it apart from a book imported on this device. +- `unsynced`: an active book with no file in S3. PRO only: upload its file through `/upload/*`, which finds the book by + uuid wherever it now lives. Never re-register these: a `PUT /` at a stale path would move them. Skip books streamed + from a media server (their file arrives when they're downloaded), books with no local file, books over 10 GiB, and + books with an upload already queued. LITE ignores this list: LITE never uploads, so every book it registered is + on it. +- Uuids come back spelled as they were sent, once each; strings that aren't uuids are left out of both lists. A + failed read is a 500, never an empty answer, which would read as "register everything". +- The body is the whole library, so this route's JSON limit is 5 MB (about 130k uuids), parsed only once the caller + is known to be on PRO or LITE; the rest of the API keeps 100 KB. Nothing is capped per request. +- Run it as the registration step of the first sync (after sign-in, and on coming back from a lapse, which a + client treats as a first sync), on LITE → PRO, and weekly, and only when nothing is waiting in the client's own + sync queue: a queued import would otherwise come back `unknown` and be registered twice. +- Until that first sync has run, no listing may delete local items it doesn't show (the first sync's own listing + included): they may be exactly the items the pass is about to register, such as books imported while sync was off. + diff --git a/src/__tests__/controllers/LibraryController.test.ts b/src/__tests__/controllers/LibraryController.test.ts index d900700..c882b24 100644 --- a/src/__tests__/controllers/LibraryController.test.ts +++ b/src/__tests__/controllers/LibraryController.test.ts @@ -372,3 +372,46 @@ describe('LibraryController — legacy routes naming a missing item', () => { expect(res400.json).toHaveBeenCalledWith({ message: 'The destination is invalid' }); }); }); + +describe('LibraryController.postLibraryStatus', () => { + let libraryService: any; + let controller: LibraryController; + const uuids = ['2c2d0f44-1111-4111-8111-111111111111', '2c2d0f44-2222-4222-8222-222222222222']; + + beforeEach(() => { + libraryService = { getItemsStatus: jest.fn() }; + controller = new LibraryController(libraryService, {} as any); + (controller as any)._logger = mockLoggerService; + mockLoggerService.log.mockClear(); + }); + + const request = () => + ({ body: { uuids }, user: { id_user: 1, email: 'user@example.com' } }) as any; + + it("answers the service's lists as they are", async () => { + libraryService.getItemsStatus.mockResolvedValue({ unknown: [uuids[0]], unsynced: [uuids[1]] }); + const res = makeRes(); + + await controller.postLibraryStatus(request(), res); + + expect(libraryService.getItemsStatus).toHaveBeenCalledWith(expect.objectContaining({ id_user: 1 }), uuids); + expect(res.json).toHaveBeenCalledWith({ unknown: [uuids[0]], unsynced: [uuids[1]] }); + }); + + it('answers 500 on a failed read, logging a count rather than the library', async () => { + libraryService.getItemsStatus.mockRejectedValue(new LibraryLookupError()); + const res = makeRes(); + + await controller.postLibraryStatus(request(), res); + + expect(res.status).toHaveBeenCalledWith(500); + expect(res.json).toHaveBeenCalledWith({ message: 'Internal error' }); + expect(mockLoggerService.log).toHaveBeenCalledWith( + expect.objectContaining({ data: { user_id: 1, count: 2 } }), + 'error', + ); + const logged = JSON.stringify(mockLoggerService.log.mock.calls); + expect(logged).not.toContain(uuids[0]); + expect(logged).not.toContain('user@example.com'); + }); +}); diff --git a/src/__tests__/database/libraryItemsActive.test.ts b/src/__tests__/database/libraryItemsActive.test.ts new file mode 100644 index 0000000..4b98871 --- /dev/null +++ b/src/__tests__/database/libraryItemsActive.test.ts @@ -0,0 +1,43 @@ +import { describe, it, expect } from '@jest/globals'; +import { getTestTransaction, createTestUser } from '../setup'; + +// Migration 20260929120000: every read filters on `active = true`, so a NULL +// `active` was an invisible third state. It's now NOT NULL, defaulting to true. +describe('library_items.active', () => { + const baseRow = (user_id: number, key: string) => ({ + user_id, + key, + title: key, + original_filename: key, + speed: 1, + actual_time: '0', + details: key, + duration: '0', + percent_completed: 0, + order_rank: 0, + type: 2, + is_finish: false, + synced: false, + }); + + it('defaults to true', async () => { + const trx = getTestTransaction(); + const user = await createTestUser(trx, { email: 'active-default@example.com' }); + + const [row] = await trx('library_items').insert(baseRow(user.id_user, 'Default.m4b')).returning('active'); + + expect(row.active).toBe(true); + }); + + it('rejects NULL', async () => { + const trx = getTestTransaction(); + const user = await createTestUser(trx, { email: 'active-null@example.com' }); + + // A savepoint, so the failed insert doesn't abort the test's transaction + await expect( + trx.transaction((savepoint) => + savepoint('library_items').insert({ ...baseRow(user.id_user, 'Null.m4b'), active: null }), + ), + ).rejects.toThrow(/null value in column "active"/); + }); +}); diff --git a/src/__tests__/middlewares/jsonBody.test.ts b/src/__tests__/middlewares/jsonBody.test.ts new file mode 100644 index 0000000..ddeba64 --- /dev/null +++ b/src/__tests__/middlewares/jsonBody.test.ts @@ -0,0 +1,49 @@ +import { describe, it, expect } from '@jest/globals'; +import express from 'express'; +import request from 'supertest'; +import { jsonBody, largeJsonBody } from '../../api/middlewares/jsonBody'; + +// server.ts parses JSON with body-parser's 100 KB limit everywhere except the routes that +// parse their own larger body after checking the caller (POST /v1/library/status). +function makeApp() { + const app = express(); + app.use(jsonBody); + app.post('/v1/library/status', largeJsonBody, (req, res) => res.json({ count: req.body.uuids.length })); + app.post('/v1/user/login', (req, res) => res.json({ keys: Object.keys(req.body) })); + app.use((err: { status?: number }, _req: express.Request, res: express.Response, _next: express.NextFunction) => { + res.status(err.status ?? 500).json({}); + }); + return app; +} + +// ~1 MB: over the 100 KB default, well under 5 MB +const library = { uuids: Array.from({ length: 27_000 }, (_, i) => `2c2d0f44-1111-4111-8111-${`${i}`.padStart(12, '0')}`) }; + +describe('jsonBody', () => { + it('lets the missing-items route parse a whole library', async () => { + const res = await request(makeApp()).post('/v1/library/status').send(library); + + expect(res.status).toBe(200); + expect(res.body).toEqual({ count: 27_000 }); + }); + + it('matches the route the way Express does: any case, trailing slash ignored', async () => { + const res = await request(makeApp()).post('/V1/Library/Status/').send(library); + + expect(res.status).toBe(200); + expect(res.body).toEqual({ count: 27_000 }); + }); + + it('keeps every other route at 100 KB', async () => { + const res = await request(makeApp()).post('/v1/user/login').send(library); + + expect(res.status).toBe(413); + }); + + it('still parses small bodies everywhere else', async () => { + const res = await request(makeApp()).post('/v1/user/login').send({ token_id: 'abc' }); + + expect(res.status).toBe(200); + expect(res.body).toEqual({ keys: ['token_id'] }); + }); +}); diff --git a/src/__tests__/services/LibraryDBItemsByUuids.test.ts b/src/__tests__/services/LibraryDBItemsByUuids.test.ts new file mode 100644 index 0000000..907056f --- /dev/null +++ b/src/__tests__/services/LibraryDBItemsByUuids.test.ts @@ -0,0 +1,38 @@ +import { describe, it, expect, beforeEach } from '@jest/globals'; +import { LibraryDB } from '../../services/db/LibraryDB'; +import { + getTestTransaction, + mockLoggerService, + createTestUser, + createTestLibraryItem, +} from '../setup'; + +// The uuids become one Postgres array literal cast to uuid[]: a single string that +// isn't a uuid would fail the cast, and with it the whole library's status. +describe('LibraryDB.getItemsByUuids', () => { + let db: LibraryDB; + + beforeEach(() => { + db = new LibraryDB(); + (db as any).db = getTestTransaction(); + (db as any)._logger = mockLoggerService; + mockLoggerService.log.mockClear(); + }); + + it('skips strings that are not uuids instead of failing the query', async () => { + const trx = getTestTransaction(); + const user = await createTestUser(trx, { email: 'items-by-uuids@example.com' }); + const book = await createTestLibraryItem(trx, { user_id: user.id_user, key: 'Book.m4b' }); + + const rows = await db.getItemsByUuids(user.id_user, ['Optional("x")', '', 'a,b}', book.uuid]); + + expect(rows?.map((row) => row.uuid)).toEqual([book.uuid]); + }); + + it('answers an empty list, not a failure, when nothing is a uuid', async () => { + const trx = getTestTransaction(); + const user = await createTestUser(trx, { email: 'items-by-uuids-none@example.com' }); + + await expect(db.getItemsByUuids(user.id_user, ['x', ''])).resolves.toEqual([]); + }); +}); diff --git a/src/__tests__/services/LibraryServiceItemsStatus.test.ts b/src/__tests__/services/LibraryServiceItemsStatus.test.ts new file mode 100644 index 0000000..c2fe54e --- /dev/null +++ b/src/__tests__/services/LibraryServiceItemsStatus.test.ts @@ -0,0 +1,209 @@ +import { describe, it, expect, beforeEach, jest } from '@jest/globals'; +import { randomUUID } from 'crypto'; +import { LibraryService, LibraryLookupError } from '../../services/LibraryService'; +import { + getTestTransaction, + mockLoggerService, + createTestUser, + createTestLibraryItem, +} from '../setup'; + +/** + * POST /v1/library/status, the clients' missing-items pass: of the uuids in a + * local library, `unknown` are the ones the server has no row for (the client + * registers them: nothing on the server holds the uuid, so the PUT can't move + * anything) and `unsynced` are active books with no file in S3 (a PRO client + * uploads them by uuid, never by path). + */ +describe('LibraryService.getItemsStatus', () => { + let service: LibraryService; + + beforeEach(() => { + service = new LibraryService(); + (service as any).db = getTestTransaction(); + (service as any)._libraryDB.db = getTestTransaction(); + (service as any)._libraryDB._logger = mockLoggerService; + (service as any)._logger = mockLoggerService; + mockLoggerService.log.mockClear(); + }); + + const user = async (email = 'status@example.com') => { + const row = await createTestUser(getTestTransaction(), { email }); + return { ...row, subscriptions: [] } as any; + }; + + it('sorts a library into unknown, unsynced, and neither', async () => { + const trx = getTestTransaction(); + const owner = await user(); + const synced = await createTestLibraryItem(trx, { user_id: owner.id_user, key: 'Synced.m4b' }); + const noFile = await createTestLibraryItem(trx, { + user_id: owner.id_user, + key: 'NoFile.m4b', + synced: false, + }); + const folder = await createTestLibraryItem(trx, { + user_id: owner.id_user, + key: 'Folder', + type: 0, + synced: false, + }); + const bound = await createTestLibraryItem(trx, { + user_id: owner.id_user, + key: 'Bound', + type: 1, + synced: false, + }); + const neverSeen = randomUUID(); + + const status = await service.getItemsStatus(owner, [ + synced.uuid, + noFile.uuid, + folder.uuid, + bound.uuid, + neverSeen, + ]); + + expect(status).toEqual({ unknown: [neverSeen], unsynced: [noFile.uuid] }); + }); + + it('counts a deleted item as seen, so a book deleted elsewhere is never registered again', async () => { + const trx = getTestTransaction(); + const owner = await user(); + const deletedBook = await createTestLibraryItem(trx, { + user_id: owner.id_user, + key: 'Deleted.m4b', + active: false, + synced: false, + }); + + const status = await service.getItemsStatus(owner, [deletedBook.uuid]); + + expect(status).toEqual({ unknown: [], unsynced: [] }); + }); + + it('merges every row holding a uuid: the active one decides, deleted ones only mark it seen', async () => { + const trx = getTestTransaction(); + const owner = await user(); + // An active synced book whose uuid an older, deleted row also carries + const synced = await createTestLibraryItem(trx, { user_id: owner.id_user, key: 'Kept.m4b' }); + await createTestLibraryItem(trx, { + user_id: owner.id_user, + key: 'Kept-old.m4b', + uuid: synced.uuid, + active: false, + synced: false, + }); + // An active unsynced book with two deleted rows under its uuid + const unsynced = await createTestLibraryItem(trx, { + user_id: owner.id_user, + key: 'Waiting.m4b', + synced: false, + }); + for (const key of ['Waiting-a.m4b', 'Waiting-b.m4b']) { + await createTestLibraryItem(trx, { + user_id: owner.id_user, + key, + uuid: unsynced.uuid, + active: false, + synced: false, + }); + } + + const status = await service.getItemsStatus(owner, [synced.uuid, unsynced.uuid]); + + expect(status).toEqual({ unknown: [], unsynced: [unsynced.uuid] }); + }); + + it('reads a NULL synced as no file in S3', async () => { + const trx = getTestTransaction(); + const owner = await user(); + const book = await createTestLibraryItem(trx, { user_id: owner.id_user, key: 'Legacy.m4b' }); + await trx('library_items').update({ synced: null }).where({ id_library_item: book.id_library_item }); + + const status = await service.getItemsStatus(owner, [book.uuid]); + + expect(status.unsynced).toEqual([book.uuid]); + }); + + it("answers another user's uuid as unknown: only the caller's rows count", async () => { + const trx = getTestTransaction(); + const owner = await user('owner@example.com'); + const other = await user('other@example.com'); + const theirs = await createTestLibraryItem(trx, { + user_id: other.id_user, + key: 'Theirs.m4b', + synced: false, + }); + + const status = await service.getItemsStatus(owner, [theirs.uuid]); + + expect(status).toEqual({ unknown: [theirs.uuid], unsynced: [] }); + }); + + it('answers in the spelling the client sent, once per uuid', async () => { + // iOS generates uppercase uuids; Postgres hands them back lowercased + const trx = getTestTransaction(); + const owner = await user(); + const book = await createTestLibraryItem(trx, { + user_id: owner.id_user, + key: 'Upper.m4b', + synced: false, + }); + const upperBook = book.uuid.toUpperCase(); + const upperUnknown = randomUUID().toUpperCase(); + + const status = await service.getItemsStatus(owner, [ + upperBook, + upperUnknown, + upperUnknown, + upperBook.toLowerCase(), + ]); + + expect(status).toEqual({ unknown: [upperUnknown], unsynced: [upperBook] }); + }); + + it('leaves strings that are not uuids out of both lists instead of failing the request', async () => { + const owner = await user(); + const neverSeen = randomUUID(); + + const status = await service.getItemsStatus(owner, [ + 'Optional("2c2d0f44-1111-4111-8111-111111111111")', + '', + neverSeen, + ]); + + expect(status).toEqual({ unknown: [neverSeen], unsynced: [] }); + }); + + it('skips the query for an empty or all-invalid list', async () => { + const owner = await user(); + const lookup = jest.spyOn((service as any)._libraryDB, 'getItemsByUuids'); + + await expect(service.getItemsStatus(owner, [])).resolves.toEqual({ unknown: [], unsynced: [] }); + await expect(service.getItemsStatus(owner, ['not-a-uuid'])).resolves.toEqual({ unknown: [], unsynced: [] }); + expect(lookup).not.toHaveBeenCalled(); + }); + + it('handles a library far past the 65,535-parameter limit in one query per half', async () => { + const trx = getTestTransaction(); + const owner = await user(); + const book = await createTestLibraryItem(trx, { + user_id: owner.id_user, + key: 'Big.m4b', + synced: false, + }); + const library = [book.uuid, ...Array.from({ length: 70_000 }, () => randomUUID())]; + + const status = await service.getItemsStatus(owner, library); + + expect(status.unsynced).toEqual([book.uuid]); + expect(status.unknown).toHaveLength(70_000); + }); + + it('throws LibraryLookupError when the read fails: "all unknown" would re-register the whole library', async () => { + const owner = await user(); + jest.spyOn((service as any)._libraryDB, 'getItemsByUuids').mockResolvedValue(null as never); + + await expect(service.getItemsStatus(owner, [randomUUID()])).rejects.toBeInstanceOf(LibraryLookupError); + }); +}); diff --git a/src/__tests__/validation/libraryStatus.test.ts b/src/__tests__/validation/libraryStatus.test.ts new file mode 100644 index 0000000..71bc063 --- /dev/null +++ b/src/__tests__/validation/libraryStatus.test.ts @@ -0,0 +1,28 @@ +import { describe, it, expect } from '@jest/globals'; +import { libraryStatusSchema } from '../../validation/libraryStatus'; + +describe('libraryStatusSchema', () => { + it('accepts any strings, so one malformed local uuid cannot fail the pass', () => { + const parsed = libraryStatusSchema.safeParse({ + uuids: ['2c2d0f44-1111-4111-8111-111111111111', 'Optional("x")', ''], + extra: true, + }); + + expect(parsed.success).toBe(true); + expect(parsed.success && parsed.data).toEqual({ + uuids: ['2c2d0f44-1111-4111-8111-111111111111', 'Optional("x")', ''], + }); + }); + + it('accepts an empty library', () => { + expect(libraryStatusSchema.safeParse({ uuids: [] }).success).toBe(true); + }); + + it.each([ + ['a missing list', {}], + ['a list that is not an array', { uuids: 'abc' }], + ['entries that are not strings', { uuids: [1, 2] }], + ])('rejects %s', (_label, body) => { + expect(libraryStatusSchema.safeParse(body).success).toBe(false); + }); +}); diff --git a/src/api/LibraryRouter.ts b/src/api/LibraryRouter.ts index cff850b..b754043 100644 --- a/src/api/LibraryRouter.ts +++ b/src/api/LibraryRouter.ts @@ -15,6 +15,8 @@ import { completeUploadSchema, abortUploadSchema, } from '../validation/multipartUpload'; +import { libraryStatusSchema } from '../validation/libraryStatus'; +import { largeJsonBody } from './middlewares/jsonBody'; const LibraryRouter = express.Router(); const controller = new LibraryController(); @@ -103,5 +105,12 @@ LibraryRouter.get('/keys', checkSubscription, requireCloudData, (req, res, next) LibraryRouter.post('/uuids', checkSubscription, requireCloudData, (req, res, next) => controller.postLibraryUuids(req, res).catch(next), ); +// The missing-items pass: the body is the client's whole library (see +// docs/multipart-uploads.md), parsed here with a 5 MB limit only once the caller +// is known to be on a tier that can use it (server.ts leaves this one path to the +// route). Hence the tier check before validation, unlike the routes above. +LibraryRouter.post('/status', checkSubscription, requireCloudData, largeJsonBody, validateBody(libraryStatusSchema), (req, res, next) => + controller.postLibraryStatus(req, res).catch(next), +); export default LibraryRouter; diff --git a/src/api/middlewares/jsonBody.ts b/src/api/middlewares/jsonBody.ts new file mode 100644 index 0000000..be506af --- /dev/null +++ b/src/api/middlewares/jsonBody.ts @@ -0,0 +1,23 @@ +import bodyParser from 'body-parser'; +import { NextFunction, Request, Response } from 'express'; + +// Routes that parse their own JSON body, with a larger limit, once they've checked the +// caller: POST /v1/library/status carries every uuid in the client's library. Everything +// else keeps body-parser's 100 KB, so an unauthenticated request can't make the server +// buffer and parse a large body. +const OWN_BODY_ROUTES = new Set(['/v1/library/status']); + +const defaultJsonBody = bodyParser.json(); + +export function jsonBody(req: Request, res: Response, next: NextFunction): void { + // Matched the way Express routes: case-insensitive, trailing slash ignored. Otherwise a + // spelling the route still accepts would be parsed here first, at 100 KB + if (OWN_BODY_ROUTES.has(req.path.toLowerCase().replace(/\/+$/, ''))) { + next(); + return; + } + defaultJsonBody(req, res, next); +} + +// For those routes, after their subscription and tier checks: about 130k uuids at ~39 bytes each +export const largeJsonBody = bodyParser.json({ limit: '5mb' }); diff --git a/src/controllers/LibraryController.ts b/src/controllers/LibraryController.ts index fda1954..446cd4a 100644 --- a/src/controllers/LibraryController.ts +++ b/src/controllers/LibraryController.ts @@ -19,6 +19,7 @@ import { StartUploadBody, listPartsQuerySchema, } from '../validation/multipartUpload'; +import { LibraryStatusBody } from '../validation/libraryStatus'; // Query-string flags arrive as strings; `?sign=false` must not read as true. // Strict on purpose. Every shipped client sends the literal `true`: iOS @@ -431,6 +432,23 @@ export class LibraryController { } } + public async postLibraryStatus( + req: IRequest, + res: IResponse, + ): Promise { + const { uuids } = req.body as LibraryStatusBody; + try { + const status = await this._libraryService.getItemsStatus(req.user, uuids); + return res.json(status); + } catch (err) { + // Identifiers only: the body is the user's whole library. + this._logger.log({ origin: 'LibraryController.postLibraryStatus', message: err.message, data: { user_id: req.user?.id_user, count: uuids.length } }, 'error'); + // A failed read is retryable, and must never read as "nothing known". + res.status(500).json({ message: 'Internal error' }); + return; + } + } + public async postLibraryUuids( req: IRequest, res: IResponse, diff --git a/src/database/migrations/20260929120000_library_items_active_not_null.ts b/src/database/migrations/20260929120000_library_items_active_not_null.ts new file mode 100644 index 0000000..fd29cba --- /dev/null +++ b/src/database/migrations/20260929120000_library_items_active_not_null.ts @@ -0,0 +1,51 @@ +import type { Knex } from 'knex'; + +// Each step commits on its own, so no statement holds an exclusive lock on +// library_items (~1.6M rows) while the table is scanned: a plain SET NOT NULL +// would scan it under ACCESS EXCLUSIVE and block the API for the whole scan. +export const config = { transaction: false }; + +const CHECK = 'library_items_active_not_null'; + +// The ALTERs below still take a brief ACCESS EXCLUSIVE lock. Behind an open +// transaction on the table, one would wait in the lock queue, and every later +// read or write on library_items would queue behind it. Give up after a few +// seconds instead: the migration is safe to run again. +const LOCK_TIMEOUT = '5s'; + +// SET LOCAL, in the statement's own transaction: with `transaction: false` each +// knex.raw may run on a different pooled connection. +async function alter(knex: Knex, sql: string): Promise { + await knex.transaction(async (trx) => { + await trx.raw(`SET LOCAL lock_timeout = '${LOCK_TIMEOUT}'`); + await trx.raw(sql); + }); +} + +// `active` was created nullable (20220515155820). Every read filters on +// `active = true`, so a NULL row is already invisible to clients: effectively +// deleted. Backfill those rows as deleted (false, not true: true would bring +// them back into users' libraries, and could collide with the active-only +// unique indexes on (user_id, key) and (uuid, user_id)), then forbid NULL. +// POST /status counts a deleted row as seen, so no answer depends on NULL. +export async function up(knex: Knex): Promise { + // A failed earlier run may have left the check behind + await alter(knex, `ALTER TABLE library_items DROP CONSTRAINT IF EXISTS ${CHECK}`); + // NOT VALID: enforced for every write from now on (so no NULL can arrive + // between the backfill and the validation), without scanning existing rows + await alter(knex, `ALTER TABLE library_items ADD CONSTRAINT ${CHECK} CHECK (active IS NOT NULL) NOT VALID`); + await knex.raw('UPDATE library_items SET active = false WHERE active IS NULL'); + // Scans the table under SHARE UPDATE EXCLUSIVE: reads and writes carry on + await alter(knex, `ALTER TABLE library_items VALIDATE CONSTRAINT ${CHECK}`); + // Postgres (12+; production runs 17) proves NOT NULL from the validated + // check, without a second scan + await alter(knex, 'ALTER TABLE library_items ALTER COLUMN active SET NOT NULL'); + await alter(knex, 'ALTER TABLE library_items ALTER COLUMN active SET DEFAULT true'); + await alter(knex, `ALTER TABLE library_items DROP CONSTRAINT ${CHECK}`); +} + +// The backfilled rows stay false: which of them were NULL isn't recorded. +export async function down(knex: Knex): Promise { + await alter(knex, `ALTER TABLE library_items DROP CONSTRAINT IF EXISTS ${CHECK}`); + await alter(knex, 'ALTER TABLE library_items ALTER COLUMN active DROP NOT NULL'); +} diff --git a/src/server.ts b/src/server.ts index f9fae9f..680c7d8 100644 --- a/src/server.ts +++ b/src/server.ts @@ -13,6 +13,7 @@ import { RestClientService } from './services/RestClientService'; import { RedisService } from './services/RedisService'; import { logger } from './services/LoggerService'; import { checkVersion } from './api/middlewares/version'; +import { jsonBody } from './api/middlewares/jsonBody'; export class Server { private readonly _logger = logger; @@ -28,7 +29,9 @@ export class Server { await this._cache.connectCacheService(); const app = express(); - app.use(bodyParser.json()); + // 100 KB, except the routes that parse their own larger body after their + // subscription check (POST /v1/library/status) + app.use(jsonBody); app.use(bodyParser.urlencoded({ extended: true })); app.use(compress()); app.use(helmet()); diff --git a/src/services/LibraryService.ts b/src/services/LibraryService.ts index 2af2b6b..dcf8ee4 100644 --- a/src/services/LibraryService.ts +++ b/src/services/LibraryService.ts @@ -1328,6 +1328,61 @@ export class LibraryService { } } + /** + * The client's missing-items pass (docs/multipart-uploads.md): of the uuids in + * its local library, which the server has never seen (the client registers + * them) and which are active books with no file in S3 (a PRO client uploads + * them by uuid). + * + * A deleted row counts as seen, so a book deleted on another device isn't + * registered again. Each uuid is answered in the spelling the client sent + * (Postgres returns uuids lowercased; iOS generates them uppercase), and + * strings that aren't uuids are left out of both lists. A failed read throws + * `LibraryLookupError`: answered as "all unknown", it would have the client + * register its whole library again. + */ + async getItemsStatus( + user: User, + uuids: string[], + ): Promise<{ unknown: string[]; unsynced: string[] }> { + const requested = new Map(); + for (const uuid of uuids) { + const normalized = uuid.toLowerCase(); + if (isValidUUID(uuid) && !requested.has(normalized)) { + requested.set(normalized, uuid); + } + } + if (requested.size === 0) return { unknown: [], unsynced: [] }; + + const rows = this.requireLookup( + await this._libraryDB.getItemsByUuids(user.id_user, [...requested.keys()]), + ); + const seen = new Set(); + const booksWithoutFile = new Set(); + for (const row of rows) { + const normalized = `${row.uuid}`.toLowerCase(); + seen.add(normalized); + if ( + row.active && + parseInt(`${row.type}`) === parseInt(LibraryItemType.BOOK) && + row.synced !== true + ) { + booksWithoutFile.add(normalized); + } + } + + const unknown: string[] = []; + const unsynced: string[] = []; + for (const [normalized, sent] of requested) { + if (!seen.has(normalized)) { + unknown.push(sent); + } else if (booksWithoutFile.has(normalized)) { + unsynced.push(sent); + } + } + return { unknown, unsynced }; + } + async processItemUUIDs( user: User, updates: ItemMatchPayload[], diff --git a/src/services/db/LibraryDB.ts b/src/services/db/LibraryDB.ts index eb57696..1d5f072 100644 --- a/src/services/db/LibraryDB.ts +++ b/src/services/db/LibraryDB.ts @@ -61,6 +61,48 @@ export class LibraryDB { } } + /** + * Every row of the user's, active or deleted, holding one of these uuids. + * Anything that isn't a uuid is skipped: the list becomes a Postgres array + * literal, cast to uuid[]. `null` means the query failed. + * + * The list is the client's whole library and has no size cap, so it goes in + * as ONE array parameter: `whereIn` binds a parameter per uuid, and Postgres + * refuses a query with more than 65,535. The active and deleted halves are + * separate branches so each is served by its partial index + * (`library_items_uuid_user_unique`, `library_items_uuid_user_inactive`), in + * one UNION ALL statement so both read the same snapshot: a row changing state + * between two statements would show up in neither. + */ + async getItemsByUuids( + user_id: number, + uuids: string[], + trx?: Knex.Transaction, + ): Promise[] | null> { + const valid = uuids.filter((uuid) => isValidUUID(uuid)); + if (valid.length === 0) return []; + try { + const db = trx || this.db; + const uuidArray = `{${valid.join(',')}}`; + const half = (query: Knex.QueryBuilder, active: boolean) => + query + .select('uuid', 'active', 'type', 'synced') + .from('library_items') + .where({ user_id, active }) + .whereRaw('uuid = ANY(?::uuid[])', [uuidArray]); + return await half(db.queryBuilder(), true).unionAll(function (this: Knex.QueryBuilder) { + half(this, false); + }, true); + } catch (err) { + this._logger.log({ + origin: 'LibraryDB.getItemsByUuids', + message: err.message, + data: { user_id, count: uuids.length }, + }); + return null; + } + } + /** `null` means the query failed; `[]` means nothing matched. Callers that * answer clients must not collapse the two (see LibraryService.requireLookup). */ async getLibrary( diff --git a/src/validation/libraryStatus.ts b/src/validation/libraryStatus.ts new file mode 100644 index 0000000..4534a95 --- /dev/null +++ b/src/validation/libraryStatus.ts @@ -0,0 +1,18 @@ +import { z } from 'zod'; + +// Request body for POST /status: every uuid in the client's local library. +// Entries only have to be strings. One malformed local uuid must not fail the +// whole request (the pass would then fail every week); LibraryService leaves +// anything that isn't a uuid out of the answer instead. +export const libraryStatusSchema = z + .object({ + uuids: z.array(z.string(), { + required_error: 'uuids is required', + invalid_type_error: 'uuids must be an array of strings', + }), + }) + .strip(); + +// Declared explicitly for the same reason as multipartUpload.ts: no +// strictNullChecks, so z.infer is too loose to trust. +export type LibraryStatusBody = { uuids: string[] };