Repository navigation
Drive stream windows from torrent v1.2.36 reader tiers - #939
Closed
pavelpikta wants to merge 245 commits into
Closed
pavelpikta wants to merge 245 commits into
pavelpikta wants to merge 245 commits into
Conversation
Signed-off-by: Pavel Pikta <pavel_pikta@epam.com>
Co-Authored-By: Pavel Pikta <devops@pavelpikta.com>
# Conflicts: # build-all.sh
# Conflicts: # .github/workflows/ts_release.yml
- Introduced a new build configuration for GStreamer (torrserver-gst) in .goreleaser.yaml and .goreleaser.local.yaml. - Updated Makefile to include new targets for building GStreamer binaries across multiple platforms. - Enhanced the flattening process to accommodate GStreamer artifacts. Signed-off-by: Pavel Pikta <devops@pavelpikta.com>
- Implemented a new CI step to verify that the Linux binary is statically linked without dynamic library dependencies. - Updated build command in torrserver.nix to ensure CGO is disabled for static builds. - Modified build constraints in GStreamer-related Go files to include 'gst' build tag for better control over GStreamer functionality. Signed-off-by: Pavel Pikta <devops@pavelpikta.com>
…tches - Deleted the 00-responsive-reader.patch and 01-no-udp-panic.patch files as they are no longer needed. - These patches contained commented-out code that has been deemed unnecessary for the current codebase. Signed-off-by: Pavel Pikta <devops@pavelpikta.com>
- Introduced .env.example for environment variable configuration. - Added docker-compose.build.yml for local image build overrides. - Created docker-compose.dlna.yml for DLNA/UPnP support with host networking. - Updated docker-compose.yml for main service configuration and health checks. Signed-off-by: Pavel Pikta <devops@pavelpikta.com>
# Conflicts: # .github/workflows/ts_release.yml # README.md # build-all.sh
# Conflicts: # build-all.sh # server/version/version.go
Updated the URL construction in ffp.go, preload.go, and ffprobe.go to consistently use HTTP without SSL checks Signed-off-by: Pavel Pikta <devops@pavelpikta.com>
Signed-off-by: Pavel Pikta <devops@pavelpikta.com>
Signed-off-by: Pavel Pikta <devops@pavelpikta.com>
Signed-off-by: Pavel Pikta <devops@pavelpikta.com>
Signed-off-by: Pavel Pikta <devops@pavelpikta.com>
Replace fragmented build tooling with Makefile-driven GoReleaser workflows and two standalone configs: `.goreleaser.local.yaml` for fast local builds and `.goreleaser.yaml` for full CI/release (Android, optional UPX). - Add Dockerfile.builder and USE_DOCKER_BUILDER=1 for reproducible builds without local Go/yarn/swag/NDK toolchains - Fix Docker builds: compile linux static binaries, correct compose context, and clarify static vs GStreamer binary usage - Split make dist (local snapshot) from make dist-full (release matrix); gate Android NDK and optional UPX behind Makefile guards - Update CI to validate both GoReleaser configs; refresh `BUILD.md` and `README.md` - Remove vendored UPX binary; use system or builder-image install instead --- Signed-off-by: Pavel Pikta <devops@pavelpikta.com>
Signed-off-by: Pavel Pikta <devops@pavelpikta.com>
Signed-off-by: Pavel Pikta <devops@pavelpikta.com>
# Conflicts: # .github/workflows/ts_release.yml # .gitignore # server/web/pages/template/html.go # server/web/pages/template/pages/asset-manifest.json # server/web/pages/template/pages/static/js/2.173efca3.chunk.js.LICENSE.txt # server/web/pages/template/pages/static/js/2.70a31402.chunk.js.LICENSE.txt # server/web/pages/template/pages/static/js/2.9d2260bd.chunk.js.LICENSE.txt # server/web/pages/template/route.go
Signed-off-by: Pavel Pikta <devops@pavelpikta.com>
Signed-off-by: Pavel Pikta <devops@pavelpikta.com>
- Introduced a new documentation file `AGENTS.md` detailing the TorrServer agent setup and operational guidelines. - Included module path specifications, dependency management, and package roles. - Provided insights into the streaming process, lock order, and handling of various components within the server architecture. - Clarified the use of dependency forks and their implications on API behavior. Signed-off-by: Pavel Pikta <devops@pavelpikta.com>
- Introduced a new skill documentation file `SKILL.md` for the principal Go TorrServer architect role. - Detailed guidelines on concurrency, race debugging, cache management, and HTTP handling within the TorrServer architecture. - Specified module paths, dependency forks, and provided instructions on locks and streaming processes. - Emphasized testing requirements and best practices for maintaining thread safety and performance. Signed-off-by: Pavel Pikta <devops@pavelpikta.com>
- Removed redundant information about the React UI and clarified the rebuild process. - Added a new section on UI boundary detailing localization and CI checks for web/src changes. - Improved clarity on lock management and operational guidelines within the TorrServer architecture. Signed-off-by: Pavel Pikta <devops@pavelpikta.com>
…erformance - Enhanced variable handling by extracting fields from the Torrent struct before locking. - Simplified data assignment logic for torrent data and poster. - Ensured thread safety by properly managing locks during data updates. - Maintained existing timestamp logic to prevent overwriting during edits. Signed-off-by: Pavel Pikta <devops@pavelpikta.com>
- Introduced an interface for mDNS announcers to improve abstraction and testing. - Replaced direct calls to zeroconf.RegisterProxy with a newAnnouncer function for better testability. - Updated tests to utilize a fake announcer, ensuring concurrent start/stop functionality without race conditions. - Refactored notifyInterval handling in DLNA to improve clarity and maintainability. Signed-off-by: Pavel Pikta <devops@pavelpikta.com>
- Added a check to create a default benchmark data file if it does not exist, preventing cache misses on the first run. - Ensured that the benchmark action can proceed smoothly by initializing necessary files. Signed-off-by: Pavel Pikta <devops@pavelpikta.com>
- Added permissions for pull requests to enable commenting on alerts and summaries. - Configured benchmark action to use the GitHub token for enhanced functionality. - Ensured comments are always generated for benchmark results, improving visibility. Signed-off-by: Pavel Pikta <devops@pavelpikta.com>
…checks - Eliminated redundant pointer checks in the TestPieceBufRecycledAndZeroed function. - Streamlined the test logic to focus on the core functionality of piece buffer recycling and zeroing. Signed-off-by: Pavel Pikta <devops@pavelpikta.com>
- Added a coverage summary step to display total coverage percentage in the GitHub step summary. - Introduced a benchmark summary step that outputs benchmark results if new entries are detected. - Updated conditions for saving benchmark history to ensure it runs on successful jobs. Signed-off-by: Pavel Pikta <devops@pavelpikta.com>
- Modified the `fetchTrackersFromURLs` function to accept a generation parameter, ensuring that stale fetches do not cancel current operations. - Enhanced tests to verify that stale tracker fetches do not interfere with ongoing requests and that the correct trackers are returned. - Updated related functions to maintain consistency with the new parameter structure. Signed-off-by: Pavel Pikta <devops@pavelpikta.com>
…line-tests ## Summary Closes races and goroutine leaks on the streaming and startup paths, then locks that behavior in with an offline race-test gate. `make test` is now `-race -count=1` for every `server/` package except generated Swagger, with a 70% total coverage floor. Benchmarks fail CI if they regress past 200%. The hot path is unchanged in shape: `GET/HEAD /stream` and `/play` still go through `Torrent.Stream` → `torrstor.Reader` → the piece cache. Piece slabs and the 64KiB stream copy buffer are pooled so cache turnover stops allocating on every read. ## Fixes - DNS cache: drop expired entries under the write lock and copy addresses before return. ffprobe's probe timer cancels with the request. The MSX start parameter is guarded by a mutex. - `CloseDB` clears the process-wide bbolt and JSON handles so the next start does not reuse a closed database. Same-length JSON is compared structurally. - Reader, cache, and torrent fields that raced under concurrent reads are guarded. `cmd`, search, and the Telegram bot start and stop through seams instead of exiting the process. DLNA listen failure, the torrent directory watch, and interface lookup stay overridable. - Shutdown closes the piece-cache watchdog and idle tracker HTTP connections. The Telegram upload queue can start again after `Stop`. - Evicted in-memory piece slabs go back to a length-keyed pool. `/stream` and `/play` copy from a pooled 64KiB buffer. A closed cache does not recycle slabs, so dropped piece RAM can return to the OS. ## Trackers and DHT - Torrent fork moves from `github.com/tsynik/torrent v1.2.34` to `github.com/lampac-nextgen/torrent v1.2.35`. uTP comes from `github.com/lampac-nextgen/utp v0.2.1`. - Tracker URLs are sanitized and merged as one extra tier. DHT bootstrap uses a bounded lookup instead of `dht.GlobalBootstrapAddrs`. ## Tests - Offline harness in `server/torr/offlinetest`: temp settings DB, loopback memStore seeder, no public DHT. - Race and lifecycle coverage for DLNA, settings, FUSE, WebDAV, the web API, MCP, and Telegram (`httptest`, offline bot). - Heap ceilings and goroutine audits for piece-cache turnover, torrent churn, aborted streams, and DLNA, Bonjour, and Telegram start/stop. - Benches for piece-buffer reuse, stream ranges, subs, torrshash, torrfs, and API playlists. The `torr` reader bench stays out of `make bench` because it blocks on `GotInfo`. - Live checks stay opt-in (`LIVE_TORRENT=1`, `LIVE_INDEXER=1`). ## Docs - `AGENTS.md` is the map of locks, forks, streaming, and test commands. Lock order is `dbMigrationLock` then `settings.mu`. `ListTorrent` and `AddTorrent` must not run while `settings.mu` is held. Idle park is 60 seconds. - UI edits belong in `web/src`. Locale keys must match across `en`, `ru`, `ua`, `bg`, `fr`, `ro`, and `zh`. Frontend CI is `yarn typecheck`, `lint`, `test`, `i18n:check`, `test:e2e`, and `build`. - The architect skill points at that file and does not restate a shorter idle timeout.
Read-only and empty settings no longer restart services or panic, and the UI sends integer settings, encoded stream URLs, and Basic auth for the media host.
The cache view paints only the visible window, and stream paths encode the characters Infuse, VLC, and IINA reject.
Split the file tree and poll cadence out of the components so encoding, abort, and a large pack can be checked without a browser.
Float settings, read-only shutdown, a missing TMDB config, and a bad cache body now fail in tests instead of only on the happy path.
The benchmark action treats every Go unit as smaller-is-better, so a faster throughput run trips the 200% alert. ns/op still fails the job.
A 200ms bench on a shared runner crosses 200% with no code change. The comparison still comments on the PR.
## Summary Settings saves, stream links, and the piece map were failing on ordinary torrents: a bad settings body restarted services or panicked, player URLs split on `#` `?` `&` and quotes, and a large remux tried to draw every piece. This locks those paths down and covers them with frontend and `server/web` tests. - `POST /settings` with `action: set` and no `sets` returns `400`. Read-only `set` and `def` return `403` and do not call `restartServices`. `SetBTSets(nil)` returns immediately. - The client rounds integer `BTSets` fields before the post. `CacheSize` stays bytes on the wire. `NaN` uses the server default. `ReaderReadAHead` is clamped to 5–100 and `PreloadCache` to 0–100. `StoreViewedInJson` is posted back and is still not a form control. - Stream and playlist URLs are built in `web/src/shared/lib/streamLinks.ts`. The filename is one encoded path segment (`!'()*` included). `link` and `index` go through `URLSearchParams`. Play stays `&play`, preload stays `&preload`, playlist stays `&m3u&fn=file.m3u`. - Desktop VLC is `vlc://` unencoded. iOS VLC, Infuse, SenPlayer, and IINA encode the absolute URL. Android VLC is an `intent://` href. An Infuse episode list longer than 8000 characters falls back to one m3u URL. Clipboard copy uses `navigator.clipboard` on HTTPS or localhost, and a textarea inside the open dialog on a plain-HTTP LAN address. - Basic userinfo is added only when the media host matches the page. Scheme and port may differ. - The piece map draws the visible window on one canvas, not one cell per piece. Polling pauses while the tab is hidden. A cancelled or unchanged `/cache` snapshot does not replace the one on screen. File rows and the folder tree stay usable on a 1000-file pack. ## Tests - Vitest covers stream and player URLs (spaces, Cyrillic, brackets, quotes, `#`, `?`, `&`, `%`), clipboard fallback, `/torrents` and `/settings` and `/cache` and `/viewed` errors (`401`, `403`, `500`, bad JSON, timeout, abort), poll intervals, a 1200-file tree, a 1000-file season filter, and a 10000-piece cache map. - Go tests cover a ranged `/download`, read-only `/shutdown`, uninitialized TMDB settings, a float inside `BTSets`, a bad `/cache` and `/viewed` body, auth account loading, WAF reload, the self-signed cert path, and the TLS connection queue address. - `server/web` is 76.5% statements under `go test -race ./web/...`. CI does not need a workflow change: `yarn test` and `make test` already pick these files up. The 70% total floor stays.
Signed-off-by: Pavel Pikta <devops@pavelpikta.com>
…#61) A probe or a player that hangs up used to pin a `/play` reader until the swarm produced bytes. `Reader.Close` does not wake an anacrolix `Read` that is blocked on `context.Background`, so the piece window and the reader stayed live after the client was gone. ffprobe made this worse: stock ffprobe waits forever on a silent socket, and each probe is another process plus another `/play` reader. - `/ffp` probes through `ProbeURLContext` with the HTTP request context. At most 4 ffprobe processes run at once. A fifth waits for a slot and gives up if the caller cancels. After a slot is free the probe still has the existing one-minute timeout. - The ffprobe command gets `-rw_timeout 30000000` (30 seconds on one socket read) so a quiet loopback `/play` cannot sit there indefinitely. - `/stream` and `/play` wrap the reader so `bufio`/`ServeContent`, which only call `Read`, pass the request context into `ReadContext`. When the client leaves, the blocked piece wait returns and `CloseReader` runs. - `/ffp` answers `504` on deadline and `408` on cancel. Any other probe error stays `400`. The probe does not hold `BTServer` or `Torrent` locks, so the loopback `/play` it dials cannot deadlock on them.
## Summary Preload is supposed to fetch an edge at each end of the file so a player can open the container: 8MiB, or one piece when the piece is larger. Both edges were dropped before the bytes reached the cache. Playback without preload is unchanged: a size probe still does not pull the tail of a huge file, and a player range request still downloads the tail on demand. ### The tail was never requested when the file fit in the budget `size` is clamped to the file length first. A 30MiB file with a 32MiB preload becomes 30MiB. The 8MiB edge is then subtracted from both ends, so the head ends at 22MiB and the tail starts at 22MiB. The tail reader started only when its offset was strictly greater than the head. 22 is not greater than 22, so the reader never started. The head read `[0, 22MiB)` and the last 8MiB was never requested. The same comparison skipped the tail whenever the two windows met or overlapped, including a budget smaller than the window on a short file. ### The last piece of a range that did run stayed incomplete Both read loops stopped when the next full 32KiB buffer would pass the end of the range. The last partial buffer was never read, so the last piece of the head and of the file never completed. ### Equal budget and window swapped the ends When the preload budget equalled the edge (for example a 16MiB piece and a 16MiB budget) the head length became 0 and only the tail was read. Raising the head to the full budget without also covering the uncovered suffix did the opposite on a file shorter than two windows: the head was read and the real end of the file was skipped. `preloadSpans` now returns `[0, head)` and `[tail, fileLen)`. If the windows overlap, `tail` starts where the head stopped, so both edges are read and the shared bytes are read once. The loops read until the end of the range and slice the last buffer to the remaining bytes, so they do not run past the head budget into the middle. On a long file with the default 32MiB budget and an 8MiB edge the total stays 24MiB at the start plus the last 8MiB. When the budget is no larger than the edge, a long file downloads both edges, one extra window at most. ### The cache could evict the last piece after it was fetched The keep window is supposed to retain the first and last 8MiB of an open file, or a whole piece when the piece is larger. The tail test was `id > es`. When the file ended on a piece boundary and the piece was at least 8MiB, that excluded the real last piece and included a piece past the end of the file. Eviction could drop the tail preload had just stored. A piece now stays when it overlaps either edge. A zero piece length returns false instead of dividing by zero. Preload still reads through the anacrolix file reader, not a cache reader. The file-edge keep window applies once a stream reader is open. For a single-file torrent the empty-reader keep window already holds the torrent start and end.
A later settings write raced with runSearchQuery reading EnableRutorSearch and EnableTorznabSearch.
…anagement - Updated the dependency for github.com/lampac-nextgen/torrent from v1.2.35 to v1.2.36. - Improved the management of reader windows in the cache system, ensuring proper allocation of download zones and readahead settings for active readers. - Adjusted preload logic to maintain download efficiency and prevent idle readers from holding unnecessary resources. Signed-off-by: Pavel Pikta <devops@pavelpikta.com>
- Eliminated the prioPieces map and associated locking mechanisms from the Cache struct, simplifying the eviction logic. - Removed tests related to priority piece management, as the functionality is no longer present. - Adjusted related methods to ensure they no longer reference priority handling, streamlining the codebase. Signed-off-by: Pavel Pikta <devops@pavelpikta.com>
Reader tiers replaced prioPieces and muPrio. The map still told the next edit to keep them.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
lampac-nextgen/torrentv1.2.36. That tag keeps the DHT close and cold-start announce, and adds reader tiers (Now / Next / Readahead / zone), dropping chunks no reader wants, and a stall guard of 3.AdjustRAnow pushesSetNext/SetReadahead/SetZone: zone is(capacity/readers)*ReaderReadAHead/100, readahead ismin(16MiB/readers, zone), next ismin(4MiB/readers, readahead). The cache no longer writes piece priorities, which would raise over those tiers. Preload sets the tail zone after the seek.-qB4390-,TotalHalfOpenConns500, andConnectionsLimit25 are unchanged. The web UI and the cache snake need no change:/cachestill reports reader ranges andPieceState.Priority.Test plan
make fmt-checkmake lint(0 issues)make test(race, torrstor 71.7%)make bench(sequential piece write/read 0 allocs/op)