Repository navigation
fix: close server races and leaks, and gate them with offline race tests - #936
Closed
pavelpikta wants to merge 219 commits into
Closed
pavelpikta wants to merge 219 commits into
pavelpikta wants to merge 219 commits into
Conversation
Signed-off-by: Pavel Pikta <pavel_pikta@epam.com>
Co-Authored-By: Pavel Pikta <devops@pavelpikta.com>
# Conflicts: # build-all.sh
# Conflicts: # .github/workflows/ts_release.yml
- Introduced a new build configuration for GStreamer (torrserver-gst) in .goreleaser.yaml and .goreleaser.local.yaml. - Updated Makefile to include new targets for building GStreamer binaries across multiple platforms. - Enhanced the flattening process to accommodate GStreamer artifacts. Signed-off-by: Pavel Pikta <devops@pavelpikta.com>
- Implemented a new CI step to verify that the Linux binary is statically linked without dynamic library dependencies. - Updated build command in torrserver.nix to ensure CGO is disabled for static builds. - Modified build constraints in GStreamer-related Go files to include 'gst' build tag for better control over GStreamer functionality. Signed-off-by: Pavel Pikta <devops@pavelpikta.com>
…tches - Deleted the 00-responsive-reader.patch and 01-no-udp-panic.patch files as they are no longer needed. - These patches contained commented-out code that has been deemed unnecessary for the current codebase. Signed-off-by: Pavel Pikta <devops@pavelpikta.com>
- Introduced .env.example for environment variable configuration. - Added docker-compose.build.yml for local image build overrides. - Created docker-compose.dlna.yml for DLNA/UPnP support with host networking. - Updated docker-compose.yml for main service configuration and health checks. Signed-off-by: Pavel Pikta <devops@pavelpikta.com>
# Conflicts: # .github/workflows/ts_release.yml # README.md # build-all.sh
# Conflicts: # build-all.sh # server/version/version.go
Updated the URL construction in ffp.go, preload.go, and ffprobe.go to consistently use HTTP without SSL checks Signed-off-by: Pavel Pikta <devops@pavelpikta.com>
Signed-off-by: Pavel Pikta <devops@pavelpikta.com>
Signed-off-by: Pavel Pikta <devops@pavelpikta.com>
Signed-off-by: Pavel Pikta <devops@pavelpikta.com>
Signed-off-by: Pavel Pikta <devops@pavelpikta.com>
Replace fragmented build tooling with Makefile-driven GoReleaser workflows and two standalone configs: `.goreleaser.local.yaml` for fast local builds and `.goreleaser.yaml` for full CI/release (Android, optional UPX). - Add Dockerfile.builder and USE_DOCKER_BUILDER=1 for reproducible builds without local Go/yarn/swag/NDK toolchains - Fix Docker builds: compile linux static binaries, correct compose context, and clarify static vs GStreamer binary usage - Split make dist (local snapshot) from make dist-full (release matrix); gate Android NDK and optional UPX behind Makefile guards - Update CI to validate both GoReleaser configs; refresh `BUILD.md` and `README.md` - Remove vendored UPX binary; use system or builder-image install instead --- Signed-off-by: Pavel Pikta <devops@pavelpikta.com>
Signed-off-by: Pavel Pikta <devops@pavelpikta.com>
Signed-off-by: Pavel Pikta <devops@pavelpikta.com>
# Conflicts: # .github/workflows/ts_release.yml # .gitignore # server/web/pages/template/html.go # server/web/pages/template/pages/asset-manifest.json # server/web/pages/template/pages/static/js/2.173efca3.chunk.js.LICENSE.txt # server/web/pages/template/pages/static/js/2.70a31402.chunk.js.LICENSE.txt # server/web/pages/template/pages/static/js/2.9d2260bd.chunk.js.LICENSE.txt # server/web/pages/template/route.go
Signed-off-by: Pavel Pikta <devops@pavelpikta.com>
Signed-off-by: Pavel Pikta <devops@pavelpikta.com>
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
…edirects Take YouROK YouROK#929/YouROK#928/YouROK#923/YouROK#930 while keeping InfoBytes and Torznab caps/Search API.
Signed-off-by: Pavel Pikta <devops@pavelpikta.com>
Signed-off-by: Pavel Pikta <devops@pavelpikta.com>
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: Pavel Pikta <devops@pavelpikta.com>
Signed-off-by: Pavel Pikta <devops@pavelpikta.com>
- Fix piece-cache RAM retention after torrent/cache Close: explicitly Release MemPiece/DiskPiece buffers, reject WriteAt/ReadAt when closed, drop buffers on MarkNotComplete, and keep `pieces` map only after buffers are cleared (regression since MatriX.143.0.11 / `c5e553aa`, which stopped nil’ing the map for race safety). - Harden concurrency: atomic `filled`, `snapshotStats` under piece mutex, remove racy `isRemove`, throttle `FreeOSMemGC` on Close/eviction; race/deadlock tests included. - Observability: detailed MemStats + per-torrent `caches` on `GET /runtime/status`, optional `--pprof`, Server Status Memory section; SW denylist for `/runtime` so PWA does not hijack the API with relative `./static` 404s. ----- Signed-off-by: Pavel Pikta <devops@pavelpikta.com>
Drop expired DNS cache entries under the write lock and copy cached addresses before returning them. Probe timeouts now cancel with the request instead of leaving a goroutine, and the MSX start parameter is guarded by a mutex. Signed-off-by: Pavel Pikta <devops@pavelpikta.com>
Cover DNS, ffprobe, DLNA, logging, settings, and web handlers with temp directories and httptest, including concurrent access. Signed-off-by: Pavel Pikta <devops@pavelpikta.com>
CloseDB now drops the process-wide database handles so a later start can reopen them, and same-length JSON is compared structurally when the bytes differ. DLNA fatal exits, the torrent directory watch, and interface lookups stay overridable without changing production behavior.
Drive the new seams with in-memory interfaces, httptest, and temp databases.
Guard cache, reader, and torrent fields that raced under concurrent reads, and let cmd, search, and the Telegram bot start and stop without process exits.
Exercise those packages with a local seeder and synthetic metainfo, and point the live checks at b0e4e3cf193431debd67b54ad7936356c0e5c6a7.
Close the piece-cache watchdog and idle HTTP connections when streaming and Telegram stop, and let the upload queue start again after Stop.
Cover piece-cache turnover, torrent churn, aborted streams, and DLNA, Bonjour, and Telegram start/stop.
Pool evicted in-memory piece slabs and serve /stream and /play ranges from a reused copy buffer so cache turnover allocates less.
…ment - Updated the DHT and torrent libraries to newer versions from lampac-nextgen. - Improved tracker management by adding functions to sanitize and merge tracker URLs. - Enhanced the BTServer lifecycle management with proper DHT context handling. - Added tests for DHT bootstrap resolution and tracker merging functionality. - Refactored existing code to improve readability and maintainability. Signed-off-by: Pavel Pikta <devops@pavelpikta.com>
- Introduced a new documentation file `AGENTS.md` detailing the TorrServer agent setup and operational guidelines. - Included module path specifications, dependency management, and package roles. - Provided insights into the streaming process, lock order, and handling of various components within the server architecture. - Clarified the use of dependency forks and their implications on API behavior. Signed-off-by: Pavel Pikta <devops@pavelpikta.com>
- Introduced a new skill documentation file `SKILL.md` for the principal Go TorrServer architect role. - Detailed guidelines on concurrency, race debugging, cache management, and HTTP handling within the TorrServer architecture. - Specified module paths, dependency forks, and provided instructions on locks and streaming processes. - Emphasized testing requirements and best practices for maintaining thread safety and performance. Signed-off-by: Pavel Pikta <devops@pavelpikta.com>
- Removed redundant information about the React UI and clarified the rebuild process. - Added a new section on UI boundary detailing localization and CI checks for web/src changes. - Improved clarity on lock management and operational guidelines within the TorrServer architecture. Signed-off-by: Pavel Pikta <devops@pavelpikta.com>
Contributor
Author
|
Closed: this targeted upstream master and included the whole fork. The branch is 15 commits on lampac-nextgen/TorrServer. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes races and goroutine leaks on the streaming and startup paths, then locks that behavior in with an offline race-test gate.
make testis now-race -count=1for everyserver/package except generated Swagger, with a 70% total coverage floor. Benchmarks fail CI if they regress past 200%.The hot path is unchanged in shape:
GET/HEAD /streamand/playstill go throughTorrent.Stream→torrstor.Reader→ the piece cache. Piece slabs and the 64KiB stream copy buffer are pooled so cache turnover stops allocating on every read.Fixes
CloseDBclears the process-wide bbolt and JSON handles so the next start does not reuse a closed database. Same-length JSON is compared structurally.cmd, search, and the Telegram bot start and stop through seams instead of exiting the process. DLNA listen failure, the torrent directory watch, and interface lookup stay overridable.Stop./streamand/playcopy from a pooled 64KiB buffer. A closed cache does not recycle slabs, so dropped piece RAM can return to the OS.Trackers and DHT
github.com/tsynik/torrent v1.2.34togithub.com/lampac-nextgen/torrent v1.2.35. uTP comes fromgithub.com/lampac-nextgen/utp v0.2.1.dht.GlobalBootstrapAddrs.Tests
server/torr/offlinetest: temp settings DB, loopback memStore seeder, no public DHT.httptest, offline bot).torrreader bench stays out ofmake benchbecause it blocks onGotInfo.LIVE_TORRENT=1,LIVE_INDEXER=1).Docs
AGENTS.mdis the map of locks, forks, streaming, and test commands. Lock order isdbMigrationLockthensettings.mu.ListTorrentandAddTorrentmust not run whilesettings.muis held. Idle park is 60 seconds.web/src. Locale keys must match acrossen,ru,ua,bg,fr,ro, andzh. Frontend CI isyarn typecheck,lint,test,i18n:check,test:e2e, andbuild.Test plan
make fmt-checkmake lintmake test(race, coverage profile atserver/coverage.out, total ≥ 70%)make bench(piece slab and stream copy buffer still pooled on-benchmem)/streamstill serves a range and the reader drops after the client disconnects