Conversation
…date The whole log of a run is stored in its mongo document with an unbounded $push. A plugin logging a long message on every page of a paginated import filled the 16MB document: the task failed, and since finish() also needs to update the document, the run could never be marked finished or killed and the kill loop retried it forever. - truncate msg/extra of a log entry beyond worker.task.maxLogEntryLength (10 000 chars, the cap already applied to axios errors) - when a $push is refused because the document is full, keep the tail of the log, write an explicit error entry and fail the task - in finish(), apply the same truncation and retry when the document is already too large (runs predating this guard)
BatLeDev
force-pushed
the
fix-run-log-size-guard
branch
from
September 15, 2026 09:45
abe46de to
05ae1af
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The whole log of a run is stored in its mongo document with an unbounded
$push. A plugin logging a ~5KB message on each of 3 000+ pages filled the 16MB document: the task failed, and sincefinish()also has to update the document, the run could never be marked finished or killed and the kill loop retried it every 20s forever (Resulting document after update is larger than 16777216).task.ts:msg/extraof a log entry are truncated beyondworker.task.maxLogEntryLength(10 000 chars, the cap already hard-coded for axios errors, now shared). When a$pushis refused because the document is full, the log is cut to its tail, an explicit error entry is written and the task fails.runs.ts:finish()applies the same truncation and retries when the document is already too large, so runs that predate this guard (the one currently stuck in production) get finished on the next kill loop iteration.runs-operations.ts: pure helpersisDocumentTooLargeErrorandtruncateLogValue, unit tested.worker.task.maxLogEntryLength/WORKER_TASK_MAX_LOG_ENTRY_LENGTH.log-flood.api.spec.tswith aprocessing-log-floodfixture plugin fills the document in ~25s and checks the run ends inerror, every entry is truncated, the guard message is present andfinish()could still write its entry.Why: a run must never become impossible to finish or kill because of its own log; a plugin that floods its log should fail with a clear message instead of wedging the worker.
Heads-up: a non-string
extrawhose serialization exceeds the cap is now stored as a truncated string instead of an object (the UI renders both). The mongo error is matched on code 17419 or/larger than \d+/in the message, since the driver wraps it in aPlan executor errormessage. On deploy,finish()will close the run currently stuck in production, keeping only its last 100 log entries.The plugin side is fixed separately in
processing-import-api(one progress entry instead of a log per page, detection of an API that ignores its pagination parameter).