[pull] master from kevoreilly:master - #517
Merged
Merged
Conversation
* Implement workaround for PanicException serialization Added a workaround for PanicException serialization failures by dynamically registering a module. This allows for successful pickle/unpickle operations. * Will's fixes + some extra * fix yara-x
Remove unused import of sys module.
* Implement workaround for PanicException serialization Added a workaround for PanicException serialization failures by dynamically registering a module. This allows for successful pickle/unpickle operations. * Will's fixes + some extra * fix yara-x * yara-x follow-ups: lint, thread-affinity tests, and a recompile backoff (#3152) * lint: fix ruff failures on this branch pebble.py's 'import sys' went unused when the PanicException stub was removed (F401), and the new engine tests have whitespace on blank lines (W293). The pre-commit ruff hook fails on both. * tests: guard the yara-x Scanner thread-affinity fix yara_x.Scanner is unsendable: PyO3 panics if one is touched from a thread other than the one that built it. Nothing covered this, because test_yara_x skips whenever yara-x is not installed -- the default, since pyproject pins yara-python -- so CI never executes the yara-x branch at all. Adds four tests (skipped without yara-x, so CI behaviour is unchanged) and a comment at the cache site recording why the Scanner must not be cached -- including per-thread, since Scanner is unsendable for drop as well as use and a cached one is dropped on the wrong thread on fork and at shutdown. * objects: back off instead of recompiling YARA rules for every file get_yara() forces a full six-category recompile whenever a category is missing, and remembers nothing -- so a category that cannot produce rules costs one full recompile per scanned file (~3s each on a production ruleset; CAPE.py and procmemory.py call this per extracted payload). Records the failure with a 300s backoff rather than a permanent skip: workers run with max_tasks=0 (no recycling), so disabling a category after one transient failure would silently return no matches for the rest of the run. A forced re-init drops the backoff for categories that actually compiled -- clearing the whole record would make each broken category forget the others and recompile on every alternating call. * Address review: real category list in tests, snapshot before iterating The tests created a 'scripts' YARA directory, but init_yara() compiles 'monitor' -- so the real category was never exercised and every run logged a missing-directory warning for it. Use the actual category tuple. Snapshot cls.yara_rules before iterating it: it is class-level and two threads can be inside a forced init at once (get_yara's fallback triggers one), so iterating it directly risks 'dictionary changed size during iteration'. Snapshotting also keeps categories injected outside the built-in list, which iterating the fixed tuple would miss. --------- Co-authored-by: William Metcalf <wmetcalf@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot] (v2.0.0-alpha.4)
Can you help keep this open source service alive? 💖 Please sponsor : )