Skip to content

fix: compile |re values with the rule's policy - #585

Open
cristianchiriac wants to merge 1 commit into
SigmaHQ:mainfrom
cristianchiriac:fix/re-modifier-policy
Open

cristianchiriac wants to merge 1 commit into
SigmaHQ:mainfrom
cristianchiriac:fix/re-modifier-policy

Conversation

@cristianchiriac

Copy link
Copy Markdown
Contributor

According to the 2.0 release notes, full regex compatibility can be restored with PythonRegexEngine, e.g. through TrustedPolicy. That currently doesn't work for |re values in rules: passing the policy when loading a rule still compiles the regex with RE2.

from sigma.rule import SigmaRule
from sigma.policy.profiles import TrustedPolicy

SigmaRule.from_yaml("""
title: Test
status: test
logsource:
    category: test
detection:
    selection:
        field|re: 'foo(?=bar)'
    condition: selection
""", policy=TrustedPolicy)
# SigmaRegularExpressionError: Regular expression 'foo(?=bar)' is invalid: b'invalid perl operator: (?='

SigmaCollection.from_yaml(..., policy=TrustedPolicy) fails the same way. SigmaRule.from_dict passes the policy to SigmaDetections.from_dict, but it stops there. Modifiers run when each SigmaDetectionItem is built, and SigmaRegularExpressionModifier.modify() creates SigmaRegularExpression(val.original) without a policy, so compile() falls back to sigma.default_policy.

Change

  • SigmaDetectionItem gets an optional policy field (compare=False, repr=False).
  • SigmaDetectionItem.from_mapping() and from_value(), and SigmaDetection.from_definition(), take an optional policy argument and pass it on.
  • SigmaDetections.from_dict() and SigmaFilter.from_dict() pass their policy down.
  • The |re modifier creates SigmaRegularExpression(..., policy=self.detection_item.policy).

All new parameters are keyword-compatible additions with a None default, so existing callers are unaffected. Without an explicit policy, behavior is unchanged: RE2 via sigma.default_policy.

Testing

  • New tests in tests/test_policy.py:
    • a rule with |re|i: 'foo(?=bar)' loads with TrustedPolicy, both via SigmaRule and SigmaCollection, and the value carries the policy;
    • the same rule is still rejected with the default policy.
    • The first test fails without the change.
  • Full suite: 1809 passed, 2 skipped (-m "not online"). mypy sigma and black --check are clean.

This came up while running pySigma-validators-sigmaHQ with pySigma 2.0. The tests of SigmahqUnsupportedRegexGroupConstructValidator need to load rules containing lookarounds, and they can't do that even with TrustedPolicy until this is fixed.

SigmaRegularExpressionModifier created SigmaRegularExpression without a
policy, so |re values were always compiled with sigma.default_policy. Rules
loaded with SigmaRule/SigmaCollection.from_yaml(..., policy=TrustedPolicy)
were still checked with RE2 and rejected as soon as they used constructs
like lookaheads that only the Python engine supports.

Pass the policy from SigmaDetections.from_dict (and SigmaFilter) through
SigmaDetection.from_definition and SigmaDetectionItem.from_mapping to the
detection item, and use it in the |re modifier.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant