Implement SigmaPolicy for enhanced security and flexibility in template processing

[thomaspatzke]

• Introduced SigmaPolicy to manage trust models and security settings for template variables and external sources.
• Updated tests to utilize SigmaPolicy for controlling template variable usage and path restrictions.
• Enhanced documentation to explain the usage and configuration of SigmaPolicy.
• Added comprehensive tests for external data source placeholder transformations, ensuring security and functionality.

[Copilot]

Copilot review overview

🔵 Needs a closer look

It is a breaking change to security-critical trust-model APIs (arbitrary code execution / external source gating) and includes accidental artifacts (a committed .bak duplicate and BOM/encoding corruption), warranting human review.

Review effort: Balanced
Findings: 2

Open (2)

• Remove unintended UTF-8 BOM from file · New
• Stale duplicate test file should be removed · New

What changed in this PR

This PR refactors pySigma's security/trust-model controls by replacing the separate per-method loader parameters (allow_template_vars, vars_allowed_paths, allow_external_sources) with a single SigmaPolicy object that is threaded through pipeline, template, finalizer, and external-source loading. This centralizes regex-engine selection and security-sensitive opt-ins in one place, aligning with the existing SigmaPolicy/default_policy infrastructure. It is a breaking public-API change (documented in a new guide) affecting security-critical code paths (arbitrary code execution via template vars, external source access).

Changes:

• Extended SigmaPolicy with allow_template_vars, vars_allowed_paths, and allow_external_sources (RE2 engine default), and migrated all loaders/transformations to read from the policy instead of individual parameters.
• Updated from_yaml/from_dict propagation (including source_path-derived vars_allowed_paths via dataclasses.replace) and regex.py to resolve the engine from a transformation's own policy before a pipeline is attached.
• Added documentation (sigma_policy guide, cross-links) and migrated all affected tests to construct SigmaPolicy; excluded tests/ from mypy.

File	Description
sigma/​policy/​__init__.py	Adds security fields + RE2 default to SigmaPolicy.
sigma/​processing/​pipeline.py	Replaces loader params with policy; derives restrict_template_path/vars_allowed_paths from effective policy.
sigma/​processing/​templates.py	TemplateBase reads vars/path settings from self.policy.
sigma/​processing/​finalization.py	NestedFinalizer propagates policy and strips untrusted keys.
sigma/​processing/​transformations/​external.py	External source gating reads policy.allow_external_sources.
sigma/​processing/​regex.py	Falls back to a transformation's own policy engine when no pipeline is set.
mypy.ini	Excludes tests/ from type checking.
docs/​guides/​sigma_policy.rst, processing_pipelines.rst, index.rst, reference/​processing.rst	New SigmaPolicy guide and cross-references.
tests/​test_*_transformations.py, test_processing_pipeline.py, test_processing_templates_security.py	Migrated tests to SigmaPolicy.
tests/​test_external_placeholder_transformations.py.bak	Accidentally committed pre-migration backup (should be removed).

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.