Chore/detect secrets perf optimizations - #294
Merged
Merged
Conversation
The call_function_with_arguments() function is called O(lines x plugins) times during scanning. Previously, make_function_self_aware() ran on every call because bound methods are new objects on each attribute access. This change adds a _plan_cache dict keyed by id(func.__func__) for bound methods (stable across calls). The injectable parameter set is computed once per distinct callable and reused on subsequent calls. Feature flag: DETECT_SECRETS_PERF_DI_CACHE=0 disables the cache for debugging.
The golden snapshot was silently regenerated from 343 to 306 findings when the optional gibberish-detector package's ML filter auto-activated after dev dependencies finished installing mid-session, masking 37 real findings. Fix: explicitly disable this filter in all scan invocations via --disable-filter. Added test_parity_oracle_filter_config_is_pinned regression guard. Fixed a bug in the golden_findings fixture that read the wrong JSON keys (filename/line_number instead of file/line) for the flat snapshot format. Re-validated Task 3's DI cache: 343 findings both enabled and disabled.
…v timing Root cause: golden snapshot was silently regenerated from 343 to 306 findings when the optional gibberish-detector ML filter auto-activated mid-session. Fixed by pinning --disable-filter detect_secrets.filters.gibberish.should_exclude_secret in all scan invocations. Restored 343/225 baseline matching original Task 0 capture. Also fixed checkov wiring: checkov's Python (pyenv 3.13.1) was silently using a stale PyPI copy of bc-detect-secrets instead of our local fork, making all prior checkov-based DI-cache comparisons meaningless. Installed our fork editable into that environment. Real full-repo (21,339 files) checkov timing, using the exact customer-facing command: checkov --framework seccommand: checkov --framework seccommand: che ------------------------------------------------------------------------------------nd---------------------------------44---------------------------------------------------------------er, zero finding-count regression
Add _could_contain_secret() pre-filter in _process_line_based_plugins(). Lines that do not match the trigger pattern skip code_snippet construction and all ~21 detector regex calls entirely. The trigger pattern is a superset of all detector patterns, verified empirically by the parity oracle (343 findings unchanged, zero regressions). Iteration note: initial version used prefix-only keyword patterns (api_key, db_pass, etc.) and length thresholds of 32-40 for entropy candidates, modeled on assumptions rather than the real detector code. This missed 12 real findings caught by the parity oracle: the actual KeywordDetector DENYLIST matches bare/compound forms (key, pass, _pass, client_key, service_key) not captured by prefix-only patterns, and the real HighEntropyStringsPlugin extraction regex has no minimum length (the entropy (the entropy (the entropy (the entropy (the th(the entropy (the entropy (the entropy (the entropy (the th(treality rather than assumption. Fixed and re-verified against the parity oracle. Feature flag: DETECT_SECRETS_PERF_PREGATE=0 disables the gate.
…ilding it on every line (DETECT_SECRETS_PERF_FILTER_CACHE flag); parity oracle 343 findings unchanged, full suite 1116 passed/1 skipped/6 xfailed
… or too uniform to exceed any threshold (DETECT_SECRETS_PERF_ENTROPY_SC flag)
…, _get_lines_from_file) to avoid eager string formatting;
`PrivateKeyDetector.analyze_line` runs for every line of every scanned
file. The file-size guard was written as a single `and` expression:
if filename not in self._analyzed_files \
and 0 < self.get_file_size(filename) < MAX_FILE_SIZE:
self._analyzed_files.add(filename)
`_analyzed_files` was only populated when the size fell inside the
scannable range. For any file at or above MAX_FILE_SIZE (8 KiB) the
membership check kept failing, so `get_file_size` -> `os.path.getsize`
fired again on every single line: a per-file operation executed
per-line. Cost scaled with (files x lines) rather than (files).
Track the files whose size has already been measured in a separate
`_sized_files` set, so the lookup happens at most once per file
regardless of the result. The subsequent whole-file read is unchanged
and still happens exactly once per file, so multi-line private keys are
detected exactly as before.
Measured on a 21,359-file repository via
`checkov --framework secrets --enable-secret-scan-all-files`:
before: real 2120s user 2014s sys 230s
after: real 1679s user 1638s sys 60s
Wall time -21%; `sys` time (the syscall fingerprint of the redundant
getsize calls) down 3.9x. Findings are identical before and after
(14 findings, same files/lines/checks).
Adds two regression tests:
- `test_get_file_size_is_called_at_most_once_per_file` - asserts the
size lookup runs at most once for a 500-line, >8 KiB file
(previously 500 times).
- `test_multiline_private_key_in_small_file_is_still_detected` -
guards the whole-file read path that multi-line key detection
depends on.
GillSami
force-pushed
the
chore/detect-secrets-perf-optimizations
branch
from
August 11, 2026 13:34
b33ff67 to
f1c07c7
Compare
Saarett
reviewed
Aug 11, 2026
GillSami
force-pushed
the
chore/detect-secrets-perf-optimizations
branch
from
August 11, 2026 14:44
96e36bb to
1a41a3f
Compare
Saarett
approved these changes
Aug 12, 2026
omriyoffe-panw
approved these changes
Aug 12, 2026
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Reduces full production-scale secrets scan time by ~3.6x (≈40 minutes → 11m21s) via six targeted, low-risk optimizations to the scanning hot path, with zero correctness regressions verified at every step.
Performance Result
Measured with the exact production command against a 21,339-file synthetic monorepo:
Net: ~3.6x faster, with identical scan findings before and after (12/12 on the synthetic repo, 343/343 on the correctness fixture) — the speedup comes from doing measurably less CPU work (confirmed via
user/sys/%CPUprofiling, not from OS or disk caching effects).What Changed (one sentence each)
c6509ad) — caches the reflection-based dependency-injection plan per callable instead of recomputing it withinspecton every single invocation.72ba48f) — skips full plugin analysis on lines that structurally cannot contain a secret, using a cheap superset check before the expensive per-plugin work.3503f9f) — caches the result ofget_filters_with_parameter()instead of rebuilding the filtered plugin list from scratch on every line.80971af) — returns early fromcalculate_shannon_entropy()for strings that are too short or insufficiently varied to ever cross any configured entropy threshold.eaffec1) — switches hot-path log calls to lazy%-style formatting so log messages aren't string-built when the log level would discard them anyway.0bce612) — looks up each file's size once instead of once per line, matching the scope of upstream PR perf(private_key): look up file size once per file instead of per line #293.(Two additional commits,
3168e33and0f08efe, are correctness-only fixes that pin filter configuration and restore/lock in the 343-finding parity baseline used to validate every change above — no performance impact themselves, but they were the safety net that caught a real regression during development, see below.)Correctness Verification
secrets-examples/, 225 files with known seeded secrets): 343/343 findings (detect-secrets CLI) and 204/204 failed checks (checkov) — identical before and after every commit, verified both by an automated parity-oracle test and an independent manual diff outside that test harness.usertime), not just faster I/O.Feature Flags
Four of the six changes are individually toggleable via environment variables (all default enabled) for easy A/B verification or rollback without a code change:
DETECT_SECRETS_PERF_DI_CACHEDETECT_SECRETS_PERF_PREGATEDETECT_SECRETS_PERF_FILTER_CACHEDETECT_SECRETS_PERF_ENTROPY_SCThe lazy-logging and PrivateKeyDetector changes are unconditional — they're behavior-preserving by construction (same log content, same detection outcome) and don't warrant a flag.
Risk Assessment
All changes are additive caching/short-circuit optimizations with no change to detection semantics — none alter what is matched, only how much redundant work is done to get there. The riskiest of the six (the pre-gate, since it's the only one capable of skipping data before any plugin sees it) is the one with the most test coverage: dedicated unit tests asserting it's a strict superset of every plugin's trigger patterns, plus the parity oracle as a continuous regression guard.