Repository navigation
quality: detect programmatic option manipulation and similar elaborator backdoors - #281
Merged
Merged
Conversation
quality.py's set_option ban was textual only; PR #278 bypassed it with `withOptions (fun o => o.set `maxRecDepth 100000)` inside a custom elaborator. Two complementary layers close the gap: - Textual: option-API tokens (withOptions, modifyOptions, KVMap/Options/ Option setters, withRecDepth, withCurrHeartbeats, ...) and gated option names (maxRecDepth, maxHeartbeats, maxSynthPendingDepth, linter.*) are banned in comment-stripped content. Validated zero-FP over all current pool sources. - Environment audit (_check_option_backdoors): a standalone `lake env lean --run` script — same pattern as the exposition extractor, extensions disabled so project notation cannot interfere — walks every declaration compiled into LeanPool.* modules, including elab_rules auxiliaries and elaborator-generated declarations the textual parser cannot enumerate. It flags references to option-manipulating constants, embedded gated option-name string literals (Name literals compile to string pieces), direct references to axiom-injecting constants (sorryAx, ofReduceBool, ...), and axioms declared inside pool modules (native_decide compiles to a generated per-theorem axiom on this toolchain; addDecl of an axiomDecl is the same hole). Fails closed via a completion marker. Validated: zero findings over the whole built pool (~10 s), and a probe module replicating the PR #278 elaborator plus obfuscated variants is fully detected. Also bans maxRecDepth in lakefile [leanOptions] alongside the existing heartbeat gate, and documents the new gates in CODE_QUALITY.md. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
|
LLM review skipped: PR touches no |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
quality.pybansset_optiontextually, but PR #278 demonstrated a bypass: a custom elaborator callingis semantically
set_option maxRecDepth 100000, invisible to the textual gate. Requested by the maintainer; this PR closes that hole with two complementary layers plus coverage for adjacent backdoor classes discovered along the way.Layer 1 — textual (fast, runs even with
--skip-lean-axioms)On comment/string-stripped content:
programmatic option manipulation is forbidden— option-API tokens:withOptions,modifyOptions,MonadWithOptions,withRecDepth,withCurrHeartbeats,elabSetOption,setOptionFromString,modifyScope,KVMap.set*/insert*/erase*,Options.set*,Option.set(IfNotSet).gated option name in code is forbidden— any code occurrence ofmaxRecDepth/maxHeartbeats/maxSynthPendingDepth/linter.*. This catches receiver-variable spellings (options.set ``maxRecDepth``) the API list can't. Word boundaries keepmaxRecDepth_boundand comment mentions legal.maxRecDepthjoins the existing heartbeat ban in[leanOptions].Layer 2 — Lean-native environment audit (
_check_option_backdoors)A standalone script run via
lake env lean --run(same pattern asscripts/exposition/Extract.lean;importModuleswith extensions disabled, so project notation cannot interfere — importing the pool normally lets e.g. MRiscX's syntax break audit parsing). It walks every declaration compiled into aLeanPool.*module — includingelab_rulesauxiliaries and elaborator-generated declarations that the textual declaration parser cannot enumerate (so#print axiomsnever sees them) — and flags any that:Nameliterals compile to string pieces, so`maxRecDepth,Name.mkStr1 "maxRecDepth", andName.mkStr2 "linter" "…"all surface;sorryAx,Lean.ofReduceBool,Lean.ofReduceNat,Lean.trustCompiler);native_decidecompiles to a generated per-theorem axiom (noofReduceBoolreference anymore), and an elaborator callingaddDecl (Declaration.axiomDecl …)is the same hole. Only the axiom kind is rejected: generated._unsafe_rec/ hygienic opaques /derivinghelpers are sanctioned compiler output and carry kernel-checked values.The audit fails closed: if the script stops compiling on a future toolchain, a missing completion marker is itself an error.
Validation
SetTheory(reflection tactic, registered simp attrs) andMRiscX(Hoare-logic elaborators) — the most metaprogramming-heavy projects — pass all arms.Lrat.leanline, and a probe module replicating its elaborator is caught in the env audit viaLean.MonadWithOptions.withOptions+Lean.Options.set+ the"maxRecDepth"literal.Name.mkSimple ("maxRec" ++ "Depth")) — invisible to any textual scan — is still caught by itsLean.KVMap.setNatreference;MonadRecDepth.withRecDepth 0(counter reset) andnative_decide's generated axiom are caught too.ruff check/formatclean.Residual risk (documented honestly)
An ephemeral backdoor that never persists a declaration (e.g.
run_cmddrivingelabCommandunderwithOptions) is invisible to the env audit — but its source must still spell a manipulator token or gated name, which the textual layer bans; assembling those from characters at runtime while also referencing no flagged constant would additionally have to dodgeevalConst-style reflection (unusable here sinceunsafeis banned). The layers make the known channels fail loudly rather than prove absence; LLM review remains the judgment backstop.🤖 Generated with Claude Code