diff --git a/.github/workflows/docs-quality.yml b/.github/workflows/docs-quality.yml index 3a7ffd97..781b2d55 100644 --- a/.github/workflows/docs-quality.yml +++ b/.github/workflows/docs-quality.yml @@ -24,8 +24,9 @@ jobs: run: uv run python tools/check_doc_images_policy.py - name: Check spelling run: uv run codespell ./docs + # pymarkdown has no configuration key for path exclusions, so .venv must be excluded on the command line. - name: Lint markdown - run: uv run pymarkdown scan -r ./docs + run: uv run pymarkdown scan -r --respect-gitignore . ci-passed: name: Docs CI passed diff --git a/CHANGELOG.md b/CHANGELOG.md index 070f6245..44a3728d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,21 +1,21 @@ -## Breaking changes +# Change Log -- Removed the nominal AST hierarchy in `renaissance.integrations.types`, including `ast_type`, `KIND_MAP`, and class-based kind finder APIs. -- Use `NodeProtocol`, `SemanticKind`, `PatternKind`, `parser_kind`, and parser-local predicates instead. +## Breaking changes -Plan for next sprints: +* Removed the nominal AST hierarchy in `renaissance.integrations.types`, including `ast_type`, `KIND_MAP`, and class-based kind finder APIs. +* Use `NodeProtocol`, `SemanticKind`, `PatternKind`, `parser_kind`, and parser-local predicates instead. +## Plan for next sprints 11-05-2026 * [X] use type hierarchy to find type concisely instead of regexp * [X] use hypothesis instead of parameterized test to get better coverage * [X] convert more complex cases of TAUT test case and reviewed the conversion by Harry -* [X] restructure with root namespace so that it can be packaged +* [X] restructure with root namespace so that it can be packaged * [X] apply ASTProtocol to Python and ~~Clang Node~~ * [X] add ADR and set up ADR discussion process * [X] update test to pytest using python refactoring * [X] created a package with callable cli * [X] expand matcher and other utils to use lst nodes * [X] convert simple case of TAUT test case and reviewed the conversion by Harry - diff --git a/CLA.md b/CLA.md index 10bf5022..e22ef477 100644 --- a/CLA.md +++ b/CLA.md @@ -6,65 +6,65 @@ By submitting any Contribution to this Project, you agree to the following terms 1. Definitions - "Project" means the software project maintained by the Maintainer. + "Project" means the software project maintained by the Maintainer. - "Maintainer" means the copyright holder(s) and any entity designated by them to manage the Project. + "Maintainer" means the copyright holder(s) and any entity designated by them to manage the Project. - "Contribution" means any source code, object code, documentation, configuration files, test material, designs, - bug fixes, enhancements, or other material intentionally submitted to the Project. + "Contribution" means any source code, object code, documentation, configuration files, test material, designs, + bug fixes, enhancements, or other material intentionally submitted to the Project. 2. Authority - You represent and warrant that: + You represent and warrant that: - (a) you are the sole author of the Contribution, or otherwise have sufficient rights to grant the permissions described in this Agreement; + (a) you are the sole author of the Contribution, or otherwise have sufficient rights to grant the permissions described in this Agreement; - (b) the Contribution does not knowingly infringe any third-party intellectual property rights; and + (b) the Contribution does not knowingly infringe any third-party intellectual property rights; and - (c) you are legally entitled to enter into this Agreement. + (c) you are legally entitled to enter into this Agreement. 3. Contribution License - You agree that all Contributions are submitted under either: - - the MIT License; or - - the BSD 3-Clause License. + You agree that all Contributions are submitted under either: + - the MIT License; or + - the BSD 3-Clause License. - By submitting a Contribution, you grant the Maintainer a perpetual, worldwide, non-exclusive, irrevocable, - royalty-free license to use, reproduce, modify, prepare derivative works of, publicly display, publicly perform, - sublicense, distribute, and otherwise exploit the Contribution. + By submitting a Contribution, you grant the Maintainer a perpetual, worldwide, non-exclusive, irrevocable, + royalty-free license to use, reproduce, modify, prepare derivative works of, publicly display, publicly perform, + sublicense, distribute, and otherwise exploit the Contribution. 4. Right to Re-License - You expressly agree that the Maintainer may: + You expressly agree that the Maintainer may: - (a) distribute the Contribution as part of the Project under the Eclipse Public License 2.0 (EPL-2.0); + (a) distribute the Contribution as part of the Project under the Eclipse Public License 2.0 (EPL-2.0); - (b) distribute the Contribution under any future version of the EPL; + (b) distribute the Contribution under any future version of the EPL; - (c) distribute the Contribution under any other open source license; and + (c) distribute the Contribution under any other open source license; and - (d) distribute, license, sublicense, or otherwise exploit the Contribution under commercial, proprietary, - closed-source, or other licensing terms. + (d) distribute, license, sublicense, or otherwise exploit the Contribution under commercial, proprietary, + closed-source, or other licensing terms. - No additional permission from the Contributor shall be required for such licensing activities. + No additional permission from the Contributor shall be required for such licensing activities. 5. Retention of Copyright - Except for the rights granted under this Agreement, ownership of copyright in the Contribution remains with the Contributor. + Except for the rights granted under this Agreement, ownership of copyright in the Contribution remains with the Contributor. 6. No Obligation - Nothing in this Agreement obligates the Maintainer to use, distribute, or accept any Contribution. + Nothing in this Agreement obligates the Maintainer to use, distribute, or accept any Contribution. 7. Disclaimer - The Contribution is provided "AS IS", without warranties or conditions of any kind, express or implied, - including without limitation any warranties of merchantability, fitness for a particular purpose, title, - or non-infringement. + The Contribution is provided "AS IS", without warranties or conditions of any kind, express or implied, + including without limitation any warranties of merchantability, fitness for a particular purpose, title, + or non-infringement. 8. Governing Law - This Agreement shall be governed by the substantive laws of the Netherlands. + This Agreement shall be governed by the substantive laws of the Netherlands. - By submitting a Contribution to the Project, you acknowledge that you have read and - agree to the terms of this Agreement. + By submitting a Contribution to the Project, you acknowledge that you have read and + agree to the terms of this Agreement. diff --git a/README.md b/README.md index 3550f538..9adf21b1 100644 --- a/README.md +++ b/README.md @@ -37,78 +37,117 @@ Contributions that are subject to additional restrictions or incompatible licens This project is experimental in nature and aims to explore various concepts and techniques to apply renaissance pattern matching -in a generic way using multiple abstract syntax trees. +in a generic way using multiple abstract syntax trees. + +## Prerequisites + +The Python dependencies, including the pinned `libclang` library used by `ClangASTNode`, are installed by `uv sync`. + +The `clang` compiler driver is a separate, external requirement: `ClangJsonASTNode` runs it as a subprocess to obtain a JSON AST dump. +It should have the same LLVM major version as the bundled library. Print the version to install: + +```powershell +uv run python -c "from importlib.metadata import version; print('.'.join(version('libclang-ng').split('.')[:3]))" +``` + +Then install that version. `winget` requires the exact version, while the other installers take the major version: + +```powershell +winget install LLVM.LLVM --version 22.1.4 +``` + +```bash +# Debian, Ubuntu, or WSL +wget https://apt.llvm.org/llvm.sh && chmod +x llvm.sh && sudo ./llvm.sh 22 +``` -## Setup for WSL ```bash -sudo apt-get install -y build-essential clang +# macOS +brew install llvm@22 ``` +Open a new terminal and check that `clang --version` reports the expected version. +On Windows the installer does not add LLVM to `PATH`, so add `C:\Program Files\LLVM\bin` to it yourself. +See [Clang prerequisites](docs/developer/modules/parser-and-ast.md#clang-prerequisites) for selecting a specific driver +when it is not on `PATH`. The code for the experiments is located in the [src](./src) folder. -# Description +## Description + This project is a generic approach to refactor code bases with a generic AST structure. It uses `TNO Renaissance` pattern matching. Currently, clang native and clang python bindings are supported. -# How to add a different binding +## How to add a different binding + You'll need to implement a concrete class for syntax_tree.ASTNode. Follow the implementations of `ClangASTNode` and `ClangJsonASTNode` as an example. If the concrete AST has a different language then also a `PatternFactory` must be added. See `CPatternFactory` for inspiration. ## Installation Procedure -To install the necessary dependencies, follow these steps: -1. **Run the Installation Script** - - Navigate to the project directory. - - Execute the `install.bat` script by double-clicking it or running the following command in the terminal: - ```sh - ./install.bat - ``` +The project is managed with [uv](https://docs.astral.sh/uv/). From the project directory: + +```sh +uv sync --group dev +``` + +This creates the virtual environment, installs the interpreter pinned by `requires-python`, and installs the development tools. ## Configuration and Verification -1. **Configure the Environment** - - Open Visual Studio Code (VSCode). - - Ensure that the Python extension is installed. - - Open the project folder in VSCode. - - alternatively in shell goto /python folder and - ```sh - code . - ``` +Two steps are needed. +By following these steps, you will have configured, installed, and verified the installation for the project. + +### Configure the Environment + +- Open Visual Studio Code (VSCode). +- Ensure that the Python extension is installed. +- Open the project folder in VSCode, or run the following command in the project directory: + + ```sh + code . + ``` -2. **Verify the Installation** - - Open the integrated terminal in VSCode. - - Run the following command to execute the tests: - ```sh - python -m unittest discover - ``` - - Check the output to ensure all tests pass successfully. +- Select the interpreter from the `.venv` directory. -By following these steps, you will have installed and verified the setup for the project. +### Verify the Installation +- Open the integrated terminal in VSCode. +- Run the following command to execute the tests: + + ```sh + uv run pytest + ``` + +- Check the output to ensure all tests pass successfully. + Failing `clang_json` tests indicate that the `clang` driver is missing from `PATH`; see [Prerequisites](#prerequisites). ## TODO An incomplete list of todo's: -* The get_properties methods of both `ClangASTNode` and `ClangJsonASTNode` are not complete yet. This might cause mismatches in the `Match_Finder` -* C++ constructs have not been tested yet -* An example of how to use includes in a `Pattern` must be added -* Tests need to be added for macro handling -* The methods `get_references` and `referred_by` must be added to `ASTNode` and implemented in the concrete classes -* Test cases for multiple match patterns need to be added. Currently, there is only one working case in the examples -* Comments in Clang appear incorrectly in the `ASTShower`. This seems to be a Clang issue, which is surprising +- The get_properties methods of both `ClangASTNode` and `ClangJsonASTNode` are not complete yet. This might cause mismatches in the `Match_Finder` +- C++ constructs have not been tested yet +- An example of how to use includes in a `Pattern` must be added +- Tests need to be added for macro handling +- The methods `get_references` and `referred_by` must be added to `ASTNode` and implemented in the concrete classes +- Test cases for multiple match patterns need to be added. Currently, there is only one working case in the examples +- Comments in Clang appear incorrectly in the `ASTShower`. This seems to be a Clang issue, which is surprising ## Usage +```bash cli +``` ### Inspect Inspect the AST of a source file. + ```bash cli inspect features/targets/demo.py pass ``` + it will show ast of demo.py and focus on 'pass' statements diff --git a/docs/developer/modules/parser-and-ast.md b/docs/developer/modules/parser-and-ast.md index 2e81b4a5..809dc514 100644 --- a/docs/developer/modules/parser-and-ast.md +++ b/docs/developer/modules/parser-and-ast.md @@ -177,6 +177,111 @@ Use the existing backend tests as the primary contract examples. Cross-backend tests establish the `NodeProtocol` matching contract; rewrite, trivia, and reparse behavior remain tests and responsibilities of the selected adapter. +## Clang prerequisites + +The repository has two independent Clang backends, and they obtain Clang in +different ways: + +| Backend | Needs | Obtained from | +| --- | --- | --- | +| `ClangASTNode` | the `libclang` library | the pinned `libclang-ng` wheel, installed by `uv sync` | +| `ClangJsonASTNode` | the `clang` or `clang++` driver | an LLVM installation on the machine | + +`ClangASTNode` works without extra setup: `clang_ast_node.py` points libclang +at the bundled native library, so its version is pinned by `pyproject.toml` +and is identical on every machine. + +`ClangJsonASTNode` runs the compiler driver as a subprocess, because +`-Xclang -ast-dump=json` is a frontend action that the library API does not +expose. That driver is not part of the Python dependencies and must be +installed separately. Without it, every `clang_json` test fails. + +### Determine the required version + +Both backends are compared against the same expectations in the cross-backend +tests, so the driver should have the same LLVM major version as the bundled +library. The bundled version is the one pinned in `pyproject.toml`. Print the +LLVM version it corresponds to: + +```powershell +uv run python -c "from importlib.metadata import version; print('.'.join(version('libclang-ng').split('.')[:3]))" +``` + +The `libclang-ng` version has four components, such as `22.1.4.2`. The first +three are the LLVM version, `22.1.4` in this example, and the fourth is the +wheel build number. + +### Install that version + +Windows, where the version must be given exactly as published. A major version +alone is rejected with `No version found matching: 22`, and +`winget show LLVM.LLVM --versions` lists the accepted values: + +```powershell +winget install LLVM.LLVM --version 22.1.4 +``` + +Debian, Ubuntu, or WSL. The distribution package is usually older than the +pinned version, so install from the LLVM apt repository, which takes the major +version and installs version-suffixed binaries such as `clang-22`: + +```bash +wget https://apt.llvm.org/llvm.sh +chmod +x llvm.sh +sudo ./llvm.sh 22 +``` + +macOS, using the versioned formula: + +```bash +brew install llvm@22 +``` + +Open a new terminal afterwards, so that the updated `PATH` is picked up, and +check that the driver is reachable and reports the expected version: + +```powershell +clang --version +``` + +On Windows the installer does not add LLVM to `PATH`, so this reports that +`clang` is not recognized even though the install succeeded. Either add +`C:\Program Files\LLVM\bin` to `PATH`, or point `RENAISSANCE_CLANG` at the +driver as described below. + +### Select a specific driver + +The driver is resolved in this order: + +1. the first entry of `extra_args`, when it names a clang executable +2. the path passed to `ClangJsonASTNode.set_compiler_path()` +3. the `RENAISSANCE_CLANG` environment variable +4. `clang`, or `clang++` for C++, found on `PATH` + +When none of these resolves to an executable driver, parsing raises +`FileNotFoundError` describing these options, rather than failing inside the +subprocess call. + +Set `RENAISSANCE_CLANG` when `PATH` does not already point at the intended +driver. This is the normal case on Windows, where the installer leaves `PATH` +alone, and on Linux, where the LLVM apt repository installs `clang-22` while +`clang` remains the distribution version: + +```bash +export RENAISSANCE_CLANG=/usr/bin/clang++-22 +``` + +```powershell +$env:RENAISSANCE_CLANG = "C:/Program Files/LLVM/bin/clang++.exe" +& $env:RENAISSANCE_CLANG --version +``` + +The major version of the resolved driver is compared with the pinned +`libclang-ng` version, and a mismatch raises a warning such as +`clang++ is LLVM 18 while libclang is pinned to LLVM 22; the ASTs may differ.` +Only the major version is compared, so any `22.x` driver satisfies a `22.1.4` +pin. No warning means the versions agree. + ## Validation Run the repository checks with the parser's optional dependencies installed: @@ -189,4 +294,5 @@ uv run pyright ./src ./test ./tools ./features uv run mkdocs build --strict ``` -For Clang integrations, make LLVM available on `PATH` before running tests. +The Clang backends additionally require the setup described in +[Clang prerequisites](#clang-prerequisites). diff --git a/features/targets/README.md b/features/targets/README.md index 2b39a96f..fceab688 100644 --- a/features/targets/README.md +++ b/features/targets/README.md @@ -1,18 +1,21 @@ -# Most useful commands: +# Most useful commands ## gcc -gcc -fdump-tree-all-raw-lineno -fdump-rtl-all-raw-lineno -o main.exe main.c +```sh +gcc -fdump-tree-all-raw-lineno -fdump-rtl-all-raw-lineno -o main.exe main.c +``` ## clang ### ast dump `clang -Xclang -ast-dump -fsyntax-only main.c > ast-dump.ast` -or +or `clang -Xclang -ast-dump -fsyntax-only main.c > ast-dump.ast` + ### preprocessing dump -`pp-trace main.c > pptrace.ast` +`pp-trace main.c > pptrace.ast` -contains all preprocessing directives and all usages. \ No newline at end of file +contains all preprocessing directives and all usages. diff --git a/src/renaissance/common/rewriter.py b/src/renaissance/common/rewriter.py index ca21b3bc..bf1a60e5 100644 --- a/src/renaissance/common/rewriter.py +++ b/src/renaissance/common/rewriter.py @@ -36,9 +36,6 @@ def replace(self, start: int, end: int, new_content: bytes) -> None: end (int): The ending index of the content to be replaced. new_content (bytes): The new content to insert in place of the old content. - Returns: - None - """ for r in self.__rewrites: # if r partially overlaps with start and end then append the new content to the existing replacement diff --git a/src/renaissance/integrations/clang/clang_json_ast_node.py b/src/renaissance/integrations/clang/clang_json_ast_node.py index 0e72aa34..c1cc8cb3 100644 --- a/src/renaissance/integrations/clang/clang_json_ast_node.py +++ b/src/renaissance/integrations/clang/clang_json_ast_node.py @@ -3,12 +3,16 @@ # create a class that inherits syntax tree ASTNode import json +import os import re +import shutil import subprocess import sys import tempfile +import warnings from collections.abc import Sequence from functools import cache +from importlib.metadata import version from pathlib import Path from typing import Any, Self, override @@ -40,6 +44,40 @@ IRRELEVANT_PROPS = {"macro_expansion", "start_point", "end_point", "source_code", "location", "type"} IRRELEVANT_NODE_KINDS = {"comment", "Comment", "MacroDefinition", "MACRO_DEFINITION", "FullComment"} VERBOSE = False +COMPILER_ENV_VAR = "RENAISSANCE_CLANG" +# The libclang bindings used by ClangASTNode are pinned by this distribution, so its version is the reference. +LIBCLANG_DISTRIBUTION = "libclang-ng" +COMPILER_HINT = ( + f"Install LLVM and put clang (clang++ for C++) on PATH, set the {COMPILER_ENV_VAR} environment variable, " + f"or call ClangJsonASTNode.set_compiler_path()." +) + + +@cache +def verify_compiler(compiler: str) -> None: + """AI: Check the clang driver runs and warn when its major version differs from the pinned libclang version.""" + arguments = [compiler, "--version"] + try: + result = subprocess.run(arguments, capture_output=True, text=True, check=False) # noqa: S603 (the compiler comes from the configuration, not from parsed input) + except OSError as error: + message = f"The clang driver {compiler} cannot be executed. {COMPILER_HINT}" + raise FileNotFoundError(message) from error + found = re.search(r"clang version (\d+)", result.stdout) + pinned = version(LIBCLANG_DISTRIBUTION).split(".")[0] + if found and found.group(1) != pinned: + message = f"{compiler} is LLVM {found.group(1)} while libclang is pinned to LLVM {pinned}; the ASTs may differ." + warnings.warn(message, stacklevel=2) + + +def resolve_compiler(file_path: Path) -> str: + """AI: Return the clang driver to invoke: the pinned path, the environment override, or the one found on PATH.""" + name = "clang++" if file_path.suffix == ".cpp" else "clang" + compiler = ClangJsonASTNode.compiler_path or os.environ.get(COMPILER_ENV_VAR) or shutil.which(name) + if compiler is None: + message = f"No clang driver found. {COMPILER_HINT}" + raise FileNotFoundError(message) + verify_compiler(compiler) + return compiler class ClangJsonASTReference: @@ -86,6 +124,12 @@ class ClangJsonASTNode(ASTNode[dict[str, Any], ClangJsonTranslationUnit]): "-ast-dump=json", "-fsyntax-only", ] + compiler_path: str | None = None + + @staticmethod + def set_compiler_path(path: str | Path | None) -> None: + """AI: Pin the clang driver used for the JSON AST dump; pass None to fall back to the environment.""" + ClangJsonASTNode.compiler_path = str(path) if path else None def __init__( self, @@ -204,8 +248,7 @@ def load( extra_args = extra_args[1:] # add clang compiler if it is not in the arguments if len(extra_args) == 0 or "clang" not in extra_args[0]: - clang = "clang++" if file_path.suffix == ".cpp" else "clang" - extra_args = [clang, *extra_args] + extra_args = [resolve_compiler(file_path), *extra_args] command = [*extra_args, *ClangJsonASTNode.parse_args] if code: diff --git a/src/renaissance/integrations/tree_sitter/README.md b/src/renaissance/integrations/tree_sitter/README.md index 176474ef..bcfddb70 100644 --- a/src/renaissance/integrations/tree_sitter/README.md +++ b/src/renaissance/integrations/tree_sitter/README.md @@ -1,6 +1,8 @@ # LST Toolkit -This toolkit provides a parser-independent Language-Specific Tree (LST) representation with pattern matching, symbol binding, and extraction capabilities. It supports Tree-sitter grammars and offers a flexible interface for analyzing Python, Java, and C++ code. +This toolkit provides a parser-independent Language-Specific Tree (LST) representation with pattern matching, symbol binding, +and extraction capabilities. +It supports Tree-sitter grammars and offers a flexible interface for analyzing Python, Java, and C++ code. --- @@ -31,6 +33,7 @@ pip install -e . ```bash pip install tree-sitter ``` + with a dash and not an underscore 1. Run the setup script to clone grammars and build the shared library: @@ -40,6 +43,7 @@ python setup_grammars.py ``` This will: + - Clone Tree-sitter grammars for Python, Java, and C++ - Build `build/my-languages.so` for use in adapters @@ -109,36 +113,30 @@ results = extractor.run(source_code) This toolkit is part of Renaissance.Py and is licensed under the Eclipse Public License 2.0 (EPL-2.0), as described in [LICENSE](../../../../LICENSE). - ## 🔌 Clang Integration for C++ For advanced C++ analysis (with preprocessing and include resolution), this toolkit supports [libclang](https://clang.llvm.org/). ### 🛠 Install Dependencies -```bash -# On Ubuntu/Debian -sudo apt install libclang-dev - -# Python bindings -pip install clang -``` +No extra installation is needed. The `libclang` library is provided by the pinned `libclang-ng` dependency, +which `uv sync` installs into the virtual environment. ### 🔧 Usage Use `ClangAdapter` instead of `TreeSitterAdapter`: ```python -from core.clang_adapter import ClangAdapter +from renaissance.integrations.clang.clang_adapter import ClangAdapter adapter = ClangAdapter() lst = adapter.parse("examples/cpp_example.cpp") -for node in lst.traverse(): - print(node) +print(lst.root.kind_key) ``` The `ClangAdapter` provides: + - Full include resolution - Macro expansion - AST node types like `FUNCTION_DECL`, `CALL_EXPR`, etc. diff --git a/src/renaissance/syntax_tree/ast_node.py b/src/renaissance/syntax_tree/ast_node.py index 85fdf345..4164a1c9 100644 --- a/src/renaissance/syntax_tree/ast_node.py +++ b/src/renaissance/syntax_tree/ast_node.py @@ -243,9 +243,6 @@ def accept(self, function: Callable[[Self], VisitorResult]) -> None: Args: function (Callable[[Self], VisitorResult]): A function that takes an ASTNode as an argument and returns a VisitorResult. - Returns: - None - """ if function(self) == VisitorResult.CONTINUE: for child in self.children: diff --git a/src/renaissance/syntax_tree/ast_rewriter.py b/src/renaissance/syntax_tree/ast_rewriter.py index bc889c82..2008dace 100644 --- a/src/renaissance/syntax_tree/ast_rewriter.py +++ b/src/renaissance/syntax_tree/ast_rewriter.py @@ -363,9 +363,6 @@ def __remove( include_whitespace (bool, optional): Whether to include surrounding whitespace in the removal. Defaults to False. include_comments (bool, optional): Whether to include surrounding comments in the removal. Defaults to False. - Returns: - None - """ if not nodes: return diff --git a/src/renaissance/syntax_tree/batch_ast_processor.py b/src/renaissance/syntax_tree/batch_ast_processor.py index 7142d055..7348aec8 100644 --- a/src/renaissance/syntax_tree/batch_ast_processor.py +++ b/src/renaissance/syntax_tree/batch_ast_processor.py @@ -43,9 +43,6 @@ def once( actions (Action | Sequence[Action]): The action or sequence of actions to apply to each item in the iterable. file_filter (Optional[str | re.Pattern], optional): A filter to apply to file names. Defaults to None. - Returns: - bool: True if processing was successful, False otherwise. - """ iterable = iterable() if callable(iterable) else iterable self.__process(iterable, actions, self.in_memory, file_filter) @@ -67,9 +64,6 @@ def repeat( file_filter (Optional[str | re.Pattern], optional): A filter to apply to the files being processed. Defaults to None. max_repeat (int, optional): The maximum number of times to repeat the processing. Defaults to 5. - Returns: - bool: True if the processing still yields changes, False otherwise. - """ self.__process(iterable_provider(), actions, self.in_memory, file_filter, max_repeat) diff --git a/test/search_strategies/prompt.md b/test/search_strategies/prompt.md index de607686..2a4b146e 100644 --- a/test/search_strategies/prompt.md +++ b/test/search_strategies/prompt.md @@ -1,47 +1,52 @@ +# PROMPT + You are to generate Python code (Python 3.14 only) that provides Hypothesis strategies to generate: - 1) (type_expr: ast.expr, value_gen: SearchStrategy[ast.expr]) pairs, where value_gen lazily produces an ast.expr value matching type_expr. - 2) an ast.arguments generator for function definitions using those (type_expr, value_gen) pairs (optional but desirable). + +1) (type_expr: ast.expr, value_gen: SearchStrategy[ast.expr]) pairs, where value_gen lazily produces an ast.expr value matching type_expr. +2) an ast.arguments generator for function definitions using those (type_expr, value_gen) pairs (optional but desirable). STRICT CONSTRAINTS / CONVENTIONS + A) Python 3.14-only codebase: - - Do NOT use: from __future__ import annotations - - Do NOT use typing.List / typing.Optional. Use built-in generics: list[T], dict[K,V], tuple[...] and union types: X | None. - - Type hints should use `list[...]`, `dict[...]`, `tuple[...]`, `ast.expr | None`. + - Do NOT use: from __future__ import annotations + - Do NOT use typing.List / typing.Optional. Use built-in generics: list[T], dict[K,V], tuple[...] and union types: X | None. + - Type hints should use `list[...]`, `dict[...]`, `tuple[...]`, `ast.expr | None`. B) Hypothesis typing: - - Every @composite strategy must type its draw parameter as DrawFn. - - The module must import DrawFn: `from hypothesis.strategies import DrawFn`. + - Every @composite strategy must type its draw parameter as DrawFn. + - The module must import DrawFn: `from hypothesis.strategies import DrawFn`. C) Naming / structure: - - Provide a uniform generator family with these PUBLIC functions only: + - Provide a uniform generator family with these PUBLIC functions only: gen_type, gen_base, gen_list, gen_dict, gen_union, gen_tuple Do NOT add separate list_type_and_value / dict_type_and_value / union_type_and_value / tuple_type_and_value wrappers. Focused tests should call gen_list/gen_union/gen_dict/gen_tuple directly. - - Do NOT pass a SearchStrategy “child” parameter around. The generators must call gen_type(depth-1) internally. + - Do NOT pass a SearchStrategy “child” parameter around. The generators must call gen_type(depth-1) internally. D) Builders: - - All AST node construction helpers must be PRIVATE and start with `_build_`. + - All AST node construction helpers must be PRIVATE and start with `_build_`. Example: use `_build_arg`, NOT `_make_arg`. - - Group all `_build_*` helpers together in one section. + - Group all `_build_*` helpers together in one section. E) Size policy: - - Provide one function: `max_len(depth: int) -> int` that returns `3 * depth`. - - NEVER inline `3 * depth` anywhere; always call max_len(depth). - - Lists/tuples/dicts must allow empty values (min size = 0). - - Unions must have minimum arms 2. + - Provide one function: `max_len(depth: int) -> int` that returns `3 * depth`. + - NEVER inline `3 * depth` anywhere; always call max_len(depth). + - Lists/tuples/dicts must allow empty values (min size = 0). + - Unions must have minimum arms 2. F) Union special-cases (must live inside gen_union): - - If depth <= 1: max number of union arms is 2 (so unions are exactly 2 arms at these depths). - - Else: max number of union arms is max_len(depth). - - Additionally: for depth <= 2, union arms must be base types only (i.e., generated from gen_base at “depth=0”). + - If depth <= 1: max number of union arms is 2 (so unions are exactly 2 arms at these depths). + - Else: max number of union arms is max_len(depth). + - Additionally: for depth <= 2, union arms must be base types only (i.e., generated from gen_base at “depth=0”). G) Performance / energy: - - When building the BitOr chain for union type expressions, avoid list slicing copies; use `islice` from itertools where appropriate. + - When building the BitOr chain for union type expressions, avoid list slicing copies; use `islice` from itertools where appropriate. H) Dict typing correctness: - - Use this builder exactly (or functionally identical): + - Use this builder exactly (or functionally identical): def _build_dict(keys: list[ast.expr], values: list[ast.expr]) -> ast.Dict: return ast.Dict(keys=list(keys), values=values) Rationale: ast.Dict.keys accepts list[expr | None], list is invariant; we copy keys to satisfy typing. I) Base types: - - Must include these base type names: bool, int, str, float, bytes, NoneType. - - NoneType must generate the instance `None` (as an ast.Constant(value=None)). - - Base type_expr must be ast.Name(id=..., ctx=Load()) using those names. (We only need AST validity; runtime execution is not required.) + - Must include these base type names: bool, int, str, float, bytes, NoneType. + - NoneType must generate the instance `None` (as an ast.Constant(value=None)). + - Base type_expr must be ast.Name(id=..., ctx=Load()) using those names. (We only need AST validity; runtime execution is not required.) REQUIRED OUTPUT API + 1) max_len(depth: int) -> int 2) gen_base(draw: DrawFn, depth: int) -> tuple[ast.expr, SearchStrategy[ast.expr]] 3) gen_list(draw: DrawFn, depth: int) -> tuple[ast.expr, SearchStrategy[ast.expr]] @@ -49,50 +54,64 @@ REQUIRED OUTPUT API 5) gen_union(draw: DrawFn, depth: int) -> tuple[ast.expr, SearchStrategy[ast.expr]] 6) gen_tuple(draw: DrawFn, depth: int) -> tuple[ast.expr, SearchStrategy[ast.expr]] 7) gen_type(draw: DrawFn, depth: int) -> tuple[ast.expr, SearchStrategy[ast.expr]] - - gen_type must dispatch among ALL five types: base, list, dict, union, tuple. - - If depth <= 0, gen_type must return a base pair (by drawing from gen_base(0) or equivalent). - - Otherwise, gen_type must draw from one_of(gen_base(depth), gen_list(depth), gen_dict(depth), gen_union(depth), gen_tuple(depth)). + - gen_type must dispatch among ALL five types: base, list, dict, union, tuple. + - If depth <= 0, gen_type must return a base pair (by drawing from gen_base(0) or equivalent). + - Otherwise, gen_type must draw from one_of(gen_base(depth), gen_list(depth), gen_dict(depth), gen_union(depth), gen_tuple(depth)). LAZINESS REQUIREMENT + - Every gen_* returns (type_expr, value_gen_strategy). value_gen must be a SearchStrategy[ast.expr] that generates the matching AST value. - Do not eagerly draw a value in the generator unless unavoidable. Prefer to return a composed strategy, e.g., st.lists(elem_vg, ...).map(_build_list). NON-VERBOSE STYLE + - Keep the code short and readable; avoid excessive scaffolding. - Avoid redundant checks that are already handled by gen_type(depth-1). - Do not introduce config objects or include_* boolean flags. TESTS (MUST GENERATE) Produce a separate test module (pytest + hypothesis) with tests that verify each generator produces what it promises: + 1) test_gen_list_generates_list: - - data.draw(gen_list(depth)) returns type_expr that is ast.Subscript with value ast.Name('list') - - value_expr = data.draw(value_gen) is ast.List + + - data.draw(gen_list(depth)) returns type_expr that is ast.Subscript with value ast.Name('list') + - value_expr = data.draw(value_gen) is ast.List + 2) test_gen_dict_generates_dict: - - type_expr is ast.Subscript with value ast.Name('dict') - - value_expr is ast.Dict and len(keys)==len(values) - - keys are ast.Constant (or None if you later add ** unpacking; currently should be Constant) + + - type_expr is ast.Subscript with value ast.Name('dict') + - value_expr is ast.Dict and len(keys)==len(values) + - keys are ast.Constant (or None if you later add ** unpacking; currently should be Constant) + 3) test_gen_tuple_generates_tuple: - - type_expr is ast.Subscript with value ast.Name('tuple') - - value_expr is ast.Tuple - - empty tuple must be reachable (not necessarily always) + + - type_expr is ast.Subscript with value ast.Name('tuple') + - value_expr is ast.Tuple + - empty tuple must be reachable (not necessarily always) + 4) test_gen_union_generates_union: - - type_expr is a BinOp chain with BitOr (at least one BinOp at root) - - flatten leaves; number of leaves: - - if depth <= 1 => exactly 2 - - if depth > 1 => between 2 and max_len(depth) + + - type_expr is a BinOp chain with BitOr (at least one BinOp at root) + - flatten leaves; number of leaves: + - if depth <= 1 => exactly 2 + - if depth > 1 => between 2 and max_len(depth) and if depth <= 2 all leaves must be ast.Name of base types only (including NoneType). - - value_expr = data.draw(value_gen) must be ast.expr - - additionally for depth <= 2, since arms are base types, value_expr should be ast.Constant whose underlying Python value type matches one of the union arms: - bool->bool, int->int, str->str, float->float, bytes->bytes, NoneType->NoneType (type(None)). + - value_expr = data.draw(value_gen) must be ast.expr + - additionally for depth <= 2, since arms are base types, value_expr should be ast.Constant whose underlying Python value type + matches one of the union arms: + bool->bool, int->int, str->str, float->float, bytes->bytes, NoneType->NoneType (type(None)). + 5) test_smoke_compile: - - for depth=0, for each gen_base/gen_list/gen_dict/gen_union/gen_tuple, build an annotated assignment: - x: = - wrap in ast.Module and compile() it. Compilation must succeed. - - Note: execution is not required. + + - for depth=0, for each gen_base/gen_list/gen_dict/gen_union/gen_tuple, build an annotated assignment: + x: = + wrap in ast.Module and compile() it. Compilation must succeed. + - Note: execution is not required. IMPORTANT: OUTPUT FORMAT + - Produce the full code for the generator module in one code block. - Produce the full code for the tests in a second code block. - Do not output in tables. - Do not add extra “focused wrapper” functions list_type_and_value/dict_type_and_value/union_type_and_value/tuple_type_and_value. -- Ensure all builder helpers are named _build_* and grouped together. \ No newline at end of file +- Ensure all builder helpers are named _build_* and grouped together.