Discover template instantiations#93
Draft
kwabenantim wants to merge 37 commits into
Draft
Conversation
Discover `template class Foo<N>;` instantiations from the source .cpp files via pygccxml, as an opt-in fallback to template_substitutions. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Extract template parameter names from the class header template declaration so default-argument substitution works for classes wrapped via template instantiation discovery. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
CastXML renders defaulted template arguments inconsistently across versions (e.g. 0.4.4 names AbstractMesh<2, 2> as AbstractMesh<2>), so read instantiation args from the source text instead. pygccxml is kept as a fallback only for templated classes the source scan cannot match. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds an opt-in mechanism to automatically discover explicit C++ template instantiations from implementation (.cpp) files to populate wrapper template substitutions, addressing Issue #73 and reducing the need for hand-written template_substitutions in YAML.
Changes:
- Introduces source-text scanning utilities to find explicit
template class Foo<...>;instantiations and template parameter names from class declarations. - Adds configuration/plumbing (
discover_template_instantiations,source_cpp_patterns,.cppfile collection) and generator logic to apply discovered instantiations, with a CastXML/pygccxml fallback. - Expands unit tests and updates example configs to demonstrate/enable discovery.
Reviewed changes
Copilot reviewed 12 out of 12 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/test_utils.py | Adds unit tests for template-arg splitting and source parsing helpers. |
| tests/test_package_info.py | Adds tests for collecting .cpp files and applying instantiation maps at package level. |
| tests/test_class_info.py | Adds tests for class-level application of discovered instantiations and inheritance of the opt-in flag. |
| examples/shapes/wrapper/package_info.yaml | Documents the new discovery option and related patterns. |
| examples/cells/dynamic/config.yaml | Enables discovery and removes now-unnecessary template_substitutions examples. |
| cppwg/utils/utils.py | Implements template instantiation discovery and template-parameter parsing from source text. |
| cppwg/parsers/source_parser.py | Refactors CastXML config creation and adds a fallback parser for instantiations in .cpp files. |
| cppwg/parsers/package_info_parser.py | Adds defaults for discover_template_instantiations and source_cpp_patterns. |
| cppwg/info/package_info.py | Collects implementation files and adds logic to decide when discovery/fallback is needed. |
| cppwg/info/class_info.py | Applies discovered instantiations to classes and recovers template parameter names from headers. |
| cppwg/info/base_info.py | Adds tri-state discover_template_instantiations to the info tree for inheritance. |
| cppwg/generators.py | Runs discovery after package init to populate template args for opted-in classes. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Add a MacroMesh class to the cells example whose explicit template instantiations are declared with a macro. The source-text scan cannot see these, so they exercise the pygccxml discovery fallback. Keep preprocessor lines when selecting candidate files so a macro that expands to `template class ...;` still flags its file. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ations Make the cells MacroMesh class use a defaulted template argument so the discovery fallback test covers the macro + defaulted-arg case, which requires CastXML >= 0.6.0 (older versions drop the defaulted arg from the instantiation name). Ubuntu 22.04 ships CastXML 0.4.4, so install 0.6.5 from CastXMLSuperbuild on that CI runner. Document the requirement in the README. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Defer implementation-file collection out of PackageInfo.init() into the discovery step, behind the uses_template_discovery() gate, so generation runs that do not use template instantiation discovery avoid a second walk of the source tree. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Move the read into utils.find_template_instantiations_in_source_file, which returns whether the file holds an explicit instantiation and the instantiations parsed from its text in a single read (stripping comments once, then deriving the preprocessor-kept candidate check and the preprocessor-stripped scan), instead of reading and stripping each candidate file twice. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Header collection: map each configured exception class to its declaring header in a single pass over the headers, instead of re-reading every header once per exception name. - CppClassInfo: memoize the template parameter names read from the class header so discovery does not read the same header twice. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a polymorphic Facet<DIM> element whose faces are Facet<DIM-1>, instantiated at <1> and <2> but not <0> (mirroring Chaste's VertexElement<0, SPACE_DIM>). Wrapping the curated Facet<2> works; wrapping the low-dim Facet<1> references the never-instantiated Facet<0> and fails to import with `undefined symbol: Facet<0>::~Facet` - a minimal, fast-building reproduction of the PyChaste discovery-superset failure. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A wrapped instantiation whose public interface takes or returns a project template type that is never instantiated (e.g. Facet<1>::GetFace returning the never-instantiated Facet<0>) fails to link/import with an undefined symbol. Add prune_uninstantiated_dependencies() to detect and drop such instantiations automatically (per instantiation, with a warning), so the boundary-layer case no longer needs hand-written template_substitutions. The check is based on explicit instantiation, not AST completeness: CastXML reports uninstantiated project templates as complete and complete library templates as incomplete, so completeness is unreliable. The "instantiated" set is the union of wrapped cpp_names and explicit `template class X;` statements in the source .cpp files; the latter keeps a class curated out of wrapping but still instantiated in source from pruning its wrapped siblings. Only references sharing a base name with an instantiated class are checked, so library types (std::vector, vtkSmartPointer, ...) are left alone. Add the Corner example to examples/cells - structurally identical to the curated Facet, but resolved by this automatic pruning instead of a template_substitutions block - and unit tests covering drop, library-type ignore, and the source-instantiation keep/drop distinction. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A type reached only through a method or constructor that will not be wrapped cannot cause an undefined symbol, so the prune must not drop an instantiation on its account. Make prune_uninstantiated_dependencies scan only the members that would actually be wrapped, applying the same config-driven exclusions the method and constructor writers apply: excluded_methods, return_type_excludes, arg_type_excludes, constructor_arg_type_excludes and calldef_excludes (matched with type_string_matches), plus skipping constructors for an abstract class that inherits from an abstract base. For example VertexMesh excludes GetFace (which returns the never-instantiated VertexElement<0, SPACE_DIM>); the prune no longer drops a VertexMesh instantiation on that basis, only for dependencies reached through members that are genuinely wrapped. Add a unit test covering a dependency reached only via an excluded method. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
parse_template_params split each signature part on a single literal space, so multiple spaces or tabs (common in YAML or formatted C++) produced empty tokens - e.g. "unsigned DIM" -> ["unsigned", "", "DIM"] - making tokens[1] an empty string and silently dropping the parameter name. Use split() with no argument to split on runs of arbitrary whitespace and discard empty tokens. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two robustness fixes to the type/signature parsing used by discovery and the prune/writer exclusions: - parse_template_params split each signature on every comma, so a comma inside a nested-template default (e.g. "<class T = std::map<int, int>, unsigned DIM>") split one parameter into two. Strip the outer angle brackets and split on top-level commas only, via split_template_args. - type_string_matches compared the pattern and type string literally, so a difference in spacing around punctuation (e.g. config "TetrahedralMesh<3, 3>" vs pygccxml "TetrahedralMesh<3,3>") defeated the match. Add canonicalize_type_whitespace, which drops whitespace except between two identifier characters, and canonicalize both operands before matching; "unsigned int" still does not match "unsignedint". Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
uses_template_discovery() gates the source .cpp tree walk and scan, but it returned True for any opted-in class with empty template_arg_lists, including untemplated ones. With discover_template_instantiations enabled at package or module level, that triggered the full scan even when no templated class could benefit. Additionally require the class to be templated: skip a class whose header is resolved (source_file_path set) and declares no template parameters. init() runs update_from_source() - which resolves source_file_path - before discovery, so this is determinable. A class whose templated-ness is unknown (no resolved header path) is treated conservatively as possibly templated, so discovery still runs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
find_template_params_in_source matched the parameter list with template<([^>]*)>, whose [^>]* stops at the first ">". A parameter default containing a nested template (e.g. "template<class T = std::map<int,int>> class Foo") ended the capture early and the following "class Foo" then failed to match, so the function returned [] and discovery lost the class's template parameter names. Find the closing ">" by matching angle brackets on depth instead: scan each "template<" for its balanced "<...>" and accept it when the wanted class/struct name follows. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
_referenced_instantiations matched template-ids with "[A-Za-z_][\w:]*<[^<>]*>", whose [^<>]* only spans an innermost argument list. An uninstantiated enclosing template whose argument is itself a template (e.g. a reference to Widget<Gadget<0>> where Gadget is not a wrapped base) was therefore never seen, so the prune could miss a drop and leave an undefined symbol. Find each template-id's closing ">" by matching angle brackets on depth, and yield template-ids at every nesting level. The project_bases filter still discards library outer types (std::shared_ptr<...>, std::vector<...>), so this only adds correct project matches. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
parse_instantiations keyed the instantiation map on the base returned by
declarations.templates.split(class_decl.name). pygccxml already reports the
class name unqualified (e.g. "Bar<2>" for ::foo::Bar<2>), but the text-scan
path (find_template_instantiations_in_source) explicitly unqualifies with
split("::")[-1] while the fallback relied on that happening to be so. Strip any
qualification defensively so both discovery paths key on the unqualified name
that matches CppClassInfo, independent of pygccxml version.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
bases_block compared each external base's unqualified pygccxml name against the external_bases config entries verbatim, so a namespace-qualified entry (e.g. "ns::AbstractFoo") silently failed to match a base named AbstractFoo. Unqualify the config entries before the membership test so both qualified and unqualified forms match. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
87dae84 to
aa47e60
Compare
collect_source_files sorted only by basename. Python's sort is stable, so files sharing a basename in different directories (e.g. multiple Foo.cpp) kept their os.walk order, which is filesystem/platform-dependent. That order flows through source_cpp_files into discovery's instantiation map and on into the generated wrapper names, so a basename collision could make wrapper generation non-reproducible across runs or platforms. Sort by (basename, full path) so shared basenames get a deterministic total order while keeping the existing primary basename ordering. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
prune_uninstantiated_dependencies scanned constructor arguments for uninstantiated dependencies but did not skip constructors excluded by constructor_signature_excludes, which CppConstructorWrapperWriter.exclude() applies. A dependency reachable only through such a constructor - one that is never emitted - could therefore prune an instantiation and needlessly shrink the wrapped surface. Skip a constructor when a constructor_signature_excludes entry matches its full signature (same arity, each argument matching its positional pattern), so the prune's constructor exclusions now match the writer's. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The CastXML fallback ran only when a templated class had no discovered args, and apply_template_instantiations returned early once args were set. So a class with some instantiations found by the text scan and others generated only by a macro (invisible to the text scan) never got the macro instantiations wrapped. - Trigger the fallback when a candidate file is macro-only (has an instantiation marker but an empty text-scan map), as well as when a class is still unresolved. - Merge fallback arg lists into a class's discovery-derived args (deduped) instead of skipping classes that already have args. template_substitutions still take precedence and are never extended (tracked via template_args_from_discovery). - Guard the CastXML defaulted-arg rendering hazard: CastXML < 0.6.0 drops defaulted trailing template arguments, so merging its arg lists into text-scanned ones could add a differently-rendered duplicate. The generator records whether CastXML preserves defaulted args (>= 0.6.0); when it does not, a merge into a class with defaulted template parameters is skipped (warning only when the fallback plausibly holds additional instantiations). Add find_template_signature_in_source / template_has_default_param helpers and tests for the merge behaviour and the defaulted-parameter detection. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- README: document the automatic pruning of instantiations that depend on an uninstantiated project type (the "Excluding ..." warnings), and note that external_bases matches with or without namespace qualification. - discover_template_instantiations docstring: the CastXML fallback no longer only covers classes the text scan missed; it also merges macro-generated instantiations into already-discovered args, gated on CastXML >= 0.6.0. - has_unresolved_template_classes docstring: it is now one of two fallback triggers (the other being a macro-only candidate file). - prune_uninstantiated_dependencies docstring: note that it honours the same config-driven exclusions as the writers. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
prune_uninstantiated_dependencies rebuilt cpp_names, py_names and decls when dropping an instantiation but left template_arg_lists unchanged, so it fell out of alignment with cpp_names. The writers index template_arg_lists by position (e.g. to substitute template parameters in default arguments), so a dropped low-dimension instantiation shifted the remaining ones onto the wrong args - e.g. a kept Foo<2> being rendered with Foo<1>'s template arguments. Prune template_arg_lists in lockstep with cpp_names/py_names/decls. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The discovery CastXML fallback was gated on has_unresolved_template_classes() and then parsed every candidate .cpp file (any file containing "template class"). For a large project that is hundreds of full CastXML runs, almost all redundant: the text scan has already captured every literal "template class X<...>;", so re-parsing those files only re-derives what is already known. Parse only macro-only files (an instantiation marker but no literal instantiation the text scan could read). A class never instantiated in the source - e.g. an abstract base - simply is not discovered; there is nothing to find. Remove the now-unused has_unresolved_template_classes. Also document that a file mixing literal and macro-generated instantiations is not fully discovered (its macro-generated ones are missed) and must be configured manually. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
log_unknown_classes reported every parsed class whose location is under the source root. When a project vendors its dependencies (boost, PETSc, VTK ...) under the source root, that is thousands of library-internal classes - slow to iterate and log, and useless noise. Report only classes declared in the collected source headers (source_hpp_files), which excludes vendored dependencies and all transitively-included headers. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The log file given by -l/--logfile was overwritten each run. Insert a "_YYYYmmdd-HHMMSS" timestamp before the suffix (cppwg.log -> cppwg_20260708-153012.log) so each run keeps its own log and runs can be compared, and log the chosen path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ed classes A templated class that is never explicitly instantiated - e.g. an abstract base only ever used as a base class or through pointers - leaves no "template class X<...>;" for the text scan or the CastXML macro fallback to find. But it is implicitly instantiated wherever a wrapped concrete class derives from it, so it appears (at the right template arguments) in that class's base declarations. Add discover_base_class_instantiations: after update_from_ns, harvest such instantiations from the base-class hierarchy of already-wrapped classes and adopt them for any opted-in, still-unresolved templated class, so abstract bases can be wrapped without hand-written template_substitutions. A harvested arg list is adopted only when its length matches the class's template parameter count, so a CastXML version that collapses a defaulted trailing argument (naming e.g. AbstractLinearPde<2, 2> as AbstractLinearPde<2>) cannot create a mis-named instantiation. The header collection is re-written afterwards so the build sees the newly wrapped bases. update_from_ns now skips (rather than aborting on) a class whose declaration is not found, so such a class can be resolved here instead of crashing generation. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The CastXML/pygccxml parse of the header collection is the dominant cost of generation and is repeated in full every run. Add an opt-in --cache [PATH] option that uses pygccxml's file_cache_t, so an unchanged parse is reused. This is safe: pygccxml invalidates the cache when the header collection, the CastXML configuration, or any transitively-included header changes (it re-hashes them). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The generated log file is now cppwg_<timestamp>.log rather than a fixed cppwg.log, so the example CI workflows that grep the log for "Unknown class" must match the timestamped name. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Base-class discovery re-wrote the header collection so newly discovered classes would appear in it. That was unnecessary - the header collection is only the CastXML parse input; the discovered classes are already in the parsed namespace (implicitly instantiated via their concrete children) and their wrappers are self-contained. Worse, the re-write ran after update_from_ns, which updates source_file for classes resolved via the typedef fallback, so the re-written include/typedef order differed from the first write - reordering the example header collections and breaking the "no wrapper changes" CI check. Remove the re-write. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The previous scoping to source_hpp_files was insufficient: source headers are collected by walking the whole source root, so a project that vendors its dependencies under the source root (e.g. PyChaste vendors boost) still had thousands of library-internal classes collected - and log_unknown_classes reported and logged them all, which is slow and noisy. Report only classes whose file is under a configured module source_location (the directories actually being wrapped), falling back to the source root for a module that wraps everything. Apply the same scope to the uninstantiated-template scan so it does not read vendored headers either. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A module that wraps everything has source_locations of None rather than an empty list, so iterating it to build the reporting scope raised a TypeError. Treat a missing value as empty (falling back to the source root). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…uning The header collection is written before parsing, so its first version cannot reflect classes discovered from the base-class hierarchy (added later) or instantiations removed by pruning. It was therefore inconsistent with the generated wrappers - e.g. it still listed template class Corner<1>; and its typedef even though Corner<1> is pruned and no wrapper is generated for it. Re-write the header collection after pruning so it reflects the final wrapped set. To keep that re-write idempotent (an earlier attempt reordered the include and typedef blocks because update_from_ns backfills source_file for typedef-fallback-resolved classes, breaking the "no wrapper changes" check), emit the includes and the instantiation/typedef pairs in sorted order rather than relying on class order. The header collection is only the CastXML parse input and is not compiled, so pruned instantiations were never needed there for symbols. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
pygccxml only consults the declarations cache in FILE_BY_FILE mode (via source_reader.read_cpp_source_file); ALL_AT_ONCE builds a synthetic #include string and parses it with no cache, so --cache had no effect. cppwg parses a single file - the header collection - which is equivalent under either mode (verified: identical output), so parse file-by-file when a cache is configured. A warm re-run then reuses the parse instead of re-running CastXML. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A module with no source_locations wraps everything, so it contributes the source root. Deciding this per module ensures one module restricting its locations does not narrow the scope for a module that wraps everything. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The --cache option reused pygccxml's parse via file_cache_t, which only takes effect in FILE_BY_FILE compilation mode. On a large project (Chaste) that mode triggers pygccxml's _relink_declarated_types step, which fails with "Unable to find out actual class definition", so wrapper generation could not complete with --cache. Remove the option and always parse the header collection in ALL_AT_ONCE mode. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
pybind11 requires a base class to be registered before any subclass, so a class must be emitted after every class it extends. Two bugs broke this: - The sort ran inside update_from_ns, before base-class discovery and pruning had finalised each class's declarations. A base whose instantiation is only added by discovery (e.g. AbstractLinearPde<1,1>, the base of AbstractLinearEllipticPde<1,1>) had no decls at sort time, so extends() returned False and the derived class kept its earlier alphabetical position. This produced an unimportable module: "referenced unknown base type". - extends() matched base classes by pygccxml declaration identity, which is not preserved across template/typedef resolution. Run the sort in generate() after discover_base_class_instantiations and prune_uninstantiated_dependencies, match bases by name (unqualified, template args stripped), and replace the ad-hoc insertion sort with a deterministic topological sort: an alphabetical baseline with alphabetical tie-breaks, so the registration order - and the generated files - are stable run to run. The shapes primitives module reorders to the now-deterministic order. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Supports #73
Discover
template class Foo<N>;instantiations from the source .cpp files via custom text parsing as an opt-in fallback to template_substitutions. Uses pygccxml as a secondary option for cases that custom text parsing can't handle e.g. template instantiations done with macros. The custom text parsing is the primary path because handling template arguments with default values is not consistent between pre- and post- 0.6 castxml versions (See also #3).Does automatic pruning to exclude classes that depend on uninstantiated templates e.g. if
Foo<1>,Foo<2>, andFoo<3>are instantiated in the C++ code butFoohas aUseLesserFoo(Foo<DIM-1>)method which makesFoo<1>depend on the uninstantiatedFoo<0>, thenFoo<1>will be pruned from wrapping automatically with a notice about this.Foo<2>is kept because it can still be wrapped asFoo<1>is instantiated in the C++ code.Excluded methods are considered during pruning. For example, if
UseLesserFoo(Foo<DIM-1>)is manually excluded, thenFoo<1>can still be wrapped as it no longer depends on the uninstantiatedFoo<0>. In that case,Foo<1>is not pruned out from wrapping.