Documentation

README.md

@@ -1,23 +1,98 @@
-# hpp-python package
+# hpp-python
 
-Provide Python bindings of the HPP software. These bindings differ from the one provided by
-`hpp-corbaserver`. They are native bindings that do not use a middleware.
-The bindings are generated by `boost::python`.
+[![Pipeline status](https://gitlab.laas.fr/humanoid-path-planner/hpp-python/badges/master/pipeline.svg)](https://gitlab.laas.fr/humanoid-path-planner/hpp-python/commits/master)
+[![Coverage report](https://gitlab.laas.fr/humanoid-path-planner/hpp-python/badges/master/coverage.svg?job=doc-coverage)](https://gepettoweb.laas.fr/doc/humanoid-path-planner/hpp-python/master/coverage/)
+[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
 
-### Compilation
+`hpp-python` provides native Python bindings for the [HPP](https://github.com/humanoid-path-planner/hpp-doc) (Humanoid Path Planner) C++ libraries (`hpp-core`, `hpp-constraints`, `hpp-pinocchio`, `hpp-manipulation`, ...), generated with [`boost::python`](https://www.boost.org/doc/libs/release/libs/python/). These bindings differ from the ones provided by `hpp-corbaserver`: they are **native, in-process bindings that do not use a CORBA middleware**, so there is no client/server split and no serialization overhead.
 
-Installation follows the standard CMake procedure:
-```sh
-git clone --recursive ...
+The resulting `pyhpp` package mirrors the C++ namespace layout:
+
+- **`pyhpp.pinocchio`** — robot model (`Device`), grippers, Lie-group utilities. Wraps `hpp-pinocchio`.
+- **`pyhpp.constraints`** — differentiable functions, transformations, implicit/explicit constraints, the hierarchical iterative solver. Wraps `hpp-constraints`.
+- **`pyhpp.core`** — planning problem, path planners, path optimizers, path validation, roadmap, steering methods. Wraps `hpp-core`.
+- **`pyhpp.manipulation`** — manipulation-specific `Device`, constraint graph, path planners/optimizers for manipulation problems, URDF/SRDF loading. Wraps `hpp-manipulation` and `hpp-manipulation-urdf`.
+- **`pyhpp.tools`** — pure-Python helpers (xacro processing, constraint-error reporting) that are not generated bindings.
+
+## Table of contents
+
+- [Module dependency graph](#module-dependency-graph)
+- [Dependencies](#dependencies)
+- [Installation](#installation)
+  - [From source with CMake](#from-source-with-cmake)
+  - [With Nix](#with-nix)
+- [Documentation generation](#documentation-generation)
+  - [Injecting Doxygen docstrings into bindings](#injecting-doxygen-docstrings-into-bindings)
+  - [API reference from stubs](#api-reference-from-stubs)
+- [License](#license)
+
+
+## Module dependency graph
+
+The `BOOST_PYTHON_MODULE` initializers `import` each other in a fixed order, which also reflects the underlying C++ library dependencies:
+
+```mermaid
+flowchart LR
+    pin["pyhpp.pinocchio\n(hpp-pinocchio)"]
+    cons["pyhpp.constraints\n(hpp-constraints)"]
+    core["pyhpp.core\n(hpp-core)"]
+    manip["pyhpp.manipulation\n(hpp-manipulation)"]
+    ext["pinocchio\n(external package)"]
+
+    ext --> pin
+    pin --> cons
+    cons --> core
+    core --> manip
+
+    style pin fill:#dbe9f4,stroke:#4a90d9
+    style cons fill:#dbe9f4,stroke:#4a90d9
+    style core fill:#dbe9f4,stroke:#4a90d9
+    style manip fill:#dbe9f4,stroke:#4a90d9
+    style ext fill:#e8f4e8,stroke:#4a9d4a
+```
+
+## Dependencies
+
+- Python ≥ 3.10, with `numpy`
+- [Boost.Python](https://www.boost.org/doc/libs/release/libs/python/) (found via `search_for_boost_python()`)
+- [`eigenpy`](https://github.com/stack-of-tasks/eigenpy)
+- [Pinocchio](https://github.com/stack-of-tasks/pinocchio) (C++ and Python)
+- HPP C++ libraries: `hpp-util`, `hpp-pinocchio`, `hpp-constraints`, `hpp-core`, `hpp-manipulation`, `hpp-manipulation-urdf`
+- Optional: [`pybind11-stubgen`](https://github.com/sizmailov/pybind11-stubgen) to generate `.pyi` type stubs (`GENERATE_PYTHON_STUBS`, default `ON`)
+- For running the test suite (`BUILD_TESTING`): [`example-robot-data`](https://github.com/Gepetto/example-robot-data), `hpp-environments`
+
+## Installation
+
+### From source with CMake
+
+```bash
+git clone https://github.com/humanoid-path-planner/hpp-python.git
 mkdir hpp-python/build
 cd hpp-python/build
-cmake -DCMAKE_INSTALL_PREFIX=... -DCMAKE_BUILD_TYPE=Release ..
+cmake -DCMAKE_INSTALL_PREFIX=<your_install_prefix> -DCMAKE_BUILD_TYPE=Release ..
 make
 make test
 make install
 ```
 
-### Generate documentation
+Relevant CMake options:
+
+- `GENERATE_PYTHON_STUBS` (default `ON`): generate `.pyi` stub files with `pybind11-stubgen` for IDE autocompletion; automatically disabled if the tool is not found.
+- `HPP_DEBUG` (default `OFF`): enable `hpp-util` debug logging (`-DHPP_DEBUG`).
+- `HPP_BENCHMARK` (default `OFF`): enable `hpp-util` benchmark output (`-DHPP_ENABLE_BENCHMARK`).
+- `BUILD_TESTING`: build and run the unit/integration test suite (requires `example-robot-data` and `hpp-environments`).
+
+### With Nix
+
+A flake is provided, pulling `hpp-constraints`, `hpp-core` and `hpp-manipulation` from their own flakes and building against [`github:gepetto/nix`](https://github.com/gepetto/nix):
+
+```bash
+nix build github:humanoid-path-planner/hpp-python
+```
+
+## Documentation generation
+
+### Injecting Doxygen docstrings into bindings
 
 Script `doc/configure.py` is used to generate documentation of Python objects from C++ objects.
 It reads the XML documentation generated by doxygen so the dependencies must provide such
@@ -33,8 +108,30 @@ ENDIF()
 
 File `doc/configure.py` contains a short documentation of how to document the bindings.
 
-### TODO
+### API reference from stubs
+
+After building and installing `pyhpp`, `pybind11-stubgen` produces `.pyi` stub files under the Python site-packages directory.
+
+**`doc/fix_stubs.py`** improves those raw Boost.Python stubs in-place by recovering proper parameter names, types, and return-type annotations from the embedded docstrings. The fixed stubs are useful on their own — IDEs (autocompletion, type checking) and static analysis tools consume them directly, independently of any documentation generation step:
+
+```bash
+python doc/fix_stubs.py path/to/site-packages/pyhpp/
+# or individual files:
+python doc/fix_stubs.py path/to/site-packages/pyhpp/core/bindings.pyi
+```
+
+Optional JSON override files can supply or correct signatures that cannot be inferred from docstrings alone (`--setter-overrides`, `--method-overrides`).
+
+**`doc/gen_api_md.py`** reads the (optionally fixed) stubs and generates one Markdown file per module — suitable for [mdbook](https://rust-lang.github.io/mdBook/) — plus an `index.md`, with syntax-coloured signatures and clickable cross-module type links:
+
+```bash
+python doc/gen_api_md.py \
+    --stubs path/to/site-packages \
+    --output path/to/docs/api/
+```
+
+Modules covered: `pyhpp.core`, `pyhpp.core.path`, `pyhpp.core.path_optimization`, `pyhpp.core.problem_target`, `pyhpp.constraints`, `pyhpp.manipulation`, `pyhpp.manipulation.urdf`, `pyhpp.manipulation.steering_method`, `pyhpp.pinocchio`, `pyhpp.pinocchio.urdf`.
+
+## License
 
-- Use doxygen to generate XML documentation of the headers included by a file.
-  Then use the generated XML doc to update the documentation of the files in `src/pyhpp`.
-  Doxygen configuration variables `INCLUDE_PATH` and `SEARCH_INCLUDES` might be helpful.
+`hpp-python` is released under the [BSD 2-Clause License](LICENSE), Copyright (c) 2018-2025, CNRS.

doc/configure.py

@@ -138,14 +138,16 @@ def indexFromNamespace(ns):
 
 
 def escape(s):
-    # return s.replace ('\n', r'\n')
-    return s.replace("\n", r"\n")
+    s = s.replace("\\", "\\\\")  # backslash → \\
+    s = s.replace('"', '\\"')  # double-quote → \"
+    s = s.replace("\n", "\\n")  # newline → \n
+    return s
 
 
 def make_doc_string(brief, detailled):
     if len(brief) == 0 or brief.isspace():
         return '"' + detailled + '"'
-    return '"' + brief + "\n" + detailled + '"'
+    return '"' + brief + "\\n" + detailled + '"'
 
 
 def make_args_string(args):
@@ -155,16 +157,40 @@ def make_args_string(args):
 def substitute(istr, ostr):
     nsPattern = re.compile(r"DocNamespace\s*\(\s*(?P<namespace>[\w:]+)\s*\)")
     classPattern = re.compile(r"DocClass\s*\(\s*(?P<class>[\w:]+)\s*\)")
+    dcdPattern = re.compile(r"DocClassDoc\s*\(\s*(?P<class>[\w:]+\s*)?\)")
     dcmPattern = re.compile(
         r"DocClassMethod\s*\(\s*(?P<method>[\w:]+)\s*(,\s*(?P<class>[\w:]+)\s*)?\)"
     )
     classDoc = None
+    currentClass = None
+    currentNamespace = None
+    index = None
     for line in map(lambda s: s.rstrip(), istr):
         for match in nsPattern.finditer(line):
             currentNamespace = match.group("namespace")
             index = indexFromNamespace(currentNamespace)
         for match in classPattern.finditer(line):
             currentClass = match.group("class")
+        for match in dcdPattern.finditer(line):
+            cn_arg = match.group("class")
+            cn = (
+                currentNamespace
+                + "::"
+                + (cn_arg.strip() if cn_arg and cn_arg.strip() else currentClass)
+            )
+            try:
+                if classDoc is None or classDoc.classname != cn:
+                    classDoc = index.classDoc(cn)
+                b, d = classDoc.getClassDoc()
+                line = line.replace(
+                    match.group(0), make_doc_string(escape(b), escape(d))
+                )
+            except Exception as e:
+                print(
+                    "Failed to find class doc for {0}: {1}".format(cn, e),
+                    file=sys.stderr,
+                )
+                line = line.replace(match.group(0), '"' + cn + '"')
         for match in dcmPattern.finditer(line):
             if match.group("class") is not None:
                 cn = currentNamespace + "::" + match.group("class")
@@ -177,7 +203,7 @@ def substitute(istr, ostr):
                 b, d, args = classDoc.getClassMethodDoc(mn)
                 line = line.replace(
                     match.group(0),
-                    escape(make_doc_string(b, d))
+                    make_doc_string(escape(b), escape(d))
                     + ", "
                     + ((make_args_string(args)) if len(args) > 0 else '""'),
                 )

doc/doxygen_xml_parser.py

@@ -124,7 +124,11 @@ def _getMember(self, sectionKind, memberDefKind, name):
         raise IndexError(msg)
 
     def getClassDoc(self):
-        return self._getDoc(self.compound)
+        b_el = self.compound.find("briefdescription")
+        d_el = self.compound.find("detaileddescription")
+        brief = etree.tostring(b_el, method="text", encoding="unicode").strip()
+        detailed = etree.tostring(d_el, method="text", encoding="unicode").strip()
+        return brief, detailed
 
     def getClassMemberDoc(self, membername):
         # member = self.compound.find ("sectiondef[@kind='public-attrib']/memberdef[@kind='variable' and name='" + methodname + "']")

doc/fix_stubs/__init__.py

@@ -0,0 +1,13 @@
+"""fix_stubs — Fix Boost.Python .pyi stubs by restoring proper signatures from docstrings.
+
+Public API:
+    fix_file(path, setter_overrides=None) -> int
+    main()                                      (CLI entry point)
+    SETTER_TYPE_OVERRIDES                       (built-in setter type table)
+"""
+
+from ._cli import main
+from ._fixer import fix_file
+from ._overrides import SETTER_TYPE_OVERRIDES
+
+__all__ = ["fix_file", "main", "SETTER_TYPE_OVERRIDES"]

doc/fix_stubs/__main__.py

@@ -0,0 +1,5 @@
+"""Allow `python -m fix_stubs` execution."""
+
+from ._cli import main
+
+main()

doc/fix_stubs/_cli.py

@@ -0,0 +1,88 @@
+"""Command-line interface for fix_stubs."""
+
+from __future__ import annotations
+
+import argparse
+import sys
+from pathlib import Path
+
+from ._fixer import fix_file
+from ._overrides import load_method_overrides, load_setter_overrides
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(
+        description="Fix Boost.Python .pyi stubs: restore proper signatures from docstrings"
+    )
+    parser.add_argument(
+        "paths",
+        nargs="+",
+        type=Path,
+        help=".pyi files or directories to fix (directories are searched recursively)",
+    )
+    parser.add_argument(
+        "--setter-overrides",
+        type=Path,
+        default=None,
+        help=(
+            "Optional JSON file with additional/overriding setter type mappings "
+            '(flat object: {"ClassName.propName": "type", ...}). '
+            "Merged on top of the built-in SETTER_TYPE_OVERRIDES table (external entries win)."
+        ),
+    )
+    parser.add_argument(
+        "--method-overrides",
+        type=Path,
+        default=None,
+        help=(
+            "Optional JSON file with additional/overriding method signature mappings "
+            '({"ClassName.method": {"ret": "Type", "params": [["name", "Type"], ...]}}). '
+            "Merged on top of the built-in METHOD_OVERRIDES table (external entries win)."
+        ),
+    )
+    parser.add_argument(
+        "--output-dir",
+        type=Path,
+        default=None,
+        help=(
+            "Write fixed stubs to this directory instead of modifying in-place. "
+            "The input directory structure is preserved relative to each input path."
+        ),
+    )
+    args = parser.parse_args()
+
+    setter_overrides = load_setter_overrides(args.setter_overrides)
+    method_overrides = load_method_overrides(args.method_overrides)
+
+    total = 0
+    total_files = 0
+    for p in args.paths:
+        if not p.exists():
+            print(f"warning: path not found, skipping: {p}", file=sys.stderr)
+            continue
+        root = p if p.is_dir() else p.parent
+        files = sorted(p.rglob("*.pyi")) if p.is_dir() else [p]
+        for f in files:
+            output_path = (
+                args.output_dir / f.relative_to(root)
+                if args.output_dir is not None
+                else None
+            )
+            try:
+                n = fix_file(
+                    f,
+                    output_path=output_path,
+                    setter_overrides=setter_overrides,
+                    method_overrides=method_overrides,
+                )
+            except Exception as e:
+                print(f"error: failed to fix {f}: {e}", file=sys.stderr)
+                continue
+            total_files += 1
+            if n:
+                print(f"{f}: {n} signature(s) fixed", file=sys.stderr)
+            total += n
+    print(
+        f"Total: {total} signature(s) fixed across {total_files} file(s)",
+        file=sys.stderr,
+    )

doc/fix_stubs/_fixer.py

@@ -0,0 +1,44 @@
+"""Orchestrate parsing, import merging, and rendering for a single .pyi file."""
+
+from __future__ import annotations
+
+from pathlib import Path
+
+from ._imports import collect_required_imports, merge_imports
+from ._parser import extract_tree
+from ._renderer import render_file
+
+
+def fix_file(
+    path: Path,
+    output_path: Path | None = None,
+    setter_overrides: dict[str, str] | None = None,
+    method_overrides: dict | None = None,
+) -> int:
+    """Fix a single .pyi file. Returns the number of signatures fixed.
+
+    If output_path is given, writes there; otherwise modifies path in-place.
+    """
+    header, tree = extract_tree(
+        path,
+        setter_overrides=setter_overrides,
+        method_overrides=method_overrides,
+    )
+
+    required_modules = collect_required_imports(tree)
+    header = merge_imports(header, required_modules)
+
+    fixed = 0
+    for cls in tree:
+        for fn in cls.functions:
+            if fn.from_docstring:
+                fixed += 1
+            elif fn.setter_overloads:
+                fixed += 1
+
+    new_content = render_file(header, tree)
+    target = output_path if output_path is not None else path
+    target.parent.mkdir(parents=True, exist_ok=True)
+    target.write_text(new_content)
+
+    return fixed