[EPIC] TTE code sharing and new python packages organization proposal #54

What is the requested feature?

Various Python tools were developed by TotalEnergies to visualize and process GEOS inputs/outputs. These codes may however benefit to the whole GEOS community (e.g., #50) and can be shared. Some common needs (e.g., #16) may also be addressed once instead of multiple times. To this end, this epic aims to share in the GEOS repository TotalEnergies Python tools and reorganize/refactor python projects to merge duplicated codes.

Is your request related to a specific problem?

• Need to share TotalEnergies Python tools that will benefit to the whole GEOS community,
• Reorganized and refactor python projects, to avoid duplicated codes and improve dependency management (cf. proposed organisation)

Describe the solution you'd like

We propose first to integrate TotalEnergies code to geosPythonPackages as they currently are, then a refactoring of the whole project to merge duplicated codes and update package dependencies.

The proposed organisation is the following, as described in the attached document:

• Data structures and processing packages:
  • geos-ats:
  • hdf5-wrapper:
  • geos-xml-tools: reader, writer, split or merge tools, check tools for geos-xml file.
  • geos-mesh: IO, statistics, procesing tools of meshes based on vtk data structure.
  • geomechanics: Data structures and geomechanical properties calculation functions
• API packages:
  • xml-vtk: Convert geos-xml to vtk objects and conversely.
  • geos-prop: Pre-procssing tools as vtk filters including mesh-doctor features
  • geos-posp: Post-processing tools as vtk filters including geos output mesh cleaning and new properties calculation filters
• Script packages:
  • geos-timehistory: plot time serie curves from geos hdf5 (and csv) files
  • geos-xml-viewer: Input object visualization tool from geos-xml based on pyvista
  • mesh-doctor: pre-processing scripts
  • geos-trame: xml interactive viewer and editor
• Paraview (PV) plugins:
  • geos-pv: paraview plugins to wrap API packages

The proposed new organisation aims at splitting processing packages from user-interface packages (i.e., script or IHM-based packages). This organisation was initially proposed by @untereiner who applied it to packages hdf5-wrapper (processing) and geos-timehistory (script).

Advantages are:

• to use same processing modules in multiple user-interface packages
• avoiding duplicated codes and easing code maintenance
• better management of package dependencies

For instance, PR #16 deals with the need to use mesh-doctor tools through Paraview in addition of current scripts. This need requires to either duplicate code or introduce a dependency of Paraview to the whole geos-mesh package, resulting in introducing a dependency of Paraview plugins to pyvista. It is not recommended since Paraview and pyvista are both user-interface modules (the former through an IHM, the latter through APIs) with similar functionalities. The integration of TotalEnergies code into geosPythonPackages will raise similar issues.

Proposed roadmap

1. TotalEnergies code sharing:

• First PR with TotalEnergies python code for GEOS post-processing (geos-posp), pre-processing (geos-prep), geos-xml-viewer, and geos-trame packages with minimal work to conform to CI/CD #56
• Settup Github CI/CD with unit tests, mypy and linting checks #58
• Create geos-geomechanics package with geomechanics classes and functions from geos-posp and update dependencies #59
• Create geos-utils package with logger, geos constants, geometry functions, ConnectionSet and update dependencies #63
• Move VTK functions and utilities to geos-mesh package #69
• Create vtk filter in geos-mesh and its associated pv plugin in geos-pv to fill partial arrays #104
• Create vtk filter from PVCreateConstantAttributePerRegion in geos-mesh move PV plugin to geos-pv #106
• Move vtk filter AttributeMappingFromCellCoords to geos-mesh + remove AttributeMappingFromCellId + move corresponding PV plugin to geos-pv #125
• Move VolumeSurfaceMeshMapper to geos-mesh utils #136
• Move TransferAttributesVolumeSurface to geos-mesh + move corresponding PV plugin to geos-pv #135
• Create vtk filter from PVMergeBlocksEnhanced in geos-mesh and move PV plugin to geos-pv #92
• Move GeosBlockExtractor to geos-processing (refactor: Move GeosBlockExtractor from geos-posp to geos-proccessing #142)
• Move GeosBlockMerge to geos-processing (refactor: Move GeosBlockMerge to geos-processing #163)
• refactor: Move existing VTK filter and PV plugin SurfaceGeomechanics to appropriate folders #131
• Move vtk filter GeomechanicsCalculator to geos-processing + move corresponding PV plugin to geos-pv #139
• Refactor the four ParaView plugins PVExtractMergeBlocksVolume* in one plugin #173
• Refactor the four ParaView plugins PVGeomechanicsWorkflowVolume* in one #174
• Create geos-pv package and add PVGeosLogReader plugin #70
• refactor: Move and refactor Mohr Circle plugin #166
• Integration of Makutu code (Revert "feat: Python interfaces to geos from Makutu repo" #47 and Rework "feat: Python interfaces to geos from Makutu repo" #52)
• Add TotalEnergies existing mesh statistic code to geos-mesh package (feat: Add mesh statistic tools #95)
• Update documentation (project README.md) to explain new project organisation and package dependencies. #61

2. Create vtk filters for mesh-doctor features #80 :

• Create vtk filter for mesh-doctor generate_global_ids in geos-mesh
• Create vtk filter for mesh-doctor collocated_nodes in geos-mesh
• Create vtk filter for mesh-doctor element_volumes in geos-mesh
• Create vtk filter for mesh-doctor fix_elements_orderings in geos-mesh
• Create vtk filter for mesh-doctor generate_fractures in geos-mesh
• Create vtk filter for mesh-doctor non_conformal in geos-mesh
• Create vtk filter for mesh-doctor self_intersecting_elements in geos-mesh
• Create vtk filter for mesh-doctor supported_elements in geos-mesh
• Create mesh-doctor package and update with vtk filters
• Create Paraview plugins for mesh-doctor vtk filters (PoC / Paraview integration for mesh_doctor  #16)
• Add mesh_stats checks and field_operations features #51 and merge with TotalEnergies codes

3. geos-xml-tools refactoring

• Gather python codes that process GEOS xml files from geos-xml-tools, geos-xml-viewer, geos-trame (Unify the XML parsing of GEOS files #76)
• Create xml-vtk package with xml-vtk conversion vtk filters
• Refactor geos-xml-viewer to update with xml-vtk package
• Refactor geos-trame to update with xml-vtk package
• (Optional) Create Paraview plugins for xml-vtk filters
• File classification  #25

PythonPackagesProposal.pdf
updatePythonPackagesProposal_09042025.pdf