Support/Linkage to the DD multiversion agnostic middleware

.codex/hooks.json

@@ -0,0 +1,15 @@
+{
+  "hooks": {
+    "PreToolUse": [
+      {
+        "matcher": "Bash",
+        "hooks": [
+          {
+            "type": "command",
+            "command": "/Users/yohann/.local/bin/graphify hook-check"
+          }
+        ]
+      }
+    ]
+  }
+}

.gitignore

@@ -18,3 +18,5 @@ test-install
 .vscode/
 *.err
 *.out
+/graphify-out
+/.idea

AGENTS.md

@@ -0,0 +1,62 @@
+# Repository Guidelines
+
+Any changes should be also reflected in CLAUDE.md
+
+## Project Structure & Module Organization
+
+This is the MATLAB high-level interface to IMAS Access Layer. MATLAB entry points live in `matlab/`; their MEX implementations and shared C helpers are in `src/`. XSLT generators at the repository root and in `common/` generate IDS-specific sources from the IMAS Data Dictionary—edit generators rather than generated output where applicable. `tests/` contains MATLAB unit, integration, and performance tests; `examples/` contains runnable MATLAB examples. Sphinx documentation is in `doc/`, and CI and cluster scripts are in `ci/` and `.github/workflows/`.
+
+## Build, Test, and Development Commands
+
+MATLAB and a C/C++ toolchain are required. CMake fetches IMAS Core and the Data Dictionary unless configured for a local development layout.
+
+```bash
+cmake -B build --preset=https -DAL_BACKEND_HDF5=ON -DAL_TESTS=ON -DAL_EXAMPLES=ON
+cmake --build build --parallel
+ctest --test-dir build --output-on-failure
+cmake --install build
+```
+
+The first command configures an HTTPS-based dependency checkout and enables the HDF5 test backend. The test command runs the CTest registrations, which invoke MATLAB's `runtests('imas_unit_tests')`. For documentation, use `ci/build_docs.sh` or configure with `-DAL_HLI_DOCS=ON -DAL_DOCS_ONLY=ON` and build the resulting tree. Do not commit build directories or generated local artifacts.
+
+For multiversion DD conversion, install IMAS-Multiversion-DD-Loader and configure with `-DAL_USE_MULTIVERSION_SHIM=ON` plus its prefix in `CMAKE_PREFIX_PATH` (or `imas-mvdd-loader_DIR`). The MATLAB MEX targets then link the shim while retaining IMAS-Core for headers and runtime loading. CTest sets `IMAS_MVDD_HLI_DD_VERSION` and `IMAS_CORE_LIBRARY`; set them explicitly when launching MATLAB outside CTest. `AL_CORE_RUNTIME_LIBRARY` overrides CTest's Core path. See `doc/doc_common/building_installing.rst` and `docs/SHIM_INTEGRATION_CONTRACT.md`.
+
+## Coding Style & Naming Conventions
+
+Follow the surrounding file's formatting: MATLAB uses two-space indentation, `function` blocks, and lower-case underscore-separated API names such as `ids_get_slice`; C uses four-space indentation and lower-case underscore-separated filenames such as `imas_mex_utils.c`. Keep MATLAB help comments immediately above public functions and use established `IMAS:<component>:<condition>` error identifiers in MEX code. Prefer focused changes; preserve the generator/source relationship when changing IDS behavior.
+
+## Testing Guidelines
+
+Add coverage in the relevant `matlab.unittest.TestCase` class, using descriptive `test...` method names. Exercise both HDF5 and MDSplus only when the change is backend-specific; HDF5 is sufficient for the standard local path. Run the narrow MATLAB suite while iterating when the built libraries are on the path, then run CTest before opening a PR. Update examples or docs when public MATLAB behavior changes.
+
+`imas_utils_unit_tests` is registered as its own backend-independent CTest suite; it runs even when neither HDF5 nor MDSplus is configured.
+
+## Skipped-path policy
+
+The shared MEX utility library owns the process-global skipped-path record and
+the refusal-band (`-1000..-1099`) tolerance chokepoint. Keep the record
+observable only through `imas_get_skipped_paths` and
+`imas_get_skipped_path_count`. Generated traversal routes only leaf data seams
+and array-of-structures opens through the chokepoint: a tolerated read leaf
+receives its default value, a tolerated read array open becomes empty, a
+tolerated write leaf is omitted, a tolerated write array open skips its
+subtree, and a tolerated delete leaf is omitted. Root read, write, and delete
+generators clear the record on entry; an `ids_put` clears once before its
+internal delete phase so both delete and write skips remain in one record.
+
+## Commit & Pull Request Guidelines
+
+Recent history uses brief, imperative summaries (for example, `Fix warnings in windows`) and conventional prefixes for automation such as `ci:`. Use one focused change per commit; include the affected component when helpful. Start work from the latest `develop` branch, as required by `CONTRIBUTING.md`. PRs should describe the behavior change, link the agreed issue, list validation performed, and include MATLAB output or screenshots when they clarify a user-facing change.
+
+## graphify
+
+This project has a knowledge graph at graphify-out/ with god nodes, community structure, and cross-file relationships.
+
+When the user types `/graphify`, use the installed graphify skill or instructions before doing anything else.
+
+Rules:
+- For codebase questions, first run `graphify query "<question>"` when graphify-out/graph.json exists. Use `graphify path "<A>" "<B>"` for relationships and `graphify explain "<concept>"` for focused concepts. These return a scoped subgraph, usually much smaller than GRAPH_REPORT.md or raw grep output.
+- Dirty graphify-out/ files are expected after hooks or incremental updates; dirty graph files are not a reason to skip graphify. Only skip graphify if the task is about stale or incorrect graph output, or the user explicitly says not to use it.
+- If graphify-out/wiki/index.md exists, use it for broad navigation instead of raw source browsing.
+- Read graphify-out/GRAPH_REPORT.md only for broad architecture review or when query/path/explain do not surface enough context.
+- After modifying code, run `graphify update .` to keep the graph current (AST-only, no API cost).

CLAUDE.md

@@ -0,0 +1,62 @@
+# Repository Guidelines
+
+Any changes should be also reflected in AGENTS.md
+
+## Project Structure & Module Organization
+
+This is the MATLAB high-level interface to IMAS Access Layer. MATLAB entry points live in `matlab/`; their MEX implementations and shared C helpers are in `src/`. XSLT generators at the repository root and in `common/` generate IDS-specific sources from the IMAS Data Dictionary—edit generators rather than generated output where applicable. `tests/` contains MATLAB unit, integration, and performance tests; `examples/` contains runnable MATLAB examples. Sphinx documentation is in `doc/`, and CI and cluster scripts are in `ci/` and `.github/workflows/`.
+
+## Build, Test, and Development Commands
+
+MATLAB and a C/C++ toolchain are required. CMake fetches IMAS Core and the Data Dictionary unless configured for a local development layout.
+
+```bash
+cmake -B build --preset=https -DAL_BACKEND_HDF5=ON -DAL_TESTS=ON -DAL_EXAMPLES=ON
+cmake --build build --parallel
+ctest --test-dir build --output-on-failure
+cmake --install build
+```
+
+The first command configures an HTTPS-based dependency checkout and enables the HDF5 test backend. The test command runs the CTest registrations, which invoke MATLAB's `runtests('imas_unit_tests')`. For documentation, use `ci/build_docs.sh` or configure with `-DAL_HLI_DOCS=ON -DAL_DOCS_ONLY=ON` and build the resulting tree. Do not commit build directories or generated local artifacts.
+
+For multiversion DD conversion, install IMAS-Multiversion-DD-Loader and configure with `-DAL_USE_MULTIVERSION_SHIM=ON` plus its prefix in `CMAKE_PREFIX_PATH` (or `imas-mvdd-loader_DIR`). The MATLAB MEX targets then link the shim while retaining IMAS-Core for headers and runtime loading. CTest sets `IMAS_MVDD_HLI_DD_VERSION` and `IMAS_CORE_LIBRARY`; set them explicitly when launching MATLAB outside CTest. `AL_CORE_RUNTIME_LIBRARY` overrides CTest's Core path. See `doc/doc_common/building_installing.rst` and `docs/SHIM_INTEGRATION_CONTRACT.md`.
+
+## Coding Style & Naming Conventions
+
+Follow the surrounding file's formatting: MATLAB uses two-space indentation, `function` blocks, and lower-case underscore-separated API names such as `ids_get_slice`; C uses four-space indentation and lower-case underscore-separated filenames such as `imas_mex_utils.c`. Keep MATLAB help comments immediately above public functions and use established `IMAS:<component>:<condition>` error identifiers in MEX code. Prefer focused changes; preserve the generator/source relationship when changing IDS behavior.
+
+## Testing Guidelines
+
+Add coverage in the relevant `matlab.unittest.TestCase` class, using descriptive `test...` method names. Exercise both HDF5 and MDSplus only when the change is backend-specific; HDF5 is sufficient for the standard local path. Run the narrow MATLAB suite while iterating when the built libraries are on the path, then run CTest before opening a PR. Update examples or docs when public MATLAB behavior changes.
+
+`imas_utils_unit_tests` is registered as its own backend-independent CTest suite; it runs even when neither HDF5 nor MDSplus is configured.
+
+## Skipped-path policy
+
+The shared MEX utility library owns the process-global skipped-path record and
+the refusal-band (`-1000..-1099`) tolerance chokepoint. Keep the record
+observable only through `imas_get_skipped_paths` and
+`imas_get_skipped_path_count`. Generated traversal routes only leaf data seams
+and array-of-structures opens through the chokepoint: a tolerated read leaf
+receives its default value, a tolerated read array open becomes empty, a
+tolerated write leaf is omitted, a tolerated write array open skips its
+subtree, and a tolerated delete leaf is omitted. Root read, write, and delete
+generators clear the record on entry; an `ids_put` clears once before its
+internal delete phase so both delete and write skips remain in one record.
+
+## Commit & Pull Request Guidelines
+
+Recent history uses brief, imperative summaries (for example, `Fix warnings in windows`) and conventional prefixes for automation such as `ci:`. Use one focused change per commit; include the affected component when helpful. Start work from the latest `develop` branch, as required by `CONTRIBUTING.md`. PRs should describe the behavior change, link the agreed issue, list validation performed, and include MATLAB output or screenshots when they clarify a user-facing change.
+
+## graphify
+
+This project has a knowledge graph at graphify-out/ with god nodes, community structure, and cross-file relationships.
+
+When the user types `/graphify`, use the installed graphify skill or instructions before doing anything else.
+
+Rules:
+- For codebase questions, first run `graphify query "<question>"` when graphify-out/graph.json exists. Use `graphify path "<A>" "<B>"` for relationships and `graphify explain "<concept>"` for focused concepts. These return a scoped subgraph, usually much smaller than GRAPH_REPORT.md or raw grep output.
+- Dirty graphify-out/ files are expected after hooks or incremental updates; dirty graph files are not a reason to skip graphify. Only skip graphify if the task is about stale or incorrect graph output, or the user explicitly says not to use it.
+- If graphify-out/wiki/index.md exists, use it for broad navigation instead of raw source browsing.
+- Read graphify-out/GRAPH_REPORT.md only for broad architecture review or when query/path/explain do not surface enough context.
+- After modifying code, run `graphify update .` to keep the graph current (AST-only, no API cost).

CMakeLists.txt

@@ -26,6 +26,7 @@ endif()
 # Use full path since CMAKE_MODULE_PATH hasn't been configured yet
 include(${CMAKE_CURRENT_SOURCE_DIR}/common/cmake/ALLocalPaths.cmake)
 option( AL_DOWNLOAD_DEPENDENCIES "Automatically download assets from the AL git repository" ON )
+option( AL_USE_MULTIVERSION_SHIM "Route MATLAB Access Layer calls through IMAS-Multiversion-DD-Loader" OFF )
 set( AL_CORE_GIT_REPOSITORY "https://github.com/iterorganization/IMAS-Core.git" CACHE STRING "Git repository of AL-core" )
 set( AL_CORE_VERSION "main" CACHE STRING "Git commit/tag/branch of AL-core" )
 
@@ -110,6 +111,31 @@ include( ALBuildDataDictionary )
 # AL core, plugins and documentation
 include( ALCore )
 
+# Keep Core available for its headers and as the library the shim opens at run
+# time, but do not put it on a MEX link line: its mirrored C symbols would
+# bypass conversion. Both al-mex and each MEX target use this one selection.
+set( AL_MATLAB_CORE_TARGET al )
+set( AL_MATLAB_SHIM_TEST_ENVIRONMENT "" )
+if( AL_USE_MULTIVERSION_SHIM AND NOT AL_DOCS_ONLY )
+  find_package( imas-mvdd-loader REQUIRED CONFIG )
+  set( AL_MATLAB_CORE_TARGET imas-mvdd-loader::imas-mvdd-loader )
+  set( AL_MATLAB_SHIM_TEST_ENVIRONMENT "IMAS_MVDD_HLI_DD_VERSION=${DD_VERSION}" )
+
+  set( AL_CORE_RUNTIME_LIBRARY "" CACHE FILEPATH
+    "AL core shared library opened by the shim at run time; empty uses the Core built here" )
+  if( AL_CORE_RUNTIME_LIBRARY )
+    if( NOT EXISTS "${AL_CORE_RUNTIME_LIBRARY}" )
+      message( FATAL_ERROR "AL_CORE_RUNTIME_LIBRARY does not exist: ${AL_CORE_RUNTIME_LIBRARY}" )
+    endif()
+    list( APPEND AL_MATLAB_SHIM_TEST_ENVIRONMENT
+      "IMAS_CORE_LIBRARY=${AL_CORE_RUNTIME_LIBRARY}" )
+  elseif( AL_DOWNLOAD_DEPENDENCIES OR AL_DEVELOPMENT_LAYOUT )
+    list( APPEND AL_MATLAB_SHIM_TEST_ENVIRONMENT
+      "IMAS_CORE_LIBRARY=$<TARGET_FILE:al>" )
+  endif()
+  message( STATUS "MATLAB Access Layer calls use the multiversion shim (${imas-mvdd-loader_DIR})" )
+endif()
+
 # Stop processing when only building documentation
 if( AL_DOCS_ONLY )
   return()
@@ -136,7 +162,7 @@ if(WIN32)
   # This should be built as a shared library (not a MEX file) with name libal-mex.so/.dll
   # It's a utility library that MEX files link to, so use add_library instead of matlab_add_mex
   add_library( al-mex SHARED ${MEX_UTIL_SOURCES} )
-  target_link_libraries( al-mex PUBLIC al Matlab::mex Matlab::mx )
+  target_link_libraries( al-mex PUBLIC ${AL_MATLAB_CORE_TARGET} Matlab::mex Matlab::mx )
   target_include_directories( al-mex PRIVATE src )
   target_include_directories( al-mex PUBLIC ${Matlab_INCLUDE_DIRS} )
 
@@ -154,9 +180,18 @@ if(WIN32)
   set_target_properties(al-mex PROPERTIES LINKER_LANGUAGE C)
 else()
   # This should be built as a shared library with name libal-mex.so
-  matlab_add_mex( NAME al-mex SRC ${MEX_UTIL_SOURCES} SHARED LINK_TO al )
+  matlab_add_mex( NAME al-mex SRC ${MEX_UTIL_SOURCES} SHARED LINK_TO ${AL_MATLAB_CORE_TARGET} )
   target_include_directories( al-mex PRIVATE src )
 endif()
+if( AL_USE_MULTIVERSION_SHIM )
+  # The shim publishes its own header, not the IMAS-Core C ABI headers used by
+  # these sources. Export Core's include paths without its link interface.
+  target_include_directories( al-mex PUBLIC
+    "$<BUILD_INTERFACE:$<TARGET_PROPERTY:al,INTERFACE_INCLUDE_DIRECTORIES>>" )
+  if( AL_DOWNLOAD_DEPENDENCIES OR AL_DEVELOPMENT_LAYOUT )
+    add_dependencies( al-mex al )
+  endif()
+endif()
 
 set_target_properties( al-mex PROPERTIES
   PREFIX "lib"
@@ -235,7 +270,7 @@ if(WIN32)
   macro( ADD_IMAS_MEX NAME SOURCES )
     # Let matlab_add_mex handle the suffix automatically
     # Or explicitly set .mexw64 if needed:
-    matlab_add_mex( NAME "mex-${NAME}" SRC ${SOURCES} LINK_TO al al-mex OUTPUT_NAME "${NAME}" )
+    matlab_add_mex( NAME "mex-${NAME}" SRC ${SOURCES} LINK_TO ${AL_MATLAB_CORE_TARGET} al-mex OUTPUT_NAME "${NAME}" )
     if(MATLAB_VERSION VERSION_GREATER_EQUAL "9.4")  # R2018a+
       set_target_properties("mex-${NAME}" PROPERTIES SUFFIX ".mexw64")
     endif()
@@ -253,7 +288,7 @@ if(WIN32)
 else()
   # Macro to add a mex target
   macro( ADD_IMAS_MEX NAME SOURCES )
-    matlab_add_mex( NAME "mex-${NAME}" SRC ${SOURCES} LINK_TO al al-mex OUTPUT_NAME "${NAME}" )
+    matlab_add_mex( NAME "mex-${NAME}" SRC ${SOURCES} LINK_TO ${AL_MATLAB_CORE_TARGET} al-mex OUTPUT_NAME "${NAME}" )
     target_include_directories( "mex-${NAME}" PRIVATE src )
     # Ensure all source-generation targets finish before this MEX target compiles.
     # Without this, parallel builds (cmake --build -j N) can race: the MEX target
@@ -328,11 +363,14 @@ set( TARGETS
   imas_deserialize
   imas_get_backendID
   imas_get_mex_params
+  imas_get_skipped_path_count
+  imas_get_skipped_paths
   imas_open
   imas_open_env
   imas_open_env_backend
   imas_serialize
   imas_set_mex_params
+  imas_test_inject_skipped_path
   imas_build_uri_from_legacy_parameters
   imas_list_all_occurrences
   imas_partial_get

CONTEXT.md

@@ -0,0 +1,38 @@
+# IMAS-MATLAB
+
+The MATLAB High Level Interface of the IMAS Access Layer. Most IDS traversal
+sources are generated at build time by XSLT stylesheets, so update the
+stylesheets rather than generated sources.
+
+Shim-side vocabulary (shim, seam, occurrence, DD-version stamp, stored DD
+version, loss log, rule, fidelity verdict) is owned by
+`IMAS-Multiversion-DD-Loader/CONTEXT.md` and is used here unchanged. The terms
+below are the ones this repository owns.
+
+## Language
+
+**Refusal band**:
+The status codes `-1000..-1099`, reserved for a shim that declines to serve a
+path. Disjoint from IMAS-Core's own `-1..-4`.
+_Avoid_: error range, MVDD codes
+
+**Tolerated refusal**:
+A refusal that the generated traversal absorbs at a single field and carries
+on past, instead of ending the operation. Only a leaf data seam and an
+array-of-structures open may tolerate one.
+_Avoid_: ignored error, swallowed refusal, soft failure
+
+**Skipped path**:
+One field the traversal left unset because of a tolerated refusal, recorded
+with the path, the status code and the refusal message.
+_Avoid_: missing field, failed path, skipped node
+
+**Partial read**:
+The outcome of a read that completed after at least one tolerated refusal. A
+positive status, because the operation succeeded and the IDS is usable.
+_Avoid_: failed read, incomplete read, degraded read
+
+**Partial put**:
+The write-side counterpart of a partial read. Covers both a refused write and
+a refused delete, which reach the caller through the same outcome.
+_Avoid_: failed put, partial write

common/cmake/ALExampleUtilities.cmake

@@ -29,7 +29,8 @@ function( set_al_example_properties TEST DISABLED USE_PLUGINS EXTRA_ENVIRONMENT
   else()
     set( P_ENV ${EXAMPLE_ENVIRONMENT_WITHOUT_PLUGINS} )
   endif()
-  set_tests_properties( ${TEST} PROPERTIES ENVIRONMENT "${P_ENV};${EXTRA_ENVIRONMENT}" )
+  set_tests_properties( ${TEST} PROPERTIES
+    ENVIRONMENT "${P_ENV};${EXTRA_ENVIRONMENT};${AL_MATLAB_SHIM_TEST_ENVIRONMENT}" )
 
   # Set fixtures: put/put_slice must run before get/get_slice
   string( TOLOWER ${TEST} TEST_LOWER )
@@ -52,4 +53,3 @@ function( error_on_missing_tests SOURCE_EXTENSION TESTS )
     endif()
   endforeach()
 endfunction()
-

delete.xsl

@@ -32,6 +32,7 @@
     <xsl:otherwise>
       fieldPath = "<xsl:value-of select="@path"/>";
       status = al_delete_data(ctx, fieldPath);
+      if (status.code &lt; 0) tolerateRefusal(&amp;status, IMAS_MEX_DELETE_OPERATION, fieldPath);
       /* Error handling */
       if (status.code &lt; 0) {
       addIdsPathInfoToErrMsg("\n ... in field <xsl:value-of select="@path"/>",0);

doc/api_ids.rst

@@ -25,3 +25,7 @@ IDS API
 .. mat:autofunction:: ids_isdefined()
 
 .. mat:autofunction:: ids_validate()
+
+.. mat:autofunction:: imas_get_skipped_paths()
+
+.. mat:autofunction:: imas_get_skipped_path_count()