Repository navigation
unexpected behaviour using on-the-fly conversion #78
Description
Activity
Comparison with al-python is irrelevant for such a case that cross versions (the al-python may not throw error but you will miss many quantities in the obtained IDS, e.g. most
_torfields will be empty, which I'm sure will give you more problem down the line, not mentioning the sign changes inpsiand other related quantities).Here, what we need to understand I guess is how the NBC
>>> equilibrium = DB.get_slice("equilibrium", 0.0, imas.ids_defs.CLOSEST_INTERP) 14:20:12 WARNING On-disk data is stored in DD 4.0.0 which has a different major version than the requested DD version (3.42.0). IMAS-Python will convert the data automatically, but this does not cover allchanges. See https://imas-python.readthedocs.io/en/latest//multi-dd.html#conversion-of-idss-between-dd-versions @db_entry.py:606 Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/work/imas/opt/EasyBuild/software/IMAS-Python/2.0.1-intel-2023b/lib/python3.11/site-packages/imas/db_entry.py", line 423, in get_slice return self._get( ^^^^^^^^^^ File "/work/imas/opt/EasyBuild/software/IMAS-Python/2.0.1-intel-2023b/lib/python3.11/site-packages/imas/db_entry.py", line 622, in _get return self._dbe_impl.get( ^^^^^^^^^^^^^^^^^^^ File "/work/imas/opt/EasyBuild/software/IMAS-Python/2.0.1-intel-2023b/lib/python3.11/site-packages/imas/backends/imas_core/db_entry_al.py", line 302, in get get_children(destination, read_ctx, time_mode, nbc_map) File "/work/imas/opt/EasyBuild/software/IMAS-Python/2.0.1-intel-2023b/lib/python3.11/site-packages/imas/backends/imas_core/db_entry_helpers.py", line 65, in get_children get_children(element, ctx, time_mode, nbc_map) File "/work/imas/opt/EasyBuild/software/IMAS-Python/2.0.1-intel-2023b/lib/python3.11/site-packages/imas/backends/imas_core/db_entry_helpers.py", line 41, in get_children new_path = nbc_map.ctxpath[path] ~~~~~~~~~~~~~~~^^^^^^ KeyError: 'ids_properties/source'Note that in DD 3.42.0
<field name="source" path="ids_properties/source" path_doc="ids_properties/source" documentation="Source of the data (any comment describing the origin of the data : code, path to diagnostic signals, processing method, ...). Superseeded by the new provenance structure." data_type="STR_0D" type="constant" lifecycle_status="obsolescent" lifecycle_version="3.34.0"/>while in DD4 there the node was removed. This looks like a bug in the convert functionality or did I miss a point @maarten-ic @prasad-sawantdesai ? (from https://github.com/iterorganization/IMAS-Python/blob/develop/imas/ids_convert.py#L64 and https://github.com/iterorganization/IMAS-Python/blob/develop/imas/ids_convert.py#L76-L77 have me thinking that such a case of a removed field shall be handled)
This looks like a bug in the convert functionality or did I miss a point
Yes, this looks like a bug with the conversion logic indeed. It's probably related to the DD3to4 conversion rule for
ids_properties/source->ids_properties/provenance(https://github.com/iterorganization/IMAS-Python/blob/develop/imas/ids_convert.py#L392).The question is: is it useful to fix this problem, or should we perhaps disable automatic conversion across DD major versions (instead of the warning that is logged currently)? The
psisign changes, for example, are not covered when doing an automatic conversion.Reacted by Olivier HoenenGood catch I missed this was in the DD3to4 specific rules. Except that this shall NOT have any impact when we are in implicit conversion, no? I need to check with an equivalent removal in an alpha IDS that happened in DD3 to verify that the same bug is not present.
Except that this shall NOT have any impact when we are in implicit conversion, no?
It shouldn't indeed: this is a bug :)
Implicit conversion is now disabled between major versions of the DD (see #82).
The reproduction scenario above now produces an error message, pointing users to the online documentation:
RuntimeError: On-disk data is stored in DD 4.0.0 which has a different major version than the requested DD version (3.42.0). IMAS-Python will not automatically convert this data for you. See the documentation for more details and fixes: https://imas-python.readthedocs.io/en/latest//multi-dd.html#loading-idss-from-a-different-major-version
If I run the following in an environment where 3.42.0 is loaded (using imas-python)
I get a crash with the message
KeyError: 'ids_properties/source'The input data is in 4.0.0 format and the conversion to 3.42.0 seems to be causing the problem.
Changing the get_slice to
does work.
The original code worked with al-python.