endianness assumptions (perhaps in code, perhaps in tests)

[llimeht]

Version: 0.12.2

The following tests currently fail on the s390x architecture which distinguishes itself as probably the only big endian architecture that the tests are run against:

131s test/sasdataloader/utest_sasdataload.py::test_load_file[test_case10] FAILED [ 84%]
131s test/sasdataloader/utest_sasdataload.py::test_load_file[test_case11] FAILED [ 84%]
132s test/sasdataloader/utest_sasdataload.py::test_load_file[test_case12] FAILED [ 84%]
132s test/sasdataloader/utest_sasdataload.py::test_load_file[test_case13] FAILED [ 84%]
133s test/sasdataloader/utest_sasdataload.py::test_load_file[test_case14] FAILED [ 84%]
133s test/sasdataloader/utest_sasdataload.py::test_load_file[test_case15] FAILED [ 84%]
133s test/sasdataloader/utest_sasdataload.py::test_load_file[test_case16] FAILED [ 84%]
133s test/sasdataloader/utest_sasdataload.py::test_load_file[test_case17] FAILED [ 84%]
133s test/sasdataloader/utest_sasdataload.py::test_load_file[test_case18] FAILED [ 84%]
133s test/sasdataloader/utest_sasdataload.py::test_load_file[test_case19] FAILED [ 84%]
133s test/sasdataloader/utest_sasdataload.py::test_load_file[test_case20] FAILED [ 84%]
133s test/sasdataloader/utest_sasdataload.py::test_load_file[test_case21] FAILED [ 84%]
133s test/sasdataloader/utest_sasdataload.py::test_load_file[test_case22] FAILED [ 85%]

These are all the XmlTestCase tests for test_file_load.

It's not clear to me at this stage whether this is the fault of the xml reader that is incorrectly reading the data, the fault of the code reading the comparison data, or the fault of the comparison test itself. (I have a feeling it's the last of those but I don't know what is in these base64-encoded blobs)

pytest does automagically diff the items for us but it's not that readable as you can imagine. The only difference from test_case10 is:

135s E             Differing items:
135s E             {'79680main_1D_2.2_10.0': {'name': '79680main_1D_2.2_10.0', 'data_contents': {'I': {'value': {'type': 'ndarray', 'enco...dowfactor', 'Qmean', 'dQl', 'dQw'], 'expected_orders': [['Q', 'I', 'dI'], ['Q', 'dQ', 'I', 'dI']]}, 'mask': None, ...}} != {'79680main_1D_2.2_10.0': {'name': '79680main_1D_2.2_10.0', 'data_contents': {'Q': {'value': {'type': 'ndarray', 'enco...dowfactor', 'Qmean', 'dQl', 'dQw'], 'expected_orders': [['Q', 'I', 'dI'], ['Q', 'dQ', 'I', 'dI']]}, 'mask': None, ...}}
135s E             
135s E             Full diff:
135s E               {
135s E                   '79680main_1D_2.2_10.0': {
135s E                       'name': '79680main_1D_2.2_10.0',
135s E                       'data_contents': {
135s E             +             'I': {
135s E             +                 'value': {
135s E             +                     'type': 'ndarray',
135s E             +                     'encoding': 'base64',
135s E             +                     'contents': 'QFVVVMmF8G9AUiKMFUyYX0BPrdl/YrauQEyML4N7SiNASYHHeaa1C0BHPL+xW1c/QETcC3gDRtxAQrXcxj8UEkBA3hsImgJ1QD6beANG3F1AO7scQyylekA5QgW8AaNuQDbFbVz6rNpANIhE0BOpKkAzEwvg3tKJQDFIf8uSOipAL3rULDye7UAtHp4bCJoCQCsuYx+KCQNAKVdl/YraukAnDKV6eGwiQCVq9PDYRNBAJEmF8G9pREAi/Sncclw+QCGdkWhysCFAIKrVe8f3e0AfKpFTefqYQB0JPIn0CihAHDEKE384xUAaIeeWfK6nQBj7L+xW1dBAF8Qd0aIeo0AW1SflIVdpQBT6W5Yoy9FAFFTmW+oLokAThoCdSVGDQBJwC8vmHQBAEd0+1SflIUARTIYFaB7NQBCQ1aW5Yo1ADuuaF23az0AN8Rcu8K5TQA12ZRbbDdhAC8JgLJCBw0AK+sHSnccmQAoqebutwJhACRDqnm7rcEAIHGff4yoGQAcCnxaxHG1ABs2uPmxMWUAF0uxrzoU0QATlf7aZhKFABLYraufVZ0ADrdBSk0rLQAMEkSmIj4ZAArrRjSXt0EABvVZs9B8hQAEC/XXiBGxAAOavzOHFgkAAmhIvrWy1QABIqbz9S/E//srVe8f3ez/+id8Rcu8LP/6ZrpJPIn0//N1jiGWUrz/7i+tbLU1AP/rAeJYT0xw/+iAH3UQTVT/6MaS9ugpSP/lhE0BOpKk/+Pq3VkMCtD/39lVcUucuP/icy31BdD8/95QcghbGFT/2fj0cwQDnP/Z6guh9LHw/9YrQPZqVQj/1KHfuTibVP/Vr7fpD/l0/9Ly+Yc/+sD/049HMEA5rP/K+V1Oj7AM/8vhsImgJ1T/zB/I8yN4rP/K6kqMFUyY/8V8Rcu8K5T/w/keZG8VYP/D9a2WpqAU/8G+yquKXOT/v2Gyon8baP/BpOerdWQw/72WdEsrd3z/wSv9tMwlCP+76sQumJnA/7hnanJkoWz/w7ALy+YdAP+5lWfbsWwg/7Ijv/io86j/sCOFQEZBLP+3TxXnyNGU/7bArQPZqVT/sDdgv114gP+nkWhysCDE/7h5TqB3A2z/uRxDLKV6eP+uEGJN0vGo/7rRKHwgDBD/obFCLMs6JP+kii7Ciyps/6jNIK+i8Fz/ltLlFMIu5P+hdu5z5oGo/6Dn3cpLEkz/jOGj9GZuyP+XwuM+/xlQ/6ZLuhK15Sj/nmdAgPmPpP+bs3yZrpJQ/5w2fkFOfuj/jVHOKO1fFP+TFNL127nQ/5mwPiDM/yD/ixvm5lOGkP+iBvrGBFuw/6obn5i3G4z/julwcYIjXP+DVFQVKwpw/4GicoYvWYj/jVYQrc0tRP+ej9n9Nvfk/3Z6QeV9nbz/eouwosqaxP+bIkJKJ2uA/3kqc3EQ5FT/hLZrYXfqHP+UkE9t/Fzc/2itzS1E3KgAAAAAAAAAA',
135s E             +                     'dtype': '>f8',
135s E             +                     'shape': [
135s E             +                         138,
135s E             +                     ],
135s E             +                 },
135s E             +                 'units': 'cm^-1',
135s E             +             },
135s E                           'Q': {
135s E                               'value': {
135s E                                   'type': 'ndarray',
135s E                                   'encoding': 'base64',
135s E             -                     'contents': 'O99PjZdugj+6SQwCK4eGPzm0yHa+n4o/uB6F61G4jj+cxCCwcmiRP9v5fmq8dJM/Gy/dJAaBlT9aZDvfT42XP5qZmZmZmZk/2c73U+Olmz8ZBFYOLbKdP1g5tMh2vp8/TDeJQWDloD/sUbgeheuhP4ts5/up8aI/K4cW2c73oz/LoUW28/2kP2q8dJMYBKY/CtejcD0Kpz+q8dJNYhCoP0oMAiuHFqk/6SYxCKwcqj+JQWDl0CKrPylcj8L1KKw/yXa+nxovrT9oke18PzWuPwisHFpkO68/VOOlm8QgsD+kcD0K16OwP/T91HjpJrE/RIts5/upsT+TGARWDi2yP+Olm8QgsLI/MzMzMzMzsz+DwMqhRbazP9NNYhBYObQ/I9v5fmq8tD9zaJHtfD+1P8P1KFyPwrU/EoPAyqFFtj9iEFg5tMi2P7Kd76fGS7c/AiuHFtnOtz9SuB6F61G4P6JFtvP91Lg/8tJNYhBYuT9CYOXQItu5P5HtfD81Xro/4XoUrkfhuj8xCKwcWmS7P4GVQ4ts57s/0SLb+X5qvD8hsHJoke28P3E9CtejcL0/wcqhRbbzvT8QWDm0yHa+P2Dl0CLb+b4/sHJoke18vz8AAAAAAADAP6jGSzeJQcA/UI2XbhKDwD/4U+Olm8TAP6AaL90kBsE/SOF6FK5HwT/wp8ZLN4nBP5huEoPAysE/PzVeukkMwj/n+6nx0k3CP4/C9Shcj8I/N4lBYOXQwj/fT42XbhLDP4cW2c73U8M/L90kBoGVwz/Xo3A9CtfDP39qvHSTGMQ/JzEIrBxaxD/P91PjpZvEP3e+nxov3cQ/H4XrUbgexT/HSzeJQWDFP28Sg8DKocU/F9nO91PjxT++nxov3STGP2ZmZmZmZsY/Di2yne+nxj+28/3UeOnGP166SQwCK8c/BoGVQ4tsxz+uR+F6FK7HP1YOLbKd78c//tR46SYxyD+mm8QgsHLIP05iEFg5tMg/9ihcj8L1yD+e76fGSzfJP0a28/3UeMk/7nw/NV66yT+WQ4ts5/vJPz0K16NwPco/5dAi2/l+yj+Nl24Sg8DKPzVeukkMAss/3SQGgZVDyz+F61G4HoXLPy2yne+nxss/1XjpJjEIzD99PzVeuknMPyUGgZVDi8w/zczMzMzMzD91kxgEVg7NPx1aZDvfT80/xSCwcmiRzT9t5/up8dLNPxSuR+F6FM4/vHSTGARWzj9kO99PjZfOPwwCK4cW2c4/tMh2vp8azz9cj8L1KFzPPwRWDi2ync8/rBxaZDvfzz+q8dJNYhDQP/7UeOkmMdA/UrgehetR0D+mm8QgsHLQP/p+arx0k9A/TmIQWDm00D+iRbbz/dTQP/YoXI/C9dA/SgwCK4cW0T+e76fGSzfRP/LSTWIQWNE/Rrbz/dR40T+amZmZmZnRP+58PzVeutE/QmDl0CLb0T+WQ4ts5/vRP+kmMQisHNI/',
135s E             +                     'contents': 'P4Jul41P3zs/hocrAgxJuj+Kn752yLQ5P464UeuFHrg/kWhysCDEnD+TdLxqfvnbP5WBBiTdLxs/l41P3ztkWj+ZmZmZmZmaP5ul41P3ztk/nbItDlYEGT+fvnbItDlYP6DlYEGJN0w/oeuFHrhR7D+i8an752yLP6P3ztkWhys/pP3ztkWhyz+mBBiTdLxqP6cKPXCj1wo/qBBiTdLxqj+pFocrAgxKP6ocrAgxJuk/qyLQ5WBBiT+sKPXCj1wpP60vGp++dsk/rjU/fO2RaD+vO2RaHKwIP7AgxJul41Q/sKPXCj1wpD+xJul41P30P7Gp++dsi0Q/si0OVgQYkz+ysCDEm6XjP7MzMzMzMzM/s7ZFocrAgz+0OVgQYk3TP7S8an752yM/tT987ZFocz+1wo9cKPXDP7ZFocrAgxI/tsi0OVgQYj+3S8an752yP7fO2RaHKwI/uFHrhR64Uj+41P3ztkWiP7lYEGJN0vI/udsi0OVgQj+6XjU/fO2RP7rhR64UeuE/u2RaHKwIMT+752yLQ5WBP7xqfvnbItE/vO2RaHKwIT+9cKPXCj1xP73ztkWhysE/vnbItDlYED+++dsi0OVgP7987ZFocrA/wAAAAAAAAD/AQYk3S8aoP8CDEm6XjVA/wMSbpeNT+D/BBiTdLxqgP8FHrhR64Ug/wYk3S8an8D/BysCDEm6YP8IMSbpeNT8/wk3S8an75z/Cj1wo9cKPP8LQ5WBBiTc/wxJul41P3z/DU/fO2RaHP8OVgQYk3S8/w9cKPXCj1z/EGJN0vGp/P8RaHKwIMSc/xJul41P3zz/E3S8an753P8UeuFHrhR8/xWBBiTdLxz/FocrAgxJvP8XjU/fO2Rc/xiTdLxqfvj/GZmZmZmZmP8an752yLQ4/xul41P3ztj/HKwIMSbpeP8dsi0OVgQY/x64UeuFHrj/H752yLQ5WP8gxJul41P4/yHKwIMSbpj/ItDlYEGJOP8j1wo9cKPY/yTdLxqfvnj/JeNT987ZGP8m6XjU/fO4/yfvnbItDlj/KPXCj1wo9P8p++dsi0OU/ysCDEm6XjT/LAgxJul41P8tDlYEGJN0/y4UeuFHrhT/LxqfvnbItP8wIMSbpeNU/zEm6XjU/fT/Mi0OVgQYlP8zMzMzMzM0/zQ5WBBiTdT/NT987ZFodP82RaHKwIMU/zdLxqfvnbT/OFHrhR64UP85WBBiTdLw/zpeNT987ZD/O2RaHKwIMP88an752yLQ/z1wo9cKPXD/PnbItDlYEP8/fO2RaHKw/0BBiTdLxqj/QMSbpeNT+P9BR64UeuFI/0HKwIMSbpj/Qk3S8an76P9C0OVgQYk4/0NT987ZFoj/Q9cKPXCj2P9EWhysCDEo/0TdLxqfvnj/RWBBiTdLyP9F41P3ztkY/0ZmZmZmZmj/Rul41P3zuP9HbItDlYEI/0fvnbItDlj/SHKwIMSbp',
135s E             -                     'dtype': '<f8',
135s E             ?                               ^
135s E             +                     'dtype': '>f8',
135s E             ?                               ^
135s E                                   'shape': [
135s E                                       138,
135s E                                   ],
135s E                               },
135s E                               'units': 'Ang^-1',
135s E             -             },
135s E             -             'I': {
135s E             -                 'value': {
135s E             -                     'type': 'ndarray',
135s E             -                     'encoding': 'base64',
135s E             -                     'contents': 'b/CFyVRVVUBfmEwVjCJSQK62Yn/ZrU9AI0p7gy+MTEALtaZ5x4FJQD9XW7G/PEdA3EYDeAvcREASFD/G3LVCQHUCmggb3kBAXdxGA3ibPkB6pSxDHLs7QG6jAbwFQjlA2qz6XG3FNkAqqRPQRIg0QInS3uALEzNAKjqSy39IMUDtnjws1HovQAKaCBueHi1AAwmKH2MuK0C62or9ZVcpQCJseHqlDCdA0ETY8PRqJUBEaW/whUkkQD5cctwp/SJAIbByaJGdIUB798d71aogQJj6eVORKh9AKAr0iTwJHUDFOH8TCjEcQKeufJbnIRpA0NVW7C/7GECjHqLRHcQXQGlXIeUn1RZA0csollv6FECiC+pb5lQUQINRSZ2AhhNAAB3mywtwEkAh5SfVPt0RQM0eaAWGTBFAjWK5pdWQEEDP2m0XmusOQFOu8C4X8Q1A2A3bFmV2DUDDgZAsYMILQCbHndLB+gpAmMCtu3kqCkBw626e6hAJQAYq499nHAhAbRyxFp8CB0BZTGw+rs0GQDSFzmvs0gVAoYSZtn/lBEBn1edqK7YEQMtKk1LQrQNAho+IKZEEA0DQ7SWN0boCQCEf9GxWvQFAbATidf0CAUCCxeHMr+YAQLVsrS8SmgBA8Uv9vKlIAEB798d71cr+PwvvchHfif4/fSJPkq6Z/j+vlGWIY938P0BNLVvri/s/HNMTlnjA+j9VE0TdByD6P1IKur2kMfo/qaROQBNh+T+0AkNWt/r4Py7nUlxV9vc/P3RBfcuc+D8VxhaCHJT3P+cAwRw9fvY/fCx96IJ69j9ClZo90Ir1P9UmTu53KPU/Xf5D+u1r9T+w/s9hvrz0P2sOEMzR4/Q/A+yjU1e+8j/VCWgibPjyPyveyDzyB/M/JlMFo5K68j/lCu9yEV/xP1jFG5lH/vA/BaipZWv98D85l+Kqsm/wP9rGn6hs2O8/DFnd6jlp8D/f3coSnWXvP0IJM23/SvA/cCamC7H67j9bKJmc2hnuP0CH+fIC7PA/CFvs9lll7j/qPCr+74jsP0uQEVDhCOw/ZTTyecXT7T9VavZAK7DtPyBe1y/YDew/MQisHFrk6T/bwB2oUx7uP55eKcsQR+4/arx0kxiE6z8EAwgfSrTuP4nOMotQbOg/m8qisIsi6T8XvOgrSDPqP7mLMEW5tOU/aqD5nLtd6D+TxJJy9znoP7KbGf1oOOM/VMa/z7jw5T9Kea2E7pLpP+ljPiDQmec/lKRrJt/s5j+6n1OQnw3nP8VXO4pzVOM/dO52vTTF5D/IPzOID2zmP6ThlLn5xuI/7BaBsb6B6D/jxi3m54bqP9eIYBxcuuM/nMJKBRXV4D9i1ouhnGjgP1FLcyuEVeM/+b1Nf/aj5z9vZ195kJ7dP7Gmsijsot4/4NqJkpDI5j8VOUTcnEreP4f6XdiaLeE/Nxd/2xMk5T8qN1FLcyvaPwAAAAAAAAAA',
135s E             -                     'dtype': '<f8',
135s E             -                     'shape': [
135s E             -                         138,
135s E             -                     ],
135s E             -                 },
135s E             -                 'units': 'cm^-1',
135s E                           },
135s E                       },

There are similar differences for the other test cases.

The full log may be available on ci.debian.net (although techbros externalising their costs on the world with scrapers for their slop machines have meant that it might not be available so easily):

https://ci.debian.net/data/autopkgtest/testing/s390x/s/sasdata/74100932/log.gz
https://ci.debian.net/data/autopkgtest/testing/s390x/s/sasdata/

[pkienzle]

The target values are stored in test/sasdataloader/json as IEEE784 little endian base 64 encoding.

AI says S390 numpy now uses IEEE784 so it is an endian only problem.

You could have the tests pass by encoding stored values as little endian regardless of type. Update here:

sasdata/sasdata/metadata.py

Lines 600 to 607 in d7d9411

	case ndarray():
	return {
	"type": "ndarray",
	"encoding": "base64",
	"contents": base64.b64encode(obj.tobytes()).decode("utf-8"),
	"dtype": obj.dtype.str,
	"shape": obj.shape,
	}

Regardless, arrays should be converted to native endian when decoding. Update here:

sasdata/sasdata/metadata.py

Lines 513 to 520 in d7d9411

	case {
	"type": "ndarray",
	"dtype": dtype,
	"encoding": "base64",
	"contents": contents,
	"shape": shape,
	}:
	return np.frombuffer(base64.b64decode(contents), dtype=dtype).reshape(shape)

[pkienzle]

On encoding use little endian (<):

obj = np.asarray(dtype=obj.dtype.newbyteorder('<'))
return { ... }

On decoding use native byte order (=):

obj = np.frombuffer(...).reshape(shape)
obj = np.asarray(dtype=obj.dtype.newbyteorder('='))
return obj

The np.asarray() call does nothing if the dtype is already correct. The alternative obj.astype(...) always makes a copy.

I tested this for numbers of various sizes (1, 2, 4, 8) and kinds (i, u, f, c), for byte strings and for unicode strings (which are stored internally as 4 byte integers). I didn't play with M=Datetime, m=Timedelta or V=Void kinds.

I don't think github CI has any big endian architectures, so someone with access to a big endian machine should make the patch and test it.

[pkienzle]

Here's a test for boolean using sympy.isprime:

In [116]: np.asarray(data:=np.array([isprime(x) for x in range(128)]), dtype=data.dtype.newbyteorder('>')).tobytes()
Out[116]: b'\x00\x00\x01\x01\x00\x01\x00\x01\x00\x00\x00\x01\x00\x01\x00\x00\x00\x01\x00\x01\x00\x00\x00\x01\x00\x00\x00\x00\x00\x01\x00\x01\x00\x00\x00\x00\x00\x01\x00\x00\x00\x01\x00\x01\x00\x00\x00\x01\x00\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00\x01\x00\x01\x00\x00\x00\x00\x00\x01\x00\x00\x00\x01\x00\x01\x00\x00\x00\x00\x00\x01\x00\x00\x00\x01\x00\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x01\x00\x01\x00\x00\x00\x01\x00\x01\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01'

In [117]: np.asarray(data:=np.array([isprime(x) for x in range(128)]), dtype=data.dtype.newbyteorder('<')).tobytes()
Out[117]: b'\x00\x00\x01\x01\x00\x01\x00\x01\x00\x00\x00\x01\x00\x01\x00\x00\x00\x01\x00\x01\x00\x00\x00\x01\x00\x00\x00\x00\x00\x01\x00\x01\x00\x00\x00\x00\x00\x01\x00\x00\x00\x01\x00\x01\x00\x00\x00\x01\x00\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00\x01\x00\x01\x00\x00\x00\x00\x00\x01\x00\x00\x00\x01\x00\x01\x00\x00\x00\x00\x00\x01\x00\x00\x00\x01\x00\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x01\x00\x01\x00\x00\x00\x01\x00\x01\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01'