Unreadable measurement

[DavidTadres]

Firstly, thank you very much for providing this python package.

One of my recordings done on a Bruker Tensor 27 spectrometer can't be read by opusreader.

The OPUS viewer displays is able to display the measurement (blue). I also show the 'readable' example in red (See below)

When reading the two files I get widely different absorption values:

from brukeropusreader import read_file

unreadable = read_file('C:\\Users\\David\\Desktop\\opus troublshooting\\unreadable.0000')
readable = read_file('C:\\Users\\David\\Desktop\\opus troublshooting\\readable.0006')

"""
unreadable['AB'][0:30]
array([1.68155816e-44, 2.80259693e-42, 1.40129846e-45, 6.27109369e-39,
       1.83670992e-40, 5.88323249e-39, 7.37636505e-39, 1.83670992e-40,
       1.40129846e-45, 8.29149843e-39, 1.83670992e-40, 4.09179152e-43,
       7.74048685e-39, 4.77544860e-39, 7.21432390e+22, 7.14469695e+31,
       2.01962467e-19, 7.43988911e+28, 3.72261848e+27, 6.68376203e+22,
       1.31832032e+25, 2.28019990e-12, 1.10184478e+24, 1.93459115e-19,
       1.57949750e-19, 2.96726393e+26, 2.69185152e+09, 2.82229577e+26,
       6.48017418e-10, 6.37359499e-10])
readable['AB'][0:30]
array([0.00488466, 0.0047653 , 0.00466482, 0.00459077, 0.00451578,
       0.00444223, 0.00436925, 0.00426781, 0.00415329, 0.00407281,
       0.0040019 , 0.00388978, 0.00375876, 0.00363817, 0.00352651,
       0.00342261, 0.00332389, 0.00322775, 0.00312658, 0.00301312,
       0.00290017, 0.00279764, 0.00268841, 0.0025628 , 0.00243541,
       0.00232032, 0.00221839, 0.00211131, 0.00198241, 0.00184446])
"""

The files can be found here: https://drive.google.com/drive/folders/1tTz6-GYi1mNkF85JtXe4k_GY7ffpiY6_?usp=sharing

Note: The files are from two different measurements which were ran consecutively. All measurement during the 'unreadable' measurement are unreadable. All measurements during the 'readable' measurement are readable.

Any idea what the problem could be? I initially thought that the file might be corrupted. However, the OPUS viewer is able to extract the correct information.

[atravert]

I've got the same issue recently. The following fixed it: PR #21

Below is how your files are read with spectrochempy (which uses brukeropusreader), with this fix.

import spectrochempy as scp

D1 = scp.read("unreadable.0000")
D2 = scp.read("readable.0006")
D = scp.stack(D1, D2)
ax = D.plot()
ax.set_ylim(-0.02, 0.05)
scp.show()

[philipp-baumann]

have you guys been thinking of reading the magic bytes with spectral type numbers and offsets in the header? This would probably universally fix such problems as above.

[atravert]

Thank you for the tip, I will look at this !

[philipp-baumann]

I have been thinking of working on that based on some earlier python package. So if you are interested on working jointly on some stuff, just let me know :-)

[atravert]

That would be great. I don't know much about the opus format -- this is why I use brukeropusreader -- , but will be glad to help if I can.

Best,
Arnaud

[philipp-baumann]

Hi Arnaud
You are on the right track with the referenced merge pull request.
We have been working further on this in a new R reader as part of the spectral-cockpit collective.
https://github.com/spectral-cockpit/opusreader2/blob/e503ce3ae51155a77a1f94bae8eff9af1092fae4/R/read_header.R#L33
There is for example atmospheric correction routines done by Bruker software. These algorithms work for water and CO2 bands, and can selectively done. If done, there are both raw reflectance/absorbance spectra and atmospherically corrected ones.
Interestingly, we found a new offset +3 that for each data block in the header (see referenced line above). This is either 0 or 64 when read as unsigned integer. :-) so what does this mean: we strongly believe that 64 stands for "raw" (relatively untreated) measurements, and 0 stands for macro processing (there is lots of macro language etc. reporting).

[philipp-baumann]

We for know call this bit of information in the header additional_type. btw, we get almost all parameters know with some more tweaks: https://github.com/spectral-cockpit/opusreader2/blob/main/R/get_nice_parameter_name.R

[atravert]

Hi Philipp,

Thank you very much for sharing.

On our side we have fixed an error when using brukeropusreader with files containing additional informations on the 'raw' AB spectrum (single channel spectra, interferograms, and acquisition parameters of the sample and reference).

our fix is here:
spectrochempy@c03aee3

The qed/brukerpousreader repository has not been updated since a while now. If this persists, and if others are interested, I could work on a python implementation of opusreader2 :-).

Best regards
Arnaud

[philipp-baumann]

Hi Arnaud,

That is great to hear. With your latest commit we will be able to get the phase spectrum, which we will happily implement soon :-)
I would the idea of a double language implementation, so we can actually jointly work on mapping new block types etc. We have been recently added support for a specific Bruker Tango NIR device, @ThomasKnecht has tweaked another special block to parse ( spectral-cockpit/opusreader2#34 ), plus read_opus() now has an informative error when there is files that fail to be parsed. Typically, this is because we do not (yet) have the full list of header mappings.

It really looks like qed.ai is not actively maintaining the package anymore. Maybe a quick email can resolve? otherwise really like the idea of opusreader2 in python!

Cheers,
Philipp

[m-ad]

Great to see that there is some active development still going on! Do you have any plans on releasing an updated opusreader package on pip or conda in the near future? @philipp-baumann @atravert

[joshduran]

I recently published a new package that can read OPUS files. It is able to read the "unreadable.0000" file (absorbance, reference, and sample spectra as well as standard metadata). However, there are two mysterious blocks in this file that do not have any block type information stored. I'm guessing this has to do with the integral analysis. If you need to extract that data, it will take some new logic to extract those undefined data blocks. Below is the file block directory as read by my package:

========================================================================================================================
                                   c:\Users\duranjm\Desktop\examples\unreadable.0000
                      Version: 920622.0    Directory start: 24    Max Blocks: 40    Num Blocks: 18

========================================================================================================================
                                                       Directory
Block Type                  Size (bytes)   Start (bytes)   Friendly Name
(0, 0, 0, 13, 0, 0)         480            24              Directory
(0, 0, 6, 0, 0, 0)          232            504             Optical Parameters
(0, 0, 4, 0, 0, 0)          116            736             Fourier Transform Parameters
(0, 0, 3, 0, 0, 0)          188            852             Acquisition Parameters
(0, 0, 10, 0, 0, 0)         284            1040            Sample Origin Parameters
(3, 1, 0, 1, 0, 0)          4876           1324            Sample Spectrum
(0, 0, 0, 0, 0, 0)          12             6200
(3, 3, 0, 4, 0, 0)          4876           6312            Absorbance
(0, 0, 0, 0, 0, 0)          112            11188
(0, 2, 2, 0, 0, 0)          552            11300           Reference Instrument Status Parameters
(0, 2, 6, 0, 0, 0)          232            11852           Reference Optical Parameters
(3, 2, 0, 1, 0, 0)          4876           12084           Reference Spectrum
(3, 2, 1, 1, 0, 0)          176            16960           Reference Data Status Parameters Spectrum
(3, 3, 0, 4, 0, 0)          2564           17136           Absorbance
(3, 3, 1, 4, 0, 0)          176            19968           Data Status Parameters Absorbance
(3, 1, 1, 1, 0, 0)          176            20144           Sample Data Status Parameters Spectrum
(0, 0, 2, 0, 0, 0)          552            20320           Instrument Status Parameters
(0, 0, 0, 0, 0, 5)          8040           20872           File Log

The two mysterious blocks have a block type of (0, 0, 0, 0, 0, 0). At first glance at the raw bytes in these undefined blocks, I can see metadata stored in these blocks, but also likely a data array. If you need to extract the info from these blocks as well (and not just the absorption and standard metadata blocks), you can open an issue in my package:

https://github.com/joshduran/brukeropus