Back to the main page.
Bug 2231 - old md4 approach for reading 4D data fails
| Status | CLOSED WONTFIX |
| Reported | 2013-08-01 16:06:00 +0200 |
| Modified | 2014-01-15 14:45:22 +0100 |
| Product: | FieldTrip |
| Component: | fileio |
| Version: | unspecified |
| Hardware: | PC |
| Operating System: | Mac OS |
| Importance: | P3 normal |
| Assigned to: | Jan-Mathijs Schoffelen |
| URL: | |
| Tags: | |
| Depends on: | |
| Blocks: | |
| See also: |
Christian Wienbruch - 2013-08-01 16:06:36 +0200
On 1 Aug 2013, at 15:58, Christian Wienbruch wrote in an email to Robert: I have a problem with the old m4d approach to read data into fieldtrip. It seems since the config file approach from Glasgow (which by the way is a pretty dangerous way to read 4D data because the sequence of channels might be changed by various 4D MSI programs and therefore onyl works reliable for the unprocessed data file) that the old appraoch does not work any longer. The error is produced by the ChannelGain in hdr.orig if you try to read data the old way. Is it possible to add an easy fix like that the ChannelGain is 1 in case old m4d files a read?
Christian Wienbruch - 2013-08-01 16:09:45 +0200
Hi Christian Could you share a (possibly small) dataset with us that would allow us to reproduce the problem? Then we can make a test script (line the ones here http://code.google.com/p/fieldtrip/source/browse/#svn%2Ftrunk%2Ftest) and add it to our testing framework at http://fieldtrip.fcdonders.nl/development/dashboard See http://fieldtrip.fcdonders.nl/faq/how_should_i_send_example_data_to_the_developers for some possible ways of sending us the data. thanks Robert
Robert Oostenveld - 2013-08-01 16:11:35 +0200
(In reply to comment #1) Hmm, I just replied while still impersonating Christian rather than as myself. Consequently the email was sent to the wrong people. @Christian, please check the previous comment on the bugzilla server.
Christian Wienbruch - 2013-08-02 16:18:09 +0200
Created attachment 499 Files concerning BUG 2231 Tried reading these files with c,rfhp0.1Hz,o,ccfbp10-40Hz-678-2,cag,c,n,fbp10-40Hz-678-2,s.m4d >> hdr = ft_read_header('c,rfhp0.1Hz,o,ccfbp10-40Hz-678-2,cag,c,n,fbp10-40Hz-678-2,s.m4d') Error using read_bti_m4d (line 128) unknown gradiometer design Error in ft_read_header (line 211) orig = read_bti_m4d(filename); Files *.refloc and *.eeglog are obsolete Thanks for your help Christian
Jan-Mathijs Schoffelen - 2013-08-03 15:13:50 +0200
Bug confirmed. It's not a problem that is related to the preferred way of reading the data from the raw data file + config (through read_4d_hdr). It rather seems that the m4d file has a format that read_bti_m4d cannot deal with. In short, the empty Eeg_Position_Information block causes the crash. I will fix this. Interesting that this has not occurred before; the function read_bti_m4d has not essentially changed since it was first written, I guess that this was around 2007. NOTE: it seems that the gradiometer references are not correctly dealt with as it is right now (in the grad-field). In FieldTrip the two coils belonging to a single gradiometer are grouped, reading the positions with read_bti_m4d seems to treat each coil as a single 'channel'. The hdr.label seems to be correct, though. In general I would eventually like to push everybody into reading the header information directly from the raw-files. If you have some inside information with respect to the config/datafile discrepancy, which can make this way of dealing with the data more robust, please don't hesitate to share this with us (In Glasgow, Gavin and I more or less reverse engineered most of the stuff, with the help of Eugene Kronberg's code). I'd prefer developing the code along these lines, because it does not need intermediate conversion of the data.
Jan-Mathijs Schoffelen - 2013-08-03 15:28:16 +0200
I did not see an error related to the ChannelGain (reported in your description). Is this an issue related to the error described in comment 3?
Jan-Mathijs Schoffelen - 2013-08-03 15:30:23 +0200
svn commit -m "bugfix - 2231: addressed issue related to reading m4d files" fileio/private/read_bti_m4d.m test/test_bug2231.m Sending fileio/private/read_bti_m4d.m Adding test/test_bug2231.m Transmitting file data .. Committed revision 8378.
Christian Wienbruch - 2013-08-03 22:35:18 +0200
Thanks for your quick look into the problem. Just a few comments. Reading the run config file works for the first processed data file (pdf), cause that is related to the acquisition setup. As various 4D MSI software tools do nasty things like changing the channel sequencce or creating new channels, the run config won't help you much reading these files. Therefore it is dangerous to use it with all pdfs. The true header information is written in a trailer at the end of each pdf. This is done by multiple 4D libraries, depending on what is written in the pdf. Unfortunately there is no easy way to read that trailer. There is is no single program, that can read it. Since 1991 I work around this trailer cause I have no information about another way to do it. So the only safe way to read all kind of 4D data is to generate an additional file like the m4d file that contains all the information needed. This is designed to be an extendable format. So if I need some more information I simply add a key followd by the information I want. That's what York did to get the data in their database. And this open format is also related to the problem with the empty EEG position. I think a added thie EEG channel position block after you've implemented that into fieldtrip. People in Konstanz did not use this possibility until recent needs mad them to come back to it. Therefor the problem has not been adressed before. To check on the described bug you can delete the piece of information from the m4d file which is related to the EEG positions and you will find the problem with the channel gains to show up, which I believe is related to what I described earlier. Sorry that I did not mention this earlier. Thanks for your continuous help
Jan-Mathijs Schoffelen - 2013-08-03 23:03:02 +0200
Hi Christian, As it stands read_4d_hdr uses a combination of the trailer of the data file, and stuff from the config file. I have always found this a bit confusing, but somehow never got it to work using information from the trailer alone. I had noticed the channel order issue on several occasions, and will investigate the code once more to see how 'unrobust' the present code is. Do you by any chance have any raw data files to share in order to test it on? How can I check for the channel gain issue? Wouldn't I need the raw data as well? (so not only the converted pdf? Cheers, JM
Christian Wienbruch - 2013-08-06 14:17:29 +0200
(In reply to comment #8) The trailer keeps information about the data format can be ( short int long int float double ) This is of course found in the m4D file. To give you an example. The acquired pdf is written to disk in short ints (16 bit) and converted to amplitude using channel gain and scale values. As soon as you process the data it will usually be stored as float (32 bit) but in the libraries there are also possibilities for long int (32 bit ) and double (64 bit). Also found in the trailer is the sequence of channels, channel names and labels and so on. This is also in the m4d file. So the only way to find out about the structure of the pdf is to read the trailer and use that information to read the pdf data section. Usually this is multiplexed (channel (changes fastest) x time). You definitely need to make use of the 4D librariers to read the trailer section. There is no other way to consider all possible variations of the data format. Therefore I would prefer to use something like the m4D file with the relevant information but then I simply copy the pdf (with all the trailer information) to actually read the data. I need an m4d file for every pdf Does this answer your question? Greetings Christian
Jan-Mathijs Schoffelen - 2013-08-07 08:59:12 +0200
Hi Christian, When I do this: >> hdr=ft_read_header('c,rfhp0.1Hz,o,ccfbp10-40Hz-678-2,cag,c,n,fbp10-40Hz-678-2,s.m4d') hdr = Fs: 678.1700 nChans: 170 nSamples: 6783 nSamplesPre: 0 nTrials: 1 label: {170x1 cell} grad: [1x1 struct] orig: [1x1 struct] chantype: {170x1 cell} chanunit: {170x1 cell} >> gain=hdr.orig.ChannelGain; >> unique(gain) ans = 1 32 So apart from channel 'X1', all ChannelGain values are 1, which is what it should be, right? Then, looking into ft_read_data, from line 247 it says: gain = hdr.orig.ChannelGain; if isfield(hdr.orig, 'ChannelUnitsPerBit') upb = hdr.orig.ChannelUnitsPerBit; else warning('cannot determine ChannelUnitsPerBit'); upb = 1; end and from line 276 it says: switch sampletype case {'short', 'long'} % include both the gain values and the integer-to-double conversion in the calibration calib = diag(gain(chanindx) .* upb(chanindx)); case {'float', 'double'} % only include the gain values in the calibration calib = diag(gain(chanindx)); otherwise error('unsupported data format'); end From the code and from the example data I don't see what may go wrong here. But perhaps I misunderstood the problem.
Jan-Mathijs Schoffelen - 2013-08-07 10:33:50 +0200
Hi Christian, When I do this: >> hdr=ft_read_header('c,rfhp0.1Hz,o,ccfbp10-40Hz-678-2,cag,c,n,fbp10-40Hz-678-2,s.m4d') hdr = Fs: 678.1700 nChans: 170 nSamples: 6783 nSamplesPre: 0 nTrials: 1 label: {170x1 cell} grad: [1x1 struct] orig: [1x1 struct] chantype: {170x1 cell} chanunit: {170x1 cell} >> gain=hdr.orig.ChannelGain; >> unique(gain) ans = 1 32 So apart from channel 'X1', all ChannelGain values are 1, which is what it should be, right? Then, looking into ft_read_data, from line 247 it says: gain = hdr.orig.ChannelGain; if isfield(hdr.orig, 'ChannelUnitsPerBit') upb = hdr.orig.ChannelUnitsPerBit; else warning('cannot determine ChannelUnitsPerBit'); upb = 1; end and from line 276 it says: switch sampletype case {'short', 'long'} % include both the gain values and the integer-to-double conversion in the calibration calib = diag(gain(chanindx) .* upb(chanindx)); case {'float', 'double'} % only include the gain values in the calibration calib = diag(gain(chanindx)); otherwise error('unsupported data format'); end From the code and from the example data I don't see what may go wrong here. But perhaps I misunderstood the problem.
Jan-Mathijs Schoffelen - 2013-10-09 16:46:20 +0200
Is this still an issue?