Back to the main page.
Bug 2888 - filetype fcdc_buffer_offline reading problems
| Status | CLOSED FIXED |
| Reported | 2015-05-04 08:31:00 +0200 |
| Modified | 2015-07-15 13:31:26 +0200 |
| Product: | FieldTrip |
| Component: | fileio |
| Version: | unspecified |
| Hardware: | PC |
| Operating System: | Windows |
| Importance: | P5 normal |
| Assigned to: | Jan-Mathijs Schoffelen |
| URL: | |
| Tags: | |
| Depends on: | 1262 |
| Blocks: | |
| See also: |
Jan-Mathijs Schoffelen - 2015-05-04 08:31:00 +0200
Ervin, a toolkit participant reported a problem with reading in data of the type 'fcdc_offline_buffer'. Some of it boils down to some messy automatic filetype detection issues + messy dataset2files conversion in ft_checkconfig. This can be easily fixed. However, upon fixing this, the issue just went to its next stage, preventing me (i.e. JM) to easily fix this. So I'd like to discuss this in a bit more detail in this thread.
Jan-Mathijs Schoffelen - 2015-05-04 14:27:37 +0200
The data that has been sent to me by Ervin contains an empty binary header file. @Ervin: can you explain how this came into existence? According to the test data we have on disk, and the low-level reading code, the binary header (i.e. the header with extension) should contain some data.
ErvinPoljac - 2015-05-04 15:17:29 +0200
(In reply to Jan-Mathijs Schoffelen from comment #1) With regard to (what it should be raw) data from 0001 file: Not sure why the header file (header without extension) is empty. The file header.txt is not empty, but it only contains channel information. With regard to the data.mat file: I added another file (sliced.mat.zip) to the dropbox folder, it contains .mat data, header and events file. This is automatically created during data collection (experiment). Unfortunately, I am not familiar with the details, since I use some toolboxes I got from other people from or department.
Jan-Mathijs Schoffelen - 2015-05-04 15:38:30 +0200
are those toolboxes you use implemented by people you can consult with, or is this N'th hand information (and possibly diluted...)? The reason I ask is, because I want to get some insight as to whether there has been some tampering with the official data format on the DCC end :-)
ErvinPoljac - 2015-05-04 16:16:16 +0200
(In reply to Jan-Mathijs Schoffelen from comment #3) I can ask Jason F. but I haven't seen him today. It is his toolbox: https://github.com/jadref/buffer_bci I am not aware of all the details of how the data is processed before saving.
Jan-Mathijs Schoffelen - 2015-05-04 21:14:05 +0200
Adding Jason to the CC'list of this issue: @Jason: Ervin is experiencing problems reading in his data, which appears to be intended to be in the format 'fcdc_buffer_offline', which Stefan Klanke implemented at some point in the course of braingain. Since at this point in time I don't have any reason to suspect that Stefan contributed dysfunctional code, I would like you guys to check whether Ervin's data is according to the expected specs. Importantly, whether the lack of data in the binary header file (he sent us some data through dropbox to reproduce the issue) is according to what should be expected, or whether it is based on some local code change on your end, when the data are written to disk. If the latter is the case, I think it is best resolved on your end, rather than in the FieldTrip code.
Jason Farquhar - 2015-05-06 13:54:37 +0200
Created attachment 711 fixed header reader which falls back on .txt file if binary header isn't available
Jason Farquhar - 2015-05-06 13:55:01 +0200
Hi, The lack of a binary header file is actually due to another bug in the 'recording' version realtime buffer binary which is the one which saves in the fcdc_offline format. Essentially, this happens when buffer is killed (e.g. by
Jan-Mathijs Schoffelen - 2015-05-06 14:05:46 +0200
Thanks Jason, that would have also been my suggested workaround, but I just wanted to check to be sure.
ErvinPoljac - 2015-05-06 14:26:00 +0200
Thanks Jason! So, replacing the existing read_buffer_offline_header.m should enable reading in the data? Or do I need an updated version of Fieldtrip, are there any other changes Jan-Mathijs?
Jason Farquhar - 2015-05-06 14:42:32 +0200
(In reply to ErvinPoljac from comment #9) Yes, replacing with this version should fix the issues. Can you try and report if it works?
Jan-Mathijs Schoffelen - 2015-05-06 14:44:32 +0200
well, there are some loose ends with respect to the correct automatic detection of the fileformat. I intend to fix this soon, but you can already have a shot at it. When you run into problems you should be explicit in defining the filetype: cfg.dataformat = 'fcdc_buffer_offline'; cfg.headerformat = 'fcdc_buffer_offline';
ErvinPoljac - 2015-05-06 16:31:12 +0200
(In reply to Jan-Mathijs Schoffelen from comment #11) Despite explicit statements: cfg.dataformat = 'fcdc_buffer_offline' and cfg.headerformat = 'fcdc_buffer_offline'; I still get the message: unsupported header format (unknown), so still not solved...
Jan-Mathijs Schoffelen - 2015-05-06 16:47:17 +0200
Did you try to explicitly specify cfg.headerfile and cfg.datafile (rather than the directory name in cfg.dataset). The issues I was referring to pertain to the non-robust way in which dataset2files tries to convert the directory name into the filenames pointing to the header and the samples respectively.
ErvinPoljac - 2015-05-06 17:04:52 +0200
I did: cfg.headerfile = 'header'; cfg.datafile = 'samples'; The full error message is: evaluating trialfunction 'ft_trialfun_general' Error using ft_read_header (line 2006) unsupported header format (unknown) Error in ft_trialfun_general (line 74) hdr = ft_read_header(cfg.headerfile, 'headerformat', cfg.headerformat); Error in ft_definetrial (line 174) [trl, event] = feval(cfg.trialfun, cfg);
Jan-Mathijs Schoffelen - 2015-05-06 17:07:00 +0200
did you clear the dataset? i.e. cfg = []; cfg.datafile = fullfile(
ErvinPoljac - 2015-05-06 17:30:58 +0200
(In reply to Jan-Mathijs Schoffelen from comment #15) Yes, this seems to produce a bit different error: Reference to non-existent field 'nSamplesPre'. Error in ft_read_header (line 2082) hdr.nSamplesPre = double(hdr.nSamplesPre); Error in ft_trialfun_general (line 74) hdr = ft_read_header(cfg.headerfile, 'headerformat', cfg.headerformat); Error in ft_definetrial (line 174) [trl, event] = feval(cfg.trialfun, cfg); So, it looks in the header file, but doesn't find what it needs?
Jan-Mathijs Schoffelen - 2015-05-06 21:23:04 +0200
Hi Ervin, I think it should work again now (when using the most recent fieldtrip) I just committed an extra fix to read_buffer_offline_header that ensures the nSamplesPre is also defined when only using the textile info. I don't know how in general FT is updated in the DCC, but tonight's ftp-version should contain the fix, or otherwise you could do a 'ft_version update' if you are running a local copy of fieldtrip
Robert Oostenveld - 2015-05-06 22:10:43 +0200
I came across this myself when testing some realtime application, and made another small improvement to the filetype detection mbp> svn commit fileio/ft_filetype.m Sending fileio/ft_filetype.m Transmitting file data . Committed revision 10380.
Jan-Mathijs Schoffelen - 2015-05-06 22:23:26 +0200
prutser
Robert Oostenveld - 2015-05-06 22:37:09 +0200
(In reply to Jan-Mathijs Schoffelen from comment #19) wie ... wat ... waar ...? Ik zat net lekker in mijn code, en dan komt jouw commit er tussen door fietsen! werkt het nu wel allemaal?
Jan-Mathijs Schoffelen - 2015-05-06 22:41:42 +0200
Voor mij werkt 't nog, ik hoop voor de jongens op het DCC ook. De woordkeus had meer betrekking op het feit dat jij laat op de avond nog zit te hobbyen.
ErvinPoljac - 2015-05-06 23:17:32 +0200
(In reply to Jan-Mathijs Schoffelen from comment #21) In ieder geval hoopvol dat het bij jou werkt, morgen probeer ik met de nieuwe versie van ftp...
Robert Oostenveld - 2015-05-07 09:31:58 +0200
I added a test script and some test data (about 10 seconds from sine2ft). There were still some cases that I was able to make more robust against misspecification of the filenames. mac011> svn commit fileio/ test/ utilities/ Sending fileio/ft_read_event.m Sending fileio/private/dataset2files.m Sending fileio/private/read_buffer_offline_header.m Sending utilities/private/dataset2files.m Transmitting file data .... Committed revision 10381.
ErvinPoljac - 2015-05-07 09:58:10 +0200
(In reply to Robert Oostenveld from comment #23) If Mathijs is using the dataset I provided ('0001'), and it is working, then I don't know what I am doing wrong: I still get the error: Warning: Couldnt open binary header file : 0001/header > In fileio/private/read_buffer_offline_header at 116 In ft_read_header at 1214 In ft_trialfun_general at 78 In ft_definetrial at 174 Reference to non-existent field 'nSamplesPre'. Error in ft_read_header (line 2102) hdr.nSamplesPre = double(hdr.nSamplesPre); Error in ft_trialfun_general (line 78) hdr = ft_read_header(cfg.headerfile, 'headerformat', cfg.headerformat); Error in ft_definetrial (line 174) [trl, event] = feval(cfg.trialfun, cfg);
Jan-Mathijs Schoffelen - 2015-05-07 10:14:47 +0200
What you're doing wrong ( :-) ) is that you do not yet use the most up-to-date version of read_buffer_offline_header... Line 117 of this file should not read: hdr=txt but it should read hdr=copyfields(txt,HDR,fieldnames(txt));
Jan-Mathijs Schoffelen - 2015-05-07 10:16:49 +0200
What you're doing wrong ( :-) ) is that you do not yet use the most up-to-date version of read_buffer_offline_header... Line 117 of this file should not read: hdr=txt but it should read hdr=copyfields(txt,HDR,fieldnames(txt));
Jan-Mathijs Schoffelen - 2015-05-07 10:17:22 +0200
sorry for the double post. Note that the HDR should read hdr
ErvinPoljac - 2015-05-07 10:24:43 +0200
(In reply to Jan-Mathijs Schoffelen from comment #27) But I replaced the read_buffer_offline_header with the new version! Anyway, this is much better, I think (after ft_preprocessing): data = hdr: [1x1 struct] label: {64x1 cell} time: {1x50 cell} trial: {1x50 cell} fsample: 2048 sampleinfo: [50x2 double] trialinfo: [50x1 double] cfg: [1x1 struct] Thanks a lot!