Back to the main page.
Bug 2211 - data.trial contains only NaN after preprocessing on neuroscan cnt files in a newer version of fieldtrip
| Status | CLOSED FIXED |
| Reported | 2013-06-25 17:34:00 +0200 |
| Modified | 2014-01-29 13:39:20 +0100 |
| Product: | FieldTrip |
| Component: | core |
| Version: | unspecified |
| Hardware: | PC |
| Operating System: | Windows |
| Importance: | P3 blocker |
| Assigned to: | Roemer van der Meij |
| URL: | |
| Tags: | |
| Depends on: | |
| Blocks: | |
| See also: |
Carmen Kung - 2013-06-25 17:34:58 +0200
I used to be able to run preprocessing successfully on the same cnt file using an older version of fieldtrip (fieldtrip-20120120). However after installing a newer version (fieldtrip-20130610), I have problems with the running preprocessing on the same cnt file. Several things did not look right. For instance, this error message ('Warning: events imported with a time shift might be innacurate') keeps coming back throughout the preprocessing. It also takes way longer to run. What is worse is that I only got NaNs in data.trial. I checked the script again and it looked fine (especially the header setting where several problems have been reported. Also, I found an older version of fieldtrip?(fieldtrip-20111231) and try it out again. The preprocessing went well and . So I wonder what could go wrong with this newer version? Thank you very much in advance! FYI, I have attached the link to the cnt file and the script. This is the link of the cnt file (https://www.dropbox.com/sh/od62sgjzvm8j9m2/go7LNwPZOK) and Below is the script till end of preprocessing: % ensure that we don't mix up subjectsclear all clear all clear subjectdata addpath('C:\Users\u488137\Documents\MATLAB\fieldtrip-20130610'); %define the filenames, parameters and other information that is subject %specific subjectdata = []; subjectdata.initial = 'HKH'; subjectdata.subjectnr = 'SS35M'; subjectdata.date = '07042012'; subjectdata.listnr = '02'; subjectdata.group = 'merger'; subjectdata.dir = 'C:\Users\u488137\Documents\'; subjectdata.subjectdir = [subjectdata.dir, subjectdata.subjectnr, '\']; subjectdata.cntdir = [subjectdata.subjectdir, 'cnt\']; mkdir([subjectdata.subjectdir, 'rawdata']); mkdir([subjectdata.subjectdir, 'preproc']); mkdir([subjectdata.subjectdir, 'figures']); mkdir([subjectdata.subjectdir, 'seg']); mkdir([subjectdata.subjectdir, 'timelock']); mkdir([subjectdata.subjectdir, 'permutation']); subjectdata.rawdatadir = [subjectdata.subjectdir, 'rawdata\']; subjectdata.preprocdir = [subjectdata.subjectdir, 'preproc\']; subjectdata.segdir = [subjectdata.subjectdir, 'seg\']; subjectdata.timelockdir = [subjectdata.subjectdir, 'timelock\']; subjectdata.permtdir = [subjectdata.subjectdir, 'permutation\']; subjectdata.figdir = [subjectdata.subjectdir, 'figures\']; addpath (subjectdata.subjectdir); addpath (subjectdata.cntdir); addpath (subjectdata.rawdatadir); addpath (subjectdata.segdir); addpath (subjectdata.timelockdir); %% Epoching cfg = []; cfg.dataset = [subjectdata.cntdir, 'MMN-', subjectdata.subjectnr, '-', subjectdata.initial, '-', subjectdata.date, '.cnt']; cfg.trialdef.eventtype = 'stimtype'; cfg.trialdef.prestim = 0.8; cfg.trialdef.poststim = 1; cfg.dataformat = 'ns_cnt32'; cfg.eventformat = 'ns_cnt32'; cfg.headerformat = 'ns_cnt32'; cfg.outputfile = [subjectdata.preprocdir, subjectdata.subjectnr, '_E.mat']; %cfg(cond).trialdef.eventvalue = triggercode(cond); cfg = ft_definetrial(cfg); cfg.bpfilter = 'yes'; cfg.bpfiltdir = 'twopass'; cfg.bpfreq = [1 20]; cfg.bpfilttype = 'but'; cfg.channel = {'EEG','-CB1','-CB2'}; ft_preprocessing(cfg);
Roemer van der Meij - 2013-06-25 19:01:53 +0200
This is really odd. Everything works fine if the data is read in first, and then filtered. When going through the code, it is actually matlab's filter.m that gives back NaNs. Up until then everything's fine. Filter.m get's the same input if you read the data and filter it in 1 step, or if you read the data in first, and then filter it separately. Filter coefficients and all, the same. So what fails (gives back only NaNs is this): *************** cfg = []; ---definetrial--- cfg.bpfilter = 'yes'; cfg.bpfiltdir = 'twopass'; cfg.bpfreq = [1 20]; cfg.bpfilttype = 'but'; cfg.channel = {'EEG','-CB1','-CB2'}; ft_preprocessing(cfg); *************** But what works (no NaNs) is this: *************** cfg = []; ---definetrial--- data = ft_preprocessing(cfg); cfg.bpfilter = 'yes'; cfg.bpfiltdir = 'twopass'; cfg.bpfreq = [1 20]; cfg.bpfilttype = 'but'; cfg.channel = {'EEG','-CB1','-CB2'}; data = ft_preprocessing(cfg,data) *************** Right now I do not see where this is coming from.
Roemer van der Meij - 2013-06-26 16:25:54 +0200
Alright, I've pinpointed the cause of the error. The data becomes NaNs because it is single precision, when it is double precision it is all fine. The cause lies in loadcnt.m, which contains the code below. This bit was more or less added in r5278, when we updated to EEGLABs most recent version: https://code.google.com/p/fieldtrip/source/detail?r=5278&path=/trunk/fileio/private/loadcnt.m ************************* if h.channeloffset <= 1 dat=fread(fid, [h.nchannels Inf], r.dataformat); if size(dat,2) < r.ldnsamples dat=single(dat); r.ldnsamples = size(dat,2); else dat=single(dat(:,1:r.ldnsamples)); end; else h.channeloffset = h.channeloffset/2; % reading data in blocks dat = zeros( h.nchannels, r.ldnsamples, 'single'); dat(:, 1:h.channeloffset) = fread(fid, [h.channeloffset h.nchannels], r.dataformat)'; counter = 1; while counter*h.channeloffset < r.ldnsamples dat(:, counter*h.channeloffset+1:counter*h.channeloffset+h.channeloffset) = ... fread(fid, [h.channeloffset h.nchannels], r.dataformat)'; counter = counter + 1; end; end ; ************************* As far as I can see, the data is always set to single precision. Also, we do not currently enforce cfg.precision (default = double) in ft_read_data, ft_preprocessing, or preproc (preproc does it after all preproccing). As far as I can see, this means that all neuroscan data that was being read in was set to single precision since Feb. 2012. If an ordinary Butterworth band-pass already gives these serious errors (output is all NaN), then I wonder why we haven't noticed this before. Robert, I added you as a CC because this had to do with loadcnt.m, and because of the possibly large consequences. As a fix/feature, I now made sure that cfg.precision is used immediately after the call to ft_read_data in ft_preprocessing. Which is something we wanted to do anyway. See bug 26 (automatically set to won't fix after inactivity). It is fixed in r8281, which will be on the ftp server tonight.