Back to the main page.

Bug 79 - ft_redefinetrial with overlapping trials (e.g. after ft_appenddata)

Reported 2010-05-07 14:27:00 +0200
Modified 2017-12-04 10:08:30 +0100
Product: FieldTrip
Component: preproc
Version: unspecified
Hardware: PC
Operating System: Linux
Importance: P3 normal
Assigned to: Jan-Mathijs Schoffelen
Depends on:
See also:

Arjen Stolk - 2010-05-07 14:27:52 +0200

Simulation of the bug using two datasets (1 contains zeros, 2 contains ones). The datasets are appended (e.g. cause system failure during recording) and then redefined (e.g. prior to the frequency analysis process). Redefinition has been attempted in two ways; 1) trl definition and 2) cfg.begsample & cfg.endsample specification. % stuff Fs = 1200; cfg = []; cfg.layout = 'CTF275.lay'; cfg.layout = prepare_layout(cfg); label = cfg.layout.label(1:273,1); % dataset 1; all ones dataset1.trial{1,1} = zeros(273,2400); dataset1.time{1,1} = (0:1/Fs:2-1/Fs); dataset1.label = label; dataset1.fsample = Fs; dataset1.cfg.previous.previous.trl(1,1) = 1; dataset1.cfg.previous.previous.trl(1,2) = 2400; dataset1.cfg.previous.previous.trl(1,3) = 0; % dataset2; all zeros dataset2.trial{1,1} = ones(273,2400); dataset2.time{1,1} = (0:1/Fs:2-1/Fs); dataset2.label = label; dataset2.fsample = Fs; dataset2.cfg.previous.previous.trl(1,1) = 1; dataset2.cfg.previous.previous.trl(1,2) = 2400; dataset2.cfg.previous.previous.trl(1,3) = 0; % append data cfg = []; dataset3 = ft_appenddata(cfg, dataset1, dataset2); % redefine trial with trl cfg = []; cfg.trl(1,1) = 1; cfg.trl(1,2) = 1200; cfg.trl(1,3) = 0; cfg.trl(2,1) = 1201; cfg.trl(2,2) = 2400; cfg.trl(2,3) = 0; dataset4 = ft_redefinetrial(cfg, dataset3); % result ??? Error using ==> fetch_data at 114 some of the requested samples occur twice in the data when commenting line 113 and 114 in fetch_data, and thus, ignoring the error; both trials in dataset4 solely contain 'ones' which means that dataset2 alone has been used for input % redefine trial with cfg.begsample and cfg.endsample cfg = []; cfg.begsample(1,1) = 1; cfg.endsample(1,1) = 1200; cfg.begsample(2,1) = 1201; cfg.endsample(2,1) = 2400; dataset5 = ft_redefinetrial(cfg, dataset3); % result trial 1 has zeros (originating from dataset1) trial 2 has ones (originating from dataset2) disadvantage of this method is that one cannot cut trials in pieces (and create more trials) but only select a part from each trial

Jan-Mathijs Schoffelen - 2010-05-17 09:23:34 +0200

The first question here is actually: What IS the bug. Arjen's problem is related to the fact that when calling ft_redefinetrial on an appended dataset (consisting of 2 or more different recordings), the appended trl-matrix contains 'overlapping' epochs. This is caught as an error in the low-level public/fetch_data. This feature of fetch_data also causes problems if a similar data structure is used in ft_databrowser (this is a separate bug assigned to Ingrid, but should be treated in this context as well). Robert pointed out to a 'philosophical' problem related to this issue. This concerns the fact that in this context fieldtrip uses the trl-matrix which is stored somewhere in the history of cfgs to a data-structure. This is against fieldtrip-philosophy, in that each data-structure should be self-explanatory. Having to dig into the past (by calling findcfg(data.cfg, 'trl') could lead to the extraction of ambiguous or ill-defined trl-matrices (as we have seen). Yet, trl-matrices have been used extensively in the past to code for trial specific parameters in the 4th until 100th columns, and by taking care of the trl-matrix in each analysis step allows us to keep track of these parameters consistently. For future development this has to be reconsidered and possibly added as a higher level field in the data-structure. (in line with Martin Vincks proposal to deal with some things in the ft_spikeXXX functions). For the time being I think a few changes need to be made to the existing code: ft_appenddata: check explicitly whether the original data structures come from the same recording. If so, append the trl-matrices. If not, append them, but replace the first 2 columns either by nans (don't know what this will be doing to the rest of the code), or by an internally consistent description (at least with respect to the trl-matrix). The second option may also have unwanted side-effects though. ft_redefinetrial: throw an error if the redefinition is based on a trl-matrix, and if the trl-matrix contained in the data is ambiguous/inconsistent/contains nans ft_databrowser: .... fetch_data: only do the overlapping samples check when this is needed Final point related to ft_redefinetrial: Arjan mentions that it may be a desired option to be able to cut out more than one epoch out of a longer one. As of yet, this functionality does not exist as far as I know, but investigate whether this is desirable, and if so, implement it.

Robert Oostenveld - 2010-05-19 13:57:28 +0200

add fields to data structure: trialdef (raw data only) trialinfo (4th column of original trl matrix and upwards) to ensure backward compatibility first build into checkdata the conversion from old-fashioned (containing cfg.previousxxx.trl) to new (containing .trialdef and .trialinfo) etc....

Jan-Mathijs Schoffelen - 2010-07-21 10:27:54 +0200

Original issue does not exist anymore after upgrading code to use trialdef and trialinfo, rather than relying on data.cfg.previous......... If appenddata is called with data from two different recordings, the trialdef will be removed