Back to the main page.

Bug 2269 - ft_rejectartifact does not work for trials with overlapping samples

Reported 2013-08-29 09:32:00 +0200
Modified 2015-07-15 13:30:47 +0200
Product: FieldTrip
Component: core
Version: unspecified
Hardware: PC
Operating System: Linux
Importance: P3 major
Assigned to: Jim Herring
Depends on:
See also:

Giorgos Michalareas - 2013-08-29 09:32:20 +0200

I have a dataset in which trials are partially overlapping. i.e. the start and end sample are: 1 1000 501 1500 1001 2000 e.t.c When I run the ft_rejectartifact with cfg.artfctdef.reject='complete'; Then I get the following error: %------------------------------------ Error using ft_fetch_data (line 121) some of the requested samples occur twice in the data Error in ft_redefinetrial (line 229) data.trial{iTrl} = ft_fetch_data(dataold, 'header', hdr, 'begsample', begsample, 'endsample', endsample, 'chanindx', 1:hdr.nChans, 'docheck', 0); %,'allowoverlap',true); Error in ft_rejectartifact (line 498) data = ft_redefinetrial(tmpcfg, data); %---------------------------------------- At this code line the command is data.trial{iTrl} = ft_fetch_data(dataold, 'header', hdr, 'begsample', begsample, 'endsample', endsample, 'chanindx', 1:hdr.nChans, 'docheck', 0); However ft_fetch_data has an input argument 'allowoverlap' which set by default to false, as in the case above where this argument is not provided. By adding the argument 'allowoverlap',true the function seems to work. I am not sure if this simple addition will wotk with all the other options though. Could you please have a look and commit a corrected version? Thank you Best Giorgos

Robert Oostenveld - 2013-08-29 09:43:05 +0200

Hi Jim, Probably you recall best what this is about, could you have a look at it? thanks Robert

Jörn M. Horschig - 2013-08-29 10:17:14 +0200

I was struggling with a samilar problem some weeks ago. I think the reason why no overlap is allowed, because trials might be affected by different filters or were demeand or whatever. If overlao is allowed, ft_fetch_data would always take the first occurence of the sample. So, for a trial with trial1=500 1500 trial2=1000 2000 ft_fetchdata would always take sample 1000 to 1500 from trial 1. That is because ft_fetch_data is a low-level function and is ignorant about what trial you are interested in. So, if you demean single trials, sample 1000 of trial 1 would have a different value (because baselined differently) from the corresponding sample of trial 2. Similar problems might arise with filter-artifacts at the edges. So given you did not filter and not demeaned or detrended, what you are proposing is perfectly fine. Otherwise, you will artificially create jump-artifact or the like, because data is taken from different trials (even if not desired) The officially proposed way to deal with these things is to use ft_rejectartifact before having the data in memory, so e.g. after ft_definetrial. On the other hand, if you know that you did not filter or demeaned/detrended or did other fancy stuff on single trial level, I would propose you change the line in your local copy and work with that for now. Generally, I think, we could change behaviour such that allowoverlap is either allowed iff the cfg does not contain evidence of filtering or demeaning/-trending, or that ft_fetch_data does a sanity check whether the overlapping segments are equal. If they are equal, then overlapping trials are not a matter of concern.

Robert Oostenveld - 2013-08-29 10:21:27 +0200

(In reply to comment #2) > Generally, I think, we could change behaviour such that ... your option 1 does not appeal to me, option 2 makes perfect sense. But I don't know whether Giorgos (FYI: he is working with me on the HCP project) will actually be helped with it, as I suspect that he has overlap of differently filtered/demeaned data.

Jim Herring - 2013-08-29 11:05:25 +0200

Actually I think the behaviour of ft_fetch_data should be changed such that it does not assume the sample timeline to correspond with one single dataline. When using ft_rejectartifact it does not make sense to think about how to combine (or choose from) trials that have overlapping sample numbers. Or, perhaps it makes more sense to change the behavior of ft_rejectartifact to operate on a trial-by-trial basis if hasdata = 'yes'.

Robert Oostenveld - 2013-08-29 11:48:36 +0200

I have just implemented the "allow for overlap if identical " option mac001> svn commit utilities/ft_fetch_data.m test/test_bug2269.m Adding test/test_bug2269.m Sending utilities/ft_fetch_data.m Transmitting file data .. Committed revision 8435.

Robert Oostenveld - 2013-08-29 11:53:15 +0200

(In reply to comment #4) regarding "assume the sample timeline to correspond with one single dataline..." people are not forced to use ft_preprocessing in combination with ft_redefinetrial. But if they do and use the cfg.trl option, there is a single sample "axis" along which the begin and end sample of each trial are expressed. That is not going to change. But nowadays people seem to be using ft_appenddata and ft_redefinetrial in ways that we did not anticipate, and also for solving problems that we seemed to have solved differently in the past... I think it would help if we could explain what most suitable strategy is for dealing with certain cases of data representation, rather than constantly trying to adjust the code to the apparent needs of the day (as I just did). In this case Giorgos probably could have used (and still might have to use) another strategy for dealing with the data that does not involve ft_fetch_data in any way.

Jörn M. Horschig - 2013-08-29 12:54:49 +0200

as said, the 'official' solution for Giorgos would be to use ft_rejectartifact before reading the data into memory. I think you, Robert, once told me that this is the way the artifact routines were initially intended to be used

Robert Oostenveld - 2013-08-29 14:17:31 +0200

(In reply to comment #7) yes.

Giorgos Michalareas - 2013-08-29 15:06:19 +0200

Ok thank you guys for all these replies. So let me summarize the main points: A- After the latest modification by Jorn, the overlap is allowed but it always selects the common interval from the first trial it occurs (as long as the same exact overlap period is fully contained in other trials). So this means that if you still have a dataset which has been demeaned,detrended or filtered or processed in any way , then you ll end up with a foreign data segment within one or more trials. Is this Correct? B- The simple case in which you want to remove the full trials when any of them coincides with a bad segment, and there is an overlap between some trials, is not going be supported any more by ft_rejectartifact. Is this right? So for such cases your advice is to write my own functions for rejection of time segments in an already processed dataset? C- Could you please put a note about this in the HELP of the ft_rejectartifact function so that other people will be able to know? I am sure there are quite a few people who are working on datasets with some partial overlapping between trials and who might be using ft_rejectartifact to remove trials with some kind of artifact from processed datasets. Thank you Giorgos