Back to the main page.
Bug 2269 - ft_rejectartifact does not work for trials with overlapping samples
| Status | CLOSED FIXED |
| Reported | 2013-08-29 09:32:00 +0200 |
| Modified | 2015-07-15 13:30:47 +0200 |
| Product: | FieldTrip |
| Component: | core |
| Version: | unspecified |
| Hardware: | PC |
| Operating System: | Linux |
| Importance: | P3 major |
| Assigned to: | Jim Herring |
| URL: | |
| Tags: | |
| Depends on: | |
| Blocks: | |
| See also: |
Giorgos Michalareas - 2013-08-29 09:32:20 +0200
I have a dataset in which trials are partially overlapping. i.e. the start and end sample are: 1 1000 501 1500 1001 2000 e.t.c When I run the ft_rejectartifact with cfg.artfctdef.reject='complete'; Then I get the following error: %------------------------------------ Error using ft_fetch_data (line 121) some of the requested samples occur twice in the data Error in ft_redefinetrial (line 229) data.trial{iTrl} = ft_fetch_data(dataold, 'header', hdr, 'begsample', begsample, 'endsample', endsample, 'chanindx', 1:hdr.nChans, 'docheck', 0); %,'allowoverlap',true); Error in ft_rejectartifact (line 498) data = ft_redefinetrial(tmpcfg, data); %---------------------------------------- At this code line the command is data.trial{iTrl} = ft_fetch_data(dataold, 'header', hdr, 'begsample', begsample, 'endsample', endsample, 'chanindx', 1:hdr.nChans, 'docheck', 0); However ft_fetch_data has an input argument 'allowoverlap' which set by default to false, as in the case above where this argument is not provided. By adding the argument 'allowoverlap',true the function seems to work. I am not sure if this simple addition will wotk with all the other options though. Could you please have a look and commit a corrected version? Thank you Best Giorgos
Robert Oostenveld - 2013-08-29 09:43:05 +0200
Hi Jim, Probably you recall best what this is about, could you have a look at it? thanks Robert
Jörn M. Horschig - 2013-08-29 10:17:14 +0200
I was struggling with a samilar problem some weeks ago. I think the reason why no overlap is allowed, because trials might be affected by different filters or were demeand or whatever. If overlao is allowed, ft_fetch_data would always take the first occurence of the sample. So, for a trial with trial1=500 1500 trial2=1000 2000 ft_fetchdata would always take sample 1000 to 1500 from trial 1. That is because ft_fetch_data is a low-level function and is ignorant about what trial you are interested in. So, if you demean single trials, sample 1000 of trial 1 would have a different value (because baselined differently) from the corresponding sample of trial 2. Similar problems might arise with filter-artifacts at the edges. So given you did not filter and not demeaned or detrended, what you are proposing is perfectly fine. Otherwise, you will artificially create jump-artifact or the like, because data is taken from different trials (even if not desired) The officially proposed way to deal with these things is to use ft_rejectartifact before having the data in memory, so e.g. after ft_definetrial. On the other hand, if you know that you did not filter or demeaned/detrended or did other fancy stuff on single trial level, I would propose you change the line in your local copy and work with that for now. Generally, I think, we could change behaviour such that allowoverlap is either allowed iff the cfg does not contain evidence of filtering or demeaning/-trending, or that ft_fetch_data does a sanity check whether the overlapping segments are equal. If they are equal, then overlapping trials are not a matter of concern.
Robert Oostenveld - 2013-08-29 10:21:27 +0200
(In reply to comment #2) > Generally, I think, we could change behaviour such that ... your option 1 does not appeal to me, option 2 makes perfect sense. But I don't know whether Giorgos (FYI: he is working with me on the HCP project) will actually be helped with it, as I suspect that he has overlap of differently filtered/demeaned data.
Jim Herring - 2013-08-29 11:05:25 +0200
Actually I think the behaviour of ft_fetch_data should be changed such that it does not assume the sample timeline to correspond with one single dataline. When using ft_rejectartifact it does not make sense to think about how to combine (or choose from) trials that have overlapping sample numbers. Or, perhaps it makes more sense to change the behavior of ft_rejectartifact to operate on a trial-by-trial basis if hasdata = 'yes'.
Robert Oostenveld - 2013-08-29 11:48:36 +0200
I have just implemented the "allow for overlap if identical " option mac001> svn commit utilities/ft_fetch_data.m test/test_bug2269.m Adding test/test_bug2269.m Sending utilities/ft_fetch_data.m Transmitting file data .. Committed revision 8435.
Robert Oostenveld - 2013-08-29 11:53:15 +0200
(In reply to comment #4) regarding "assume the sample timeline to correspond with one single dataline..." people are not forced to use ft_preprocessing in combination with ft_redefinetrial. But if they do and use the cfg.trl option, there is a single sample "axis" along which the begin and end sample of each trial are expressed. That is not going to change. But nowadays people seem to be using ft_appenddata and ft_redefinetrial in ways that we did not anticipate, and also for solving problems that we seemed to have solved differently in the past... I think it would help if we could explain what most suitable strategy is for dealing with certain cases of data representation, rather than constantly trying to adjust the code to the apparent needs of the day (as I just did). In this case Giorgos probably could have used (and still might have to use) another strategy for dealing with the data that does not involve ft_fetch_data in any way.
Jörn M. Horschig - 2013-08-29 12:54:49 +0200
as said, the 'official' solution for Giorgos would be to use ft_rejectartifact before reading the data into memory. I think you, Robert, once told me that this is the way the artifact routines were initially intended to be used
Robert Oostenveld - 2013-08-29 14:17:31 +0200
(In reply to comment #7) yes.