Back to the main page.
Bug 2726 - ft_math takes very long to compute operations
| Status | CLOSED WONTFIX |
| Reported | 2014-10-06 16:50:00 +0200 |
| Modified | 2019-08-10 12:28:56 +0200 |
| Product: | FieldTrip |
| Component: | core |
| Version: | unspecified |
| Hardware: | PC |
| Operating System: | Linux |
| Importance: | P5 normal |
| Assigned to: | |
| URL: | |
| Tags: | |
| Depends on: | |
| Blocks: | |
| See also: |
Diego Lozano Soldevilla - 2014-10-06 16:50:32 +0200
Created attachment 663 profiler output Recently, ft_math is going very slow when one does simple operations. Here an example: % create some test data cfg=[]; cfg.layout='CTF275.lay'; lay = ft_prepare_layout(cfg); lab = ft_channelselection('MEG',lay.label); freq1.label = lab; freq1.freq = 5:1:90; freq1.time = 1:52; freq1.dimord = 'chan_freq_time'; freq1.powspctrm = ones(length(freq1.label),length(freq1.freq),length(freq1.time)); freq1.cfg = struct([]); freq2 = freq1; freq2.powspctrm = ones(length(freq1.label),length(freq1.freq),length(freq1.time)).*2; %% this one takes ~16 seconds tic; cfg=[]; cfg.parameter = 'powspctrm'; cfg.operation = '(x1-x2)./(x1+x2)'; out = ft_math(cfg,freq1,freq2); toc; %% this one takes 0.00 seconds tic; out=freq1; out.powspctrm = (freq1.powspctrm-freq2.powspctrm)./(freq1.powspctrm+freq2.powspctrm); toc; I use matlab profiler to check where's the bottleneck and line 384 (y = arrayfun(operation, arginval{:}); ) is called more than a million times (see attachment): profile on -history cfg=[]; cfg.parameter = 'powspctrm'; cfg.operation = '(x1-x2)./(x1+x2)'; out = ft_math(cfg,freq1,freq2); p = profile('info'); profview(0,p) Any ideas?
Robert Oostenveld - 2014-10-06 17:20:29 +0200
I also noticed this while working on the natmeg stats tutorial. In my case it was cfg.operation='log10' which was fast, but 'log10(x1)' which was slow, also in arrayfun. Do you know whether this is a recent change in speed?
Diego Lozano Soldevilla - 2014-10-06 22:13:14 +0200
(In reply to Robert Oostenveld from comment #1) Hi Robert, I had the feeling this was recent to me but I downloaded a fieldtrip from sept 1st and the problem is still there... No clue when it started
Robert Oostenveld - 2014-10-07 10:08:32 +0200
I checked the following code ----- nrpt = 10; nchan = 10; nfreq = 100; ntime = 100; data = []; data.dimord = 'rpt_chan_freq_time'; data.label = arrayfun(@num2str, 1:nchan, 'uniformoutput', false); data.freq = 1:nfreq; data.time = 1:ntime; data.powspctrm = rand(nrpt, nchan, nfreq, ntime); cfg = []; cfg.parameter = 'powspctrm'; cfg.operation = 'log10'; stopwatch = tic; data1 = ft_math(cfg, data); time1 = toc(stopwatch); cfg = []; cfg.parameter = 'powspctrm'; cfg.operation = 'log10(x1)'; stopwatch = tic; data2 = ft_math(cfg, data); time2 = toc(stopwatch); disp(time1); disp(time2); ----- The problem is partially in arrayfun, and partially in the use of an anonymous function. See here for a demonstration: >> operation = @(x) log10(x) >> tic; arrayfun(operation, data.powspctrm); toc Elapsed time is 3.501949 seconds. >> tic; arrayfun(@log10, data.powspctrm); toc Elapsed time is 1.048518 seconds. >> tic; log10(data.powspctrm); toc Elapsed time is 0.017312 seconds. The reason for using arrayfun is that the output and input dimensions should be identical, i.e. x1/x2 should be implemented as x1./x2 and not as a matrix division. I don't see how we can speed it up, given these constraints. Knowing that cfg.operation='log10' is much faster than cfg.operation='log10(x1)' Do you agree with this verdict? If so, I suggest to mark it as "RESOLVED->WONTFIX".
Diego Lozano Soldevilla - 2014-10-07 14:57:43 +0200
(In reply to Robert Oostenveld from comment #3) Hi Robert, Fully agree with your verdict. Still I cannot explain why it was faster in the past (might be it was only in my fantasy ;) ). Still I'll try to think about how to improve speed because now I won't use a function that takes 17secs to compute a simple operation (x1-x2)./(x1+x2) (my case 75 meg datasets...)
Eelke Spaak - 2014-10-11 11:54:36 +0200
(In reply to Robert Oostenveld from comment #3) If the main reason for using arrayfun is that the operation needs to be elementwise (i.e. ./ rather than /), then we should be able to come up with an alternative. There is e.g. the builtin function vectorize(), which inserts a . before any ^, / or *. If that won't do it (I don't know ft_math's architecture that well), I also have some code lying around that does a more sophisticated parsing of arbitrary arithmetical expressions, which takes care of the elementwise thing, automatically identifies constants, functions, and variables in the expression, and that sort of stuff. I think it's quite important that we don't "wontfix" this, because these huge performance differences will cause people to stick with plain matlab, rather than use ft_math.