Back to the main page.

Bug 668 - ft_artifact_zvalue causes a steadily increasing memory use

Reported 2011-05-13 21:55:00 +0200
Modified 2011-09-14 14:33:30 +0200
Product: FieldTrip
Component: core
Version: unspecified
Hardware: PC
Operating System: Mac OS
Importance: P1 minor
Assigned to: Jörn M. Horschig
Depends on:
See also:

Robert Oostenveld - 2011-05-13 21:55:54 +0200

... which causes it to run out of memory on a ctf151 datasetr on my 4GB laptop! Once upon a time, it used to work so efficient memory wise, but this problem suggests that we cannot even process a 275 channel dataset on a 8GB computer any more. This needs discussion in the FT meeting.

Robert Oostenveld - 2011-05-13 21:57:14 +0200

changed status from multiple bugs from NEW into ASSIGNED

Robert Oostenveld - 2011-05-13 21:58:09 +0200

changed status from multiple bugs from NEW to ASSIGNED

Jan-Mathijs Schoffelen - 2011-05-15 20:06:00 +0200

Are you passing a data structure as second input argument? In other words: is the function required to fetch or read the data?

Robert Oostenveld - 2011-05-15 20:59:29 +0200

reading from disk

Jörn M. Horschig - 2011-07-26 12:04:04 +0200

Might the problem be caused just by storing the data? In my case, 137 trials, this is already ~1.5gigs: Name Size Bytes Class Attributes dat 1x137 1408594544 cell The rest of the code looks quite memory-efficient. Btw, in case you give data as an input argument, the data is stored again in a dat-matrix, which also is extremely inefficient. If we do not want to store the data in memory, then we need to apply the preprocessing algorithm twice to the data (thus the function will be twice as slow), once for calculating the mean and standard devitation, and once for transforming the amplitude-values to z-values. Should I add a cfg-option which allows this? I think it might be preferable to do on low-memory machines.

Jörn M. Horschig - 2011-08-29 17:06:36 +0200

As discussed some weeks ago, I fixed this so that ft_artifact_zvalue can be called with cfg.memory='low', meaning that it will require way less memory (but more computation time). If strcmp(cfg.memory, 'low') then it will not store all the preprocessed data, but in return do the desired preprocessing two times, in two different loops

Robert Oostenveld - 2011-09-14 14:33:30 +0200

I closed all the bugs that were in the status RESOLVED. This includes the ones that we just discussed in the weekly fieldtrip meeting, but also the bugs that we did not discuss.