Back to the main page.

Bug 1429 - ft_componentanalysis returns slightly different results from the stored test results

Reported 2012-04-17 12:02:00 +0200
Modified 2013-11-03 19:57:00 +0100
Product: FieldTrip
Component: core
Version: unspecified
Hardware: PC
Operating System: Windows
Importance: P3 normal
Assigned to: Eelke Spaak
Depends on:
See also:

Eelke Spaak - 2012-04-17 12:02:32 +0200

While checking out test_ft_componentanalysis, I tracked its failing down to line 30: assert(isequalwithequalnans(data, datanew)); It turns out that (at least) the .unmixing field is slightly different in the two versions. The difference is on the order of 10^-10, though, while the weights are in the order of 10^-1, so probably this is not a 'real' bug. I would not know how to fix it, though.

Johanna - 2012-04-17 12:11:57 +0200

Hi Eelke, for which 'k' does this occur? I just did svn-update and my previous version (from last week) of test_ft_componentanalysis went through all the cfg.method options...does this new version also? (it seems only 'pca' is now tested). I ask because I did make a change with the .unmixing specific to DSS method, about 2 months ago, but that shouldn't affect PCA.

Eelke Spaak - 2012-04-17 12:16:11 +0200

(In reply to comment #1) The one I'm using indeed only checks PCA (which might be an issue in itself, of course: why not test every method?). The assertion fails for k = 1,2,3 and probably also the rest (which I haven't checked).

Roemer van der Meij - 2012-05-16 14:01:09 +0200

my suspicion is that this is related to the scaling of the data, that has been introduced a while ago. Before, the data entered the bss-algorithms unscaled, i.e. it could have values of 10e-12, whereas they are scaled to 10e0. See whether temporarily commenting out the scaling line reproduces the problem. If it indeed is a numerical accuracy issue, probably it is not a real bug and the reference data on disk should be updated with the most recent version of ft_componentanalysis

Jan-Mathijs Schoffelen - 2012-10-24 13:29:31 +0200

this issue is related to test_ft_componentanalysis failing. The general discussion in FT-meeting (24-Oct-2012) went into the direction of questioning the usefulness of comparing newly computed data against data stored on disk. The issue also applies to test_ft_timelockanalysis and test_ft_freqanalysis. In general, users should be able to expect that the numerical output does not change (unless the earlier version had a bug in the algorithm); another discussion is whether the implementation of the algorithm is correct. In the case of componentanalysis, it was questioned whether the reference data should be used at all for the second purpose. Better to use 'generated' data in this case. The latter purpose suggests that many test-functions need to be extended as to also test the correctness of the implementation.

Jim Herring - 2013-11-03 19:57:00 +0100

In response to Robert's e-mail from 2-11-2013 on the recent regression related errors I had a look at test_ft_componentanalysis. It is now failing for k=12 due to slight differences between the calculated components and the data on disk. The error does not appear when I comment out the scaling so I think the current failing of test_ft_componentanalysis is related to the discussion in this bug.