Back to the main page.
Bug 2342 - ft_connectivity_csd2transfer: zigzags
| Status | CLOSED FIXED |
| Reported | 2013-10-28 14:28:00 +0100 |
| Modified | 2014-01-15 14:47:22 +0100 |
| Product: | FieldTrip |
| Component: | core |
| Version: | unspecified |
| Hardware: | PC |
| Operating System: | Windows |
| Importance: | P3 normal |
| Assigned to: | Jan-Mathijs Schoffelen |
| URL: | |
| Tags: | |
| Depends on: | |
| Blocks: | |
| See also: |
Jan-Mathijs Schoffelen - 2013-10-28 14:28:10 +0100
Get a better understanding about the zigzags that sometimes show up when doing spectral factorization. Some insights: -In the past there was an issue with the DC-bin, causing a lot of trouble: this has been addressed in current versions of the code (i.e. explicit demeaning prior to spectral estimation, and applying an appropriate weight when creating the two-sided spectra) -Craig and JM at some point found out that the initialization of the algorithm sometimes caused the zigzags (i.e. sometimes with a cholesky decomposition of a random matrix there was a zigzag, while taking a 'data driven' initialization got rid of it, or the other way around). Georgios and JM discussed: -It could well be a numerical issue, that is amplified by the sequential fft's and ifft's (wit intermediate zeroing out of part of the spectrum, potentially causing discontinuities). -It could be caused by a suboptimal implementation of the fft algorithm used -> action point for Georgios -It could be caused by the signals being too collinear (i.e. possibly almost identical at high frequencies). Adding a tiny bit of uncorrelated noise would address this. -> action point for JM -To keep in mind (if the discontinuity in the spectrum is the cause of all this: investigate of post-hoc tapering of the zero'd spectrum (i.e. convolving with the fft of a taper) smoothes out the discontinuity. At some point Martin and JM discussed: -In the algorithm, at some point the spectral matrix is sandwiched between the inverse of the transfer matrix. If this inverse is not well behaved it could lead to numerical problems as well
Craig Richter - 2013-10-28 14:52:37 +0100
We can also bring Dhamala, Ding and Rangarajan into the discussion at any point. I think they would be interested in helping and were before. So once we develop a test set which reproduces the zigzags, it might be a good idea to send it to them and see if they can see where things get pathological.
Jan-Mathijs Schoffelen - 2013-10-30 12:45:09 +0100
[[this comment was removed]]
Jan-Mathijs Schoffelen - 2013-10-30 12:59:04 +0100
I created a test function on fieldtrip/test/test_bug2342.m Initial conclusion: padding on synthetic data makes a bit of a difference, but does not help in the extreme case, when the time series are collinear (leading to huge zigzags, probably because the inverse of the 'psi' matrix blows up).
Jan-Mathijs Schoffelen - 2013-10-30 13:04:24 +0100
getting back to my initial conclusion: there's a typo in the function cfg.padding should read cfg.pad. effect of padding still to be investigated...
Martin Vinck - 2013-10-30 19:14:19 +0100
(In reply to comment #2) hi JM, could you please remove the quoted mail from this exchange? It was not intended to be public.
Jan-Mathijs Schoffelen - 2013-10-31 20:14:43 +0100
Wordt aan gewerkt, maar lijkt minder makkelijk dan ik dacht. Verzoek ligt (via Robert) bij TG.
Jan-Mathijs Schoffelen - 2013-11-04 09:29:00 +0100
Created attachment 552 granger with zigzag
Jan-Mathijs Schoffelen - 2013-11-04 09:29:15 +0100
Created attachment 553 granger with zigzag smoothed
Jan-Mathijs Schoffelen - 2013-11-04 09:29:47 +0100
Created attachment 554 granger with proper weighting of DC bin prior to NPSF
Jan-Mathijs Schoffelen - 2013-11-04 09:36:23 +0100
With one of the kids keeping me awake last night, I realized that there may be an issue regarding how the sfactorization_wilson code creates the two-sided spectral density matrix from the one-sided input. The code assumes all frequency bins to be weighted with 1, whereas ft_freqanalysis explicitly weights the non-DC and non-Nyquist frequency bins by a factor of 2, to account for the 'negative frequencies'. I therefore weighted the DC-bin twice as heavy when creating the two-sided spectral density, and computed Granger causality for a pair of channels (based on real data) with massive zigzags (i.e. Granger with zigzag figure). The result is shown in the attachment "Granger with proper weighting of DC bin prior to NPSF". I will incoporate this change in the code. Fortunately, the results don't look altogether too different (for this channel pair) when comparing it with the smoothed zigzags (figure attached: Granger with zigzag smoothed).
Robert Oostenveld - 2013-11-14 09:31:08 +0100
(In reply to comment #5) Comment #2 has been removed.
Jan-Mathijs Schoffelen - 2013-11-14 09:33:55 +0100
(In reply to comment #11) Great, thanks very much on behalf of Martin.
Jan-Mathijs Schoffelen - 2013-11-14 09:35:42 +0100
With my DC-bin correction I have not seen any significant zigzags of late (in combination with Martin's suggestion to use more frequency bins). I find the tapering of the gamp a bit ad hoc, and would not support this as a standard solution. Given the above, I suggest to set this bug as fixed for the moment. @Georgios: feel free to re-open if you feel like addressing this a bit further along the lines we discussed.
Martin Vinck - 2013-11-14 09:44:41 +0100
(In reply to comment #13) Hi JM Thanks for removing that! I agree that my 'solution' is a bit ad-hoc. At least it should be theoretically motivated, which I think is quite possible, and I'm working on this - I'll let you know when that worked out. Best Martin.
Jan-Mathijs Schoffelen - 2013-11-14 09:48:53 +0100
Hi Martin, Could you quickly see how it now behaves (without the tapering) after my change to the DC-bin weighting? PS: if you are in for a joint project, let me know!
Giorgos Michalareas - 2013-11-14 19:04:26 +0100
JM wrote: "With my DC-bin correction I have not seen any significant zigzags of late (in combination with Martin's suggestion to use more frequency bins). I find the tapering of the gamp a bit ad hoc, and would not support this as a standard solution. " Regarding the above it seems that the reduction of zigzags comes from 2 modifications: - DC-bin correction - Use of more frequency bins A first question is : Did u try what happens only with the DC-bin correction without the larger number of frequencies? Is the zig zag reduced dramatically just by the DC-bin correction? I am mostly asking to get an idea about the validity of existing results , computed prior to the DC-bin correction. A second question regarding the larger number of frequency bins is the following: Are the zigzags reduced if you just keep the frequency range(meaningfully wide) fixed and just increase the resolution? i.e. freqRange1= 1:1:100 as compared to freqRange2=1:0.1:100. Or is it that you need a larger frequency range? i.e freqRange1=1:1:100 as compared to freqRange2=1:1:500 ? I ll give it also a try this weekend. Giorgos
Martin Vinck - 2013-11-14 23:05:19 +0100
(In reply to comment #15) hi JM It seems you corrected a bug indeed. This helps a bit as the DC would cause additional ringing. However, the zig-zagging issue is not fully solved. In the test script I sent you by mail you'll see that the zig-zagging is still there. It's not just the zig-zagging but in some cases of zig-zagging the algorithm completely goes astray. As you can see the 'solution' I proposed strongly reduces the problem (if my test script is correct, at least) although as you'll see there are cases where it fails. Best Martin.