Back to the main page.
Bug 1953 - Clarify the correctm naming for holm, holms, fdr
| Status | CLOSED FIXED |
| Reported | 2013-01-24 15:25:00 +0100 |
| Modified | 2014-03-12 12:21:33 +0100 |
| Product: | FieldTrip |
| Component: | core |
| Version: | unspecified |
| Hardware: | Other |
| Operating System: | Mac OS |
| Importance: | P3 normal |
| Assigned to: | Jörn M. Horschig |
| URL: | |
| Tags: | |
| Depends on: | |
| Blocks: | |
| See also: |
Robert Oostenveld - 2013-01-24 15:25:46 +0100
This came up in the EDA and needs to be clarified. I suggest to make a faq and check the cfg.correctm option in the stats functions.
Jörn M. Horschig - 2013-01-24 16:35:01 +0100
cfg.correctm ft_statistics_montecarlo string, apply multiple-comparison correction, 'no', 'max', cluster', 'bonferoni', 'holms', 'fdr' (default = 'no') The names look alright to me. I'm gonna check what they are doing (in terms of source code) and write a FAQ. Actually, after talking to Rene and looking at Wikipedia again it seems like Holm's is controlling the FWER but not the FDR
Robert Oostenveld - 2013-01-24 16:42:27 +0100
is the person named holm or holms, and his method holm's or holms'
Jörn M. Horschig - 2013-01-24 16:49:56 +0100
He's called Holm, the method is usally referred to as Holm-Bonferroni method. So you think it might be better to called it holm rather than holms? (which would then stand for Holm's method) Btw, I just realized, bonferoni with one r is incorrect, the guy's called Bonferroni (Carlo Emilio)
Robert Oostenveld - 2013-01-24 17:15:15 +0100
(In reply to comment #3) so two names to fix then. That should be using ft_checkconfig with renamedvalue.
rené scheeringa - 2013-01-24 17:21:52 +0100
In relation to the discussion in the EDA, I just read the wikipedia on Holm-Bonferoni. If the entry on HOLM-bonferroni is correct, Jorn got Holm-Bonferroni and Hochberg mixed up. Apparently Holm-Bonferroni can be applied in any case, while Hochberg may only be applied when observations are independent or "under some forms of positive dependence". Which version is implemented in fieldtrip under Holm?
Jörn M. Horschig - 2013-01-25 08:08:07 +0100
wikipedia is pretty inconsistent, cause it explains the Holm method different depending on what page of wikipedia you are. I'm gonna check some papers to get a grip on this.
Robert Oostenveld - 2013-01-25 12:10:32 +0100
(In reply to comment #6) I would expect http://en.wikipedia.org/wiki/Holm–Bonferroni_method to be authoritative.
Jörn M. Horschig - 2013-01-25 12:15:55 +0100
Could be, but http://en.wikipedia.org/wiki/Familywise_error_rate#Holm.27s_step-down_procedure_.281979.29 The scientific article from Holm might be better than wikipedia ;)
Robert Oostenveld - 2013-01-25 12:26:40 +0100
(In reply to comment #8) Although phrased slightly differently, these two descriptions are consistent, right? Or do I overlook something?
Jörn M. Horschig - 2013-01-25 12:35:12 +0100
no, the FWER page says Holm looks for the largest k such that P_k > (...). The FWER page actually makes the point that the Hochberg procedure is different from Holm, cause Hochberg looks for the smallest k such that P_k < (...), but the Holm-bonferroni page states that that is the algorithm that searches for the smallest k such that P_k < (...). Pretty confusing.
Robert Oostenveld - 2013-01-25 13:52:08 +0100
(In reply to comment #10) http://en.wikipedia.org/wiki/Familywise_error_rate#Holm.27s_step-down_procedure_.281979.29 states Let R be the largest k such that [[ something is smaller than or equal to... ]] Or find the last R=k such that [[ something is smaller than or equal to... ]] It then rejects H(1) up to H(R) For H(R) the P is smaller than ... For H(R+1) the P would be larger than ... and http://en.wikipedia.org/wiki/Holm–Bonferroni_method states Let k be the minimal index such that [[ something is larger than... ]] Or find the first k where [[ something is larger than... ]] It then rejects H(1) up to H(k-1). For H(k-1) the P is smaller than ... For H(k) the P would be larger than ... Is that not exactly the same? The confusion might be due to the <= and > comparisons [1 2 3 4 5 6 7 8 9] <=5 returns 1:5, i.e. 5 as last [1 2 3 4 5 6 7 8 9] >5 returns 6:9, i.e. 6 as first So R=k in the first is equal to k-1 in the second comparison.
Roemer van der Meij - 2013-01-25 14:01:52 +0100
I agree with Robert. It is unfortunate that the statement is mirrored in the two explanations. On FDR vs FWER (the rate's, not the method of correction, like the following: Controlling FWER is a form of simultaneous inference, where all inference made in a family are jointly corrected up to a pre-specified error rate. Depending on the definition of the family, the researcher might choose a different form of inference: For example, simultaneous inference may be too conservative for certain large-scale problems that are currently being addressed by science. For such problems, a selective inference approach might be more suitable, since it assumes that any sub-group of hypotheses from the large scale group can be viewed as a family. Selective inference is usually performed by controlling the FDR (false discovery rate criteria). FDR controlling procedures are more powerful (i.e. less conservative) procedures than the familywise error rate (FWER) procedures (such as the Bonferroni correction), at the cost of increasing the likelihood of false positives within the rejected hypothesis. From the wikipedia page on FWER: http://en.wikipedia.org/wiki/Familywise_error_rate#Simultaneous_inference_vs._selective_inference
Roemer van der Meij - 2013-01-25 14:02:40 +0100
(In reply to comment #12) I meant to say (as a header): On FDR vs FWER (the rate's, not the method of correction), I like the following:
Jörn M. Horschig - 2013-01-25 16:45:08 +0100
Created attachment 416 Difference between Holm and Hochberg method according to Wikipedias FWER page (In reply to comment #11) no, I don't think you are right. The simplified example Robert gave is not appropriate, because we are shooting at a moving target here. See the attached figure. If you prove me wrong, I would be happy though. So, for me the question remains, which of the two Wiki-pages is correct.
rené scheeringa - 2013-01-25 17:44:58 +0100
So what I think the labels in de the figure are mixed up. Holm-Bonferroni is 6, while Hochberg is 8. Holm-Bonferroni is the more conservative method that is applicable in any case, while for Hochberg you should be able to assume independence (or special cases of positive dependence apperently). This is also how I was thought the Holm method years ago in Groningen.
Robert Oostenveld - 2013-01-28 10:15:05 +0100
(In reply to comment #14) I see. The "problem" is in 6 and 8 being below and 7 being above the line, which I had not considered in comment #11. That makes the two descriptions of Holm's procedure inconsistent. On identifies R=k=8 as the largest below the line, and rejects H0 for 1:8. The other identifies k=7 as the smallest above the line and rejects H0 for 1:6. Conclusion: we have two description of Holm's procedure that are inconsistent. Hochberg is also described twice. I find the page http://en.wikipedia.org/wiki/Familywise_error_rate to be least precise as it suggests a step-wise procedure but does not make the steps explicit, whereas the separate page on Holm explains it irrespective of the stepping. ---- I found this explicit comparison http://www.stat.osu.edu/~jch/PDF/HuangHsuPreprint.pdf which is consistent with Holm's resulting in 1:6 and Hochberg in 1:8. It suggests that http://en.wikipedia.org/wiki/Holm–Bonferroni_method (which also mentions Hochberg) is correct. It also states: "Holm’s method is based on the Bonferroni inequality and is valid regardless of the joint distribution of the test statistics. Hochberg’s method is more powerful than Holm’s method, but the test statistics need to be independent or have a distribution with multivariate total positivity of order two or a scale mixture thereof for its validity (Sarkar, 1998)." ----- I guess we now have enough insight to fix the implementation in "ft_statistics_analytic.m" around line 94.
Jörn M. Horschig - 2013-01-31 11:53:39 +0100
the algorithm actually is step-wise, and imho it's also explained that way on both pages. Indepently of who's to name/blame, the step-up procedure starts at the lowest p and stops as soon as a p-value is above the threshold, the step-down procedure starts at the largest and stops as soon a p-value is below the threshold. In any case, the current FT implementation does not obey to any of these two. It just takes all p-values below it's individual threshold: mask = pvals<=(cfg.alpha ./ ((length(pvals):-1:1)')); I changed this to: k = find(pvals > (cfg.alpha ./ ((length(pvals):-1:1)')), 1, 'first'); mask = (1:length(pvals))'<k; This way, it is: a) consistent with http://en.wikipedia.org/wiki/Holm%E2%80%93Bonferroni_method#The_Holm-Bonferroni_method b) in the same nomenclature as that page c) easily changeable to the Hochberg method in case we are proven wrongly. svn ci ft_statistics_montecarlo.m -m "bugfix #1953 - Holm-Bonferroni adjustment corrected, cfg.clusterm options are more correctly named now (bonferroni and holm rather than bonferoni and holms)" Sending ft_statistics_montecarlo.m Transmitting file data . Committed revision 7429.
Robert Oostenveld - 2013-01-31 12:07:13 +0100
(In reply to comment #17) mac001> svn commit ft_statistics_* Sending ft_statistics_analytic.m Sending ft_statistics_montecarlo.m Transmitting file data .. Committed revision 7430. I also corrected the holm method for analytic probability (just like for montecarlo method), and fixed the r-rr in bonferroni. Small change in whitespace in ft_statistics_montecarlo
Robert Oostenveld - 2013-01-31 12:08:04 +0100
(In reply to comment #17) why not implement hochberg while you are at it?
Jörn M. Horschig - 2013-01-31 12:26:33 +0100
okidoki ;) svn ci ft_statistics_montecarlo.m ft_statistics_analytic.m -m "enhancement #1953 - Hochberg's method implemented" Sending ft_statistics_analytic.m Sending ft_statistics_montecarlo.m Transmitting file data .. Committed revision 7431. I explicitly state that this is not the Bejamini-Hochberg method in the fprintf statement. It might be confused with that one. Also, I'm gonna edit Wikipedia's FWER site so that people might not get confused there (I hope the wiki admins see the inconsistency immediately)
rené scheeringa - 2013-01-31 12:49:54 +0100
Created attachment 417 Original Holm paper (1979)
rené scheeringa - 2013-01-31 12:52:31 +0100
Maybe something we should have done some mails ago, but I downloaded the original Holm paper from 1979. Scheme 1 is actually very clear about the procedure. See attachment.
Robert Oostenveld - 2013-01-31 13:12:39 +0100
(In reply to comment #21) thanks I have added it to the reference list http://fieldtrip.fcdonders.nl/references_to_implemented_methods?&#statistics_for_eeg-_and_meg-data perhaps also add the one from Kiebel (?) that you referred to in the presentation.