Back to the main page.
Bug 3365 - invalid spm_bwlabel mex mac file under matlab2017b
| Status | CLOSED WONTFIX |
| Reported | 2017-10-21 07:03:00 +0200 |
| Modified | 2019-08-10 12:37:20 +0200 |
| Product: | FieldTrip |
| Component: | external |
| Version: | unspecified |
| Hardware: | PC |
| Operating System: | Mac OS |
| Importance: | P5 normal |
| Assigned to: | Arjen Stolk |
| URL: | |
| Tags: | |
| Depends on: | |
| Blocks: | |
| See also: |
Arjen Stolk - 2017-10-21 07:03:51 +0200
The spm_bwlabel mex mac file that works under MATLAB 2015b, no longer works under MATLAB 2017b. This is an SPM8 mex file (spm_bwlabel.mexmaci64). When replacing it with the SPM12 equivalent, all runs fine. However, it's seem impossible to enforce the use of SPM12 when calling ft_freqstatistics. When calling the following, rmpath(spm('dir')); ft_hastoolbox('spm12',1); the same error shown below appears, suggesting it tried to use the SPM8 mexfile. Proposed (untested) solution #1: allow the use of SPM12 (i.e. why does it still use spm8 if spm12 is explicitly added to the path?) Proposed (untested) solution #2: fix/replace the spm8 mexfile by the spm12 mexfile. Potential downside is that this mexfile maybe not supports other MATLAB versions (@Guillaume?) The error: Warning: adding /Users/arjsto/Documents/MATLAB/fieldtrip/external/spm8 toolbox to your MATLAB path Invalid MEX-file '/Users/arjsto/Documents/MATLAB/fieldtrip/external/spm8/spm_bwlabel.mexmaci64': dlopen(/Users/arjsto/Documents/MATLAB/fieldtrip/external/spm8/spm_bwlabel.mexmaci64, 6): Library not loaded: @loader_path/libmex.dylib Referenced from: /Users/arjsto/Documents/MATLAB/fieldtrip/external/spm8/spm_bwlabel.mexmaci64 Reason: image not found. Error in findcluster (line 87) [labelmat(spatdimlev, :, :), num] = spm_bwlabel(double(reshape(onoff(spatdimlev, :, :), nfreq, ntime)), 6); % the previous code contained a '4' for input Error in clusterstat (line 201) posclusobs = findcluster(reshape(postailobs, [cfg.dim,1]),channeighbstructmat,cfg.minnbchan); Error in ft_statistics_montecarlo (line 348) [stat, cfg] = clusterstat(cfg, statrand, statobs); Error in ft_freqstatistics (line 193) [stat, cfg] = statmethod(cfg, dat, design); 87 [labelmat(spatdimlev, :, :), num] = spm_bwlabel(double(reshape(onoff(spatdimlev, :, :), nfreq, ntime)), 6); % the previous code contained a '4' for input
Arjen Stolk - 2017-10-21 07:05:12 +0200
Created attachment 850 test script
Robert Oostenveld - 2017-10-22 14:29:54 +0200
did you use the cfg.spmversion option? By default that is spm8, you can specify spm12.
Robert Oostenveld - 2017-10-22 14:31:24 +0200
I think you can also use 'spm8up', in which case it will be happy to find spm12 on the path.
Arjen Stolk - 2017-10-23 02:06:49 +0200
Thanks, I didn't know one could use spmversion with stats functions as well. For a proof of concept, cfg.spmversion = 'spm12'; stat = ft_freqstatistics(cfg, A, B); Warning: adding /Users/arjsto/Documents/MATLAB/fieldtrip/external/spm12 toolbox to your MATLAB path found 1 positive clusters in observed data found 0 negative clusters in observed data computing clusters in randomization computing clusters in randomization 1000 from 1000 using a cluster-based method for multiple comparison correction the returned probabilities and the thresholded mask are corrected for multiple comparisons the call to "ft_freqstatistics" took 1 seconds and required the additional allocation of an estimated 9 MB
Robert Oostenveld - 2017-10-23 06:16:59 +0200
(In reply to Arjen Stolk from comment #4) You can only use it in stats functions as of recently, when I encountered the same issue as you. I have this in my startup.m global ft_default ft_default.spmversion = 'spm12' ft_defaults % this loads the rest of the defaults The global ft_default variable is always merged with the cfg of each FT function, so this ensures that all FT functions use spm12.
Arjen Stolk - 2017-10-23 06:40:20 +0200
Excellent, that is useful. I presume we're not making spm12 the default given that it it would produce potentially mismatching results in people unknowingly using spm8 before such a transition and spm12 after.
Guillaume - 2017-10-23 17:20:26 +0200
Instead of copying the compiled MEX files from SPM12 to SPM8, I would recompile the SPM8 MEX files with MATLAB R2017a or R2017b. In my experience they are backward compatible with older MATLAB versions.
Arjen Stolk - 2017-10-27 00:10:47 +0200
Reopening this thread to discuss whether we should not recompile the spm8 files ourselves (or test/pull the ones contributed by Alan Bush, https://github.com/fieldtrip/fieldtrip/pull/555). If I understand correctly from Guillaume, compiling with v2017b would keep the files backward-compatible. Essentially, a bulk of the spm8-mex functionalities that are in fieldtrip have become inaccessible to those using v2017b (and perhaps also 2017a). From my limited but issue-free experience with using spm12 in v2017b, I cannot tell as of yet whether the spm12 mex files need recompilation too.
Robert Oostenveld - 2017-10-27 09:48:08 +0200
(In reply to Arjen Stolk from comment #8) We currently have 5 platforms to worry about - win32 - win64 - linux32 - linux64 - osx64 Ideally I would update the mex files for all platforms, and if we can confirm that the mex files both work with 2017b and 2012b (i.e. 5 years old). The only platform that is really easy to test for me is linux64: I have a script that kicks off the complete test batch on all possible MATLAB versions on our compute cluster. But for all other platforms that would be too heavy, so we should figure out a single (or a few) test scripts that should be sufficient and that do not take 12 hours of execution time in total. But given that we don't have reports of problems on platforms other than OSX, I suppose that we should only consider that. For OSX I am a bit worried that there might be an interaction between MATLAB version and operating system version, since the system libraries will be different between OS versions. See https://en.wikipedia.org/wiki/MacOS_version_history. I can test on El Captain 10.11.6. Who can test on newer OSX/MacOS versions?
Jan-Mathijs Schoffelen - 2017-10-27 13:13:54 +0200
I have a macbook pro running 10.12.6, but currently have only matlab2015b installed...
Robert Oostenveld - 2017-10-27 15:34:54 +0200
(In reply to Jan-Mathijs Schoffelen from comment #10) We have the same license, so I could share my installed versions with you. In the past it worked fine to simply copy the MATLAB_xxx.app folder. The only challenge is the size, together it is 16 GB. mac011> du -hs MATLAB_R20???.app 4.5G MATLAB_R2012b.app 12G MATLAB_R2017b.app I propose we do that when we are both at the DCCN, by making a temp network share on my computer, or by doing an rsync in the terminal (i.e. I would type my ssh password on your computer at the start of the transfer).
Arjen Stolk - 2017-10-27 18:00:36 +0200
Running the latest 10.12.6 as well, and v2015b and v2017b installed. Ehmm, quick check to make sure we're not reinventing the wheel: @Guillaume, does spm provide forward compatibility for spm8? That is, do you guys already have recompiled and tested the mex files in spm8 to make it compatible with v2017b (and earlier, on different platforms)? If so, that would make our lives a bit easier. :)
Robert Oostenveld - 2018-01-24 12:50:49 +0100
should we be updating the mex files? should we include a warning in ft_hastoolbox, e.g. show something when it is called to add external/spm or external/gifti to the path. should we be documenting this as a faq on the FT wiki? -> I opt for yes.
Robert Oostenveld - 2018-01-24 12:51:02 +0100
*** Bug 3402 has been marked as a duplicate of this bug. ***
Robert Oostenveld - 2018-01-24 12:59:37 +0100
(In reply to Robert Oostenveld from comment #13) in terms of documentation we now have http://www.fieldtriptoolbox.org/tag/mex?do=showtag&tag=mex @Arjen, you want to add a FAQ on spm mex files?
Arjen Stolk - 2018-01-24 17:53:34 +0100
should we be updating the mex files? > can we not fix the mex-files that we know of are throwing issues with the newer matlab versions (i.e. mexmaci64), rather than to preemptively recompile each and every mex-file that are possibly still fine? as for the spm8 mex-files on mac, it'd important to test the new ones on matlab v2017 and ad v2012 - just a note. should we include a warning in ft_hastoolbox, e.g. show something when it is called to add external/spm or external/gifti to the path. > this would create another layer of things that need to be adjusted when update mex-files. it would be easy for the warning to become misaligned with the current state of the code should we be documenting this as a faq on the FT wiki? -> I opt for yes. > such documentation should entail not only how to create standalone workarounds (recompiling) but also how those new mex-files can be tested (see above) in order to allow for the solution to be proposed on github
Robert Oostenveld - 2018-01-24 18:06:46 +0100
(In reply to Arjen Stolk from comment #16) From my usage tracking I can see that although 2017b seems the most used version (40% of the users), the majority of FT users is not using 2017b yet. There are multiple 2011a users (also recently). We have to live with this http://www.fieldtriptoolbox.org/development/guidelines/code#ensure_that_it_runs_on_older_matlab_versions so we need to find a strategy that does not piss of old-time users. perhaps make a fieldtrip/external/spm12-xxx perhaps make a fieldtrip/external/spm12-yyy with xxx and yyy being the MATLAB versions used for compilation? or we can make a faq like "if spm mex files fail, please delete the mex files, go to ftp.fieldtriptoolbox.org and download spm12-mexfiles-2017b.zip instead" Note that I have no reason to assume that the problem is limited to maci64. So the solution probably should be implemented for win32+win64 and linux32+64 as well.
Arjen Stolk - 2018-01-24 18:19:32 +0100
I think you have more insight and experience in this than I. There are also the known code differences between older and newer matlab versions, which we also cannot support in the form of say, separate ft_freqanalysis or subroutines therein? In terms of reproducibility of today's analyses in the near future (e.g. 5-10y) it would indeed be nice to have all possible versions of the toolbox being available still. However, things might become quite fragmented in that we would for instance also need a fieldtrip/external/spm12-zzz with matlab 2018? Seems there's a hard to judge tradeoff, from my perspective at least, between compatibility and efficiency.. Either way, the solution needs to be such that the code can still easily be updated from a developer's perspective. Too many patches and versions with obscure and deter this process.
Robert Oostenveld - 2018-01-24 18:25:05 +0100
(In reply to Arjen Stolk from comment #18) the code currently is compatible (tested and all) with 2012b. Even up to 2009b most of it runs. And keeping it compatible is not always easy. We have the code guidelines and fieldtrip/compat/matlabltxxxxx for less-than xxxxx version. All handled automagically inside Fieldtrip. But external is *external* to fieldtrip. I want to draw the line somewhere w.r.t. me taking care of the code. I propose you simply document what you know on a FT FAQ and that you point to the spm mex files that Guillaume or other people from the SPM team provide. I wish we had something like https://www.npmjs.com for MATLAB...
Arjen Stolk - 2018-01-24 19:09:09 +0100
(In reply to Robert Oostenveld from comment #19) Expanded an already existing FAQ related to this. See the bottom of this wiki: http://www.fieldtriptoolbox.org/faq/matlab_complains_about_a_missing_or_invalid_mex_file_what_should_i_do
Arjen Stolk - 2018-01-24 19:17:20 +0100
On a more general note, replacing or recompiling (external) MEX files won't fool git? That is, this needs to be done each time again when one updates their git master branch, making it a not so preferred solution for developers and users that continually update their fieldtrip version. As for SPM8, the use of SPM12 is a favorable workaround that does not require replacing/recompiling files. But for the GIFTI MEX file, there's currently no alternative.
Robert Oostenveld - 2018-01-25 17:25:30 +0100
(In reply to Arjen Stolk from comment #21) if you replace your local mex files and do not want your difference to show up when you type git diff, you can add a "fieldtrip/.gitignore" file. You can also add the .gitignore file itself to the list of files that is to be ignored. You can also maintain a full spm outside of fieldtrip. When you add that to your path prior to FT calling ft_hastoolbox(spm), it will prevail. In that case you can leave the fieldtrip/external/spm as it is.
Arjen Stolk - 2018-01-25 18:18:42 +0100
Thanks. Though, I'd prefer to have fieldtrip complete and up-to-date rather than creating solutions around it just to support backward-compatibility to the stone age. But I guess there's fewer people using the latest software than the older..
Robert Oostenveld - 2018-09-21 10:16:08 +0200
this is an SPM issue. The work-around for fieldtrip users is documented on http://www.fieldtriptoolbox.org/faq/matlab_complains_about_a_missing_or_invalid_mex_file_what_should_i_do#known_issues_with_spm_mex_files_and_workarounds
Guillaume - 2018-09-21 16:36:47 +0200
MEX file issues should be resolved using the latest version of SPM12: https://github.com/spm/spm12 or the maintenance version of SPM8: https://github.com/spm/spm8/tree/maint Otherwise, a recompilation is required. A Makefile is provided with SPM and instructions are available here: https://www.wikibooks.org/wiki/SPM#Installation
Robert Oostenveld - 2018-09-21 16:49:31 +0200
(In reply to Guillaume from comment #25) thanks for the update, I added them all to http://www.fieldtriptoolbox.org/faq/matlab_complains_about_a_missing_or_invalid_mex_file_what_should_i_do#known_issues_with_spm_mex_files_and_workarounds