Back to the main page.
Bug 3439 - OpenMEEG test does not pass
| Status | CLOSED FIXED |
| Reported | 2018-07-12 19:44:00 +0200 |
| Modified | 2018-07-19 18:26:22 +0200 |
| Product: | FieldTrip |
| Component: | forward |
| Version: | unspecified |
| Hardware: | All |
| Operating System: | All |
| Importance: | P5 critical |
| Assigned to: | Sarang Dalal |
| URL: | |
| Tags: | |
| Depends on: | |
| Blocks: | 3440 |
| See also: |
Sarang Dalal - 2018-07-12 19:44:07 +0200
There's a report that the OpenMEEG test script does not produce the expected results: https://lists.gforge.inria.fr/pipermail/openmeeg-info/2018/000254.html I have confirmed this also happens for me with OpenMEEG v2.3 and 2.4, on Matlab R2016a and Matlab R2018a.
Sarang Dalal - 2018-07-12 19:54:14 +0200
Test script updated in https://github.com/fieldtrip/fieldtrip/pull/740
Robert Oostenveld - 2018-07-13 09:39:07 +0200
I am running regression tests every night. All test functions in fieldtrip/test get executed on our linux cluster. This is using (our default) openmeeg/2.2.0 and I see in the latest logs test_headmodel_openmeeg PASSED in 38 seconds using If you want to see how the tests are executed, please go to https://github.com/fieldtrip/dashboard, where a cron job starts schedule-tests.sh, which in turn schedules a large batch of run-test.sh jobs. Each job starts MATLAB and executes one of the fieldtrip/test/test_xxx.m functions.
Sarang Dalal - 2018-07-13 10:57:45 +0200
Thanks Robert. It seems the only relevant test in that directory is test_headmodel_openmeeg.m It does produce sensible plots when I run it manually, but it seems the test would only check for a crash of the script rather than erroneous output. The test script I was referring to is external/openmeeg/testOpenMEEGeeg.m, which also checks that the output is within a certain tolerance of expected results.
Robert Oostenveld - 2018-07-13 12:38:57 +0200
(In reply to Sarang Dalal from comment #3) I also see test_bug1042.m test_bug1082.m test_bug1368.m test_bug1756.m test_bug686.m test_bug70.m test_headmodel_openmeeg.m that have some relation to openmeeg. Furthermore, there are failed_eeg_leadfield_units.m failed_ft_prepare_headmodel.m failed_tutorial_headmodel_eeg.m obsolete_bug2338.m obsolete_headmodel_openmeeg_new_old.m which are not executed.
Robert Oostenveld - 2018-07-13 12:43:44 +0200
(In reply to Sarang Dalal from comment #3) Regarding external/openmeeg/testOpenMEEGeeg.m, on line https://github.com/fieldtrip/dashboard/blob/f7b2a8d1142e0fe819b2c3c5e80e72a992b04adc/schedule-tests.sh#L55 I see that it only tests when the script is called test/test_*, i.e. in a "test" dir and with an underscore. If you want the script to execute regularly, you should move the script into openmeeg/test, and rename it to test_openmeeg.m.
Sarang Dalal - 2018-07-13 13:01:50 +0200
While the other test functions make some OpenMEEG calls, it seems they likewise would only fail if there's an actual crash. It looks like testOpenMEEGeeg.m may be the only one that tests whether the output yields the expected values, so I think that's a good suggestion to move it so it automatically runs. @Alex, as far as I see, this test function exists only locally in FieldTrip? Or do we need to worry about syncing with the OpenMEEG repository? (I ask because it's currently located in external/openmeeg)
Alexandre Gramfort - 2018-07-13 13:53:35 +0200
testOpenMEEGeeg.m was written before the testing system of fieldtrip was setup AFAIK. we already test the C++ openmeeg code in our repo. testOpenMEEGeeg.m tests the FieldTrip integration. I think the test should be run with OpenMEEG release version. Alex
Sarang Dalal - 2018-07-13 14:09:06 +0200
I think @Robert would have to help with implementing testing with different OpenMEEG versions, since the automated tests run on a Donders server that probably has only a single version (2.2?) currently installed.
Sarang Dalal - 2018-07-13 14:09:59 +0200
If no objections Alex, I'll open a PR later today to move to the tests directory then.
Sarang Dalal - 2018-07-13 17:00:20 +0200
moved to tests: https://github.com/fieldtrip/fieldtrip/pull/743