Back to the main page.
Bug 1407 - finalize the egi_mff implementation
| Status | CLOSED WONTFIX |
| Reported | 2012-04-04 08:57:00 +0200 |
| Modified | 2015-02-11 10:40:01 +0100 |
| Product: | FieldTrip |
| Component: | fileio |
| Version: | unspecified |
| Hardware: | PC |
| Operating System: | All |
| Importance: | P4 enhancement |
| Assigned to: | Robert Oostenveld |
| URL: | |
| Tags: | |
| Depends on: | 1536 |
| Blocks: | 2466 |
| See also: |
Robert Oostenveld - 2012-04-04 08:57:10 +0200
EGI is about to make a final release of their Java implementation and MATLAB code. This bug is a placeholder for the testing and for finalizing the inclusion in FieldTrip.
Robert Oostenveld - 2012-04-04 09:00:47 +0200
see also bug #645 about the speed of reading the xml file containing the events
Robert Oostenveld - 2012-04-04 11:10:44 +0200
I have added the EGI files to fieldtrip/external/egi_mff/demoMFF.m fieldtrip/external/egi_mff/mff_getEEGFilename.m fieldtrip/external/egi_mff/mff_getObject.m fieldtrip/external/egi_mff/mff_getSummaryInfo.m fieldtrip/external/egi_mff/mff_micros2Sample.m fieldtrip/external/egi_mff/read_mff_data.m fieldtrip/external/egi_mff/read_mff_event.m fieldtrip/external/egi_mff/read_mff_header.m fieldtrip/external/egi_mff/write_mff_data.m fieldtrip/external/egi_mff/write_mff_event.m and fieldtrip/external/egi_mff/MFF-1.0.jar Subsequently I have added the line /Volumes/Data/roboos/matlab/fieldtrip/external/egi_mff/MFF-1.0.jar to /Applications/MATLAB_R2010b.app/toolbox/local/classpath.txt Note that this required administrator access to the computer, which in this case I luckily had on this particular computer, but which in general is probably not the case for all users. I made the required changes to the ft_read_xxx functions Sending fileio/ft_read_data.m Sending fileio/ft_read_event.m Sending fileio/ft_read_header.m Sending test/test_bug1407.m Transmitting file data .... Committed revision 5584. Note that I also made a test function, which you can see http://code.google.com/p/fieldtrip/source/browse/trunk/test/test_bug1407.m
Robert Oostenveld - 2012-04-04 11:15:16 +0200
Using the test that is implemented in test_bug1407 and the test data provided along with the EGI zip file, I get the following error >> hdr = ft_read_header('ATD256_3.ses.mff', 'headerformat', 'egi_mff_v2'); LocalSignalFile: error: bad optional header length: 64032 ??? Java exception occurred: java.lang.NegativeArraySizeException at com.egi.services.mff.utility.LocalSignalFile.signalBlock(LocalSignalFile.java:350) at com.egi.services.mff.utility.LocalSignalFile.loadSignalBlocks(LocalSignalFile.java:146) at com.egi.services.mff.utility.LocalSignalFile.unmarshal(LocalSignalFile.java:76) at com.egi.services.mff.api.MFFResource.loadResource(MFFResource.java:100) Error in ==> mff_getObject at 38 theObject.loadResource(); Error in ==> mff_getSummaryInfo>getEEGBlocks at 89 binObj = mff_getObject(com.egi.services.mff.api.MFFResourceType.kMFF_RT_Signal, EEGFile, filePath); Error in ==> mff_getSummaryInfo at 42 [binObj blocks] = getEEGBlocks(mfffileObj, filePath); Error in ==> read_mff_header at 12 summaryInfo = mff_getSummaryInfo(filePath); Error in ==> ft_read_header at 873 hdr = read_mff_header(filename); The same error happens when trying to read events.
Robert Oostenveld - 2012-04-04 11:18:01 +0200
Using the test that is implemented in test_bug1407 and another test dataset provided along with the EGI zip file, I get the following error >> hdr = ft_read_header('Long64ChannelWithEvents.mff', 'headerformat', 'egi_mff_v2'); MFFRootDoc: error: java.io.FileNotFoundException: /Volumes/Data/roboos/matlab/Long64ChannelWithEvents.mff/epochs.xml (No such file or directory) ??? Java exception occurred: com.egi.services.mff.utility.ResourceUnmarshalException: could not obtain resources at com.egi.services.mff.utility.XMLUtil.getDocumentFromURI(XMLUtil.java:101) at com.egi.services.mff.utility.XMLLocalFileResource.unmarshal(XMLLocalFileResource.java:497) at com.egi.services.mff.api.MFFResource.loadResource(MFFResource.java:100) Error in ==> mff_getObject at 38 theObject.loadResource(); Error in ==> mff_getSummaryInfo>getEpochInfos at 94 epochList = mff_getObject(com.egi.services.mff.api.MFFResourceType.kMFF_RT_Epochs, 'epochs.xml', filePath); Error in ==> mff_getSummaryInfo at 56 [epochType epochBeginSamps epochNumSamps epochFirstBlocks epochLastBlocks epochLabels epochTime0] = getEpochInfos(filePath, sampRate); Error in ==> read_mff_header at 12 summaryInfo = mff_getSummaryInfo(filePath); Error in ==> ft_read_header at 873 hdr = read_mff_header(filename); This suggests a path problem. I am in the correct directory: >> pwd ans = /Volumes/Data/roboos/matlab/fieldtrip/external/egi_mff/test >> ls ATD256_3.ses.mff Long64ChannelWithEvents.mff NS1000sps.mff NS500Sine6Hz.mff If I specify the full path to the file, it works >> hdr = ft_read_header('/Volumes/Data/roboos/matlab/fieldtrip/external/egi_mff/test/Long64ChannelWithEvents.mff', 'headerformat', 'egi_mff_v2') hdr = Fs: 250 nChans: 65 nSamplesPre: 0 nSamples: 3087000 nTrials: 1 label: {65x1 cell} chantype: {1x65 cell} chanunit: {1x65 cell} orig: [1x1 struct] This error is one that I can catch and deal with in the ft_read_xxx code.
Robert Oostenveld - 2012-04-04 11:29:53 +0200
(In reply to comment #4) I made an enhancement to ensure that the filename includes the full path manzana> svn commit ft_read_*m Sending ft_read_data.m Sending ft_read_event.m Sending ft_read_header.m Transmitting file data ... Committed revision 5585. Note that it does not deal with windows yet.
Robert Oostenveld - 2012-04-04 11:33:19 +0200
continuing with the same Long64ChannelWithEvents.mff test dataset from the EGI zip file (which now works for the header), I run into >> dat = ft_read_data('Long64ChannelWithEvents.mff', 'headerformat', 'egi_mff_v2', 'dataformat', 'egi_mff_v2') ??? Index exceeds matrix dimensions. Error in ==> read_mff_data at 36 data = data(:,beginSample:beginSample + (endInd-beginInd)); Error in ==> ft_read_data at 712 dat = read_mff_data(filename, 'sample', begsample, endsample, chanindx, hdr); In this case it should have returned the whole data as Nchans * Nsamples matrix. Some work needs to be done here.
Robert Oostenveld - 2012-04-04 11:37:31 +0200
again with the Long64ChannelWithEvents.mff test dataset the reading of events seems to work >> event = ft_read_event('Long64ChannelWithEvents.mff', 'headerformat', 'egi_mff_v2', 'eventformat', 'egi_mff_v2') event = 1x86 struct array with fields: type sample value offset duration timestamp sampleRemainder durationRemainder but there are two fields that should not be there. The presence of these fields does not comply to fieldtrip standards and therefore they have to be removed in ft_read_event. I have added the following code to that effect % clean up the fields in the event structure fn = fieldnames(event); fn = setdiff(fn, {'type', 'sample', 'value', 'offset', 'duration', 'timestamp'}); for i=1:length(fn) event = rmfield(event, fn{i}); end manzana> svn commit ft_read_event.m Sending ft_read_event.m Transmitting file data . Committed revision 5586.
Robert Oostenveld - 2012-04-04 11:44:24 +0200
reading the header for the 3rd and 4th dataset from the list works dataset = { 'ATD256_3.ses.mff' 'Long64ChannelWithEvents.mff' 'NS1000sps.mff' 'NS500Sine6Hz.mff' }; >> i = 3; >> hdr = ft_read_header(dataset{i}, 'headerformat', headerformat) hdr = Fs: 1000 nChans: 257 nSamplesPre: 0 nSamples: 60660000 nTrials: 1 label: {257x1 cell} chantype: {1x257 cell} chanunit: {1x257 cell} orig: [1x1 struct] >> i = 4; >> hdr = ft_read_header(dataset{i}, 'headerformat', headerformat) hdr = Fs: 1000 nChans: 257 nSamplesPre: 0 nSamples: 77113 nTrials: 1 label: {257x1 cell} chantype: {1x257 cell} chanunit: {1x257 cell} orig: [1x1 struct] Also reading the events seems to work (i.e. does not give an obvious error) >> i = 3; >> event = ft_read_event(dataset{i}, 'headerformat', headerformat, 'eventformat', eventformat) event = 1x2 struct array with fields: type sample value offset duration timestamp >> i = 4; >> event = ft_read_event(dataset{i}, 'headerformat', headerformat, 'eventformat', eventformat) event = 1x2 struct array with fields: type sample value offset duration timestamp Reading the data fails for the 3rd test set >> i = 3; >> dat = ft_read_data(dataset{i}, 'headerformat', headerformat, 'dataformat', dataformat); ??? Index exceeds matrix dimensions. Error in ==> read_mff_data at 36 data = data(:,beginSample:beginSample + (endInd-beginInd)); Error in ==> ft_read_data at 712 dat = read_mff_data(filename, 'sample', begsample, endsample, chanindx, hdr); Reading the data seems to work for the 4th test set >> i = 4; >> dat = ft_read_data(dataset{i}, 'headerformat', headerformat, 'dataformat', dataformat); The size of the data is consistent with the header, and the channels seem to contain a sine-wave.
Robert Oostenveld - 2012-04-04 11:47:59 +0200
the summary sofar is that dataset1: fails on hdr, event and data dataset2: fails on data dataset3: fails on data dataset4: is ok Please look in http://code.google.com/p/fieldtrip/source/browse/trunk/test/test_bug1407.m or the local copy of that file in your fieldtrip SVN checkout (*). *) note that test scripts are not present in the daily zipped fieldtrip release version from our ftp server, but only in the svn version.
Robert Oostenveld - 2012-04-07 15:51:21 +0200
###### copy and paste from email ###### Looking at egi_micros2Sample it gets clear to me that the remainder is expressed in microseconds. One simple implementation that I see for solution 1 (i.e. use timestamps) is to do event.timestamp = event.sample + (sampleRemainder/1e6)*Fsample; this causes the timestamps to be expressed in samples, i.e. event.sample is the rounded-off version of event.timestamp. The change to the header then is simply hdr.FirstSampleTimeStamp = 1; hdr.TimeStampPerSample = 1; Another implementation would express the timestamps in seconds, and then event.timestamp = (event.sample-1)/Fsample + (sampleRemainder/1e6); hdr.FirstSampleTimeStamp = 0; hdr.TimeStampPerSample = 1/Fsample; % how many seconds are there per sample or express it in microseconds as event.timestamp = 1e6*(event.sample-1)/Fsample + sampleRemainder; hdr.FirstSampleTimeStamp = 0; hdr.TimeStampPerSample = 1e6/Fsample; % how many microseconds are there per sample Either case is very simple and can be done on the fieldtrip side of the code, so would not require any changes [in the EGI provided code].
Robert Oostenveld - 2012-05-11 21:36:10 +0200
On 18 Apr 2012, at 20:44, Renner Tom wrote: There's now a newer revision, here: ftp://ftp.egi.com/pub/users/trenner/MFF_API_2012-04-09.zip This newer release has some important bug fixes and improvements; you should definitely use it instead of the older, April 03 version. -- Done. I have now added the matlab code and jar file to fieldtrip/external/egi_mff and updated to this latest version. This means that it will now be released along with FieldTrip.
Robert Oostenveld - 2012-05-11 22:09:40 +0200
I have just updated the test script and removed the corrupt test file (which is now also not in the EGI zip file any more). see http://code.google.com/p/fieldtrip/source/browse/trunk/test/test_bug1407.m mbp> svn commit test_bug1407.m Sending test_bug1407.m Transmitting file data . Committed revision 5772. On a native computer the test script results in the following error Undefined variable "com" or class "com.egi.services.mff.api.MFFResourceType.kMFF_RT_MFFFile". Error in mff_getSummaryInfo (line 39) mfffileObj = mff_getObject(com.egi.services.mff.api.MFFResourceType.kMFF_RT_MFFFile, [], filePath); Error in read_mff_header (line 12) summaryInfo = mff_getSummaryInfo(filePath); Error in ft_read_header (line 882) hdr = read_mff_header(filename); Error in test_bug1407 (line 39) hdr = ft_read_header(dataset{i}, 'headerformat', headerformat); This is due to the jar file not being correctly installed. At this moment I could go and dig into my personal emails, but of course it is required that the user (where user~=me) gets proper instructions. So the mff_xxx functions have to be extended with a try catch around the first access to the java object, and if the java object fails, the user should get an informative error and instructions. I suggest to implement a "mff_checkinstallation" function that performs this check and that contains all end-user instructions.
Robert Oostenveld - 2012-05-31 14:50:47 +0200
note to self (from the email) if ~(exist('com.egi.services.mff.api.MFFFactory') == 8) error('the EGI_MFF format requires that the jar file on your classpath') end if ~usejava('jvm') error('the EGI_MFF format requires that the java virtual machine is available') end should be added to ft_read_header, data and event.
Robert Oostenveld - 2012-06-05 23:42:21 +0200
(In reply to comment #13) the two additional checks have been implemented: mbp-2> svn commit ft_read*m Sending ft_read_data.m Sending ft_read_header.m Transmitting file data .. Committed revision 5915.
Robert Oostenveld - 2012-06-06 18:33:16 +0200
(In reply to comment #14 and comment #12) The ugly error message is after the update now replaced (on a native MATLAB installation) by ??? Error using ==> ft_read_header at 880 the EGI_MFF format requires the "MFF-x.y.jar" file to be on your classpath, see http://fieldtrip.fcdonders.nl/getting_started/egi Error in ==> test_bug1407 at 43 hdr = ft_read_header(dataset{i}, 'headerformat', headerformat); ---- If I do javaclasspath('/Volumes/Data/roboos/matlab/fieldtrip/external/egi_mff/java/MFF-1.0.jar') then test_bug1407 runs through without errors. See also http://code.google.com/p/fieldtrip/source/browse/trunk/test/test_bug1407.m for direct access to the test code. The FIXME on line 54 still needs to be addressed. Furthermore, visual inspection is needed by others to ensure that the data is meaningful.
Robert Oostenveld - 2012-06-06 18:40:12 +0200
it turns out to be possible to update the classpath on the fly, see http://www.mathworks.nl/help/techdoc/ref/javaclasspath.html So there is no reason to require a static entry in the (often user-non-writable) global classpath.txt file. I have changed the code accordingly, that will save the users a few headaches. manzana> svn commit ft_read*m Sending ft_read_data.m Sending ft_read_header.m Transmitting file data .. Committed revision 5923.
Ingrid Nieuwenhuis - 2012-06-07 23:46:29 +0200
I've just started testing with reading in a hdr like this: hdr_v2 = ft_read_header([cur_path_MFF, 'NS_EMR_002_S1'], 'headerformat', 'egi_mff_v2'); I'm at newest revision (way past 5923) but do get the following error: Error using ft_read_header (line 885) the EGI_MFF format requires the "MFF-x.y.jar" file to be on your classpath, see http://fieldtrip.fcdonders.nl/getting_started/egi So automatic adding did not work for me. I'll now try to add a path to the MFF-1.0.jar (java) file by typing “edit classpath.txt” on my matlab prompt and add the path to the .jar file, as explained on the FieldTrip wiki
Ingrid Nieuwenhuis - 2012-06-07 23:53:34 +0200
I can't add the path. When I type edit classpath.txt a file does open. But the first line says: # DO NOT MODIFY THIS FILE. IT IS AN AUTOGENERATED FILE. If I subsequently add the jar to the file anyway by adding this line to the bottom of my file: C:\Users\Ingrid\Documents\MATLAB\fieldtrip\external\egi_mff\java\MFF-1.0.jar When I subsequently save the file, I get a popup access denied (while I have administrator rights). Also, the explanation on the wiki is quite brief. I was unsure whether I should add the line on the top, or the bottom of the file. Also, whether my path is correct. The other entries all start with $matlabroot. I wasn't sure whether I should just do C:\ like I did. Could the explanation on the wiki be expanded to make it very explicit?
Ingrid Nieuwenhuis - 2012-06-08 00:04:18 +0200
I've found out what goes wrong with the automatic adding of the javaclasspath in ft_read_header line 879 and furter: if ~(exist('com.egi.services.mff.api.MFFFactory', 'class')==8) % try to add it on the fly to the dynamic classpath javaclasspath(fullfile(fileparts(which('mff_getEEGFilename')), 'java', 'MFF-1.0.jar')) end The path that is generated with this code is: 'C:\Users\Ingrid\Documents\MATLAB\java\MFF-1.0.jar' on my computer. However, the correct path should be: 'C:\Users\Ingrid\Documents\MATLAB\fieldtrip\external\egi_mff\java\MFF-1.0.jar' I've now added it manually with: javaclasspath('C:\Users\Ingrid\Documents\MATLAB\fieldtrip\external\egi_mff\java\MFF-1.0.jar') However, when I try to read the hdr again with: hdr_v2 = ft_read_header([cur_path_MFF, 'NS_EMR_002_S1'], 'headerformat', 'egi_mff_v2'); I now get a new error: Error using javaObject No constructor with appropriate signature exists in Java class com.egi.services.mff.api.MFFFile Error in mff_getSummaryInfo (line 3) mfffileObj = javaObject('com.egi.services.mff.api.MFFFile', filePath, true); Error in read_mff_header (line 2) summaryInfo = mff_getSummaryInfo(filePath); Error in ft_read_header (line 895) hdr = read_mff_header(filename); I have no clue what's this means or what to do about it.
Robert Oostenveld - 2012-06-08 11:45:38 +0200
manzana> svn commit test_bug1407.m Sending test_bug1407.m Transmitting file data . Committed revision 5986. I have just added an explicit comparison to the test script, see http://code.google.com/p/fieldtrip/source/diff?spec=svn5986&r=5986&format=side&path=/trunk/test/test_bug1407.m There is (at least) a problem in dataset 1, the last channel is labeled E65 in one and VREF in the other implementation. And the number of events is different (84 versus 86). This needs to be addressed by someone who knows more about the actual data format to decide whether implementation v1 or v2 should be considered faulty. I suggest that someone at EGI looks at this.
Ingrid Nieuwenhuis - 2012-06-11 18:30:17 +0200
My previous reported errors with the EGI test files are resolved and were indeed due to path to old code. Testing reading hdr: 1) EGI test files Long64ChannelWithEvents.mff and NS1000sps.mff read in without problems now. About labeling REF channel, I propose to keep it consistent with the layouts. 2) reading of my own MFF data: A) reading of my earliest subjects (Sept 2011), measured with older version of NetStation (but MFF data) Results in the following error: Error in mff_getSummaryInfo (line 39) mfffileObj = mff_getObject(com.egi.services.mff.api.MFFResourceType.kMFF_RT_MFFFile, [], filePath); Output argument "summaryInfo" (and maybe others) not assigned during call to "C:\Users\Ingrid\Documents\MATLAB\fieldtrip\external\egi_mff\mff_getSummaryInfo.m>mff_getSummaryInfo". Error in read_mff_header (line 12) summaryInfo = mff_getSummaryInfo(filePath); Error in ft_read_header (line 895) hdr = read_mff_header(filename); So the v2 code does not seem to be backward compatible for older MFF data. the v1 code works fine. I'm happy to supply EGI with these data for testing purposes. B) reading recent MFF data: hdr created without errors. problems: i)The PIB channels (in signal2) are not read in! ii)minor: v1:nSamples: 1339965, v2:nSamples: 1.3400e+009. I haven't compared the hdrs in detail, I'm happy to supply my data to any one who has more time for systematic testing. Testing reading events: 1) when I call event_v2 = ft_read_event('C:\Users\Ingrid\Documents\EGI_hdEEGdocs\MFF_API_2012-04-09\testfiles\Long64ChannelWithEvents.mff', 'headerformat', 'egi_mff_v2') The event still becomes read in by v1. In ft_read_event the headerformat is egi_mff_v2, but for some reason, if I subsequently stop in db on line 100 in ft_read_header, the headerformat has become egi_mff. No-one else had the same problem? I haven't updated FT since Friday (will do now), but I didn't hear anyone about this before... My testing 30 minutes of today are used up. I can test more tomorrow. And again, I'm happy to supply EGI with my data (containing PIB channels and old MFF format), just let me know where to put it. Cheers, Ingrid
Ingrid Nieuwenhuis - 2012-06-11 18:37:09 +0200
My previous reported errors with the EGI test files are resolved and were indeed due to path to old code. Testing reading hdr: 1) EGI test files Long64ChannelWithEvents.mff and NS1000sps.mff read in without problems now. About labeling REF channel, I propose to keep it consistent with the layouts. 2) reading of my own MFF data: A) reading of my earliest subjects (Sept 2011), measured with older version of NetStation (but MFF data) Results in the following error: Error in mff_getSummaryInfo (line 39) mfffileObj = mff_getObject(com.egi.services.mff.api.MFFResourceType.kMFF_RT_MFFFile, [], filePath); Output argument "summaryInfo" (and maybe others) not assigned during call to "C:\Users\Ingrid\Documents\MATLAB\fieldtrip\external\egi_mff\mff_getSummaryInfo.m>mff_getSummaryInfo". Error in read_mff_header (line 12) summaryInfo = mff_getSummaryInfo(filePath); Error in ft_read_header (line 895) hdr = read_mff_header(filename); So the v2 code does not seem to be backward compatible for older MFF data. the v1 code works fine. I'm happy to supply EGI with these data for testing purposes. B) reading recent MFF data: hdr created without errors. problems: i)The PIB channels (in signal2) are not read in! ii)minor: v1:nSamples: 1339965, v2:nSamples: 1.3400e+009. I haven't compared the hdrs in detail, I'm happy to supply my data to any one who has more time for systematic testing. Testing reading events: 1) when I call event_v2 = ft_read_event('C:\Users\Ingrid\Documents\EGI_hdEEGdocs\MFF_API_2012-04-09\testfiles\Long64ChannelWithEvents.mff', 'headerformat', 'egi_mff_v2') The event still becomes read in by v1. In ft_read_event the headerformat is egi_mff_v2, but for some reason, if I subsequently stop in db on line 100 in ft_read_header, the headerformat has become egi_mff. No-one else had the same problem? I haven't updated FT since Friday (will do now), but I didn't hear anyone about this before... My testing 30 minutes of today are used up. I can test more tomorrow. And again, I'm happy to supply EGI with my data (containing PIB channels and old MFF format), just let me know where to put it. Cheers, Ingrid
Ingrid Nieuwenhuis - 2012-06-11 18:48:08 +0200
In my impatience I created a double comment (bugzilla is slow!). Comment 21 == comment 22 = true;
Robert Oostenveld - 2012-06-12 09:53:44 +0200
On 11 Jun 2012, at 23:33, Colin Davey wrote: > The name of the channel is in the sensor layout xml file in the .mff directory. That's where our reader gets it. > > We should probably continue to get the name from the .xml file, and if we don't like VREF, then we should change what goes into the .xml file. It might also be possible that user-generated montages could result in different channel names. When and if that happens, it would be reflected in the .xml file. So, once again, the .mff reader should get these names directly from the data. > I have changed the v1 implementation so that it also gets the VREF from the sensorInfo.xml file. manzana> svn commit ft_read_header.m Sending ft_read_header.m Transmitting file data . Committed revision 6032.
Ingrid Nieuwenhuis - 2012-06-18 01:33:07 +0200
For v1, a bug was introduced in channel labeling, because also labels of COM and placeholders were grabbed from layout. Now only type 0 (EEG) and type 1 (REF) labels are grabbed from sensor layout, not type 2 (COM and placeholders). Also, when no name is available from sensorlayout, and type is 1 (meaning REF), electrode is now labeled "REF" instead of Exxx to be as consistent as possible.
Ingrid Nieuwenhuis - 2012-06-18 02:03:44 +0200
I've looked into why if I try to read an event with v2, v1 is still used. I call read_event as follows: event_v2 = ft_read_event('C:\Users\Ingrid\Documents\EGI_hdEEGdocs\MFF_API_2012-04-09\testfiles\Long64ChannelWithEvents.mff', 'headerformat', 'egi_mff_v2') However on line 109 in ft_read_event, eventformat comes back empty: % get the options eventformat = ft_getopt(varargin, 'eventformat'); and therefore gets set to v1 in line 112 if isempty(eventformat) % only do the autodetection if the format was not specified eventformat = ft_filetype(filename); end I tried to debug the ft_getopt function, to see why it doesn't return eventformat = 'egi_mff_v2', but when I put a debugging stop, I get the popup message that it's shadowed by ft_getopt.mexw64 file. Could this explain the different behavior on different systems?
Robert Oostenveld - 2012-06-18 14:10:02 +0200
(In reply to comment #26) I made the separate bug 1536 for this. One of the windows users in the FT team will soon look into this and fix it.
Ingrid Nieuwenhuis - 2012-06-18 18:49:45 +0200
Note, The ft_getopt thing is resolved, I no can call the read_event with the v2 option. However, when I call ft_read_event with v2, I get the following error: event_v2 = ft_read_event('C:\Users\Ingrid\Documents\EGI_hdEEGdocs\MFF_API_2012-04-09\testfiles\Long64ChannelWithEvents.mff', 'eventformat', 'egi_mff_v2') Warning: adding C:\Users\Ingrid\Documents\MATLAB\fieldtrip\external\egi_mff toolbox to your Matlab path Undefined variable "com" or class "com.egi.services.mff.api.MFFResourceType.kMFF_RT_MFFFile". Error in mff_getSummaryInfo (line 39) mfffileObj = mff_getObject(com.egi.services.mff.api.MFFResourceType.kMFF_RT_MFFFile, [], filePath); Error in read_mff_event (line 10) summaryInfo = mff_getSummaryInfo(filePath); Error in ft_read_event (line 808) event = read_mff_event(filename, hdr);
Ingrid Nieuwenhuis - 2012-06-25 19:58:42 +0200
I've now also tried reading event with the v2 version on my work PC (Windows 7, 64 bit, Matlab 2011b). I also get an error here, again in line 39 of mff_getSummaryInfo. Error in mff_getSummaryInfo (line 39) mfffileObj = mff_getObject(com.egi.services.mff.api.MFFResourceType.kMFF_RT_MFFFile, [], filePath); Output argument "summaryInfo" (and maybe others) not assigned during call to "C:\Users\Ingrid\Documents\MATLAB\fieldtrip\external\egi_mff\mff_getSummaryInfo.m>mff_getSummaryInfo". Error in read_mff_event (line 10) summaryInfo = mff_getSummaryInfo(filePath); Error in ft_read_event (line 808) event = read_mff_event(filename, hdr); SUMMARY COMMENT 28 and 29: I can't read events with v2 version on both my computers (both Windows 7, 64 bits). I get an error in mff_getSummaryInfo (line 39). Either Undefined variable "com" or class "com.egi.services.mff.api.MFFResourceType.kMFF_RT_MFFFile". or Output argument "summaryInfo" (and maybe others) not assigned during call to "C:\Users\Ingrid\Documents\MATLAB\fieldtrip\external\egi_mff\mff_getSummaryInfo.m>mff_getSummaryInfo".
Robert Oostenveld - 2012-08-08 11:50:09 +0200
I have changed http://fieldtrip.fcdonders.nl/getting_started/egi?&#installing_the_egi_java_implementation from the instruction to edit classpath into th einstruction to use javaaddpath. This was suggested to me by one of the users and in general is a better approach (as it does not require admin rights).
Roemer van der Meij - 2012-10-10 15:34:56 +0200
Just as a reminder from the bug binge, test_bug1407 fails during testing.
Eelke Spaak - 2012-11-28 16:13:19 +0100
Another bug binge reminder (28-nov-2012): test_bug1407 still fails. Stack trace: java.io.FileNotFoundException: /home/common/matlab/fieldtrip/data/test/bug1407/Long64ChannelWithEvents.mff/signal1.bin (Permission denied) at java.io.RandomAccessFile.open(Native Method) at java.io.RandomAccessFile.
Joseph Dien - 2013-09-24 22:10:39 +0200
Ah, it hadn't occurred to me there might be a bugzilla entry for the mff implementation. My apologies. Here is a series of e-mails I've been sending out on my work with the mff code. I've made some fixes to the mff_v1 code regarding it not computing event times correctly, not registering multiple segments correctly (where there is a delay between segments), and not including events only within the period of the epoch. Hopefully these changes are acceptable. When I tried to run test_bug1407, it failed: >> test_bug1407 Warning: could not determine filetype of testfiles/NS1000sps > In fileio/private/warning_once at 158 In ft_filetype at 1152 In test_bug1407 at 27 Error using test_bug1407 (line 27) Assertion failed. I don't see anyone else reporting this behavior. Maybe I'm doing something wrong? This is with fieldtrip-20130923 under Matlab 2013b on OS X 10.8.5. The first resolves as egi_mff and the second as unknown. E-mail chain follows: I also tried mff_v1 with an average file in mff format but got the following error: reading xml files to obtain header info... [Fatal Error] history.xml:56:28: Invalid byte 2 of 2-byte UTF-8 sequence. ************************************************************** No data were read. The error message was: Error using xmlread (line 98) Java exception occurred: org.xml.sax.SAXParseException; systemId: file:/Volumes/Data2/EEG-fMRI/demo%2020130911%201214.ave.mff/history.xml; lineNumber: 56; columnNumber: 28; Invalid byte 2 of 2-byte UTF-8 sequence. at org.apache.xerces.parsers.DOMParser.parse(Unknown Source) at org.apache.xerces.jaxp.DocumentBuilderImpl.parse(Unknown Source) at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:177) ************************************************************** I tried also with mff_v2 and got: ************************************************************** No data were read. The error message was: Error using mff_getObject (line 36) Java exception occurred: java.lang.NoClassDefFoundError: joptsimple/OptionParser at com.egi.services.mff.api.LocalMFFFactoryDelegate.openResourceAtURI(LocalMFFFactoryDelegate.java:451) at com.egi.services.mff.api.MFFFactory.openResourceAtURI(MFFFactory.java:192) Caused by: java.lang.ClassNotFoundException: joptsimple.OptionParser at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at com.mathworks.jmi.CustomURLClassLoader.findClass(ClassLoaderManager.java:760) at java.lang.ClassLoader.loadClass(ClassLoader.java:423) at java.lang.ClassLoader.loadClass(ClassLoader.java:356) ... 2 more ************************************************************** The average file was generated and translated into mff using 4.5.5a3 so perhaps it's an obsolete build (I'm not sure what the current build is). So anyway, I think this is as far as I can take it till Colin gets back. Phan tells me he's not in town yet so I'll just wait for now. Cheers! Joe On Sep 24, 2013, at 12:06 PM, Joseph Dien <jdien07@mac.com> wrote: Hey Colin, welcome back! Some questions and bug reports for you. To everyone else, I'm cc'ing everyone who has been on this thread but feel free to ask me to drop you off if I'm spamming you. 1) I've tracked down the error with the bcg corrected mff file to being due to the timing in the xml.epochs.epoch field being in picoseconds rather than microseconds. This is causing both mff_v1 and mff_v2 to error out. Since both the regular (microseconds) and the bcg corrected (picoseconds) versions were generated by Mark Moran using the same copy of NetStation, this would appear to be a NetStation bug rather than a Matlab bug. You'll have to ask him what version of NetStation he was running and how he generated the bcg file. 2) I'm seeing references to "blocks" in the mff file. What are they about? 3) In the sample file Mark generated for us, the signal2 file only has the ECG channel in it and yet the nsignals field for signal2 equals 2. What is the second channel? Or is that a bug? At any rate, it'd be good to know if we can just drop the second channel. Thanks! Joe On Sep 19, 2013, at 3:37 AM, Robert Oostenveld <r.oostenveld@donders.ru.nl> wrote: Hi Joe, If you have a suggested fix to the v1 code, please add it to the code base directly. fieldtrip/test/test_bug1407.m is a script that performs some tests. At this moment it fails due toa discrepancy between the v1 and v2 implementation for the events. This might be related to the issue you are addressing. I have just added some more lines to the test script to try to safeguard the v1 implementation. If you wish, you can send me your data and I can add it to the test. Otherwise, you might just use the same strategy for testing the implementation on your dataset. best regards, Robert On 19 Sep 2013, at 1:36, Joseph Dien wrote: I tried to read in an mff file using the v1 code and ran into some bugs. The symptom was that an event sample was larger than the number of data samples. In part, this seems to be because the code for handling multiple epochs is looking for an xml.epoch field when what I'm seeing in my sample file is a xml.epochs field. I can enter in this correction to the codebase if that seems appropriate. Even after making the correction, hdr.nSamples does not correspond to the contents of epochdef. I'm going to need more information about the structure of the mff file format before I can take this further. Alternatively, I could wait for the v3 release. I'd rather be able to navigate around the mff file structure myself though and verify things. Is there any documentation I can refer to? Thanks! Joe On Sep 17, 2013, at 11:07 PM, Joseph Dien <jdien07@mac.com> wrote: Hi, I'd also like to access the ECG channel in mff files from fMRI-EEG sessions. I just tried the mff_v2 routine and it doesn't include the ECG data. Is there some way to access it alongside the EEG channels? I tried the mff_v1 routine and it did come up with a 258th ECG channel, as well as a 259th "unknown" channel. Any idea what this unknown channel is? It apparently was in the signal2.bin file: Warning: found less lables in xml.pnsSet than channels in signal 2, labeling with s2_unknownN instead In ft_read_header at 868 In ft_read_event at 724 In ep_readData at 1470 Also, the mff_v1 works under Matlab2013b. Are there any known issues with this version? It looks like I'll be needing to use it until the v2 issues have been worked out. Cheers! Joe On Sep 16, 2013, at 2:27 AM, Robert Oostenveld <r.oostenveld@donders.ru.nl> wrote: Hi Joe, The problem seems to reside in the Java code, not in MATLAB, so I am not able to help you out with that aspect. EGI programming expertise is needed here. Regarding your second point: it that anything that fieldtrip/fileio can be of help for? On linux there is of course the same issue. In general I have tried treating the mff directory just like the ds-directory that CTF uses for a MEG dataset (e.g. used in the online tutorials), i.e. prevent people from having to specify files in the directory but rahter consistently specify the directory name (assuming that the expected files are in there with a well defined file naming scheme). best Robert On 14 Sep 2013, at 4:36, Joseph Dien wrote: Hi, a new issue just came up. I just tried reading a couple mff files using the FieldTrip code (EGI's mff_v2 version) with Matlab 2013b and it crashed with the following error message but it did fine with Matlab 2013a (on the same computer and everything else the same I believe). ************************************************************** No data were read. The error message was: Error using mff_getObject (line 36) Java exception occurred: java.lang.NoClassDefFoundError: joptsimple/OptionParser at com.egi.services.mff.api.LocalMFFFactoryDelegate.openResourceAtURI(LocalMFFFactoryDelegate.java:451) at com.egi.services.mff.api.MFFFactory.openResourceAtURI(MFFFactory.java:192) Caused by: java.lang.ClassNotFoundException: joptsimple.OptionParser at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at com.mathworks.jmi.CustomURLClassLoader.findClass(ClassLoaderManager.java:760) at java.lang.ClassLoader.loadClass(ClassLoader.java:423) at java.lang.ClassLoader.loadClass(ClassLoader.java:356) ... 2 more ************************************************************** I'm using: OS X 10.8.4 FieldTrip fieldtrip-20130912 Can anyone diagnose this one? Also, the problem with Windows not knowing how to handle mff package files remains an issue.
Joseph Dien - 2013-09-26 02:38:26 +0200
I've run into a question on how to represent event timing. To begin with my understanding of the convention, based on the ft_read_event header, the event.sample field is intended to be the count from the start of the session. If the data is segmented into epochs, then the field is still counted from the beginning of the recording session. If there were samples between the epochs that were discarded, then determining how the event relates to its epoch starts becoming more difficult to ascertain. The convention is to always have a "trigger" event that essentially keeps track of the recording time for the first sample of the epoch and thus for all the events within the epoch. This convention does not appear to distinguish between recording stops in unsegmented data and intertrial periods in segmented data, or at least I haven't seen any such documentation. So with respect to the present mff_v1 code, I'm a little confused as it doesn't seem to comply with my understanding of the fieldTrip convention. More concretely, should the event.sample field contain the time of the event with respect to the start of the recording session (real time), the start of the data file not counting discarded samples (file time), or the start of the segment (epoch time)? The current code uses file time in the .sample field and the information needed to compute real time is held in the orig.epochdef structure, which is not part of the formal FieldTrip data structure. This question is also relevant to my last edit of ft_read_data because it was based on the presumption that begsample and endsample are supposed to be in file time, not in real time. If they are supposed to be in real time, then I need to undo that edit. Or to put it another way, the original mff_v1 code in ft_read_data seems to presume real time while the mff_v1 code in ft_read_event seems to presume file time, so I need to know which timing convention to use so I can make them coherent with each other (unless data are supposed to be specified in real time and events are supposed to be specified in file time and thus nothing needed changing?). Thanks for any help you can give me! Joe
Robert Oostenveld - 2013-09-26 09:55:40 +0200
(In reply to comment #34) Hi Joe, To address your question specifically: > More concretely, should the event.sample field contain the time of the event with > respect to the start of the recording session (real time), the start of the > data file not counting discarded samples (file time), or the start of the >segment (epoch time)? it should be "file time". Let me now elaborate, in the hope that it clarifies the general design of the fieldtrip fileio code. event.sample relates to the sample index into the file, disregarding any internal structure in the file. Different data formats that are supported by fieldtrip allow for 1) fully continuous data 2) continuous recordings that are stored in blocks, with no gaps in between (e.g. EDF) 3) epoched recordings, i.e. a fixed block representation with known or unknown gaps in between 4) continuous recordings with an occasional break (e.g. a pause) 1 and 2 are continuous or pseudo-continuous respectively 2 and 3 can most of the times not be distinguished based on the file content, but can sometimes be dinstinguished by external information (e.g. EDF is meant to represent continuous data, but for CTF it can be either continuous or have gaps in between). 4 has segments ("trials" in fieldtrip-speak) of unequal length. To support all of these with ft_read_header, ft_read_data and ft_read_event, some conventions have been adopted. Say we do hdr = ft_read_header(filename) event = ft_read_event(filename) Then in all cases hdr.nSamples*ndr.nTrials reflects the total number of samples present in the datafile. Each sample in the datafile is indexed, starting from 1, up to hdr.nSamples*ndr.nTrials. In case 1, hdr.nTrials = 1 In case 2 and 3, hdr.nTrials is usually the number of segments that can be represented in the file and hdr.nSamples is the number of samples in a segment (fixed for all segments). In case 3, hdr.nSamplesPre is sometimes known from the file and is returned as non-zero (by default it is zero) in case 4, hdr.nSamples does not apply as a single number to the variable length segments. To deal with the more detailed structure in the file, the output of ft_read_event is needed. In case 1, there are events for each trigger. In case 2, there are events for each trigger, but there is also a "trial" event for each segment in the file In case 3, there are events for each trigger, but there is also a "trial" event for each segment in the file. In case 4, it is the same as case 2, but sometimes it has another name (e.g. BrainVision calls them "New Segment" markers). So if in case 3 you want to know where a trigger is relative to its corresponding trial, you have to look in the events at both the trigger AND at the trial. As example, lets look at the Subject01.ds CTF MEG dataset that is used throughout many tutorials. Note that it is sampled according to case 3 at 300 Hz. >> event = ft_read_event('Subject01.ds') event = 1343x1 struct array with fields: type sample value offset duration There are 1343 events, but not 1343 trials, nor triggers >> event(1) ans = type: 'trial' sample: 1 value: [] offset: -300 duration: 900 There is a trial, starting at sample 1 in the datafile and ending at sample 900, i.e. a 3-second trial. >> event(2) ans = type: 'backpanel trigger' sample: 301 value: 1 offset: [] duration: [] There is a "sort of" trigger at sample 301. It happens to lie in the first trial. See also the next one: >> event(3) ans = type: 'STIM' sample: 301 value: 65536 offset: [] duration: [] There is another "sort of" trigger at sample 301. It happens to lie in the first trial. The particular system has two 16 bit TTL input ports, one on the front and one on the back of the cabinet. Together they form a 32-bit STIM channel. Event(2) represent the upper 16 bits of the 32-bit word. People expect from their presentation computer (connected with a cable to the back) that they sent trigger value 1 rather than 65536, hence we represent both. Note that this is CTF specific, but other systems have similar peculiarities (like Biosemi using status bits for the battery charge, etc). >> event(4) ... Let me skip this one. It comes from the CTF markerfile and is irrelevant and confusing for now. >> event(5) ans = type: 'trial' sample: 901 value: [] offset: -300 duration: 900 Here is another trial. It starts at sample 901 and ends at 1800. >> event(6) ans = type: 'backpanel trigger' sample: 1201 value: 3 offset: [] duration: [] Again a trigger, now at sample 1201, which happens to be in the 2nd trial. >> event(7) ans = type: 'STIM' sample: 1201 value: 196608 offset: [] duration: [] The same trigger, represented as 32 bits. What you see is that all timing is represented relative to the onset of the file, not to the onset of the recording, as there are unknown gaps in this file. If you want to know which trigger (or triggers) happen in which trial, you have to make some combinatorial MATLAB code (i.e. the trialfun). In the trialfun you write the code that parses the sequence of events, combining them where needed. E.g. you might have trials, each with a stimulus and a response event, but in some trials there is no response trigger because the subject was too late. The flexibility of interpreting the datastream as represented on disk in terms of the experimental manipulations during the experiment and as represented (sometimes partially) in the trigger sequence requires expert (=end-user) knowledge. FieldTrip by itself cannot make sense out of all possible "event" sequences. Hence it represents all event information and leaves it to the end-user to decipher it. This is indeed not simple, and I don't know of any software that makes it really simple. Some commercial packages have some sort of macro-language to do it, fieldtrip just uses MATLAB itself. The most simple cases are supported by fieldtrip out of the box. Once the user has deciphered all events and decided on the segments of data that are of interest for the analysis, we end up calling ft_read_data to read those segments of interest from the file. That function does not care about the internal timing structure in the file. It only wants to have a unique indexing to determine which part to read: and that is where we again use samples. Those samples relate to the file on disk, not the time in the experiment. Ideally you would use the time of the experiment. But we only have the time in the file. Some files have an explicit representation of time and allow you to determine how large the gaps are (in case 3), or allow you to distinguish with case 2 (having gaps of zero length). But most manufacturer file formats don't have an explicit representation of time, only the explicit representation of samples on disk. Hence that is what we have to use time in the file. I like to distinguish wall-clock-time (as measured on a clock in seconds) and file time (measured in seconds relative to the start of the file on disk). If I record for 5 minutes, pause for 4 minutes and continue for 3 minutes, I get a file (dataset) with 8 minutes in total of file time. The ft_read_xxx functions don't use seconds but (integer) samples to express file time. The first sample is 1 (not 0), the last sample in the file is 8*60*fsample.
Joseph Dien - 2013-09-26 21:48:54 +0200
Wow, thanks for taking the time to provide this really clear explanation! Just to make sure I'm not misunderstanding (since I'm not sure we're defining "file time" in the same way), let me confirm. To make this concrete, if there is a dataset where the sampling rate is 250Hz, data is collected for one second (250 samples), there is a pause of half a second (125 samples) between trials and then a further one second recording (250 samples), an event at the start of the second epoch could be indexed in the following ways: real time (sample 376, with respect to the start of the experiment) file time (sample 251, with respect to the samples for which there is data in the file) epoch time (sample 1, with respect to the "trial" event denoting the start of the epoch) So in this case, we should be using option #2? In that case, the present code is what it needs to be (whew!). If this is the intent for FieldTrip I/O in general, perhaps I should modify the documentation in the ft_read_event header to clarify this point? Thanks again! Joe
Joseph Dien - 2013-10-09 21:28:35 +0200
Just added support for segmented data to the egi_mff_v1 (when the epochs are all the same size, will assume that the data is segmented and the data will be output in 3D format rather than issuing error message. I'd also like to reiterate that I'm uncomfortable with the egi_mff_v1 event timing being in file time whereas the ft_read_event header indicates that event timing should be in real time. It's going to be unnecessarily confusing if some of the fileio code is written one way and the rest of it is written the other way. If we could have an official ruling which way to go, it would be really helpful. :)
Joseph Dien - 2013-10-09 22:18:54 +0200
oh hold on, thinking about if further, I see that the "trial" scheme allows for both "real time" and "file time". So the take home message is I need to add "trial" events to the egi_mff_v1 code to finish adding support for segmented data.
Robert Oostenveld - 2014-02-23 13:44:51 +0100
enhancement - allow reading over epoch boundaties for mff_v1, to make it consistent with v2. Return hdr.orig.epochdef also in case of a single epoch, add the epochs to the event in ft_read_event. Sending fileio/ft_read_data.m Sending fileio/ft_read_event.m Sending fileio/ft_read_header.m Sending test/test_bug1407.m Transmitting file data .... Committed revision 9222. I also removed the check on the exact equality of the events in v1 and v2, the count is identical but the type and offset are not. However, it would be good to check that the samples are identical...
Robert Oostenveld - 2014-02-23 13:55:09 +0100
(In reply to Robert Oostenveld from comment #39) in the test script I added a test on event.sample between mff_v1 and v2. This presently fails, indicating a problem with the code. Sending test/test_bug1407.m Transmitting file data . Committed revision 9223. A discrepancy is observed for the first dataset (Long64ChannelWithEvents.mff, with 86 events, out of which 38 are one sample later in the v1 implementation than in the v2 implementation. This suggests a rounding off error in either one of th eimplementations. This is for example one that differs: K>> v1_evt{1}(9) ans = type: 'ECI_TCPIP_55513' sample: 341 offset: 0 duration: 2.5000e-04 value: 'stm-' orig: [1x1 struct] K>> v2_evt{1}(9) ans = type: 'stm-' sample: 340 value: [] offset: [] duration: 1 timestamp: [] This needs to be looked into by one of the EGI experts.
Robert Oostenveld - 2014-02-25 22:32:06 +0100
On 25 Feb 2014, at 22:23, Colin Davey wrote: In any case, regarding the off-by-1 difference between v1 and v2, I believe v2 is correct. The way Net Station assigns events to samples when an event may fall between the samples, is as follows: An event is assigned to a sample if the event time is >= the sample time, and < the next-sample time. Is that the logic that v1 is attempting? If v1 is associating a sample with the time period spanning half a sample before to half a sample after, that could explain the difference.
Robert Oostenveld - 2014-02-25 22:43:09 +0100
(In reply to Robert Oostenveld from comment #41) The observed effect makes sense if one is doing floor(sample) and the other round(sample), where sample is the fractional sample in between two integer samples. There is this line of code in ft_read_event on line 804: %in samples, relative to start of recording eventOffset = round((eventSDV - begSDV)*24*60*60*hdr.Fs); from which subsequently the event.sample gets computed. Right now I don't have access to the test data, but when I have, I will change this to floor() and try again.
Colin Davey - 2014-03-19 23:09:05 +0100
I propose closing this bug report, as it contains a variety of issues, most if not all have been resolved, and which pertain to an old version of the software. Any objections?
Colin Davey - 2014-03-19 23:12:51 +0100
As I'm learning this bug reporting system, I'm wondering, who is authorized to close this bug, whether by the bug reporting software itself, or the FT policies.
Joseph Dien - 2014-07-24 06:01:07 +0200
Right now the mff_v1 code gives the reference site the label REF to be consistent with the montage (according to the notes in this thread). However, the output of ft_senslabel expects EGI reference sites to have the label Cz, so it's not being recognized as an EEG site. Should we change the mff_v1 code to Cz to be consistent with ft_senslabel or should we change ft_senslabel to be consistent with the montage?
Joseph Dien - 2014-07-24 06:46:14 +0200
I should also note that, at least in the header of the mff file that I am currently using for testing, this channel is labeled "VREF" in one field and "vertex reference" in another. Given that FieldTrip appears to be set up for consistent labeling, my suggestion is to adopt a convention for the vertex reference electrode on the FieldTrip side. Since the EGI data is usually rereferenced to either mean mastoid or average reference, at which point it is functionally simply an electrode like any other recording electrode, my suggestion is to simply label it E257 (for 256-channel data) to be consistent with the rest of the labels (which are labeled E1-E256). I could add code to ft_readHeader that would look for these different reference electrode names and change them to this standard label (and do so for each of the EGI file formats). I would then also change ft_senslabel accordingly. Please let me know if this proposal is acceptable to everyone.
Joseph Dien - 2014-07-25 02:57:09 +0200
Please note that this issue of reference channel labeling is pertinent to all EGI datasets, so it needs to be addressed even if we retire the mff_v1 code.
Robert Oostenveld - 2015-01-26 12:28:02 +0100
(In reply to Joseph Dien from comment #47) I looked at the reference electrode naming confusion. I don't think fieldtrip should be imposing a standard here, it just has to work with whatever is provided by the users. I removed the Cz label from ft_senslabel. The electrode collections in there are only used in heuristic approaches to determine which type of data acquisition system and which type of sensor array it is. The most important use is in ft_senstype, where it checks for an 80% match or more between data labels and the predefined lists in ft_senslabel. Adding multiple possible reference channels (Cz, REF, VREF, etc) will reduce that overlap, hence I think it is better to exclude the reference channel from the list of known channels in the EGI systems. I also looked at the current support, and realise that we are still not at a compatibility level where the code v2 can replace v1. I don't know the fraction of the files with the different sub-version numbers of the mfg dataset, but I mainly have test data on which the v2 reader fails. It is not clear to me how windows or linux users (or mac users without the EGI software, like me) would have to convert the files to a later sub-version. Given lack of further progress on this, I suggest to change the resolution to WONTFIX. Some stuff works, some does not. Users will have to study the code and documentation (including this publicly available bug report) if they run into problems with their data.
Robert Oostenveld - 2015-01-26 12:40:26 +0100
I removed the reference electrode from the known channels (as it is not consistently named). I updated the test script so that it does not result in an error, but a warning. I fixed a small TODO item in the ft_read_xxx functions for adding the full path to the filename on windows machines. ----- The status of the test script is now: Warning: the egi_mff_v2 implementation fails for /home/common/matlab/fieldtrip/data/test/bug1407/Long64ChannelWithEvents.mff > In test_bug1407 at 76 Warning: the egi_mff_v2 implementation fails for /home/common/matlab/fieldtrip/data/test/bug1407/NS1000sps.mff > In test_bug1407 at 76 Warning: the egi_mff_v2 implementation fails for /home/common/matlab/fieldtrip/data/test/bug1407/NS500Sine6Hz.mff > In test_bug1407 at 76 not comparing v1 and v2 for dataset 1 not comparing v1 and v2 for dataset 2 not comparing v1 and v2 for dataset 3 ------ roboos@mentat001> svn commit Sending fileio/ft_chantype.m Sending fileio/ft_read_data.m Sending fileio/ft_read_event.m Sending fileio/ft_read_header.m Sending forward/ft_senslabel.m Sending test/test_bug1407.m Transmitting file data ...... Committed revision 10123.