Back to the main page.
Bug 1795 - integrate new plexon readers into fieldtrip/fileio
| Status | ASSIGNED |
| Reported | 2012-10-28 09:54:00 +0100 |
| Modified | 2016-09-13 15:11:07 +0200 |
| Product: | FieldTrip |
| Component: | fileio |
| Version: | unspecified |
| Hardware: | PC |
| Operating System: | All |
| Importance: | P3 normal |
| Assigned to: | Thomas Hartmann |
| URL: | |
| Tags: | |
| Depends on: | |
| Blocks: | |
| See also: |
Robert Oostenveld - 2012-10-28 09:54:16 +0100
Thomas mentioned that he has revamped the support for reading data from plexon files. I know that there are still some issues with the existing implementation, therefore I think it would be good to consider merging Thomas' new implementation in fieldtrip, or to replace the existing implementation all together. To do 1) make an inventory of the old and existing tickets on bugzilla that pertain to this 2) make an inventory of the existing test data sets 3) make an inventory of the existing test scripts ...
Robert Oostenveld - 2012-10-28 09:57:52 +0100
regarding item 1 (inventorize existing bugzilla entries): I found bug 258, bug 322, bug 323, bug 909, bug 1194, bug 1300 that pertain to the plexon format.
Robert Oostenveld - 2012-10-28 10:01:30 +0100
regarding item 2 (inventorize available test data): ls -al /home/common/matlab/fieldtrip/data/test/original/lfp/plexon -rw-r--r-- 1 jansch fieldtrip 126125140 Apr 11 2011 p213parall.nex mac001> pwd /Volumes/Data/roboos/data/dataformat/testdata/plexon mac001> find . -type f ./fpieper-wideband-data.ddt ./p021parall.nex ./p213parall.nex ./PlexonNeuroshareSampleFile/NSSample.plx ./PlexonNeuroshareSampleFile/Readme.txt ./test.nex ./test1.ddt ./test1.plx ./thiwom/p075par.plx ./thiwom/p075st1.plx ./thiwom/p075st2.plx ./thiwom/p075tun.plx ./thiwom/plx_0500_2units.nex ./thiwom/plx_0500_2units.plx ./thiwom/plx_1000_2units.nex ./thiwom/plx_1000_2units.plx ./thiwom/plx_3000_2units.nex ./thiwom/plx_3000_2units.plx ./thiwom/r190parTWO.plx ./thiwom/readme.txt so there are nex, plx and ddt files that are associated with plexon.
Robert Oostenveld - 2012-10-28 10:15:19 +0100
The existing test scripts I could find that do something with plexon data are test_tutorial_spike test_tutorial_spikefield which both use ftp/tutorial/spike/p029_sort_final_01.nex Furthermore, there is ref_datasets which points to p213parall.nex. This is subsequently used in mac001> grep -l ref_datasets test_*.m test_bug1306.m test_bug1599.m test_ft_channelrepair.m test_ft_componentanalysis.m test_ft_freqanalysis.m test_ft_megplanar.m test_ft_megrealign.m test_ft_multiplotTFR.m test_ft_prepare_neighbours.m test_ft_preprocessing.m test_ft_read_header.m test_ft_read_sens.m test_ft_sourceanalysis.m test_ft_timelockanalysis.m test_ft_timelockanalysis_new.m test_historical.m In short: there is very limited testing of the explicit characteristics of the plexon file formats. I think it makes sense to write a new test script that explicitly makes use of a variety of files and ensures that the results from all ft_read_xxx functions are correct.
Thomas Hartmann - 2012-10-29 15:20:49 +0100
Created attachment 356 Patch to include new plx reading functions. Diff from svn version 121029
Thomas Hartmann - 2012-10-29 15:26:13 +0100
alright... took today to wrap up my changes to fieldtrip to have it read plx files and make a patch out of it. if you only merge the patch, you will notice no difference, as the code checks whether the Plexon Offline SDK is installed. this can be downloaded from here: http://www.plexon.com/assets/downloads/sdk/ReadingPLXandDDTfilesinMatlab-mexw.zip after unpacking to some directory, go to the mexPlex folder and issue the build_and_verify_mexPlex command. this builds the mex files needed for operation. then add the root folder of the Plexon SDK to your matlab path. i hope you like it and it works. if there are any issue, i am glad to solve them! thomas
Robert Oostenveld - 2012-10-29 15:43:02 +0100
(In reply to comment #5) As I understand it now, the mex files can be compiled for any platform?
Thomas Hartmann - 2012-10-29 15:49:32 +0100
(In reply to comment #6) dont know about mac, but i can confirm that it runs on 64bit linux. havent checked for windows yet...
Robert Oostenveld - 2012-10-29 17:17:53 +0100
(In reply to comment #6) Let me answer myself. I downloaded it and had a look: there is cpp code and there seem to be no restrictions on the re-distribution. I suggest to make a stripped down version available in fieldtrip/external/plexon with only the mex files, the PDF and other READMEs and the required plx_ matlab wrappers. Furthermore, I suggest to implement the two versions in a similar fashion as egi_mff or the different CTF readers. So switch fileformat case {'plexon_plx_v1' 'plexon_plx'} do the old stuff case {'plexon_plx_v2'} do the new stuff and then when everything is confirmed to work as it should, the 'plexon_plx' case (which is the default in case the user does not specify anything, since that is what ft_filetype reports) can be moved to the v2 section. This gives a longer time window for development, keeps the code separate, and facilitates future maintenance and expansions. Agreed? If so, then you would have to make some small reorganizations again to your code... I have already added your plx_orig_header to fileio/private. The only changes I made were some comments in the help, and I replaced the ~ output variable with a dummy. Old matlab versions (which we still want to support) don't understand the ~.
Thomas Hartmann - 2012-10-30 11:03:48 +0100
(In reply to comment #8) yeah, this sounds like a good strategy, though i have some questions: 1. would the ft_hastoolbox function have to be patched or how would be integrate the plexon toolbox into the system? 2. how would i use the alternative (i.e. my) plx reading functions from high-level functions such as ft_preprocessing if i would restructure the code as you suggest?
Thomas Hartmann - 2012-10-30 11:05:29 +0100
(In reply to comment #9) oh, i just saw that you alread patched ft_hastoolbox...
Robert Oostenveld - 2012-11-08 10:43:26 +0100
Hi Thomas, What is the status of this? I would expect some suggestions from you w.r.t. an insertion of case 'plexon_plx_v2' into the ft_read_data/header/event function and then get the associated low-level functions.
Thomas Hartmann - 2012-11-08 11:05:47 +0100
(In reply to comment #11) well, i am just working on another urgent matter. depending whether i can solve that today i can restructure the code tomorrow or on monday. i still need to know how to use the alternative reading functions from high-level functions in order to test the code before i submit it...
Robert Oostenveld - 2012-11-08 11:14:34 +0100
(In reply to comment #12) > i still need to know how to use the alternative reading functions from > high-level functions in order to test the code before i submit it... ft_preprocessing and the likes, that read data from disk have cfg.headerformat and cfg.dataformat, which you woudl specify as plexon__plx_v2 to end up at the right location in the switch-ladder.
Thomas Hartmann - 2012-11-30 11:43:02 +0100
Created attachment 383 new patches for current nightly fieldtrip to add plexon plx reading functions as 'plexon_plx_v2'.
Thomas Hartmann - 2012-11-30 11:45:06 +0100
Created attachment 384 stripped down version of the plexon offline SDK for fieldtrip
Thomas Hartmann - 2012-11-30 11:46:05 +0100
sorry that it took me so long... as suggested, i have restructured the code. for my personal test-cases, it works nicely....
Ian Andolina - 2014-01-10 14:24:57 +0100
Hi, I was just about to submit a bug then saw this. First off I can confirm that the current latest Plexon SDK mex files compiles on OS X as well. Also the current SDK has recently added read support for their latest PL2 format, which offers an order of magnitude faster reading speeds than their previous format. The PL2 support uses matlab .P files so works cross-platform without needing to compile mex files etc. I was going to write my own new file importer for PLX/PL2 files but suspect you've done most of the hard work, so I'd be happy to beta test if I can help (I am completely naive to fieldtrip however)...
Ian Andolina - 2014-01-10 14:39:25 +0100
I should also add that the PL2 support is "transparent" in that they've made the existing .m file wrappers call either their PLX or PL2 readers and make sure the same info gets returned (apart from a few bugs here and there). As long as you use their .m file wrappers to get data it should just "work" (I certainly didn't have to rewrite any of my code to support the PL2 files)...
Robert Oostenveld - 2014-01-13 10:32:21 +0100
(In reply to Ian Andolina from comment #18) Hi Ian, Thanks for chiming in. That is a good reason to take this further, as Thomas and I have not been actively working on this. What I'd like to see is a regression test, i.e. an explicit check that "stuff" gets better and does not get worse. We have many of these tests at http://code.google.com/p/fieldtrip/source/browse/#svn%2Ftrunk%2Ftest and find them extremely valuable, especially once the core Nijmegen team does not know the details as well as some (less committed to the project) external expert. In this case there are various options to be tested. I have not looked into the details recently, but in general for a certain file we have reader version 1 and version 2, and we would like to ascertain that v2 is not inconsistent with v1, except for cases where we know that v2 is better than v1 (e.g. where v1 is buggy or incomplete). This requires (again in general) 1) test data 2) a mechanism to switch between the various versions 3) a script that loops over the test data with the various versions @Thomas: does this sound like a plan?
Ian Andolina - 2014-01-15 15:30:34 +0100
I'm happy to provide some test data .plx and .pl2 files. I don't really understand what kind of testing framework you use so am probably little use there...
Ian Andolina - 2014-01-29 11:18:24 +0100
I have a fork with Thom's changes as a branch pullable to the current master here: https://github.com/iandol/fieldtrip/compare/plexonv2?expand=1