Back to the main page.
Bug 3239 - ft_read_event cannot import from .nex files where any variables have 0 timestamps
| Status | ASSIGNED |
| Reported | 2017-01-31 18:14:00 +0100 |
| Modified | 2017-02-09 19:17:16 +0100 |
| Product: | FieldTrip |
| Component: | fileio |
| Version: | unspecified |
| Hardware: | Macintosh |
| Operating System: | Mac OS |
| Importance: | P5 normal |
| Assigned to: | Teresa Madsen |
| URL: | https://www.dropbox.com/s/acfa828vvl109yy/18141_FR1_S2_SPK_sorted.nex?dl=0 |
| Tags: | |
| Depends on: | |
| Blocks: | |
| See also: |
Suzanne - 2017-01-31 18:14:59 +0100
ft_read_event returns following warning when using a .nex file consisting of units and events (no continous - LFP - channels). Example file is added. Warning: No continuous variables found - using hdr.filheader.frequency > In read_nex_event (line 47) In ft_read_event (line 1946) Error using struct Array dimensions of input '4' must match those of input '2', or be scalar. Error in read_nex_event (line 145) tmp = struct('sample',num2cell(evt.tim), 'value',cell(Nevent,1), ... Error in ft_read_event (line 1946) event = read_nex_event(filename);
Teresa Madsen - 2017-01-31 19:09:53 +0100
This is due to
Robert Oostenveld - 2017-01-31 19:45:38 +0100
this is a "feature". Events are expressed in samples relative to the (constant sampling rate) continuous channels. The sample count is consistent with the sampling frequency returned by ft_read_header. I realize that this is suboptimal for processing spike-only data. I am welcoming suggestions (that should include not only code, but also documentation) to improve this.
Teresa Madsen - 2017-01-31 20:01:07 +0100
Oops! Sorry for the useless comment. I started to explain my hypothesis, but then decided to just assign myself the bug and test it before exploring further. Turns out I can't replicate your bug with just any old .nex file without continuous variables, as this ran through successfully: >> [event] = ft_read_event('S:\Teresa\PlexonData Backup\200\200_091006_0001_final_spksONLY.nex') Warning: No continuous variables found - using hdr.filheader.frequency > In read_nex_event (line 47) In ft_read_event (line 1946) event = 1917×1 struct array with fields: sample value timestamp type duration offset You said "example file is added" but I don't see where. Please run this line before trying again: >> dbstop if error Matlab will drop you into the function where the error occurs, giving you access to the variables involved. Tell me about all the variables used as inputs to the function struct in read_nex_event (line 145). For example, here's what shows up if I type: K>> evt evt = struct with fields: tim: [1755×1 double] K>> Nevent Nevent = 1755 K>> whos timestamp Name Size Bytes Class Attributes timestamp 1755x1 14040 double K>> hdr.varheader(evtvarnum(ev)) ans = struct with fields: typ: 1 version: 100 nam: 'Event003 ' offset: 12964288 cnt: 1755 wirenumber: 0 unitnumber: 0 gain: 0 filter: 0 xpos: 0 ypos: 0 wfrequency: 0 adtomv: 0 numsmp: 0 nummrk: 0 mrklen: 0 It sounds like your size(evt.tim) ~= [Nevent, 1], and I'm trying to figure out why. Please send me your smallest file where this happens via a file transfer service (http://www.pcmag.com/article/343582/how-to-send-large-files).
Teresa Madsen - 2017-01-31 20:06:55 +0100
(In reply to Robert Oostenveld from comment #2) Robert, it's not just the warning - she's getting an actual error that prevents her from importing events. She emailed this to me (since I helped her with the previous step) before I suggested she post the bug here: "My next step would be to extract also the timestamps of the events recorded in my recordings, which I formatted like this: >> [event18141] = ft_read_event('/Users/suzannevdveldt/Documents/export/18141_FR1_S1_SPK_sorted.nex’) However, the fr_read_event does not seem to recognize the events in my file, which I do not fully understand, as when I check my plexon recs with NeuroExplorer the events seem to be there."
Suzanne - 2017-01-31 20:24:08 +0100
(In reply to Teresa Madsen from comment #3) >> dbstop if error Drops me in line 145, checking the following inputs shows that all seem to be empty. K>> evt evt = tim: [] K>> Nevent Nevent = 0 K>> whos timestamp Name Size Bytes Class Attributes timestamp 0x0 0 double K>> hdr.varheader(evtvarnum(ev)) ans = typ: 1 version: 100 nam: 'KBD1 ' offset: 11368 cnt: 0 wirenumber: 0 unitnumber: 0 gain: 0 filter: 0 xpos: 0 ypos: 0 wfrequency: 0 adtomv: 0 numsmp: 0 nummrk: 0 mrklen: 0 I will attach the smallest file. Thanks for your help!
Teresa Madsen - 2017-01-31 20:40:32 +0100
(In reply to Suzanne from comment #0) It might also be helpful to make sure your NEX files are properly formatted by testing them with NeuroExplorer's own Matlab import functions: http://www.neuroexplorer.com/downloads/HowToReadAndWriteNexAndNex5FilesInMatlab.zip After unzipping the file somewhere on your Matlab path, run and show me the output of: [nexFile] = readNexFile('/Users/suzannevdveldt/Documents/export/18141_FR1_S1_SPK_sorted.nex') nexFile.events{:}
Suzanne - 2017-01-31 21:04:45 +0100
(In reply to Teresa Madsen from comment #6) I came across this too, and this works fine for all these .nex files, including the read out of the events. [s18141_FR1_S2] = readNexFile('/Users/suzannevdveldt/Documents/export/18141_FR1_S2_SPK_sorted. s18141_FR1_S2 = version: 104 comment: [1x0 char] freq: 40000 tbeg: 0 tend: 3.6113e+03 events: {40x1 cell} markers: {5x1 cell} neurons: {7x1 cell}
Robert Oostenveld - 2017-01-31 21:11:51 +0100
(In reply to Suzanne from comment #7) The original design of ft_read_header, data and event is that of continuously sampled channels with an equal sampling rate, all starting at the same moment in time and no holes in the recording. Implicitly this assumes that sample indices (starting at 1) can be used as timestamps. Not all electophys formats are compatible with this, and most have an explicit (high frequency) timestamps. Furthermore, spikes and waveform snippets don't fit this design (hence ft_read_spike). Work-arounds have accumulated over the years. With FT we try to keep the interface consistent over all data formats (otherwise it becomes a nightmare to support). Most important are the timestamps, if you can get those, you might get away with setting the sample numbers (event.sample) to nan.
Teresa Madsen - 2017-01-31 21:19:46 +0100
(In reply to Suzanne from comment #5) Yeah, your NEX file doesn't have timestamps in most events. I added checks for that, so now it will warn you when it's skipping a variable, but then move on without errors. https://github.com/fieldtrip/fieldtrip/pull/314
Teresa Madsen - 2017-01-31 21:25:10 +0100
(In reply to Robert Oostenveld from comment #8) Robert, I already added a simple fix for that problem when I made the last round of changes to read_nex_event. If there are no continuous variables, it gives that warning and then moves on just fine, using the sampling rate in the file header instead of the continuous variable header. That change was documented: % If there are no continuous variables in the file, the system sampling % frequency is used throughout, so % event.sample = timestamp; The warning at the beginning of Suzanne's error message was confusing, but the error was actually unrelated. I edited the title of the bug to reflect that.
Suzanne - 2017-01-31 21:28:19 +0100
(In reply to Teresa Madsen from comment #9) Great, it works perfectly. Thank you Teresa!
Robert Oostenveld - 2017-02-01 09:49:18 +0100
I have downloaded the linked file and made a local copy on our network drive. For reference: if you would want to use it in a matlab test script that we can (automatically) run here, please use dccnpath('/home/common/matlab/fieldtrip/data/test/bug3239/18141_FR1_S2_SPK_sorted.nex') That should locate the file on our network share, but should also locate the same file on your disk (assuming it is in the present working directory).
Robert Oostenveld - 2017-02-01 09:54:34 +0100
This is what we have as new files in our test (and tutorial) directories roboos@mentat001> pwd /home/common/matlab/fieldtrip/data roboos@mentat001> find . -name \*.nex ./ftp/tutorial/spike/p029_sort_final_01.nex ./ftp/tutorial/spikefield/p029_sort_final_01.nex ./test/original/lfp/plexon/p213parall.nex ./test/bug3239/18141_FR1_S2_SPK_sorted.nex where one file is present twice (since used in two tutorials). I will add a test script like this filename = { ... } for i=1:numel(filename) hdr = ft_read_header(filename{i}); dat = ft_read_data(filename{i}); evt = ft_read_event(filename{i}); spike = ft_read_spike(filename{i}); % I guess this should never error end
Robert Oostenveld - 2017-02-08 22:54:57 +0100
(In reply to Robert Oostenveld from comment #13) I merged https://github.com/fieldtrip/fieldtrip/pull/314 Before closing, I still want to implement a generic nex reader test script as mentioned in the previous commit.