Back to the main page.
Bug 2029 - Speed problems after the latest read_neuralynx_ncs update
| Status | CLOSED FIXED |
| Reported | 2013-03-05 13:06:00 +0100 |
| Modified | 2019-08-10 12:03:26 +0200 |
| Product: | FieldTrip |
| Component: | fileio |
| Version: | unspecified |
| Hardware: | All |
| Operating System: | Linux |
| Importance: | P3 normal |
| Assigned to: | |
| URL: | |
| Tags: | |
| Depends on: | |
| Blocks: | |
| See also: |
George Dimitriadis - 2013-03-05 13:06:19 +0100
After the read_neuralynx_ncs was updated in the morning of 04/03/2013 the speed at which I am able to access my neuralynx data dropped by about a factor of 10. Nothing gives an error, it just takes about 10 times more to do anything that requires reading from neuralynx files.
Roemer van der Meij - 2013-03-05 13:28:17 +0100
(related to bug 1998)
Martin Vinck - 2013-03-05 14:39:24 +0100
taken care of this by reading in much less blocks now, should be fixed. see bug 1998
Robert Oostenveld - 2013-03-05 14:59:47 +0100
George, can you please report on whether the speed is now acceptable again?
Martin Vinck - 2013-03-06 07:17:52 +0100
I actually don't get why it was a factor 10. We read out only the timestamps per block once moving the pointer by 512 every time. So relative to reading the data that should only give a small increase in computational load but not a factor 10. What do you think RObert?
George Dimitriadis - 2013-03-06 08:10:55 +0100
Hi guys, I just started testing my code with the latest update and it looks as if it is behaving ok (pretty much like before). Martin, regarding the slow down. I do preprocessing on two data sets each having 72 channels captured at around 8KHz and I cut them at 400 trials each dataset where each trial is 0.7secs long. That process took with the original code (and it seems with the newest one) around half an hour (roughly, I never actually timed it). With the delay that same process would take more than 5 hours. Even reading the buffer in when it was doing the trialfunction was extremely slow (minute or two instead of just a few seconds).
George Dimitriadis - 2013-03-06 08:12:11 +0100
In the last sentence of the previous message I meant header not buffer
Robert Oostenveld - 2013-03-06 09:14:33 +0100
> We read out only the timestamps per block once moving the pointer by 512 every > time. Reading a few bytes every 512 samples in the file (1044 bytes) means that with a modern filesystem on a large hard disk that every (file system) logical block has to be touched. E.g. see http://support.microsoft.com/kb/140365 for windows. So the head of the hard disk has to pass along every block to get a few bytes out. Fseeking is fast if you skip a significant portion of the file (i.e. the head does not have to move to each block on the hard disk platter), but skipping 1044 bytes is not significant. George's dataset is 72*8000*4=2304000 bytes per second, or 7.9 GB per hour. Reading every 1044th byte from such a bulk of data takes time. So his observation does not surprise me.
Martin Vinck - 2013-03-06 09:16:26 +0100
I think the key thing is that read_neuralynx_ncs is called for every entry of the cfg.trl Because if it would be called only once (it's doing the same thing every time) it would be a lot faster and the load would never change by more than a factor 2 (but probably less).