Back to the main page.
Bug 2999 - implement support for NIRS ISS Imagent data format
| Status | ASSIGNED |
| Reported | 2015-11-05 13:58:00 +0100 |
| Modified | 2017-01-31 11:09:53 +0100 |
| Product: | FieldTrip |
| Component: | fileio |
| Version: | unspecified |
| Hardware: | PC |
| Operating System: | Windows |
| Importance: | P5 normal |
| Assigned to: | Jörn M. Horschig |
| URL: | |
| Tags: | |
| Depends on: | |
| Blocks: | |
| See also: |
Matteo Caffini - 2015-11-05 13:58:58 +0100
General info about NIRS ISS Imagent export data format: 1) Files are .txt with a header and some structure 2) You can export intensity data and/or HbO-HbR concentrations data 3) some sort of channel info (pairs, not geometry) is found in the header
Jörn M. Horschig - 2015-11-05 14:41:38 +0100
Buongiorno Matteo, thanks for getting started ;) could you upload an example .txt file? Just a few secs of data would be fine, but preferably multichannel data, so that we can better infer the structure. Might be a bit premature, but please also check the code guidelines: http://www.fieldtriptoolbox.org/development/guidelines Groetjes, Jörn
Matteo Caffini - 2015-11-25 10:24:39 +0100
Created attachment 755 ISS Imagent ACDC ISS Imagent output file. It contains a header and AC-DC-Phase data for each channel.
Matteo Caffini - 2015-11-25 10:25:11 +0100
Created attachment 756 ISS Imagent conc ISS Imagent output file. It contains conc data for each channel.
Matteo Caffini - 2015-11-25 10:37:22 +0100
Hi, sorry for the delay, now I'm ready to dive into this. :-) I uploaded two files of my ISS Imagent output (one in ACDC form and one transformed into concentrations). 1) Text structure (tab separated columns) of the ASCII files may vary, depending on the channel configuration in use, but the overall structure is maintained. 2) There are plenty of options in the ISS software to configure the output files. We should play with those to make sure we can read as many cases as we can. Ciao! Teo
Jörn M. Horschig - 2015-11-25 11:20:33 +0100
Hi Teo, I guess you already found all the files I wrote for our devices (external/artinis). Maybe you can start with reading the read_bucn_* files in fileio/private. They were written once ago to support NIRS data for the UCL NIRS system. These files are fairly rudimentary, and you would need some more robust content/error handling. But they give you somewhat of an overview how you can tackle reading in of the files. As the read_bucn_ files are low level function, they actually do not obey to all the FT code guidelines (see http://www.fieldtriptoolbox.org/development/guidelines/code, but you do not need to worry too much about these, yet). Let me know if I can help you with anything! Best, Jörn PS: I assigned the bug to me, so that it gets out of the 'unassigned bug list'
Matteo Caffini - 2015-11-26 12:11:47 +0100
Created attachment 757 Read ISS Imagent (conc files) header Read ISS Imagent conc files and provides useful data (in a hdr struct) for further labeling and reshaping of data. Jörn, I split HbO and HbR labels, gimme a feedback about this. :-) Ciao, Teo
Jörn M. Horschig - 2015-11-26 12:34:07 +0100
Comment on attachment 757 Read ISS Imagent (conc files) header Hi Teo, looks good! For a first start ;) Please do not split label into labelHbR and labelHbO. This is the definition of a header structure in FieldTrip: http://www.fieldtriptoolbox.org/reference/ft_read_header If you want, you can use hdr.orig for the additional information, but it should not be relied on that that information is present. We use for our data channel labelling probably a different rationale than Imagent does, i.e. our channels are called 'Rx8b-Tx15 [O2Hb]' and 'Rx8b-Tx15 [HHb]'. We settled on these conventions to standardize the use of NIRS data in FT. According to this convention all 'HbR' channels are affixed with [HHb] and all 'HbO' channels are affixed with [O2Hb]. If data contains raw ODs, the affix represents the respective wavelength (e.g. [750nm]). That way, channels of different concentraions can always be selected with ft_channelselection by e.g. asking for '*[O2Hb]' channels. You are of course not obliged to take over this receiver/transmitter naming convention, but I think it would be very useful to use the affix [] convention also for Imagent data. What do you think about that? For NIRS data, Robert and we also thought of a nice way to transform from concentrations to ODs and back. This would require ah hdr.opto structure, but I am not sure whether all information is stored in the imagent header file. hdr.opto contains information about e.g. wavelengths, mapping of wavelengts/laser to transmitter, what transmitter-receiver combination constitute what channel, etc. Is all that information present in your data? Btw, to answer the last question in your code: data is channel x samples, labels are channel x 1, and time is 1 x samples. time is a row vector, because that way you can intuitively see that the first dimension of the data refers to the label-field, and the second dimension of your data refers to the time-field .
Matteo Caffini - 2015-11-26 15:12:42 +0100
Thank you Jörn for the info, I had the feeling that the label split would have been a bad idea. :-) Your channel "[]" naming convention is ok, and I can adapt to it. Concerning OD to conc (and maybe also intensity to OD?), the Imagent raw data are intensity data in the AC-DC-Phase form (it's a frequency domain device) and probably we're going to need a different handling of this. So we have a few data types, like: 1) "Intensity", "AC", "DC", "Phase" (these data are coming from receivers and can be labeled "Rx8b [750nm]") 2) "OD", "HHb", "O2Hb" (these data are computed for Rx-Tx pairs, using the probe geometry and can be labeled "Rx8b-Tx15 [HHb]") I have to check how much of the probe geometry can be saved in my raw data txt files. Ciao, Teo ps I like [HbO] and [HbR] just because are strings of the same length... :-)
Matteo Caffini - 2015-11-27 11:20:46 +0100
I'm not sure how to deal with non channel columns that have a sample by sample entry (such as flag, mark, group, DigAux, ...). I think the header reader should keep track of this difference and the data reader should only read data from "real" channels. Then, chanindx may or may not be used for subselection (Should I make it a non-mandatory input? Should I put a default "all" possibility?). Sometimes a few non channel columns (e.g. DigAux) will be read by the event reader. Ciao, Teo
Jörn M. Horschig - 2015-11-30 10:20:22 +0100
Hi Teo, can the three data types be combined in one way or another? Usually, data.trial is just a real-valued time-series. Frequency domain data could be represented as complex numbers, but I am not sure if this would suffice here. What exactly does Intensity represent? Is it the raw laser output strength, or really something the receiver is measuring (similar to CW systems)? Are you suggesting to have something like (e.g. Tx2-Rx8 [Intensity ], Tx2-Rx8 [DC] , etc? Maybe you can upload a .mat file with a parsed FieldTrip header so that I get a better impression. The default for channels is usually 'all', e.g. in ft_preprocessing. It would make sense if by default 'all' returns also non-channel columns. I made a chantype 'nirs', which can be used to distinguish NIRS channels from the rest (see ft_chantype). That way, for example ft_preprocessing can be used with cfg.channel = 'nirs' to ignore the non-nirs channels. Does that sound good? Best, Jörn
Matteo Caffini - 2015-12-01 12:54:03 +0100
Hi Jörn, sorry my bad, maybe I messed up with the terminology. Let's define a few quantities concerning raw data. Instruments raw output (may be in text files or binary files) is usually stored in the form of: 1) Intensity (or fluence rate, photons per second per unit area or a similar physical quantity) seen by the receivers, in continuous wave instruments 2) AC-DC-Phase (at one or more than one modulation frequencies) seen by the receivers (usually "lock-in style"), in frequency domain instruments 3) Temporal spread function (intensity over time), in time domain instruments Then each time sample is represented by: 1) one real number in CW 2) three real numbers (or, same thing, DC+phasor) in FD 3) lots of real numbers in TD Of course this happens for each wavelength, so since most of us use two, twofold. Do we need two labels to completely describe each column content (something like [DC][750nm], [AC][830nm], ...)? Chantype "nirs" to group [O2Hb] and [HHb] channels is great! Ciao, Teo
Jörn M. Horschig - 2015-12-01 13:29:36 +0100
Hi Teo, thanks for the clarification post ;) TD data would indeed be somewhat complicated to deal with, so it's good that we do not have to deal with that (yet). It sounds to me like a good plan to have "[DC][760nm]", "[AC][760nm]" and "[Phase][760nm]" suffixes. Right now, I do not see any clever way to combine these three measures. Let's give Robert and JM some time to read up on your last comment and intervene on this approach if necessary. Feel free to go ahead if they have not responded by the end of this week :) Best, Jörn
Matteo Caffini - 2015-12-03 11:07:02 +0100
Hi Jörn, thanks for the feedback. Since we have a holiday in a few days here, I'll get back into coding on Wednesday, Dec 9th or so. Plenty of time for everyone to think about smart ways to label our data. No hurry, as they say, Rome wasn't built in a day. :-) Ciao! Teo
Matteo Caffini - 2016-02-23 11:40:57 +0100
Hi Jörn, hi everyone sorry I'm on and off with this project and I know it's not good, I'm doing too many things here in our lab. I'm reading there's a parallel discussion about FT and HOMER2 and I'd like to chat a little about what we need, what we have and what to add. Step back then. :-) My usual fNIRS workflow (probably not just mine...) usually involves all or many of these steps: 1) Design study and design proper probe geometry and pairs 2) Collect data 3) Pre-process data: import raw data in some sort of environment (usually MATLAB), compute HHb and HbO concentrations (using probe geometry and pairs), detrending, baseline subtraction, trial definition, trial selection, ... 4) Process data: trial average, group average, ... 5) Compare snippets of data, compute statistics, ... Normally I use HOMER2 and a few scripts of my own for 1-4. I use FT for 5 (FT could also be used for 4). Where should we develop new scripts? How can we handle things as simple as possible? Ciao, Teo
Jörn M. Horschig - 2016-02-23 11:58:17 +0100
Hi Teo, not sure with what you mean with 'where should we develop new scripts'. Scripts which are experiment-specific should be located at the researcher's computer. FT should only contain general functions (e.g. functions for filtering, trial selection, statistical tests, etc.). Generally it's wise to think in form of an analysis protocol (http://www.fieldtriptoolbox.org/tutorial/introduction#making_an_analysis_protocol). Ideally, you should be able to trace back each of these steps to an FT function. If a function is missing from FT, but you consider it a crucial step of your analysis (say, reference channel correction), let me know and we can discuss if it's indeed missing or whether current functions can be used for that. If such a function is missing, we for sure find a place for that somewhere. I will assist in that. Any concrete examples? Best, Jörn
Matteo Caffini - 2016-02-23 13:44:35 +0100
Hi Jörn, nothing exp-specific, I was just wondering at which point should a fNIRS researcher step into FT. For example, now I step in at 5 and it's ok. Here we're trying to implement support for ISS data in order to be able to step in at 3 (before or after conc computation complicates things a bit, but that's another story). Ciao, Teo
Jörn M. Horschig - 2016-02-23 14:07:26 +0100
Hey Teo, ah, I see ;) well, the reading functions I wrote for our systems indeed allow stepping into FT right after 2), i.e. preprocessing, filtering, segmenting, etc. I also wrote that OD2Conc function, so that everything essential can be done in FT. I actually also want to aim for that, but at the moment lack the time for working on this further. Your ISS-reading function would allow the user to also get in there at 3), wouldn't it? Of course, it's also nice to have an import function for later points in time. Anything goes ;)
Jörn M. Horschig - 2016-09-12 16:25:25 +0200
Hi Teo, no long time no update. Just wanted to check whether you ran into issues, or just have not updated us on the progress. It would be nice to hear back from you. Best, Jörm
Matteo Caffini - 2016-09-15 10:50:37 +0200
Hi Jörn, yeah sorry, I've been carried away a little bit (and mostly moved to Python, I confess :-D). I was able to process ISS fNIRS data with FT, stepping in at 5 after doing the trials averaging manually (matching the FT data structure). Still I've no general code to step in "earlier" (but the easy reading function I attached here last year, that actually doesn't do much). How do we move on? Ciao, Teo
Jörn M. Horschig - 2016-09-15 17:00:35 +0200
Hi Teo, depends on you :) If you want to share the code here, that'd be something, or maybe on the fieldtrip wiki. You could for example add a section here (Robert, please disagree if you feel this clutters things too much): http://www.fieldtriptoolbox.org/getting_started at other data, then describe the system and data a bit (copy&paste from what's posted here) and then add the code with some comments as an example. Of course you can also try starting from top again and implement a nice and neat reading function, but I guess that is not very realistic given that what you already got right now is good enough for you, or is it? Best, Jörn
Matteo Caffini - 2016-09-19 21:42:07 +0200
Hi Jörn, I will get back to the work I've done and describe the way I handled my data, in the form of a wiki. BTW, thanks for the call, this may be useful to others. Let's see what comes out. :-) Ciao, Teo
Jörn M. Horschig - 2016-09-20 09:39:58 +0200
great, thanks for willing to contribute!
Jörn M. Horschig - 2016-12-20 13:11:40 +0100
Hey Teo, any update? Any chance this will still be done?
Jörn M. Horschig - 2017-01-18 09:59:58 +0100
Dear Teo, it would be great if you could indicate if you still plan to upload part of your code, or if you have other priorities currently and consider it unlikely to do this the next weeks. No hard feeling if you have other priorities currently, but it would be good to let us know so that we know whether to close this bug or not. Best regards, Jörn
Matteo Caffini - 2017-01-31 10:05:25 +0100
Dear Jörn, sorry for the late reply. I must admit that don't have time for this now, but I have a colleague that is willing to do this in my place. :-) I'll point him to this thread and will work with him to prepare a wiki-like article about how we applied FieldTrip methods to our fNIRS data. Ciao! Teo