Back to the main page.
Bug 3059 - implement, extend, and document ECoG/sEEG visualization tools
| Status | CLOSED FIXED |
| Reported | 2016-02-01 19:31:00 +0100 |
| Modified | 2017-09-21 22:13:21 +0200 |
| Product: | FieldTrip |
| Component: | plotting |
| Version: | unspecified |
| Hardware: | PC |
| Operating System: | Mac OS |
| Importance: | P5 normal |
| Assigned to: | Roemer van der Meij |
| URL: | |
| Tags: | |
| Depends on: | 28373275 |
| Blocks: | |
| See also: | http://bugzilla.fieldtriptoolbox.org/show_bug.cgi?id=3334 |
Roemer van der Meij - 2016-02-01 19:31:14 +0100
This bug is intended as a discussion and memo thread for the implementation of tools to visualize human and animal brain surface recordings (electrocorticography, ECoG; etc) and brain depth recordings (stereotactic EEG, sEEG; etc). The goal of these tools is first and foremost to aid the researcher in data exploration of highly individualized results (i.e. subjects with varying electrode coverage). If applicable in the end, also to generate (raw material for) figures for publication. A distinction will be made between brain SURFACE and brain DEPTH recordings where applicable. A distinction will be made between visualizing SCALAR results and VECTOR/MATRIX results. Typical SCALAR results: - power in a single frequency band (+ e.g. some statistical valuation) - inter-trial coherence in a single frequency band - intra-electrode phase-amplitude coupling between two single frequencies Typical VECTOR/MATRIX results: - event-related potentials, 1d vector of amplitude over time - time-frequency representations of power, 2d matrix of power over frequencies, over time SCALAR results could be visualized by e.g.: - plotting an electrode marker with a size/color - (SURFACE) coloring of the surrounding pial surface with transparency of coloring - (DEPTH) colored overlay of the surrounding grey matter of sliced MRI with transparency VECTOR/MATRIX results could be visualized by e.g.: - (SURFACE) a subplot type square oriented towards the viewer on top of pial surface - (DEPTH) a subplot type square oriented towards the viewer on top of a sliced MRI SCALAR tools will be implemented first. A highly useful tool in all of the above situations will be to automatically detect the viewpoints of the pial surface and/or the slices of the MRI in which the results are visible. An SVD on the electrode locations will likely be the main tool. Some of the above will be rotatable (i.e. the SCALAR results), but this not certain for others. Automatic detection might result in multiple viewpoints per subject. It is acceptable for this to be a little crude, as long as it improves the visualization. All plotting functions should allow for a specific viewpoint and depth/surface as input, and if absent try to find them automatically.
Roemer van der Meij - 2016-02-01 19:31:57 +0100
Added dependency to Arjen's main ecog thread
Roemer van der Meij - 2016-02-01 19:40:06 +0100
First step, implement SCALAR visualization. To follow our current naming scheme, a candidate name is: ft_intracranplotER This function will take as main non-cfg input traditional output data, such as that from ft_freqanalysis, ft_timelockanalysis, etc. A second non-cfg input will be the MRI or pial surface. This can be a single subject's brain or a template brain. This main input data should contain the TAL/MNI coordinates of the subject at hand, in an data.elec field.
Arjen Stolk - 2016-02-01 19:42:02 +0100
:D
Roemer van der Meij - 2016-02-01 20:01:24 +0100
(In reply to Roemer van der Meij from comment #2) W.r.t. to function naming, the ER/TFR naming distinction doesn't hold well. We can: (1) choose to maintain it, as it still holds a little bit, in order to be congruent with other plotting functions (2) use a different scheme...(?) (3) use a main interface function ft_intracranplot, and split into scal/1d/2d as lower level handling functions (4) ...?
Roemer van der Meij - 2016-02-02 02:42:02 +0100
roemer$ svn commit ft_intracranplotER.m -m "initial commit" Adding ft_intracranplotER.m Transmitting file data . Committed revision 11159. First version has been added. It's a bit barebones right now, and only does surface electrodes currently, not a lot of input data allowed, etc. The coloring of the cortex works, but needs fine tuning (w.r.t. smoothing and distance dropoff of the influence of electrodes). Probably better to go with a Gaussian smoothing with a FWHM of about 10-30mm or so. I'll hopefully get a chance to work on it again soon, there's a few immediate things higher on my priority list for the time being alas (and in the past months I'm afraid). It can be run with the following test-code: (not commited as test script, until it matures) ********* From the directory that contains the ECoG example subject ********* % load elec and pial load('SubjectECoG_elec.mat'); pial = ft_read_headshape({'SubjectECoG_lh.pial','SubjectECoG_rh.pial'},'concatenate','yes'); % as a workaround add chantype to elec elec.chantype = repmat({'surf'},[numel(elec.label) 1]); % create some random data freq = []; freq.label = elec.label; freq.powspctrm = rand(numel(freq.label),1); freq.dimord = 'chan_freq'; freq.freq = 1; freq.elec = elec; % plot cfg = []; cfg.method = 'both'; % 'point','area' cfg.anatomypial = pial; cfg.parameter = 'powspctrm'; cfg.paramsel = 1; cfg.paramlim = 'minmax'; cfg.channel = ft_channelselection(elec.label,'*G*'); % for now, only select probable surface electrodes ft_intracranplotER(cfg,freq);
Arjen Stolk - 2016-02-02 04:49:45 +0100
Hey Roemer, Nice to see this moving forward. How does the surface plotting work? And, is it different from how data is plotted with ft_sourceplot using cfg.method = 'surface'? This approach uses a headshape as input (cfg.surffile if I'm not mistaken), interpolates data in-between say 1cm distant grid points (source-estimated activity), and then projects that data onto the surface. Ft_sourceplot also allows plotting volumetric data (cfg.method = 'ortho'). In a way ft_electrodeplacement is a pimped version of ft_sourceplot, allowing plotting electrodes within the orthoplots. I guess this functionality could serve our purposes of plotting depths too? Moreover, I would leave the distinction between grids and depths up to the user. For instance, myself I keep track of the labels of the grids and depths for each subject, and use ft_channelselection to select the channels I want. This separation is already needed at the first stages of analysis, when re-referencing the data, separately per grids/depths. For example (selecting elecs belonging to either the LPG or the LTG grids): grids = ft_channelselection({'*LPG*','LTG'}, raw.label); cfg = []; cfg.channel = grids; grid_data = ft_selectdata(cfg, raw); % select ECoG grid channels I also came across a couple of cases where the labelling was mixed up. That is, those labels need to be adjusted to each case, and it is easy to make mistakes here. No problem for the user to deal with these idiosyncrasies, but for ft functionality to automatically detect what elec belongs to what grid/shaft is going to be a b**ch (beach, of course). Hope these (limited) experiences help, Ciao!
Roemer van der Meij - 2016-02-02 19:56:51 +0100
(In reply to Arjen Stolk from comment #6) Not automatically detect, but specified by the user in ft_electrodeplacement of course. The naming scheme at Irvine and some bay area ECoG happens to be useful right now, but this not necessarily so of course. The reason specifying this makes sense to me, is because the distinction for the end-user is very important. When you're busy plotting and interpreting the data, this distinction is really unavoidable, there's no way around it. As a comparison, this to me is similar to the distinction between planar and axial gradiometers, each have their distinct way of handling. So, I'd really like to make a case for this distinction ;). It shouldn't be hard to add to ft_electrodeplacement? I based the function off of ft_sourceplot and the other plotting functions. For the depths I'll for sure use the functionality in ft_sourceplot! The biggest hurdle is not the actual plotting, but determining the precise slice that it should be plotted in. I think, this can be done automatically, which for sure would save a hell of a lot of time for the end-user. Think of it, how would you determine the slice for a set of electrodes now? Awful :). For the plotting by coloring the area underneath the electrodes I use functionality from ft_sourceinterpolate (and sourceplot), but the methods of interpolation I used so far were not very satisfying. This is due to higher precision and uneven sampling, I think. Not difficult to fix though, a guassian smudge worked great for me in the past (but I couldn't quite get it right yesterday). I'm not very happy with the name yet (the ER part).
Arjen Stolk - 2016-02-02 20:25:49 +0100
"Not automatically detect, but specified by the user in ft_electrodeplacement of course." > What about re-referencing, which does not have access to this elec information? I'd leave all this book-keeping up to the user (e.g. see example channelselection code above), because it's idiosyncratic and perhaps too categorical with regards to future developments (in which many different types of electrodes may see the light). The user has to therefore keep in mind what data he/she is working with, not us having to update every time a new electrode type sees the light. "I based the function off of ft_sourceplot and the other plotting functions." > So it should be possible to build this back in? Do have a screenshot of the plots actually - very curious? ;) "The biggest hurdle is not the actual plotting, but determining the precise slice that it should be plotted in." > To capture a whole electrode shaft you mean, rather than just one? Sounds tricky indeed. Perhaps, when plotting an electrode shaft on an MR, it's possible to weight the electrodes sizes as a function of their distances to the actual slice, i.e. to give a feeling of 3D?
Roemer van der Meij - 2016-02-02 22:24:08 +0100
> What about re-referencing, which does not have access to this elec information? I'd leave all this book-keeping up to the user (e.g. see example channelselection code above), because it's idiosyncratic and perhaps too categorical with regards to future developments (in which many different types of electrodes may see the light). The user has to therefore keep in mind what data he/she is working with, not us having to update every time a new electrode type sees the light. I'm not following you I'm afraid. There's only 2 types, inside and outside, each requiring a very different way of visualization, purely by nature of being penetrating the brain vs not penetrating the brain. I don't see the relation with rereferencing either, which is very idiosyncratic (some do a ref per grid, some for all). In all my plotting over the years however, it's always been depths vs surface. One can be plotted on a pial surface, the other cannot. And recognizing depths vs surface comes for free when using ft_electrodeplacement, as the person using it will always have to have a schematic nearby that contains this information anyway. > So it should be possible to build this back in? Do have a screenshot of the plots actually - very curious? ;) What do you mean? :p Build what back in? You mean doing everything in ft_sourceplot? Hmmmm, not thought of that. > To capture a whole electrode shaft you mean, rather than just one? Sounds tricky indeed. Perhaps, when plotting an electrode shaft on an MR, it's possible to weight the electrodes sizes as a function of their distances to the actual slice, i.e. to give a feeling of 3D? That's a good idea, but you'd still need an approximate slice fitting the two angles of the electrode? With slice btw, I don't mean the traditional sagittal vs axial vs (?) slices, but a 2D plane cutting through the 3D space which should match (reasonably) with the plane of the electrode. The usefulness of this all is limited, because often, one would not necessarily plot depths on a brain, but rather determine the anatomical region (amy, hipp, etc) based on the single subject MRI, and plot as a function of those labels. Nevertheless, the automatic detection of dominant orientations/viewpoints is most useful for surface grids, and than the depth plane detection comes (roughly) for free.
Arjen Stolk - 2016-02-02 23:45:42 +0100
"There's only 2 types, inside and outside, each requiring a very different way of visualization, purely by nature of being penetrating the brain vs not penetrating the brain." > Okay, why not using ft_selectdata or cfg.channel prior to plotting? As you say, it needs to be plotted separately anyway, or do you want you pop-up different plots (e.g., a ortho and surface) at the same time? If I still haven't convinced you, proceed as you wish, and we can see later on whether and how to build additional functionality in ft_electrodeplacement.
Roemer van der Meij - 2016-02-03 01:23:25 +0100
(In reply to Arjen Stolk from comment #10) Manual selection via ft_channelselection it would be it we don't save this information somewhere. There are situations where I'd want to plot both at the same time, but they will of course pop up different windows. This it would also do for multiple viewpoints, one fig per viewpoint. In the example patient grids-only, it would be one 2-3 viewpoints IIRC. One for the main grid, one from the bottom, perhaps one from the right bottom. Of course, there would always be an option to plot all in one fig (for surface and depth separately) and which can then be rotated. Now that I think of it, I haven't really made the usefulness of automating explicit. The whole thought of mine for automating is that often I want to try some analysis tweak on say, 10 subjects, which I then want to quickly flip through to get an overview of the effects (or saving on disk and doing the same). Manually rotating is very unsatisfying, and I've notice it strongly limits how much I can see and experiment in a given time span. Hence, the idea of (crudely) automatically detecting the viewpoints. As analogy, we have MEG and EEG layouts that are 2D for the same reason, rotating each recording manually to see all the data is costly in terms of time, and our own 'pattern recognition'. The depth vs surface electrode labels is just an additional little step in making things easy. It feels a bit low level, which would be nice if FT could handle. Like, FT detects EEG vs MEG electrodes, detects which CTF system were using, etc. And I'd say surf vs depth would fit well in this line of detection (as it partially free during placement). But let's see and let it rest a bit, it's not that important, maybe in a few weeks it will come to us :).
Roemer van der Meij - 2016-02-03 23:59:27 +0100
(thought I had taken it already)
Roemer van der Meij - 2016-02-08 21:06:38 +0100
@Arjen/Robert For better development practice, I removed ft_intracranialplotER from the master branch on github fieldtrip/fieldtrip. There is a branch called ecogvisual on roemervandermeij/filter, were I'll continue development. Arjen, you can use the following to add the branch to your own local repository. You can then push changes to your own fieldtrip fork, and use a pull request to to alert me of the changes, so that I can incorporate them as well. git remote add ftforkroemer https://github.com/roemervandermeij/fieldtrip.git git pull ftforkroemer ecogvisual pit push origin ecogvisual >(this pushes the branch to your own fork, assuming its called origin) Wijzigen kan je dan zo doen: git checkout ecogvisual **wijziging** git add ft_intracranialplotER.m git commit -m 'wijziging' >(je kan ook direct committen, maar dit is niet aan te raden, omdat het dan onduidelijk is wat je precies commit) git push origin ecogvisual *git pull request vanuit mijn repo Een andere optie voor dergelijk werk, is misschien 'protected' branches Robert. Je kan de master van fieldtrip/fieldtrip 'protected' maken, met een admin-check voor pull request (volgens mij zoals het nu ook is). De rest van het team kan dan vrij branches maken en wijzigen, en dan hoeven we niet via elkaars fork te gaan. Nadeel is wel dat iedereen alle branches kan wijzigen (behalve master dus). (Ik dacht overigens eerst ook dat het via github teams kon, maar dat bleek niet zo, vandaar dat er nu een 'spook' team aanwezig is.) Ik ben er nog niet echt over uit wat mijn voorkeur zou zijn, misschien goed om in een FT-meeting te bespreken?
Arjen Stolk - 2016-02-08 23:10:08 +0100
Thanks, mate. I'm honestly a bit reluctant to create these additional forks give my current git level of experience. But please keep me updated 'conceptually', also for discussion on how to best integrate your advances into existing functionality.
Roemer van der Meij - 2016-02-09 01:19:54 +0100
I thought we could use something like the following, which I stole from ft_volumerealign. However, it is clear the electrodes have partially 'sunk' into the brain. Which I was surprised about. For sure there could be some noise in the procedure, but it extracts 'brain' (defined as csf, white, grey), as such it should fit? Or am I interpreting ft_volumesegment incorrectly? Regardless, it made me wonder how successful the coregistration was. Could you add a step to the tutorial to check this? Is there a way we can plot both of them on top of each other? I added a fixme for the tutorial for this. ************* % read MRI mri = ft_read_mri('SubjectECoG_MR_preproc.mgz'); mri.coordsys = 'tal'; % segment mri and extract brain cfg = []; cfg.method = 'spm8'; cfg.output = 'brain'; segmented = ft_volumesegment(cfg, mri); % use extracted brain to generate surface cfg = []; cfg.tissue = 'brain'; cfg.method = 'isosurface'; cfg.numvertices = inf; brain = ft_prepare_mesh(cfg, segmented); % plot mesh figure ft_plot_mesh(brain, 'edgecolor', 'none', 'facecolor', 'cortex') camlight(0,0); lighting phong material dull set(gca,'DataAspectRatio',[1 1 1], 'PlotBoxAspectRatio', [1 1 1]); alpha(.7) hold on % add electrodes plot3(elec.elecpos(:,1),elec.elecpos(:,2),elec.elecpos(:,3),'o','markerfacecolor','r','markeredgecolor',[0 0 0]) *************
Roemer van der Meij - 2016-02-09 01:21:19 +0100
Please ignore above comment, wrong thread :/.
Robert Oostenveld - 2017-03-23 17:15:20 +0100
this issue has partially been resolved by https://github.com/fieldtrip/fieldtrip/pull/335 and https://github.com/fieldtrip/fieldtrip/pull/382 I just made some more changes to the layout stuff in the master (which is now again mature and stable afaik) mac011> git commit ft_prepare_layout.m test/test_ft_prepare_layout.m [master cbde14c] ENH - allow multiple headshapes for left/right hemishpere, extended the example script with ECoC + sEEG test case 2 files changed, 97 insertions(+), 55 deletions(-) mac011> git push upstream master You're about to push master, is that what you intended? [y|n] y Counting objects: 13, done. Delta compression using up to 4 threads. Compressing objects: 100% (13/13), done. Writing objects: 100% (13/13), 3.38 KiB | 0 bytes/s, done. Total 13 (delta 9), reused 0 (delta 0) remote: Resolving deltas: 100% (9/9), completed with 7 local objects. To github.com:fieldtrip/fieldtrip.git 5bb8f0d..cbde14c master -> master
Robert Oostenveld - 2017-03-23 17:26:26 +0100
(In reply to Robert Oostenveld from comment #17) Please check https://github.com/fieldtrip/fieldtrip/pull/382#issuecomment-288775221 Does this now conclude the non-functional (*) part of the iEEG plotting? I think not, as there was also discussion (and partial(?) implementation) of separate versus appended elec structures, and idem for layouts. @Roemer: remember the unfolded paper box with 6 sides? The layout code seems to be able to do it now (i.e. each of the 6 sides). Is there more that needs to be done? *) there is also the functional aspect, i.e. how to color-code or otherwise visualize functional data.
Roemer van der Meij - 2017-03-23 20:05:17 +0100
Very neat reorganization Robert. It's a shame the eeg detection didn't work, but it's also a hard problem. The cfg.mask/outline options are a good solution. There's a couple of fixes I'll make to the headshape outline, I'll submit that in a bit. I think it's pretty solid after that. I'll have a look at the columnX outlines, which had a bug in there I think, and make a row version too. Another thing I can make is an ft_appendlayout, which we talked about as well. Not a difficult function to make, in essence it just a rescaling of each layout and combining them in a certain way. Do you think it's useful next to using simple subplots? One downside of subplots is that they tend to have very wide gaps between plots, not great real estate usage for scenarios. Another is unnecessary looping, if one wants to plot e.g. TFRs for grids (headshape) and depths (row/column) in one figure. Right now I'm thinking lay = ft_appendlayout(cfg,lay1,lay2,lay3,...,layN) cfg.method = [NxM] cfg.method = 'square' cfg.method = 'row' cfg.method = 'column' cfg.skipscale cfg.skipcomnt Where the latter three are just ways of creating the first. The first indicates where each lay should go, with a 0 where non should be. For example [1 2 3; 4 5 6] --> 6 lays in 2x3 row-wise order [1 2;3 4; 5 6]' --> 6 lays in 2x3 column-wise order [1 2 3; 4 0 5; 6 7 8] --> 8 lays in 3x3 with a hole in the middle etc This can in principle be used for situations with different datatypes (e.g. grad/planar neuromag), by not generating a new window and giving the same lay, which holds for most plotting functions I think. Still though, there will be only be one COMNT/SCALE box. If multiple are desired, subplots are more applicable.
Robert Oostenveld - 2017-03-24 09:25:57 +0100
(In reply to Roemer van der Meij from comment #19) The dashboard test batch detected some errors on which I am presently working. All relatively minor, but need to be fixed. ---- roboos@mentat001> git commit -a [master e506238] FIX - resolved some errors in layout generation, these were detected by running the test batch 5 files changed, 17 insertions(+), 9 deletions(-) roboos@mentat001> git push upstream master Warning: untrusted X11 forwarding setup failed: xauth key data not generated Warning: No xauth data; using fake authentication data for X11 forwarding. X11 forwarding request failed on channel 0 Counting objects: 14, done. Delta compression using up to 4 threads. Compressing objects: 100% (8/8), done. Writing objects: 100% (8/8), 2.27 KiB | 0 bytes/s, done. Total 8 (delta 6), reused 0 (delta 0) remote: Resolving deltas: 100% (6/6), completed with 6 local objects. To git@github.com:fieldtrip/fieldtrip.git 0512e03..e506238 master -> master ---- I realize that especially the MRI based outline+mask are slow, because it calls the sub function twice. That needs improvement. I will add the non-topographic layouts to the test.
Roemer van der Meij - 2017-03-24 23:39:16 +0100
I added two small tweaks, see PR https://github.com/fieldtrip/fieldtrip/pull/385 1) Allows proper projection of headshape/mri if cfg.projection ~= orthographic or if viewpoint = [], using cfg.rotate and elproj.m as in subfunction sens2lay 2) Makes sure the headshape/mri outline is reused if possible
Robert Oostenveld - 2017-09-15 09:44:36 +0200
Hi Roemer, I have the idea this is largely done, except for some last (and ongoing) refinements. If agreed, please close this issue and open new ones (or PRs) when new ideas pop up. Robert
Roemer van der Meij - 2017-09-21 22:12:24 +0200
Part of the ideas on top nicely implemented by Arjen, closing. Will reopen if I find a way to make little multiplot squares on a surface that stay oriented towards the 'camera'.