Back to the main page.
Bug 1747 - the reading functions can support URL syntax, download on the fly
| Status | CLOSED WONTFIX |
| Reported | 2012-09-26 22:32:00 +0200 |
| Modified | 2019-08-10 12:29:27 +0200 |
| Product: | FieldTrip |
| Component: | fileio |
| Version: | unspecified |
| Hardware: | PC |
| Operating System: | Mac OS |
| Importance: | P3 normal |
| Assigned to: | Eelke Spaak |
| URL: | |
| Tags: | |
| Depends on: | |
| Blocks: | |
| See also: | http://bugzilla.fcdonders.nl/show_bug.cgi?id=1038 |
Robert Oostenveld - 2012-09-26 22:32:27 +0200
this was sugegsted by Boris in bug 1038#3 > Maybe some ideas can be borrowed from Python's DataSource: > http://docs.scipy.org/doc/numpy/reference/generated/numpy.DataSource.html > It implements caching and opening from URLs on top of Robert's proposal. I have added the decoding of SFTP and FTP identifiers and implemented the support for SFTP. It works in principle, however rapidly fails in case a datafile needs to be complemented with another (unspecified one). roboos@dccn-c007> svn commit Sending fileio/ft_read_atlas.m Sending fileio/ft_read_data.m Sending fileio/ft_read_event.m Sending fileio/ft_read_header.m Sending fileio/ft_read_headshape.m Sending fileio/ft_read_mri.m Sending fileio/ft_read_sens.m Sending fileio/ft_read_spike.m Sending fileio/ft_read_vol.m Adding fileio/private/fetch_url.m Sending fileio/private/filetype_check_uri.m Transmitting file data ........... Committed revision 6552.
Robert Oostenveld - 2012-09-26 22:33:49 +0200
To be done: - implement support for FTP - implement support for HTTP - possibly implement support for SMB (imagine downloading a file on the fly on mentat from your windows desktop)
Eelke Spaak - 2012-09-27 08:31:21 +0200
Would it be an idea to just use Java's URL? I understand that Matlab provides full access to the Java API. Java transparently supports literally everything we could want: file, smb, http, ftp, sftp, and even zip/gzip/tar given the correct handlers. Plus, it's fairly trivial to implement your own handlers (as opposed to it being probably quite messy in procedural code).
Robert Oostenveld - 2012-09-27 09:49:32 +0200
(In reply to comment #2) yes, that is what I had in mind for the FTP and HTTP version. I did not know that Java does sftp, so that is why I implemented that one with a simple system call. How to wrap the java code in the MATLAB code is not yet clear. E.g. MATLAB has an ftp object that apparently wraps something. ---------- >> help ftp FTP Create an FTP object. FTP(host,username,password) returns an FTP object. If only a host is specified, it defaults to "anonymous" login. There is also an object for http-style URLs ---------- >> help urlread URLREAD Returns the contents of a URL as a string. S = URLREAD('URL') reads the content at a URL into a string, S. If the server returns binary data, the string will contain garbage. ... s = urlread('http://www.mathworks.com') s = urlread('ftp://ftp.mathworks.com/README') s = urlread(['file:///' fullfile(prefdir,'history.m')]) The FT specific interface that I have now defined is in fileio/private/fetch_url.m ------------- % FETCH_URL checks the filename and downloads the file to a local copy in % case it is specified as an Universal Resource Locator. It returns the % name of the temporary file on the local filesystem. % % Use as % filename = fetch_url(filename) % % In case the filename does not specify an URL, it just returns the original % filename. Anything that can be put inside that function would in principle be fine with me. The only
Robert Oostenveld - 2012-09-27 09:50:12 +0200
This works >> s = urlread('ftp://ftp.fcdonders.nl/pub/fieldtrip/00README.txt') But this not >> s = urlread('sftp://mentat001//home/common/matlab/fieldtrip/data/ftp/00README.txt') ??? Error using ==> urlread at 74 Either this URL could not be parsed or the protocol is not supported.
Robert Oostenveld - 2012-09-27 09:52:33 +0200
(In reply to comment #4) Here it gets a bit unclear ... >> s = urlread('ftp://ftp.fcdonders.nl/pub/fieldtrip/fieldtrip-lite-20120926.zip'); >> whos s Name Size Bytes Class Attributes s 1x19346980 38693960 char Am I now able to simply do filename = tempfile fid = fopen(filename, 'wb') fwrite(fid, s); fclose(s) and then have the same file on the local filesystem? This is something to be checked.
Robert Oostenveld - 2012-09-27 10:07:27 +0200
(In reply to comment #5) downloading into a MATLAB variable and then saving to a file can become problematic w.r.t. memory for large files
Eelke Spaak - 2014-01-29 15:02:12 +0100
Marking this as wontfix for now, it seems a feature that should be in matlab's file handling routines, but they are not as modern as they should be. Anyone who feels like it should feel free to reassign to themselves :)