Back to the main page.
Bug 449 - improve the release procedure on the ftp server to avoid many zip files that are the same
| Status | CLOSED FIXED |
| Reported | 2011-01-27 12:08:00 +0100 |
| Modified | 2011-02-12 11:50:09 +0100 |
| Product: | FieldTrip |
| Component: | release |
| Version: | unspecified |
| Hardware: | PC |
| Operating System: | Mac OS |
| Importance: | P1 normal |
| Assigned to: | Simon Oosthoek |
| URL: | |
| Tags: | |
| Depends on: | |
| Blocks: | |
| See also: |
Robert Oostenveld - 2011-01-27 12:08:40 +0100
Every evening a zip file is made and copied to the ftp server. This also happens if the code (i.e. the content of the zip file) did not change, e.g. in the weekends. The consequence is that there are many zip files that are identical. The problem is not so much with the fieldtrip-YYYMMDD files, but more so with the pub/fieldtrip/modules where the separate mini-toolboxes are avaialble as stand-alone packages (this is largely symbolic, because most people will download a complete FT copy, but nevertheless important). it would be nice if the ftp uplaod script would only upload if the file had changed
Robert Oostenveld - 2011-01-27 12:09:37 +0100
(In reply to comment #0) example on the ftp server manzana> ftp ftp Connected to dccn-l007.fcdonders.nl. 220 Welcome to F.C. Donders FTP service. 331 Please specify the password. 230 Login successful. Remote system type is UNIX. Using binary mode to transfer files. ftp> ls pub/fieldtrip/modules 229 Entering Extended Passive Mode (|||29882|) 150 Here comes the directory listing. ... -rw-r--r-- 1 500 500 49723 Jan 07 21:15 preproc-20110107.zip -rw-r--r-- 1 500 500 49723 Jan 08 21:15 preproc-20110108.zip -rw-r--r-- 1 500 500 49723 Jan 09 21:15 preproc-20110109.zip -rw-r--r-- 1 500 500 49723 Jan 10 21:15 preproc-20110110.zip -rw-r--r-- 1 500 500 49723 Jan 11 21:15 preproc-20110111.zip -rw-r--r-- 1 500 500 49723 Jan 12 21:15 preproc-20110112.zip -rw-r--r-- 1 500 500 49723 Jan 13 21:15 preproc-20110113.zip -rw-r--r-- 1 500 500 49723 Jan 14 21:15 preproc-20110114.zip -rw-r--r-- 1 500 500 49723 Jan 15 21:15 preproc-20110115.zip -rw-r--r-- 1 500 500 49723 Jan 16 21:15 preproc-20110116.zip -rw-r--r-- 1 500 500 49723 Jan 18 21:15 preproc-20110118.zip -rw-r--r-- 1 500 500 49723 Jan 19 21:15 preproc-20110119.zip -rw-r--r-- 1 500 500 49755 Jan 20 21:15 preproc-20110120.zip -rw-r--r-- 1 500 500 49755 Jan 21 21:15 preproc-20110121.zip -rw-r--r-- 1 500 500 49755 Jan 22 21:15 preproc-20110122.zip -rw-r--r-- 1 500 500 49755 Jan 23 21:15 preproc-20110123.zip -rw-r--r-- 1 500 500 49755 Jan 24 21:15 preproc-20110124.zip -rw-r--r-- 1 500 500 49755 Jan 25 21:15 preproc-20110125.zip -rw-r--r-- 1 500 500 53441 Jan 26 21:15 preproc-20110126.zip -rw-r--r-- 1 500 500 16934 Nov 23 21:19 specest-20101123.zip -rw-r--r-- 1 500 500 16959 Nov 24 21:19 specest-20101124.zip -rw-r--r-- 1 500 500 16959 Nov 25 21:19 specest-20101125.zip -rw-r--r-- 1 500 500 16959 Nov 26 21:19 specest-20101126.zip -rw-r--r-- 1 500 500 16951 Nov 27 21:19 specest-20101127.zip -rw-r--r-- 1 500 500 16951 Nov 28 21:19 specest-20101128.zip -rw-r--r-- 1 500 500 16951 Nov 29 21:19 specest-20101129.zip -rw-r--r-- 1 500 500 16951 Nov 30 21:19 specest-20101130.zip -rw-r--r-- 1 500 500 16951 Dec 01 21:19 specest-20101201.zip -rw-r--r-- 1 500 500 16951 Dec 02 21:19 specest-20101202.zip -rw-r--r-- 1 500 500 16951 Dec 03 21:19 specest-20101203.zip -rw-r--r-- 1 500 500 16951 Dec 04 21:19 specest-20101204.zip -rw-r--r-- 1 500 500 16951 Dec 05 21:19 specest-20101205.zip -rw-r--r-- 1 500 500 16951 Dec 06 21:19 specest-20101206.zip -rw-r--r-- 1 500 500 16951 Dec 07 21:19 specest-20101207.zip -rw-r--r-- 1 500 500 16951 Dec 08 21:19 specest-20101208.zip -rw-r--r-- 1 500 500 16951 Dec 09 21:19 specest-20101209.zip -rw-r--r-- 1 500 500 16951 Dec 10 21:19 specest-20101210.zip -rw-r--r-- 1 500 500 16951 Dec 11 21:19 specest-20101211.zip -rw-r--r-- 1 500 500 16951 Dec 12 21:19 specest-20101212.zip -rw-r--r-- 1 500 500 16951 Dec 13 21:19 specest-20101213.zip ...
Robert Oostenveld - 2011-01-27 12:11:35 +0100
the scripts that do it are in ~roboos/bin and are named release-ftp, release-fileio etcetera, and are called from a cronjob on mentat004. It relies on a ~/.netrc file for password-less ftp and the script content is like this ------------------------------------ #!/bin/sh TODAY=`date +%Y%m%d` TRUNK=$HOME/fieldtrip/release/trunk # this is the first to run every evening, so it should do a full svn update cd $TRUNK && svn update cd $HOME/fieldtrip/release || exit 1 rsync -ar --copy-links --delete --exclude .svn $TRUNK/ release-ftp || exit 1 mv release-ftp fieldtrip-$TODAY zip -r daily/fieldtrip-$TODAY.zip fieldtrip-$TODAY zip -r daily/fieldtrip-lite-$TODAY.zip fieldtrip-$TODAY -x '*.mat' mv fieldtrip-$TODAY release-ftp MD5_LATEST=`cat daily/fieldtrip-latest.zip | md5sum` MD5_TODAY=`cat daily/fieldtrip-$TODAY.zip | md5sum` if [ "$MD5_LATEST" = "$MD5_TODAY" ] ; then # the current release has not been updated compared to the previous rm daily/fieldtrip-$TODAY.zip rm daily/fieldtrip-lite-$TODAY.zip exit else # the current release is an updated version rm -f daily/fieldtrip-latest.zip ln daily/fieldtrip-$TODAY.zip daily/fieldtrip-latest.zip rm -f daily/fieldtrip-lite-latest.zip ln daily/fieldtrip-lite-$TODAY.zip daily/fieldtrip-lite-latest.zip ftp ftp.fcdonders.nl << EOF put daily/fieldtrip-$TODAY.zip /pub/fieldtrip/fieldtrip-$TODAY.zip put daily/fieldtrip-lite-$TODAY.zip /pub/fieldtrip/fieldtrip-lite-$TODAY.zip EOF #cd $HOME/fieldtrip/release || exit 1 #rsync -ar --copy-links --delete --exclude .svn --exclude data --exclude wiki /home/common/matlab/fieldtrip/ release-ftp || exit #mv release-ftp fieldtrip-$TODAY #zip -r daily/fieldtrip-$TODAY.zip fieldtrip-$TODAY #zip -r daily/fieldtrip-lite-$TODAY.zip fieldtrip-$TODAY -x '*.mat' #mv fieldtrip-$TODAY release-ftp # #ftp ftp.fcdonders.nl << EOF #put daily/fieldtrip-$TODAY.zip /pub/fieldtrip/fieldtrip-$TODAY.zip #put daily/fieldtrip-lite-$TODAY.zip /pub/fieldtrip/fieldtrip-lite-$TODAY.zip #EOF fi
Simon Oosthoek - 2011-02-10 16:54:12 +0100
Release module (generic): #!/bin/sh MODULE=$(basename $0 .sh | sed 's/release-\(.*\)/\1/') TODAY=$(date +%Y%m%d) BASEDIR=$HOME/fieldtrip/release TRUNK=$BASEDIR/trunk MD5FILE=$BASEDIR/.tarmd5-release-$MODULE cd $TRUNK # && svn update cd $HOME/fieldtrip/release || exit 1 rsync -ar --copy-links --delete --exclude .svn $TRUNK/$MODULE/ release-$MODULE || exit 1 CURRMD5=$(tar cf - release-$MODULE |md5sum |awk '{print $1}') LASTMD5=$(cat $MD5FILE) if [ "x$CURRMD5" = "x$LASTMD5" ] then # the current release has not been updated compared to the previous exit 0 else # the current release is an updated version echo $CURRMD5 > $MD5FILE mv release-$MODULE $MODULE-$TODAY zip -r daily/$MODULE-$TODAY.zip $MODULE-$TODAY mv $MODULE-$TODAY release-$MODULE rm -f daily/$MODULE-latest.zip ln daily/$MODULE-$TODAY.zip daily/$MODULE-latest.zip ftp ftp.fcdonders.nl <<-EOF put daily/$MODULE-$TODAY.zip /pub/fieldtrip/modules/$MODULE-$TODAY.zip EOF fi Release ftp (fieldtrip): #!/bin/sh TODAY=$(date +%Y%m%d) BASEDIR=$HOME/fieldtrip/release TRUNK=$BASEDIR/trunk MD5FILE=$BASEDIR/.tarmd5-release-ftp cd $TRUNK && svn update cd $HOME/fieldtrip/release || exit 1 rsync -ar --copy-links --delete --exclude .svn $TRUNK/ release-ftp || exit 1 CURRMD5=$(tar cf - release-ftp |md5sum |awk '{print $1}') LASTMD5=$(cat $MD5FILE) if [ "x$CURRMD5" = "x$LASTMD5" ] then # the current release has not been updated compared to the previous exit 0 else # the current release is an updated version echo $CURRMD5 > $MD5FILE mv release-ftp fieldtrip-$TODAY zip -r daily/fieldtrip-$TODAY.zip fieldtrip-$TODAY zip -r daily/fieldtrip-lite-$TODAY.zip fieldtrip-$TODAY -x '*.mat' mv fieldtrip-$TODAY release-ftp rm -f daily/fieldtrip-latest.zip ln daily/fieldtrip-$TODAY.zip daily/fieldtrip-latest.zip rm -f daily/fieldtrip-lite-latest.zip ln daily/fieldtrip-lite-$TODAY.zip daily/fieldtrip-lite-latest.zip ftp ftp.fcdonders.nl <<-EOF put daily/fieldtrip-$TODAY.zip /pub/fieldtrip/fieldtrip-$TODAY.zip put daily/fieldtrip-lite-$TODAY.zip /pub/fieldtrip/fieldtrip-lite-$TODAY.zip EOF fi