Duncan Murdoch wrote:
On 28/07/2010 8:10 PM, Ray Brownrigg wrote:
NOTE: Now submitted to R-devel, as this seems more appropriate. I may have spoken too soon about this having been fixed. (see below). If I create another "unusual but not 'invalid'" filename in the R subdirectory, the behaviour is different from that reported below, and is similar to the original poster's output (the third "unlink" command, where "xyz" was "~"): circa> ls -al RColorBrewer/R total 140 -rwxr-xr-x 1 ray ecs 43988 Apr 17 2005 ColorBrewer.R~* -rw-r--r-- 1 ray ecs 0 Jul 29 09:57 residuals.MCMCglmm.R?xyz
Ray clarified to me that this filename was "residuals.MCMCglmm.R" preceded by 3 spaces and followed by a carriage return and "xyz".
drwxr-xr-x 2 ray ecs 4096 Jul 29 12:02 ./ drwxr-xr-x 5 ray ecs 4096 Jul 29 11:49 ../ -rwxr-xr-x 1 ray ecs 43988 Jul 29 09:57 ColorBrewer.R* -rwxr-xr-x 1 ray ecs 43988 Apr 17 2005 ColorBrewer.R~* -rw-r--r-- 1 ray ecs 0 Jul 29 09:58 residuals.MCMCglmm.R circa> circa> circa> R CMD build RColorBrewer * checking for file 'RColorBrewer/DESCRIPTION' ... OK * preparing 'RColorBrewer': * checking DESCRIPTION meta-information ... OK * checking whether 'INDEX' is up-to-date ... NO * use '--force' to overwrite the existing 'INDEX' * removing junk files unlink RColorBrewer/R/ ColorBrewer.R~ unlink RColorBrewer/R/ColorBrewer.R unlink RColorBrewer/R/ residuals.MCMCglmm.R xyz
That certainly looks bad. I can't reproduce it on Windows; it doesn't allow that filename. So I'll have to leave this for a Unix-alike user.
I have been following this from the sideline, because I suck really bad
when it comes to Perl programming. However....
I'm seeing this stuff in the build script:
## Remove exclude files.
open(EXCLUDE, "< $exclude");
while(<EXCLUDE>) {
rmtree(glob($_));
}
close(EXCLUDE);
Now this comes after
find(\&find_exclude_files, "$pkgname");
which AFAICT prints a number of file names into EXCLUDE. Now if one of
those file names contain a wildcard, I conjecture that the glob() can
make weird things happen. I don't think we want to glob there, do we?
Another issue is that EXCLUDE seems unprotected against file names with
embedded newlines. Something like find's -print0 would be handy...
Duncan Murdoch
unlink RColorBrewer/R/residuals.MCMCglmm.R unlink RColorBrewer/R/ColorBrewer.R~ rmdir RColorBrewer/R * checking for LF line-endings in source and make files * checking for empty or unneeded directories * building 'RColorBrewer_1.0-3.tar.gz' circa> Ray Brownrigg On Thu, 29 Jul 2010, Ray Brownrigg wrote:
On Thu, 29 Jul 2010, Duncan Murdoch wrote:
On 28/07/2010 10:01 AM, Jarrod Hadfield wrote:
Hi Marc, Thanks for the info on recovery - most of it can pieced together from backups but a quick, cheap and easy method of recovery would have been nicer. My main concern is that this could happen again and that the "bug" is not limited to R 2.9. I would think that an accidental carriage return at the end of a file name (even a temporary one) would be a reasonably common phenomenon (I'm surprised I hadn't done it before).
If you can put together a recipe to reproduce the problem (or a less extreme version of R deleting files it shouldn't), we'll certainly fix it. But so far all we've got are guesses about what might have gone wrong, and I don't think anyone has been able to reproduce the problem on current R.
Duncan: It looks to me like it has already been fixed, if indeed that was the problem. In R-2.10.1, I tried to reproduce the problem (using RColorBrewer, since that was the smallest package I have a local copy of), and the build produced this: * removing junk files * excluding invalid files from 'RColorBrewer' Subdirectory 'R' contains invalid file names: residuals.MCMCglmm.R xyz where the space shown between the "R" and the "xyz" was a newline character. [I didn't dare try using a "~" :-)] Ray Brownrigg
Duncan Murdoch
Cheers, Jarrod On 28 Jul 2010, at 14:04, Marc Schwartz wrote:
Jarrod,
Noting your exchange with Martin, Martin brings up a point that
certainly I missed, which is that somehow the tilde ('~') character
got into the chain of events. As Martin noted, on Linuxen/Unixen
(including OSX), the tilde, when used in the context of file name
globbing, refers to your home directory. Thus, a command such as:
ls ~
will list the files in your home directory. Similarly:
rm ~
will remove the files there as well. If the -rf argument is added,
then the deletion becomes recursive through that directory tree,
which appears to be the case here.
I am unclear, as Martin appears to be, as to the steps that caused
this to happen. That may yet be related in some fashion to Duncan's
hypothesis.
That being said, the use of the tilde character as a suffix to
denote that a file is a backup version, is not limited to Fedora or
Linux, for that matter. It is quite common for many text editors
(eg. Emacs) to use this. As a result, it is also common for many
applications to ignore files that have a tilde suffix.
Based upon your follow up posts to the original thread, it would
seem that you do not have any backups. The default ext3 file system
that is used on modern Linuxen, by design, makes it a bit more
difficult to recover deleted files. This is due to the unlinking of
file metadata at the file system data structure level, as opposed to
simply marking the file as deleted in the directory structures, as
happens on Windows.
There is a utility called ext3undel
(http://projects.izzysoft.de/trac/ext3undel ), which is a wrapper of
sorts to other undelete utilities such as PhotoRec and foremost. I
have not used it/them, so cannot speak from personal experience. Thus
it would be a good idea to engage in some reviews of the
documentation and perhaps other online resources before proceeding.
The other
consideration is the Catch-22 of not copying anything new to your
existing HD, for fear of overwriting the lost files with new data. So
you would need to consider an approach of downloading these utilities
via another computer and then running them on the computer in
question from other media, such as a CD/DVD or USB HD.
A more expensive option would be to use a professional data recovery
service, where you would have to consider the cost of recovery
versus your lost time. One option would be Kroll OnTrack UK
(http://www.ontrackdatarecovery.co.uk/ ). I happen to live about a
quarter mile from their world HQ here in a suburb of Minneapolis. I
have not used them myself, but others that I know have, with good
success. Again, this comes at a
potentially substantial monetary cost.
The key is that if you have any hope to recover the deleted files,
you not copy anything new onto the hard drive in the mean time.
Doing so will decrease the possibility of file recovery to near 0.
As Duncan noted, there is great empathy with your situation. We have
all gone through this at one time or another. In my case, it was
perhaps 20+ years ago, but as a result, I am quite anal retentive
about having backups, which I have done for some time on my systems,
hourly.
HTH,
Marc Schwartz
On Jul 28, 2010, at 5:55 AM, Jarrod Hadfield wrote:
Hi Martin, I think this is the most likely reason given that the name in the DESCRIPTION file does NOT have a version number. Even so, it is very easy to misname a file and then delete it/change its name (as I've done here) and I hope current versions of R would not cause this problem. Perhaps Fedora should not use ~ as its back up file suffixes? Cheers, Jarrod On 28 Jul 2010, at 11:41, Martin Maechler wrote:
Jarrod Hadfield <j.hadfield at ed.ac.uk> on Tue, 27 Jul 2010 21:37:09 +0100 writes:
Hi, I ran R (version 2.9.0) CMD build under root in Fedora (9). When it tried to remove "junk files" it removed EVERYTHING in my local account! (See below). Can anyone tell me what happened,
the culprit may lay here:
* removing junk files unlink MCMCglmm_2.05/R/ residuals.MCMCglmm.R ~
where it seems that someone (you?) have added a newline in the filname, so instead of 'residuals.MCMCglmm.R~' you got 'residuals.MCMCglmm.R ~' and the unlink / rm command interpreted '~' as your home directory. But I can hardly believe it. This seems explanation seems a bit doubtful to me.. ...
and even more importantly if I can I restore what was lost.
well, you just get it from the backup. You do daily backups, do you? Regards, Martin Maechler, ETH Zurich
______________________________________________ R-help at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Peter Dalgaard Center for Statistics, Copenhagen Business School Phone: (+45)38153501 Email: pd.mes at cbs.dk Priv: PDalgd at gmail.com