Error: C stack usage is too close to the limit when using list.files()

Sat, Sep 28, 2013 5:06 PM

The issue is not symbolic links per se, but ones that form loops.
Note that you can detect such loops by running 'find -L ...' and 
looking for the error messages.  (find by default does not follow
any symbolic links, which can be a problem also.)

It is a shortcoming of the current version of list.files().

Bill Dunlap
Spotfire, TIBCO Software
wdunlap tibco.com

-----Original Message-----
From: jgrn307 at gmail.com [mailto:jgrn307 at gmail.com] On Behalf Of Jonathan
Greenberg
Sent: Saturday, September 28, 2013 10:51 AM
To: William Dunlap
Cc: r-help
Subject: Re: [R] Error: C stack usage is too close to the limit when using list.files()

Thanks all -- ok, so the symbolic link issue is a distinct
possibility, but fundamentally doesn't solve the issue since most
users will have symbolic links on their machines SOMEPLACE, so a full
drive scan will run into these issues --  is list.files calling find,
or is it using a different algorithm?  This seems like a shortcoming
in the list.files algorithm -- is there a better solution (short of a
System call, which I'm still not sure will work on Macs without Xcode
-- a colleague of mine did NOT have Xcode, and reported not being able
to run find from the command line) -- perhaps a different package?

--j

On Fri, Sep 27, 2013 at 3:08 PM, William Dunlap <wdunlap at tibco.com> wrote:

Toss a couple of extra files in there and you will see the output grow exponentially.

% touch dir/IMPORTANT_1 dir/subdir/IMPORTANT_2

and in R those two new files cause 82 more strings to appear in list.file's output:

nchar(list.files("dir", recursive=TRUE))

 [1]  11  18  33  40  55  62  77  84  99 106 121 128 143 150 165 172 187 194 209
[20] 216 231 238 253 260 275 282 297 304 319 326 341 348 363 370 385 392 407 414
[39] 429 436 451 458 473 480 495 502 517 524 539 546 561 568 583 590 605 612 627
[58] 634 649 656 671 678 693 700 715 722 737 744 759 766 781 788 803 810 825 832
[77] 847 854 869 876 891 898 901

'find', by default, does not following symbolic links.

% find dir
dir
dir/subdir
dir/subdir/IMPORTANT_2
dir/subdir/linkToUpperDir
dir/IMPORTANT_1

The -L option makes it follow them, but it won't follow loops:

% find -L dir
dir
dir/subdir
dir/subdir/IMPORTANT_2
find: File system loop detected; `dir/subdir/linkToUpperDir' is part of the same file

system loop as `dir'.

dir/IMPORTANT_1

Bill Dunlap
Spotfire, TIBCO Software
wdunlap tibco.com

-----Original Message-----
From: r-help-bounces at r-project.org [mailto:r-help-bounces at r-project.org] On

Behalf

Of William Dunlap
Sent: Friday, September 27, 2013 12:56 PM
To: Jonathan Greenberg; r-help
Subject: Re: [R] Error: C stack usage is too close to the limit when using list.files()

Do you have some symbolic links that make loops in your file system?
list.files() has problems with such loops and find does not.  E.g.,  on a Linux box:

% cd /tmp
% mkdir dir dir/subdir
% cd dir/subdir
% ln -s ../../dir linkToUpperDir
% cd /tmp
% R --quiet

list.files("dir", recursive=TRUE, full=TRUE)

[1]

"dir/subdir/linkToUpperDir/subdir/linkToUpperDir/subdir/linkToUpperDir/subdir/linkToU

pperDir/subdir/linkToUpperDir/subdir/linkToUpperDir/subdir/linkToUpperDir/subdir/linkT

oUpperDir/subdir/linkToUpperDir/subdir/linkToUpperDir/subdir/linkToUpperDir/subdir/li

nkToUpperDir/subdir/linkToUpperDir/subdir/linkToUpperDir/subdir/linkToUpperDir/subdi

r/linkToUpperDir/subdir/linkToUpperDir/subdir/linkToUpperDir/subdir/linkToUpperDir/su

bdir/linkToUpperDir/subdir/linkToUpperDir/subdir/linkToUpperDir/subdir/linkToUpperDir

/subdir/linkToUpperDir/subdir/linkToUpperDir/subdir/linkToUpperDir/subdir/linkToUpper

Dir/subdir/linkToUpperDir/subdir/linkToUpperDir/subdir/linkToUpperDir/subdir/linkToUp

perDir/subdir/linkToUpperDir/subdir/linkToUpperDir/subdir/linkToUpperDir/subdir/linkTo

UpperDir/subdir/linkToUpperDir/subdir/linkToUpperDir/subdir/linkToUpperDir/subdir/lin

kToUpperDir/subdir/linkToUpperDir/subdir/linkToUpperDir"

system("find dir")

dir
dir/subdir
dir/subdir/linkToUpperDir

Bill Dunlap
Spotfire, TIBCO Software
wdunlap tibco.com

-----Original Message-----
From: r-help-bounces at r-project.org [mailto:r-help-bounces at r-project.org] On

Behalf

Of Jonathan Greenberg
Sent: Friday, September 27, 2013 12:13 PM
To: r-help
Subject: [R] Error: C stack usage is too close to the limit when using list.files()

R-helpers:

I'm running a file search on my entire drive (Mac OS X) using:

files_found <-

list.files(dir="/",pattern=somepattern,recursive=TRUE,full.names=TRUE)

where somepattern is a search pattern (which I have confirmed via a
unix "find / -name somepattern" only returns ~ 3 results).

I keep getting an error:

Error: C stack usage is too close to the limit

when running this command.  Any ideas on 1) how to fix this or 2) if
there is an alternative to using list.files() to accomplish this
search without resorting to an external package?

Cheers!

--jonathan


--
Jonathan A. Greenberg, PhD
Assistant Professor
Global Environmental Analysis and Remote Sensing (GEARS) Laboratory
Department of Geography and Geographic Information Science
University of Illinois at Urbana-Champaign
259 Computing Applications Building, MC-150
605 East Springfield Avenue
Champaign, IL  61820-6371
Phone: 217-300-1924
http://www.geog.illinois.edu/~jgrn/
AIM: jgrn307, MSN: jgrn307 at hotmail.com, Gchat: jgrn307, Skype: jgrn3007

______________________________________________
R-help at r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

______________________________________________
R-help at r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

Error: C stack usage is too close to the limit when using list.files()

Thread (9 messages)