An embedded and charset-unspecified text was scrubbed... Name: not available URL: <https://stat.ethz.ch/pipermail/r-help/attachments/20130714/284aec8f/attachment.pl>
Does R ever stop responding without a message?
7 messages · L S, David Winsemius, jim holtman +1 more
On Jul 14, 2013, at 2:40 PM, L S wrote:
Hi, Have any of you ever encountered a situation where R stops processing an instruction but does not give a "not responding" message? The reason I ask is I am working in RStudio (Mac OS/X 10.7, 1.8 Ghz i7, 4 GB DDR3) and the instruction I entered in the command line pane is still being processed since last night around 7 PM. I expected it to take on the order of 8-12 hours to complete, but I'm nearing 24 hours with no progress, no messages, etc. The command is essentially splitting the a ppp (marked point pattern) into a tessellation/grid. There are about a couple million points in the pattern and each has a vector of four marks. That said, I processed a similar command with 700,000 points in the pattern (each point with a vector of two marks) yesterday and it took only around 3 hours. Is there anyway I can be certain the command is being processed?
Well, this answer only addresses whether the R program is "active": Use the Activity Monitor.app looking at teh CPU panel. The process name is R and, this being a Mac, you even get a micro-R-logo. At "idling speed" it will only display 1% or so. When it is active the cpu-%-age will "bounce around" near 100%.
I don't want to abort prematurely if I know it will go to completion. How long would you give it before you knew for certain it would not complete?
If you have pushed your process over the bounds of available RAM and it is now using "virtual memory", there is really no way to estimate the complettion time. I generally give up after about 15 or 20 minutes. This does mean that the best practice will have been to make sure you have saved you work before terminating the session. The R GUI is helpful in this regard because it automatically backs up open source documents at intervals. Unfortunately that is not also true for the history session, which is only updated at a normal termination of a session.
Any guidance you could offer would be much appreciated. Thanks, Lily [[alternative HTML version deleted]]
This is not the correct mailing list for this question, and you should also read what the Posting Guide says about not using HTML format. -- David Winsemius Alameda, CA, USA
An embedded and charset-unspecified text was scrubbed... Name: not available URL: <https://stat.ethz.ch/pipermail/r-help/attachments/20130714/4afb260e/attachment.pl>
An embedded and charset-unspecified text was scrubbed... Name: not available URL: <https://stat.ethz.ch/pipermail/r-help/attachments/20130714/bc9bd03f/attachment.pl>
On Jul 14, 2013, at 8:24 PM, L S wrote:
Thanks Jim and David for your helpful feedback. I still have not terminated RStudio (and it still has not gone to completion). A few observations I forgot to mention is that the red "stop" icon is showing in RStudio so I am unable to enter any new commands. Also there is a blinking cursor under the "> command" I ran with no > before the cursor.
I thought I mentioned that these questions are off-topic on Rhelp. They should be directed to a different mailing list, either the Mac-SIG-mailing list or the RStudio list.
David--Looking at the activity monitor, RStudio is using anywhere from 0.4-1% of CPU while another process called rsession is using anywhere from 17-23% CPU. Does this mean it has given up processing the R command?
No. In fact the opposite, but you obviously have a great deal more patience than I do. I would long ago have shutdown R (using the force-quit functions from the "Apple menu" and then restarted.) I generally notice around 100% in the % "CPU column" in the table whin R is active, although the scrolling "CPU Usage" graphic at teh bottom of thepage will max out at 25% because the R process is not distributed across all 4 cores.
If so, is there anything I can do to ensure this won't happen next time I try to run the command with my data?
If you do not tell us exactly what the command was _and_ the size and structure of the data involved, it is unlikely that we can tell you much of anything.
(Also, how can you tell if the process is using virtual memory?)
One of the columns in the CPU activity monitor is labeled "Virtual Mem". If it is being used the number will change.
Jim--What I'm running now is just a single instruction using the split function, so I would not be able to put in the progress messages. But, that's a great idea! I will keep that in mind if I run into problems with scripts in the future. Thank you. On Sun, Jul 14, 2013 at 6:46 PM, jim holtman <jholtman at gmail.com> wrote: If you are writing a script that you know will take a long time to process, "pepper" it with "progress" reports so you know what part of the script it is in and when it is going around loop. On some of my long scripts, I will print out a message every n'th time through the loop so that I know if it making progress. I position them so that I get out a message every minute or so. Also make sure the GUI is not buffered, or better, follow each status with a 'flush.console()' to put the message out to the screen. This will give you a feeling of what progress you are making. On Sun, Jul 14, 2013 at 5:40 PM, L S <losedaghat at gmail.com> wrote: Hi, Have any of you ever encountered a situation where R stops processing an instruction but does not give a "not responding" message? The reason I ask is I am working in RStudio (Mac OS/X 10.7, 1.8 Ghz i7, 4 GB DDR3) and the instruction I entered in the command line pane is still being processed since last night around 7 PM. I expected it to take on the order of 8-12 hours to complete, but I'm nearing 24 hours with no progress, no messages, etc. The command is essentially splitting the a ppp (marked point pattern) into a tessellation/grid. There are about a couple million points in the pattern and each has a vector of four marks. That said, I processed a similar command with 700,000 points in the pattern (each point with a vector of two marks) yesterday and it took only around 3 hours. Is there anyway I can be certain the command is being processed? I don't want to abort prematurely if I know it will go to completion. How long would you give it before you knew for certain it would not complete? Any guidance you could offer would be much appreciated. Thanks, Lily
David Winsemius Alameda, CA, USA
An embedded and charset-unspecified text was scrubbed... Name: not available URL: <https://stat.ethz.ch/pipermail/r-help/attachments/20130715/149698ed/attachment.pl>
It is a (very) large marked point process object that is being split (see original post), and that is some other kind of structure. So I think all bets are off, in terms of what kind of duration to expect. The folks on r-sig-geo might be able to help with suggestions for faster approaches. In terms of monitoring the process, I have a couple of suggestions. The first is to turn on profiling (see ?Rprof). Unless things changed in R 3.x.x, Rprof writes to a file as it profiles, and one could watch the file grow to determine that processing is still taking place. Admittedly, it's a cheap trick, and runs the risk of slowing the process, especially if the profile file gets big. The other is to do the split manually with a loop. This way, one could insert cat() statements to track the progress. This is probably what I would do first. Indeed, one could also write timestamps and track the rate at which splitting is taking place (i.e., is it getting slower and slower). -Don
Don MacQueen Lawrence Livermore National Laboratory 7000 East Ave., L-627 Livermore, CA 94550 925-423-1062 On 7/15/13 2:50 AM, "Jim Holtman" <jholtman at gmail.com> wrote: >a 'split' function should not take hours to complete. I will not even >wait a couple of minutes for one until I look at other approaches. Can >you at least show what the command looks like and what the structure of >the data is. Are you trying to split a dataframe on multiple criteria? >I tend to use 'data.table' for some of these operation, or split the >indices of a dataframe instead of the data. My own criteria is that if a >single base function call is taking more than a minute, I see if there >are options. > >This is also a case of "tell me what you want to do, not how you want to >do it". > >Sent from my iPad > >On Jul 14, 2013, at 23:24, L S <losedaghat at gmail.com> wrote: > >> Thanks Jim and David for your helpful feedback. I still have not >>terminated RStudio (and it still has not gone to completion). A few >>observations I forgot to mention is that the red "stop" icon is showing >>in RStudio so I am unable to enter any new commands. Also there is a >>blinking cursor under the "> command" I ran with no > before the cursor. >> >> David--Looking at the activity monitor, RStudio is using anywhere from >>0.4-1% of CPU while another process called rsession is using anywhere >>from 17-23% CPU. Does this mean it has given up processing the R >>command? If so, is there anything I can do to ensure this won't happen >>next time I try to run the command with my data? >> >> (Also, how can you tell if the process is using virtual memory?) >> >> Jim--What I'm running now is just a single instruction using the split >>function, so I would not be able to put in the progress messages. But, >>that's a great idea! I will keep that in mind if I run into problems >>with scripts in the future. >> >> Thank you. >> >> >> On Sun, Jul 14, 2013 at 6:46 PM, jim holtman <jholtman at gmail.com> wrote: >> If you are writing a script that you know will take a long time to >>process, "pepper" it with "progress" reports so you know what part of >>the script it is in and when it is going around loop. On some of my >>long scripts, I will print out a message every n'th time through the >>loop so that I know if it making progress. I position them so that I >>get out a message every minute or so. Also make sure the GUI is not >>buffered, or better, follow each status with a 'flush.console()' to put >>the message out to the screen. This will give you a feeling of what >>progress you are making. >> >> >> On Sun, Jul 14, 2013 at 5:40 PM, L S <losedaghat at gmail.com> wrote: >> Hi, >> >> Have any of you ever encountered a situation where R stops processing an >> instruction but does not give a "not responding" message? >> >> The reason I ask is I am working in RStudio (Mac OS/X 10.7, 1.8 Ghz i7, >>4 >> GB DDR3) and the instruction I entered in the command line pane is still >> being processed since last night around 7 PM. I expected it to take on >>the >> order of 8-12 hours to complete, but I'm nearing 24 hours with no >>progress, >> no messages, etc. >> >> The command is essentially splitting the a ppp (marked point pattern) >>into >> a tessellation/grid. There are about a couple million points in the >> pattern and each has a vector of four marks. That said, I processed a >> similar command with 700,000 points in the pattern (each point with a >> vector of two marks) yesterday and it took only around 3 hours. >> >> Is there anyway I can be certain the command is being processed? I >>don't >> want to abort prematurely if I know it will go to completion. How long >> would you give it before you knew for certain it would not complete? >> >> Any guidance you could offer would be much appreciated. >> >> Thanks, >> Lily >> >> [[alternative HTML version deleted]] >> >> ______________________________________________ >> R-help at r-project.org mailing list >> https://stat.ethz.ch/mailman/listinfo/r-help >> PLEASE do read the posting guide >>http://www.R-project.org/posting-guide.html >> and provide commented, minimal, self-contained, reproducible code. >> >> >> >> -- >> Jim Holtman >> Data Munger Guru >> >> What is the problem that you are trying to solve? >> Tell me what you want to do, not how you want to do it. >> > > [[alternative HTML version deleted]] > >______________________________________________ >R-help at r-project.org mailing list >https://stat.ethz.ch/mailman/listinfo/r-help >PLEASE do read the posting guide >http://www.R-project.org/posting-guide.html >and provide commented, minimal, self-contained, reproducible code.