The problem is that the device chosen by dev.new() depends on the GUI.
You can see the code that does this in grDevices:::.onLoad. So in fact
with noRstudioGD=TRUE, the decision is identical to what it is in R:
you only get X11 if your GUI is X11 or Tk, you get pdf otherwise.
It's pretty common to use R on a machine where X11 won't work, so this
makes sense.
But not if running from RStudio always causes the X11 check to fail. If the function is used outside RStudio, there's little point in specifying noRStudioGD=TRUE in the first place.
If you consider the code used to determine whether quartz() or X11() is available
dsp <- Sys.getenv("DISPLAY")
if (.Platform$OS.type == "windows")
windows
else if (.Platform$GUI == "AQUA" || ((!nzchar(dsp) || grepl("^/tmp/launch-", dsp)) && .Call(C_makeQuartzDefault)))
quartz
else if (nzchar(dsp) && .Platform$GUI %in% c("X11", "Tk"))
X11
else defdev
you can see that it checks for a DISPLAY variable and assumes that X11 can be used if it is set. Wouldn't it be just as safe to add RStudio to the list of accepted .Platform$GUIs?
In my case (Mac OS X 10.10.5), I'd like to get a quartz device. The problem here is that I have XQuartz installed, so DISPLAY is always set and looks like this
/private/tmp/com.apple.launchd.2wKas4wzPe/org.macosforge.xquartz:0
while dev.new() checks for ^/tmp/launch-; I suppose the DISPLAY variable has changed between different versions of Mac OS X or XQuartz. Perhaps the additional patterns could just be added to the grepl() call?