cerr and cout not working calling c++ from R
Just a short follow up to the list.
It turns out that the strange behavior I was experiencing had nothing to do
with cout and cerr, but ostreams in general. For some reason, at a minimum,
the operators:
ostream& operator<< (long& val );
ostream& operator<< (double& val );
were not functioning properly and were causing my problem.
To get around this, I simply had to convert types like these to strings
before passing them into the ostreams with << operators.
I don't know if this was due to a mistake in which libraries I linked to or
which headers are being read when I make my project. However, it seems that
I have ruled out the possibility of this being unique to 64-bit longs as the
doubles are standard types on any machine. Anyway, I appreciate everyone's
help, and if anyone has trouble with this type of thing in the future, a
simple couple of functions I used to convert the long or double values to
strings were:
string getStringFromDouble(double d)
{
char*doubleValue = (char*)calloc(1000,sizeof(char));
sprintf(doubleValue,"%f",d);
string returnValue(doubleValue);
free(doubleValue);
return(returnValue);
}
string getStringFromLong(long i)
{
char*intValue = (char*)calloc(1000,sizeof(char));
sprintf(intValue,"%ld",i);
string returnValue(intValue);
free(intValue);
return(returnValue);
}
I'm not sure why this was necessary, but it solves my issue.
Thanks again for all the help--I learned a lot about streams and stream
buffers in the process.
Thanks,
Sean
On 5/8/11 12:56 AM, "Sean Robert McGuffee" <sean.mcguffee at gmail.com> wrote:
Thanks, Simon, That link may prove to be very useful. I think what I can do is make a similar version of a buffer to one in that example and then have it conditionally write to Rprintf or cout and/or REprintf or cerr, depending on whether I'm running a c++ program from the command line or launching a function from R. Then I can just replace all of my cout and cerr terms with this new buffer or two in global space. Before I call those functions in R, I'll set them to write to R before I launch any code that uses it. This should allow me to leave all my code exactly how it is with a minor replacement of one or two terms for another. So, if the errors in my performance are only caused by writing to cout and/or cerr from R, then they can be completely avoided by all of those substitutions writing to Rprintf and REprintf. This seems like it is going to work exactly as I'm hoping. Thanks again, Sean On Sat, May 7, 2011 at 6:13 PM, Simon Urbanek <simon.urbanek at r-project.org> wrote:
Sean, just to clarify - this is not about something "not working" or any leaks. cout/cerr do exactly what they are supposed to, it's just in most cases not what you want. The main reason is that R does not necessarily use stdin/stdout so cout/cerr may have nothing to do with the R console output. The effects of using cout/cerr vary by the actual setup in which you run R as I was explaining earlier. In the most complex setup (default for interactive R on OS X, actually) you have: 1) stdin/stdout/stderr - those are entirely independent of the R console 2) ReadConsole/WriteConsole - this is how R interacts with the console 3) C++ cin/cout/cerr - by default a buffered link to stdin/out/err So as you see the reason for cout being strongly discouraged is simply because it has no relationship with R and produces no output in R itself. What makes things seemingly puzzling is that some UI implementations in R use stdin/out/err in Read/WriteConsole, so suddenly you are dealing with three independent streams and buffers (e.g., cout, WriteCondole and stdout all eventually ending in the same place but at different times). If your final output is a tty then they all get synchronized at the end into the same tty, but since each of them has a buffer, the result is random at the best. Due to the interactions, you must be careful not to mess things up with any additional low-level stream manipulations. For example, the R.app GUI doesn't use stdout/err for console at all, since it is driven by text views instead, so it uses exclusively Read/WriteConsole (and so does the Windows GUI). However, it also monitors stdout/err for external output (e.g. from system()) and uses two separate pipes to merge that output into the console with different colors. So if you were to start to mess with the descriptors for stdout/err you could break that setup. That said, if you want a solution for C++ you should simply write a stream or buffer that uses Rprintf/REprintf for output. I was assuming that others have done that before, but Dirk's response suggests that it is not as common. If you choose to do it, you may find this link helpful: http://groups.google.com/group/comp.lang.c++/msg/1d941c0f26ea0d81 Cheers, Simon On May 7, 2011, at 12:42 AM, Sean Robert McGuffee wrote:
This is a good idea as to converting to stringstream's because it could let me remove cout and cerr from the equation. At the moment, I get that feeling from the experts that having cout and cerr is causing a bug by not mixing with R. I have other cases where algorithms that avoid cout and cerr aren't working, but as far as my experience with bugs in c code goes, a memory issue in one place can make everything go haywire, and an random cout or cerr somewhere that I don't catch could maybe be the source of the issue. I also like this idea of piping and then calling Rprintf. Along the same lines, I could create a string and then either spit to cout or cerr or spit to Rprintf depending on some sort of setting. One of the key reasons I have been hanging on to cout and cerr is that I want to keep all of my c++ code identical to the status quo as far as the command line interface is concerned. On the other hand, maybe I can just globally substitute cout and cerr to random file streams and avoid the conflict with stdout and stderr. I take it that it isn't ostreams per say causing the problem but that it's something specific about cout and cerr not mixing with R. Then maybe I can pass their buffers to cout, cerr, or Rprintf, depending on preferences later. What I would really like to do maybe be beyond my grasp, but that would be to integrate ostreams into R. I am toying with an idea of mimicking cout and cerr with some new classes that are almost verbatim copies of cout and cerr with substituted names. Then maybe inside their internals I can more easily create conditional access to Rprintf calls. What I like about this strategy is that I could do a global substitution of cout and cerr with new objects of more useful functionality, and that way I might not have to re-write all my c++ code which constantly calls on these standard streams. Plus, if I get something generic to work, maybe I can pass it on to Dirk so others won't have to deal with this stuff in the future. I find this fascinating that the output may be tty. I see now that the device on my Mac that /dev/stdout points to is a tty. I have never encountered tty before, but I have a vague memory of a term like that back in the day of dial up! Oh the fun things I'm learning. Just to clarify though, is the only way my code "mixes IO systems between R and C++" by way of cout and cerr happening to be synchronized with stderr and stdout by default? Or is there something inherent in ostreams in general that doesn't mesh with R? I guess I find it a little bit surprising that calling ios_base::sync_with_stdio(false) and redirecting those streams to other files doesn't dodge that bullet. I know that doesn't break the ostreams themselves because it works fine from the command line interface. I guess I'll have to delve more into how c++ works behind the scenes to figure that part out. I think Dirk has a really good suggestion to comment out all of my cout and cerr calls and see if still get a bug. Obviously that wouldn't be any permanent fix for future users who want to link code with cout and cerr, but it would help me identify if this is the only source of error. It might take me till past the weekend to get to the bottom of some of these things, but I'll post what I find out. Thanks again--I appreciate everyone's useful suggestions. Sean On Fri, May 6, 2011 at 10:44 PM, Dirk Eddelbuettel <edd at debian.org> wrote:
On 6 May 2011 at 22:09, Simon Urbanek wrote:
| Sean,
|
| the path form the console is not under your control and it depends
heavily on the internal settings in R (is console enabled, is R
interactive, is the output a tty or a file ...) - that's why you have to
use Rprint/REprintf - that's the only path that is guaranteed to work.
Anything else is settings-specific and not guaranteed to work (it may
appear to work in limited settings, but will not work in others). Note
that, for example, WriteConsole is only used if R is not fed via I/O pipes
- that may be what baffled you in that case.
|
| So AFAICS the only C++ solution (other than using Rprintf/REprintf
directly) is to create streams to pipes whose ends will call
Rprintf/REprintf with the content. Anything else is not general. You may
want to ask Rcpp people (Dirk is CC'd anyway ;)) - I'd imagine they must
have thought of it as it seems very natural to request in a C++ interface
... (but I don't use Rcpp so I don't know).
Rcpp makes no claims about reimplementing IO, or providing new IO
facilities.
Rcpp is clearly bound by the R API upon which it builds; the R
documentation
(which I quoted in my earlier email in this thread) suggests to expect
nasty
things when one mixes IO systems between R and C++.
And that seems to be what Sean is experiencing; I for one am not surprised.
It was never clear from Seans's mails exactly how much time he has had for
reading of the available documentation, his approach seems to me mostly
driven by a vast and vague set of assumption on his part (eg the rout/rerr
business he comes back to), as well as a lot energy and a willingness to
just
try this. ?Which is perfectly fine, but it can also fail.
I'd suggest to start small with as little IO as possible. Maybe even
comment
all IO out as much as possible. Or redo it as strstream and feed that to
Rprintf.
Dirk
| Cheers,
| Simon
|
|
|
| On May 6, 2011, at 3:19 PM, Sean Robert McGuffee wrote:
|
| > Hi,
| >
| > Thanks for the comments so far.
| >
| > I've been going through the code extensively and seem to be having
trouble
| > reproducing what R is doing, so this confuses me. For example, when I
write
| > out the pointer values for ptr_R_WriteConsole type functions, I can't
find a
| > function that matches what is actually being called.
| >
| > REprintf("Rstd_WriteConsole=%ld, ptr_R_WriteConsole=%ld,
| > R_WriteConsole=%ld\n", Rstd_WriteConsole, ptr_R_WriteConsole,
| > R_WriteConsole);
| >
| > It might be something like RTcl_WriteConsole too. However, I can't seem
to
| > find the headers I would need to try to find it's pointer value.
Anyway,
| > these are often hidden/protected/static which is an aspect of code that
I
| > only understand in concept. The concept I understand about this type of
| > stuff is that if you provide a better way of doing things, then maybe
you
| > hide some stuff that doesn't involve the better way. However, in this
case,
| > the hidden stuff is just making it difficult for me to debug and get to
| > bottom of what is going on. Anyway, I get different results if I call
on the
| > functions that simply use stdout when trying to reproduce Rprintf
| > functionality. I'm only doing that to try to get to the bottom of what
R is
| > doing with stdout and stderr. I also am having a similar issue with
trying
| > to understand what GNU GCC is doing with how it sync's cout and cerr
with
| > stdout and stderr. Maybe R is intercepting some of these components.
Anyway,
| >
| > I was hoping to get around the
| > "5.6 Interfacing C++ code
| > ? ======================== ? [...]
| > ? ? ?Using C++ iostreams, as in this example, is best avoided. ?There
is
| > ? no guarantee that the output will appear in the R console, and indeed
it
| > ? will not on the R for Windows console. ?Use R code or the C entry
points
| > ? (*note Printing::) for all I/O if at all possible."
| > if at all possible. My reason is that I have 76386 lines of c++ code
that
| > I'm trying to interface with R, and it literally might be faster for me
to
| > update R rather than modify all of that code to suit R which seems to
be
| > lacking a c++ ostream interface.
| >
| > Ideally, I'd write rout and rerr versions of cout and cerr for R to
become
| > more compatible with c++. I have had some trouble figuring a lot of
this
| > hidden stuff out. Strategies I have considered are the following:
| > 1) Write macros that conditionally replace cerr and cout with Rprintf
| > somehow throughout all of my c++ code that I'm trying to interface with
R.
| > 2) Recreate the std::cerr and std::cout type of classes but maybe add
some
| > extra functionality for callbacks to when i/o events take place to
somehow
| > try to fix things.
| > 3) Make a scratch version of R source code that has no hidden aspects
to try
| > to investigate what is going on a little better.
| >
| > Also, I should mention that I found an unanswered post from someone who
had
| > a similar problem when trying to use cerr and cout when accessing
functions
| > from dynamically linked libraries. I wonder, is it possible that R
hasn't
| > completely connected to these libraries? I mean, it's doing some sort
of
| > magic where it finds a pointer to a function, but maybe c++ has a lot
going
| > on in the background related to these ostream types that could be
getting
| > messed up. Maybe some aspects of the global environment might be
different
| > somehow? I directly link to my libraries via a c++ compiler when I get
the
| > identical functions to work from main in c++.
| >
| > One final thing I'm considering is that the R manual also mentions
something
| > about garbage collection and some circumstances under which it's
important
| > to explicitly request pointers to be maintained. Is it possible that R
is
| > eating global pointers used by std::ostream ?
| >
| > All suggestions are valued and if anyone has recommendations on
strategy,
| > please let me know.
| >
| > Thanks,
| > Sean
| >
| >
| > On 5/6/11 2:50 PM, "Davor Cubranic" <cubranic at stat.ubc.ca> wrote:
| >
| >> On 2011-05-06, at 11:41 AM, Dirk Eddelbuettel wrote:
| >>
| >>> | I?m trying to call some of my c++ code from R and seem to be having
an
| >>> issue
| >>> | with streams, although that?s just one obvious sign of something
different
| >>> | in performance between calling the same function from main in c++
vs.
| >>> | calling the same function from R. [...]
| >>> In a nutshell, R 'owns' your console. That is part of the Faustian
bargain.
| >>
| >> But isn't he redirecting cout to a file, so the C++ output is not to
the
| >> console?
| >>
| >> Davor
| >
| >
| > On 5/6/11 2:41 PM, "Dirk Eddelbuettel" <edd at debian.org> wrote:
| >
| >>
| >> On 6 May 2011 at 14:21, Sean Robert McGuffee wrote:
| >> | Hi,
| >> |
| >> | Sorry, I just tried posting this but I had it in the wrong format of
text,
| >> | so this is a cleared format repost:
| >> |
| >> | I?m trying to call some of my c++ code from R and seem to be having
an issue
| >> | with streams, although that?s just one obvious sign of something
different
| >> | in performance between calling the same function from main in c++
vs.
| >> | calling the same function from R. I?m not getting any signs of
errors from
| >> | R, but the behavior is different somehow and in a way that breaks my
| >> | algorithms. In both cases of launching a function from c++ or R, I
redirect
| >> | cout to a file using a streambuf. In particular, I can see progress
stop
| >> | when launching from R at a point when cout gets truncated when
starting to
| >> | write out iterator values in a std::vector<string>. Then, I find no
way to
| >> | write anything else to cout. When I write those values to Rprintf,
they are
| >> | there and they are as expected. I?ve tried setting
| >> | ios_base::sync_with_stdio(false) in case R is messing with stdout
and stderr
| >> | somehow, but that made no difference. I?m not quite sure how R uses
those C
| >> | FILE* streams. Anyway, I am wondering if anyone has a hunch as to
what might
| >> | be going on. At this point I don?t know if it might be an issue with
R not
| >> | completely handling c++ from a library or if it is an issue with
cout in
| >> | particular or if there is something else I?m missing. Please let me
know if
| >> | you have any suggestions for me.
| >>
| >> Doug Bates sometimes remarks that, once all other alternatives are
exhausted,
| >> one could consider reading the manual. And indeed, 'Writing R
Extensions' has
| >> this to say:
| >>
| >> ? 5.6 Interfacing C++ code
| >> ? ========================
| >>
| >> ? [...]
| >>
| >> ? ? ?Using C++ iostreams, as in this example, is best avoided. ?There
is
| >> ? no guarantee that the output will appear in the R console, and
indeed it
| >> ? will not on the R for Windows console. ?Use R code or the C entry
points
| >> ? (*note Printing::) for all I/O if at all possible.
| >>
| >> In a nutshell, R 'owns' your console. That is part of the Faustian
bargain.
| >>
| >> Dirk
| >>
| >> --
| >> Gauss once played himself in a zero-sum game and won $50.
| >> ? ? ? ? ? ? ? ? ? ? ?-- #11 at http://www.gaussfacts.com
| >
| > ______________________________________________
| > R-devel at r-project.org mailing list
| > https://stat.ethz.ch/mailman/listinfo/r-devel
| >
| >
|
--
Gauss once played himself in a zero-sum game and won $50.
? ? ? ? ? ? ? ? ? ? ?-- #11 at http://www.gaussfacts.com